Automating Production with WebWorks Automap

By Simon Bate, guest speaker at the February local FrameMaker SIG meeting

The marketing materials for WebWorks AutoMap promise that AutoMap makes automated production of documentation a breeze. In many ways, the product delivers on this promise. Here at Symantec, AutoMap is currently producing 37,000 pages of documentation a night for more than 240 documents (21,000 pages in PDF and 16,000 pages in HTML). AutoMap isn’t even breaking a sweat.

However, Quadralay omits many things from its marketing materials and product documentation that you need to know before setting up a successful AutoMap implementation. This article:

• Gives an overview of the production needs at the Symantec Data Center Management Group (DCMG)
• Describes the overall workings of AutoMap
• Describes the two ways of using AutoMap
• Describes how I got AutoMap to do what we needed at Symantec

My experience is with using AutoMap on Windows systems. Although AutoMap is available on UNIX systems, this article does not address UNIX issues specifically.

Build Automation at Symantec DCMG

When I was hired at Veritas (now Symantec), my initial responsibilities included building the documents for the CommandCentral Storage and CommandCentral Service products.

At the time, the documents included nine books and five help systems, comprising about 4,000 pages. The FrameMaker sources were stored in Visual SourceSafe; the resulting PDFs and Help Zips were archived in CVS.

I built the docs once a week. The build had to be at night because I had to check the source files and didn’t want to compete with the writers for access to the files. It took me three hours or more to complete one build.

I’m always looking for a way to get computers to do my work for me. FrameScript is a great way to automate FrameMaker tasks. After writing a few FrameScripts to help automate the build, I was able to reduce the build time to about an hour. Because I removed the human element, the scripts also made the builds much more reliable and consistent.

Looking for even more ways to automate things, I proposed that our team purchase WebWorks AutoMap, which would allow us to run the builds every night of the week. My manager suggested that once we had AutoMap set up, we might be able to offer AutoMap services to other groups.

Three months later, our first complete AutoMap system was online, processing 128 jobs for many Veritas products. The jobs comprised 19,000 PDF pages and another 4,000 HTML pages.

Since that time, I’ve improved performance and throughput by adding automated build technologies, such as Apache Ant. These enhancements have brought us to the 37,000 pages I mentioned above.

The jobs run completely unattended between 6:00 pm and 9:00 am. There is still plenty of time on the server for additional jobs; the jobs consume only 5 3/4 hours of machine time.

AutoMap now fetches source files from VSS, CVS, and file servers. It generates output in Acrobat, Microsoft HTML Help, WebWorks Help, and other DHTML Help sets. Some output documents are stored in CVS, and other documents are copied to file servers using FTP.

AutoMap Components

Quadralay AutoMap has three major components:

• AutoMap Server - Enables you to run WebWorks Publisher from a command line.
• AutoMap Administrator - Allows you to create and maintain AutoMap jobs. You also use Administrator to run jobs and schedule them for unattended execution. As we’ll see, the AutoMap Administrator isn’t always necessary; its operation can be replaced by scripts and a scheduling program.
• WebWorks License Manager - Controls concurrent users of WebWorks licenses (WebWorks Publisher and WebWorks PDF template). At times I’ve been exasperated by the poor documentation for this software. The Help contains only a cursory explanation on how the software works and does not describe real-world tasks, such as how to move License Manager to another server. More recent Quadralay products do not use License Manager.

To use AutoMap, you must have at least one license of WebWorks Publisher 2003.

AutoPDF

If you are using AutoMap Administrator, the AutoPDF template is the one additional component essential for a complete solution. As the name indicates, AutoPDF allows you to use AutoMap to generate PDF files.

If you use AutoPDF, you need to know two things:

• You cannot use the AutoPDF template in WebWorks Publisher. This situation makes testing rather difficult. You need to use WebWorks publisher to make changes to the template, but you must test your modified template in an AutoMap job.
• The AutoPDF template requires a license. AutoPDF licenses are managed by the WebWorks License Manager. To schedule simultaneous AutoMap jobs that use AutoPDF, the total number of simultaneously running AutoMap PDF jobs cannot exceed the number of available WebWorks 2003 or AutoPDF licenses.

Note that you don’t need AutoPDF if you are running AutoMap from a script. It’s easier and cheaper to run a FrameScript to save books as PDF.

AutoMap Client

The one piece of software that isn’t necessary for document automation is the AutoMap Client. It allows users on any machine on your network to run WebWorks jobs for their FrameMaker files. The users don’t need to have WebWorks Publisher or AutoMap installed on their machines, just the AutoMap Client.

While this sounds like a useful tool, it runs into problems when you’ve got variables, conditions, and WebWorks user macros that need to be set for each job. I find that these are essential to producing useful drafts.

Although that’s my experience, others may find that AutoMap Client works well in their environment.

AutoMap Concepts

Using AutoMap is similar to using WebWorks Publisher, but there are some differences. Let’s first take a look at using WebWorks Publisher.

Creating Jobs in WebWorks Publisher

You begin in WebWorks Publisher by using the wizard to create a project. In the wizard, specify the name of the template and the name of the book file. The wizard then guides you through modifying paragraph, character, table, and other mappings and configuring the show/hide settings on conditional text.

When you’re done with the wizard, you have a .wdt or .wfp file (depending on your WebWorks version), which you can then use to generate output.

Creating Jobs in AutoMap

The basic information you provide to AutoMap is similar to the information you provide to WebWorks Publisher. To create an AutoMap job, you specify the name of the project, the WebWorks template, the book file, and other related information.

Your template must already have the appropriate paragraph, character, and other mappings. It’s best to create or maintain the template with WebWorks Publisher, then refer to it from AutoMap.

After providing the basic information, AutoMap creates a .wfp file (AutoMap works only with WebWorks Publisher 2003). AutoMap then uses the .wfp file to generate the output.

One major difference between AutoMap and WebWorks Publisher is that each time you run a job in AutoMap, it creates a new .wfp file. There are certain things that you cannot alter that you might be used to altering in WebWorks Publisher, such as user macros and individual changes to files in the Support directory. When I started looking at AutoMap, I was somewhat concerned by these limitations, but it turned out that they just required a different way of doing things.

After testing your job a few times, use the Job Scheduler feature of AutoMap Administrator to schedule your job for unattended operation.

Preparing Files for AutoMap

AutoMap is great at running WebWorks Publisher. But to create reasonable output from FrameMaker sources, you need to consider all the steps that go into document production. These include: setting variables and conditions, refreshing text insets, setting page and chapter numbering, generating tables of contents and indexes, copy fitting the generated files, setting change bars, and so on.

Rather than requiring users to check in “print-ready” files every night, I used FrameScripts to automate these document preparation steps. I started with my build automation FrameScripts, then added a number of command-line arguments so that I could control the processing.

AutoMap Administrator vs. Custom Scripts

There are two different approaches to creating and scheduling AutoMap jobs:

• Use AutoMap Administrator to create, maintain, schedule, and run AutoMap jobs.
• Write your own batch files or scripts to run AutoMap jobs. You can schedule those jobs using the Windows Task Scheduler.

AutoMap Administrator

As long as your documents meet certain criteria, AutoMap Administrator provides a convenient way of creating and scheduling jobs. AutoMap Administrator is great when:

• The source FrameMaker files are directly accessible through the file system (either local to the machine where AutoMap runs, or on a file server).
• The locations for the final work products (PDF files or Help sets) are accessible through the file system.
• You don’t need to do any pre-processing or post-processing on the files.

Administrator allows you to set up pre- and post-processing on files, but this limits other useful capabilities of the AutoMap Administrator.

The initial display in AutoMap Administrator shows the jobs you created and when they are scheduled to run.

AutoMap Administrator

To modify or create a job, use the Job Editor. Use this tabbed dialog to specify all the standard WebWorks job parameters, such as template, job name, file locations, language, and so on. The first two tabs are the most interesting.

From the General tab, specify the name of the job, the source type, the WebWorks template, and a post-processing script (if any).

Job editor General tab

From the Projects tab, you can specify multiple projects that are combined into one Help set. For each project, specify the name of the project, project-to-project linking context, locale and encoding, and the FrameMaker source files used by the project. You can either specify the files directly (if they’re locally accessible) or specify a script to retrieve and pre-process the input files.

Job editor Projects tab

Note the words “Custom Properties” on the dialog. It took me a while to figure out that “Custom Properties” is a clickable button.

When you click on Custom Properties, the panel contents change to display a set of properties specific to your template and project. If you are using the AutoPDF template, this portion of the dialog is quite important.

Custom properties

Editing job.xml Files

For each job in AutoMap administrator, AutoMap creates a directory. The information about each job is stored in an XML file (named job.xml) in the job’s directory.

After using AutoMap Administrator for a while, I found that editing the job.xml file was a much faster and easier way of making many job changes.

Also, when I needed to create a new job, it was quite easy to clone an existing job by copying and renaming the job directory and then editing the job.xml file.

In particular, there’s not enough room in the AutoMap Job Editor dialogs to see the pre- and post-processing script calls.

Pre- and Post-Processing

AutoMap Administrator allows you to run scripts to perform actions before and after AutoMap processing. Scripts are useful when:

• The source for your input is not on your local file system.
• The final destination for your output is not on your local file system.
• You need to modify files before AutoMap processing (such as using FrameScript to update the Table Of Contents).
• You need to modify files after AutoMap processing (such as zipping WebWorks Help files).

Unfortunately, Quadralay did not include much documentation explaining how to invoke the scripts. If you need to implement scripts in AutoMap, the following section of this article should be helpful. (As a disclaimer, in the absence of useful documentation, I might not have found the best way to perform these tasks. If you have a better way to accomplish these tasks, let me know.)

To specify a pre- or post-processing script in the Job Editor, you must:

1. Wrap your script in a Windows Script File (.wsf).
2. Use cscript.exe to run the script.

Windows scripting is the high-powered, modern-day equivalent of batch files of the DOS days. I won’t go far into Windows Scripting here. However, if you’re interested in Windows Scripting, I highly recommend that you visit the Microsoft Scripting Website.

A Windows Script File is nothing more than an XML wrapper around a regular script (VBScript, JScript, PerlScript, Python, and so on). As long as the interpreter for the scripting language is installed on your machine, you can use Windows Script Files.

AutoMap uses environment variables to pass two essential pieces of information to the pre-processing script (these are documented in the AutoMap Help):

• WWPROJECTDIR tells your script where to put the files to be processed by the AutoMap job.
• WWFILELIST is a path to a file. Your script must write the name of the FrameMaker book file (or FrameMaker files) to be processed by the AutoMap job.

There are no parallel environment variables for the post-processing scripts.

When you invoke a script from the Job Editor’s fields, use the syntax:
cscript.exe yourfile.wsf [“arg1” [...] ]

Running AutoMap from Scripts or Batch Files

There are two executables for AutoMap server:wpuba.exe andwpjob.exe. Both of these executables are well documented in the AutoMap Help. Compared to using AutoMap Administrator, running AutoMap from scripts is much more straight-forward.

All your pre- and post-processing steps can go in the same script or batch job. You can use whatever scripting language you want to build the documents; the scripts do not have to be in Windows script (-+.wsf+-) files.

If you generate output from a single FrameMaker book, you run wpuba.exe twice:

• The first time, you usewpuba.exe to create the project. This is just like using the Project Wizard in WebWorks Publisher to create a.wfp file.
• The second time, you usewpuba.exe to generate the output.

To generate merged output from multiple FrameMaker books, use both wpuba.exe andwpjob.exe:

• Usewpuba.exe to create AutoMap jobs for each of your source books.
• Then usewpjob.exe to generate the output.

AutoMap at Symantec

Our first AutoMap implementation used AutoMap Administrator, but I eventually migrated all the work to scripts that call wpuba.exe from Ant. (Ant is a Java-based software build tool similar to Make. Ant’s major advantage over Make is that it incorporates many modern-day scripting concepts. See http://ant.apache.org.)

AutoMap Administrator was useful for getting started with AutoMap. With that familiarity came an understanding of how I could avoid inefficiencies by writing my own scripts. These scripts also allowed me to avoid a number of other frustrations I had with AutoMap Administrator.

In my current implementation, Ant fetches sources from source control, pre-processes files with FrameScript, invokes wpuba.exe to create the project files and generate output, post-processes the output (if necessary), and moves the output to a source control facility.

File Organization

In our file organization, all documents for one team are stored in one directory (books and generated files at the top level directory; sections, insets, and graphics in subdirectories). The books use shared files and insets for common bits of text.

To build one book under AutoMap Administrator, I had to fetch all files from the directory (fetching individual files would present too many maintenance problems).

The logical flow for AutoMap Administrator is: pre-process, generate, post-process. For Symantec DCMG docs, these steps expanded to: fetch, prepare for print (FrameScript), generate, zip if necessary, and post.

This model works for one book. However, to process multiple individual books with individual outputs, the flow needs to be:

• Fetch sources.
• Prepare first book; generate first book
• Prepare second book; generate second book
• Repeat until all books are done.
• Zip the output, if necessary.
• Post the output.

This flow is difficult to capture in WebWorks Administrator. In Ant, it’s a natural fit.

Other AutoMap Administrator Issues

If you give AutoMap Administrator the name of the book file it is to process, you can use the Conditions and Variables tabs of the Job Editor to query the FrameMaker files for condition settings and variable definitions. Both of these tabs have a Scan Documents button that is intended to fetch conditions or variables from your documents. Unfortunately, these buttons do not work if you get your documents via a pre-processing script.

This glitch is unfortunate because modifying variables from WebWorks would be very useful.

There are a number of little annoyances in the GUI, in addition to the stealth button (“Custom Properties”) in the Projects Tab.

The Next Run and Last Run columns in the AutoMap Administrator do a simple ASCII sort, rather than a date/time sort, which results in hours sorted in the order: 1:00, 10:00, 11:00, 12:00, 2:00. There is a workaround: look in the Windows Task Scheduler, which is where you’ll find the AutoMap jobs. The Next Run Time and Last Run Time columns in the Task Scheduler sort correctly.

The pre- and post-processing scripts required a number of parameters (file location, CVS info, pre-processing information, and so on). Editing a lengthy list of parameters in the small pre- and post-processing script fields is virtually impossible. The workaround: edit the job.xml file directly.

User Macros and AutoMap

When developing WebWorks templates for use with WebWorks publisher, I employ user macros for information that might change from project to project. When others use the template, they plug their project-specific information into the user macro definition.

However, because AutoMap creates a new project file each time it generates output, this application of user macros is no longer viable. Wpuba.exe does accept a UserMacro parameter, but it’s limited to a single user macro (-UserMacro name=definition).

Instead, you pass dynamic information to templates using Project Properties. (-props name=definition;name=definition).

When I modified my WebWorks templates for AutoMap, I replaced many user macro definitions with the $GET_PROJECTPROP(); function.

There is one further caveat. As shown in the syntax example, the semi-colon (;) is the property separator character. Unfortunately, there is no way to escape a semi-colon if you need to pass one as part of your data.

Communicating with Writers via a Wiki

If you’re using AutoMap to build documents for a large number of writers, it’s good to develop a way of communicating and storing the special requirements for each job.

A Wiki allows writers to describe their jobs to me. I can refer to the Wiki pages whenever I need, and writers can modify their specifications if necessary. A notifier in the Wiki tells me when a page has changed.

I created a page with the basic set of information I needed, including contact information for the project lead, source file location, book file names, expected output names, and so on. The leads then copied the empty page, created their own pages from the empty page, and filled in the necessary information.

This system worked quite well (and still works today). If I were to change anything about this system, I would make the information a bit more compact, rather than spread across a number of tables.

But There’s More…

Our writers found that having PDF files and Help sets built for them every day was extremely useful. Rather than dropping everything to produce a quick review draft, they point requesters to the most recent build.

Because the documents are updated daily, writers solve production problems as they develop documents. They then have more time to focus on last-minute content changes, rather than production issues.

In addition to generating documents daily, using AutoMap has other benefits:

• Documents generated by AutoMap can be made available through a Web site or FTP site, allowing anyone access to any of the documents generated by AutoMap.
• Apache Ant opens the door to any number of additional build steps. It is easy to add new pre- and post-processing steps or change existing ones. For example, we will soon incorporate a controlled language checker into our processes. It will be easy to incorporate this step in Ant.

To summarize, WebWorks AutoMap is an extremely useful tool for performing unattended documentation builds. Out of the box, AutoMap can generate reasonable documents. By adding the power of scripting, the results can be amazing.

References

• AutoMap, AutoPDF template, WebWorks Publisher are products of Quadralay Corporation: http://www.quadralay.com
• FrameMaker and Adobe Acrobat are products of Adobe Corporation: http://www.adobe.com.
• FrameScript is a product of Finite Matters, Limited: http://www.framescript.com
• Apache Ant is available from http://ant.apache.org/
• For information on Microsoft Scripting, see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnanchor/html/scriptinga.asp

Simon can be reached at simon_bate at symantec dot com.

If you liked this article, you might also enjoy: Single-Sourcing with FrameMaker and SiberSafe FrameMaker 9 Review Meeting Recap: NC-FUN SIG Talks About Conditional Text

Other editions...