WorkShop Teamware: Solutions Guide: - Using Makefiles With TeamWare

# Using Makefiles With TeamWare |

This topic describes how to write makefiles that take full advantage of Configuring and DMake, the `make` utility that is bundled with the Sun WorkShop TeamWare release. Both Configuring and DMake are designed to be compatible with existing makefiles and methods.

Configuring does not depend on any specific release of `make` or on any other system modeler, but your site's makefile hierarchy may affect the way you organize your Configuring workspaces. In addition, poorly written makefiles may keep you from realizing the full potential of DMake.

## Source and Makefile Hierarchies

In order to build and test successfully in a development workspace, a developer must bring over all the files required for the build. In most cases, one or more makefiles are among the files required for the build. Finding the correct makefiles may not be obvious if they reside in directories far from the local build directory, which is the case for many large projects.

The following sections describe the different ways three software development groups structured their makefile hierarchies. The makefile organizations range from casual to formal, and each organization has different implications for Configuring. In each case, the source directory tree was structured along modular lines, with one module or deliverable in each subdirectory of the tree.

### Self-Contained Makefiles

The first case describes the Bantam project. The project was small and the number of developers assigned to it were few. The Bantam group decided to use a makefile and source tree organization in which the makefile in each directory was self-contained. This strategy was regarded as sufficient because the project contained few modules and its source directory tree was simple. Also, because the group was small, coordination between developers was easy. When necessary, preprocessor, compiler, and linker flags were set globally for each build with a few environment variables.

    Click for closeup.

#### Figure 24 Source and Makefile Organization with Self-Contained Makefiles

The simplicity of this organization had several advantages. First, Configuring bringovers were obvious because all source, including the makefile required for the build, was present at each directory level -- a bringover of all files in a directory was sufficient to build successfully. Second, when makefiles were not allowed to grow too large, they were easy to debug because they contained no included files.

A major disadvantage of this organization was that minor unforeseen changes to the build (a new location for include files, for example) required a change to each makefile. Also, style consistency in makefiles was difficult to enforce, which resulted in makefiles that were difficult to maintain.

A final disadvantage became apparent as the Bantam project matured. Over time, the project became larger than first envisioned, and the source tree eventually required so many large makefiles that understanding and maintaining them became difficult.

### Direct Inclusion of Makefiles

The second case describes the Midpoint project, which was larger than the Bantam project and required more formal organization. The Midpoint group settled on a strategy of including selected high-level makefiles in low-level makefiles; the top-level makefiles set flags and macros that were used throughout each build. By setting these flags in top-level makefiles, the group was able to centrally control such parameters as the installation location of targets, the locations of special libraries and include files, and the optimization level of compilers.

Some low-level makefiles included makefiles from intermediate directories in the source tree. For example, one family of commands had so much in common that the makefile from a common parent directory was included in the makefile for each command. This case is shown in Figure 25, where `cmd.Makefile` is included in the makefile for the `first_cmd` module.

The group had to be careful not to include high-level makefiles more than once in lower-level makefiles. For example, Figure 25 shows `master.Makefile` being included in the lowest level makefile. It should not be included in an intermediate makefile such as `cmd.Makefile` that is also included at the lowest level, although such multiple inclusions rarely lead to fatal errors.

    Click for closeup.

#### Figure 25 High-Level Makefiles Included Directly in Low-Level Makefiles

The major advantage of this organization was that the Midpoint group could set flags, macros, and directories common to all makefiles at the highest level; a single change in `master.Makefile` applied to all the makefiles in the hierarchy. Makefile style was easier to enforce -- in fact, it had to be enforced, at least to the degree that all makefiles used the included top-level makefiles the same way. Individual makefiles were smaller in the Midpoint project than in the Bantam project, but debugging makefiles became more difficult in some cases because of the inclusions.

When the Midpoint group adopted Configuring, another disadvantage of the makefile organization became apparent: a bringover of any module required that the included makefiles also be brought over. The group handled the difficulty by customizing the default File List Program (FLP) so that it brought over not only the contents of the source directory but also automatically brought over the included makefiles. The FLP list of included makefiles had to be updated manually when a new high-level makefile was included in the makefile for a low-level module.

### Nested Inclusion of Makefiles

The last case describes the experience of the Hefty project, which was large and required the most formal strategy in its makefile organization. The Hefty group decided to require that each makefile include the makefile immediately above it in the source tree. The most general makefiles were at the top level. Descending the source tree, each subdirectory contained a makefile with increasingly specific build instructions. The formal organization ensured that high-level makefiles were included only once in lower-level makefiles.

    Click for closeup.

#### Figure 26 Nested Makefiles -- Each Makefile Includes the Makefile Above It

As with the strategy adopted by the Midpoint group, an advantage of the Hefty makefile organization was that makefile style was easy to enforce. Another advantage was that each individual makefile was small. However, because of the large number of nested includes, understanding and debugging makefiles at the lowest levels sometimes proved difficult.

When the Hefty group adopted Configuring, they recognized that each developer had to be sure to bring over the entire list of included makefiles in order to build successfully. The listing of makefiles was automated by a custom FLP and script. This makefile organization and use of the custom FLP is similar to that used by the SunOSTM group. See "Using File List Programs" on page 32 for a description of that group's experience.

## More About Configuring and Makefiles

When Configuring copies source files between workspaces in bringover and putback transactions, it relies on SCCS to identify, control, and track the files. In fact, Configuring operates exclusively on SCCS files, so the decision about what files to place under SCCS control is important.

Projects that use `make` to manage builds usually treat makefiles as source files, placing them under SCCS control and copying them between workspaces during bringovers and putbacks.

### Makefile Contents

A makefile contains a list of derived modules (usually files), called targets, and the modules required to build them (called dependencies). A target may be the final result of a long series of compilations (a top-level target) or an intermediate result (secondary target) used to build a top-level target. Typical secondary targets are the object (.`o`) files produced by C compilers. A makefile may also contain explicit rules describing how a target is to be built from its dependencies.

### Makefiles as Derived Source

One form of secondary target is the derived source file, a source file that is generated automatically near the beginning of a build. At some sites, makefiles or parts of makefiles are written automatically at the beginning of the build process by a makefile generator and are themselves derived source.

Some insights into generated makefiles can be gained from the experience of the Concoct project development group. This group inherited a build environment that included generated makefiles. When the group adopted Configuring, they needed to find a way to enforce consistency in their generated makefiles.

### Generated Makefiles

As the Concoct group reviewed the original rationale for using makefile generators, they realized that the main reason was to read the contents of one or more directories, identify the source files in them, and explicitly list those file names in the generated makefile. For example, one makefile generator scanned the build directory for C++ source files of the form `*.cc`, built a list, and wrote a makefile that included the list explicitly.

This approach relieved developers from hand editing makefiles whenever they created or renamed a source file. However, when a generated makefile was placed under SCCS control, the following permission conflict resulted: subsequent builds tried to overwrite the makefile even though it was read-only as a consequence of being under SCCS control. This condition created a problem because a makefile that is not under SCCS control is not put back into the parent workspace following a build.

### Controlling Generated Makefiles

The Concoct group actually used three different makefile generators:

- Generator 1 -- `imake`, a compiled binary that read a template, a set of `cpp` macro functions, and an input file in each build directory. Using these text files as instructions, `imake` generated makefiles.

- Generator 2 -- A script that performed the same function as `imake`, but used different input files.

- Generator 3 -- A script that generated makefiles without reading input instructions. The script simply listed the build directory contents and treated all listed files as source.

The group realized that it could control its generated makefiles indirectly by placing the inputs to the makefile generators (or the generators themselves if they were scripts and not compiled binaries) under SCCS control, as illustrated in Figure 27.

    Click for closeup.

#### Figure 27 Putting Makefile Generators Under SCCS Control

In each case, only true source (either the input to the makefile generator or the generator itself) and not derived source (the generated makefiles) were under SCCS control. As a result, new makefiles were built consistently and the information used to generate makefiles was propagated between workspaces.

The group agreed on a policy to ensure that makefile consistency would not be compromised:

- Each build would create generated makefiles anew, and the generated makefiles would not be edited by hand after being built.

- If a generated makefile had to be edited, an automated way to edit it would be provided. The script that automated the editing would be placed under SCCS control and treated as source.

- Generated makefiles would be deleted after each build (with the `make` `clean` pseudotarget) along with other secondary targets such as object (`*.o`) and library (`*.a`) files.

### Devguide as Makefile Generator

The Concoct group produces software for the XViewTM environment and uses the Open WindowsTM Developer's Guide (Devguide) to develop the Graphical User Interface (GUI) for its product. Devguide generates a makefile as part of its output, and so may be considered a makefile generator. Devguide consists of a GUI Design Editor and a set of postprocessors that convert the output of the editor into C or C++ source code for use with a variety of GUI libraries and toolkits.

The GUI Design Editor provides a graphical way for developers to design user interfaces without writing code. After the interface has been designed on the screen, the result is saved to a GIL (Guide Interchange Language) file, an intermediate text-based representation of the interface. The GIL file is a source file -- the GUI Design Editor that creates it is analogous to the text editor a developer might use to write an ordinary source file. Because the Concoct group's executables are targeted for an XView application, they run the postprocessor GXV (Guide-to-X View) on the GIL file to generate XView source in the form of a header file, two source files, and a makefile. These derived sources are used by `make` to build the desired executable.

The use of GXV (or one of the other Devguide postprocessors) presents the same challenge to source code control as a makefile generator -- the files actually used by `make` to build the executable are derived sources. Because they are derived during the build process, they should not be placed under SCCS control. The GIL file created by the GUI Design Editor, however, can be considered true source because it is generated by the developer prior to the build process. As true source, the GIL file can be placed under SCCS control, and Configuring will bring it over and put it back along with other source.

The first step in the Concoct group's automated build process is to run the postprocessor GXV to regenerate the derived source. Once created, the derived source is never hand edited and is deleted along with other intermediate targets following the build.

In deciding not to hand edit derived source, the group avoided the following complications: if the derived source were to be hand edited, it would have to be placed under SCCS control (to make it eligible for Configuring bringovers). In order to allow the derived source to be rederived, the makefile would have to be enhanced to check out the derived source (making it writable), the hand edits would have to be merged into the checked out file, and the result checked back in to SCCS. Only then could the project build begin.

### Makefiles Using Built-In `make` Features

Following a major release milestone, the Concoct group decided to rewrite its entire code base. They took advantage of the occasion to rewrite their makefiles as well in order to eliminate their makefile generators (with the exception of Devguide). They discovered that they could achieve the same goals by writing makefiles that exploited the built-in features of `make` (dynamic macros, pattern-matching and implicit rules, conditional macro assignment, and so on). These makefiles could be placed under SCCS control and propagated between workspaces as true source.

In designing its makefile hierarchy, the Concoct group adopted the makefile inclusion strategy used by the Midpoint group (shown in Figure 25 on page 96). In this strategy, one or more top-level makefiles are included in each makefile in the directory hierarchy.

The example makefiles are not explained in great detail. Refer to the `make` man page and the manuals SunOS 5.0 Programming Utilities and SunOS 5.0 User's Guide for more information on `make` and makefiles.

#### Master Makefile

The master makefile shown in Figure 28 defines several macros (`LDLIBS`, `CC`, `CPPFLAGS`, and so on). It also sets the special-function target .`KEEP_STATE` for all makefiles. Because the master makefile is included in all other makefiles, these definitions apply throughout each build.

    # master.Makefile, to be included in all
    # subordinate makefiles.
    .KEEP_STATE:
    LDLIBS = -L/usr/lib
    CC = cc
    CPPFLAGS = -DSUN5_x
    CFLAGS = -g -xs
    LINK.c = ${CC} ${CFLAGS} ${CPPFLAGS} ${LDFLAGS}

#### Figure 28 Top-Level Makefile for the Concoct Project

#### Low-Level Makefile

A typical low-level makefile for the Concoct project is shown in Figure 29. This makefile is used to build an executable named `cogitate`.

Comments in the file explain the constructs of its targets, dependencies, and rules. Note that the makefile automatically absorbs any new C source files (files with a .`c` suffix in their names) that may be introduced in the build directory. This capability, implemented with a pattern replacement macro, helps eliminate the need for a makefile generator.

    # First, include the master makefile.
    # This low-level makefile assumes that master.Makefile
    # is two directories above the current directory.
    TOP	= ../..
    include ${TOP}/master.Makefile
    # The next macro definition uses command
    # substitution to list all C source files in
    # the current directory.
    CSRCS:sh = ls *.c
    # The next pattern replacement macro creates
    # a list of object file names, based on the
    # source file names.
    COBJS = ${CSRCS:%.c=%.o}
    # The next rule uses a dynamic macro:
    # $@ represents the name of the current target, in
    # this case "cogitate." The rule links all object
    # files and produces an executable named "cogitate".
    cogitate : ${COBJS}
    	${LINK.c} -o $@ ${COBJS} ${LDLIBS}
    # Next, the targets and dependencies are listed by
    # means of a pattern matching rule.
    # The last build rule uses a dynamic macro:
    # $< is the name of a dependency file.
    %.o: %.c
    	${CC} ${CFLAGS} ${CPPFLAGS} -c $<

#### Figure 29 Low-Level Makefile for Building the `cogitate` Executable

## Using Makefiles With DMake

DMake, the `make` utility bundled with Sun WorkShop TeamWare, can build several targets simultaneously in parallel processes. This new capability contrasts with earlier versions of `make`, which build targets one at a time in the sequence in which they appear in the makefile. DMake is a syntactic superset of the standard SunTM version of `make`. It supports all the syntax of standard `make` and works with existing makefiles.

### Improving Performance With DMake

Most build processes are I/O bound, which means that they are limited by the speed at which data can be read and written from mass storage devices, not by CPU performance. Therefore, building targets in parallel results in performance increases on all machines, even single-processor ones, but the greatest improvements are seen in the new classes of multiprocessor servers and workstations.

### Listing Dependencies Explicitly in Makefiles

Most makefiles that were written for earlier versions of `make` will work with DMake. When problems occur, they usually result from a reliance on the order in which targets appear in a makefile to establish the build order. These problems can be avoided by explicitly listing each target's dependencies rather than relying on the build of another target to bring the dependencies up to date. For example, consider the following makefile fragment, which the Concoct group inherited from an early version of the project:

    all:prog1 prog2
    prog1: prog1.o aux.o

    	$(LINK.c) prog1.o aux.o -o prog1
    prog2: prog2.o

    	$(LINK.c) prog2.o aux.o -o prog2

When built in serial, the target `aux.o` was built as a dependent of `prog1` and was up to date for the build of `prog2`. However, when built in parallel, the link of `prog2` sometimes began before `aux.o` had been built, and was therefore incorrect. The group corrected the dependency list for the `prog2` target to read:

    prog2: prog2.0 aux.o

### Controlling DMake With Special Targets

Other cases of the implicit ordering of dependencies were more difficult for the Concoct group to identify and correct. The ParallelMake User's Guide discusses these cases in detail. In the end, the group used the following special-function targets to handle their most difficult problems:

- `.WAIT` -- When inserted into a dependency list, the special-function target`.WAIT` causes DMake to complete the processing of all prior dependents in the list before proceeding with the following dependents. For example,

    	all: hdrs .WAIT libs functions

  causes DMake to wait until the `hdrs` dependency is built before building `libs` and `functions`.

- `.NO_PARALLEL:` -- The target `.NO_PARALLEL:` causes a list of targets to be built in serial rather than in parallel. For example,

    	.NO_PARALLEL: prog1 prog2

  When all targets must be built serially, `.NO_PARALLEL:` can be specified without arguments. If there are exceptions that can be built in parallel, they can be identified with the `.PARALLEL:` special target.

- `.PARALLEL:` -- When `.NO_PARALLEL:` is used without arguments, `.PARALLEL:` is used to identify targets that can be built in parallel. For example,

    	.NO_PARALLEL:

    	.PARALLEL: prog3 prog4

  causes all targets to be built serially with the exception of `prog3` and `prog4`, which can be built in parallel.

### Using DMake Serially

Finally, DMake can be forced to behave serially for all targets by using the `-R` option. While this option guarantees compatibility with working existing makefiles, it sacrifices the improved performance possible with DMake.