WorkShop Teamware: Solutions Guide: -

# Managing SunOS Development With TeamWare |

This case study describes how the Sun WorkShop TeamWareTM code management software is being used in a large software development project. The study is taken from the experience of a Sun Microsystems® business unit, in its ongoing development of the SunOSTM operating system component of the Solaris® operating environment.

## Project Description

The source base for the SunOS project is large -- more than one million lines of code contained in over twelve thousand files. The staffing for the project is correspondingly large, with more than 100 developers working at sites across California and Colorado.

Management recognized during the project's planning stages that developers would need parallel access to the most current source -- the classic one-developer-at-a-time access allowed by SCCS (the Source Code Control System) was too limiting for their needs. As a more capable alternative, they adopted Configuring, the TeamWare component that manages groups of files for groups of developers.

Configuring provides an isolated workspace for each developer and automatically records the change history of each source file. By judiciously setting up their hierarchy of Configuring workspaces, the group allows each developer access to the newest stable source code at any time. If two developers edit copies of the same file, Configuring detects the differences and presents them graphically in the Merging tool. One or both of the developers merge the changes into a single file that is then returned to the common source base.

Five integration engineers coordinate and maintain the Configuring workspace hierarchy for the project. Only a fraction of their time is spent on Configuring issues, however. Their additional duties include:

- Building the current SunOS release and packaging it for delivery to Release Engineering
- Managing earlier SunOS releases, including the build and delivery of patches
- Maintaining the makefile hierarchy used to build the project

The integration engineering staff generally agrees that TeamWare tools have reduced time-to-market for the SunOS product by lowering the overhead required to manage source code during the product's development phase. The tools are also being used to maintain earlier product releases.

## Case Study Overview

This case study details the following aspects of the strategy used by the SunOS group in adopting and deploying Configuring and the other TeamWare tools:

- Organizing Configuring workspaces -- The organization of the Configuring workspace hierarchy affects the building and testing of executables at the developer, integration, and release levels.

- Automating builds within Configuring -- The directory structure within workspaces and the organization of makefiles both affect the ability of the `make` utility to smoothly automate the compilation and linking of programs. The SunOS group makes use of custom scripts and File List Programs (FLPs) to set environment variables for makefiles and to bring header files and libraries into workspaces automatically.

- Coordinating network and operating system features with Configuring -- The SunOS group takes advantage of operating system features to enhance Configuring source file access control and to facilitate changing to workspace directories. They also use the Configuring notification feature to mail policy statements to developers.

## Basic Configuring Concepts

This section defines five Configuring terms that will help you understand the case study.

Workspace -- A directory (and its subdirectories) that has been designated for use by Configuring. Configuring identifies a workspace by the presence of a subdirectory named `Codemgr_wsdata`.

Workspace hierarchy -- An organization of parent and child workspaces, much like a standard directory hierarchy. Each workspace can have a parent and one or more children, in the same way that a directory has a parent directory and subdirectories.

Putback -- A transaction between a child workspace and its parent, in which changes to files from the child are copied ("put back") into the parent.

Bringover -- A transaction between a child workspace and its parent, in which changes to files from the parent are copied ("brought over") into the child.

Reparent -- An operation that changes the parent of a child workspace.

## Organizing the Workspace Hierarchy

SunOS development is an ongoing project, so the workspace hierarchy was organized to allow for successive, sometimes overlapping, releases. Central to the organization is the concept of "the train," a constantly evolving top-level workspace that always represents the current group development effort. The train analogy results from the way product-release workspaces are generated, and is explored more fully on pages 14-16.

Beneath the top-level workspace, called `train`, is a group of sibling integration workspaces (see Figure 1). Developers bring files into their private workspaces from these integration workspaces. They also put back to the integration workspaces, never to `train` directly. Because the integration workspaces restrict access to the release-level workspace, the SunOS group calls them gates. In each group, a senior-level developer is designated as the gatekeeper. The gatekeeper is responsible for maintaining the group's gate workspace, making sure that the source it contains is without error before it is put back to the release-level workspace.

    Click for closeup.

#### Figure 1 Basic Workspace Hierarchy

Most integration workspaces are organized functionally: developers working on a related group of commands or libraries transact with a single integration workspace. Other integration workspaces represent physical organization: developers working at a remote site transact with a single integration workspace, which puts back to `train` overnight. (Remote integration workspaces communicate with `train` through the NFS® distributed file system by means of a dedicated communications link, which slows down transactions somewhat.)

The organization shown in Figure 1 is an ideal; the actual hierarchy at any given time may look quite different because the SunOS group usually reparents workspaces during building and testing, as described in the following sections. The ability to reparent workspaces results in a great deal of flexibility. The hierarchy is a statement of group policy, not a rigid organizational structure, and can easily be revised.1

### Building and Testing Executables

Each developer in the SunOS development group is expected to build (compile and link) and test the new code he writes. Custom File List Programs (FLPs) and hierarchical makefile structures help ensure that every development workspace builds with the most recent header files and libraries. After a successful build and test, the developer puts back his new source files into the integration workspace. If a conflict occurs because another developer's changes were put back while the first developer was working, the first developer resolves the conflicts with the Merging tool, and the resulting source files are once again compiled, linked, and tested.

A large software development project is almost always organized into functional modules, and the SunOS project is no exception. Because of this modularity, the code that has been put back to an integration workspace cannot be guaranteed to build and test successfully even though each developer tests new code in his own workspace. While a development workspace usually tests one bug fix at a time, an integration workspace can accumulate dozens or even hundreds of putbacks between each build-test cycle. To make certain that the interaction between all these changes is correct, a senior developer is given responsibility for building and testing at the integration workspace level. Only when the testing is completed is the new code put back into the release workspace to become part of the product.

Building in a parent workspace, whether it be integration or release, requires that no child workspaces put back during the build. If putbacks were allowed, the source code base could change in the midst of compilation, almost assuring errors. This requirement clashes with the goal of letting each developer transact with his integration workspace at any time -- a goal that increases productivity and decreases wait time for developers. Configuring solves the problem by allowing reparenting of workspaces, as discussed in the next section.

### Isolating Test Workspaces

The SunOS development group creates identical copies ("clones") of integration workspaces in which to build and test. After a successful test of the software, reparenting is generally used to put back changes to the release workspace. Depending on the demands of the moment, testing is done in a clone of one gate or in a temporary composite workspace formed by cloning one gate, then reparenting the clone to a second gate and updating it with files from the second gate. This section describes both approaches.

#### Testing in a Single Integration Workspace

Consider the gatekeeper who is responsible for maintaining the integration workspace `gate_1`. The workspace has accumulated enough changes to warrant putting them back into `train`. Before putting back, however, the gatekeeper must do three things:

  1. Bring over any new source files from `train` that have been put back from other integration workspaces. This step also involves resolving any conflicts that may result from the bringover.

  2. Compile and link the source code in `gate_1`.

  3. Test the resulting executables.

If the gatekeeper brings over directly from `train` into `gate_1`, she is likely to encounter conflicts. As she resolves these conflicts (by merging source files), she will be forced to lock `gate_1` to keep developers from putting back while she works.

Instead of locking the gate workspace, the SunOS group's gatekeepers use a cloning technique that allows developers access to the gate while they resolve conflicts and fix bugs in the source base. The following figure (Figure 2) shows the steps the gatekeeper would perform to integrate `gate_1` with the release workspace `train`.

####

    Figure  2	 Integrating an Integration Workspace Into a Release Workspace

    Create a clone of gate_1 with a bringover create command (\xac ).

    Click for closeup.

    Reparent to train (\xad ), and then bring over into the clone (\xae ).

    After the gatekeeper resolves the conflicts that result from bringing over updates from train, gate_1_clone is no longer an exact image of gate_1 -- it has been modified by the most recent changes other integration workspaces made in train.

    Click for closeup.

    The gatekeeper must now build and test the source code in gate_1_clone to make sure it is correct (\xaf ). Building in gate_1_clone requires locking it against putbacks, but this constraint inconveniences no one because gate_1_clone has no children.

    The build and test may reveal inconsistencies and bugs that will have to be fixed. As the gatekeeper fixes the errors, the contents of gate_1_clone diverge from gate_1.

    When the gatekeeper is satisfied with the results of her testing, she puts back from gate_1_clone to train (\xb0 ) rather than reparenting and putting back to gate_1.

    When changes are put back from the clone directly into train, they become available immediately to all other integration workspaces, including gate_1. A bringover from train to gate_1 brings gate_1 up to date (\xb1 ).

    Click for closeup.

Note that in the day or two the gatekeeper spent building and testing in `gate_1_clone`, developers may have been putting new changes into `gate_1`. The best method for synchronizing `gate_1` and `gate_1_clone` in that event involves three steps, as described in Figure 3.

#### Figure 3 Synchronizing the Integration Environment With its Clone

Reparent `gate_1_clone` to `gate_1` (\xb1 ). Bring over to update `gate_1_clone` (\xb2 ), resolving conflicts as required. | Click for closeup.
Put back to `gate_1`, (\xb3 ). | Click for closeup.

The cloning technique just outlined allows development to move forward at the same time a large, complex workspace is being tested and integrated into the release level. It's disadvantage comes when a developer puts back a large functional module into the integration workspace while testing is going on in the clone. In such a case, the final bringover and putback (\xb2 and \xb3 , Figure 3) can result in many conflicts and a great deal of work for the gatekeeper. These cases can be avoided by polling developers about their work in progress and scheduling the build and test operations for lull times following a developer milestone putback event.

#### Testing in Merged Workspaces

With a few additional steps, the process discussed in the last section can be applied to merging several gates, building and testing, and then reparenting and putting back to the release workspace. If the gates do not contain a lot of new code, gatekeepers can save time by merging the gates before building and testing them: instead of building and testing a clone of each gate, the gatekeepers build and test once for all the merged workspaces.

Figure 4 shows the process for two integration workspaces.

#### Figure 4 Merging Two Workspaces and Putting Back to the Release Level

A gatekeeper creates a clone of `gate_1` by bringing over into a new workspace called `gate_merge` (\xac ). | Click for closeup.
The gatekeeper reparents to `gate_2` (\xad ) and brings over changes, which may create conflicts that must be resolved in `gate_merge` (\xae ). After the conflicts are resolved, the gatekeeper builds and tests in `gate_merge` (\xaf ). | Click for closeup.
When testing is done, the gatekeeper reparents to `train` (\xb0 ) and puts back into the release workspace (\xb1 ). | Click for closeup.
To synchronize `gate_1` and `gate_2` with `gate_merge`, the gatekeeper could bring over from `train`. However, that bringover could create conflicts that would need to be resolved in `gate_1` and `gate_2`, which would lock out developers from those integration workspaces. A better approach is separately to reparent and put back from `gate_merge` into `gate_1` (\xb2 ) and `gate_2` (\xb3 ). If the putbacks are blocked, the gatekeeper can perform new bringovers and resolve conflicts in `gate_merge`, leaving the integration workspaces free for developer transactions. | Click for closeup.

Clearly, more than two integration workspaces can be merged in the manner just described in Figure 4. Projects of lesser complexity may extend workspace merging to include all integration workspaces in the project, periodically merging, building, testing, and putting back to the release level. See the TW Topic "Workspace Hierarchy Strategies for Software Development and Release on page 37" for a discussion of this approach.

### Managing Release-Level Workspaces

Many of the considerations that apply to integration-level workspaces also apply to the release level -- providing transaction access for child workspaces during build and test cycles, for example. Additional issues at the release level involve providing for alpha, beta, and customer releases, multiple simultaneous customer releases, and maintenance (patch) releases.

As a workspace nears release, the group relies on a senior developer who is assigned to the workspace. This person decides when to merge changes into the workspace and when testing has been completed satisfactorily. The position is one of considerable responsibility. Integrators at the release level are aided by Configuring and network features that control access to source files and email notifications when workspace transactions occur.

#### Posting Notifications

Configuring can send email messages when an interworkspace transaction occurs (bringover, putback) or when a workspace is deleted, moved, or reparented. Integrators at the release level set up these notifications so that they are alerted whenever an interesting transaction occurs in workspaces for which they have responsibility.

The SunOS group arranged to have a special email message sent to team members who bring over files from release-level workspaces. The message is a policy statement outlining the restrictions on putting back files to the release level and explaining the release-gateway-developer workspace hierarchy. This message helps ensure that new developers do not bring over source directly from the release level into their private workspaces, from which they will not be allowed to put back to the release level.

#### Controlling Access to Source Files

Controlling access to workspaces is important, particularly at the release level. Configuring, through the `access_control` file in the `Codemgr_wsdata` directory, allows only specified users and net groups to perform transactions with a workspace (bringovers and putbacks) and operations on a workspace (reparentings, deletions, and so on).

Although workspace access can be controlled through Configuring commands, there is no special access control for workspaces at the file system level. Configuring does not limit the ability of a user to change directories to the workspace across the NFS distributed file system, check out a file from SCCS, edit it, and check it back in. This possibility is especially worrisome at the release workspace level, where a developer might accidentally check in a file.

To overcome this problem, the SunOS group selectively sets the permissions of the directories in its workspaces. For example, the `Codemgr_wsdata` directory is writable by anyone because workspace transactions require the ability to write to a locks file in that directory. However, the `usr` directory, which is the root of the source base, is read-only on most file systems so that it cannot be written to casually. It is read-write to only one machine, which developers must log into remotely (`rlogin`) in order to put back. The machine itself can be password protected from remote logins by unauthorized users, but more important is the fact that qualified developers must perform the extra step of logging into the machine before putting back. This extra step protects from accidental source file checkins by ensuring a conscious step on the part of developers before they put back source files.

#### Building and Testing at the Release Level

At the release level, building and testing is performed in a clone workspace just as it is at the integration level. However, procedures at the release level are more formalized because risks to the code base are higher. Figure 5 illustrates the process. By cloning a workspace (\xac ) for test activity, the release workspace itself is free for transactions with integration workspaces.

    Click for closeup.

#### Figure 5 Release Level Building and Testing

The clone workspace is write-only -- it never puts back to the release workspace. If a bug arises during testing, it is evaluated in the clone workspace and narrowed down to an area of functionality corresponding to an integration workspace. The bug is fixed at the integration level and put back to the release level.

A typical bug fix scenario is shown in Figure 6.

#### Figure 6 Release-level Bug Fixing

Here, a bug has been identified as having originated in the gate_1 workspace. A clone is made of`gate_1 (¨)` so that `gate_1` itself can be free for developer transactions. The bug is fixed and tested in `gate_1_clone` (\xad ). | Click for closeup.
The clone is reparented to `train`, and the new code is put back to the release level (\xae ). From release level, the new code is brought over into `train_clone` (\xaf ) where it is tested again (\xb0 ). | Click for closeup.

Because the SunOS operating system is compiled for more than one architecture (SPARC® and Intel® x86 architectures at this writing), a build clone is created for each architecture.

#### Managing Test Releases

If the `train` clone shown in Figure 6 is created for a test release, it can be named `Alpha` or `Beta` and maintained for the period of the test release. If it is created only to establish a stable release workspace, it can be deleted after testing is completed.

#### Managing Multiple Customer Releases

The SunOS group publishes an ongoing set of releases, handling at least two releases concurrently. As one release enters beta testing and First Customer Ship (FCS), development and alpha testing begins on the next release. A modified cascade workspace structure is used to support the releases (see the TW Topic "Workspace Hierarchy Strategies for Software Development and Release on page 37" for a more general discussion of workspace hierarchies).

The release-level workspace `train` is always the top of the current development hierarchy. For each release, `train` is stabilized and cloned. The old `train` workspace is renamed to designate its release number and the clone is renamed `train`. Active integration workspaces are reparented to the new `train`, and development continues on the next release. Figure 7 illustrates the process for Release 3.0.

####

    Figure  7	 Leaving a Release Behind While the train Moves On

    Before Release 3.0

    Release-level workspace train is cloned and
    named train_clone (\xac ).

    Click for closeup.

    After Release 3.0

    Workspace train is renamed Release_3.0, and workspace train_clone is renamed train (\xad ).

    The integration-level children of Release_3.0 are reparented to train (\xae ).

    An alternative reparenting technique, for hierarchies with many integration workspaces, is to name the train clone Release_3.0, then reparent Release_3.0 to Release_2.0 and train to Release_3.0.

    Click for closeup.

By reparenting all the existing development and integration workspaces to the new `train`, any work in progress that was not finished for the last release (the development of a new feature, for example) will automatically be applied to the next release.2

The analogy to a train is apt when the workspace hierarchy is viewed over time. Each release begins as a snapshot of the current `train`, and a string of releases takes on the appearance of a series of boxcars left behind by `train`, which proceeds along the track. From a developer's perspective, "the train" is always the active development workspace with the most recent code, always moving on to the next release (Figure 8).

    Click for closeup.

#### Figure 8 A String of Releases Left Behind by `train`

#### Managing Maintenance Releases

Of course, each release does not remain frozen in time. As bugs are reported against a particular release, child workspaces are created in which to fix the bug. Moreover, a bug fix applied to a shipping release may also be applied to later release workspaces, including `train`. This is the normal case for the SunOS project because each succeeding release is a superset of the previous release.

Note that the arrows between the release workspaces in Figure 8 show a one-way information flow. Bug fixes from earlier releases are brought over into later releases, but no code is ever put back into an earlier release. The major reason for this formality is to avoid introducing incompatible features from a recent release into older releases.

In a similar vein, a child of a late release is never allowed to reparent into the hierarchy of an earlier release because of the risk of putting back a late feature into an earlier release. Figure 9 illustrates what can go wrong.

####

    Figure  9	 Problems in Reparenting From a Recent Release to an Earlier Release

    A senior developer creates a child of Release 3.0 in order to fix a bug (\xac ).

    Some of the code he brings over implements Feature X, unknown to him. Feature X has been developed for Release 3.0, but is not intended for Release 2.0.

    He fixes the bug, which is entirely unrelated to Feature X, and decides to apply his bug fix to Release 2.0.

    Click for closeup.

    He reparents to an integration workspace under Release 2.0 (\xad ) and puts back his changes, inadvertently putting back Feature X at the same time.

    The next time the integration workspace puts back to Release 2.0, it will include Feature X in the files it puts back.

    Click for closeup.

The inadvertent contamination of earlier releases illustrated in Figure 9 is difficult to guard against and can be difficult to fix. The best defense is to strongly discourage such reparenting, as the SunOS group does. Instead, they fix the bug in the earliest release to which it applies and bring the changes forward along the release chain.

#### Bringing Changes Into a Recent Release

When changes are brought over from an early release workspace into a later release, they are almost always brought over into a clone of the later release workspace (not into the later release directly). For example, when the person responsible for release-level integration wants to bring changes from the Release 2.0 workspace into Release 3.0, she creates a clone of Release 3.0, reparents to Release 2.0, and brings over the changes into the clone (Figure 10). After resolving conflicts and thoroughly testing the changes in the Release 3.0 clone, she puts back to the Release 3.0 workspace.

If the build reveals a major problem, the bug is investigated in the clone and narrowed down to one area of functionality in one of the release workspaces. The bug is fixed in the integration workspace where the functionality originated, and the change is put back into the release workspace. The change is then brought over into the build clone, and build and test begins again.

####

    Figure  10	 Bringing Changes from Release 2.0 into Release 3.0

    To bring Release 2.0 changes into Release 3.0, a release-level integrator first creates a clone of Release 3.0 (\xac ).

    Click for closeup.

    Release_3.0_clone is reparented to Release 2.0, and a bringover update is performed on Release_3.0_clone (\xad ).

    If the bringover creates conflicts, they are resolved in Release_3.0_clone.

    After the bringover, Release_3.0_clone becomes a merge of Release 2.0 and Release 3.0.

    Click for closeup.

    After building and testing the source in Release_3.0_clone, the clone is reparented to Release_3.0 and changes are put back.

    Click for closeup.

By merging Releases 2.0 and 3.0 in a clone of Release 3.0, the release-level integrator leaves Release 3.0 free for putbacks from its child bug-fix workspaces. The penalty for allowing these putbacks is that the putback from the clone into Release 3.0 (\xae ) may be blocked. In that case, the integrator must bring over updates from Release 3.0 into the clone, resolve conflicts, build, test, and put back again.

When the early release workspaces are active with bug fixes, the SunOS group merges changes into the current release every two or three days because the group is concerned about bringing bug fixes into recent releases as quickly as possible. And, as with all such merging, the more time that elapses between merges, the more difficult the merge becomes.

## Automating Builds Within Configuring

Maintaining logical, consistent source and makefile hierarchies is important to any large project, whether or not it uses Configuring. Like most large projects, the SunOS project organizes its source files into directories along functional lines. The makefile hierarchy follows the source file organization: each directory contains one default makefile, and each makefile may include other makefiles from parent directories.

Staff engineers established guidelines that encourage developers to observe certain conventions when writing makefiles. For example, developers are requested to use designated environment variables in their makefiles and to include a master makefile that defines system-wide make targets.

To smoothly automate the build process in any workspace, SunOS integration engineers used two mechanisms to provide Configuring with insight into the source and makefile structures: they wrote an initialization script called `ws`, and they provided custom File List Programs (FLPs).

### Organizing the Workspace Directory Structure

The root of each Configuring workspace is defined by the environment variable `CODEMGR_WS`. This variable contains the absolute path name of the workspace, and can be thought of as the relative root of the workspace. Beneath this relative root are three top-level directories:

- `Codemgr_wsdata`, which is found at the top of every Configuring workspace

- `usr`, which is the root of the source base directory tree

- `proto`, a prototype area that contains files used during the linking stage of a build and where new executables are installed

The `proto` directory has no makefiles because no builds are performed directly in it or its subdirectories.

### Organizing the Source Directory Structure

Source files are contained in a directory hierarchy under `$CODEMGR_WS/usr/src`. However, `/usr/src` is never referred to directly by any makefile; instead, it is the value of the `SRC` environment variable, which is used instead. The full path name of the top-level source tree in an active workspace is therefore `$CODEMGR_WS/$SRC`. This indirection allows the entire source tree to be moved without altering any makefile or script.

Figure 11 shows part of the top-level source tree. Besides directories for commands, device drivers, header files, and so on, the top-level directory contains a makefile that is used to build the entire product, and a master makefile that is included in other makefiles in the source tree.

    Click for closeup.

#### Figure 11 Partial Directory Tree for SunOS Source Hierarchy

### Coordinating Makefiles

The SunOS group had the following goals in mind when it wrote its guidelines for writing makefiles:

- The makefiles have a common internal structure to make debugging easy. For example, all makefiles contain the `all` and `install` targets.

- Each makefile includes a higher-level makefile to make sure important variables are inherited. The highest-level makefile, `Makefile`.`master`, is included in all makefiles.

- Builds are as simple as possible, and incremental builds are supported.

- It is easy to detect if and where a build fails.

- The makefiles are optimized to be used with DMake.

An important guideline that realizes these goals is to have makefiles share as much code and as many rules as possible. Because all makefiles include `Makefile`.`master`, any change there (a change to the value of the `CFLAGS` variable, for example) will affect all other makefiles.

The following example shows how the policy of including parent makefiles helps keep each makefile simple. The example shows the build makefile for the `ls` command. It includes two higher-level makefiles, `Makefile`.`cmd` and `Makefile`.`targ`.

`Makefile`.`cmd` contains variables and rules common to the builds of all commands in the `$SRC/cmd` directory. It includes the top-level makefile, `Makefile`.`master`. The second included makefile, `Makefile`.`targ`, specifies system-wide install targets.

    #

    #ident	"@(#)Makefile	1.7	92/12/15 SMI"

    #

    # Copyright (c) 1992 by Sun Microsystems, Inc.

    #

    PROG= ls

    include ../Makefile.cmd

    LDLIBS += -lgen -lw -lintl

    .KEEP_STATE:

    all: $(PROG)

    install: all $(ROOTPROG)

    clean:

    lint:	lint_PROG

    include ../Makefile.targ

#### Figure 12 Makefile for the `ls` Command

### Using DMake

Large software projects typically consist of multiple independent modules that can be built in parallel. DMake supports concurrent processing of targets on a single machine; this concurrence can significantly reduce the time required to build a large project.

Table 1 shows typical times required to build the SunOS source base on three different machines. On a SPARCstationTM 2 system, the use of DMake improves performance by over 30 percent. Larger improvements are realized on higher-performance machines.

#### Table 1 Build Times for SunOS Operating System, Make vs. DMake

Build Machine | Make | DMake | Percent Improvement
SPARCstation 2 | 15 hours | 10 hours | 33
SPARCstation 10 | 10 hours | 4.5 hours | 55
SPARCcenterTM 2000 | 7 hours | 1 hour | 86

### The `ws` Utility

The SunOS group wrote the `ws` utility to automatically initialize the build environment within a Configuring workspace. It configures an environment to build the SunOS executables, sets environment variables for the workspace, and spawns a shell for the environment that has been set up.

During a build, `make` relies on makefiles to define search paths for the proper header files, libraries, and installation directories. Makefiles construct these search paths by absorbing environment variables. For the makefiles to work properly, each developer must set several environment variables. Setting them manually is an error-prone chore for the developer, a chore that is eliminated by `ws`.

Header and library files undergoing development are installed in a workspace directory named `proto`. When `ws` configures the environment, it sets environment variables that define the `proto` directory to be used during the build. If the build involves installing files, `ws` also defines a (perhaps different) `proto` directory as the install target.

The `ws` utility was written as a shell script, which makes it both portable and easy to modify.

#### Using `ws`

Consider the developer responsible for the `ls` command. Only a few source files are required to build the `ls` executable, and these are the files she brings over into her workspace when she begins work (Figure 13). `CODEMGR_WS` is the environment variable that contains the absolute path name of the current workspace.

    Click for closeup.

#### Figure 13 Directory Structure for `ls` Development Workspace

When the developer builds the `ls` executable, she does not want to use the native system library and header files. Instead, she wants to use the project-specific files that have been installed in `proto`. If `proto` is empty or does not exist in her workspace, she must direct `make` to search for the directory in ancestor workspaces during the build. The `ws` utility automates setting of these search paths to ensure that a build always uses the proper libraries and headers.

By setting four environment variables (named `PROTO1`, `PROTO2`, `PROTO3`, and `TERMPROTO`), the developer can set compiler flags to control the build search paths. For the `ls` case, she might want to search for header files and libraries first in the `proto` directory in her development workspace, then in the integration workspace used by the library group, then in her development workspace's parent (integration) workspace, and finally in the release workspace. She would set her `PROTO` variables to search as shown in Figure 14.

    Click for closeup.

#### Figure 14 Example Search Path for Files in `proto` Directory

The search path shown in Figure 14 might provide the developer with the latest versions of the libraries -- those produced by the libraries group -- but might not provide her with the most stable versions. Those versions would presumably be in the release workspace. Or, if she and her fellow developers in the commands group decide to build with the same libraries, she would want those libraries to be in her parent workspace ("commands" in Figure 14).

If the necessary files cannot be found in any `proto` directory, the search can default to the native file system libraries, issue a warning message, or both. Whatever search sequence she decides on, `ws` provides an easy way to specify it.

The following fragment from the `ws` script first shows the search for a `protodefs` file, which the user can edit to set up her `PROTO` environment variables. If no `protodefs` file exists, the `PROTO` variables are set to default values. The script then constructs the `ENVCPPFLAGS` and `ENVLDLIBS` environment variables from the `PROTO` variables. The `ENVCPPFLAGS` and `ENVLDLIBS` variables are absorbed by makefiles to direct the compiling and linking of the program.

####

    Figure  15	 Part of the ws Script That Sets Environment Variables

    ...

    CM_DATA=Codemgr_wsdata

    wsosdir=$CODEMGR_WS/$CM_DATA/sunos

    protofile=$wsosdir/protodefs

    if [ ! -f $protofile ]; then

    	if [ ! -w $CODEMGR_WS/$CM_DATA ]; then

    		#

    		# The workspace doesn't have a protodefs file

    		# and this script cannot create one.

    		# Tell user and use /tmp instead.

    		#

    		echo "Unable to create the proto defaults file

    				($protofile)."

    		# Create a protodefs file in in /tmp

    		wsosdir=/tmp

    		protofile=$wsosdir/protodefs

    	fi

    	if [ ! -d $wsosdir ]; then

    		mkdir $wsosdir

    	fi

    	cat << PROTOFILE_EoF > $protofile

    #!/bin/sh

    #

    #	Set default proto areas for this workspace

    #	NOTE: This file was initially automatically generated.

    #

    #	Feel free to edit this file.  If this file is removed

    #	it will be rebuilt containing default values.

    #

    #	The variable CODEMGR_WS is available to this script.

    #

    #	PROTO1 is the first proto area searched

    #	and is typically set to a proto area associated

    #	with the workspace.  The ROOT environment variable

    #	is set to the same as PROTO1.  If you will be doing

    #	make installs, this proto area needs to be writable.

    #

    #	PROTO2 and PROTO3 are set to proto areas to search

    #	before the search proceeds to the local machine

    #	or the proto area specified by TERMPROTO.

    #

    #	TERMPROTO (if specified) is the last place searched.

    #	If TERMPROTO is not specified the search will end

    #	at the local machine.

    #

    PROTO1=\$CODEMGR_WS/proto

    if [ -f "\$CODEMGR_WS/Codemgr_wsdata/parent" ]; then

    	#

    	# If this workspace has a Configuring parent,

    	# then set PROTO2 to point to the parent's

    	# proto space.

    	#

    	parent=\\Qworkspace parent \$CODEMGR_WS\\Q

    	if [ "\$parent" != "" ]; then

    	   PROTO2=\$parent/proto

    	fi

    fi

    PROTOFILE_EoF

    fi

    . $protofile

    #	The next line means you don't have to type make -e

    #	each time you build

    MAKEFLAGS=e; export MAKEFLAGS

    #

    #	Set up the CPPFLAGS and LDLIBS environment variables

    #

    MACH=\Quname -p\Q

    ROOT=/proto/root_${MACH}                             # default

    ENVCPPFLAGS1=

    ENVCPPFLAGS2=

    ENVCPPFLAGS3=

    ENVCPPFLAGS4=

    ENVLDLIBS1=

    ENVLDLIBS2=

    ENVLDLIBS3=

    if [ "$PROTO1" != "" ]; then       # first proto area specifed

    	ROOT=$PROTO1

    	ENVCPPFLAGS1=-I$ROOT/usr/include

    	export ENVCPPFLAGS1

    	ENVLDLIBS1="-L$ROOT/usr/ccs/lib -L$ROOT/usr/lib"

    	export ENVLDLIBS1

    	if [ "$PROTO2" != "" ]; then

                                       # second proto area specifed

    		ENVCPPFLAGS2=-I$PROTO2/usr/include

    		export ENVCPPFLAGS2

    		ENVLDLIBS2="-L$PROTO2/usr/ccs/lib -L$PROTO2/usr/lib"

    		export ENVLDLIBS2

    		if [ "$PROTO3" != "" ]; then

                                        # third proto area specifed

    			ENVCPPFLAGS3=-I$PROTO3/usr/include

    			export ENVCPPFLAGS3

    			ENVLDLIBS3="-L$PROTO3/usr/ccs/lib \

    							-L$PROTO3/usr/lib"

    			export ENVLDLIBS3

    		fi

    	fi

    fi

    export ROOT

    if [ "$TERMPROTO" != "" ]; then	# fallback area specifed

    	ENVCPPFLAGS4="-Y I,$TERMPROTO/usr/include"

    	export ENVCPPFLAGS4

    	ENVLDLIBS3="$ENVLDLIBS3 -Y
    P,$TERMPROTO/usr/ccs/lib:$TERMPROTO/usr/lib"

    	export ENVLDLIBS3

    fi

    ...

#### Shielding Developers From Unstable Files

If a developer sets up his build search path to refer to a copy of a library or header file located in another workspace's `proto` directory, he is exposed to the changes other developers make to the file. Any change in the file will be immediately visible the next time he performs a build. However, there are times when a developer does not want to see such changes -- when he is diagnosing a bug, for example. A developer can do one of two things to shield himself from changes:

- He can use the file in the `proto` directory of the release workspace during a build. Files there are not updated as often as those in other workspaces, and are typically well tested.

- He can bring over a complete copy of the SunOS source base and build his own `proto` directory to use during builds. This option gives him full control of the `proto` directory used during a build.

### Other Custom Utilities

The SunOS group wrote several other scripts to simplify manipulation of source files controlled by Configuring. The most interesting are `sccsrm` and `sccsmv`.

#### `sccsrm`

Files in workspaces controlled by Configuring can be renamed but not deleted. The reason for this seeming paradox is obvious if you consider that a file that has been deleted from one workspace will reappear when it is put back from a child workspace where it has not been deleted. By renaming a file to a hidden file name -- `.del-filename`, for example -- the file appears to be deleted, and the rename propagates to other workspaces, where the file will also appear to be deleted.

The `sccsrm` script initially renames a file to a `.del*` name in the current directory; another script is run periodically to move `.del*` files to an area in a directory hierarchy parallel to (and with the same structure as) the source hierarchy. See the TW Topic "Deleting Files From the Configuring Tool Workspaces on page 89" for details on this strategy.

Because users are not required to remember the path name of the directory that stores deleted files when they rename a file they want to delete, the `sccsrm` utility helps avoid mistakes. Figure 16 shows the entire script for `sccsrm.`

#### Figure 16 Script for the `sccsrm` Utility

    #!/usr/bin/sh
    USAGE="usage: sccsrm [-f] <filename> ..."
    #
    #ident "@(#)sccsrm								1.4			93/03/30 SMI"
    #
    # This script is to be used to remove files from any CodeManager
    # workspace. It will do this by moving the specified file,
    # and its corresponding s-dot file, to a .del-<file>-\Qdate\Q
    # format.
    #
    # The only way to remove files under the CodeManager is
    # through the rename mechanism - it is not enough to
    # simply \Qrm' the file.
    #
    message() {
    	if [ ${F_FLAG} -eq 0 ]; then
    	echo "$*"
    	fi
    }
    #
    # LC_ALL=C is set so that the this script will work no matter
    # which localization you have installed on your machine. Some
    # localizations can cause the output of \Qdate' and other commands
    # to vary.
    #
    LC_ALL="C"; export LC_ALL
    date=\Q/usr/bin/date +%h-%d-%y\Q
    F_FLAG=0
    #
    # Parse options...
    #
    set -- \Qgetopt f $*\Q
    if [ $? != 0 ]; then
     echo $USAGE
     exit 2
    fi
    for i in $*
    do
     case $i in
     -f) F_FLAG=1; shift;;
     --) shift; break;;
     esac
    done
    if [ $# -eq 0 ]; then
     message $USAGE
     exit 1
    fi
    #
    # Process s-dot files.
    #
    for file in $*
    do
     new_file="${file}-${date}"
     #
     # If there is a deleted file of the same name, append the pid
     # to the name.
     if [ -f SCCS/s..del-${new_file} -o -d .del-${new_file} ]; then
     new_file="${new_file}.$$"
     fi
     if [ -f SCCS/s.$file ]; then
     if [ -f SCCS/p.${file} ]; then
     if [ ${F_FLAG} -eq 0 ]; then
    	echo "warning: ${file} is checked out for editing, \
    	all edits will be lost - continue (y/n)"
    	read ans
    	while [ \Qexpr $ans : "^[YyNn]"\Q -eq 0 ]
    	do
    	echo "warning: ${file} is checked out for editing, \
    	all edits will be lost - continue (y/n)"
    	read ans
    	done
    	else
    	ans="y"
    	fi
    	if [ \Qexpr $ans : "^[Yy]"\Q -eq 1 ]; then
     rm -f SCCS/p.${file}
    	rm -f ${file}
    	else
    	continue
    	fi
     fi
     if [ -f ${file} ]; then
     mv ${file} .del-${new_file}
     fi
     mv SCCS/s.${file} SCCS/s..del-${new_file}
     elif [ -d ${file} -a ${file} != "SCCS" ]; then
     mv ${file} .del-${new_file}
     else
     message "${file}: not an SCCS file"
     fi
    done

#### `sccsmv`

Because moving SCCS-controlled files is so similar to deleting them in the Configuring environment, the utility `sccsmv` was written as a companion to `sccsrm`.

`sccsmv` renames files the same way `sccsrm` does, but with `sccsmv` a user can specify the new name of the file. In practical terms, this utility merely saves the extra step of moving both the g-file (sometimes called the clear file) and the SCCS history file when performing a rename. However, because Configuring acts only on SCCS history files, errors are eliminated by moving them automatically whenever a g-file name is given to `sccsmv`.

### Using File List Programs

When a developer brings over a file into her workspace, she explicitly specifies the file she wants to move. When she specifies a directory, a default File List Program (FLP) sees to it that all the files in the directory are brought over. The default FLP accomplishes this feat by listing the contents of the directory and writing them to `stdout`, from which the bringover and putback commands read them. Users can replace the default FLP with a custom FLP that writes any file names they want to `stdout`, enabling them to bring over (and put back) files without naming them explicitly.

The SunOS group has found custom FLPs to be an easy way to ensure that all the files needed to build an executable are available in a development workspace.

#### Default File List Program

Once again, consider the developer responsible for the `ls` command. With the default FLP (named `def`.`dir`.`flp`) in force, the developer would explicitly bring over the `ls` and `proto` directories, resulting in a development workspace directory structure like the one shown in Figure 17. (The `Codemgr_wsdata` directory is present in every workspace.)

    Click for closeup.

#### Figure 17 Development Workspace for the `ls` Command (Default FLP)

One shortcoming of the workspace shown in Figure 17 is obvious when you consider the makefile for `ls` shown on page 22. That makefile includes two other makefiles, `Makefile`.`cmd` and `Makefile`.`install` from the `cmd` directory. `Makefile`.`cmd`, in turn, includes `Makefile`.`master`. None of these files were brought over by the default FLP, so an attempt to build the `ls` executable will fail in the workspace shown in Figure 17.

A less obvious problem is that other files in the source base may need to be built before `ls`, as determined by the `ls` makefile. If such files exist, they must be brought over explicitly when using the default FLP.

#### Custom File List Program

The SunOS group has customized the default FLP so that it executes scripts that identify the files that are required for a specific build. The required files are listed by scripts named `inc`.`flp` and `req`.`flp`, which are stored in the same directory as the source files required for the build.

Using the custom FLP, the developer specifies only the `/usr/src/cmd/ls` directory in his bringover command. The custom FLP looks in the directory and lists its contents recursively just as the default FLP does. In addition, it looks for a script named `inc`.`flp` or `req`.`flp`. If one or both of these scripts is among the contents of the directory, it is executed. The executed scripts list additional files required to build the executable, and the names are read by the bringover command. In the case of `ls`, the scripts would name the `proto` directory and the makefiles required for the build. Figure 18 shows the resulting workspace.

    Click for closeup.

#### Figure 18 Development Workspace for the `ls` Command (Custom FLP)

The `inc`.`flp` script lists files that do not reside under the current directory but are required in order to build the current directory. Header (include) files are the best examples of such files. The `req`.`flp` script lists other files that are not included but are nevertheless required to build `ls` -- most importantly, makefiles. The `inc`.`flp` script is associated with the source directory structure, while the `req`.`flp` script is associated with the makefile structure.

When a developer brings over `/usr/src/cmd/ls`, the custom FLP starts in the current directory and searches for the `req`.`flp` and `inc`.`flp` scripts. It searches for `req`.`flp` in ancestor directories and for `inc`.`flp` in descendant directories, executing the scripts when it finds them (Figure 19). The `req`.`flp` script lists files that are required for any build in a subdirectory. For example, the `req`.`flp` script in the directory `/usr/src/cmd` writes out `Makefile`.`cmd` and `Makefile`.`install`, which are used to build all commands.

    Click for closeup.

#### Figure 19 The Custom FLP Searches Directories for `req`.`flp` and `inc`.`flp`

Similarly, when the custom FLP executes the `req`.`flp` script in the `/usr/src` directory, it writes out the name `Makefile`.`master`, the makefile that is used to build every executable in the source base.

If the developer brings over `/usr/src/lib/lib`.`c`, the custom FLP encounters a different `req`.`flp`. The `req`.`flp` in `/usr/src/lib` writes out `Makefile`.`lib`, which is required to build every library. `Makefile`.`master` is brought over when the custom FLP executes the `req`.`flp` in the ancestor directory `/usr/src`.

The `inc`.`flp` script performs a similar function for files elsewhere in the file system. When a build depends on files in the brought-over directory, its subdirectories, or in distant directories, `inc.flp` provides the file names to the bringover command.

The custom FLP and the scripts it executes help automate complicated Configuring transactions. However, the scripts themselves are static descriptions of the dependencies of build targets and must be edited when the directory structure or build target dependencies change.

## Registering Public Workspaces With NIS

The SunOS group takes advantage the automounter and the Network Information Service (NIS), both standard parts of the Solaris operating environment, to reduce the amount of information developers must remember daily.

The automounter (`automount`(8)) is a daemon that automatically mounts an NFS directory as needed. It monitors attempts to access directories that are associated with an NIS automount map, along with any directories or files that reside under them. When a user attempts to access a file or directory, the daemon mounts the required NFS file system.

By registering the SunOS public workspaces (integration and release-level) in the automount map, the SunOS group is able to assign them abstract, net-wide names. The abstract names relieve users from having to remember the true path name of workspaces that must be accessed often.

For example, a workspace physically located at
`server:/export/here/there/everywhere/sunos1
`is normally accessed by the automounter with the command
`cd /net/server/export/here/there/everywhere/sunos1
`In contrast, after a map name for the workspace has been entered in the automounter map, SunOS developers can access the same workspace with the command
`cd /workspace/sunos1`

If the physical location of a workspace must be changed because of disk space limitations or other reasons, users do not need to learn the new path name. The new physical location is entered into the automounter map, but the abstract name does not change.

1 Developers working in close collaboration can even reparent to each other's workspaces in order to solve a problem jointly, then put back to the integration workspace when the problem is solved.

2 And, if it is not finished for the next release, it can be applied to the one after that.