






                 Recursive Make Considered Harmful

                            [4mPeter[24m [4mMiller[0m
                      millerp@canb.auug.org.au



                              [1mABSTRACT[0m
         For large UNIX projects, the traditional method of
         building the project is to use recursive [4mmake.[24m  On
         some  projects,  this results in build times which
         are unacceptably large, when all you want to do is
         change one file.    In examining the source of the
         overly long build times, it became evident that  a
         number of apparently unrelated problems combine to
         produce the delay, but on analysis  all  have  the
         same root cause.
         This paper explores a number of problems regarding
         the use of recursive [4mmake,[24m and shows that they are
         all  symptoms  of the same problem.  Symptoms that
         the UNIX community have long accepted as a fact of
         life,  but  which  need not be endured any longer.
         These problems include recursive [4mmake[24ms which  take
         "forever"  to  work out that they need to do noth-
         ing, recursive [4mmake[24ms which do  too  much,  or  too
         little, recursive [4mmake[24ms which are overly sensitive
         to changes in the source code and require constant
         Makefile intervention to keep them working.
         The  resolution  of these problems can be found by
         looking at what [4mmake[24m does, from first  principles,
         and  then  analyzing  the  effects  of introducing
         recursive [4mmake[24m to  this  activity.   The  analysis
         shows  that  the problem stems from the artificial
         partitioning of the build into  separate  subsets.
         This,  in  turn,  leads to the symptoms described.
         To avoid the symptoms, it  is  only  necessary  to
         avoid the separation; to use a single [4mmake[24m session
         to build the whole project, which is not quite the
         same as a single Makefile.
         This  conclusion  runs counter to much accumulated
         folk wisdom in building large  projects  on  UNIX.
         Some  of  the  main objections raised by this folk
         wisdom are examined and  shown  to  be  unfounded.
         The  results  of actual use are far more encourag-
         ing, with routine development performance improve-
         ments  significantly  faster  than  intuition  may
         indicate, and without the intuitvely expected com-
         promise of modularity.  The use of a whole project
         [4mmake[24m is not as difficult to put into  practice  as
         it may at first appear.






    Peter Miller          31 December 2013                Page 1





    AUUGN'97                   Recursive Make Considered Harmful


               +------------------------------------+
               |Miller, P.A. (1998), [4mRecursive[24m [4mMake[24m |
               |[4mConsidered[24m [4mHarmful,[24m                 |
               |AUUGN Journal of AUUG Inc.,  19(1), |
               |pp. 14-25.                          |
               +------------------------------------+



    [1m1.  Introduction[0m

    For  large  UNIX  software  development projects, the tradi-
    tional methods of building the project use what has come  to
    be  known  as "recursive [4mmake[24m."  This refers to the use of a
    hierarchy of directories containing  source  files  for  the
    modules  which  make  up the project, where each of the sub-
    directories contains a [4mMakefile[24m which  describes  the  rules
    and instructions for the [4mmake[24m program.  The complete project
    build is done by arranging for  the  top-level  Makefile  to
    change directory into each of the sub-directories and recur-
    sively invoke [4mmake.[0m

    This paper explores some  significant  problems  encountered
    when  developing  software projects using the recursive [4mmake[0m
    technique.  A simple solution is offered, and  some  of  the
    implications of that solution are explored.

    Recursive [4mmake[24m results in a directory tree which looks some-
    thing like this:
                          +++
                          ++[4mP[24m+[4mr[24m|[4moject[0m
                           ++++Mmaokdeufliel1e
                           |++ +|Makefile
                           | + +|source1.c
                           | + +|[4metc...[0m
                           ++++m+odule2
                             + +|Makefile
                             + +|source2.c
                             + +|[4metc...[0m

    This hierarchy of modules can be  nested  arbitrarily  deep.
    Real-world  projects  often  use two- and three-level struc-
    tures.

    [1m1.1.  Assumed Knowledge[0m

    This paper assumes that the reader is familiar with develop-
    ing  software  on  UNIX, with the [4mmake[24m program, and with the
    issues of C programming and include file dependencies.

    This paper assumes that you have installed GNU Make on  your
    system  and are moderately familiar with its features.  Some
    -----------
    Copyright (C) 1997 Peter Miller



    Peter Miller          31 December 2013                Page 2





    AUUGN'97                   Recursive Make Considered Harmful


    features of [4mmake[24m described below may not be available if you
    are using the limited version supplied by your vendor.

    [1m2.  The Problem[0m

    There  are  numerous  problems with recursive [4mmake[24m, and they
    are usually observed daily in practice.  Some of these prob-
    lems include:

    +o It is very hard to get the [4morder[24m of the recursion into the
      sub-directories correct.  This order is very unstable  and
      frequently needs to be manually "tweaked."  Increasing the
      number of directories, or  increasing  the  depth  in  the
      directory tree, cause this order to be increasingly unsta-
      ble.

    +o It is often necessary to do more than one  pass  over  the
      sub-directories  to  build  the whole system.  This, natu-
      rally, leads to extended build times.

    +o Because the builds take so long, some dependency  informa-
      tion  is omitted, otherwise development builds take unrea-
      sonable lengths of time, and the developers are  unproduc-
      tive.  This usually leads to things not being updated when
      they need to be, requiring frequent  "clean"  builds  from
      scratch, to ensure everything has actually been built.

    +o Because inter-directory dependencies are either omitted or
      too hard to express, the Makefiles are  often  written  to
      build [4mtoo[24m [4mmuch[24m to ensure that nothing is left out.

    +o The  inaccuracy of the dependencies, or the simple lack of
      dependencies, can result in a product which  is  incapable
      of  building  cleanly,  requiring  the build process to be
      carefully watched by a human.

    +o Related to the above, some projects are incapable of  tak-
      ing  advantage  of various "parallel make" impementations,
      because the build does patently silly things.

    Not all projects experience all of  these  problems.   Those
    that  do  experience  the problems may do so intermittently,
    and dismiss the problems as unexplained  "one  off"  quirks.
    This  paper  attempts  to bring together a range of symptoms
    observed over long practice, and presents a systematic anal-
    ysis and solution.

    It  must be emphasized that this paper does not suggest that
    [4mmake[24m itself is the problem.  This paper is working from  the
    premise  that  [4mmake[24m  does [1mnot [22mhave a bug, that [4mmake[24m does [1mnot[0m
    have a design flaw.  The problem is not in [4mmake[24m at all,  but
    rather  in  the  input given to [4mmake[24m - the way [4mmake[24m is being
    used.




    Peter Miller          31 December 2013                Page 3





    AUUGN'97                   Recursive Make Considered Harmful


    [1m3.  Analysis[0m

    Before it is possible to address these  seemingly  unrelated
    problems, it is first necessary to understand what [4mmake[24m does
    and how it does it.  It is then  possible  to  look  at  the
    effects recursive [4mmake[24m has on how [4mmake[24m behaves.

    [1m3.1.  Whole Project Make[0m

    [4mMake[24m  is  an  expert system.  You give it a set of rules for
    how to construct things, and a  target  to  be  constructed.
    The rules can be decomposed into pair-wise ordered dependen-
    cies between files.  [4mMake[24m takes the rules and determines how
    to  build  the  given target.  Once it has determined how to
    construct the target, it proceeds to do so.

    [4mMake[24m determines how to build the target  by  constructing  a
    [4mdirected[24m  [4macyclic[24m  [4mgraph,[24m  the DAG familiar to many Computer
    Science students.  The vertices of this graph are the  files
    in  the  system,  the edges of this graph are the inter-file
    dependencies.  The edges of the graph are  directed  because
    the  pair-wise  dependencies  are  ordered;  resulting in an
    [4macyclic[24m graph - things which look like loops are resolved by
    the direction of the edges.

    This  paper  will use a small example project for its analy-
    sis.  While the number of files in this  example  is  small,
    there  is  sufficient  complexity  to demonstrate all of the
    above recursive [4mmake[24m problems.  First, however, the  project
    is presented in a non-recursive form.
                           +++
                           ++[4mP[24m+[4mr[24m|[4moject[0m
                            + +|Mmaakienf.icle
                            + +|parse.c
                            + +|parse.h
                              -

    The Makefile in this small project looks like this:

                    +--------------------------+
                    |OBJ = main.o parse.o      |
                    |prog: $(OBJ)              |
                    |  $(CC) -o $@ $(OBJ)      |
                    |main.o: main.c parse.h    |
                    |  $(CC) -c main.c         |
                    |parse.o: parse.c parse.h  |
                    |  $(CC) -c parse.c        |
                    +--------------------------+
    Some  of  the  implicit  rules  of  [4mmake[24m  are presented here
    explicitly, to assist the reader in converting the  Makefile
    into its equivalent DAG.

    The  above  Makefile  can be drawn as a DAG in the following
    form:



    Peter Miller          31 December 2013                Page 4





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[0m
                                prog



                          main.o   parse.o


                      main.c   parse.h  parse.c



    [40mThis is an [4macyclic[24m graph because of the arrows which express[0m
    [40mthe  ordering  of  the  relationship  between the files.  If[0m
    [40mthere [4mwas[24m a circular dependency according to the arrows,  it[0m
    [40mwould be an error.[0m

    [40mNote that the object files (.o) are dependent on the include[0m
    [40mfiles (.h) even though it is the source files (.c) which  do[0m
    [40mthe  including.  This is because if an include file changes,[0m
    [40mit is the object files which are out-of-date, not the source[0m
    [40mfiles.[0m

    [40mThe  second part of what [4mmake[24m does it to perform a [4mpostorder[0m
    [40mtraversal of the DAG.  That is, the dependencies are visited[0m
    [40mfirst.  The actual order of traversal is undefined, but most[0m
    [4m[40mmake[24m implementations work down the graph from left to  right[0m
    [40mfor  edges  below the same vertex, and most projects implic-[0m
    [40mitly rely on this behavior.  The last-time-modified of  each[0m
    [40mfile is examined, and higher files are determined to be out-[0m
    [40mof-date if any of the lower files on which they  depend  are[0m
    [40myounger.   Where a file is determined to be out-of-date, the[0m
    [40maction associated with the relevant graph edge is  performed[0m
    [40m(in the above example, a compile or a link).[0m

    [40mThe  use of recursive [4mmake[24m affects both phases of the opera-[0m
    [40mtion of [4mmake:[24m it causes [4mmake[24m to construct an inaccurate DAG,[0m
    [40mand  it  forces [4mmake[24m to traverse the DAG in an inappropriate[0m
    [40morder.[0m

    [1m[40m3.2.  Recursive Make[0m

    [40mTo examine the effects of recursive [4mmake[24ms, the above example[0m
    [40mwill  be  artificially segmented into two modules, each with[0m
    [40mits own Makefile, and a top-level Makefile  used  to  invoke[0m
    [40meach of the module Makefiles.[0m

    [40mThis example is intentionally artificial, and thoroughly so.[0m
    [40mHowever, all "modularity" of all projects is artificial,  to[0m
    [40msome  extent.  Consider: for many projects, the linker flat-[0m
    [40mtens it all out again, right at the end.[0m

    [40mThe directory structure is as follows:[0m




    Peter Miller          31 December 2013                Page 5





    AUUGN'97                   Recursive Make Considered Harmful


    [40m                      +++[0m
                          [40m++[4mP[24m+[4mr[24m|[4moject[0m
                           [40m++++Maanktefile[0m
                           [40m|++ +|Makefile[0m
                           [40m| - +|main.c[0m
                           [40m++++b+ee[0m
                             [40m+ +|Makefile[0m
                             [40m+ +|parse.c[0m
                             [40m+ +|parse.h[0m

    [40mThe top-level Makefile  often  looks  a  lot  like  a  shell[0m
    [40mscript:[0m

                  [40m+-------------------------------+[0m
                  [40m|MODULES = ant bee              |[0m
                  [40m|all:                           |[0m
                  [40m|  for dir in $(MODULES); do \  |[0m
                  [40m|    (cd $$dir; ${MAKE} all); \ |[0m
                  [40m|  done                         |[0m
                  [40m+-------------------------------+[0m
    [40mThe ant/Makefile looks like this:[0m

                  [40m+------------------------------+[0m
                  [40m|all: main.o                   |[0m
                  [40m|main.o: main.c ../bee/parse.h |[0m
                  [40m|  $(CC) -I../bee -c main.c    |[0m
                  [40m+------------------------------+[0m
    [40mand the equivalent DAG looks like this:[0m
    [40m[0m
                               [40mmain.o[0m



                          [40mmain.c    parse.h[0m

    [40mThe bee/Makefile looks like this:[0m

                   [40m+----------------------------+[0m
                   [40m|OBJ = ../ant/main.o parse.o |[0m
                   [40m|all: prog                   |[0m
                   [40m|prog: $(OBJ)                |[0m
                   [40m|  $(CC) -o $@ $(OBJ)        |[0m
                   [40m|parse.o: parse.c parse.h    |[0m
                   [40m|  $(CC) -c parse.c          |[0m
                   [40m+----------------------------+[0m
    [40mand the equivalent DAG looks like this:[0m











    Peter Miller          31 December 2013                Page 6





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[0m
                              [40mprog[0m



                        [40mmain.o    parse.o[0m


                             [40mparse.h  parse.c[0m



    [40mTake  a  close look at the DAGs.  Notice how neither is com-[0m
    [40mplete - there are vertices and edges  (files  and  dependen-[0m
    [40mcies) missing from both DAGs.  When the entire build is done[0m
    [40mfrom the top level, everything will work.[0m

    [40mBut what happens when small  changes  occur?   For  example,[0m
    [40mwhat would happen if the parse.c and parse.h files were gen-[0m
    [40merated from a parse.y yacc grammar?  This would add the fol-[0m
    [40mlowing lines to the bee/Makefile:[0m

                    [40m+--------------------------+[0m
                    [40m|parse.c parse.h: parse.y  |[0m
                    [40m|  $(YACC) -d parse.y      |[0m
                    [40m|  mv y.tab.c parse.c      |[0m
                    [40m|  mv y.tab.h parse.h      |[0m
                    [40m+--------------------------+[0m
    [40mAnd the equivalent DAG changes to look like this:[0m
    [40m[0m
                              [40mprog[0m



                        [40mmain.o    parse.o[0m


                             [40mparse.h  parse.c[0m



                                  [40mparse.y[0m



    [40mThis  change  has  a  simple  effect:  if parse.y is edited,[0m
    [40mmain.o will [1mnot [22mbe constructed correctly.  This  is  because[0m
    [40mthe DAG for ant knows about only some of the dependencies of[0m
    [40mmain.o, and the DAG for bee knows none of them.[0m

    [40mTo understand why this happens, it is necessary to  look  at[0m
    [40mthe  actions [4mmake[24m will take [4mfrom[24m [4mthe[24m [4mtop[24m [4mlevel.[24m  Assume that[0m
    [40mthe project is in a self-consistent state.  Now edit parse.y[0m
    [40min such a way that the generated parse.h file will have non-[0m



    Peter Miller          31 December 2013                Page 7





    AUUGN'97                   Recursive Make Considered Harmful


    [40mtrivial differences.  However, when the  top-level  [4mmake[24m  is[0m
    [40minvoked,  first ant and then bee is visited.  But ant/main.o[0m
    [40mis [4mnot[24m recompiled, because  bee/parse.h  has  not  yet  been[0m
    [40mregenerated  and  thus  does not yet indicate that main.o is[0m
    [40mout-of-date.  It is not until bee is visited by  the  recur-[0m
    [40msive  [4mmake[24m  that parse.c and parse.h are reconstructed, fol-[0m
    [40mlowed by parse.o.  When the program  is  linked  main.o  and[0m
    [40mparse.o  are  non-trivially incompatible.  That is, the pro-[0m
    [40mgram is [4mwrong.[0m

    [1m[40m3.3.  Traditional Solutions[0m

    [40mThere are three traditional fixes for the above "glitch."[0m

    [1m[40m3.3.1.  Reshuffle[0m

    [40mThe first is to manually tweak the order of the  modules  in[0m
    [40mthe  top-level  Makefile.  But why is this tweak required at[0m
    [40mall?  Isn't [4mmake[24m supposed to be an expert system?   Is  [4mmake[0m
    [40msomehow flawed, or did something else go wrong?[0m

    [40mTo answer this question, it is necessary to look, not at the[0m
    [40mgraphs, but the [4morder[24m [4mof[24m [4mtraversal[24m of the graphs.  In  order[0m
    [40mto operate correctly, [4mmake[24m needs to perform a [4mpostorder[24m tra-[0m
    [40mversal, but in separating the DAG into two pieces, [4mmake[24m  has[0m
    [40mnot  been  [4mallowed[24m  to  traverse  the graph in the necessary[0m
    [40morder - instead the project has dictated an order of traver-[0m
    [40msal.   An order which, when you consider the original graph,[0m
    [40mis plain [4mwrong.[24m  Tweaking the  top-level  Makefile  corrects[0m
    [40mthe order to one similar to that which [4mmake[24m could have used.[0m
    [40mUntil the next dependency is added...[0m

    [40mNote that "make -j" (parallel build) invalidates many of the[0m
    [40mordering  assumptions  implicit  in  the reshuffle solution,[0m
    [40mmaking it useless.  And then there are all  of the sub-makes[0m
    [40mall doing their builds in parallel, too.[0m

    [1m[40m3.3.2.  Repetition[0m

    [40mThe  second  traditional  solution  is to make more than one[0m
    [40mpass in the top-level Makefile, something like this:[0m

                  [40m+-------------------------------+[0m
                  [40m|MODULES = ant bee              |[0m
                  [40m|all:                           |[0m
                  [40m|  for dir in $(MODULES); do \  |[0m
                  [40m|    (cd $$dir; ${MAKE} all); \ |[0m
                  [40m|  done                         |[0m
                  [40m|  for dir in $(MODULES); do \  |[0m
                  [40m|    (cd $$dir; ${MAKE} all); \ |[0m
                  [40m|  done                         |[0m
                  [40m+-------------------------------+[0m





    Peter Miller          31 December 2013                Page 8





    AUUGN'97                   Recursive Make Considered Harmful


    [40mThis doubles the length of time  it  takes  to  perform  the[0m
    [40mbuild.   But that is not all: there is no guarantee that two[0m
    [40mpasses are enough!  The upper bound of the number of  passes[0m
    [40mis  not  even  proportional  to the number of modules, it is[0m
    [40minstead proportional to the  number  of  graph  edges  which[0m
    [40mcross module boundaries.[0m

    [1m[40m3.3.3.  Overkill[0m

    [40mWe  have  already  seen an example of how recursive [4mmake[24m can[0m
    [40mbuild too little, but another common problem is to build too[0m
    [40mmuch.  The third traditional solution to the above glitch is[0m
    [40mto add even [4mmore[24m lines to ant/Makefile:[0m

                    [40m+--------------------------+[0m
                    [40m|.PHONY: ../bee/parse.h    |[0m
                    [40m|../bee/parse.h:           |[0m
                    [40m|    cd ../bee; \          |[0m
                    [40m|    make clean; \         |[0m
                    [40m|    make all              |[0m
                    [40m+--------------------------+[0m
    [40mThis means that whenever main.o is made, parse.h will always[0m
    [40mbe  considered to be out-of-date.  All of bee will always be[0m
    [40mrebuilt including parse.h, and  so  main.o  will  always  be[0m
    [40mrebuilt, [4meven[24m [4mif[24m [4meverything[24m [4mwas[24m [4mself[24m [4mconsistent.[0m

    [40mNote that "make -j" (parallel build) invalidates many of the[0m
    [40mordering assumptions implicit in the overkill solution, mak-[0m
    [40ming  it  useless, because all of the sub-makes are all doing[0m
    [40mtheir builds ("clean" then "all")  in  parallel,  constantly[0m
    [40minterfering with each other in non-deterministic ways.[0m

    [1m[40m4.  Prevention[0m

    [40mThe  above  analysis  is based on one simple action: the DAG[0m
    [40mwas artificially separated  into  incomplete  pieces.   This[0m
    [40mseparation  resulted  in  all  of  the  problems familiar to[0m
    [40mrecursive [4mmake[24m builds.[0m

    [40mDid [4mmake[24m get it wrong?  No.  This is a case of  the  ancient[0m
    [40mGIGO  principle:  [4mGarbage[24m [4mIn,[24m [4mGarbage[24m [4mOut.[24m  Incomplete Make-[0m
    [40mfiles are [4mwrong[24m Makefiles.[0m

    [40mTo avoid these problems, don't break the  DAG  into  pieces;[0m
    [40minstead, use one Makefile for the entire project.  It is not[0m
    [40mthe recursion itself which is harmful, it  is  the  crippled[0m
    [40mMakefiles  which  are used in the recursion which are [4mwrong[24m.[0m
    [40mIt is not a deficiency of [4mmake[24m itself that recursive [4mmake[24m is[0m
    [40mbroken,  it does the best it can with the flawed input it is[0m
    [40mgiven.[0m

         [40m"[4mBut,[24m [4mbut,[24m [4mbut...[24m  [4mYou[24m [4mcan't[24m [4mdo[24m [4mthat![24m" I hear  you[0m
         [40mcry.   "[4mA[24m [4msingle[24m Makefile [4mis[24m [4mtoo[24m [4mbig,[24m [4mit's[24m [4munmain-[0m
         [4m[40mtainable,[24m [4mit's[24m [4mtoo[24m [4mhard[24m [4mto[24m [4mwrite[24m [4mthe[24m [4mrules,[24m [4myou'll[0m



    Peter Miller          31 December 2013                Page 9





    AUUGN'97                   Recursive Make Considered Harmful


    [40m     [4mrun[24m  [4mout[24m [4mof[24m [4mmemory,[24m [4mI[24m [4monly[24m [4mwant[24m [4mto[24m [4mbuild[24m [4mmy[24m [4mlittle[0m
         [4m[40mbit,[24m [4mthe[24m [4mbuild[24m [4mwill[24m [4mtake[24m [4mtoo[24m [4mlong.[24m  [4mIt's[24m [4mjust[24m  [4mnot[0m
         [4m[40mpractical.[24m"[0m

    [40mThese  are  valid  concerns,  and  they frequently lead [4mmake[0m
    [40musers to the conclusion that re-working their build  process[0m
    [40mdoes  not  have any short- or long-term benefits.  This con-[0m
    [40mclusion is based on ancient, enduring, false assumptions.[0m

    [40mThe following sections will address each of  these  concerns[0m
    [40min turn.[0m

    [1m[40m4.1.  A Single [22mMakefile [1mIs Too Big[0m

    [40mIf  the  entire project build description were placed into a[0m
    [40msingle Makefile this would certainly be true, however modern[0m
    [4m[40mmake[24m  implementations have [4minclude[24m statements.  By including[0m
    [40ma relevant fragment from each module, the total size of  the[0m
    [40mMakefile  and  its  include files need be no larger than the[0m
    [40mtotal size of the Makefiles in the recursive case.[0m

    [1m[40m4.2.  A Single [22mMakefile [1mIs Unmaintainable[0m

    [40mThe complexity of using a single  top-level  Makefile  which[0m
    [40mincludes a fragment from each module is no more complex than[0m
    [40min the recursive case.  Because the DAG  is  not  segmented,[0m
    [40mthis  form  of  Makefile becomes less complex, and thus [4mmore[0m
    [40mmaintainable, simply because fewer "tweaks" are required  to[0m
    [40mkeep it working.[0m

    [40mRecursive  Makefiles  have a great deal of repetition.  Many[0m
    [40mprojects solve this by using include files.  By using a sin-[0m
    [40mgle  Makefile  for  the  project,  the need for the "common"[0m
    [40minclude files disappears - the single Makefile is the common[0m
    [40mpart.[0m

    [1m[40m4.3.  It's Too Hard To Write The Rules[0m

    [40mThe only change required is to include the directory part in[0m
    [40mfilenames in a number of places.  This is because  the  [4mmake[0m
    [40mis  performed  from  the  top-level  directory;  the current[0m
    [40mdirectory is not the one in which the file  appears.   Where[0m
    [40mthe  output file is explicitly stated in a rule, this is not[0m
    [40ma problem.[0m

    [40mGCC allows a -o option in conjunction with  the  -c  option,[0m
    [40mand  GNU Make knows this.  This results in the implicit com-[0m
    [40mpilation rule placing  the  output  in  the  correct  place.[0m
    [40mOlder  and dumber C compilers, however, may not allow the -o[0m
    [40moption with the -c option, and will leave the object file in[0m
    [40mthe  top-level  directory ([4mi.e.[24m the wrong directory).  There[0m
    [40mare three ways for you to fix this: get GNU  Make  and  GCC,[0m
    [40moverride  the  built-in  rule  with one which does the right[0m
    [40mthing, or complain to your vendor.[0m



    Peter Miller          31 December 2013               Page 10





    AUUGN'97                   Recursive Make Considered Harmful


    [40mAlso, K&R C compilers will start  the  double-quote  include[0m
    [40mpath  (#include  "[4mfilename.h[24m")  from  the current directory.[0m
    [40mThis will not do what you want.  ANSI C compliant C  compil-[0m
    [40mers,  however,  start the double-quote include path from the[0m
    [40mdirectory in which the source file appears; thus, no  source[0m
    [40mchanges are required.  If you don't have an ANSI C compliant[0m
    [40mC compiler, you should consider installing GCC on your  sys-[0m
    [40mtem as soon as possible.[0m

    [1m[40m4.4.  I Only Want To Build My Little Bit[0m

    [40mMost  of  the  time,  developers are deep within the project[0m
    [40mtree and they edit one or two files and  then  run  [4mmake[24m  to[0m
    [40mcompile  their  changes  and try them out.  They may do this[0m
    [40mdozens or hundreds of times a day.  Being  forced  to  do  a[0m
    [40mfull project build every time would be absurd.[0m

    [40mDevelopers  always have the option of giving [4mmake[24m a specific[0m
    [40mtarget.  This is always the case, it's just that we  usually[0m
    [40mrely  on  the  default target in the Makefile in the current[0m
    [40mdirectory to shorten the command line for us.  Building  "my[0m
    [40mlittle bit" can still be done with a whole project Makefile,[0m
    [40msimply by using a specific target, and an alias if the  com-[0m
    [40mmand line is too long.[0m

    [40mIs  doing  a  full project build every time so absurd?  If a[0m
    [40mchange made in a module has repercussions in other  modules,[0m
    [40mbecause  there  is  a dependency the developer is unaware of[0m
    [40m(but the Makefile is aware of), isn't  it  better  that  the[0m
    [40mdeveloper  find out as early as possible?  Dependencies like[0m
    [40mthis [4mwill[24m be found, because the DAG is more complete than in[0m
    [40mthe recursive case.[0m

    [40mThe  developer is rarely a seasoned old salt who knows every[0m
    [40mone of the million lines  of  code  in  the  product.   More[0m
    [40mlikely the developer is a short-term contractor or a junior.[0m
    [40mYou don't want implications like these to blow up after  the[0m
    [40mchanges are integrated with the master source, you want them[0m
    [40mto blow up on the developer in some nice safe  sand-box  far[0m
    [40maway from the master source.[0m

    [40mIf  you  want to make "just your little" bit because you are[0m
    [40mconcerned that performing a full project build will  corrupt[0m
    [40mthe  project  master  source, due to the directory structure[0m
    [40mused in your project, see the "Projects  [4mversus[24m  Sand-Boxes"[0m
    [40msection below.[0m

    [1m[40m4.5.  The Build Will Take Too Long[0m

    [40mThis  statement  can  be  made from one of two perspectives.[0m
    [40mFirst, that a whole project [4mmake[24m, even  when  everything  is[0m
    [40mup-to-date,  inevitably  takes a long time to perform.  Sec-[0m
    [40mondly, that these inevitable delays are unacceptable when  a[0m
    [40mdeveloper  wants  to  quickly  compile and link the one file[0m



    Peter Miller          31 December 2013               Page 11





    AUUGN'97                   Recursive Make Considered Harmful


    [40mthat they have changed.[0m

    [1m[40m4.5.1.  Project Builds[0m

    [40mConsider a hypothetical project with 1000 source (.c) files,[0m
    [40meach  of which has its calling interface defined in a corre-[0m
    [40msponding include (.h) file with defines,  type  declarations[0m
    [40mand  function  prototypes.   These 1000 source files include[0m
    [40mtheir own interface definition, plus the  interface  defini-[0m
    [40mtions  of any other module they may call.  These 1000 source[0m
    [40mfiles are compiled into 1000 object  files  which  are  then[0m
    [40mlinked  into  an  executable  program.  This system has some[0m
    [40m3000 files which [4mmake[24m must be told about, and be told  about[0m
    [40mthe  include  dependencies, and also explore the possibility[0m
    [40mthat implicit rules (.y -> .c for example) may be necessary.[0m

    [40mIn order to build the DAG, [4mmake[24m must "stat" 3000 files, plus[0m
    [40man additional 2000 files or so, depending on which  implicit[0m
    [40mrules  your  [4mmake[24m  knows  about  and  your Makefile has left[0m
    [40menabled.  On the author's humble 66MHz i486 this takes about[0m
    [40m10  seconds; on native disk on faster platforms it goes even[0m
    [40mfaster.  With NFS over 10MB Ethernet it takes about 10  sec-[0m
    [40monds, no matter what the platform.[0m

    [40mThis  is an astonishing statistic!  Imagine being able to do[0m
    [40ma single file compile, out of 1000 source files, in only  10[0m
    [40mseconds, plus the time for the compilation itself.[0m

    [40mBreaking  the  set of files up into 100 modules, and running[0m
    [40mit as a recursive [4mmake[24m takes about 25 seconds.  The repeated[0m
    [40mprocess  creation  for the subordinate [4mmake[24m invocations take[0m
    [40mquite a long time.[0m

    [40mHang on a minute!  On real-world  projects  with  less  than[0m
    [40m1000 files, it takes an awful lot longer than 25 seconds for[0m
    [4m[40mmake[24m to work out that  it  has  nothing  to  do.   For  some[0m
    [40mprojects,  doing  it in only 25 minutes would be an improve-[0m
    [40mment!  The above result tells us that it is not  the  number[0m
    [40mof  files  which is slowing us down (that only takes 10 sec-[0m
    [40monds), and it is not the repeated process creation  for  the[0m
    [40msubordinate  [4mmake[24m  invocations  (that  only takes another 15[0m
    [40mseconds).  So just what [4mis[24m taking so long?[0m

    [40mThe traditional solutions  to  the  problems  introduced  by[0m
    [40mrecursive [4mmake[24m often increase the number of subordinate [4mmake[0m
    [40minvocations beyond the minimum described here; [4me.g.[24m to  per-[0m
    [40mform multiple repetitions (3.3.2), or to overkill cross-mod-[0m
    [40mule dependencies (3.3.3).  These can take a long time,  par-[0m
    [40mticularly  when combined, but do not account for some of the[0m
    [40mmore spectacular build times; what else is taking so long?[0m

    [40mComplexity of the Makefile is what is taking so long.   This[0m
    [40mis covered, below, in the [4mEfficient[24m [4mMakefiles[24m section.[0m




    Peter Miller          31 December 2013               Page 12





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[1m4.5.2.  Development Builds[0m

    [40mIf, as in the 1000 file example, it only takes 10 seconds to[0m
    [40mfigure out which one of the files needs  to  be  recompiled,[0m
    [40mthere is no serious threat to the productivity of developers[0m
    [40mif they do a whole-project [4mmake[24m as opposed to a  module-spe-[0m
    [40mcific  [4mmake[24m.  The advantage for the project is that the mod-[0m
    [40mule-centric developer is reminded  at  relevant  times  (and[0m
    [40monly  relevant  times)  that  their work has wider ramifica-[0m
    [40mtions.[0m

    [40mBy consistently using C include files which contain accurate[0m
    [40minterface  definitions (including function prototypes), this[0m
    [40mwill produce compilation errors in many of the  cases  which[0m
    [40mwould result in a defective product.  By doing whole-project[0m
    [40mbuilds, developers discover such errors very  early  in  the[0m
    [40mdevelopment  process, and can fix the problems when they are[0m
    [40mleast expensive.[0m

    [1m[40m4.6.  You'll Run Out Of Memory[0m

    [40mThis is the most interesting response.  Once long ago, on  a[0m
    [40mCPU far, far away, it may even have been true.  When Feldman[0m
    [40m[feld78] first wrote [4mmake[24m it was 1978 and  he  was  using  a[0m
    [40mPDP11.  Unix processes were limited to 64KB of data.[0m

    [40mOn  such  a  computer, the above project with its 3000 files[0m
    [40mdetailed in the whole-project Makefile, would  probably  [4mnot[0m
    [40mallow the DAG and rule actions to fit in memory.[0m

    [40mBut  we  are not using PDP11s any more.  The physical memory[0m
    [40mof modern computers exceeds 10MB for  [4msmall[24m  computers,  and[0m
    [40mvirtual  memory  often exceeds 100MB.  It is going to take a[0m
    [40mproject with  hundreds  of  thousands  of  source  files  to[0m
    [40mexhaust  virtual  memory on a [4msmall[24m modern computer.  As the[0m
    [40m1000 source file example takes less  than  100KB  of  memory[0m
    [40m(try  it,  I did) it is unlikely that any project manageable[0m
    [40min a single directory tree on a  single  disk  will  exhaust[0m
    [40myour computer's memory.[0m

    [1m[40m4.7.  Why Not Fix The DAG In The Modules?[0m

    [40mIt  was  shown in the above discussion that the problem with[0m
    [40mrecursive [4mmake[24m is that the DAGs are incomplete.  It  follows[0m
    [40mthat  by  adding the missing portions, the problems would be[0m
    [40mresolved without  abandoning  the  existing  recursive  [4mmake[0m
    [40minvestment.[0m

    +o[40m The  developer needs to remember to do this.  The problems[0m
      [40mwill not affect the  developer  of  the  module,  it  will[0m
      [40maffect the developers of [4mother[24m modules.  There is no trig-[0m
      [40mger to remind the developer to do this, other than the ire[0m
      [40mof fellow developers.[0m




    Peter Miller          31 December 2013               Page 13





    AUUGN'97                   Recursive Make Considered Harmful


    +o[40m It  is  difficult to work out where the changes need to be[0m
      [40mmade.  Potentially every Makefile in  the  entire  project[0m
      [40mneeds  to  be  examined  for  possible  modifications.  Of[0m
      [40mcourse, you can wait for your fellow  developers  to  find[0m
      [40mthem for you.[0m

    +o[40m The include dependencies will be recomputed unnecessarily,[0m
      [40mor will be interpreted incorrectly.  This is because  [4mmake[0m
      [40mis string based, and thus "." and "../ant" are two differ-[0m
      [40ment places, even when you are in the ant directory.   This[0m
      [40mis  of concern when include dependencies are automatically[0m
      [40mgenerated - as they are for all large projects.[0m

    [40mBy making sure that each Makefile is complete, you arrive at[0m
    [40mthe  point  where  the Makefile for at least one module con-[0m
    [40mtains the equivalent of  a  whole-project  Makefile  (recall[0m
    [40mthat these modules form a single project and are thus inter-[0m
    [40mconnected), and there is no need for the recursion any more.[0m

    [1m[40m5.  Efficient Makefiles[0m

    [40mThe central theme of this paper is the [4msemantic[24m side-effects[0m
    [40mof artificially separating a Makefile into the pieces neces-[0m
    [40msary  to perform a recursive [4mmake[24m.  However, once you have a[0m
    [40mlarge number of Makefiles,  the  speed  at  which  [4mmake[24m  can[0m
    [40minterpret this multitude of files also becomes an issue.[0m

    [40mBuilds can take "forever" for both these reasons: the tradi-[0m
    [40mtional fixes for the separated DAG may be building too  much[0m
    [4m[40mand[24m your Makefile may be inefficient.[0m

    [1m[40m5.1.  Deferred Evaluation[0m

    [40mThe text in a Makefile must somehow be read from a text file[0m
    [40mand understood by [4mmake[24m so that the DAG can  be  constructed,[0m
    [40mand  the  specified  actions attached to the edges.  This is[0m
    [40mall kept in memory.[0m

    [40mThe input language for Makefiles is deceptively  simple.   A[0m
    [40mcrucial  distinction  that  often  escapes  both novices and[0m
    [40mexperts alike is that [4mmake[24m's input language is  [4mtext[24m  [4mbased,[0m
    [40mas  opposed  to  token  based,  as is the case for C or AWK.[0m
    [4m[40mMake[24m does the very least possible to process input lines and[0m
    [40mstash them away in memory.[0m

    [40mAs an example of this, consider the following assignment:[0m

                    [40m+--------------------------+[0m
                    [40m|OBJ = main.o parse.o      |[0m
                    [40m+--------------------------+[0m
    [40mHumans  read  this  as  the  variable OBJ being assigned two[0m
    [40mfilenames "main.o" and "parse.o".  But [4mmake[24m does not see  it[0m
    [40mthat  way.   Instead   OBJ  is  assigned  the [4mstring[24m "main.o[0m
    [40mparse.o".  It gets worse:[0m



    Peter Miller          31 December 2013               Page 14





    AUUGN'97                   Recursive Make Considered Harmful


    [40m                +--------------------------+[0m
                    [40m|SRC = main.c parse.c      |[0m
                    [40m|OBJ = $(SRC:.c=.o)        |[0m
                    [40m+--------------------------+[0m
    [40mIn this case humans expect [4mmake[24m to assign two  filenames  to[0m
    [40mOBJ,  but  [4mmake[24m  actually assigns the string "$(SRC:.c=.o)".[0m
    [40mThis is because it is a [4mmacro[24m language with deferred evalua-[0m
    [40mtion, as opposed to one with variables and immediate evalua-[0m
    [40mtion.[0m

    [40mIf this does not seem too problematic, consider the  follow-[0m
    [40ming Makefile:[0m

                   [40m+-----------------------------+[0m
                   [40m|SRC = $(shell echo 'Ouch!' \ |[0m
                   [40m|  1>&2 ; echo *.[cy])        |[0m
                   [40m|OBJ = \                      |[0m
                   [40m|  $(patsubst %.c,%.o,\       |[0m
                   [40m|    $(filter %.c,$(SRC))) \  |[0m
                   [40m|  $(patsubst %.y,%.o,\       |[0m
                   [40m|    $(filter %.y,$(SRC)))    |[0m
                   [40m|test: $(OBJ)                 |[0m
                   [40m|  $(CC) -o $@ $(OBJ)         |[0m
                   [40m+-----------------------------+[0m
    [40mHow  many  times  will the shell command be executed?  [1mOuch![0m
    [40mIt will be executed [4mtwice[24m just to construct the DAG,  and  a[0m
    [40mfurther [4mtwo[24m times if the rule needs to be executed.[0m

    [40mIf  this shell command does anything complex or time consum-[0m
    [40ming (and it usually does) it will  take  [4mfour[24m  times  longer[0m
    [40mthan you thought.[0m

    [40mBut  it  is  worth looking at the other portions of that OBJ[0m
    [40mmacro.  Each time it is named, a huge amount  of  processing[0m
    [40mis performed:[0m

    +o[40m The  argument  to  [4mshell[24m is a single string (all built-in-[0m
      [40mfunctions take a single string argument).  The  string  is[0m
      [40mexecuted  in  a sub-shell, and the standard output of this[0m
      [40mcommand is read back in, translating newlines into spaces.[0m
      [40mThe result is a single string.[0m

    +o[40m The  argument to [4mfilter[24m is a single string.  This argument[0m
      [40mis broken into two strings at the first comma.  These  two[0m
      [40mstrings are then each broken into sub-strings separated by[0m
      [40mspaces.  The first set are the patterns,  the  second  set[0m
      [40mare  the  filenames.   Then,  for each of the pattern sub-[0m
      [40mstrings, if a filename sub-string matches it,  that  file-[0m
      [40mname  is  included  in the output.  Once all of the output[0m
      [40mhas been found, it is re-assembled into  a  single  space-[0m
      [40mseparated string.[0m

    +o[40m The  argument  to [4mpatsubst[24m is a single string.  This argu-[0m
      [40mment is broken into three strings at the first and  second[0m



    Peter Miller          31 December 2013               Page 15





    AUUGN'97                   Recursive Make Considered Harmful


    [40m  commas.   The third string is then broken into sub-strings[0m
      [40mseparated by spaces, these are the filenames.   Then,  for[0m
      [40meach  of  the filenames which match the first string it is[0m
      [40msubstituted according to the second string.  If a filename[0m
      [40mdoes  not match, it is passed through unchanged.  Once all[0m
      [40mof the output has been generated, it is re-assembled  into[0m
      [40ma single space-separated string.[0m

    [40mNotice how many times those strings are disassembled and re-[0m
    [40massembled.  Notice how many  ways  that  happens.   [4mThis[24m  [4mis[0m
    [4m[40mslow.[24m   The  example  here names just two files but consider[0m
    [40mhow inefficient this would be for 1000 files.  Doing it [4mfour[0m
    [40mtimes becomes decidedly inefficient.[0m

    [40mIf  you  are using a dumb [4mmake[24m that has no substitutions and[0m
    [40mno built-in functions, this cannot bite you.  But  a  modern[0m
    [4m[40mmake[24m  has  lots  of  built-in  functions and can even invoke[0m
    [40mshell commands on-the-fly.  The  semantics  of  [4mmake[24m's  text[0m
    [40mmanipulation  is  such  that  string manipulation in [4mmake[24m is[0m
    [40mvery CPU intensive, compared to performing the  same  string[0m
    [40mmanipulations in C or AWK.[0m

    [1m[40m5.2.  Immediate Evaluation[0m

    [40mModern  [4mmake[24m  implementations  have  an immediate evaluation[0m
    [40m":=" assignment operator.  The above example can be re-writ-[0m
    [40mten as[0m

                  [40m+------------------------------+[0m
                  [40m|SRC := $(shell echo 'Ouch!' \ |[0m
                  [40m|  1>&2 ; echo *.[cy])         |[0m
                  [40m|OBJ := \                      |[0m
                  [40m|  $(patsubst %.c,%.o,\        |[0m
                  [40m|    $(filter %.c,$(SRC))) \   |[0m
                  [40m|  $(patsubst %.y,%.o,\        |[0m
                  [40m|    $(filter %.y,$(SRC)))     |[0m
                  [40m|test: $(OBJ)                  |[0m
                  [40m|  $(CC) -o $@ $(OBJ)          |[0m
                  [40m+------------------------------+[0m
    [40mNote  that [4mboth[24m assignments are immediate evaluation assign-[0m
    [40mments.  If the first  were  not,  the  shell  command  would[0m
    [40malways  be  executed  twice.   If  the  second were not, the[0m
    [40mexpensive substitutions would be performed  at  least  twice[0m
    [40mand possibly four times.[0m

    [40mAs  a rule of thumb: always use immediate evaluation assign-[0m
    [40mment unless you knowingly want deferred evaluation.[0m

    [1m[40m5.3.  Include Files[0m

    [40mMany Makefiles perform the same text processing (the filters[0m
    [40mabove,  for  example)  for  every  single  [4mmake[24m run, but the[0m
    [40mresults of the processing rarely change.   Wherever  practi-[0m
    [40mcal,  it is more efficient to record the results of the text[0m



    Peter Miller          31 December 2013               Page 16





    AUUGN'97                   Recursive Make Considered Harmful


    [40mprocessing into a file, and have the Makefile  include  this[0m
    [40mfile.[0m

    [1m[40m5.4.  Dependencies[0m

    [40mDon't  be  miserly  with include files.  They are relatively[0m
    [40minexpensive to read, compared to $(shell),  so  more  rather[0m
    [40mthan less doesn't greatly affect efficiency.[0m

    [40mAs  an  example of this, it is first necessary to describe a[0m
    [40museful feature of GNU Make: once a Makefile  has  been  read[0m
    [40min, if any of its included files were out-of-date (or do not[0m
    [40myet exist), they are re-built, and then [4mmake[24m  starts  again,[0m
    [40mwhich  has  the  result that [4mmake[24m is now working with up-to-[0m
    [40mdate include files.  This feature can be exploited to obtain[0m
    [40mautomatic  include  file  dependency tracking for C sources.[0m
    [40mThe obvious way to implement it, however, has a subtle flaw.[0m

                    [40m+--------------------------+[0m
                    [40m|SRC := $(wildcard *.c)    |[0m
                    [40m|OBJ := $(SRC:.c=.o)       |[0m
                    [40m|test: $(OBJ)              |[0m
                    [40m|  $(CC) -o $@ $(OBJ)      |[0m
                    [40m|include dependencies      |[0m
                    [40m|dependencies: $(SRC)      |[0m
                    [40m|  depend.sh $(CFLAGS) \   |[0m
                    [40m|    $(SRC) > $@           |[0m
                    [40m+--------------------------+[0m
    [40mThe depend.sh script prints lines of the form[0m

         [4m[40mfile[24m.o: [4mfile[24m.c [4minclude[24m.h ...[0m

    [40mThe  most  simple  implementation of this is to use [4mGCC,[24m but[0m
    [40myou will need an equivalent awk script or C program  if  you[0m
    [40mhave a different compiler:[0m

                    [40m+--------------------------+[0m
                    [40m|#!/bin/sh                 |[0m
                    [40m|gcc -MM -MG "$@"          |[0m
                    [40m+--------------------------+[0m
    [40mThis  implementation  of tracking C include dependencies has[0m
    [40mseveral serious flaws, but the one most commonly  discovered[0m
    [40mis  that  the  dependencies file does not, itself, depend on[0m
    [40mthe C include files.  That is, it is not re-built if one  of[0m
    [40mthe  include  files  changes.   There  is no edge in the DAG[0m
    [40mjoining the dependencies vertex to any of the  include  file[0m
    [40mvertices.   If  an  include  file changes to include another[0m
    [40mfile (a nested include), the dependencies will not be recal-[0m
    [40mculated,  and potentially the C file will not be recompiled,[0m
    [40mand thus the program will not be re-built correctly.[0m

    [40mA classic build-too-little problem, caused  by  giving  [4mmake[0m
    [40minadequate  information,  and  thus  causing  it to build an[0m
    [40minadequate DAG and reach the wrong conclusion.[0m



    Peter Miller          31 December 2013               Page 17





    AUUGN'97                   Recursive Make Considered Harmful


    [40mThe traditional solution is to build too much:[0m

                    [40m+--------------------------+[0m
                    [40m|SRC := $(wildcard *.c)    |[0m
                    [40m|OBJ := $(SRC:.c=.o)       |[0m
                    [40m|test: $(OBJ)              |[0m
                    [40m|  $(CC) -o $@ $(OBJ)      |[0m
                    [40m|include dependencies      |[0m
                    [40m|.PHONY: dependencies      |[0m
                    [40m|dependencies: $(SRC)      |[0m
                    [40m|  depend.sh $(CFLAGS) \   |[0m
                    [40m|    $(SRC) > $@           |[0m
                    [40m+--------------------------+[0m
    [40mNow, even if  the  project  is  completely  up-do-date,  the[0m
    [40mdependencies will be re-built.  For a large project, this is[0m
    [40mvery wasteful, and can be a major contributor to [4mmake[24m taking[0m
    [40m"forever" to work out that nothing needs to be done.[0m

    [40mThere  is  a  second problem, and that is that if any [4mone[24m of[0m
    [40mthe C files changes, [4mall[24m of the C files will  be  re-scanned[0m
    [40mfor  include dependencies.  This is as inefficient as having[0m
    [40ma Makefile which reads[0m

                    [40m+--------------------------+[0m
                    [40m|prog: $(SRC)              |[0m
                    [40m|  $(CC) -o $@ $(SRC)      |[0m
                    [40m+--------------------------+[0m
    [40mWhat is needed, in exact analogy to the C case, is  to  have[0m
    [40man  intermediate form.  This is usually given a ".d" suffix.[0m
    [40mBy exploiting the fact that more than one file may be  named[0m
    [40min  an  include  line, there is no need to "link" all of the[0m
    [40m".d" files together:[0m

                    [40m+--------------------------+[0m
                    [40m|SRC := $(wildcard *.c)    |[0m
                    [40m|OBJ := $(SRC:.c=.o)       |[0m
                    [40m|test: $(OBJ)              |[0m
                    [40m|  $(CC) -o $@ $(OBJ)      |[0m
                    [40m|include $(OBJ:.o=.d)      |[0m
                    [40m|%.d: %.c                  |[0m
                    [40m|  depend.sh $(CFLAGS) \   |[0m
                    [40m|    $*.c > $@             |[0m
                    [40m+--------------------------+[0m

    [40mThis has one more thing to fix:  just  as  the  object  (.o)[0m
    [40mfiles  depend  on the source files and the include files, so[0m
    [40mdo the dependency (.d) files.[0m

         [4m[40mfile[24m.d [4mfile[24m.o: [4mfile[24m.c [4minclude[24m.h[0m

    [40mThis means tinkering with the depend.sh script again:[0m






    Peter Miller          31 December 2013               Page 18





    AUUGN'97                   Recursive Make Considered Harmful


    [40m            +-----------------------------------+[0m
                [40m|#!/bin/sh                          |[0m
                [40m|gcc -MM -MG "$@" |                 |[0m
                [40m|sed -e 's@^\(.*\)\.o:@\1.d \1.o:@' |[0m
                [40m+-----------------------------------+[0m

    [40mThis method of determining include file dependencies results[0m
    [40min  the  Makefile  including  more  files  than the original[0m
    [40mmethod, but opening files is less expensive than  rebuilding[0m
    [40mall  of  the dependencies every time.  Typically a developer[0m
    [40mwill edit one or two files before re-building;  this  method[0m
    [40mwill  rebuild  the  [4mexact[24m  dependency file affected (or more[0m
    [40mthan one, if you edited an include file).  On balance,  this[0m
    [40mwill use less CPU, and less time.[0m

    [40mIn  the case of a build where nothing needs to be done, [4mmake[0m
    [40mwill actually do  nothing,  and  will  work  this  out  very[0m
    [40mquickly.[0m

    [40mHowever,  the above technique assumes your project fits eni-[0m
    [40mtrely within the one directory.  For  large  projects,  this[0m
    [40musually  isn't  the  case.   This  means  tinkering with the[0m
    [40mdepend.sh script again:[0m

                [40m+-----------------------------------+[0m
                [40m|#!/bin/sh                          |[0m
                [40m|DIR="$1"                           |[0m
                [40m|shift 1                            |[0m
                [40m|case "$DIR" in                     |[0m
                [40m|"" | ".")                          |[0m
                [40m|gcc -MM -MG "$@" |                 |[0m
                [40m|sed -e 's@^\(.*\)\.o:@\1.d \1.o:@' |[0m
                [40m|;;                                 |[0m
                [40m|*)                                 |[0m
                [40m|gcc -MM -MG "$@" |                 |[0m
                [40m|sed -e "s@^\(.*\)\.o:@$DIR/\1.d \  |[0m
                [40m|$DIR/\1.o:@"                       |[0m
                [40m|;;                                 |[0m
                [40m|esac                               |[0m
                [40m+-----------------------------------+[0m
    [40mAnd the rule needs to change, too, to pass the directory  as[0m
    [40mthe first argument, as the script expects.[0m

                    [40m+---------------------------+[0m
                    [40m|%.d: %.c                   |[0m
                    [40m|  depend.sh `dirname $*` \ |[0m
                    [40m|    $(CFLAGS) $*.c > $@    |[0m
                    [40m+---------------------------+[0m
    [40mNote  that  the  .d  files will be relative to the top level[0m
    [40mdirectory.  Writing them so that they can be used  from  any[0m
    [40mlevel is possible, but beyond the scope of this paper.[0m






    Peter Miller          31 December 2013               Page 19





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[1m5.5.  Multiplier[0m

    [40mAll of the inefficiencies described in this section compound[0m
    [40mtogether.  If you do 100 Makefile interpretations, once  for[0m
    [40meach module, checking 1000 source files can take a very long[0m
    [40mtime - if the interpretation requires complex processing  or[0m
    [40mperforms  unnecessary  work, or both.  A whole project [4mmake[24m,[0m
    [40mon the other hand, only needs to interpret  a  single  Make-[0m
    [40mfile.[0m

    [1m[40m6.  Projects [4m[22mversus[24m [1mSand-boxes[0m

    [40mThe  above discussion assumes that a project resides under a[0m
    [40msingle directory tree, and this is often  the  ideal.   How-[0m
    [40mever,  the realities of working with large software projects[0m
    [40moften lead to weird and wonderful  directory  structures  in[0m
    [40morder  to  have  developers working on different sections of[0m
    [40mthe project without taking complete copies and thereby wast-[0m
    [40ming precious disk space.[0m

    [40mIt  is  possible to see the whole-project [4mmake[24m proposed here[0m
    [40mas impractical, because it does not match the evolved  meth-[0m
    [40mods of your development process.[0m

    [40mThe  whole-project [4mmake[24m proposed here does have an effect on[0m
    [40mdevelopment methods: it can give  you  cleaner  and  simpler[0m
    [40mbuild  environments  for  your  developers.  By using [4mmake[24m's[0m
    [40mVPATH feature, it is possible to copy only those  files  you[0m
    [40mneed  to  edit  into  your private work area, often called a[0m
    [4m[40msand-box.[0m

    [40mThe simplest explanation of what VPATH does is  to  make  an[0m
    [40manalogy  with  the  include file search path specified using[0m
    [40m-I[4mpath[24m options to the  C  compiler.   This  set  of  options[0m
    [40mdescribes  where to look for files, just as VPATH tells [4mmake[0m
    [40mwhere to look for files.[0m

    [40mBy using VPATH, it is possible to "stack"  the  sand-box  [4mon[0m
    [4m[40mtop[24m [4mof[24m the project master source, so that files in the sand-[0m
    [40mbox take precedence, but it is the union of  all  the  files[0m
    [40mwhich [4mmake[24m uses to perform the build.[0m
    [47m[0m[40m                  +          +[0m
                     [40m+[4m[0m[4mM[24m[40m+[4m[0m[4master[24m [4mSour[24m[40m+[4m[0m[4mc[24m[40m+[4m[0m[4me[0m
                     [40m+   [0mmain.c [40m+   [4m[0m[4mCombined[24m [4mView[0m
                    [40m+   [0mparse.y[40m+       [0mmain.c
                     [4m[47mSand-Box[24m    [40m+     [0mparse.y
                      [47mmain.c    [40m++   [0mvariable.c
                                [40m+[0m
                    [47mvariable.c [40m+[0m


    [40mIn  this  environment, the sand-box has the same tree struc-[0m
    [40mture as the project master source.  This  allows  developers[0m
    [40mto  safely  change  things  across separate modules, [4me.g.[24m if[0m



    Peter Miller          31 December 2013               Page 20





    AUUGN'97                   Recursive Make Considered Harmful


    [40mthey are changing a module interface.  It  also  allows  the[0m
    [40msand-box  to be physically separate - perhaps on a different[0m
    [40mdisk, or under their home directory.   It  also  allows  the[0m
    [40mproject master source to be read-only, if you have (or would[0m
    [40mlike) a rigorous check-in procedure.[0m

    [40mNote: in addition to adding a VPATH line to your development[0m
    [40mMakefile, you will also need to add -I options to the CFLAGS[0m
    [40mmacro, so that the C compiler uses the  same  path  as  [4mmake[0m
    [40mdoes.   This  is  simply done with a 3-line Makefile in your[0m
    [40mwork area - set a macro, set the VPATH, and then include the[0m
    [40mMakefile from the project master source.[0m

    [1m[40m6.1.  VPATH Semantics[0m

    [40mFor  the above discussion to apply, you need to use GNU make[0m
    [40m3.76 or later.  For versions of GNU Make earlier than  3.76,[0m
    [40myou  will  need  Paul  Smith's  VPATH+  patch.   This may be[0m
    [40mobtained from  ftp://ftp.wellfleet.com/netman/psmith/gmake/.[0m

    [40mThe  POSIX  semantics  of  VPATH are slightly brain-dead, so[0m
    [40mmany other [4mmake[24m implementations are too  limited.   You  may[0m
    [40mwant to consider installing GNU Make.[0m

    [1m[40m7.  The Big Picture[0m

    [40mThis  section  brings  together all of the preceding discus-[0m
    [40msion, and presents the example  project  with  its  separate[0m
    [40mmodules,  but  with a whole-project Makefile.  The directory[0m
    [40mstructure is changed little from the recursive case,  except[0m
    [40mthat  the  deeper  Makefiles are replaced by module specific[0m
    [40minclude files:[0m
                          [40m+++[0m
                          [40m++[4mP[24m+[4mr[24m|[4moject[0m
                           [40m++++Maanktefile[0m
                           [40m|++ +|module.mk[0m
                           [40m| - +|main.c[0m
                           [40m++++b+ee[0m
                           [40m| + +|module.mk[0m
                           [40m| + +|parse.y[0m
                           [40m+ +|depend.sh[0m


    [40mThe Makefile looks like this:[0m

                  [40m+-------------------------------+[0m
                  [40m|MODULES := ant bee             |[0m
                  [40m|# look for include files in    |[0m
                  [40m|#   each of the modules        |[0m
                  [40m|CFLAGS += $(patsubst %,-I%,\   |[0m
                  [40m|  $(MODULES))                  |[0m
                  [40m|# extra libraries if required  |[0m
                  [40m|LIBS :=                        |[0m
                  [40m+-------------------------------+[0m



    Peter Miller          31 December 2013               Page 21





    AUUGN'97                   Recursive Make Considered Harmful


    [40m              +-------------------------------+[0m
                  [40m|# each module will add to this |[0m
                  [40m|SRC :=                         |[0m
                  [40m|# include the description for  |[0m
                  [40m|#   each module                |[0m
                  [40m|include $(patsubst %,\         |[0m
                  [40m|    %/module.mk,$(MODULES))    |[0m
                  [40m|# determine the object files   |[0m
                  [40m|OBJ :=                    \    |[0m
                  [40m|  $(patsubst %.c,%.o,     \    |[0m
                  [40m|    $(filter %.c,$(SRC))) \    |[0m
                  [40m|  $(patsubst %.y,%.o,     \    |[0m
                  [40m|    $(filter %.y,$(SRC)))      |[0m
                  [40m|# link the program             |[0m
                  [40m|prog: $(OBJ)                   |[0m
                  [40m|  $(CC) -o $@ $(OBJ) $(LIBS)   |[0m
                  [40m|# include the C include        |[0m
                  [40m|#   dependencies               |[0m
                  [40m|include $(OBJ:.o=.d)           |[0m
                  [40m|# calculate C include          |[0m
                  [40m|#   dependencies               |[0m
                  [40m|%.d: %.c                       |[0m
                  [40m|  depend.sh `dirname $*.c` \   |[0m
                  [40m|    $(CFLAGS) $*.c > $@        |[0m
                  [40m+-------------------------------+[0m
    [40mThis looks absurdly large, but it has all of the common ele-[0m
    [40mments  in  the  one place, so that each of the modules' [4mmake[0m
    [40mincludes may be small.[0m

    [40mThe ant/module.mk file looks like:[0m

                    [40m+--------------------------+[0m
                    [40m|SRC += ant/main.c         |[0m
                    [40m+--------------------------+[0m
    [40mThe bee/module.mk file looks like:[0m

                    [40m+--------------------------+[0m
                    [40m|SRC += bee/parse.y        |[0m
                    [40m|LIBS += -ly               |[0m
                    [40m|%.c %.h: %.y              |[0m
                    [40m|  $(YACC) -d $*.y         |[0m
                    [40m|  mv y.tab.c $*.c         |[0m
                    [40m|  mv y.tab.h $*.h         |[0m
                    [40m+--------------------------+[0m

    [40mNotice that the built-in rules are used for the C files, but[0m
    [40mwe  need  special  yacc  processing  to get the generated .h[0m
    [40mfile.[0m

    [40mThe savings in this example  look  irrelevant,  because  the[0m
    [40mtop-level  Makefile is so large.  But consider if there were[0m
    [40m100 modules, each with only a  few  non-comment  lines,  and[0m
    [40mthose specifically relevant to the module.  The savings soon[0m
    [40madd up to a total size often [4mless[24m [4mthan[24m the  recursive  case,[0m



    Peter Miller          31 December 2013               Page 22





    AUUGN'97                   Recursive Make Considered Harmful


    [40mwithout loss of modularity.[0m

    [40mThe equivalent DAG of the Makefile after all of the includes[0m
    [40mlooks like this:[0m
    [40m[47m[40m[0m
                                [40mprog[0m



                          [47mmain.o   parse.o[0m
                            [40mmain.d|  parse.d|[0m

                      [40mmain.c   parse.h  parse.c[0m



                                   [40mparse.y[0m



    [40mThe vertexes and edges for the include file dependency files[0m
    [40mare also present as these are important for [4mmake[24m to function[0m
    [40mcorrectly.[0m

    [1m[40m7.1.  Side Effects[0m

    [40mThere are a couple of desirable side-effects of using a sin-[0m
    [40mgle Makefile.[0m

    [40m+o  The  GNU  Make -j option, for parallel builds, works even[0m
    [40mbetter than before.  It can find even more unrelated  things[0m
    [40mto do at once, and no longer has some subtle problems.[0m

    [40m+o The general make -k option, to continue as far as possible[0m
    [40meven in the face of errors, works even better  than  before.[0m
    [40mIt can find even more things to continue with.[0m

    [1m[40m8.  Literature Survey[0m

    [40mHow  can  it be possible that we have been misusing [4mmake[24m for[0m
    [40m20 years?  How can it be possible that  behavior  previously[0m
    [40mascribed to [4mmake[24m's limitations is in fact a result of misus-[0m
    [40ming it?[0m

    [40mThe author only started thinking about the  ideas  presented[0m
    [40min  this  paper when faced with a number of ugly build prob-[0m
    [40mlems on utterly different projects, but  with  common  symp-[0m
    [40mtoms.   By  stepping  back from the individual projects, and[0m
    [40mclosely examining the thing they had  in  common,  [4mmake[24m,  it[0m
    [40mbecame  possible  to see the larger pattern.  Most of us are[0m
    [40mtoo caught up in the minutiae of  just  getting  the  rotten[0m
    [40mbuild  to  work that we don't have time to spare for the big[0m
    [40mpicture.  Especially when the item in  question  "obviously"[0m
    [40mworks, and has done so continuously for the last 20 years.[0m



    Peter Miller          31 December 2013               Page 23





    AUUGN'97                   Recursive Make Considered Harmful


    [40mIt  is  interesting  that the problems of recursive [4mmake[24m are[0m
    [40mrarely mentioned in the very books Unix programmers rely  on[0m
    [40mfor accurate, practical advice.[0m

    [1m[40m8.1.  The Original Paper[0m

    [40mThe  original  [4mmake[24m  paper [feld78] contains no reference to[0m
    [40mrecursive [4mmake,[24m let alone any discussion as to the  relative[0m
    [40mmerits of whole project [4mmake[24m over recursive [4mmake.[0m

    [40mIt is hardly surprising that the original paper did not dis-[0m
    [40mcuss recursive [4mmake[24m, Unix projects at the time  usually  [4mdid[0m
    [40mfit into a single directory.[0m

    [40mIt  may  be this which set the "one Makefile in every direc-[0m
    [40mtory" concept so firmly in the collective  Unix  development[0m
    [40mmind-set.[0m

    [1m[40m8.2.  GNU Make[0m

    [40mThe GNU Make manual [stal93] contains several pages of mate-[0m
    [40mrial concerning recursive [4mmake,[24m however  its  discussion  of[0m
    [40mthe  merits or otherwise of the technique are limited to the[0m
    [40mbrief statement that[0m

         [40m"This technique is useful when you want  to  sepa-[0m
         [40mrate makefiles for various subsystems that compose[0m
         [40ma larger system."[0m

    [40mNo mention is made of the problems you may encounter.[0m

    [1m[40m8.3.  Managing Projects with Make[0m

    [40mThe Nutshell Make book [talb91] specifically promotes recur-[0m
    [40msive [4mmake[24m over whole project [4mmake[24m because[0m

         [40m"The  cleanest  way  to build is to put a separate[0m
         [40mdescription file in each directory, and  tie  them[0m
         [40mtogether  through  a  master description file that[0m
         [40minvokes [4mmake[24m recursively.  While  cumbersome,  the[0m
         [40mtechnique  is  easier  to  maintain than a single,[0m
         [40menormous file that covers  multiple  directories."[0m
         [40m(p. 65)[0m

    [40mThis  is  despite the book's advice only two paragraphs ear-[0m
    [40mlier that[0m

         [40m"[4mmake[24m is happiest when you keep all your files  in[0m
         [40ma single directory." (p. 64)[0m

    [40mYet the book fails to discuss the contradiction in these two[0m
    [40mstatements, and goes on to describe one of  the  traditional[0m
    [40mways  of  treating the symptoms of incomplete DAGs caused by[0m
    [40mrecursive [4mmake[24m.[0m



    Peter Miller          31 December 2013               Page 24





    AUUGN'97                   Recursive Make Considered Harmful


    [40mThe book may give us a clue as to  why  recursive  [4mmake[24m  has[0m
    [40mbeen  used  in  this  way for so many years.  Notice how the[0m
    [40mabove quotes confuse the concept of  a  directory  with  the[0m
    [40mconcept of a Makefile.[0m

    [40mThis  paper suggests a simple change to the mind-set: direc-[0m
    [40mtory trees, however deep, are places to store  files;  Make-[0m
    [40mfiles are places to describe the relationships between those[0m
    [40mfiles, however many.[0m

    [1m[40m8.4.  BSD Make[0m

    [40mThe tutorial for BSD Make [debo88] says nothing at all about[0m
    [40mrecursive  [4mmake[24m,  but  it  is  one of the few which actually[0m
    [40mdescribed, however briefly, the relationship between a Make-[0m
    [40mfile and a DAG (p. 30).  There is also a wonderful quote[0m

         [40m"If  [4mmake[24m doesn't do what you expect it to, it's a[0m
         [40mgood chance the makefile is wrong." (p. 10)[0m

    [40mWhich is a pithy summary of the thesis of this paper.[0m

    [1m[40m9.  Summary[0m

    [40mThis paper presents a number of related problems, and demon-[0m
    [40mstrates  that  they are not inherent limitations of [4mmake[24m, as[0m
    [40mis commonly believed,  but  are  the  result  of  presenting[0m
    [40mincorrect  information to [4mmake[24m.  This is the ancient [4mGarbage[0m
    [4m[40mIn,[24m [4mGarbage[24m [4mOut[24m principle at work.  Because  [4mmake[24m  can  only[0m
    [40moperate  correctly with a complete DAG, the error is in seg-[0m
    [40mmenting the Makefile into incomplete pieces.[0m

    [40mThis requires a shift in thinking: directory [4mtrees[24m are  sim-[0m
    [40mply a place to hold files, Makefiles are a place to remember[0m
    [40mrelationships between files.  Do not confuse the two because[0m
    [40mit is as important to accurately represent the relationships[0m
    [40mbetween files in different directories as it is to represent[0m
    [40mthe relationships between files in the same directory.  This[0m
    [40mhas the implication that there should be exactly  one  Make-[0m
    [40mfile for a project, but the magnitude of the description can[0m
    [40mbe managed by using a [4mmake[24m include file in each directory to[0m
    [40mdescribe  the subset of the project files in that directory.[0m
    [40mThis is just as modular as having a Makefile in each  direc-[0m
    [40mtory.[0m

    [40mThis  paper  has shown how a project build and a development[0m
    [40mbuild can be equally brief for a whole-project [4mmake[24m.   Given[0m
    [40mthis  parity  of time, the gains provided by accurate depen-[0m
    [40mdencies mean that this process will, in fact, be faster than[0m
    [40mthe recursive [4mmake[24m case, and more accurate.[0m







    Peter Miller          31 December 2013               Page 25





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[1m9.1.  Inter-dependent Projects[0m

    [40mIn organizations with a strong culture of re-use, implement-[0m
    [40ming whole-project [4mmake[24m can present  challenges.   Rising  to[0m
    [40mthese challenges, however, may require looking at the bigger[0m
    [40mpicture.[0m

    +o[40m A module may be shared between two  programs  because  the[0m
      [40mprograms  are  closely related.  Clearly, the two programs[0m
      [40mplus the shared module belong to  the  same  project  (the[0m
      [40mmodule  may  be self-contained, but the programs are not).[0m
      [40mThe dependencies must be explicitly stated, and changes to[0m
      [40mthe  module  must result in both programs being recompiled[0m
      [40mand re-linked as appropriate.  Combining them all  into  a[0m
      [40msingle  project  means  that whole-project [4mmake[24m can accom-[0m
      [40mplish this.[0m

    +o[40m A module may be shared between two projects  because  they[0m
      [40mmust  inter-operate.  Possibly your project is bigger than[0m
      [40myour current directory structure implies.   The  dependen-[0m
      [40mcies  must be explicitly stated, and changes to the module[0m
      [40mmust result in both  projects  being  recompiled  and  re-[0m
      [40mlinked  as  appropriate.  Combining them all into a single[0m
      [40mproject means that whole-project [4mmake[24m can accomplish this.[0m

    +o[40m It  is  the  normal  case  to  omit the edges between your[0m
      [40mproject and the operating system or other installed  third[0m
      [40mparty tools.  So normal that they are ignored in the Make-[0m
      [40mfiles in this paper, and they are ignored in the  built-in[0m
      [40mrules of [4mmake[24m programs.[0m
      [40mModules shared between your projects may fall into a simi-[0m
      [40mlar category: if they change, you  will  deliberately  re-[0m
      [40mbuild  to  include their changes, or quietly include their[0m
      [40mchanges whenever the next build  may  happen.   In  either[0m
      [40mcase,  you  do  not explicitly state the dependencies, and[0m
      [40mwhole-project [4mmake[24m does not apply.[0m

    +o[40m Re-use may be better served if the module were used  as  a[0m
      [40mtemplate,  and  divergence between two projects is seen as[0m
      [40mnormal.  Duplicating the module in each project allows the[0m
      [40mdependencies  to  be explicitly stated, but requires addi-[0m
      [40mtional effort if maintenance is  required  to  the  common[0m
      [40mportion.[0m

    [40mHow to structure dependencies in a strong re-use environment[0m
    [40mthus becomes an exercise in [4mrisk[24m [4mmanagement[24m.   What  is  the[0m
    [40mdanger  that  omitting  chunks  of  the  DAG  will harm your[0m
    [40mprojects?  How vital is it to rebuild if a  module  changes?[0m
    [40mWhat  are  the consequences of [4mnot[24m rebuilding automatically?[0m
    [40mHow can you tell when a rebuild is necessary if  the  depen-[0m
    [40mdencies  are  not  explicitly  stated?   What are the conse-[0m
    [40mquences of forgetting to rebuild?[0m





    Peter Miller          31 December 2013               Page 26





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[1m9.2.  Return On Investment[0m

    [40mSome of the techniques presented in this paper will  improve[0m
    [40mthe speed of your builds, even if you continue to use recur-[0m
    [40msive [4mmake[24m.  These are not the focus of this paper, merely  a[0m
    [40museful detour.[0m

    [40mThe  focus  of this paper is that you will get more accurate[0m
    [40mbuilds of your project if you use whole-project [4mmake[24m  rather[0m
    [40mthan recursive [4mmake[24m.[0m

    +o[40m The  time  for  [4mmake[24m  to work out that nothing needs to be[0m
      [40mdone will not be more, and will often be less.[0m

    +o[40m The size and complexity of the total Makefile  input  will[0m
      [40mnot be more, and will often be less.[0m

    +o[40m The  total  Makefile  input is no less modular than in the[0m
      [40mresursive case.[0m

    +o[40m The difficulty of maintaining  the  total  Makefile  input[0m
      [40mwill not be more, and will often be less.[0m

    [40mThe disadvantages of using whole-project [4mmake[24m over recursive[0m
    [4m[40mmake[24m are often un-measured.  How much time is spent figuring[0m
    [40mout  why  [4mmake[24m  did  something unexpected?  How much time is[0m
    [40mspent figuring out that [4mmake[24m [1mdid [22msomething unexpected?   How[0m
    [40mmuch  time is spent tinkering with the build process?  These[0m
    [40mactivities are often  thought  of  as  "normal"  development[0m
    [40moverheads.[0m

    [40mBuilding  your  project is a fundamental activity.  If it is[0m
    [40mperforming poorly, so are development, debugging  and  test-[0m
    [40ming.  Building your project needs to be so simple the newest[0m
    [40mrecruit can do it immediately with only  a  single  page  of[0m
    [40minstructions.   Building  your project needs to be so simple[0m
    [40mthat it rarely needs any development effort at all.  Is your[0m
    [40mbuild process this simple?[0m



















    Peter Miller          31 December 2013               Page 27





    AUUGN'97                   Recursive Make Considered Harmful


    [40m[1m10.  References[0m


         [1m[40mdebo88: [22mAdam de Boor (1988).  [4mPMake[24m [4m-[24m [4mA[24m [4mTutorial[24m.  Uni-[0m
    [40mversity of California, Berkeley[0m

         [1m[40mfeld78: [22mStuart I. Feldman (1978).  [4mMake[24m [4m-[24m [4mA[24m [4mProgram[24m [4mfor[0m
    [4m[40mMaintaining[24m  [4mComputer[24m [4mPrograms[24m.  Bell Laboratories Computing[0m
    [40mScience Technical Report 57[0m

         [1m[40mstal93: [22mRichard M. Stallman and Roland McGrath  (1993).[0m
    [4m[40mGNU[24m [4mMake:[24m [4mA[24m [4mProgram[24m [4mfor[24m [4mDirecting[24m [4mRecompilation[24m.  Free Soft-[0m
    [40mware Foundation, Inc.[0m

         [1m[40mtalb91: [22mSteve Talbott (1991).  [4mManaging[24m  [4mProjects[24m  [4mwith[0m
    [4m[40mMake,[24m [4m2nd[24m [4mEd[24m.  O'Reilly & Associates, Inc.[0m

    [1m[40m11.  About the Author[0m

    [40mPeter  Miller  has worked for many years in the software R&D[0m
    [40mindustry, principally on UNIX systems. In that time  he  has[0m
    [40mwritten  tools  such as Aegis (a software configuration man-[0m
    [40magement system) and Cook (yet  another  [4mmake[24m-oid),  both  of[0m
    [40mwhich  are freely available on the Internet.  Supporting the[0m
    [40muse of these tools  at  many  Internet  sites  provided  the[0m
    [40minsights which led to this paper.[0m

    [40mPlease  visit  http://www.canb.auug.org.au/~millerp/  if you[0m
    [40mwould like to look at some of the author's free software.[0m




























    Peter Miller          31 December 2013               Page 28


