8. The Makefile architecture

8. The `Makefile` architecture
Prev |  | Next

## 8. The `Makefile` architecture

make is great if everything works—you type gmake install and lo! the right things get compiled and installed in the right places. Our goal is to make this happen often, but somehow it often doesn't; instead some weird error message eventually emerges from the bowels of a directory you didn't know existed.

The purpose of this section is to give you a road-map to help you figure out what is going right and what is going wrong.

### 8.1. Debugging

Debugging `Makefile`s is something of a black art, but here's a couple of tricks that we find particularly useful. The following command allows you to see the contents of any make variable in the context of the current `Makefile`:

    $ make show VALUE=HS_SRCS

where you can replace `HS_SRCS` with the name of any variable you wish to see the value of.

GNU make has a `-d` option which generates a dump of the decision procedure used to arrive at a conclusion about which files should be recompiled. Sometimes useful for tracking down problems with superfluous or missing recompilations.

### 8.2. A small project

To get started, let us look at the `Makefile` for an imaginary small `fptools` project, `small`. Each project in `fptools` has its own directory in `FPTOOLS_TOP`, so the `small` project will have its own directory `FPOOLS_TOP/small/`. Inside the `small/` directory there will be a `Makefile`, looking something like this:

    # Makefile for fptools project "small"

    TOP = ..
    include $(TOP)/mk/boilerplate.mk

    SRCS = $(wildcard *.lhs) $(wildcard *.c)
    HS_PROG = small

    include $(TOP)/target.mk

this `Makefile` has three sections:

1. The first section includes [2] a file of “boilerplate” code from the level above (which in this case will be `FPTOOLS_TOP/mk/boilerplate.mk`). As its name suggests, `boilerplate.mk` consists of a large quantity of standard `Makefile` code. We discuss this boilerplate in more detail in Section 8.5, “The main `mk/boilerplate.mk` file”.

   Before the `include` statement, you must define the make variable `TOP` to be the directory containing the `mk` directory in which the `boilerplate.mk` file is. It is not OK to simply say

       include ../mk/boilerplate.mk  # NO NO NO

   Why? Because the `boilerplate.mk` file needs to know where it is, so that it can, in turn, `include` other files. (Unfortunately, when an `include`d file does an `include`, the filename is treated relative to the directory in which gmake is being run, not the directory in which the `include`d sits.) In general, every file `foo.mk` assumes that `$(TOP)/mk/foo.mk` refers to itself. It is up to the `Makefile` doing the `include` to ensure this is the case.

   Files intended for inclusion in other `Makefile`s are written to have the following property: after `foo.mk` is `include`d, it leaves `TOP` containing the same value as it had just before the `include` statement. In our example, this invariant guarantees that the `include` for `target.mk` will look in the same directory as that for `boilerplate.mk`.

2. The second section defines the following standard make variables: `SRCS` (the source files from which is to be built), and `HS_PROG` (the executable binary to be built). We will discuss in more detail what the “standard variables” are, and how they affect what happens, in Section 8.8, “The main `mk/target.mk` file”.

   The definition for `SRCS` uses the useful GNU make construct `$(wildcard $pat$)`, which expands to a list of all the files matching the pattern `pat` in the current directory. In this example, `SRCS` is set to the list of all the `.lhs` and `.c` files in the directory. (Let's suppose there is one of each, `Foo.lhs` and `Baz.c`.)

3. The last section includes a second file of standard code, called `target.mk`. It contains the rules that tell gmake how to make the standard targets (Section 7.7, “Standard Targets”). Why, you ask, can't this standard code be part of `boilerplate.mk`? Good question. We discuss the reason later, in Section 8.4, “Boilerplate architecture”.

   You do not have to `include` the `target.mk` file. Instead, you can write rules of your own for all the standard targets. Usually, though, you will find quite a big payoff from using the canned rules in `target.mk`; the price tag is that you have to understand what canned rules get enabled, and what they do (Section 8.8, “The main `mk/target.mk` file”).

In our example `Makefile`, most of the work is done by the two `include`d files. When you say gmake all, the following things happen:

- gmake figures out that the object files are `Foo.o` and `Baz.o`.

- It uses a boilerplate pattern rule to compile `Foo.lhs` to `Foo.o` using a Haskell compiler. (Which one? That is set in the build configuration.)

- It uses another standard pattern rule to compile `Baz.c` to `Baz.o`, using a C compiler. (Ditto.)

- It links the resulting `.o` files together to make `small`, using the Haskell compiler to do the link step. (Why not use ld? Because the Haskell compiler knows what standard libraries to link in. How did gmake know to use the Haskell compiler to do the link, rather than the C compiler? Because we set the variable `HS_PROG` rather than `C_PROG`.)

All `Makefile`s should follow the above three-section format.

### 8.3. A larger project

Larger projects are usually structured into a number of sub-directories, each of which has its own `Makefile`. (In very large projects, this sub-structure might be iterated recursively, though that is rare.) To give you the idea, here's part of the directory structure for the (rather large) GHC project:

    $(FPTOOLS_TOP)/ghc/
      Makefile
      mk/
        boilerplate.mk
        rules.mk
       docs/
        Makefile
        ...source files for documentation...
       driver/
        Makefile
        ...source files for driver...
       compiler/
        Makefile
        parser/...source files for parser...
        renamer/...source files for renamer...
        ...etc...

The sub-directories `docs`, `driver`, `compiler`, and so on, each contains a sub-component of GHC, and each has its own `Makefile`. There must also be a `Makefile` in `$(FPTOOLS_TOP)/ghc`. It does most of its work by recursively invoking gmake on the `Makefile`s in the sub-directories. We say that `ghc/Makefile` is a non-leaf `Makefile`, because it does little except organise its children, while the `Makefile`s in the sub-directories are all leaf `Makefile`s. (In principle the sub-directories might themselves contain a non-leaf `Makefile` and several sub-sub-directories, but that does not happen in GHC.)

The `Makefile` in `ghc/compiler` is considered a leaf `Makefile` even though the `ghc/compiler` has sub-directories, because these sub-directories do not themselves have `Makefile`s in them. They are just used to structure the collection of modules that make up GHC, but all are managed by the single `Makefile` in `ghc/compiler`.

You will notice that `ghc/` also contains a directory `ghc/mk/`. It contains GHC-specific `Makefile` boilerplate code. More precisely:

- `ghc/mk/boilerplate.mk` is included at the top of `ghc/Makefile`, and of all the leaf `Makefile`s in the sub-directories. It in turn `include`s the main boilerplate file `mk/boilerplate.mk`.

- `ghc/mk/target.mk` is `include`d at the bottom of `ghc/Makefile`, and of all the leaf `Makefile`s in the sub-directories. It in turn `include`s the file `mk/target.mk`.

So these two files are the place to look for GHC-wide customisation of the standard boilerplate.

### 8.4. Boilerplate architecture

Every `Makefile` includes a `boilerplate.mk` file at the top, and `target.mk` file at the bottom. In this section we discuss what is in these files, and why there have to be two of them. In general:

- `boilerplate.mk` consists of:

  - Definitions of millions of make variables that collectively specify the build configuration. Examples: `HC_OPTS`, the options to feed to the Haskell compiler; `NoFibSubDirs`, the sub-directories to enable within the `nofib` project; `GhcWithHc`, the name of the Haskell compiler to use when compiling GHC in the `ghc` project.

  - Standard pattern rules that tell gmake how to construct one file from another.

  `boilerplate.mk` needs to be `include`d at the top of each `Makefile`, so that the user can replace the boilerplate definitions or pattern rules by simply giving a new definition or pattern rule in the `Makefile`. gmake simply takes the last definition as the definitive one.

  Instead of replacing boilerplate definitions, it is also quite common to augment them. For example, a `Makefile` might say:

      SRC_HC_OPTS += -O

  thereby adding “`-O`” to the end of `SRC_HC_OPTS`.

- `target.mk` contains make rules for the standard targets described in Section 7.7, “Standard Targets”. These rules are selectively included, depending on the setting of certain make variables. These variables are usually set in the middle section of the `Makefile` between the two `include`s.

  `target.mk` must be included at the end (rather than being part of `boilerplate.mk`) for several tiresome reasons:

  - gmake commits target and dependency lists earlier than it should. For example, `target.mk` has a rule that looks like this:

        $(HS_PROG) : $(OBJS)
              $(HC) $(LD_OPTS) $< -o $@

    If this rule was in `boilerplate.mk` then `$(HS_PROG)` and `$(OBJS)` would not have their final values at the moment gmake encountered the rule. Alas, gmake takes a snapshot of their current values, and wires that snapshot into the rule. (In contrast, the commands executed when the rule “fires” are only substituted at the moment of firing.) So, the rule must follow the definitions given in the `Makefile` itself.

  - Unlike pattern rules, ordinary rules cannot be overriden or replaced by subsequent rules for the same target (at least, not without an error message). Including ordinary rules in `boilerplate.mk` would prevent the user from writing rules for specific targets in specific cases.

  - There are a couple of other reasons I've forgotten, but it doesn't matter too much.

### 8.5. The main `mk/boilerplate.mk` file

If you look at `$(FPTOOLS_TOP)/mk/boilerplate.mk` you will find that it consists of the following sections, each held in a separate file:

`config.mk`

  is the build configuration file we discussed at length in Section 7.3, “Getting the build you want”.

`paths.mk`

  defines make variables for pathnames and file lists. This file contains code for automatically compiling lists of source files and deriving lists of object files from those. The results can be overriden in the `Makefile`, but in most cases the automatic setup should do the right thing.

  The following variables may be set in the `Makefile` to affect how the automatic source file search is done:

  `ALL_DIRS`

    Set to a list of directories to search in addition to the current directory for source files.

  `EXCLUDED_SRCS`

    Set to a list of source files (relative to the current directory) to omit from the automatic search. The source searching machinery is clever enough to know that if you exclude a source file from which other sources are derived, then the derived sources should also be excluded. For example, if you set `EXCLUDED_SRCS` to include `Foo.y`, then `Foo.hs` will also be excluded.

  `EXTRA_SRCS`

    Set to a list of extra source files (perhaps in directories not listed in `ALL_DIRS`) that should be considered.

  The results of the automatic source file search are placed in the following make variables:

  `SRCS`

    All source files found, sorted and without duplicates, including those which might not exist yet but will be derived from other existing sources. `SRCS` can be overriden if necessary, in which case the variables below will follow suit.

  `HS_SRCS`

    all Haskell source files in the current directory, including those derived from other source files (eg. Happy sources also give rise to Haskell sources).

  `HS_OBJS`

    Object files derived from `HS_SRCS`.

  `HS_IFACES`

    Interface files (`.hi` files) derived from `HS_SRCS`.

  `C_SRCS`

    All C source files found.

  `C_OBJS`

    Object files derived from `C_SRCS`.

  `SCRIPT_SRCS`

    All script source files found (`.lprl` files).

  `SCRIPT_OBJS`

    “object” files derived from `SCRIPT_SRCS` (`.prl` files).

  `HSC_SRCS`

    All `hsc2hs` source files (`.hsc` files).

  `HAPPY_SRCS`

    All `happy` source files (`.y` or `.hy` files).

  `OBJS`

    the concatenation of `$(HS_OBJS)`, `$(C_OBJS)`, and `$(SCRIPT_OBJS)`.

  Any or all of these definitions can easily be overriden by giving new definitions in your `Makefile`.

  What, exactly, does `paths.mk` consider a “source file” to be? It's based on the file's suffix (e.g. `.hs`, `.lhs`, `.c`, `.hy`, etc), but this is the kind of detail that changes, so rather than enumerate the source suffices here the best thing to do is to look in `paths.mk`.

`opts.mk`

  defines make variables for option strings to pass to each program. For example, it defines `HC_OPTS`, the option strings to pass to the Haskell compiler. See Section 8.7, “Pattern rules and options”.

`suffix.mk`

  defines standard pattern rules—see Section 8.7, “Pattern rules and options”.

Any of the variables and pattern rules defined by the boilerplate file can easily be overridden in any particular `Makefile`, because the boilerplate `include` comes first. Definitions after this `include` directive simply override the default ones in `boilerplate.mk`.

### 8.6. Platform settings

There are three platforms of interest when building GHC:

The build platform

  The platform on which we are doing this build.

The host platform

  The platform on which these binaries will run.

The target platform

  The platform for which this compiler will generate code.

These platforms are set when running the `configure` script, using the `--build`, `--host`, and `--target` options. The `mk/config.mk` file defines several symbols related to the platform settings (see `mk/config.mk` for details).

We don't currently support build & host being different, because the build process creates binaries that are both run during the build, and also installed.

If host and target are different, then we are building a cross-compiler. For GHC, this means a compiler which will generate intermediate .hc files to port to the target architecture for bootstrapping. The libraries and stage 2 compiler will be built as HC files for the target system (see Section 10, “Porting GHC” for details.

More details on when to use BUILD, HOST or TARGET can be found in the comments in `config.mk`.

### 8.7. Pattern rules and options

The file `suffix.mk` defines standard pattern rules that say how to build one kind of file from another, for example, how to build a `.o` file from a `.c` file. (GNU make's pattern rules are more powerful and easier to use than Unix make's suffix rules.)

Almost all the rules look something like this:

    %.o : %.c
          $(RM) $@
          $(CC) $(CC_OPTS) -c $< -o $@

Here's how to understand the rule. It says that something`.o` (say `Foo.o`) can be built from something`.c` (`Foo.c`), by invoking the C compiler (path name held in `$(CC)`), passing to it the options `$(CC_OPTS)` and the rule's dependent file of the rule `$<` (`Foo.c` in this case), and putting the result in the rule's target `$@` (`Foo.o` in this case).

Every program is held in a make variable defined in `mk/config.mk`—look in `mk/config.mk` for the complete list. One important one is the Haskell compiler, which is called `$(HC)`.

Every program's options are are held in a make variables called `<prog>_OPTS`. the `<prog>_OPTS` variables are defined in `mk/opts.mk`. Almost all of them are defined like this:

    CC_OPTS = \
      $(SRC_CC_OPTS) $(WAY$(_way)_CC_OPTS) $($*_CC_OPTS) $(EXTRA_CC_OPTS)

The four variables from which `CC_OPTS` is built have the following meaning:

`SRC_CC_OPTS`:

  options passed to all C compilations.

`WAY_<way>_CC_OPTS`:

  options passed to C compilations for way `<way>`. For example, `WAY_mp_CC_OPTS` gives options to pass to the C compiler when compiling way `mp`. The variable `WAY_CC_OPTS` holds options to pass to the C compiler when compiling the standard way. (Section 8.10, “Way management” dicusses multi-way compilation.)

`<module>_CC_OPTS`:

  options to pass to the C compiler that are specific to module `<module>`. For example, `SMap_CC_OPTS` gives the specific options to pass to the C compiler when compiling `SMap.c`.

`EXTRA_CC_OPTS`:

  extra options to pass to all C compilations. This is intended for command line use, thus:

      $ gmake libHS.a EXTRA_CC_OPTS="-v"

### 8.8. The main `mk/target.mk` file

`target.mk` contains canned rules for all the standard targets described in Section 7.7, “Standard Targets”. It is complicated by the fact that you don't want all of these rules to be active in every `Makefile`. Rather than have a plethora of tiny files which you can include selectively, there is a single file, `target.mk`, which selectively includes rules based on whether you have defined certain variables in your `Makefile`. This section explains what rules you get, what variables control them, and what the rules do. Hopefully, you will also get enough of an idea of what is supposed to happen that you can read and understand any weird special cases yourself.

`HS_PROG`.

  If `HS_PROG` is defined, you get rules with the following targets:

  `HS_PROG`

    itself. This rule links `$(OBJS)` with the Haskell runtime system to get an executable called `$(HS_PROG)`.

  `install`

    installs `$(HS_PROG)` in `$(bindir)`.

`C_PROG`

  is similar to `HS_PROG`, except that the link step links `$(C_OBJS)` with the C runtime system.

`LIBRARY`

  is similar to `HS_PROG`, except that it links `$(LIB_OBJS)` to make the library archive `$(LIBRARY)`, and `install` installs it in `$(libdir)`.

`LIB_DATA`

  …

`LIB_EXEC`

  …

`HS_SRCS`, `C_SRCS`.

  If `HS_SRCS` is defined and non-empty, a rule for the target `depend` is included, which generates dependency information for Haskell programs. Similarly for `C_SRCS`.

All of these rules are “double-colon” rules, thus

    install :: $(HS_PROG)
          ...how to install it...

GNU make treats double-colon rules as separate entities. If there are several double-colon rules for the same target it takes each in turn and fires it if its dependencies say to do so. This means that you can, for example, define both `HS_PROG` and `LIBRARY`, which will generate two rules for `install`. When you type gmake install both rules will be fired, and both the program and the library will be installed, just as you wanted.

### 8.9. Recursion

In leaf `Makefile`s the variable `SUBDIRS` is undefined. In non-leaf `Makefile`s, `SUBDIRS` is set to the list of sub-directories that contain subordinate `Makefile`s. It is up to you to set `SUBDIRS` in the `Makefile`. There is no automation here—`SUBDIRS` is too important to automate.

When `SUBDIRS` is defined, `target.mk` includes a rather neat rule for the standard targets (Section 7.7, “Standard Targets” that simply invokes make recursively in each of the sub-directories.

These recursive invocations are guaranteed to occur in the order in which the list of directories is specified in `SUBDIRS`. This guarantee can be important. For example, when you say gmake boot it can be important that the recursive invocation of make boot is done in one sub-directory (the include files, say) before another (the source files). Generally, put the most independent sub-directory first, and the most dependent last.

### 8.10. Way management

We sometimes want to build essentially the same system in several different “ways”. For example, we want to build GHC's `Prelude` libraries with and without profiling, so that there is an appropriately-built library archive to link with when the user compiles his program. It would be possible to have a completely separate build tree for each such “way”, but it would be horribly bureaucratic, especially since often only parts of the build tree need to be constructed in multiple ways.

Instead, the `target.mk` contains some clever magic to allow you to build several versions of a system; and to control locally how many versions are built and how they differ. This section explains the magic.

The files for a particular way are distinguished by munging the suffix. The “normal way” is always built, and its files have the standard suffices `.o`, `.hi`, and so on. In addition, you can build one or more extra ways, each distinguished by a way tag. The object files and interface files for one of these extra ways are distinguished by their suffix. For example, way `mp` has files `.mp_o` and `.mp_hi`. Library archives have their way tag the other side of the dot, for boring reasons; thus, `libHS_mp.a`.

A make variable called `way` holds the current way tag. `way` is only ever set on the command line of gmake (usually in a recursive invocation of gmake by the system). It is never set inside a `Makefile`. So it is a global constant for any one invocation of gmake. Two other make variables, `way_` and `_way` are immediately derived from `$(way)` and never altered. If `way` is not set, then neither are `way_` and `_way`, and the invocation of make will build the “normal way”. If `way` is set, then the other two variables are set in sympathy. For example, if `$(way)` is “`mp`”, then `way_` is set to “`mp_`” and `_way` is set to “`_mp`”. These three variables are then used when constructing file names.

So how does make ever get recursively invoked with `way` set? There are two ways in which this happens:

- For some (but not all) of the standard targets, when in a leaf sub-directory, make is recursively invoked for each way tag in `$(WAYS)`. You set `WAYS` in the `Makefile` to the list of way tags you want these targets built for. The mechanism here is very much like the recursive invocation of make in sub-directories (Section 8.9, “Recursion”). It is up to you to set `WAYS` in your `Makefile`; this is how you control what ways will get built.

- For a useful collection of targets (such as `libHS_mp.a`, `Foo.mp_o`) there is a rule which recursively invokes make to make the specified target, setting the `way` variable. So if you say gmake Foo.mp_o you should see a recursive invocation gmake Foo.mp_o way=mp, and in this recursive invocation the pattern rule for compiling a Haskell file into a `.o` file will match. The key pattern rules (in `suffix.mk`) look like this:

      %.$(way_)o : %.lhs
            $(HC) $(HC_OPTS) $< -o $@

  Neat, eh?

- You can invoke make with a particular `way` setting yourself, in order to build files related to a particular `way` in the current directory. eg.

      $ make way=p

  will build files for the profiling way only in the current directory.

### 8.11. When the canned rule isn't right

Sometimes the canned rule just doesn't do the right thing. For example, in the `nofib` suite we want the link step to print out timing information. The thing to do here is not to define `HS_PROG` or `C_PROG`, and instead define a special purpose rule in your own `Makefile`. By using different variable names you will avoid the canned rules being included, and conflicting with yours.

[2] One of the most important features of GNU make that we use is the ability for a `Makefile` to include another named file, very like cpp's `#include` directive.

Prev |  | Next
7. Working with the build system | Home | 9. Building the documentation