The Lost Art of the Makefile

I work on a lot of Javascript projects. The fashion in Javascript is to use build tools like Gulp or Webpack that are written and configured in Javascript. I want to talk about the merits of Make (specifically GNU Make).

Make is a general-purpose build tool that has been improved upon and refined continuously since its introduction over forty years ago. Make is great at expressing build steps concisely and is not specific to Javascript projects. It is very good at incremental builds, which can save a lot of time when you rebuild after changing one or two files in a large project.

Make has been around long enough to have solved problems that newer build tools are only now discovering for themselves.

Despite the title of this post, Make is still widely used. But I think that it is underrepresented in Javascript development. You are more likely to see a Makefile in a C or C++ project, for example.

My guess is that a large portion of the Javascript community did not come from a background of Unix programming, and never had a good opportunity to learn what Make is capable of.

I want to provide a quick primer here; I will go over the contents of the Makefile that I use with my own Javascript projects.

You can find the complete file here.

When to stick with Webpack

The job that Webpack does is quite specialized. If you are writing a frontend app and you need code bundling you should absolutely use Webpack (or a similar tool like Parcel).

On the other hand if your needs are more general Make is a good go-to tool. I use Make when I am writing a client- or server-side library, or a Node app. Those are cases where I do not benefit from the specialized features in Webpack.

Why does Javascript need a build step?

Let’s quickly address the question of why someone would want a build step in a project that does bundle code.

I want to be able to write Stage 4 ECMAScript while targeting browsers or recent stable versions of Node. I also like to include Flow type annotations in my code, and I want to distribute type definitions with my code; but I want the code that I distribute to be plain Javascript. So I use Make to transpile code using Babel.

Introducing the Makefile

Make looks for a file called Makefile in the current directory. A Makefile is a list of tasks that generally look like this:

target_file: prerequisite_file1 prerequisite_file2
	shell command to build target_file (must be indented with tabs, not spaces)
	another shell command (these commands are called the "recipe")

Unless you specify otherwise, Make assumes that the target (target_file in this example) and prerequisites (prerequisite_file1 and prerequisite_file2) are files or directories. You can ask Make to build a target from the command line like this:

$ make target_file

If the target_file does not exist, or if prerequisite_file1 or prerequisite_file2 have been modified since target_file was last built, Make will run the given shell commands. But first Make will check to see if there are recipes in the Makefile for prerequisite_file1 and prerequisite_file2 and build or rebuild those if necessary.

A practical example of a Makefile rule

A minimal project might have a file called src/index.js. We want a rule that tells Make to transpile that file and write the result to lib/index.js. But Make looks at things the other way around: Make expects to be told the desired result, and it uses rules to work out how to produce that result. So we write a Makefile with a rule where the target is lib/index.js and src/index.js is a prerequisite:

lib/index.js: src/index.js
	mkdir -p $(dir $@)
	babel $< --out-file $@ --source-maps

The recipe uses babel to produce lib/index.js using src/index.js as input. The shell commands in a Makefile recipe are almost exactly what you would type in bash - but note that Make substitutes variables and expressions prefixed with $ before commands are executed. You can escape a $ in a recipe command by doubling it (e.g. cd $$HOME). In the recipe above there are two special variables: $< is a shorthand for the list of prerequisites (src/index.js in this case) and $@ is the target (lib/index.js). We will see why those variables are indispensable in a moment.

The mkdir -p line creates the lib/ directory in case it does not already exist. The function dir extracts the directory portion from a file path. So $(dir $@) is read as “the path to the directory that contains the file referenced by $@”.

Generalized rules

When we add more files to the project it would be tedious to write a Makefile target for each Javascript file. A target and it’s prerequisites can include wildcards to create a pattern:

lib/%: src/%
	mkdir -p $(dir $@)
	babel $< --out-file $@ --source-maps

This tells Make that any file path that begins with lib/ can be built using the given steps, and that the target depends on a matching path under src/. Whatever string Make substitutes in the position of the % in the target, it substitutes the same string for % on the prerequisite’s side. Now it becomes clear why the variables $< and $@ are necessary: we won’t know what the values of those variables will be until the rule is invoked.

Why invoke Babel separately for each source file?

Babel can transpile all files in a directory tree with one invocation. But the rule above will run babel separately for every file under src/. There is some startup time overhead every time babel runs; so invoking babel many times is slower when building from a fresh checkout. But thanks to Make’s talent for incremental builds separate invocations make incremental builds much faster. When we ask Make to transpile all files under src/ it will skip files that already have up-to-date results under lib/. I run incremental builds far more often than full builds so I appreciate the speedup!

Edit: several commenters on Hacker News (falcolas, Jtsummers, jlg23, nzoschke) point out that Make can run tasks in parallel. Because Make rules explicitly list dependencies for each target Make knows which tasks can be run in parallel safely. Using the command make --jobs=4 will run up to four instances of Babel at once, which can offset some of the performance loss of running a separate instance of Babel for each source file.

Locating Babel

I have the above rule in my Makefile with one small change:

babel := node_modules/.bin/babel

lib/%: src/%
	mkdir -p $(dir $@)
	$(babel) $< --out-file $@ --source-maps

The babel executable is provided by the babel-cli NPM package. I prefer to install babel-cli as a project dev dependency, which causes babel executable to be installed at the path node_modules/.bin/babel. That way anyone who wants to build my project does not have to take a special step to install babel-cli globally. But then babel will not be in the executable $PATH on most machines. To avoid typing out the path to the executable I assign the location of babel to a variable in the Makefile (babel := node_modules/.bin/babel), and use Make’s variable substitution to splice that path into recipe commands.

(Pro tip: you can add node_modules/.bin to your shell $PATH like this: PATH="node_modules/.bin:$PATH". That makes it easy to run executables installed by dependencies of the project in your current directory. Executables installed with the project will take precedence over executables installed globally. NPM automatically makes this $PATH adjustment when you run NPM scripts. I type out the path to babel in my Makefile because I do not want to assume that other people have made the same $PATH modification, and I do not always run make from an NPM script.)

Transpiling the whole project

With the above rule in place you can transpile a Javascript source file with this command:

$ make lib/index.js  # outputs lib/index.js and lib/index.js.map

Make finds the matching target in your Makefile (lib/%), expands the wildcard, finds the source matching file by expanding src/%, and runs babel. But you probably do not want to run make manually for every source file. What you want is to be able to just type make and have it transpile all source files. Remember that Make needs to be told the results that you want. To do that, first compute a list of all source files and assign it to a variable:

src_files := $(shell find src/ -name '*.js')

The expression on the right side of that assignment uses Make’s built-in shell function to run an external shell command. In this case we use the find command to recursively list all files under src/ that have the extension .js. You could use another command like [fd][] - but find is more likely to be installed on your colleagues’ workstations and on your CI server.

That gives us a list of files that we have. But we need to tell Make which files we want. For every file under src/ we want a transpiled file with a matching path under lib/. We can compute that list by applying Make’s patsubst function to the path of every source file:

transpiled_files := $(patsubst src/%,lib/%,$(src_files))

The substitution expression uses % as a wildcard in the same way as the rule that we wrote earlier.

Now we can define a target that lists the files that we want as prerequisites. When we request that target, Make will automatically build a transpiled result for every source file:

all: $(transpiled_files)

The target name all is special: When you run make with no target specified it will evaluate the all target by default. This is a case where the target is not a file or directory - all is just a label. You should declare non-file targets in your Makefile like this so that Make does not waste time or confuse itself trying to find matching files in your project:

.PHONY: all clean

Oh yeah, you probably want a way to remove build artifacts so that you can build cleanly. With this target you can run make clean to do that:

clean:
	rm -rf lib

Automatically install node modules when package.json changes

Make is powerful enough to accomplish pretty much any task that you can imagine. Do you ever pull updates to a project, and find out after some debugging that you forgot to run yarn install to update your dependencies? You can catch that with Make! When you run yarn install the result is that the node_modules directory is created or updated. You can add a rule for the node_modules target to represent that fact to Make. The state of node_modules depends on the content of package.json and yarn.lock, so those files should be listed as prerequisites:

node_modules: package.json yarn.lock
	yarn install  # could be replaced with `npm install` if you prefer

This change to the all target adds node_modules as a prerequisite:

all: node_modules $(transpiled_files)

Now Make will run yarn install if and only if package.json or yarn.lock has changed since the last build. I put node_modules before $(transpiled_files) just in case new dependencies include items such as updates to Babel modules that might affect the way that project files should be built.

Watch files and rebuild on changes

Every build tool should have a watch-files-for-changes option for rapid development. You can get that effect by pairing Make with a general purpose file-watching tool:

$ yarn global add watch
$ watch make src/

Just make sure that you do not watch lib/ or you will get into an infinite build loop.

Using Make to distribute Flow type definitions

I mentioned that I often use Flow to type-check my projects. I want to distribute plain Javascript, but I also want anyone consuming my library who also uses Flow to benefit from my type annotations. Flow supports that by looking for a files with the .js.flow extension. For example when you import a module called User the Javascript runtime looks for a file called User.js; Flow will additionally look for a file called User.js.flow in the same directory, which should be the original source file with type annotations. My Makefile copies every file under src/ to the corresponding path under lib/ and adds a .flow extension according to this rule:

lib/%.js.flow: src/%.js
	mkdir -p $(dir $@)
	cp $< $@

To make sure that Flow runs this step for all source files I compute the list of .flow files that I expect the same way that we computed the list of transpiled files that we expect:

flow_files := $(patsubst %.js,%.js.flow,$(transpiled_files))

And I include flow_files in the prerequisites of the all task:

all: node_modules $(flow_files) $(transpiled_files)

Going further

Make has many capabilities that I have not touched on here. For example Make supports macros that can compute rules on-the-fly for especially complex use cases. And a Makefile can delegate to targets in other Makefiles, which is useful when distributing Make libraries, or for multi-tiered projects where a build process involves combining artifacts from building multiple subprojects. There is much information to be found in the Gnu Make Manual.