AmigaOS GNU Make

Being the avid AmigaOS 4.x coder you may have noticed a couple of versions of GNU make in your SDK right now. Since I am the current maintainer I thought I'd share some of my experiences with the ubiquitous make utility and explain exactly why there are two versions instead of just one in the process.
What is Make?
From the GNU Make Manual, "The make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them."

Once you have moved beyond the usual "Hello World" type programs you'll find you have a whole bunch of source code files which, if you are using C or C++, includes a bunch of header files as well. When your program gets a bit bigger you'll want to subdivide your files into directories as well. Keeping track of which files need to be compiled when other files have been changed is rather boring. Recompiling everything each and every time becomes unrealistic as the program gets bigger and bigger. What you need is some automatic way to recompile only those files which really need to be recompiled and that is what a tool like make will help you to do. Of course the make utility itself is totally ignorant of what you are doing so you need to describe your project via a makefile like this one for Hello World:

hello: hello.c
->gcc -o hello hello.c

This will create an executable named hello using the file named hello.c as input. When hello.c is modified it will compile the file and output the hello executable. Nice and simple.

Beginners should note the '->' in the examples in this article which represents a tab character. Any line which starts with a tab is a command line but it is normally invisible when editing makefiles. This confuses everyone at one point or another.

The make utility has a long history and you'll find thousands of references out there on the internet.
The make utility is rather simple and not very bright. Essentially, it boils down to a string parser that reads file date stamps and executes commands. The trick is to feed make the correct information so it can effect intelligent decisions while executing your commands. Before getting into makefiles and how to correctly specifying dependencies, you need to understand exactly how make executes your commands.

The make utility itself cannot execute commands. What it really does is pass commands from your makefile to the shell which then executes the command. It is the shell that is doing the work, not make. The way in which make chooses a shell to execute your commands with differs depending on which version of make you are using (make or gmake) as supplied in the SDK. The utility named make will always execute commands using the default AmigaOS shell which is built into the operating system. The gmake utility will always execute commands using the abc-shell (named sh) which is a Bourne compatible shell.

Things are a bit more interesting because you also have the option to specify which shell to use to execute commands with via the SHELL variable which is further explained in the GNU make manual. Understanding precisely which shell is being used to execute your commands is absolutely critical because if you use the wrong shell, your makefile is not going to do anything productive for you.
POSIX Projects
The POSIX standard for make specifies the default shell must be sh compatible so most documentation and projects will assume this as well. Non-POSIX operating systems such as AmigaOS may default to any shell they wish which is what make does by default. For your porting projects, gmake is almost always what you want.
There are further differences between make and gmake besides the default shell used to execute commands. The gmake utility is focused on POSIX compliance and tries hard not to include any AmigaOS-specific features. For example, gmake always uses POSIX-style paths which includes . and .. and expects * to be a shell wildcard. The make utility by contrast includes AmigaOS features such as mixed path styles and may include more in the future if requested. For example, the @@ command line continuation token can be used to perform compound commands with make such as:

.PHONY: all
->if exists make @@ \
->->list make @@ \

Now you know which shell is executing the commands in your makefile and which version of make is which. But do you really understand how to specify dependencies correctly? For example, the following makefile is fundamentally broken:

.PHONY: all setup
all: setup out/test

->makedir out

out/test: test.c
->gcc -o out/test test.c

How many of you think make will execute the makedir command before the gcc command? You are both right and wrong. In fact, make can choose which target to build in any order it wishes. It just happens that by default it builds targets in left to right order when performing a non-parallel build. Don't go limiting yourself to one command at a time when you know your Amiga was just born to multi-task!

Parallel builds are supported and can significantly reduce your build times. To use parallel builds, first fix your dependencies to remove any implied ordering problems like in the previous example. Then fire off your builds with the --jobs (or -j) option to specify the maximum number of parallel jobs. How much time you really save depends on a lot of things including the amount of RAM you have, underlying file system, compiler optimization levels, source file coupling and source file sizes. Generally speaking, the larger the project, the more benefit you'll see with parallel compiles.

During my own testing, I found most projects did not benefit from using parallel make when run on a single AmigaOS machine. In fact, it sometimes extended the time by a second or two. If only there was some way for AmigaOS to support multiple CPUs right now you may be thinking.
Want an easy way to record your compile times? AmigaShell includes the new _RunTime variable which gives you the number of seconds used by the last executed command. Here is a small script I created called tmake which will time my make for me and print out the execution time after it is finished:

.key line/m
.bra {
.ket }

make {line}
echo $_RunTime "seconds"
The parallel building option is best exploited when you get into distributed compiles. That is, using a pool of computers in a network to compile your sources in parallel. I'll be exploring this subject further in another article on my experiences with the distcc tools which are also included with your SDK. Feel free to jump in now if you have a couple of AmigaOS machines or perhaps a PC or two with a cross-compiler installed.

Now you are parallel compiling your projects. But not everything is running in parallel. It is very likely you are a victim of the infamous recursive make build system. I don't want to get bogged down in details here but do yourself a favour and read Recursive Make Considered Harmful by Peter Miller. It is not at all a new idea but it seems most projects are addicted to recursive make which only hurts you when you want to reduce your build times with parallel builds. I recently converted one of my own large projects from recursive make to a single makefile and it really wasn't that difficult or that strange in the end. I too was convinced recursive make was the only way especially for large projects but that just isn't true. With recursive makefiles vanquished from my project I can now catcomp, compile, link, etc. all in parallel where before things would get bogged down while one makefile waited for another to finish. Once you start using parallel builds and distcc more you'll notice just how ignorant recursive make really is.

Very shortly after all the parallel compile excitement dies down you'll realize that source code file dependency management is a real pain to do manually. Fortunately, all the tools you need to easily and correctly manage your source code file dependencies are present in your SDK. All you will need is gcc and your favourite GNU make flavour.

The simplest and most complete solution (there are many solutions floating around the internet) is to do something like the following in your project makefile:

SRCS := file1.c file2.c file3.c
CCOPTS := -mcrt=newlib -DNDEBUG -DBLAH

%.o: %.c
->gcc -MM -MP -MT $*.d -MT $@ -MF $*.d $(CCOPTS) $<
->gcc -c $(CCOPTS) $< -o $@

-include $(SRCS:.c=.d)

Now that is a quite a bunch of funny looking options there so I'll go through this line by line in detail to explain things.

SRCS := file.c file2.c file3.c

Nothing too strange here. Note the use of the := operator which is an immediate assignment. An ordinary = operator is a deferred assignment and is almost always not what you intended to say. Using := is not only faster but it also avoids unwanted side effects so it is worth your while to read up on it if you are not already familiar. See the GNU make manual for more details.

CCOPTS := -mcrt=newlib -DNDEBUG -DBLAH

It is important to pass identical options which affect the C/C++ preprocessor to any invocation of the compiler. If you don't your auto-generated dependencies may be incorrect as the dependency generation takes a different #ifdef path in your source files.

%.o: %.c

This is just a pattern rule which is used by default to build .o files from .c files. Are you still using old-fashioned suffix rules (e.g. .o.c:) in your makefiles? Time to get updated cavedude. See the GNU make manual for details.

gcc -MM -MP -MT $*.d -MT $@ -MF $*.d $(CCOPTS) $<

This is the magic part. The -MM option tells the compiler to output only dependency rules suitable for make and at the same time skipping any system includes (e.g. SDK:include). The -MP option will output dummy rules to work around errors make would give you if you had removed a header file. The -MT $*.d option will ensure the target includes the dependency file itself which is important when you delete files from your project. The -MT $@ option just outputs the target file. The -MF $*.d option is the output filename without which output would go to stdout by default. The $(CCOPTS) ensures we pass the same options to the dependency generation as the actual compilation step. The $< is just an automatic make variable representing the .c file itself.

You'll find the gcc options fully documented in the GCC manual. What isn't documented is how you might put them together which is what is covered here. I personally found that by taking a look at the .d files files that this command outputs and playing with the options I was able to fully understand what this command was doing. So before giving up on this wacky looking command, take a look at the .d output files and hopefully things will become much more clear.

gcc -c $(CCOPTS) $< -o $@

This should look familiar to any make aficionado. The $< automatic variable is the .c source file while the $@ automatic variable is the .o target file. Using the -o option is not a requirement but it enables you to specify an alternate directory to place the output file to avoid directory clutter.

-include $(SRCS:.c=.d)

This line includes all those magic .d files that were produced by the gcc -MM... commands. The $(SRCS:.c=.d) part is a common idiom which just substitutes the suffixes in the list of files. If any of the .d files do not yet exist they are simply skipped thanks to the - option in front of the include keyword which is critical when you first run your makefile and no .d files exist yet.

You can build from this basic setup and get more elaborate such as storing all those annoying .d and .o files in a separate directory, etc. There are other dependency generation options as well you may want to play with. Solutions are left to the reader as an exercise.

Of course I didn't come up with this idea in the first place and you can find more information about it by exploring the net. Once article I found quite useful is Advanced Auto-Dependency Generation but it has not been updated for the latest version of gcc included in the OS4 SDK which now has more options for controlling dependency generation. Using those -MT options you can avoid the use of sed and similar tools entirely. I also found Paul's Rules of Makefiles quite useful for laying down a good set of rules for makefiles to live by.

I hope the inclusion of two different but similar GNU make implementations didn't confuse too many of the budding AmigaOS developers out there. I felt it was a necessary evil to satisfy both the native AmigaOS coder and those porting applications from POSIX platforms. Your feedback would be appreciated so feel free to contact me.