Yet Another Makefile Tutorial - Part II

This blog continues where 'Yet Another Makefile Tutorial' left off. Remember sources are found at https://bitbucket.org/FernLeaf_07/makefiletutorial.

NOTE: The make files in the tutorial are NOT POSIX compliant.

We have a general purpose make file, makefile10a. The source is put in the source directory SRC_DIR. Object files are built in OBJ_DIR, and the program is built in TARGET. Includes files are found in SRC_DIR.

For a project with one flat directory of source, makefile10a does its job.

What about projects where there are sub-directories to SRC_DIR?

The problem with makefile10a is that it uses the make function wildcard to find the source files in SRC_DIR. wildcard does not recursively descend SRC_DIR looking for files. We need another function to perform the recursive search.

make does not have such a function, but we can use external commands. The external command to use is find.

BTW, vpath does not perform a recursive search for files. This is not a problem, yet.

find

find is a very complicated program. We are going to use a relatively simple form.

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

find is started by shell. find is told to start the search for *.c files in the SRC_DIR. find recursively searches for files. -type f makes sure that find only retrieves files and not directories. It is rare that a directory could end in *.c, but -type f makes sure that directory is not included.

We update makefile10a to use find.

#makefile11

SRC_DIR := src

OBJ_DIR := obj

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

DEPS := $(OBJS:.o=.d)

TARGET_DIR := target

TARGET := $(TARGET_DIR)/aprogram

vpath %.c $(SRC_DIR)

.PHONY: all clean

all: $(TARGET)

clean:

@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d

@rm -f $(TARGET)

@rmdir $(OBJ_DIR)

@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)

gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c

gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)

_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \

 do              \

 mkdir -p $$d;   \

 done)

-include $(DEPS)

Debugging Make Files

The debugging of make files is difficult. make does have a -d switch which is used for showing how make searches for targets and dependencies.

make does not have a switch to output its variables. The function info that will output trace messages while make is running.

$(info message)

We can use any string or variable to create message.

$(info SRC_LIST=$(SRC_LIST))

$(info OBJS=$(OBJS))

$(info DEPS=$(DEPS))

The function error can be substituted for the function info. When the function error is encountered, the message is output and make stops. warning is also available, but not needed for this debugging tutorial.

$(error STOP HERE!)

makefile11a is updated to use info.

#makefile11a

SRC_DIR := src

OBJ_DIR := obj

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

DEPS := $(OBJS:.o=.d)

$(info SRC_LIST=$(SRC_LIST))

$(info OBJS=$(OBJS))

$(info DEPS=$(DEPS))

TARGET_DIR := target

TARGET := $(TARGET_DIR)/aprogram

vpath %.c $(SRC_DIR)

.PHONY: all clean

all: $(TARGET)

clean:

@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d

@rm -f $(TARGET)

@rmdir $(OBJ_DIR)

@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)

gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c

gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)

_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \

 do              \

 mkdir -p $$d;   \

 done)

-include $(DEPS)

Sub-Directories

We update our example by putting suba.c, subb.c, and subc.c files in the sub-directory sub. We also move all the include files to their own directory INC_DIR.

See updates in makefile12.

#makefile12 - does not work

SRC_DIR := src

OBJ_DIR := obj

INC_DIR := inc

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

DEPS := $(OBJS:.o=.d)

TARGET_DIR := target

TARGET := $(TARGET_DIR)/aprogram

vpath %.c $(SRC_DIR)

.PHONY: all clean

all: $(TARGET)

clean:

@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d

@rm -f $(TARGET)

@rmdir $(OBJ_DIR)

@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)

gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c

gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)

_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \

 do              \

 mkdir -p $$d;   \

 done)

-include $(DEPS)

Unfortunately, makefile12 do not work.

The problem is the old problem of directories do not exist that gcc expects to exist. In this case, the directory obj/sub does not exist.

find found all the source files, complete with sub-directory names, but there are no sub-directories in obj to match those in src.

How to solve this problem?

There are two possible solutions. First, find all the sub-directories in the SRC_LIST of files and create the corresponding sub-directories in OBJ_DIR, or remove all the sub-directory names from the file names in OBJS.

The first solution creates a mirror image of sub-directories in OBJ_DIR found in SRC_DIR.

The second solution will create a flat listing of object files in OBJ_DIR. No necessarily a bad solution.

Both solutions are acceptable. Both solutions shows using additional features of make.

Sub-Directories Under OBJ_DIR

First solution involves using the make functions patsubst, sort, and dir. We saw patsubst in the previous tutorial. sort and dir are new.

dir is a function that extracts the directory portion of a filename.

sort is a function that sorts a list and removes duplicates.

patsubst is the pattern substitution function.

SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

Let's explain the command from the inside out.

$(dir $(SRC_LIST))

This changes SRC_LIST from this

src/main.c src/suba/suba.c src/subb/subb.c src/subc/subc.c 

to this.

src/ src/sub/ src/sub/ src/sub/

There is one entry for each *.c file. We have the same sub-directory three times, in this example. If there were 50 files in a sub-directory, that sub-directory would appear 50 times.

sort is used to remove the duplicates. We really don't care about the sorting feature, we really just want the duplicates removed.

$(sort $(dir $(SRC_LIST)))

changes the SRC_LIST into this.

src/ src/sub/

Now we have a list of the sub directories.

Remove the trailing slash from the directory names. We will see why later.

$(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

Finally, the list of directories where we find source files.

src src/sub

Change src to obj and we have the list directories for object files.

OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))

Change how we create REQUIRED_DIRS and we are done.

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)

makefile13 shows all the updates for the first solution.

#makefile13

SRC_DIR := src

OBJ_DIR := obj

INC_DIR := inc

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

DEPS := $(OBJS:.o=.d)

TARGET_DIR := target

TARGET := $(TARGET_DIR)/aprogram

vpath %.c $(SRC_DIR)

.PHONY: all clean

all: $(TARGET)

clean:

@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d

@rm -f $(TARGET)

@rmdir $(OBJ_DIR)

@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)

gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c

gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)

_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \

 do              \

 mkdir -p $$d;   \

 done)

-include $(DEPS)

We solved the directory problem for the first solution. We now have a mirror image of sub-directories in the OBJ_DIR as is found in SRC_DIR.

Let's check the clean command.

The rm command does not remove directories recursively. We have a sub-directory under obj, sub. The sub directory has to be removed before we can remove obj.

rm has a switch to recursively delete directories recursively

@rm -rf $(OBJ_DIR)
@rm -rf $(TARGET_DIR)

Updating makefile13 the final make file is this.

#makefile13

SRC_DIR := src

OBJ_DIR := obj

INC_DIR := inc

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

DEPS := $(OBJS:.o=.d)

TARGET_DIR := target

TARGET := $(TARGET_DIR)/aprogram

vpath %.c $(SRC_DIR)

.PHONY: all clean

all: $(TARGET)

clean:

@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d

@rm -f $(TARGET)

@rm -rf $(OBJ_DIR)

@rm -rf $(TARGET_DIR)

$(TARGET): $(OBJS)

gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c

gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)

_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \

 do              \

 mkdir -p $$d;   \

 done)

-include $(DEPS)

makefile13 is the final make file. We have sources in SRC_DIR and its sub-directories. We have include files in INC_DIR. We generate object files in OBJ_DIR and mirror image sub-directories of SRC_DIR, and the target is generated in TARGET_DIR.

clean works correctly.

Sub-Directories - Second Solution, Flat OBJ_DIR

The second solution is to create a flat directory of object files.

We want the filenames from the SRC_LIST, not the directories.

The make function notdir extracts the filename, not the directory.

OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))

We now have a list of files object files.

obj/main.o obj/suba.o obj/subb.o obj/subc.o

Unfortunately changing OBJS in makefile12 produces this error.

The problem we have is that make can't find suba.c. It is in a sub-directory. The file name of the pattern is obj/suba.o, not obj/sub/suba.o which is changed to sub/suba.c. With vpath looking in SRC_DIR, the file suba.c is found in SRC_DIR/sub/suba.c.

We need to give vpath the list of source directories. Use SRC_DIR_LIST from makefile13.

SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

So update OBJS, add SRC_DIR_LIST to makefile12, update vpath, we get makefile14.

#makefile14
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))
DEPS := $(OBJS:.o=.d)
$(info OBJS=$(OBJS))
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR_LIST)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)


-include $(DEPS)

This update works.

Update makefile14 for clean the same as makefile13. The final result is this.

#makefile14
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR_LIST)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)

@rm -rf $(OBJ_DIR)

  @rm -rf $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)

The second solution is now complete.

Summary

Fini.

We have two files, makefile13, and makefile14, that are general purpose make files for a project with sources in one directory, object files in another, and the target in a third directory.

Future

The make files handle just one source type *.c. What if you have *.cc, or *.cxx files? 

If you just have *.cc instead of *.c or *.cxx instead of *.c, then simply change the find command to use *.cc or *.cxx.

What if you have both *.c and *.cxx? Typically c files are compiled differently than *.cxx files. There are different switches supplied to gcc.

The next part in this blog on make files will look at updating makefile13 to handle multiple file extensions, groups of files that need different compilation, and excluding source files from the build.

See the next blog. Yet Another Make Tutorial III