Monday, July 27, 2015

Using GDB with a target processor that does not have hardware breakpoints.

--This is an expansion of a previous blog GDB and Symbol Tables.--

GNU Debugger, GDB is one of the most popular debuggers today.

It has a lot of features. One of those features that doesn't have a lot discussion are the convenience variables and user-defined commands.

I work with firmware. GDB can be used with processors that are not the host processor where you write and compiler your code. However, additional GDB commands must be used to provide this feature.

I found a very good use for both of these neglected features when working on firmware on legacy (old) hardware.

Motorla 68000

Microprocessors today support hardware breakpoints. The microprocessor provides these breakpoints as part of its structure, no additional support or tools required, JTAG enabled and all that rot.

But what happens when you have to work with legacy devices. The Motorola 68000 series microprocessor is such an example.

When the M68K was first introduced, the tool to use was an ICE, In-Circuit Emulator. A large connector, the same as the CPU, was inserted into the circuit board instead of the M68K, and an M68K was placed on top of this connector. The ICE device then controlled ALL the pins of the M68K. The ICE also had RAM that mirrored the memory space on the circuit board. In essence, a complete clone.

ICEs were expensive. One could still use an ICE today, if you can find one, and it is still working. :)

A simpler method was developed for the M68K, and later families of Motorola processors (today Freescale), BDM, Background Debug Mode.

BDM is a serial bus dedicated to debugging supported by a simple header.

Using BDM, one can read and write registers, read and write RAM, read from ROM and reflash programs, but it has very limited instruction debugging. This makes debugging complex programs difficult. BDM was a predecessor to JTAG.

Enter the convenience variables and user-defined commands.

GDB Server

To use GDB with any target hardware, one needs a GDB server to translate the GDB commands to those supported by the target hardware, along with sending the commands over a wire from the host to the target and back.

This is done in two ways, one is with a software GDB server, and one is with a hardware GDB server. With a software GDB server, a GDB server program is run on the host and it talks to the target via dedicated hardware. With a hardware GDB server, there is a hardware device that communicates with host, via Ethernet or RS-232 and has a built-in GDB server as well as its own set of commands. No software running on the host.

In either choice, you must inform GDB where the GDB server is located.

.GDBINIT

.gdbinit is the default file that GDB reads before it starts debugging your program. If the file doesn't exist, GDB doesn't complain about the missing file, but GDB won't talk to the hardware where you want to debug your program.

.gdbinit is a file that establishes the environment you want GDB to use with the target. You specify the DDR memory register values. You specify valid memory address ranges. You specify chip selects, etc. Anything that needs to be specified to be able to read and write to registers and memory of the target hardware.

Your program does all this, but GDB needs to talk to the hardware before your program starts.

In addition to configuring the remote target hardware, GDB needs to know where the hardware is.

target remote 192.168.0.10

This would be a typical command line added to .gdbinit, one of the first lines of the file.

One does not have use the file name .gdbinit. The command switch -x <filename> can be used to specify a file other than .gdbinit. On Linux systems, .gdbinit will be a hidden file. On Windows systems it is sometimes difficult to create a file with a period.


Abatron BDI2000

Abatron, of Switzerland, produces a device, BDI2000, that supports the Motorola BDM and the CPU32 family of Motorola processors.

The Abatron BDI2000 is an example of a hardware GDB server.

Abatron BDI2000 User's Manual

[BTW, this device is no longer manufactured. Abatron has announced in 2015 that they will be closing their doors. There are several listings on eBay. ]

 The BDI2000 supports two native commands, ti and tc.

The ti command steps to the next machine instruction.

The tc command runs until there is a change in the flow of instructions. Essentially stop when there is a jump instruction.

The BDI2000 does not disassemble machine instructions. We use GDB to perform this task.

The Motorola 68332 starts at location 0. The first 32-bit value is top of the stack. The second 32-bit value is the address of the first instruction.

Upon 'reset' using the BDI2000, the 68332 program counter is at location 0x4.

How does one debug code written for this hardware?

How does one use GDB with such hardware?

You say "Can't you just use software breakpoints?". Yes, if the program is running out of RAM. If the program runs out of flash or EEProm, software breakpoints are not possible. A software break point requires the modification of the break location to replace the desired instruction with a BREAK interrupt instruction. Flash does not allow this substitution.

An alternative is to substitute a RAM chip for the Flash chip and download the program with each power on.

But let's say we don't want to change the hardware. We need to test and debug it the way it is.

Today with C/C++ compilers debugging in assembly is only for the most hard core programmers or difficult hardware bugs.

Convenience Variables and User-Defined Commands

Using convenience variables and user-defined commands, we are going to create break point commands, break on memory change, and other commands.

The GDB command that allows access to the native BDI2000 commands is monitor or mon for short.

To list the M68K registers, enter

 mon rd

GDB has the equivalent

  info reg

But mon rd has a very compact presentation. GDB presents each register on a separate line.

mon reset will perform a soft reset of the CPU.

The BDI2000 converts the GDB command si into ti and the ni into ti. The normal n and s commands do not work as they expect native hardware breakpoints or software breakpoints.

We will simulate s and n with user-defined commands. The user-defined commands are added to .gdbinit.

User-Define Commands

The format of a user-defined command is

define <command>
<body of command>
end
document <command>
<documentation>
end

The documentation written with the command is echoed when you type the GDB command help <command>.
All the user-defined commands are listed with the GDB command help user-defined.


Convenience Variables

A convenience variable is defined as

set <variable> = <expression>

Reset, Read, Info

Let's define three user-defined commands, mreset, mrd, and minfo.

define mreset
mon reset
flushregs
end
document mreset
Send the BDI2000 command 'reset'
'flushregs' is required as GDB does not
know what 'mon reset' did to the target
end

define mrd
mon rd
end
document mrd
Send the BDI2000 command 'rd'. Reads better than 'info reg'
end

define minfo
mon info
end
document minfo
Send the BDI2000 command 'info'
end

mreset is a short cut for the two commands mon reset and flushregs.

User-defined commands are very useful to define repetitive commands.

mrd is a repetitive command to use BDI2000 to output registers instead of GDB, as the output is more compact.

minfo is again another short cut command.

A more useful command is beefcore.

define beefcore
  mon mm 0xe00000 0xdeadbeef 0x10000
  mon mm 0xe10000 0xdeadbeef 0x10000
  mon mm 0xe20000 0xdeadbeef 0x10000
  mon mm 0xe30000 0xdeadbeef 0x10000
  mon mm 0xe40000 0xdeadbeef 0x10000
  mon mm 0xe50000 0xdeadbeef 0x10000
  mon mm 0xe60000 0xdeadbeef 0x10000
  mon mm 0xe70000 0xdeadbeef 0x10000
end
document beefcore
Fill RAM (0xe00000), 1MB, with 0xdeadbeef
end

Fills RAM with the same data, 0xdeadbeef.

Now that we done the easy commands, let's build more complicated ones.

dxi

First let's make a short cut dxi.

# dxi
# dxi <count>
# dxi <count> <startaddress>
define dxi
if $argc == 2
  x /$arg0i $arg1
end
if $argc == 1
  x /$arg0i $pc
end
if $argc == 0
  x /20i $pc
end
end
document dxi
Output $arg0 instructions from current PC
If $arg0 is not supplied, output 20 instructions
end

# starts a comment in .gdbinit.

The user-defined command supports if/then/else and while constructs.

Just like a function, arguments can be passed to a user-defined command. $argc is the variable with the number of arguments. Each argument is $argx where x is the argument number starting with zero.

The command dxi has three forms,

dxi

dxi <number>

dxi <number> <address>

$pc is the built-in GDB variable for the current program counter.

With no arguments, dni executes the GDB command x /20i $pc. Output 20 assembly instructions starting at the program counter.

dxi <number> changes the 20 to <number>. Notice this is a string substitution. <number> is not a number in the 'int' sense. It is a string that GDB interprets as a number.

Finally, dxi <number> <address> outputs <number> of assembly instructions starting at <address>.

dxi is a short cut command, but with some variability.

BMEM - Break on Change of Contents of An Address

Below are three command bmemc, bmems, and bmeml. bmemloop is used by all three other commands. This shows GDB user-defined commands support subroutines.

Each command executes an si command until the value at the specified address changes a specified number of times.

bmemc examines 8-bit values, bmems examines 16-bit values, and bmeml examines 32-bit values.

The logical of each command is the same.

Now the specific syntax of user-defined commands that is not well documented is shown.

#bmemloop ptr count
#internal function, called by bmemc, bmemw, and bmeml
define bmemloop
  if $argc == 2
     set $loop = $arg1
     set $bmemlooprd = *($arg0)
     set $bmemlooporg = *($arg0)
     while ($loop > 0)
       while ($bmemlooprd==$bmemlooporg)
         si
         set $bmemlooprd=*($arg0)
       end
       set $loop = $loop - 1
       if $loop == 0
         loop_break
       end
       set $bmemlooporg = *($arg0)
     end
  end
end
document bmemloop
First argument specified address to watch
Second argument specifies number of times to loop
  for each change of the specified address
bmemloop 10000 5 - Loop until address 10000 changes 5 times
end

define bmemc
  if $argc>=1
    disable disp 1
    set $bmemcx=(char*)($arg0)
    set $bmemccount = 1
    if $argc == 2
      set $bmemccount = $arg1
    end
    bmemloop $bmemcx $bmemccount
    enable disp 1
  end
end
document bmemc
bmemc runs until the byte at the address specified changes
end

define bmemw
  if $argc>=1
    disable disp 1
    set $bmemwx=(short*)($arg0)
    set $bmemwcount = 1
    if $argc == 2
      set $bmemwcount = $arg1
    end
    bmemloop $bmemwx $bmemwcount
    enable disp 1
  end
end
document bmemw
bmemw runs until the short (16-bit) at the address specified changes
end

define bmeml
  if $argc>=1
    disable disp 1
    set $bmemlx=(long*)($arg0)
    set $bmemlcount = 1
    if $argc == 2
      set $bmemlcount = $arg1
    end
    bmemloop $bmemlx $bmemlcount
    enable disp 1
  end
end
document bmeml
bmeml runs until the long (32-bit) at the address specified changes

Let's discuss some of the syntax.

set $bmemcx=(char*)($arg0)
set $bmemwx=(short*)($arg0)
set $bmemlx=(long*)($arg0)

These three lines each declare a convenience variable. Start with a dollar sign and create a name, start with a letter. The syntax that is not obvious is that the address specified as the first argument, arg0, of the command has to be casted to the appropriate type, a pointer of a specific width.

The syntax is the same as that of C.

The three convenience variables are now pointers.

We can pass the pointer to bmemloop, a subroutine, along with an optional count.

The enable and disale commands are used to suppress GDB output while the bmemloop is running. The si command outputs data each time it is executed.

To compare the value of two addresses we need pointers. bmemloop creates two convenience variables, bmemlooporg, the original value pointed to by arg0 and bmemlooprd, the value pointed to by arg0 after each si command.

bmemw 20000

This command will execute the si command until the 16-bit value at 20000 changes.

bmemc 0x30000 10

This command will execute the si command until the 8-bit value at 0x30000 changes 10 times.

Note that the address is hexadecimal. The cast to a pointer understands hexadecimal notation.

Performance

The commands bmemc, bmemw, and bmeml work, but they are slow. We are using the GDB interpreter to do something, that is today, done by hardware, but it does work.

Break on an Address - brk

Now let's create a break point user-defined command, brk.

# brk
# brk <dest_address>
# brk <dest_address> <count>
# brk <dest_address> <count> <offset>
define brk
  if $argc >= 1
    disable disp 1
    set $brkcount = 1
    if $argc >=2
      set $brkcount = $arg1
    end
    set $targ = (long)($arg0)
    if $argc == 3
      set $targ = $targ + $arg2
    end
    set $brkx = (unsigned char *)($targ)
    print $brkx
    if $brkcount > 0
      set $loop = 0
      while $brkx == $pc
        si
      end
      while ($loop < $brkcount)
        while ($brkx != $pc)
          si
        end
        set $loop = $loop +1
        if $loop == $brkcount
          loop_break
        end
        si
      end
    else
      print "Loop count cannot be negative"
    end
    enable disp 1
  end
  if $argc == 0
    si
  end
end
document brk
brk <dest_address> [count offset]
Send 'si' commands until <dest_address> reached
If count specified, send 'si' commands until <dest_address> is reached <count> times.
If offset specified, add <offset> to <dest_address> to make break address.
   When using <offset>, must specify count.
   brk $ReadSwitches 1 0x20 - break at $ReadSwitches+0x20 once.
end

The user-define command brk accepts one to three arguments. The first argument, mandatory, is the address we want to break on. The second optional argument is how many times we want to break before stopping. The third optional argument is an offset to add to the address to specify the break point address.

while ($brkx != $pc)

Execute si commands until the program counter is the same as the specified address.

Again the enable and disable commands are used.

brk is no speed demon, but it works.

The use of convenience variables and user-defined commands creates GDB commands that emulate break point commands.

Monday, July 20, 2015

Yet Another Makefile Tutorial - Part II

This blog continues where 'Yet Another Makefile Tutorial' left off. Remember sources are found at https://bitbucket.org/FernLeaf_07/makefiletutorial.

NOTE: The make files in the tutorial are NOT POSIX compliant.

We have a general purpose make file, makefile10a. The source is put in the source directory SRC_DIR. Object files are built in OBJ_DIR, and the program is built in TARGET. Includes files are found in SRC_DIR.

For a project with one flat directory of source, akefile10a does its job.

What about projects where there are sub-directories to SRC_DIR?

The problem with makefile10a is that it uses the make function wildcard to find the source files in SRC_DIR. wildcard does not recursively descend SRC_DIR looking for files. We need another function to perform the recursive search.

make does not have such a function, but we can use external commands. The external command to use is find.

BTW, vpath does not perform a recursive search for files. This is not a problem, yet.

find

find is a very complicated program. We are going to use a relatively simple form.

SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)

find is started by shell. find is told to start the search for *.c files in the SRC_DIR. find recursively searches for files. -type f makes sure that find only retrieves files and not directories. It is rare that a directory could end in *.c, but -type f makes sure that directory is not included.

We update makefile10a to use find.

#makefile11
SRC_DIR := src
OBJ_DIR := obj
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)

Debugging Make Files

The debugging of make files is difficult. make does have a -d switch which is used for showing how make searches for targets and dependencies.

make does not have a switch to output its variables. The function info that will output trace messages while make is running.

$(info message)

We can use any string or variable to create message.

$(info SRC_LIST=$(SRC_LIST))
$(info OBJS=$(OBJS))
$(info DEPS=$(DEPS))

The function error can be substituted for the function info. When the function error is encountered, the message is output and make stops. warning is also available, but not needed for this debugging tutorial.

$(error STOP HERE!)

makefile11a is updated to use info.

#makefile11a
SRC_DIR := src
OBJ_DIR := obj
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
$(info SRC_LIST=$(SRC_LIST))
$(info OBJS=$(OBJS))
$(info DEPS=$(DEPS))
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)














Sub-Directories

We update our example by putting suba.c, subb.c, and subc.c files in the sub-directory sub. We also move all the include files to their own directory INC_DIR.

See updates in makefile12.

#makefile12 - does not work
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)


Unfortunately, makefile12 do not work.








The problem is the old problem of directories do not exist that gcc expects to exist. In this case, the directory obj/sub does not exist.

find found all the source files, complete with sub-directory names, but there are no sub-directories in obj to match those in src.

How to solve this problem?

There are two possible solutions. First, find all the sub-directories in the SRC_LIST of files and create the corresponding sub-directories in OBJ_DIR, or remove all the sub-directory names from the file names in OBJS.

The first solution creates a mirror image of sub-directories in OBJ_DIR found in SRC_DIR.

The second solution will create a flat listing of object files in OBJ_DIR. No necessarily a bad solution.

Both solutions are acceptable. Both solutions shows using additional features of make.

Sub-Directories - First Solution

First solution involves using the make functions patsubst, sort, and dir. We saw patsubst in the previous tutorial. sort and dir are new.

dir is a function that extracts the directory portion of a filename.

sort is a function that sorts a list and removes duplicates.

patsubst is the pattern substitution function.

SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

Let's explain the command from the inside out.

$(dir $(SRC_LIST))

This changes SRC_LIST from this

src/main.c src/suba/suba.c src/subb/subb.c src/subc/subc.c 

to this.

src/ src/sub/ src/sub/ src/sub/

There is one entry for each *.c file. We have the same sub-directory three times, in this example. If there were 50 files in a sub-directory, that sub-directory would appear 50 times.

sort is used to remove the duplicates. We really don't care about the sorting feature, we really just want the duplicates removed.

$(sort $(dir $(SRC_LIST)))

changes the SRC_LIST into this.

src/ src/sub/

Now we have a list of the sub directories.

Remove the trailing slash from the directory names. We will see why later.

$(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

Finally, the list of directories where we find source files.

src src/sub

Change src to obj and we have the list directories for object files.

OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))

Change how we create REQUIRED_DIRS and we are done.

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)

makefile13 shows all the updates for the first solution.

#makefile13
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)







We solved the directory problem for the first solution. We now have a mirror image of sub-directories in the OBJ_DIR as is found in SRC_DIR.

Let's check the clean command.







The rm command does not remove directories recursively. We have a sub-directory under obj, sub. The sub directory has to be removed before we can remove obj.

rm has a switch to recursively delete directories recursively

@rm -rf $(OBJ_DIR)
@rm -rf $(TARGET_DIR)

Updating makefile13 the final make file is this.


#makefile13
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJ_DIR_LIST := $(patsubst $(SRC_DIR)%,$(OBJ_DIR)%,$(SRC_DIR_LIST))
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rm -rf $(OBJ_DIR)
@rm -rf $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR_LIST) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)

makefile13 is the final make file. We have sources in SRC_DIR and its sub-directories. We have include files in INC_DIR. We generate object files in OBJ_DIR and mirror image sub-directories of SRC_DIR, and the target is generated in TARGET_DIR.

clean works correctly.

Sub-Directories - Second Solution, Flat OBJ_DIR

The second solution is to create a flat directory of object files.

We want the filenames from the SRC_LIST, not the directories.

The make function notdir extracts the filename, not the directory.

OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))

We now have a list of files object files.

obj/main.o obj/suba.o obj/subb.o obj/subc.o

Unfortunately changing OBJS in makefile12 produces this error.







The problem we have is that make can't find suba.c. It is in a sub-directory. The file name of the pattern is obj/suba.o, not obj/sub/suba.o which is changed to sub/suba.c. With vpath looking in SRC_DIR, the file suba.c is found in SRC_DIR/sub/suba.c.

We need to give vpath the list of source directories. Use SRC_DIR_LIST from makefile13.



SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))

So update OBJS, add SRC_DIR_LIST to makefile12, update vpath, we get makefile14.

#makefile14
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))
DEPS := $(OBJS:.o=.d)
$(info OBJS=$(OBJS))
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR_LIST)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rmdir $(OBJ_DIR)
@rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)


-include $(DEPS)

This update works.





Update makefile14 for clean the same as makefile13. The final result is this.

#makefile14
SRC_DIR := src
OBJ_DIR := obj
INC_DIR := inc
SRC_LIST := $(shell find $(SRC_DIR) -name "*.c" -type f)
SRC_DIR_LIST := $(patsubst %/,%,$(sort $(dir $(SRC_LIST))))
OBJS := $(patsubst %.c,$(OBJ_DIR)/%.o,$(notdir $(SRC_LIST)))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR_LIST)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
@rm -rf $(OBJ_DIR)
  @rm -rf $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD -I $(INC_DIR) $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do              \
 mkdir -p $$d;   \
 done)

-include $(DEPS)

The second solution is now complete.

Summary

Fini.

We have two files, makefile13, and makefile14, that are general purpose make files for a project with sources in one directory, object files in another, and the target in a third directory.

Future

The make files handle just one source type *.c. What if you have *.cc, or *.cxx files? 

If you just have *.cc instead of *.c or *.cxx instead of *.c, then simply change the find command to use *.cc or *.cxx.

What if you have both *.c and *.cxx? Typically c files are compiled differently than *.cxx files. There are different switches supplied to gcc.

The next part in this blog on make files will look at updating makefile13 to handle multiple file extensions, groups of files that need different compilation, and excluding source files from the build.

See the next blog. Yet Another Make Tutorial III

Sunday, July 19, 2015

True = False, Python 2.7

This a quick blog on an anomaly in Python 2.7.

At the >>> prompt, enter

True = False

True

Now think about what you just did, for a second, or a minute.







Now the question is how to correct it?  [see below]

This is an academic exercise. You would never really do this, right?


Is Python 3 better?






Much more sensible, no?

Use 3.+ where you can, n'est pas?





Wednesday, July 15, 2015

Yet Another Make Tutorial

YAMT, an abbreviation that won't stick.

This tutorial on make may be too little, too late, as other tools are replacing make, Python, CMake, etc.

But I will push on with this tutorial to create a fairly general purpose production make file.

The reader is expected to have a bit of working knowledge of using gcc from the command line, how to create an object file and how to link them together to form a program.

At the end of this tutorial is a list of links where some of the features of make I use were found, along with other references.

All the files shown in the tutorial are found at this BitBucket site. https://bitbucket.org/FernLeaf_07/makefiletutorial.

Introduction

If you landed here not knowing much about make, this very brief introduction to make will get us started. If you are versed in the world of make, ctrl-f to 'Advanced Make'. If you just want the punch line, ctrl-f to 'makefile10a'.

make is a program that builds things, usually programs. make saves time. If the input file to make is written properly, then just those files that need to be compiled since the last compilation will be compiled. The purpose of using make is to define how a program is built and to save time building a program after an update to the source.

NOTE: The make files in this tutorial are NOT POSIX compliant.

Target and Dependency

The fundamental concept of make is that files are targets and have dependencies.

If a file is a target and it doesn't exist or is older in terms of date time than its associated dependency files, then a command is executed to create a new version of the target file.

The syntax to define a target/dependency pair is

target : dependency ; command
<tab>command

The target is first, starting in column 1, followed by a colon, followed by one or more dependencies.

Each target/dependency/command is separated by at least one blank line.

The command to execute can follow the dependency list, separated by a semi-colon, or put on the next line. The common convention is to put the command on the next line.

If the command is put on the next line, then a tab character MUST BE THE ONLY CHARACTER before the command.

Again, if the command is on its own line, the tab character MUST BE THE ONLY CHARACTER before the command.

make does not output good error messages when the tab character is omitted or a space is used instead.

With whatever editor you use, display spaces and tabs so that you make sure you have the tab where it belongs.

NOTE: Make sure you use a TAB CHARACTER to start the command line of a target/dependency definition.

targets don't have to be files, but usually are.

dependencies don't have to be files, but usually are.

There can be more than one target, but this tutorial will not use that feature.

make does not execute the commands in a make file sequentially. The make file defines target/dependency pairs.  At least one target/dependency pair has to have associated with it a command, otherwise the make file will do nothing. When make sees that the target is out of date with respect to its dependencies,  the command or commands are executed. When these commands are executed has no relationship to its position in the file.

Note: make really doesn't like spaces in filenames. Many have asked 'the Internet' how to work around this problem. Any exercise for the reader or a future blog.

Make Command Line Syntax.

A bit of make command line syntax.

make

This command assumes there is a file name makefile in the current directory and all will be the default .PHONY word. makefile.mak can also be used. (all and .PHONY discussed soon.)

make all

Same as the make command above.

In this tutorial, each make file will get its own unique name.

make -f makefile1

The -f argument informs make to use makefile1 instead of makefile as the input file.

There is only one input file to make, at the command line.

Simple

gcc main.c

This command causes gcc to compile main.c and create a.out (or a.exe for the Cygwin/MinGW crowd).

No need for make here. Just one file. No way to save time. Just the one file to compile.

gcc main.c suba.c subb.c subc.c

This command will build another a.out from the list of files. Now the need for make and a make file begins.

If you change just suba.c, then all the other files are built too. The same amount of time is used no matter how big or small the changes.

A shell script (or batch) can hold the complete command to save typing, but not time. Just ./buildit.sh (buildit.bat).

But what if your project has dozens and dozens of files? Instead of taking mere moments to build, it will take an hour, even if all you added was one comment.

Let's break down the long gcc command.

gcc -c main.c
gcc -c suba.c
gcc -c subb.c
gcc -c subc.c
gcc main.o suba.o subb.o subc.o -o aprogram

Again, a shell script could be created to execute all 5 lines, but make can do better.

The five commands above builds our program, aprogram, one step at a time. This is the key to a make file. Break the build of your program into individual steps.

Here is the source files we will be using for this tutorial. The program is a scaffolding. It doesn't do anything. It is just code that has relationships to illustrate make and make's features.

// main.c
#include "suba.h"
#include "subb.h"
#include "subc.h"
int main(void)
{
    suba();
    subb();
    subc();

    return 1;
}

// suba.c
#include "suba.h"
void suba(void)
{
}

// subb.c
#include "subb.h"
void subb(void)
{
}

// subc.c
#include "subc.h"
void subc(void)
{
}

// suba.h
void suba(void);

// subb.h
void subb(void);

// subc.h
void subc(void);

Here is the make file version of the above 5 commands.

# makefile1
.PHONY: all clean

all: aprogram

clean:
@rm -f *.o
@rm -f aprogram*

aprogram: main.o suba.o subb.o subc.o
gcc $^ -o $@

main.o: main.c
gcc -c $<

suba.o: suba.c
gcc -c $<

subb.o: subb.c
gcc -c $<

subc.o: subc.c
gcc -c $<

Let's examine each line of the make file and get familiar with terminology and structure.

.PHONY - This keyword tells make where to start. Each of the words on the .PHONY line are arguments to the make program to instruct make where to start.

.PHONY keywords define the top of a dependency tree.

all is a reserved .PHONY word. If make is executed with no arguments, all is the default .PHONY word.

clean is a word, used by convention, to remove all files built by using the all keyword.

Other keywords can be defined. No additional keywords will be defined in this tutorial.

all: aprogram

This line defines our first target/dependency pair. all is the target. aprogram is the dependency. There is no command associated with this pair. The statement defines the dependency pair and the start of the dependency tree.

We will skip clean for now.

aprogram: main.o suba.o subb.o subc.o
<tab>gcc $^ -o $@

This is the first dependency pair with a target, a dependency list, and a command. (note the <tab> reminding you to use a tab character before the command.

The $^ and $@ are make automatic variables. make has several automatic variables. They define strings based on the targets and dependencies.

$^ is the list of dependencies, all of them.

$@ is the target.

Expanding the command we would get the following.

gcc main.o suba.o subb.o subc.o -o aprogram

The automatic variables save typing, and later, we will see they are mandatory when using patterns to define dependency pairs.

We have defined all as a target. We have defined aprogram as its dependency. We have defined aprogram as a target, the next level in the dependency tree. We have defined a list of object files as dependencies of aprogram.

Now we define each object file as a target.

main.o : main.c
<tab>gcc -c $<

suba.o : suba.c
<tab>gcc -c $<

subb.o : subb.c
<tab>gcc -c $<

subc.o : subc.c
<tab>gcc -c $<

Each object file has a dependency on a c file. The command to create the target object file, compile the c file is the same for each object and c file pair.

$< is the automatic variable that represents the first dependency. Initially it looks like we could also use $^, as there is only one dependency, but later we will see it is better to use $<.

The c files do not need a dependency. They do not need to be created. They already exist.

Thus the dependency tree is complete.

clean

clean:

This is a target/dependency pair with no dependency. Thus the commands associated with it are always executed.

@rm -f *.o
@rm -f aprogram*

The command rm (remove files) remove all the object files. The -f switch says to not report an error if the file does not exist. The @ in front of rm suppresses any output from the program.

The second line removes aprogram or aprogram.exe, doesn't matter.

clean removes all files created by all.

Execute makefile1

make -f makefile1

Running the above command we get the following output.








We see that all four files are compiled and aprogram is built.

touch suba.c

make -f makefile1







Now we see the power of make. The touch command updated the date time of suba.c. suba.c is now newer than its object file. make scans the target/dependencies and determines that the command associated with suba.o needs to be executed. Since suba.o is a dependency of aprogram, the command for aprogram will be executed next.

Only two commands are executed instead of five.

make -f makefile1 clean

This command specifies clean as the top level dependency .PHONY word.





Include Files

What about the include files?

An observant reader will notice that the include files are not listed in makefile1.

Let's update that now.

# makefile2
.PHONY: all clean

all: aprogram

clean:
@rm -f *.o
@rm -f aprogram*

aprogram: main.o suba.o subb.o subc.o
gcc $^ -o $@

main.o: main.c suba.h subb.h subc.h
gcc -c $<

suba.o: suba.c suba.h
gcc -c $<

subb.o: subb.c subb.h
gcc -c $<

subc.o: subc.c subc.h
gcc -c $<

[Note: Red is used to show how the make files are updated from step to step in this tutorial.]

Each object file now has the corresponding include file as a dependency file.

We now have more than one dependency for each object file, but we still want the command to compile just the c file. The automatic variable $< still uses just the c file, not the additional include files added (as long as the c file remains the first dependency).

touch suba.h

make -f makefile2








If suba.h is changed, then both suba.o and main.o are rebuilt, thus aprogram is rebuilt.

Note that repeating the command, make reports nothing to be done, a test the make file is correct.

Simple Summary

For really simple programs, I mean really simple programs, makefile2 provides a guide to creating useful make files.

But production quality programs are not this simple.

The problem with makefile2 is the file is brittle. If a new include file is created, or a new c file is created or deleted, the dependency tree has to be updated, by hand. This is tedious and error prone.

The rest of the tutorial is how to create a make file that automates the building of the list of files to be compiled and linked.

Advanced Make

Variables

Make is an interpretive programming language. Not a 'typical' programming language, but a programming language. (probably not Turing complete :) ).

Just like all programming languages, make has variables. We saw a few automatic variables above.

Let's change makefile2 to use a variable or two.

# makefile3
OBJS = main.o suba.o subb.o subc.o
TARGET = aprogram
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f *.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

main.o: main.c suba.h subb.h subc.h
gcc -c $<

suba.o: suba.c suba.h
gcc -c $<

subb.o: subb.c subb.h
gcc -c $<

subc.o: subc.c subc.h
gcc -c $<

The string of object files is now a variable, OBJS. The name of our program is now a variable, TARGET.

To use a variable name, the name is preceded by $( and followed by ). The automatic variables were preceded  with a $. So any variable in make is referenced by using a $. An automatic variable is one of several letters. A user defined variable is surrounded with parentheses. Brackets { } can be used in place of parentheses, but parentheses are much more commonly used.

Variables will play a major role in the final production quality make file.

makefile3 is no less brittle than makefile2.


Patterns

make allows for patterns to be used in defining a target/dependency pair.

%.o : %.c

The %.o pattern states any file that ends in a two characters .o is a target. The corresponding dependency file uses the same % string for the root name but changes the .o to .c.

Let's use the above pattern to change makefile3.

# makefile4
OBJS = main.o suba.o subb.o subc.o
TARGET = aprogram
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f *.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

%.o: %.c
gcc -c $<

makefile4 is a lot shorter than makefile3. The automatic variable $< is required when using a pattern as the actual name of the dependency is not known until make runs.

A bit of automation has been added. If we add a new c file or delete a c file, we only have to add or remove from the corresponding object file in OBJS.

Let's review how the dependency tree is built. all is dependent on $(TARGET). $(TARGET) is dependent on the list $(OBJS). The pattern dependency then generates the c file dependencies for each object file in the $(OBJS) list. The object file targets used by the pattern come from the make file. Remember the object files do not exist the first time.

But we have lost the dependency relationship with the include files.







Touching  suba.h does not result in rebuilding suba.o and aprogram.

DEPS

Let's do a quick and dirty remedy.

#makefile5
OBJS := main.o suba.o subb.o subc.o
DEPS := suba.h subb.h subc.h
TARGET := aprogram
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f *.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

%.o: %.c $(DEPS)
gcc -c $<


We created the variable DEPS to hold all the include files. We made this list a dependent on any object file. DEPS is a variable name, used by convention, for include files.

This update works. However, if just an include file is changed, then all the objects and aprogram are rebuilt. aprogram is built properly, but make builds more object files than is necessary.












We will come back and update the make file so that the minimum number of object files are built for just an include file change.


Recursively Defined and Simply Defined Variables

Another subtle, but important change was made. Equal signs (=) were changed to colon equal signs (:=).

make has two types of variables, recursively defined variables and simply defined variables. If the string being assigned to a variable can be determined without recursion, statically, then use the simply defined variable notation, colon equal (:=). It is faster than recursively defined variables.

All variables in this tutorial are simply defined variables. It was difficult to find a good example of a recursively defined variable, and they are not needed here.

Directories

Another observant reader will notice that the object files and aprogram are being created in the same directory as the source files.

This is considered poor form. It is messy. Production make files have the object files in one directory, the source in another directory, and the program in a third directory. A convention is followed where the object files and the program are put in the same directory. This tutorial will have the program and object files in separate directories.

We move all the sources to a sub-directory named src. The object files will be placed in the sub-directory obj, and the program will be placed in the sub-directory target.

Several make tutorials talk about it being easiest to design a make file when the make file runs in the directory where the object files are to be built.

Yes, this can be easier when building the make file. The make file is just like any other source file. It will be archived in source control. If it exists in a directory where all the other files are intermediate, temporary files, one can accidentally delete all the files in the obj directory, requiring the make file to be retrieved from the source control archive.

The following make file is expected to be run in the parent directory of src, obj, and target.

Let's update the various filenames in the make file with the new directory names.

#makefile6 - doesn't work
OBJS := obj/main.o obj/suba.o obj/subb.o obj/subc.o
DEPS := src/suba.h src/subb.h src/subc.h
TARGET := target/aprogram
.PHONY: all clean

all: $(TARGET)

clean:
@rm –f obj/*.o
@rm –f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

obj/%.o: %.c $(DEPS)
gcc –c $< -o $@


The include files have the src directory pre-pended. The object files have the obj directory pre-pended. aprogram has target pre-pended.

However, as the comment states, makefile6 doesn't work.






There are two reasons makefile6 does not work. A simple change will correct one problem.

Change


obj/%.o: %.c $(DEPS)
  gcc –c $< -o $@

to


obj/%.o: src/%.c $(DEPS)
  gcc –c $< -o $@


This change illustrates the literal nature of the make pattern.

obj/%.o specifies any file in obj where the pattern (%) is the root portion of the name.

%.c then says take the found object root name and simply change .o to .c.

If we start with obj/main.o for a target. The dependency is then main.c.

gcc -c main.c -o obj/main.o

The source files do not exist in the parent directory, thus make cannot find the dependency.

With the update of src/%.c the command is now

gcc -c src/main.c -o obj/main.o







There is still an error. This time from gcc, not make. The error 'No such file or directory' is being reported because the obj directory does not exist, not that the file obj/main.o does not exist. A bit confusing, but par for make error messages.

gcc expects the obj directory (and the target directory) to exist before it will write main.o into the obj directory. gcc does not create directories if they are missing.


VPATH and vpath

We take a step back and solve the issue not finding the sources a different way than adding src/ to the pattern.

Enter make's vpath feature.

make contains a variable and a function to define a virtual path to look for dependencies.

VPATH is a reserved variable. It may contain a list of one or more directories to look for dependencies.

vpath, all lowercase, is a function that has two arguments, a pattern and a list of directories.

There is only one VPATH variable. There may be as many vpath pattern/directory pairs as you need.

Update makefile6 to use vpath.

#makefile7 - does not work
OBJS := obj/main.o obj/suba.o obj/subb.o obj/subc.o
DEPS := src/suba.h src/subb.h src/subc.h
TARGET := target/aprogram
vpath %.c src
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f obj/*.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

obj/%.o: %.c $(DEPS)
gcc -c $< -o $@

Now the command

gcc -c main.c -o obj/main.o

is still generated, but vpath instructs make to look in the src directory if a dependent file with a .c extension is not found by default.

makefile7 still does not work, We'll fix the missing directories next.





Creating Missing Directories

The make file must define the directories that need to be created and create them prior to compiling the c files.

There are two directories to create, obj and target. We need to have a command to create these directories, if they don't exist.

mkdir is the command to create a directory. mkdir is an external command. It is not built into make, just like gcc.

mkdir -p <directory> will create <directory> if it does not exist and not report an error if it does.

There are two approaches to updating the make file to create the missing directories. Approach 1 is to add the missing directories to the dependency tree. Approach 2 is to use variables to create the directories.

Missing Directory - Approach 1

all: required_dirs $(TARGET)

required_dirs : $(OBJ_DIR) $(TARGET_DIR)

$(OBJ_DIR) :
<tab>@mkdir –p $@

$(TARGET_DIR) :
<tab>@mkdir –p $@

The new target, required_dirs is added to the list of dependencies for all. Each directory in turn has a target/command pair. Without a dependency the command, mkdir, is always executed. If the directory does not exist, it is created.

Missing Directory - Approach 2


REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
mkdir -p $$d; \
done)


The variable REQUIRED_DIRS is now a list of the required directories.
_MKDIRS is a variable that is assigned the result of the shell command.

make allows scripts to be run using the function shell.
Note that shell is surrounded by $( ). Essentially the return code of shell is a variable.

The above shell script is a do loop that iterates over the list of directories, creates a shell variable d and creates each directory if it does not exist. The $$ is so that d is used as a shell variable, not a make file variable.

Note: Tabs are not required here. This is a definition of variables, not a target/pair dependency.

Let's incorporate Approach 2 into makefile7.

#makefile8
OBJ_DIR := obj
OBJS := $(OBJ_DIR)/main.o $(OBJ_DIR)/suba.o $(OBJ_DIR)/subb.o $(OBJ_DIR)/subc.o
DEPS := src/suba.h src/subb.h src/subc.h
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c src
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c $(DEPS)
gcc -c $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
 do \ mkdir -p $$d; \  done)

Added the variables OBJ_DIR and TARGET_DIR. Updated OJBS and TARGET to use these new variables. Finally added the variables to run the shell script to create the missing directories, if needed.

The mess is cleaned up. The object files are created in the obj directory. aprogram is created in the target directory and gcc does not complain about directories missing.



Wildcard

Now let's start to get some automation into the make file.

There is a make function, wildcard, which will create a list of files from a pattern.

SRC_DIR := src
$(wildcard $(SRC_DIR)/*.c)

This function looks for all c files in the SRC_DIR directory and creates a space separated list of all the files found.

Let's assign this list to the variable SRC_LIST.

SRC_LIST := $(wildcard $(SRC_DIR)/*.c)

SRC_LIST can be used to create the list of object files.

Using the make function patsubt, short for pattern substitution, we can create OBJS from SRC_LIST.

OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))

Automation. Whatever files are in SRC_DIR, the corresponding list of object files is created.

#makefile9
SRC_DIR := src
OBJ_DIR := obj
SRC_LIST := $(wildcard $(SRC_DIR)/*.c)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(SRC_DIR)/suba.h $(SRC_DIR)/subb.h $(SRC_DIR)/subc.h
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o
@rm -f $(TARGET)*

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c $(DEPS)
gcc -c $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
mkdir -p $$d; \
done)

Good make file. Simply add a new c file or delete the file from SRC_DIR and the appropriate object file will be created and linked.

However, the include files are still not automated.

Let's finally solve this problem.

Automated Include Files

There are three features that are required to automate the include files. One feature is from gcc and two features are from make.

gcc -MMD switch

gcc has a command line switch -MMD. It is specifically designed to assist in the creation of automated include file dependencies.

Using -MMD, gcc creates, in addition to the object file, an include dependency file, *.d file, in the same directory as the object file.

The include dependency file is a make file with no commands. The target is the object file created. The dependencies are the include files used by the c file compiled. The include file dependencies is the complete include calling tree. If include files are nested in include files, the nested files are in the dependency list.






Next we need a list of these include dependency files.

Short cut string replacement

DEPS := $(OBJS:.o=.d)

The above assignment changes every .o in OJBS to .d. This is simpler, but not as flexible, as patsubst.

include function

Now that we have created the include dependency files and we have a list of these files, we need to put these files into the dependency tree.

The make function include will perform this step.

-include $(DEPS)

include causes the files listed to be included within the make file. As each include dependency file has a target and dependency pair, each pair now part of the dependency tree.

The target is the object file, which is already dependent on the c file via the pattern. The object files are now also dependent on the include files associated with the c file, as generated by gcc.

The minus sign in front of include is important. If the include dependency file does not exist, include will not report an error. This will be true the first time make is run or after a clean.

Notice that the order of dependencies is definitely not the order the dependencies are found in the file.

#makefile10
SRC_DIR := src
OBJ_DIR := obj
SRC_LIST := $(wildcard $(SRC_DIR)/*.c)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
mkdir -p $$d; \
done)

-include $(DEPS)

Added -MMD switch to gcc. Updated the DEPS variable and included all the include dependency files.













The include file dependency problem solved.

There is one shortcoming with makefile10OBJ_DIR and TARGET_DIR are not deleted when clean is run. We said clean removes all files created by all. The directories are created by all, but not removed by clean.

#makefile10a
SRC_DIR := src
OBJ_DIR := obj
SRC_LIST := $(wildcard $(SRC_DIR)/*.c)
OBJS := $(patsubst $(SRC_DIR)/%.c, $(OBJ_DIR)/%.o, $(SRC_LIST))
DEPS := $(OBJS:.o=.d)
TARGET_DIR := target
TARGET := $(TARGET_DIR)/aprogram
vpath %.c $(SRC_DIR)
.PHONY: all clean

all: $(TARGET)

clean:
@rm -f $(OBJ_DIR)/*.o $(OBJ_DIR)/*.d
@rm -f $(TARGET)
 @rmdir $(OBJ_DIR)
 @rmdir $(TARGET_DIR)

$(TARGET): $(OBJS)
gcc $^ -o $@

$(OBJ_DIR)/%.o: %.c
gcc -c -MMD $< -o $@

REQUIRED_DIRS := $(OBJ_DIR) $(TARGET_DIR)
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
mkdir -p $$d; \
done)

-include $(DEPS)

Fin. makefile10a is completely automated. Give a list of source files in the SRC_DIR directory that will link, as object files, to create aprogram, no changes to makefile10a are required as new c files or include files are added or deleted.














Include File Dependency Details

You may be wondering if the update for include files works for all circumstances. Let's discuss it in a bit more detail.

When the make file is run for the first time, there are no *.d files, so the only dependent file for an object file is the c file. Since the object file does not exist, the c file is compiled.

The second running of the make file, the *.d files exist and the include files are added to the dependencies of the object file. If an include file is changed, then the appropriate object file is created.

If an include is added or deleted, then some file, a c file or another include file must have been changed. The appropriate object file is created.

Finally, what if a c file is deleted, the corresponding object file is not deleted. No it is not, but the old object file is no longer a member of the list of object files, OBJS, used to link aprogram. An extra file exists in the OBJ_DIR directory, but does not get used.

The next time clean is run the extra object files will be removed.

Again, makefile10a is very good general purpose make file.

Change the target and you are ready to use makefile10a for another program.

There are is, however, a limitation with makefile10a. The sources must all be in SRC_DIR. There can be no sub-directories to SRC_DIR containing source. The problem is wildcard does not recursively descend directories looking for files.

The next posting of this blog shows how to extend makefile10a to use sub-directories of SRC_DIR.

Next post: Yet Another Make Tutorial II

References and Links

This is a listing of various URLs that discusses section of this document and where the ideas came from.




Note about putting –include at the end of the file comes from here. http://www.microhowto.info/howto/automatically_generate_makefile_dependencies.html