The Blog of Fernleaf: Number of Processors, Linux, and Make

[Much of this post is from here, http://www.binarytides.com/linux-cpu-information/]

For those who use Windows, there is an environment variable that comes pre-loaded, NUMBER_OF_PROCESSORS.

This variable contains the number of processors that a program might want to know to allow for parallel operations.

But in this day and age, the number of actual processors and the number reported may differ by a factor of two. The technology that allows this is called hyperthreading. Hyperthreading has been around for more than ten years.

A CPU chip may have two physical cores, but with hyperthreading, the operating system is presented with four processors.

Under Windows, NUMBER_OF_PROCESSORS is the hyperthreading number.

Under Linux, we can run a set of commands and get both.

# Get the number of actual processors, not hyperthreaded processors
NUMBEROFPROCESSORS=`cat /proc/cpuinfo | grep 'core id' | wc -l`
echo "Number of Processors="$NUMBEROFPROCESSORS

# Get the number of hyperthreaded processors
NUMBEROFHTTPROCESSORS=`cat /proc/cpuinfo | grep processor | wc -l`
echo "Number of Hyperthreaded Processors="$(($NUMBEROFHTTPROCESSORS-$NUMBEROFPROCESSORS))

The environment variable NUMBEROFPROCESSORS is created by the output of the file /proc/cpuinfo being piped to grep, which looks for the string processor or core id. The result is then piped to wc which counts the number of lines grep found.

My results are as follows. I'm running on an Intel(c) Core(tm)2 T9400. It does not support hyperthreading.

Number of Processors=2
Number of Hyperthreaded Processors=0

We can use this information with the -j argument for make.

# Get the number of actual processors, not hyperthreaded processors

NUMBEROFPROCESSORS=`cat /proc/cpuinfo | grep processor | wc -l`

make -j$NUMBEROFPROCESSORS

Now make will use the number of processors (with hyperthreading). We don't have to change the make script each time we get more cores on our VM or laptop.

It is mentioned (can't find the URL) that -j has no effect when running under MS-DOS. Don't know if this has changed. The version of Windows is not mentioned. nmake, supplied with Visual Studio, is said to be available for multi-core (parallel) operation. StackOverflow discussion here.

Using -j, in general, assumes that the build is not a highly dependent build. Lots of leaves in the build tree.

More details on cpuinfo can be found here.

Nice discussion on the why's and wherefore's of processors, physical ids, and cores here.

Busy - Sluggish

Now that you have enabled make to use all the available processors, be prepared for your system to get sluggish.

On a build machine you want to build fast. On a development machine you want to do something else, but you just told make to use all the available CPUs. Ctrl-C might not even be effective, for a while.

Your mileage may vary. Forewarned is forearmed.

Number of Processors + 1

There are many discussions that state the value for the make argument -j should be the number of processors +1, even +2. The idea is that make (and compiling) is an I/O intensive activity. While waiting for one compile to finish, another can be prepared.

At this URL, Agostino Sarubbo, has shown that one should use the number of processors and no more.

The reason for this would be a subject of a separate blog. Exercise for the reader. :)