PDA

View Full Version : Makefile secondary expansion



milli-961227
02-08-2015, 11:23 AM
Hey folks,

I've got a question about secondary expansion using the GNU implementation of Make, therefore I thought the Linux section would be the right one.

In short, I'm trying to create a macro that is able to build some target rules which then can be used by other targets.


.SECONDEXPANSION:

# ${1} - Source files (Prerequisites)
# ${2} - A procedure that is used to generate the target name
# ${3} - The command that is invoked by every target rule
define rules
$(foreach file,${1},${2}:${file};${3};)
endef


$(call rules,test.cc,$$(shell echo test.o),echo $$@ requires $$^)

This should generate the target rule

test.o:test.cc;echo $@ requires $^;

Output when running Make:

# What I expected:
test.o requires test.cc

# Actual output:
$ requires test.cc

Could anyone please tell me why the above is not working as expected and how to fix it? Thanks in advance!

Nominal Animal
02-08-2015, 09:26 PM
Why do you need secondary expansion? Your example does not. Plain


$(shell echo test.o): $(shell echo test.cc); @echo $@ requires $^

provides the equivalent of


test.o: test.cc
@echo $@ requires $^

which when run, outputs


test.cc requires test.o



Let's say I have a growing collection of plug-in modules, and I want to save some effort in maintaining them. They all use the same command to build (if they didn't, I'd put them into a subdirectory with their own Makefile). Let's say the list of C plug-in modules to be built is built part-by-part, thus:


PLUGINS = foo.so
PLUGINS += bar.so
PLUGINS += baz.so

The rule needed to build them is simple:


$(PLUGINS): %.so: %.c
$(CC) $(CFLAGS) -fPIC -shared $^ $(LDFLAGS) -ldl -Wl,-soname,$@ -o $@
What I'm trying to say is that I cannot see what you need the secondary expansion for. Maybe some more context on what kind of problem you're trying to solve, is in order?

milli-961227
02-09-2015, 02:58 AM
Thanks for your answer so far, though it didn't help me very much..

What I want is to allow a user to compile a variable number of source files and link them together with my static library in order to build a binary executable.


make test sources="test.cc test-class.cc" output=bin/test

My problem is now that one could provide a a source file looking like `../../../project/main.cc'. Therefore, I cannot simply put a `build/test/' prefix in front of the path and substitute `.cc' by `.o' in order to generate the name of the object file. Putting the object file into the same directory as the source file could also fail due to lacking write permissions.

My current approach is to simply hash every path given (which is unique) and generate object files like `build/test/a7462dc634fbe16640590be885e3fdea.o'.

But now a simple pattern rule like this

${OBJECTS}: %.o: %.cc
wont work anymore, because requested source files like `build/test/a7462dc634fbe16640590be885e3fdea.cc' do (of course) not exist.

In order to solve this, I came up with the idea to create a function that automatically creates one target rule for every object file.



$(call rules,\
test.cc test-class.cc,\ # Source files
$(firstword $(shell echo ${file} | md5sum)).o\ # Rule to generate object file names
@echo $$@ requires $$^) # Command of every rule

# Should expand to (and be usable like):

f6350e8e03106d7161e6a36849436de0.o: test.cc
@echo $@ requires $^

93e09f2e4d202855e5d8c919fbb57119o.: test-class.cc
@echo $@ requires $^

milli-961227
02-09-2015, 03:47 AM
This is what I've got so far and it even works partially. When calling the `rule' function manually for each source file it works well, whereas I get errors when doing this with a loop.



# ${1} - Source file
# ${2} - Conversion function
# ${3} - Command function
define rule
$(call ${2},${1}):${1};$(call ${3})
endef


conversion = $(firstword $(shell echo ${1} | md5sum)).o
command = @echo $$@ requires $$^


SOURCES := test.cc src/a.cc
OBJECTS := $(foreach file,${SOURCES},$(call conversion,${file}))


all: ${OBJECTS}


# Works well
$(call rule,test.cc,conversion,command)
$(call rule,src/a.cc,conversion,command)


# Doesn't work
#$(foreach file,${SOURCES},$(call rule,${file},conversion,command))


The first one prints out


f6350e8e03106d7161e6a36849436de0.o requires test.cc
5304837f2022cc1a49c80abafb7d3cfd.o requires src/a.cc


While the second one fails with


f6350e8e03106d7161e6a36849436de0.o requires test.cc 5304837f2022cc1a49c80abafb7d3cfd.o:src/a.cc
/bin/sh: @echo: command not found
mk-2:28: recipe for target 'f6350e8e03106d7161e6a36849436de0.o' failed
make: *** [f6350e8e03106d7161e6a36849436de0.o] Error 127

Nominal Animal
02-09-2015, 06:05 AM
Thanks for your answer so far, though it didn't help me very much..
Your own fault. If you want an answer for your actual problem, you should ask for it, and not ask about some detail on your non-working solution.


What I want is to allow a user to compile a variable number of source files and link them together with my static library in order to build a binary executable.

make test sources="test.cc test-class.cc" output=bin/test

Now we're getting somewhere.

I'd use a Makefile similar to


CC := gcc
CFLAGS := -Wall
LD := $(CC)
LDFLAGS :=

define TESTNAME
test-$(firstword $(shell md5sum -b $(1))).o
endef

define TESTRULE
$(call TESTNAME,$(1)): $(1)
$$(CC) $$(CFLAGS) $$^ -c -o $$@
endef

SOURCES := empty.c
OUTPUT := bin/test
OBJECTS := $(foreach src,$(SOURCES),$(call TESTNAME,$(src)))

.PHONY: all clean test

all: clean test

clean:
rm -f *.o bin/*

$(foreach src,$(SOURCES),$(eval $(call TESTRULE,$(src))))

$(OUTPUT): $(OBJECTS)
$(LD) $^ $(LDFLAGS) -o $(OUTPUT)

test: $(OUTPUT)

which will happily build a test binary using

make test SOURCES="../../foo/bar.c baz.c" OUTPUT=bin/test

Note I derive the object file name for each source file from the MD5 has of its contents. This allows the user to specify multiple sources with the same name but in different directories.

The idea here is that the TESTNAME function generates the object file name from a given source file name.

We list all the object file names for SOURCES in OBJECTS, so that we can let make worry about which object files need to be compiled, and which ones are new enough.

The TESTRULE function is the prototype rule to compile each source file object. The first parameter is the source file name, the second the object file name.

To emit the rules for the source files, we use the foreach loop near the bottom. Because TESTRULE function should be parsed as Makefile syntax, we use $(eval $(call TESTRULE,...)) (https://www.gnu.org/software/make/manual/html_node/Eval-Function.html).

If you want to see the generated rules, run

make test SOURCES="../../foo/bar.c baz.c" -p | sed -ne '/^#/ d; /^test-/,/^$/ p'

Above, each source file is compiled as a separate compilation unit. If you want, you could just compile them all at once, in which case a single rule for the final binary suffices:

CC := gcc
CFLAGS := -Wall
LD := $(CC)
LDFLAGS :=

define TESTNAME
test-$(firstword $(shell md5sum -b $(1))).o
endef

SOURCES := empty.c
OUTPUT := bin/test
OBJECTS := $(foreach src,$(SOURCES),$(call TESTNAME,$(src)))

.PHONY: all clean test

all: clean test

clean:
rm -f *.o bin/*

$(OUTPUT): $(SOURCES)
$(CC) $(CFLAGS) $^ $(LDFLAGS) -o $(OUTPUT)

test: $(OUTPUT)


Questions?

milli-961227
02-09-2015, 07:17 AM
Wow thank you, it's working now!


Because TESTRULE function should be parsed as Makefile syntax, we use $(eval $(call TESTRULE,...)) (https://www.gnu.org/software/make/manual/html_node/Eval-Function.html).
This was the key for me, didn't know that the `$(eval ...)' function is required for strings to be interpreted as Makefile syntax. All i had to do is to change the following:



# Didn't work
$(foreach file,${SOURCES},$(call rule,${file},conversion,command))

# Added `$(eval ...)' - Working now
$(foreach file,${SOURCES},$(eval $(call rule,${file},conversion,command)))



Note I derive the object file name for each source file from the MD5 has of its contents. This allows the user to specify multiple sources with the same name but in different directories.

No bad idea but wouldn't this lead to crashes if the user specified files with equal contents (empty files for example)? Ill try to combine hashes of filenames and contents in the names of object files.

Big thanks again, you really helped me a lot!

Nominal Animal
02-09-2015, 07:46 AM
No bad idea but wouldn't this lead to crashes if the user specified files with equal contents (empty files for example)?
Good point -- I just realized you could always get a hash collision. Unlikely, but possible. So,


define TESTNAME
test-$(firstword $(shell echo $(realpath $(1)) | md5sum -b))-$(firstword $(shell md5sum -b $(1))).o
endef
would use test-MD5 hash of real absolute path to file-MD5 hash of file contents.o style object file names.

Using realpath helps catch the case when the user specifies the same file twice accidentally, using two different paths. If you want to allow that (duplicate source files using different paths), use $(firstword $(shell echo $(1) | md5sum -b)) instead.