How to optimize parallel build of subdirectories using GNU Make?

Euklides

I am working in a large project where we have several subdirectories from which we build libs/archives. Most of the subdirectories do NOT have dependecies against eachother but some do.

So for example, this is the general form:

TARG1=targ1/
TARG2=targ2/
TARG3=targ3/

SUBDIRS= lib1\
         lib2\
         lib3\
         ...

.PHONY: $(TARG1) $(TARG2) $(TARG3)

The build order must be as follows: TARG1 -> TARG2-> TARG3 -> SUBDIRS

We have structured the targets like this:

build_libs:
      $(MAKE) subdirs

subdirs: $(SUBDIRS)

$(SUBDIRS): $(TARG3)
      $(MAKE) -C $@

$(TARG3): $(TARG2)
      $(MAKE) -C $@

$(TARG2): $(TARG1)
      $(MAKE) -C $@

$(TARG1):
      $(MAKE) -C $@

Now as I see it, this would allow for maximum parallelization possible since when having built the targets (TARG1 -> TARG3), Make can build all libs in SUBDIRS simultaneously. But perhaps there is a better method to optimize this? The downside with this approach, which I see, is that we get a much larger Makefile since we must specify a recipe for every target which has a dependecy against another target. Is there some way to avoid this and still allow for full parallelization?

The other approach would be to build the targets like this:

SUBDIRS = targ1 targ2 targ3 lib1 lib2 lib3

subdirs:
        for dir in $(SUBDIRS); do \
          $(MAKE) -C $$dir; \
        done

Now the good thing with this approach is that the dependencies can easily be specified just by writing the targets in the correct order. However Make would not be able to fully utilize the parallelization since it would have to complete every sub-directory before moving on to the next one.

Perhaps my assumptions are wrong so therefore I ask this question how to best organize the targets to allow for optimal parallelization? And as a side-question how to organize the targets to avoid code duplication?

MadScientist

You don't have to duplicate the recipes, if they are all the same. Recipes and prerequisites don't have to be defined at the same time, if you don't want to. You could rewrite your first example as:

build_libs:
        $(MAKE) subdirs

$(SUBDIRS) $(TARG1) $(TARG2) $(TARG3):
        $(MAKE) -C $@

subdirs: $(SUBDIRS)
$(SUBDIRS): $(TARG3)
$(TARG3): $(TARG2)
$(TARG2): $(TARG1)

I don't really know why you have the build_libs recursion at all; why not just have build_libs: subdirs?

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

How to optimize parallel build of subdirectories using GNU Make?

From Dev

How to optimize GNU parallel for this use?

From Dev

How to write multicore sorting using GNU Parallel

From Dev

using GNU Parallel for pagination

From Dev

How to force certain groups of targets to run in parallel with GNU Make?

From Dev

How to uninstall GNU parallel?

From Dev

How to use GNU Parallel

From Dev

how do i build libraries in subdirectories using cmake?

From Dev

How to execute one command in multiples directories using "GNU parallel"?

From Dev

GNU Parallel: How do determine job "slot" you're using?

From Dev

How to correctly pass arguments to curl using GNU parallel in BASH

From Java

&& in bash script using GNU "parallel"

From Dev

Python pipeline using GNU Parallel

From Dev

Parallelise rsync using GNU Parallel

From Dev

Python pipeline using GNU Parallel

From Dev

Using variables as inputs in GNU parallel

From Dev

gnu make: how to automatically list and build all targets

From Dev

How to include GNU make auto dependecies with custom build steps

From Dev

How do I run the same exact command, N number of times in parallel using GNU parallel?

From Java

How to insert sleep in GNU parallel?

From Dev

How to assign variables in gnu parallel?

From Dev

How to use GNU parallel with gunzip

From Java

How to optimize webpack's build time using prefetchPlugin & analyse tool?

From Dev

Trigger an action on timeout using gnu parallel

From Dev

Running bash script using gnu parallel

From Dev

Terminal Command to run tests using GNU Parallel

From Dev

Using if else and xargs to pipe into gnu parallel

From Dev

Boosting the grep search using GNU parallel

From Dev

How to optimize spark sql to run it in parallel

Related Related

  1. 1

    How to optimize parallel build of subdirectories using GNU Make?

  2. 2

    How to optimize GNU parallel for this use?

  3. 3

    How to write multicore sorting using GNU Parallel

  4. 4

    using GNU Parallel for pagination

  5. 5

    How to force certain groups of targets to run in parallel with GNU Make?

  6. 6

    How to uninstall GNU parallel?

  7. 7

    How to use GNU Parallel

  8. 8

    how do i build libraries in subdirectories using cmake?

  9. 9

    How to execute one command in multiples directories using "GNU parallel"?

  10. 10

    GNU Parallel: How do determine job "slot" you're using?

  11. 11

    How to correctly pass arguments to curl using GNU parallel in BASH

  12. 12

    && in bash script using GNU "parallel"

  13. 13

    Python pipeline using GNU Parallel

  14. 14

    Parallelise rsync using GNU Parallel

  15. 15

    Python pipeline using GNU Parallel

  16. 16

    Using variables as inputs in GNU parallel

  17. 17

    gnu make: how to automatically list and build all targets

  18. 18

    How to include GNU make auto dependecies with custom build steps

  19. 19

    How do I run the same exact command, N number of times in parallel using GNU parallel?

  20. 20

    How to insert sleep in GNU parallel?

  21. 21

    How to assign variables in gnu parallel?

  22. 22

    How to use GNU parallel with gunzip

  23. 23

    How to optimize webpack's build time using prefetchPlugin & analyse tool?

  24. 24

    Trigger an action on timeout using gnu parallel

  25. 25

    Running bash script using gnu parallel

  26. 26

    Terminal Command to run tests using GNU Parallel

  27. 27

    Using if else and xargs to pipe into gnu parallel

  28. 28

    Boosting the grep search using GNU parallel

  29. 29

    How to optimize spark sql to run it in parallel

HotTag

Archive