BaseTools: ninja build system for faster builds

[xpahos]

Description

The current makefiles are executed sequentially. The VfrCompile build also includes an antlr build that generates code over a long period of time. Ninja allows to parallelise builds of different utilities and build them 2.5 times faster.

• Breaking change?
  • Breaking change - Does this PR cause a break in build or boot behavior?
  • Examples: Does it add a new library class or move a module to a different repo.
• Impacts security?
  • Security - Does this PR have a direct security impact?
  • Examples: Crypto algorithm change or buffer overflow fix.
• Includes tests?
  • Tests - Does this PR include any explicit test code?
  • Examples: Unit tests or integration tests.

How This Was Tested

[] for i in $(seq 1 500); do make -C BaseTools clean >/dev/null 2>&1; time make -C BaseTools >/dev/null; done

real	0m22.448s
real	0m22.244s
real	0m22.269s
real	0m22.344s
real	0m22.224s

[] for i in $(seq 1 500); do make -C BaseTools clean >/dev/null 2>&1; time make -C BaseTools ninja >/dev/null; done

real	0m9.488s
real	0m9.453s
real	0m9.582s
real	0m9.605s
real	0m9.644s

Integration Instructions

N/A

[bexcran]

I've been wanting to switch the build system to Ninja for a while, so this is a step in the right direction!
PatchCheck is failing:

Code format is not valid:
 * Trailing whitespace found
   File: BaseTools/Source/C/LzmaCompress/GNUmakefile
   Line: override NINJA_CXXSRCS := 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/app.makefile
   Line: NINJA_DEPS = 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/app.makefile
   Line:   NINJA_DEPS +=  || 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/header.makefile
   Line: NINJA_DIR = 
 * Trailing whitespace found
   File: BaseTools/Source/C/VfrCompile/GNUmakefile
   Line: 	@echo "build $(NINJA_DIR)/VfrUtilityLib.o: cxx_vfr_$(NINJA_ID) $(NINJA_DIR)/VfrUtilityLib.cpp || $(NINJA_DIR)/EfiVfrParser.o" >> build.ninja 

[xpahos]

I've been wanting to switch the build system to Ninja for a while, so this is a step in the right direction! PatchCheck is failing:

Code format is not valid:
 * Trailing whitespace found
   File: BaseTools/Source/C/LzmaCompress/GNUmakefile
   Line: override NINJA_CXXSRCS := 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/app.makefile
   Line: NINJA_DEPS = 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/app.makefile
   Line:   NINJA_DEPS +=  || 
 * Trailing whitespace found
   File: BaseTools/Source/C/Makefiles/header.makefile
   Line: NINJA_DIR = 
 * Trailing whitespace found
   File: BaseTools/Source/C/VfrCompile/GNUmakefile
   Line: 	@echo "build $(NINJA_DIR)/VfrUtilityLib.o: cxx_vfr_$(NINJA_ID) $(NINJA_DIR)/VfrUtilityLib.cpp || $(NINJA_DIR)/EfiVfrParser.o" >> build.ninja 

I was thinking about bazel/buck2, they are much more flexible than Ninja. But it would be hard to support two build systems. Ninja seems to be good enough to be a meta-generated layer that runs in parallel.

[bexcran]

I was thinking about bazel/buck2, they are much more flexible than Ninja. But it would be hard to support two build systems. Ninja seems to be good enough to be a meta-generated layer that runs in parallel.

I'm hoping we can switch entirely from GNU Make to Ninja in future.
Ninja seems a good choice because Qemu also uses it, and so it's something I think people are likely to already have installed.

[lgao4]

This change is just for BaseTools. Have you evaluated to apply ninja for platform build?

[xpahos]

This change is just for BaseTools. Have you evaluated to apply ninja for platform build?

I can't figure out how to do the dependency graph properly yet. I found a hack to generate compile_commands.json, so far I'm trying to do something with it. The problem is that ninja doesn't have the most convenient dependencies. When building a large graph, the dependencies look like an endless series of objects. I wanted to start with BaseTools as they are more close to the Linux world. It's still difficult with the main project, as it's a project with a lot of history, and in some places I don't really understand why some things are done the way they are. But as a quick option, I'll try to make a similar compile_commands.json.