Skip to content

gniuk/linux-compile-commands

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GENERATE LINUX KERNEL COMPILATION DATABASE

Prerequisites

A modern Linux host

Though Ubuntu 16.04 used here, any modern Linux Distributions should work. Tested on Arch Linux.

compiledb

python2 or python3 compiledb command

$ pip install compiledb

Linux 2.0.27

  1. decompress the source tar ball
    $ mkdir linux-2.0.27 && tar -xvf linux-2.0.27.tar.xz -C linux-2.0.27 --strip-components 1
        
  2. make menuconfig
    $ make menuconfig
        

    Notice: the path of the ncurses library on your host system may be not at the position as old linux expected.

    make -C scripts/lxdialog all
    -e
    >> Unable to find the Ncurses libraries.
    >>
    >> You must have Ncurses installed in order
    >> to use 'make menuconfig'
    Makefile:34: recipe for target 'ncurses' failed
        

    To fix it, install ncurses lib via your package manager and then modify scripts/lxdialog/Makefile:29 as follows:

    all: ncurses lxdialog -> all: lxdialog
        

    Let gcc find libncurses automatically. Then rerun make menuconfig.

    https://raw.githubusercontent.com/gniuk/linux-compile-commands/master/image/linux-2.0.27_menuconfig.png

  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. tune two Makefiles to support dry-run without error
    a. fs/Makefile:
       11 # L_OBJS    = $(join $(SUB_DIRS),$(SUB_DIRS:%=/%.o))
       12 L_OBJS     =
    
    b. net/Makefile:
       52 # L_OBJS             := socket.o protocols.o sysctl_net.o $(join $(SUB_DIRS),$(SUB_DIRS:%=/%.o))
       53 L_OBJS       := socket.o protocols.o sysctl_net.o
        
  5. generate build log
    $ LANGUAGE=en make V=1 -j1 bzImage modules --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun.

  6. generate compile_commands.json
    $ compiledb < build-log.txt
        
  7. [OPTIONAL] tune compile_commands.json

    Add more general CFLAGS, some index backend engines might need them.

    $ sed -i.bak '/m486/a \ \ \ "-march=i386",\n\ \ \ "-m32",' compile_commands.json
        

Linux 2.0.33

  1. decompress the source tar ball
    $ mkdir linux-2.0.33 && tar -xvf linux-2.0.33.tar.xz -C linux-2.0.33 --strip-components 1
        
  2. make menuconfig
    $ make menuconfig
        

    Notice: the path of the ncurses library on your host system may be not at the position as old linux expected.

    make -C scripts/lxdialog all
    -e
    >> Unable to find the Ncurses libraries.
    >>
    >> You must have Ncurses installed in order
    >> to use 'make menuconfig'
    Makefile:34: recipe for target 'ncurses' failed
        

    To fix it, install ncurses lib via your system package manager and then modify scripts/lxdialog/Makefile:29 as follows:

    all: ncurses lxdialog -> all: lxdialog
        

    Let gcc find libncurses automatically. Fix a bug in scripts/lxdialog/lxdialog.c:60 before going on.

    const char *title = NULL; -> char *title = NULL;
        

    Then rerun make menuconfig.

  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. tune two Makefiles to support dry-run without error
    a. fs/Makefile:
       11 # L_OBJS    = $(join $(SUB_DIRS),$(SUB_DIRS:%=/%.o))
       12 L_OBJS     =
    
    b. net/Makefile:
       52 # L_OBJS             := socket.o protocols.o sysctl_net.o $(join $(SUB_DIRS),$(SUB_DIRS:%=/%.o))
       53 L_OBJS       := socket.o protocols.o sysctl_net.o
        
  5. generate build log
    $ LANGUAGE=en make V=1 -j1 bzImage modules --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun.

  6. generate compile_commands.json
    $ compiledb < build-log.txt
        
  7. [OPTIONAL] tune compile_commands.json

    Add more general CFLAGS, some index backend engines might need them.

    $ sed -i.bak '/m486/a \ \ \ "-march=i386",\n\ \ \ "-m32",' compile_commands.json
        

Linux 2.2.14

  1. decompress the source tar ball
    $ mkdir linux-2.2.14 && tar -xvf linux-2.2.14.tar.xz -C linux-2.2.14 --strip-components 1
        
  2. make menuconfig
    $ make ARCH=i386 menuconfig
        

    Notice: the path of the ncurses library on your host system may be not at the position as old linux expected.

    make -C scripts/lxdialog all
    -e
    >> Unable to find the Ncurses libraries.
    >>
    >> You must have Ncurses installed in order
    >> to use 'make menuconfig'
    Makefile:34: recipe for target 'ncurses' failed
        

    To fix it, install ncurses lib via your system package manager and then modify scripts/lxdialog/Makefile:28 as follows:

    all: ncurses lxdialog -> all: lxdialog
        

    Let gcc find libncurses automatically. Then rerun make ARCH=i386 menuconfig.

  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. generate build log
    $ LANGUAGE=en make V=1 ARCH=i386 -j1 bzImage modules --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun.

  5. generate compile_commands.json
    $ compiledb < build-log.txt
        
  6. [OPTIONAL] tune compile_commands.json

    Add more general CFLAGS, some index backend engines might need them.

    $ sed -i.bak '/m486/a \ \ \ "-march=i386",\n\ \ \ "-m32",' compile_commands.json
        

Linux 2.4.0

  1. decompress the source tar ball
    $ mkdir linux-2.4.0 && tar -xvf linux-2.4.0.tar.xz -C linux-2.4.0 --strip-components 1
        
  2. make menuconfig
    $ make ARCH=i386 menuconfig
        
  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. generate build log
    $ LANGUAGE=en make V=1 ARCH=i386 -j1 bzImage modules --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun.

  5. generate compile_commands.json
    $ compiledb < build-log.txt
        
  6. tune compile_commands.json

    Some index backend engines may not work well using this compile_commands.json, since some CFLAGS needed by modern x86_64 compilers are missing in old kernel config. Add -m32 to the compile_commands.json:

    $ sed -i.bak '/march=i686/a \ \ \ "-m32",' compile_commands.json
        

Linux 2.4.18

  1. decompress the source tar ball
    $ mkdir linux-2.4.18 && tar -xvf linux-2.4.18.tar.xz -C linux-2.4.18 --strip-components 1
        
  2. make menuconfig
    $ make ARCH=i386 menuconfig
        
  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. generate build log
    $ LANGUAGE=en make V=1 ARCH=i386 -j1 bzImage modules --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun.

  5. generate compile_commands.json
    $ compiledb < build-log.txt
        
  6. tune compile_commands.json

    Some index backend engines may not work well using this compile_commands.json, since some CFLAGS needed by modern x86_64 compilers are missing in old kernel config. Add -m32 to the compile_commands.json:

    $ sed -i.bak '/march=i686/a \ \ \ "-m32",' compile_commands.json
        

Linux 2.6.11

  1. decompress the source tar ball
    $ mkdir linux-2.6.11 && tar -xvf linux-2.6.11.tar.xz -C linux-2.6.11 --strip-components 1
        
  2. make menuconfig
    $ make ARCH=i386 menuconfig
        
  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. fix a bug in Makefile
    drivers/media/dvb/b2c2/Makefile:4: *** missing separator.  Stop.
    scripts/Makefile.build:311: recipe for target 'drivers/media/dvb/b2c2' failed
    
      4 # obj-$(CONFIG_DVB_B2C2_USB) + = b2c2-usb.o
      5 obj-$(CONFIG_DVB_B2C2_USB) += b2c2-usb.o
        
  5. generate build log
    $ LANGUAGE=en make V=1 ARCH=i386 -j1 --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun. The fail of the final linkage of vmlinux does not matter, since the total compilation has finished.

  6. generate compile_commands.json
    $ compiledb < build-log.txt
        

Linux 2.6.24

  1. decompress the source tar ball
    $ mkdir linux-2.6.24 && tar -xvf linux-2.6.24.tar.xz -C linux-2.6.24 --strip-components 1
        
  2. make menuconfig

    choose ARCH, i386 or x86_64

    $ make ARCH=i386 menuconfig
        

    or just use a common default config, and skip step 3.

    $ make ARCH=i386 defconfig
        

    Note: The Makefile in src root dir has syntax error using modern make, fix that first.

    434 config %config: scripts_basic outputmakefile FORCE
    435         $(Q)mkdir -p include/linux include/config
    436         $(Q)$(MAKE) $(build)=scripts/kconfig $@
    
    -->
    
    config: scripts_basic outputmakefile FORCE
            $(Q)mkdir -p include/linux include/config
            $(Q)$(MAKE) $(build)=scripts/kconfig $@
    %config: scripts_basic outputmakefile FORCE
            $(Q)mkdir -p include/linux include/config
            $(Q)$(MAKE) $(build)=scripts/kconfig $@
    
    1506 / %/: prepare scripts FORCE
    1507         $(cmd_crmodverdir)
    1508         $(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \
    1509         $(build)=$(build-dir)
    
    -->
    
    /: prepare scripts FORCE
            $(cmd_crmodverdir)
            $(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \
            $(build)=$(build-dir)
    %/: prepare scripts FORCE
            $(cmd_crmodverdir)
            $(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1) \
            $(build)=$(build-dir)
        
  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis
  4. generate build log
    $ LANGUAGE=en make V=1 ARCH=i386 -j1 --dry-run |& tee build-log.txt
        

    This does not compile the kernel actually, just dryrun. The fail of the final linkage of vmlinux does not matter, since the total compilation has finished.

  5. generate compile_commands.json
    $ compiledb < build-log.txt
        

Linux 2.6.34

  1. decompress the source tar ball
    $ mkdir linux-2.6.34 && tar -xvf linux-2.6.34.tar.xz -C linux-2.6.34 --strip-components 1
        
  2. make menuconfig

    choose ARCH, i386 or x86_64

    $ make ARCH=i386 menuconfig
        

    This config will base on your host’s /boot/config of the host kernel.

    or just use a common default config

    $ make ARCH=i386 defconfig
        
  3. config the kernel as you want, or use the linux-x.x.x-config in this repo as a basis

    You may need to make menuconfig again after make defconfig to disable the “Device Drivers -> Graphics support -> Bootup logo”, which causes the dryrun fail prematurely. If you want a real compilation of the kernel source, just skip this step after make defconfig.

  4. [OPTIONAL] prepare a real compilation of the kernel if you want

    a. install gcc-4.x multilib to support the compiling, here I use 4.6, 4.9 should be ok, not tested

    b. modify a Makefile to support gcc 4.x to compile

    arch/x86/vdso/Makefile
     28 # VDSO_LDFLAGS_vdso.lds = -m elf_x86_64 -Wl,-soname=linux-vdso.so.1 \
     29 VDSO_LDFLAGS_vdso.lds = -m64 -Wl,-soname=linux-vdso.so.1 \
    
     72 # VDSO_LDFLAGS_vdso32.lds = -m elf_i386 -Wl,-soname=linux-gate.so.1
     73 VDSO_LDFLAGS_vdso32.lds = -m32 -Wl,-soname=linux-gate.so.1
        

    c. perl scritps may need to be modified to support more recent perl interpreter, e.g.

    kernel/timeconst.pl
     373         # if (!defined(@val)) {
     374         if (!@val) {
        
  5. generate build log

    note: [a] and [b] are exclusive, use one of them according to real compile or dryrun.

    [a]. do a real compilation of the kernel and get the build log, this requires step[4]

    $ LANGUAGE=en make V=1 CC=gcc-4.6 ARCH=i386 -j4 |& tee build-log.txt
        

    ARCH x86_64 should be the same as i386, the gcc-4.x multilib version should be used if both i386 and x86_64 need to be supported.

    [b]. get the build log using make dryrun, if not real compilation

    Before we can dryrun, “Device Drivers -> Graphics support -> Bootup logo” should be disabled.

  6. generate compile_commands.json
    $ compiledb < build-log.txt
        

Linux 3.x - latest

  1. The 3.x and 4.x versions should be the same as 2.6.34. If dryrun fails, fix the problems or JUST DO A REAL [CROSS] COMPILATION on your host. The difference may be that the gcc version used is varied.
  2. Since kernel v5, scripts/gen_compile_commands.py can be used to generate the compile_commands.json natively. Just compile the kernel, and run the script. e.g.
    $ make ARCH=x86_64 defconfig
    $ make -j8
    $ scripts/gen_compile_commands.py
        

Bonus: Linux 0.12

  1. decompress the source tar ball
    $ tar -xvf linux-0.12.tar.gz
        
  2. generate build log

    note: [a] and [b] are exclusive, use one of them.

    [a]. use linux-0.12-gen_build_log.sh to generate the build log

    $ cp /PATH/TO/linux-0.12-gen_build_log.sh linux-0.12/
    $ cd linux-0.12 && bash ./linux-0.12-gen_build_log.sh
        

    [b]. or use linux-0.12-gen_build_log.mk to generate the build log

    $ cp /PATH/TO/linux-0.12-gen_build_log.mk linux-0.12/
    $ cd linux-0.12 && make -f linux-0.12-gen_build_log.mk |& tee build-log.txt
        
  3. generate compile_commands.json
    $ compiledb < build-log.txt
        
  4. [OPTIONAL] add -m32
    $ sed -i.bak '/nostdinc/a \ \ \ "-m32",' compile_commands.json
        

Dry-Run vs Real Compilation

The compile database of a real compilation get all the files involved in the compilation. The compile database of dry-run might miss some seperate targets despite all the kernel vmlinux compilation commands that successfully generated. The missing targets are mostly in arch/$ARCH/boot/, and some helping tools and scripts. Files in arch/$ARCH/boot of ancient kernel source are mostly ASM files, which are not able to be indexed by clang based C/C++ indexers. Routines or symbols in the .S asm files can be easily found via grep tools like ripgrep . The arch/$ARCH/boot of relatively new kernels can be indexed via a real compilation.

So we consider the compile_commands.json from dry-run a good enough compilation database when indexing ancient kernel source.

About

generate compile_commands.json for old linux kernel source

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published