INSTALL

== Basic build ==

1. Edit config.mk. Follow the comments there.
   Optionally you can change compiler settings in include_[GCC|CLANG|ICC|MIC|*].mk.
   Please note that only the default compiler flags GCC are supported and tested.
   For 32 bit builds the only supported compiler setting is GCCX86.
   For ARM builds the possible compiler settings are GCCARMv8 and ARMCLANG.
   For POWER builds the possible compiler settings are GCCPOWER and XLC.
2. make (Builds hwloc, lua, Likwid libraries, access daemons and likwid-bench)
3. make install (this is required for likwid-pin and if you use the accessDaemon)

Only the default flags set are tested. As it is not possible to test all
compiler setting variants the Intel icc compiler and Clang is only build tested.
A basic function test is done for the icc binary. The only variant fully tested
is gcc with default compiler flags. It is therefore recommended to use gcc with
the default flags. If you want to use and build the Fortran interface you can mix
GCC with the Intel Fortran Compiler (default setup). You can change the Fortran
compiler in make/include_[GCC|CLANG|ICC|MIC].mk.

*NOTICE*

All generated files are located in the [GCC|ICC|CLANG|MIC|GCCX86|*] build directory.
This includes the dependency files, object files. The
generated source files and the pas and assembly files for likwid-bench are build
in bench/[GCC|ICC|CLANG|MIC|GCCX86].
If you debug your likwid-bench benchmarks you can look at all
intermediate build files and also the final assembly code.

== Build on Xeon Phi (Knights Corner) ==
This section is _only_ for Knights Corner, for Knights Landing use default
installation method!
For builds for the Xeon Phi (Knights Corner) coprocessor, the accessDaemon and
the frequency daemon are disabled. Moreover, the access mode is set to 'direct'.
This was made because it is important to run as few processes as possible on the
Xeon Phi and the accessDaemon would start one process per hardware thread.
In order to build Likwid for the Xeon Phi processor, you have to change the
RPATHS variable in make/include_MIC.mk to point to the folder with the Intel
libraries like libimf.so. This is crucial because when using an suid-root
executable, the LD_LIBRARY_PATH gets lost but Likwid still needs to know where
the Intel libraries reside.
After installation change the owner of likwid-lua to root and set the suid-root
bit for likwid-lua:
chown root <BINPATH>/likwid-lua
chmod u+s <BINPATH>/likwid-lua
Afterwards Likwid can be used as anywhere else.

== Known problems ==

On very old systems with old kernels (< 2.6.7) or old glibc versions likwid
is build with reduced funtionality. This includes missing support for NUMA
and pinning.
likwid-setFrequencies can only be used if the acpi_cpufreq module is loaded. It
is not possible to fix the frequency with the intel_pstate module.

== Additional Targets ==

make clean     -  clean the object directory
make distclean -  clean also the executables/libraries
make uninstall -  delete installed files
make docs      -  generate html documentation using doxygen
make local     -  set paths in Lua files to work from current directory
                  (for testing only! Uses already installed access daemons and
                  libraries. Often you have to set the LD_LIBRARY_PATH to the
                  contain the current folder)

== Dependencies ==
Most parts of the Likwid suite do not have external dependencies that need to be
installed before you can build Likwid. If external libraries are used, they are
shipped with Likwid.

Included dependencies:
- hwloc
- Lua
- Perl Template toolkit
- GOTCHA (experimental)


Build dependencies:
- C compiler (commonly gcc, but clang and icc are also possible)
- make
- Perl

Runtime dependencies for likwid-perfscope:
- gnuplot

For the HTML documentation you further need doxygen.

== Build  accessDaemon ==

Change path for the accessDaemon:

1. Edit config.mk and configure path in ACCESSDAEMON variable. You can overwrite
   it later in likwid.cfg
2. Set the desired default ACCESSMODE. You can overwrite this on the command
   line or likwid.cfg.
2. make will also build the accessDaemon
3. Install with (sudo) make install

With the standard make install target the daemon will also be installed in
to the path in $ACCESSDAEMON. It also sets the user to root and the suid bit.

== Setup of msr module ==

likwid-perfctr, likwid-powermeter, likwid-agent, require the Linux msr kernel
module. This module is part of most standard distro kernels. You have to be root
to do the initial setup.

1. Check if the msr module is loaded with 'lsmod | grep msr'.
   There should be an output.
2. If the module is not loaded, load it with 'modprobe msr'. For automatic
   loading at startup consult your distros documentation how to do so, commonly
   by adding 'msr' to /etc/modules.
3. Adopt access rights on the msr device files for normal user. To allow
   everybody access you can use 'chmod o+rw /dev/cpu/*/msr'.
   This is only recommended on save single user desktop systems and might be not
   enough to grant access to anybody because of POSIX capabilites or other
   security features of your distro.

As a general access to the msr registers is not desired on security sensitive
systems, you can either implement a more sophisticated access rights settings
with e.g. setgid. A common solution used on many other device files, e.g. for
audio, is to introduce a group and make a chown on the msr device files to that
group or use dbus rules. Now if you execute likwid-perfctr with setgid on that
group the executing user can use the tool but cannot directly write or read the
msr device files.

A secure solution is to use the accessDaemon, which encapsulates the access to
the msr device files and performs an address check for allowed registers. For
more information how to setup look at the HTML documentation.

A demo for a root exploit involving the msr device files was published. As
a consequence the security settings for access to the msr device files are
tightened in recent kernels. The exploit used a specify register to alter the
entry point for the current process to a malware. The daemon grants access only
to hardware performance counter related registers.

Just setting the file access rights or using suid root on the access daemon is
not sufficient anymore for some distros. You have to register your binary at the
libcap now to get access. This is only necessary if above setup does not work.

You register the necessary capability by calling

sudo setcap cap_sys_rawio+ep EXECUTABLE

on the executables. Not all file systems support capabilities.
The only feasable way is to register the likwid-accessD and proxy all access
over it.

If you want to avoid the suid-root likwid-accessD, it is still possible to use
direct mode access for normal users. Set the capabilities on the LIKWID binaries
as described above and additionally assign rw file permissions to the user on
/dev/cpu/*/msr (with groups, ACLs, etc.). However, beware of the security risk.

In case access to MSRs is still not possible, you can additionally set
cap_dac_override. Please be aware that this poses a severe security risk, since
it allows accessD (accessdaemon mode) or one of the LIKWID binaries (direct mode)
to read and write all files on the system. This is easily exploitable to become
root. Do not do this unless you know what you are doing.

Notes regarding Intel Sapphire Rapids (and possibly newer) CPUs:
In order to support Uncore Discovery, LIKWID needs read and write access to
/dev/mem. If required, set the capabilities the same way as you do for MSR device
files. Please be reminded that user access to /dev/mem leaves the door open for
anyone who wants to exploit the system. Accordingly, the usage of accessD with
suid-root is highly recommended.

Update for Linux kernel 5.9 and newer:
With Linux 5.9, the msr kernel module got some security fixes. The major change for LIKWID is, that now all MSR are non-writable by default. In order to change that, you have to change the boot options of your operating system to contain msr.allow_writes=on to enable writes again. This affects only ACCESSMODE=direct and ACCESSMODE=accessdaemon. If you use the perf_event backend, you don't have to change anything.

Update for Linux kernel 5.10 and newer:
We got reports, that with Linux 5.10 the PCI accesses are also restricted by security mechanisms. In order to fix this, the access daemon requires an additinal capabilities flag: sudo setcap cap_sys_admin,cap_sys_rawio=ep EXECUTABLE

If you have still problems please let me know on the likwid mailing list:
http://groups.google.com/group/likwid-users

== Building LIKWID with perf_event kernel backend ==
When setting USE_PERF_EVENT in config.mk, the perf_event kernel interface is
used to programm the counters instead of the way through the msr kernel module
and PCI devices. The event and counter lists are the same.
The perf_event interface handles the counter allocation itself, thus the counters
provided to LIKWID might not be the ones perf_event uses internally.
The support is tested for the core-local fixed- and general-purpose counters. The
paranoid setting requires to be 1 or less.
It is also able to use it for Uncore counters but the support is experimental
and requires a paranoid setting of 0 or less.
Currently no counter options are supported.

paranoid setting: /proc/sys/kernel/perf_event_paranoid


== Linking applications with LIKWID lib ==
Commonly the LIKWID C library can be used as all other libraries by setting the
include (-I) and library path (-L) and add -llikwid to the command line.

==== Internal hwloc library ====
The included hwloc library shouldn't make a problem as all hwloc functions are
prefixed with 'likwid_'.

==== Internal Lua library and interpreter ====
The included Lua libray might cause problems because the functions are not
renamed and might interfere with the other Lua library functions.
Maybe it works if the versions of Lua are the same (5.3.2).
The Lua interpreter is renamed at installation to likwid-lua, so does not
overwrite other interpreters.

==== Building LIKWID with support for Nvidia GPUs ====
The Nvidia backend is optional and needs to be activated in config.mk. Make sure
that the CUDA lib/inc paths and the lib/inc paths to CUPTI are adjusted accoring
to your system. The library paths need to be in LD_LIBRARY_PATH at runtime.