Exploratory debug symbols & core dump analysis for Manager (DEB) #23453

ncvicchi · 2024-05-15T23:27:41Z

Related issue
#21736
#21739

Description

This issue aims to make a exploratory session of current #9913 phase 1 development for the Manager instance of the DEB package.

Verifications should be performed on the following issues to check end to end the process from generation of symbols to core/crash dump analysis with them.

These verifications must be performed by a different collaborator than the originally assigned to the issue, and a full detail of procedures, logs and results must be provided.
Evidence of success must be provided as well.

Goals

Verify that binary and debug symbols packages are correctly generated by following the current documentation
Verify that binary & debug symbols packages are automatically uploaded to their designated locations.
Verify that packages perform a successful installation by following the current documentation.
Verify that the installed manager instance runs succesfully.
Verify that a core dump is generated on simulated failure.
Verify that debug symbols are suitable to debug/analyze the core/crash dumps.
Verify that the documentation used during all the exploratory is adequate, correct and complete.
Verify that core dump generation can be enabled and disabled just by following the proper documentation.

DoD

Packages for binaries and debug symbols are generated by following documentation.
Packages are confirmed to be uploaded to their designated location.
Installation is tested and validated.
Installed manager behaves as expected.
A simulation of failure is performed and as a result a core dump is generated.
Core dump is successfully analyzed by using the corresponding debug symbols.
No documentation errors are found or left uncorrected.
Core dump generation is validated to be enabled or disabled just by following the proper documentation.
Extensive evidence and documentation of the exploratory is provided

Approval
DRI Name: @ncvicchi
Objective: Generate debug symbols

Leoquicenoz · 2024-05-16T23:10:32Z

Update

Cloning the wazuh/wazuh repository

git clone https://github.com/wazuh/wazuh.git

Install the necessary dependencies to compile and to generate the .deb package.
Generate the .deb packages, this is done by executing the command sudo ./generate_package.sh -t manager --system deb in the directory wazuh/packages, during the execution of this command the compilation is done and then the generation of the packages.

Compilation

General settings:
    TARGET:             server
    V:                  
    DEBUG:              
    DEBUGAD             
    INSTALLDIR:         /var/ossec
    DATABASE:           
    ONEWAY:             no
    CLEANFULL:          no
    RESOURCES_URL:      https://packages.wazuh.com/deps/25
    EXTERNAL_SRC_ONLY:  
    HTTP_REQUEST_BRANCH:v1.0.0
User settings:
    WAZUH_GROUP:        wazuh
    WAZUH_USER:         wazuh
USE settings:
    USE_ZEROMQ:         no
    USE_GEOIP:          no
    USE_PRELUDE:        no
    USE_INOTIFY:        no
    USE_BIG_ENDIAN:     no
    USE_SELINUX:        yes
    USE_AUDIT:          yes
    DISABLE_SYSC:       no
    DISABLE_CISCAT:     no
    IMAGE_TRUST_CHECKS: 1
    CA_NAME:            DigiCert Assured ID Root CA
Mysql settings:
    includes:           
    libs:               
Pgsql settings:
    includes:           
    libs:               
Defines:
    -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT
Compiler:
    CFLAGS            -pthread -Iexternal/pacman/lib/libalpm/ -Iexternal/libarchive/libarchive -Wl,--start-group -Iexternal/audit-userspace/lib -g -DNDEBUG -O2 -DOSSECHIDS -DUSER="wazuh" -DGROUPGLOBAL="wazuh" -DLinux -DINOTIFY_ENABLED -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DIMAGE_TRUST_CHECKS=1 -DCA_NAME='DigiCert Assured ID Root CA' -DENABLE_SYSC -DENABLE_CISCAT -DENABLE_AUDIT -pipe -Wall -Wextra -std=gnu99 -I./ -I./headers/ -Iexternal/openssl/include -Iexternal/cJSON/ -Iexternal/libyaml/include -Iexternal/curl/include -Iexternal/msgpack/include -Iexternal/bzip2/ -Ishared_modules/common -Ishared_modules/dbsync/include -Ishared_modules/rsync/include -Iwazuh_modules/syscollector/include  -Idata_provider/include  -Iexternal/libpcre2/include -Iexternal/rpm//builddir/output/include -Isyscheckd/include -Ishared_modules/router/include -Ishared_modules/content_manager/include -Iwazuh_modules/vulnerability_scanner/include -I./shared_modules/ 
    LDFLAGS           '-Wl,-rpath,/../lib' -pthread -lrt -ldl -O2 -Lshared_modules/dbsync/build/lib -Lshared_modules/rsync/build/lib  -Lwazuh_modules/syscollector/build/lib -Ldata_provider/build/lib -Lsyscheckd/build/lib
    LIBS              -lrt -ldl -lm 
    CC                gcc
    MAKE              make
make[3]: Leaving directory '/build_wazuh/manager/wazuh-manager-4.9.0/src'

Done building server

Package generation

Finished running lintian.

WARNING generated by debuild:
Making debian/rules executable!

+ get_checksum 4.9.0 36f93d8 no
+ wazuh_version=4.9.0
+ short_commit_hash=36f93d8
+ base_name=wazuh-manager_4.9.0-0
+ symbols_base_name=wazuh-manager-dbg_4.9.0-0
+ [[ amd64 == \p\p\c\6\4\l\e ]]
+ deb_file=wazuh-manager_4.9.0-0_amd64.deb
+ symbols_deb_file=wazuh-manager-dbg_4.9.0-0_amd64.deb
+ [[ no == \n\o ]]
++ sed 's/\.deb/_36f93d8&/'
+ deb_file=wazuh-manager_4.9.0-0_amd64_36f93d8.deb
++ sed 's/\.deb/_36f93d8&/'
+ symbols_deb_file=wazuh-manager-dbg_4.9.0-0_amd64_36f93d8.deb
+ pkg_path=/build_wazuh/manager
+ [[ no == \y\e\s ]]
+ find /build_wazuh/manager -type f -name 'wazuh-manager*deb' -exec mv '{}' /var/local/wazuh/ ';'
++ ls -Art /wazuh/packages/output/
++ tail -n 1
+ echo 'Package wazuh-manager-dbg_4.9.0-0_amd64.deb added to /wazuh/packages/output/.'
Package wazuh-manager-dbg_4.9.0-0_amd64.deb added to /wazuh/packages/output/.
+ return 0
+ return 0
+ clean 0
+ exit_code=0
+ find /wazuh/packages/debs/amd64/manager '(' -name '*.sh' -o -name '*.tar.gz' -o -name 'wazuh-*' ')' '!' -name docker_builder.sh -exec rm -rf '{}' +
+ exit 0

ls -la output/
-rw-r--r-- 1 root root 122806878 may 15 16:35 wazuh-manager_4.9.0-0_amd64.deb
-rw-r--r-- 1 root root  20885282 may 15 16:35 wazuh-manager-dbg_4.9.0-0_amd64.deb

Installation of the manager and debug symbols: sudo dpkg -i wazuh-manager_4.9.0-0_amd64.deb and sudo dpkg -i wazuh-manager-dgb_4.9.0-0_amd64.deb.

Restarting wazuh

/var/ossec/bin/wazuh-control restart
2024/05/16 22:02:30 wazuh-modulesd:router: INFO: Loaded router module.
2024/05/16 22:02:30 wazuh-modulesd:content_manager: INFO: Loaded content_manager module.
wazuh-clusterd not running...
Killing wazuh-modulesd...
Killing wazuh-monitord...
Killing wazuh-logcollector...
Killing wazuh-remoted...
Killing wazuh-syscheckd...
Killing wazuh-analysisd...
wazuh-maild not running...
Killing wazuh-execd...
Killing wazuh-db...
Killing wazuh-authd...
wazuh-agentlessd not running...
wazuh-integratord not running...
wazuh-dbd not running...
wazuh-csyslogd not running...
Killing wazuh-apid...
Wazuh v4.9.0 Stopped
Starting Wazuh v4.9.0...
Started wazuh-apid...
Started wazuh-csyslogd...
Started wazuh-dbd...
2024/05/16 22:02:33 wazuh-integratord: INFO: Remote integrations not configured. Clean exit.
Started wazuh-integratord...
Started wazuh-agentlessd...
Started wazuh-authd...
Started wazuh-db...
Started wazuh-execd...
Started wazuh-analysisd...
Started wazuh-syscheckd...
Started wazuh-remoted...
Started wazuh-logcollector...
Started wazuh-monitord...
2024/05/16 22:02:42 wazuh-modulesd:router: INFO: Loaded router module.
2024/05/16 22:02:42 wazuh-modulesd:content_manager: INFO: Loaded content_manager module.
Started wazuh-modulesd...
Completed.

It can be seen that the manager installed correctly and is running

Enabling and configuring coredump

Install systemd-coredump: sudo apt install systemd-coredump
Configure systemd-coredump

sudo nano /etc/systemd/coredump.conf

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the coredump.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/coredump.conf' to display the full config.
#
# See coredump.conf(5) for details.

[Coredump]
#Storage=external
#Compress=yes
#ProcessSizeMax=2G
#ExternalSizeMax=2G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=

Restart systemd-coredump: sudo systemctl restart systemd-coredump*
Check status systemd-coredump

systemctl status systemd-coredump*

● systemd-coredump.socket - Process Core Dump Socket
     Loaded: loaded (/lib/systemd/system/systemd-coredump.socket; static)
     Active: active (listening) since Thu 2024-05-16 15:07:44 UTC; 28min ago
       Docs: man:systemd-coredump(8)
     Listen: /run/systemd/coredump (SequentialPacket)
   Accepted: 0; Connected: 0;
     CGroup: /system.slice/systemd-coredump.socket

May 16 15:07:44 ubuntu2204.localdomain systemd[1]: systemd-coredump.socket: Dea>
May 16 15:07:44 ubuntu2204.localdomain systemd[1]: Closed Process Core Dump Soc>
May 16 15:07:44 ubuntu2204.localdomain systemd[1]: Stopping Process Core Dump S>
May 16 15:07:44 ubuntu2204.localdomain systemd[1]: Listening on Process Core Du>

Generate crash of one of Wazuh's modules

wazuh-analysisd crash

See process details, in this case PID 1636 corresponds to the wazuh-analysisd module.

ps -f -p 1636
wazuh       1636       1     0       14:34      ?        00:00:02      /var/ossec/bin/wazuh-analysisd

Generating the crash (The SIGSEGV signal (Segmentation Fault) generates a segmentation fault crash.

sudo kill -SIGSEGV 1636

Look for the generated dump file.

ls /var/lib/systemd/coredump
core.wazuh-analysisd.114.f77b35a11c5e4dd8bc60024d1871199b.1636.1715885520000000.zst

I use zstd to unzip the file and rename it, but first I have to install zstd because I don't have it installed.

sudo apt install zstd

zstd -d /var/lib/systemd/coredump/core.wazuh-analysisd.114.f77b35a11c5e4dd8bc60024d1871199b.1636.1715885520000000.zst -o core.wazuh-analysisd.1636
/var/lib/systemd/coredump/core.wazuh-analysisd.114.f77b35a11c5e4dd8bc60024d1871199b.1636.1715885520000000.zst: 289509376 bytes

Debug crash dump

gdb output

gdb /var/ossec/bin/wazuh-analysisd core.wazuh-analysisd.1636

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/ossec/bin/wazuh-analysisd...
Reading symbols from /usr/lib/debug//var/ossec/bin/wazuh-analysisd...

warning: Can't open file /bin/wazuh-analysisd during file-backed mapping note processing

warning: Can't open file /lib/libwazuhext.so during file-backed mapping note processing
[New LWP 1636]
[New LWP 1638]
[New LWP 1639]
[New LWP 1644]
[New LWP 1645]
[New LWP 1646]
[New LWP 1647]
[New LWP 1648]
[New LWP 1649]
[New LWP 1650]
[New LWP 1653]
[New LWP 1654]
[New LWP 1655]
[New LWP 1656]
[New LWP 1658]
[New LWP 1660]
[New LWP 1664]
[New LWP 1665]
[New LWP 1667]
[New LWP 1668]
[New LWP 1640]
[New LWP 1641]
[New LWP 1642]
[New LWP 1643]
[New LWP 1651]
[New LWP 1652]
[New LWP 1657]
[New LWP 1659]
[New LWP 1661]
[New LWP 1662]
[New LWP 1663]
[New LWP 1666]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/ossec/bin/wazuh-analysisd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb2317d57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffdc175bef0, rem=rem@entry=0x7ffdc175bef0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78	../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
[Current thread is 1 (Thread 0x7fb23267f740 (LWP 1636))]
(gdb) bt
#0  0x00007fb2317d57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffdc175bef0, rem=rem@entry=0x7ffdc175bef0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007fb2317da677 in __GI___nanosleep (req=req@entry=0x7ffdc175bef0, rem=rem@entry=0x7ffdc175bef0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2  0x00007fb2317da5ae in __sleep (seconds=0, seconds@entry=1) at ../sysdeps/posix/sleep.c:55
#3  0x000000000042e350 in OS_ReadMSG (m_queue=<optimized out>, m_queue@entry=4) at analysisd/analysisd.c:1095
#4  0x0000000000408deb in main (argc=<optimized out>, argv=<optimized out>) at analysisd/analysisd.c:852
(gdb) info locals
sc_cancel_oldtype = 0
sc_ret = <optimized out>
r = <optimized out>

aritosteles · 2024-05-20T11:15:44Z

Replicated previous test:

1. Cloned wazuh/wazuh repository:

- git clone https://github.com/wazuh/wazuh.git
- checkout enhancement/9913..epic

2. Installed dependencies:

- Docker: https://documentation.wazuh.com/current/deployment-options/docker/docker-installation.html
- Gcc, Cmake, etc: https://documentation.wazuh.com/current/deployment-options/wazuh-from-sources/wazuh-server/index.html

3. Generated rpm packages:

./generate_package.sh -t manager --system deb

4. Installed the manager and debug symbols:

sudo dpkg -i wazuh-manager_4.9.0-0_amd64.deb
sudo dpkg -i wazuh-manager-dgb_4.9.0-0_amd64.deb.

5. Restart wazuh:

/var/ossec/bin/wazuh-control restart

6. Enabling and configuring coredump:

- Install systemd-coredump: 
    sudo apt install systemd-coredump
- Configure systemd-coredump:
    sudo nano /etc/systemd/coredump.conf

 - Restart systemd-coredump: 
    sudo systemctl restart systemd-coredump*

 - Check status systemd-coredump
    systemctl status systemd-coredump*

7- Get list of wazuh processes:

ps -ef | grep wazuh

8- Generate crash in one of wazuh processes:

sudo kill -SIGSEGV 473322

9- Look for generated dump file

ls /var/lib/systemd/coredump

10- Install zstd and unzip file:

sudo apt install zstd

11- gdb output:

gdb /var/ossec/bin/wazuh-analysisd wazuh-analisysd-core-dump

root@Ubuntu-template:/home/ariel/wazuh/packages/output# gdb /var/ossec/bin/wazuh-analysisd wazuh-analisysd-core-dump
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/ossec/bin/wazuh-analysisd...
Reading symbols from /usr/lib/debug//var/ossec/bin/wazuh-analysisd...

warning: Can't open file /bin/wazuh-analysisd during file-backed mapping note processing

warning: Can't open file /lib/libwazuhext.so during file-backed mapping note processing
[New LWP 473322]
[New LWP 473334]
[New LWP 473335]
[New LWP 473338]
[New LWP 473361]
[New LWP 473340]
[New LWP 473344]
[New LWP 473346]
[New LWP 473331]
[New LWP 473349]
[New LWP 473352]
[New LWP 473333]
[New LWP 473341]
[New LWP 473355]
[New LWP 473360]
[New LWP 473343]
[New LWP 473363]
[New LWP 473345]
[New LWP 473348]
[New LWP 473364]
[New LWP 473366]
[New LWP 473351]
[New LWP 473367]
[New LWP 473354]
[New LWP 473358]
[New LWP 473368]
[New LWP 473359]
[New LWP 473373]
[New LWP 473362]
[New LWP 473374]
[New LWP 473372]
[New LWP 473375]
[New LWP 473377]
[New LWP 473376]
[New LWP 473329]
[New LWP 473330]
[New LWP 473332]
[New LWP 473336]
[New LWP 473337]
[New LWP 473339]
[New LWP 473342]
[New LWP 473347]
[New LWP 473350]
[New LWP 473353]
[New LWP 473356]
[New LWP 473357]
[New LWP 473365]
[New LWP 473369]
[New LWP 473370]
[New LWP 473371]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by '/var/ossec/bin/wazuh-analysisd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007b584bae57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fffcce7c6f0, rem=rem@entry=0x7fffcce7c6f0)
at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
78 ../sysdeps/unix/sysv/linux/clock_nanosleep.c: No such file or directory.
[Current thread is 1 (Thread 0x7b584caa9740 (LWP 473322))]
(gdb) bt
#0 0x00007b584bae57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fffcce7c6f0, rem=rem@entry=0x7fffcce7c6f0)
at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1 0x00007b584baea677 in __GI___nanosleep (req=req@entry=0x7fffcce7c6f0, rem=rem@entry=0x7fffcce7c6f0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2 0x00007b584baea5ae in __sleep (seconds=0, seconds@entry=1) at ../sysdeps/posix/sleep.c:55
#3 0x000000000042e350 in OS_ReadMSG (m_queue=, m_queue@entry=4) at analysisd/analysisd.c:1095
#4 0x0000000000408deb in main (argc=, argv=) at analysisd/analysisd.c:852
(gdb) info locals
sc_cancel_oldtype = 0
sc_ret =
r =
(gdb)

ncvicchi · 2024-05-20T20:02:00Z

Blocked until access to wazuh internal repositories is provided

ncvicchi · 2024-05-29T21:12:53Z

Exploratory has been repeated with almost identical results on Ubuntu 24.04.

Some minor fixes were implemented in scripts and updated documentation was followed instead (which requires to clone wazuh/wazuh repository instead of wazuh/wazuh-packages as it was before, as it major difference).

Automatic upload is still failing and the cause is being tracked down. Both packages seem to be generated but not being found later.

ncvicchi mentioned this issue May 15, 2024

Exploratory extensive testing to ensure reliability and performance. #21739

Closed

6 tasks

ncvicchi added type/enhancement New feature or request level/subtask labels May 15, 2024

ncvicchi assigned Leoquicenoz May 16, 2024

aritosteles self-assigned this May 20, 2024

ncvicchi assigned ncvicchi and unassigned Leoquicenoz May 20, 2024

Dwordcito unassigned aritosteles May 28, 2024

Dwordcito changed the title ~~Exploratory extensive testing to ensure reliability and performance of debug symbols & core dump analysis for Manager (DEB)~~ Exploratory debug symbols & core dump analysis for Manager (DEB) May 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exploratory debug symbols & core dump analysis for Manager (DEB) #23453

Exploratory debug symbols & core dump analysis for Manager (DEB) #23453

ncvicchi commented May 15, 2024 •

edited by aritosteles

Leoquicenoz commented May 16, 2024

aritosteles commented May 20, 2024 •

edited

ncvicchi commented May 20, 2024

ncvicchi commented May 29, 2024

Exploratory debug symbols & core dump analysis for Manager (DEB) #23453

Exploratory debug symbols & core dump analysis for Manager (DEB) #23453

Comments

ncvicchi commented May 15, 2024 • edited by aritosteles

Description

Goals

DoD

Leoquicenoz commented May 16, 2024

Update

aritosteles commented May 20, 2024 • edited

1. Cloned wazuh/wazuh repository:

2. Installed dependencies:

3. Generated rpm packages:

4. Installed the manager and debug symbols:

5. Restart wazuh:

6. Enabling and configuring coredump:

7- Get list of wazuh processes:

8- Generate crash in one of wazuh processes:

9- Look for generated dump file

10- Install zstd and unzip file:

11- gdb output:

ncvicchi commented May 20, 2024

ncvicchi commented May 29, 2024

ncvicchi commented May 15, 2024 •

edited by aritosteles

aritosteles commented May 20, 2024 •

edited