Skip to content

Setting up a dunedaq v5.1.0 Development Area

Jump to bottom
Gordon Crone edited this page Jul 1, 2024 · 24 revisions

01-July-2024 - Work in progress (the following steps have been verified to work) (Thanks to Kurt for making a wonderful "getting started" template)

Reference links:

  1. create a new software area based on the latest nightly build (see step 1.iv for the exact dbt-create command to use)

    1. The steps for this are based on the latest instructions for daq-buildtools

    2. As always, you should verify that your computer has access to /cvmfs/dunedaq.opensciencegrid.org

    3. If you are using one of the np04daq computers, and need to clone packages, add the following lines to your .gitconfig file (once you do this, there will be no need to activate the web proxy each time you log in, and this means that you won't forget to disable it...):

      [http]
  proxy = http://np04-web-proxy.cern.ch:3128
  sslVerify = false

    4. Here are the steps for creating the new software area:

      cd <directory_above_where_you_want_the_new_software_area>
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest_v5
dbt-create -n NFD_DEV_240620_A9 <work_dir>
cd <work_dir>

    Do not add -q to the dbt-create command as we need a local Python environment to install drunc

    1. Please note that if you are following these instructions on a computer on which the DUNE-DAQ software has never been run before, there are several system packages that may need to be installed on that computer. These are mentioned in this script. To check whether a particular one is already installed, you can use a command like yum list libzstd and check whether the package is listed under Installed Packages.

  2. add the needed repositories to the /sourcecode area.

    1. decide if you want the very latest code, or a more stable set of packages that has been verified to work.

      Run this command to select the very latest code

      export use_very_latest_dunedaq_code=1

    2. clone the repositories (the following block has some extra directory checking; it can all be copy/pasted into your shell window)

      # change directory to the "sourcecode" subdir, if possible and needed
if [[ -d "sourcecode" ]]; then
    cd sourcecode
fi
# double-check that we're in the correct subdir
current_subdir=`echo ${PWD} | xargs basename`
if [[ "$current_subdir" != "sourcecode" ]]; then
    echo ""
    echo "*** Current working directory is not \"sourcecode\", skipping repo clones"
else
    # finally, do the repo clone(s)

    # We always get appmodel so that we can look at the configurations
    # If you want to run the dfmodules unit tests, clone dfmodules as well
    #if [ ${use_very_latest_dunedaq_code:-0} -eq 1 ]; then
        git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
        git clone https://github.com/DUNE-DAQ/dfmodules.git -b develop
        git clone https://github.com/DUNE-DAQ/drunc.git -b develop
    #else
    #fi
    cd ..
fi

  3. setup the work area, possibly install the latest nanorc version, and build the software

    source env.sh
dbt-build -j 20
dbt-workarea-env

  4. install drunc

    cd sourcecode/drunc
pip install .
cd -

  5. dfmodules contains unit tests which have been updated to use OKS, they can be run with

    dbt-unittest-summary.sh

  6. When you return to working with the software area after logging out, the steps that you'll need to redo are the following:

    • cd <work_dir>
    source ./env.sh
dbt-build  # if needed
dbt-workarea-env  # if needed

Running the test-session example configuration from appmodel with drunc

If you are not running at EHN1 you will need to change the session-env-connectivity-server Variable in the session to point to a local instance of the connectivity server by settting its value attribute approriately (e.g. "localhost"). If you have podman or docker available pn your system, you can start a local connectivty server as simply as

 podman pull ghcr.io/DUNE-DAQ/connectivityserver
 podman run  -p127.0.0.1:30005:5000  dune-daq/connectivityserver

Drunc Configuration

We use a configuration coming from the drunc/data directory to start up the process_manager. The choices for the test configuration include

  • data/process-manager-no-kafka.json - no kafka connection (for drunc development)
  • data/process-manager-CERN-kafka.json - connect to the kafka instance running on the np04 cluster
  • data/process-manager-k8s.json - K8s version of drunc
  • data/process-manager-pocket-kafka.json - docker version of kafka (recommended for expert use only) Then to run the drunc CLI, run
drunc-unified-shell file://<path/to/configuration/file/configuration_file_name>

where <path/to/configuration/file/configuration_file_name> is either the relative or absolute path + configuration file name found in the $workDir/sourcecode/drunc/data/. For first time setup this will likely be (from your working directory)

drunc-unified-shell file://sourcecode/drunc/data/process-manager-no-kafka.json

This sets up drunc's unified shell.

Boot

You can now boot test-session in the same terminal using

boot test/config/test-session.data.xml test-session

In this command

  • boot - drunc-unified-shell command that parses the configuration file and looks for the session name
  • test/config/test-session.data.xml - appmodel configuration file
  • test-session - session name

Finite State Machine (FSM) sequences

The FSM will then take all the commands for operating the run control. An outline of the operating instructions can be found in https://github.com/DUNE-DAQ/drunc/wiki/Running-drunc.

Running the test-session example configuration from appmodel with nanorc

  1. We still need a configuration "boot.json" file to instantiate applications in NanoRC. Create a configuration directory (e.g. test_config) and create a link to the sample boot.json:

    mkdir test_config && cd test_config
if [ -f ../sourcecode/appdal/test/boot.json ];then
  ln -s ../sourcecode/appdal/test/boot.json
else
  wget https://github.com/DUNE-DAQ/appdal/raw/develop/test/boot.json
fi
cd ..

    A few notes on the sample file:

    • Since this is a pre-generated boot.json, it defaults to starting the Connectivity Service on port 15000.
    • This configuration starts the ru-01, ru-02, ru-03, df-01, df-02, df-03, dfo-01, and ta-01 apps defined in the OKS database
    • The OKS "database" file is in sourcecode/appdal/test/config and contains the rest of the configuration information.
    • You can test the SmartDaqApplication generation of modules and connections using 
      listApps test-session sourcecode/appdal/test/config/test-session.data.xml
generate_modules_test test-session <app_name> sourcecode/appdal/test/config/test-session.data.xml # <app_name> is one of the apps listed by listApps

  2. nanorc --partition-number <num> <config name> test-session boot conf wait 10 start_run <run number> wait 60 stop_run scrap terminate

    • e.g. nanorc --partition-number 2 test_config test-session boot conf wait 10 start_run 111 wait 60 stop_run scrap terminate
    • or, you can simply invoke nanorc --partition-number 2 test_config test-session by itself and input the commands individually
    • Note that the session name "test-session" refers to the OKS configuration database https://github.com/DUNE-DAQ/appdal/blob/develop/test/config/test-session.data.xml

The test-session Example Configuration (And setting up your own)

  • The current example configuration is here: https://github.com/DUNE-DAQ/appmodel/blob/develop/test/config/test-session.data.xml
  • It consists of several files in both appmodel/test/config (configuration-specific) and appmodel/config/appmodel (somewhat more generic)
    1. test-session.data.xml is the main entry point for this configuration; it includes the other data files and defines the high-level objects like the Session, the DetectorConfig, and some of the Service entries.
    2. hosts.data.xml defines the hosts and processing resources
    3. wiecconfs.data.xml defines the "front end" electronics configuration
    4. ru-segment.data.xml defines the readout applications (ru-01, ru-02, ru-03)
    5. df-segment.data.xml defines the dataflow applications (df-01, df-02, df-03 and dfo-01) and their module configurations except
    6. trigger-segment.data.xml defines the trigger applications (tc-maker-1, mlt and hsi-to-tc-app) and its module configurations
    7. data-store-params.data.xml defines the output file configuration for the dataflow apps
    8. fsm.data.xml defines the state machine and the supported transition commands
    9. connections.data.xml defines the Network and Queue connection rules used to generate appropriate endpoints in the SmartDaqApplications
    10. moduleconfs.data.xml contains DAQ Module configuration objects for readout, dataflow, and trigger
  • These files are placed in the install/appmodel/share/test/config directory upon build (via the daq_install() CMake command, based on their location in appmodel/test/config), where OKS can find them
  • To create a new configuration, we are currently limited to copying files from an existing configuration, but tools are in development to generate the readout map from an existing one.

Editing database files

  • The data and schema XML files can be edited manually or by running the graphical dbe editors dbe_main and schemaeditor. To enable use of the dbe editors, you must first run the command spack load dbe then dbe_main -f /path/to/file for editing data files or schemaeditor -f /path/to/file for editing schema files.

Running with CRP4 as example with real hardware

  1. Log into the EHN1 DAQ cluster.
  2. setup a work area with a fresh nightly (NFD_DEV_240701_A9++) as above
  3. Clone the EHN1 config repository and source its setup_db_path.sh script
  4. Start drunc as above but boot sessions/crp4-session.data.xml crp4-oks-session
  5. You will be able to look at monitoring metrics and errors using grafana (v5 dashboards).

Useful DBT and Spack commands for software areas

  • dbt-release-info # prints out the release type and name, and the base release name (version)
  • dbt-pkg-info <dunedaq_package_name> # prints out the package version and commit hash used by the release
  • dbt-src-status # prints out the branch names of source repos under sourcecode, and marks those with local changes with "*"
  • spack find --loaded -N <external_package_name>, e.g. spack find --loaded -N boost # prints out the version of the specified external package that is in use in the current software area
  • spack info fddaq # prints out the packages that are included in the fddaq bundle for the current software area
  • spack info dunedaq # prints out the packages that are included in the dunedaq (common) bundle for the current software area

Also see here.

Monitoring the system

When running with nanorc, metrics reports appear in the info_*.json files that are produced (e.g. info_dataflow_<portno>.json). We can collate these, grouped by metric name, using python -m opmonlib.info_file_collator info_*.json (default output file is opmon_collated.json).

It is also possible to monitor the system using a graphic interface.

Steps to enable and view TRACE debug messages

Here are suggested steps for enabling and viewing debug messages in the TRACE memory buffer:

  • set up your software area, if needed (e.g. cd <work_dir>; source ./dbt-env.sh ; dbt-workarea-env)
  • export TRACE_FILE=$DBT_AREA_ROOT/log/${USER}_dunedaq.trace
    • this tells TRACE which file on disk to use for its memory buffer, and in this way, enables TRACE in your shell environment and in subsequent runs of the system with nanorc.
  • run the application using the nanorc commands described above
    • this populates the list of already-enabled TRACE levels so that you can view them in the next step
  • run tlvls
    • this command outputs a list of all the TRACE names that are currently known, and which levels are enabled for each name
    • TRACE names allow us to group related messages, and these names typically correspond to the name of the C++ source file
    • the bitmasks that are relevant for the TRACE memory buffer are the ones in the "maskM" column
  • enable levels with tonM -n <TRACE NAME> <level>
    • for example, tonM -n DataWriter DEBUG+5 (where "5" is the level that you see in the TLOG_DEBUG statement in the C++ code)
  • re-run tlvls to confirm that the expected level is now set
  • re-run the application
  • view the TRACE messages using tshow | tdelta -ct 1 | more
    • note that the messages are displayed in reverse time order

A couple of additional notes:

  • For debug statements in our code that look like TLOG_DEBUG(5) << "test, test";, we would enable the output of those messages using a shell command like tonM -n <TRACE_NAME> DEBUG+5. A couple of notes on this...
    • when we look at the output of the bitmasks with the tlvls command, bit #5 is going to be offset by the number of bits that TRACE and ERS reserve for ERROR, WARNING, INFO, etc. messages. At the moment, the offset appears to be 8, so the setting of bit "DEBUG+5" corresponds to setting bit #13.
    • when we view the messages with tshow, one of the columns in its output shows the level associated with the message (the column heading is abbreviated as "lvl"). Debug messages are prefaced with the letter "D", and they include the number that was specified in the C++ code. So, for our example of level 5, we would see "D05" in the tshow output for the "test, test" messages.
  • There are many other TRACE 'commands' that allow you to enable and disable messages. For example,
    • tonMg <level> enables the specified level for all TRACE names (the "g" means global in this context)
    • toffM -n <TRACE NAME> <level> disables the specified level for the specified TRACE name
    • toffMg <level> disables the specified level for all TRACE names
    • tlvlM -n <TRACE name> <mask> explicitly sets (and un-sets) the levels specified in the bitmask
Clone this wiki locally