Running Telegraf in a jail and collecting host metrics - stuck with Telegraf simply "Loading Config"

[b-neufeld]

Here's a head-scratcher.

I'm allllmost done rebuilding my TrueNAS-hosted Ubuntu/Docker VM with jails. Everything is working, even finnicky Frigate.... except for seemingly-simple Telegraf!

What I want to do seems fairly simple: Use Telegraf in a Docker container with a mount or two to read host metrics (see here under "Monitoring the Docker Engine Host") and pump them back to my influxdb.

Please bear with me... this might seem like a Telegraf issue but I had it working on the Ubuntu VM and the TrueNAS SCALE host in tandem before this... so I think it's something to do with Jailmaker and/or mount configuration.

Here's the first part of my jail config:

startup=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind-ro='/:/hostfs'                      # Pass through a read-only copy of the host system
        --bind='/mnt/vesta/apps/telegraf/ceres:/hostconfig'        # where Telegraf logs go 
        --bind='/mnt/vesta/apps/docker/data-telegraf:/app/data'     # for Dockge
        --bind='/mnt/vesta/apps/docker/stacks-telegraf:/opt/stacks'     # for Dockge
...

The one thing I wonder about is binding '/' inside the jail.

Docker compose for Telegraf:

version: "3.8"
services:
  telegraf:
    image: telegraf:latest 
    container_name: telegraf_ceres
    #user: root:root
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /hostconfig/telegraf.conf.condensed:/etc/telegraf/telegraf.conf    # mount the config file - this works.
      - /hostconfig:/hostconfig         # passing this through from the jail config for log file storage
      - /hostfs:/hostfs                 # passing through from jail config for 
    environment:
      - HOST_ETC=/hostfs/etc
      - HOST_PROC=/hostfs/proc
      - HOST_SYS=/hostfs/sys
      - HOST_VAR=/hostfs/var
      - HOST_RUN=/hostfs/run
      - HOST_MOUNT_PREFIX=/hostfs
    ports:
      - 8125:8125
    restart: unless-stopped
networks: {}

Here's a stripped-down Telegraf config I've been testing with (trying to eliminate something dumb like a typo as a source of error):

[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  debug = true
  quiet = false
  logtarget = "file"
  logfile = "/hostconfig/telegraf.log"
  logfile_rotation_interval = "24h"
  logfile_rotation_max_size = "2MB"
  logfile_rotation_max_archives = 10
  hostname = "ceres"
  omit_hostname = false

[[outputs.influxdb_v2]]
  urls = ["http://myhost:8086"]
  token = "mytoken~~~~~~~~~~~~~~~~~
  organization = "myorg"
  bucket = "mybucket"

[[inputs.mem]]

No mater what I do, I see messages like 2024-05-11T04:41:29Z I! Loading config: /etc/telegraf/telegraf.conf in the container logs and nothing after that, despite enabling debug mode and disabling quiet mode on Telegraf.

It seems like it's stuck loading the config, but I've ruled out file access issues (Telegraf can see the right config file). I thought Telegraf might not have permissions to write to the logfile, but I've tried changing the config to send errors to stderr.

Any ideas? Thanks!

[Jip-Hop]

Maybe try the simpler setup first as described in "Monitoring docker containers" on the Docker Hub page. Then you can see if it will load the config file and you're not yet involving host storage. Build it out from there...

I don't see anything strange in your config files at first glance though.

[b-neufeld]

The suggestion to try a simpler setup was a good one.

Zeroing in on the issue. The one thing that seems to freeze the startup of the container on loading config is my [[outputs.influxdb_v2]] configuration. Which is strange because this is exactly copy & pasted from my pre-jail Telegraf config file.

[[outputs.influxdb_v2]]
   urls = ["http://172.16.1.97:8086/"]
   token = "mytoken~~~~~~~~~~~~~~~~"
   organization = "myorg"
   bucket = "mybucket"

If I comment out the token line, things actually happen:

05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loading config: /etc/telegraf/telegraf.conf
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Starting Telegraf 1.30.2 brought to you by InfluxData the makers of InfluxDB
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Available plugins: 233 inputs, 9 aggregators, 31 processors, 24 parsers, 60 outputs, 6 secret-stores
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loaded inputs: mem
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loaded aggregators: 
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loaded processors: 
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loaded secretstores: 
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Loaded outputs: influxdb_v2
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! Tags enabled: host=ceres
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ceres", Flush Interval:10s
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z D! [agent] Initializing plugins
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z D! [agent] Connecting outputs
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z D! [agent] Attempting connection to [outputs.influxdb_v2]
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z D! [agent] Successfully connected to outputs.influxdb_v2
05/11/2024
10:50:29 PM
2024-05-12T04:50:29Z D! [agent] Starting service inputs
05/11/2024
10:50:39 PM
2024-05-12T04:50:39Z E! [outputs.influxdb_v2] When writing to [http://172.16.1.97:8086]: failed to write metric to truenas (401 Unauthorized): unauthorized: unauthorized access

... but with the token line included, it hangs on the first "Loading config".

I'm not certain this is a Jailmaker issue... but it's strange the config works elsewhere, but not in the jail. I will keep digging later.

[b-neufeld]

No big update but I have done more testing to zero in on this weird "token" parameter behaviour on the influxdb_v2 output plugin, and I've cross-posted this issue to the telegraf community forums to see if I can get some application-specific ideas.

[b-neufeld]

Found it! Got a great reply to my post on the telegraf forums above. --capability=CAP_IPC_LOCK is needed as a systemd_nspawn_user_args argument because telegraf requires lockable memory.

Still fiddling with the telegraf config but I think this was the big barrier.