Filter transform getting rate limited interally

[danthegoodman1]

I've got the following values.yml file:

transforms:
  logFilter:
    type: filter
    inputs:
      - "kubernetes_logs"
    condition: "length(parse_regex!(.kubernetes.pod_name, r'.*deployment.*')) > 0"
  # jsonFlatten:
  #   type: remap
  #   inputs:
  #     - "logFilter"
  #   source: " .message = parse_json!(.message)\n"

sinks:
  # Adjust as necessary. By default we use the console sink
  # to print all data. This allows you to see Vector working.
  # /docs/reference/sinks/
  stdout:
    type: console
    inputs: ["logFilter"]
    target: "stdout"
    encoding: "json"

  lokiSink:
    type: loki
    inputs:
      - logFilter
    endpoint: http://loki.default.svc.cluster.local:3100
    encoding:
      codec: json
    healthcheck:
      enabled: true
    labels:
      forwarder: vector
      podname: "{{kubernetes.pod_name}}"
      nodename: "{{kubernetes.pod_node_name}}"
      messagetext: "{{message.message}}"

And I am getting errors:

Aug 09 16:54:18.635  WARN transform{component_kind="transform" component_name=logFilter component_type=filter}: vector::internal_events::conditions: VRL condition execution failed. internal_log_rate_secs=120
Aug 09 16:54:18.635  WARN transform{component_kind="transform" component_name=logFilter component_type=filter}: vector::internal_events::conditions: Internal log [VRL condition execution failed.] is being rate limited.

How do I remove this rate limit?

[jszwedko]

Hi @danthegoodman1 !

That rate limit you see there is specific to the log message from Vector. We typically rate limit error messages to avoid flooding people's logs in the event of catastrophic failure. Vector is not actually rate limiting events flowing through your topology. I opened #8635 to try to clarify this.

[danthegoodman1]

Will try the contains when I get an opportunity, thanks for the tip!

[jszwedko]

@danthegoodman1 I might suggest putting a remap transform in front of your filter transform and logging events missing that key.

Something like:

if !exists(.kubernetes.pod_name) {
  log(.)
}

That may help shine some light on why those events are missing that key.

I could also be wrong and the filter transform was failing for a different reason, but I can't think of one.

[danthegoodman1]

@jszwedko Will do! And looks like you might have pinged the wrong person by accident XD

[danthegoodman1]

@jszwedko Just got the following:

Aug 12 17:56:36.481 ERROR vector::topology: Configuration error. error=Transform "logFilter":                                                                                                                              │
│ error[E100]: unhandled error                                                                                                                                                                                               │
│   ┌─ :1:1                                                                                                                                                                                                                  │
│   │                                                                                                                                                                                                                        │
│ 1 │ contains(.kubernetes.pod_name, "deployment.")                                                                                                                                                                          │
│   │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                          │
│   │ │                                                                                                                                                                                                                      │
│   │ expression can result in runtime error                                                                                                                                                                                 │
│   │ handle the error case to ensure runtime success                                                                                                                                                                        │
│   │                                                                                                                                                                                                                        │
│   = see documentation about error handling at https://errors.vrl.dev/#handling                                                                                                                                             │
│   = learn more about error code 100 at https://errors.vrl.dev/100                                                                                                                                                          │
│   = see language documentation at https://vrl.dev

This is how I was doing it:

transforms:
  remapLogWeirdness:
    type: remap
    inputs:
      - "kubernetes_logs"
    source: |-
      if !exists(.kubernetes.pod_name) {
        log(.)
      }
  logFilter:
    type: filter
    inputs:
      - "kubernetes_logs"
    condition: contains(.kubernetes.pod_name, "deployment.")

[jszwedko]

Ah, sorry, I didn't actually test out my suggestion before I gave it. It should be contains!(.kubernetes.pod_name, "deployment")