Converting protobuf bytes to JSON

[hanshuebner]

Hello,

I'm trying to convert some protobuf messages to JSON for debugging purposes, and I noticed that the resulting JSON contains the literal bytes of bytes fields in protobuf messages. Is there a way to have protobuf bytes fields be encoded as base64 strings instead?

My vector.toml file:

[sources.http]
type = "http_server"
address = "0.0.0.0:4318"
strict_path = false
decoding.codec = "protobuf"
decoding.protobuf.desc_file = "trace.desc"
decoding.protobuf.message_type = "opentelemetry.proto.trace.v1.TracesData"

[sinks.console]
type = "console"
inputs = [ "*" ]
encoding.codec = "json"

JSON output sample (elided and reformatted):

{
  "path": "/v1/traces",
  "resource_spans": [
    {
      "schema_url": "",
      "scope_spans": [
          "spans": [
            {
              "attributes": [],
              "dropped_attributes_count": 0,
              "dropped_events_count": 0,
              "dropped_links_count": 0,
              "end_time_unix_nano": 1703091504825214464,
              "events": [],
              "flags": 0,
              "kind": "SPAN_KIND_INTERNAL",
              "links": [],
              "name": "kong.header_filter.plugin.opentelemetry",
              "parent_span_id": "J@���͡L",
              "span_id": "�b�4\u000f��\n",
              "start_time_unix_nano": 1703091504825208064,
              "status": {
                "code": "STATUS_CODE_UNSET",
                "message": ""
              },
              "trace_id": "۸��پ_�yğ�Hl��",
              "trace_state": ""
            }
          ]
        }
      ]
    }
  ],
  "source_type": "http_server",
  "timestamp": "2023-12-20T16:58:24.834998494Z"
}

Notice the parent_span_id, span_id and trace_id containing the unencoded byte data.

Thank you!
Hans

[hanshuebner]

I came up with this VRL transform myself:

[transforms.convert_ids]
type = "remap"
inputs = [ "http" ]
source = '''
. = map_values(., true) -> |value| {
                      if is_object(value) {
                         if exists(value.trace_id) { value.trace_id, err = encode_base16(value.trace_id)}
                         if exists(value.parent_span_id) { value.parent_span_id, err = encode_base16(value.parent_span_id) }
                         if exists(value.span_id) { value.span_id, err = encode_base16(value.span_id) }
                      }
                      value
}

It seems that the bytes from the protobuf input data are just carried into JSON strings with no conversion applied. I was not sure whether VRL strings basically are byte arrays in that they can contain anything. Apparently, that is the case.