How do I add fixed attributes to the expanded array?

[PanHywel]

How do I add fixed attributes to the expanded array

Vector Version

vector 0.15.0

[sources.httpSources]
type = "http"
address = "172.16.1.41:7001"
path = "/client-lua"
encoding = "json"
headers = ["x-forwarded-for"]


[transforms.parse_logs]
type = "remap"
inputs = ["httpSources"]
source = '''
. = .msg
.clientIp = .x-forwarded-for
'''


[sinks.print]
 type = "console"
 inputs = ["parse_logs"]
 encoding.codec = "json"

{
   "x-forwarded-for":"192.168.1.1",
   "msg":[
     {
       "name":"tom"
     },
     {
       "name":"mimi"
     }
  ]
}

I hope to get：

{
  "clientIp":"192.168.1.1",
  "name":"tom"
},
{
  "clientIp":"192.168.1.1",
  "name":"mimi"
}

[spencergilbert]

source = '''
. = .msg
.clientIp = .x-forwarded-for
'''

The first line here will replace the entire event (.) with the the .msg field, if present. The following line should set a root level field named clientIp to the contents of the .x-forwarded-for field (presumably a part of the .msg key you used above?).

Could you share an example of an event that would be going into the remap transform here?

EDIT: if that second snippet is the example input, I think you may just want to use unnest on .msg

source = """
. = unnest!(.msg)
.clientIp = del(.x-forwarded-for)
"""

EDIT2: 💭 might require two remaps:

[transforms.parse_logs]
type = "remap"
inputs = ["httpSources"]
source = '''
. = unnest(.msg)
'''

[transforms.parse_logs]
type = "remap"
inputs = ["parse_logs"]
source = '''
. |= .msg
.clientIp = ."x-forwarded-for"
'''

[spencergilbert]

I'll have a look in a little bit here, and see what I can come up with locally - I'm pretty certain we can make it simpler 🙂

[spencergilbert]

I think you got pretty well close actually, this is a little simpler.

[transforms.explode]
    type = "remap"
    inputs = ["parse"]
    source = """
      . = unnest!(.msg)
    """
[transforms.remap]
    type = "remap"
    inputs = ["explode"]
    source = """
      ua = del(."User-Agent")
      xff = del(."x-forwarded-for")
      . = .msg
      ."User-Agent" = ua
      ."x-forwarded-for" = xff
    """

[PanHywel]

@spencergilbert Thanks, I have another new question. When I use http as input source and distribute it through Nginx in front, I find that there are a lot of error logs in Nginx. I don't know if this is limited by the performance of the http module. But I didn't find any relevant information about the configuration in the configuration documentation.

The flow is as follows.
Nginx upstream => vector http => transforms => kafka

Nginx error message.

connect() failed (111: Connection refused) while connecting to upstream

To add：My Nginx requests are very heavy, and not all of them are rejected, but most of them are

[spencergilbert]

🤔 Would you mind opening another discussion with the configuration for vector and what's being sent to the HTTP source?

[PanHywel]

Okay, I'll start a new discussion #8700