an error occur when vector cant query geo info from geo_city_lite.mmdb

[saltfishh]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

• version
  • vector 0.38

hi, I push data from kafka by vector, and I used transformer to conver ip info, and write it to file.
here is all my vector config:

most times, it works well, but few times, it will reflect on error:

2024-05-10T05:56:56.496641Z ERROR transform{component_kind="transform" component_id=geo_ip_transfer component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "get_enrichment_table_record" at (36:123): IP not found" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
and then i will get an wrong result.

Can I detect when the transformer is doing the conversion, and if an error is encountered, give .geo_ip a default value? just like:

  .geo_ip,err = get_enrichment_table_record!("geoip_table",{"ip": .dissect.remote_ip})
  if (err !=null){
    .geo_ip={"city_name":"unknon"}
  }

In this way, I can get the correct results.

Thanks!

Configuration

sources:
  dev_kafka:
    type: kafka
    bootstrap_servers: 10.11.1.170:9092
    topics:
      - web-logs
    group_id: web-logs
transforms:
  geo_ip_transfer:
    type: remap
    inputs:
      - dev_kafka
    source: |
      . = parse_json!(.message)
      .geo_ip = get_enrichment_table_record!("geoip_table",{"ip": .dissect.remote_ip})
      del(.@metadata)
sinks:
  to_file:
    inputs: ["geo_ip_transfer"]   
    type: file
    encoding:
      codec: json
    path: ./business-%Y-%m-%d.log
enrichment_tables:
  geoip_table:
    path: "./GeoLite2-City.cdn.mmdb"
    type: geoip
    locale: zh-CN

### Version

0.38

### Debug Output

```text
2024-05-10T06:59:02.137959Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::topology::builder: Sink starting.
2024-05-10T06:59:02.139301Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::utilization: utilization=0.6459871647614359
2024-05-10T06:59:02.152279Z DEBUG vector::sources::kafka: Consuming partition web-logs:0.
2024-05-10T06:59:07.139394Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::utilization: utilization=0.06459877335472873
2024-05-10T06:59:10.187682Z DEBUG transform{component_kind="transform" component_id=geo_ip_transfer component_type=remap}: vector::utilization: utilization=0.00042708953430647956
2024-05-10T06:59:10.188359Z ERROR transform{component_kind="transform" component_id=geo_ip_transfer component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"get_enrichment_table_record\" at (36:106): IP not found" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true
2024-05-10T06:59:12.139095Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::utilization: utilization=0.006754124998050746
2024-05-10T06:59:17.139247Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::utilization: utilization=0.0006754655981937604
2024-05-10T06:59:22.139401Z DEBUG sink{component_kind="sink" component_id=to_file component_type=file}: vector::utilization: utilization=0.00006760145813916612

Example Data

• raw data from kafka

{
  "@timestamp": "2024-05-10T06:11:45.833Z",
  "message": {
    "dissect": {
      "UA": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
      "access_date": "2024-04-09",
      "access_time": "18:00:27",
      "http_code": "302",
      "http_host": "abc.com",
      "referer": "-",
      "remote_ip": "103.190.179.11",
      "request_bytes": "113",
      "request_method": "GET",
      "resp_bytes": "267",
      "resp_cost": "0",
      "uri": "/index.do"
    }
  }
}

Additional Context

No response

References

No response

[jszwedko]

Hi @saltfishh ,

You can check for errors like:

  .geo_ip,err = get_enrichment_table_record("geoip_table",{"ip": .dissect.remote_ip})
  if (err !=null){
    .geo_ip={"city_name":"unknon"}
  }

That is: remove the ! from the function call. The ! tells VRL to abort if the function fails; otherwise it'll return an err value. This is documented here: https://vector.dev/docs/reference/vrl/expressions/#function-call

[saltfishh]

Hi @saltfishh ,

You can check for errors like:

  .geo_ip,err = get_enrichment_table_record("geoip_table",{"ip": .dissect.remote_ip})
  if (err !=null){
    .geo_ip={"city_name":"unknon"}
  }

That is: remove the ! from the function call. The ! tells VRL to abort if the function fails; otherwise it'll return an err value. This is documented here: https://vector.dev/docs/reference/vrl/expressions/#function-call

it works, thanks for that!