[exporter/datadog] invalid api token causes failure when using logs agent exporter

[csquire]

Component(s)

exporter/datadog

What happened?

Description

When the feature gate enabled for the Datadog logs agent exporter, an invalid Datadog API key will cause the collector to exit.

Requires this feature gate to be enabled:
#32327

This is the summary of what I believe is happening:

1. The collector defaults the logging endpoint address to https://http-intake.logs.datadoghq.com/
2. The Datadog Agent code used by the collector is performing a check to see if HTTP connectivity works before deciding on HTTP or TCP
3. If HTTP works, TCP is not attempted
4. If HTTP fails (including if the api key is invalid), the Datadog Agent code attempts TCP
   This is where the error occurs. When the API key is invalid, the HTTP connection validation fails and TCP is attempted. Parsing the default endpoint of https://http-intake.logs.datadoghq.com/ is attempted, but the code expects a <host>:<port> format and errors due to port not being a number.

Steps to Reproduce

Run collector with feature gate enabled:

RUN_ARGS=--feature-gates=exporter.datadogexporter.UseLogsAgentExporter make run

Expected Result

The collector continues to run, even if log forwarding is not possible

Actual Result

The collector crashes with the included log error

Collector version

v0.100.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    start_at: beginning
    include:
      - ../../local/app.log

exporters:
  datadog:
    api:
      key: xyz
      
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [datadog]

Log output

Error: failed to build pipelines: failed to create "datadog" exporter for data type "logs": failed to create logs agent: Invalid endpoints: could not parse https://http-intake.logs.datadoghq.com: strconv.Atoi: parsing "//http-intake.logs.datadoghq.com": invalid syntax
2024/05/22 14:33:20 collector server run finished with error: failed to build pipelines: failed to create "datadog" exporter for data type "logs": failed to create logs agent: Invalid endpoints: could not parse https://http-intake.logs.datadoghq.com: strconv.Atoi: parsing "//http-intake.logs.datadoghq.com": invalid syntax

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.