PoC use of AWS Lambda Telemetry API

[simitt]

Research how the recently announced AWS Lambda Telemetry API can be used for collecting APM data from the lambda service.
A PoC should be built for comparing the current lambda extension to the possibilities via the new API. Focus should be on how the APM data compare and usability.

[estolfo]

I've completed a POC of using the Telemetry API in the aws lambda extension, and these are the findings:

• The Telemetry API is a superset of the Logs API and is recommended in place of it
  • Implementation of subscription to Telemetry API is here
  • The Logs API will probably be deprecated
• The functionality is equivalent to the Logs API
  • Send subscription request, then listen on a certain host:port for data
• Differences include the amount and type of data being sent
  • Additional Event types and Object types
    • Note Span objects and Tracing data
• The Telemetry API documentation gives guidance on how to translate data into OTel spans here
• The spanId reported in Telemetry API documents is a kind of transaction ID for the overall request to the function
• The APM agent will create/propagate its trace id to downstream services but the Lambda runtime won't parse the agent's trace id, if the downstream service is a Lambda function.
• The tracing info is available via the Telemetry API if "Active tracing" is turned to ON for a function
• Only two spans are reported right now as "tracing" data
  • The responseDuration span reported in the Telemetry API is visualized in their traces explorer as the "intialization" segment (i.e. span)
  • The responseLatency span reported in the Telemetry API is visualzed in their traces explorer as the 'invocation" segment (i.e. span)

In conclusion, we should use the Telemetry API going forward. We can also use the two spans reported to more accurately timestamp the transaction for the lambda function invocation and to show the two "initialization" and "invocation" phases of the lambda function as spans in our own UI.

Open questions:

• Is there an extra span when SnapStart is used?
• Are all metrics we use from the Logs API available via the Telemetry API?