Skip to content

Telemetryapi receiver slow startup time (~3.5-4 seconds) #2100

@jousby

Description

@jousby

Describe the bug
With a hello world nodejs lambda function with 512Mb of memory Im seeing the telemetry api receiver take over 3.5 seconds to initialize during the function cold start init phase.

Steps to reproduce

  • Create a hello world nodejs function
  • Add the opentelemtry-lambda collector and nodejs layers
  • Add a custom colllectory.yaml to the function bundling step that enables the telemetryapi receiver ie
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 'localhost:4317'
      http:
        endpoint: 'localhost:4318'
  # This receiver is used to capture the logs sent to stdout by the Lambda function
  # This is how we capture logs generated by the Powertools Logger.
  telemetryapi:

Also add the receiver to a valid logs pipeline to capture the telemetry from the receiver

  • Deploy the function behind and api gateway
  • Make some calls to the endpoint

What did you expect to see?
Functionally everything is working, the telemetry api receiver is a big win. I was expecting this to startup in less than a second though.

What did you see instead?
The init phase for the telemetry receiver is consistently over three seconds:

Image

What version of collector/language SDK version did you use?

  • 0.19.0 (collector layer)
  • 0.19.0 (nodejs layer)

What language layer did you use?
nodejs

Additional context
With the telemetryapi receiver turned on Im seeing overall coldstart init times of ~4+ seconds. With an instrumented custom authorizer function on the gateway I'm seeing a worst case scenario for a hello world function of ~10.5 second round trip time for a cold start (4.5 seconds for the auth function to spin up and then 4.5 seconds for the helloworld function to spin up).

My serverless mircoservice architecture can tolerate the occasional blow out in a cold start time but when cold starts are consistently this long it means that hitting my stack when cold causes service to service interactions to quickly hit the function timeouts (20-30 seconds) and exceeds user tolerable page load times.

I like/need the telemetryapi receiver functionality. I would like the function cold start init phase telemetry it provides shipped to my non cloudwatch observability stack. My code is currently using a logging framework that emits to stdout and the telemtryapi receiver allows me to capture this and ship it to somewhere other than cloudwatch.

For my usecase I can't use the telemetryapi receiver with the current start times without resorting to provisioned concurrency/managed instances which are imperfect solutions.

Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions