[Bug]: `file_put_contents(): Write of 198 bytes failed with errno=28 No space left on device`

[matthewnessworthy]

Bug report

Since 1.8.x, after accumulating a number of requests, Docker PHP based Lambdas are failing with No space left on device errors.

Dockerfile PHP setup

FROM laravelphp/vapor:php84-arm

ARG DD_VERSION=1.8.3

RUN curl -Lo /tmp/datadog-setup.php https://github.com/DataDog/dd-trace-php/releases/download/${DD_VERSION}/datadog-setup.php \
    && php /tmp/datadog-setup.php --php-bin=all \
    && rm /tmp/datadog-setup.php

Datadog ENV config

DD_AGENT_HOST=datadog-agent.production-application.local
DD_TAGS=source:php
DD_AUTOFINISH_SPANS=true
DD_SERVICE=laravel
DD_ENV=prod
DD_APM_ENABLED=true
DD_TRACE_HEADER_TAGS=x-mobile-app-version,mobile-app-version
DD_TRACE_HTTP_CLIENT_SPLIT_BY_DOMAIN=true
DD_TRACE_HTTP_POST_DATA_PARAM_ALLOWED=*
DD_TRACE_CLIENT_IP_ENABLED=true
DD_LOGS_INJECTION=true
DD_PROFILING_ENABLED=false
DD_EXCEPTION_DEBUGGING_ENABLED=true
DD_EXCEPTION_REPLAY_ENABLED=true

The agent is running on an internal AWS Fargate cluster with internal networking setup (hence DD_AGENT_HOST=datadog-agent.production-application.local)

PHP version

8.4.6

Tracer or profiler version

1.8.3

Installed extensions

No response

Output of phpinfo()

No response

Upgrading from

dd-trace 1.7.2

[bwoebi]

Hey @matthewnessworthy,

With dd-trace-php 1.8.0 on lambdas we start writing to /tmp/libdatadog instead of /dev/shm, which was failing.

Now my question would be: do you see many / large files in /tmp/libdatadog? If yes, can you tell which / their contents?

[matthewnessworthy]

@bwoebi i could try and take a look
is there some way to disable this?

[bwoebi]

I don't think that it's possible to disable it fully.

[matthewnessworthy]

@bwoebi scandir(/tmp/libdatadog): Failed to open directory: No such file or directory

[matthewnessworthy]

@bwoebi i tried version 1.10.0 and the same error is occurring, this is directly blocking us from upgrading the datadog agent beyond 1.7.2, which was working fine to send APM traces and logs to an external endpoint (in this case an ECS fargate task running the DD agent)

[matthewnessworthy]

@bwoebi following up on this, it's still a blocker for us upgrading past 1.7.x

[matthewnessworthy]

@bwoebi sorry to keep bringing this up, but it's still a blocker for us and i don't know how to resolve it

[github-actions]

This issue has been marked as stale because it has not had activity in the past half year. It will be closed in 7 days if no further activity occurs. Feel free to reopen it if you are still working on it.

[github-actions]

This issue has been closed because it has not had activity over the past half year. Feel free to reopen it if you are still working on it.