fix(llmobs): openai-java payload mapping for responses, tool metadata, and prompt tracking#10644
Open
fix(llmobs): openai-java payload mapping for responses, tool metadata, and prompt tracking#10644
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.07 s) : 0, 1070094
Total [baseline] (11.13 s) : 0, 11129669
Agent [candidate] (1.059 s) : 0, 1059294
Total [candidate] (11.09 s) : 0, 11089855
section appsec
Agent [baseline] (1.25 s) : 0, 1250383
Total [baseline] (11.128 s) : 0, 11127977
Agent [candidate] (1.247 s) : 0, 1247120
Total [candidate] (11.082 s) : 0, 11082107
section iast
Agent [baseline] (1.23 s) : 0, 1229848
Total [baseline] (11.345 s) : 0, 11344740
Agent [candidate] (1.227 s) : 0, 1226710
Total [candidate] (11.47 s) : 0, 11470333
section profiling
Agent [baseline] (1.191 s) : 0, 1190616
Total [baseline] (11.035 s) : 0, 11034797
Agent [candidate] (1.213 s) : 0, 1212884
Total [candidate] (11.201 s) : 0, 11200529
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.234 ms) : 0, 1234
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (633.528 ms) : 0, 633528
BytebuddyAgent [candidate] (628.248 ms) : 0, 628248
AgentMeter [baseline] (29.312 ms) : 0, 29312
AgentMeter [candidate] (29.105 ms) : 0, 29105
GlobalTracer [baseline] (258.554 ms) : 0, 258554
GlobalTracer [candidate] (256.843 ms) : 0, 256843
AppSec [baseline] (31.764 ms) : 0, 31764
AppSec [candidate] (31.438 ms) : 0, 31438
Debugger [baseline] (59.857 ms) : 0, 59857
Debugger [candidate] (59.265 ms) : 0, 59265
Remote Config [baseline] (603.679 µs) : 0, 604
Remote Config [candidate] (582.774 µs) : 0, 583
Telemetry [baseline] (8.69 ms) : 0, 8690
Telemetry [candidate] (8.612 ms) : 0, 8612
Flare Poller [baseline] (10.35 ms) : 0, 10350
Flare Poller [candidate] (7.965 ms) : 0, 7965
section appsec
crashtracking [baseline] (1.193 ms) : 0, 1193
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (658.629 ms) : 0, 658629
BytebuddyAgent [candidate] (658.654 ms) : 0, 658654
AgentMeter [baseline] (12.057 ms) : 0, 12057
AgentMeter [candidate] (11.984 ms) : 0, 11984
GlobalTracer [baseline] (260.062 ms) : 0, 260062
GlobalTracer [candidate] (258.463 ms) : 0, 258463
IAST [baseline] (24.205 ms) : 0, 24205
IAST [candidate] (23.926 ms) : 0, 23926
AppSec [baseline] (178.486 ms) : 0, 178486
AppSec [candidate] (178.042 ms) : 0, 178042
Debugger [baseline] (66.119 ms) : 0, 66119
Debugger [candidate] (65.568 ms) : 0, 65568
Remote Config [baseline] (578.02 µs) : 0, 578
Remote Config [candidate] (566.936 µs) : 0, 567
Telemetry [baseline] (9.072 ms) : 0, 9072
Telemetry [candidate] (8.838 ms) : 0, 8838
Flare Poller [baseline] (3.628 ms) : 0, 3628
Flare Poller [candidate] (3.587 ms) : 0, 3587
section iast
crashtracking [baseline] (1.211 ms) : 0, 1211
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (797.988 ms) : 0, 797988
BytebuddyAgent [candidate] (795.727 ms) : 0, 795727
AgentMeter [baseline] (11.341 ms) : 0, 11341
AgentMeter [candidate] (11.36 ms) : 0, 11360
GlobalTracer [baseline] (247.835 ms) : 0, 247835
GlobalTracer [candidate] (247.262 ms) : 0, 247262
IAST [baseline] (25.152 ms) : 0, 25152
IAST [candidate] (25.18 ms) : 0, 25180
AppSec [baseline] (27.271 ms) : 0, 27271
AppSec [candidate] (26.421 ms) : 0, 26421
Debugger [baseline] (62.564 ms) : 0, 62564
Debugger [candidate] (63.291 ms) : 0, 63291
Remote Config [baseline] (532.802 µs) : 0, 533
Remote Config [candidate] (528.949 µs) : 0, 529
Telemetry [baseline] (14.988 ms) : 0, 14988
Telemetry [candidate] (14.817 ms) : 0, 14817
Flare Poller [baseline] (4.87 ms) : 0, 4870
Flare Poller [candidate] (4.886 ms) : 0, 4886
section profiling
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.203 ms) : 0, 1203
BytebuddyAgent [baseline] (688.129 ms) : 0, 688129
BytebuddyAgent [candidate] (701.396 ms) : 0, 701396
AgentMeter [baseline] (8.639 ms) : 0, 8639
AgentMeter [candidate] (8.81 ms) : 0, 8810
GlobalTracer [baseline] (216.584 ms) : 0, 216584
GlobalTracer [candidate] (220.458 ms) : 0, 220458
AppSec [baseline] (32.186 ms) : 0, 32186
AppSec [candidate] (33.141 ms) : 0, 33141
Debugger [baseline] (64.153 ms) : 0, 64153
Debugger [candidate] (64.358 ms) : 0, 64358
Remote Config [baseline] (588.212 µs) : 0, 588
Remote Config [candidate] (597.44 µs) : 0, 597
Telemetry [baseline] (9.887 ms) : 0, 9887
Telemetry [candidate] (10.767 ms) : 0, 10767
Flare Poller [baseline] (3.56 ms) : 0, 3560
Flare Poller [candidate] (3.636 ms) : 0, 3636
ProfilingAgent [baseline] (94.369 ms) : 0, 94369
ProfilingAgent [candidate] (96.093 ms) : 0, 96093
Profiling [baseline] (94.938 ms) : 0, 94938
Profiling [candidate] (96.669 ms) : 0, 96669
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064559
Total [baseline] (8.919 s) : 0, 8919250
Agent [candidate] (1.074 s) : 0, 1074408
Total [candidate] (8.847 s) : 0, 8846986
section iast
Agent [baseline] (1.237 s) : 0, 1236675
Total [baseline] (9.572 s) : 0, 9571802
Agent [candidate] (1.23 s) : 0, 1229624
Total [candidate] (9.562 s) : 0, 9561541
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.219 ms) : 0, 1219
BytebuddyAgent [baseline] (633.366 ms) : 0, 633366
BytebuddyAgent [candidate] (638.9 ms) : 0, 638900
AgentMeter [baseline] (29.466 ms) : 0, 29466
AgentMeter [candidate] (29.523 ms) : 0, 29523
GlobalTracer [baseline] (257.544 ms) : 0, 257544
GlobalTracer [candidate] (260.767 ms) : 0, 260767
AppSec [baseline] (31.591 ms) : 0, 31591
AppSec [candidate] (32.146 ms) : 0, 32146
Debugger [baseline] (58.629 ms) : 0, 58629
Debugger [candidate] (59.498 ms) : 0, 59498
Remote Config [baseline] (591.132 µs) : 0, 591
Remote Config [candidate] (603.755 µs) : 0, 604
Telemetry [baseline] (8.59 ms) : 0, 8590
Telemetry [candidate] (8.844 ms) : 0, 8844
Flare Poller [baseline] (7.223 ms) : 0, 7223
Flare Poller [candidate] (6.511 ms) : 0, 6511
section iast
crashtracking [baseline] (1.209 ms) : 0, 1209
crashtracking [candidate] (1.211 ms) : 0, 1211
BytebuddyAgent [baseline] (804.025 ms) : 0, 804025
BytebuddyAgent [candidate] (797.843 ms) : 0, 797843
AgentMeter [baseline] (11.572 ms) : 0, 11572
AgentMeter [candidate] (11.351 ms) : 0, 11351
GlobalTracer [baseline] (248.731 ms) : 0, 248731
GlobalTracer [candidate] (248.545 ms) : 0, 248545
IAST [baseline] (25.375 ms) : 0, 25375
IAST [candidate] (25.182 ms) : 0, 25182
AppSec [baseline] (26.597 ms) : 0, 26597
AppSec [candidate] (26.4 ms) : 0, 26400
Debugger [baseline] (62.67 ms) : 0, 62670
Debugger [candidate] (62.536 ms) : 0, 62536
Remote Config [baseline] (531.597 µs) : 0, 532
Remote Config [candidate] (524.816 µs) : 0, 525
Telemetry [baseline] (14.824 ms) : 0, 14824
Telemetry [candidate] (14.886 ms) : 0, 14886
Flare Poller [baseline] (4.853 ms) : 0, 4853
Flare Poller [candidate] (5.028 ms) : 0, 5028
LoadParameters
See matching parameters
SummaryFound 5 performance improvements and 3 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section baseline
no_agent (1.165 ms) : 1154, 1176
. : milestone, 1165,
iast (3.174 ms) : 3131, 3217
. : milestone, 3174,
iast_FULL (5.968 ms) : 5908, 6028
. : milestone, 5968,
iast_GLOBAL (3.665 ms) : 3608, 3722
. : milestone, 3665,
profiling (2.084 ms) : 2064, 2105
. : milestone, 2084,
tracing (1.821 ms) : 1804, 1837
. : milestone, 1821,
section candidate
no_agent (1.207 ms) : 1194, 1219
. : milestone, 1207,
iast (3.191 ms) : 3146, 3236
. : milestone, 3191,
iast_FULL (5.691 ms) : 5635, 5747
. : milestone, 5691,
iast_GLOBAL (3.717 ms) : 3645, 3790
. : milestone, 3717,
profiling (2.04 ms) : 2020, 2060
. : milestone, 2040,
tracing (1.762 ms) : 1748, 1777
. : milestone, 1762,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section baseline
no_agent (18.479 ms) : 18294, 18664
. : milestone, 18479,
appsec (18.554 ms) : 18368, 18741
. : milestone, 18554,
code_origins (18.899 ms) : 18706, 19092
. : milestone, 18899,
iast (18.011 ms) : 17833, 18190
. : milestone, 18011,
profiling (20.26 ms) : 20049, 20470
. : milestone, 20260,
tracing (17.672 ms) : 17494, 17850
. : milestone, 17672,
section candidate
no_agent (18.132 ms) : 17949, 18315
. : milestone, 18132,
appsec (19.782 ms) : 19578, 19987
. : milestone, 19782,
code_origins (17.706 ms) : 17526, 17885
. : milestone, 17706,
iast (18.686 ms) : 18499, 18873
. : milestone, 18686,
profiling (19.044 ms) : 18855, 19233
. : milestone, 19044,
tracing (17.722 ms) : 17547, 17897
. : milestone, 17722,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section baseline
no_agent (15.569 s) : 15569000, 15569000
. : milestone, 15569000,
appsec (14.612 s) : 14612000, 14612000
. : milestone, 14612000,
iast (17.88 s) : 17880000, 17880000
. : milestone, 17880000,
iast_GLOBAL (17.695 s) : 17695000, 17695000
. : milestone, 17695000,
profiling (14.983 s) : 14983000, 14983000
. : milestone, 14983000,
tracing (15.028 s) : 15028000, 15028000
. : milestone, 15028000,
section candidate
no_agent (15.244 s) : 15244000, 15244000
. : milestone, 15244000,
appsec (15.003 s) : 15003000, 15003000
. : milestone, 15003000,
iast (17.915 s) : 17915000, 17915000
. : milestone, 17915000,
iast_GLOBAL (17.946 s) : 17946000, 17946000
. : milestone, 17946000,
profiling (15.284 s) : 15284000, 15284000
. : milestone, 15284000,
tracing (15.065 s) : 15065000, 15065000
. : milestone, 15065000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~028d64f1f8, baseline=1.61.0-SNAPSHOT~4fd66d45a9
dateFormat X
axisFormat %s
section baseline
no_agent (1.478 ms) : 1467, 1490
. : milestone, 1478,
appsec (3.781 ms) : 3560, 4003
. : milestone, 3781,
iast (2.255 ms) : 2186, 2324
. : milestone, 2255,
iast_GLOBAL (2.305 ms) : 2235, 2374
. : milestone, 2305,
profiling (2.108 ms) : 2052, 2164
. : milestone, 2108,
tracing (2.061 ms) : 2007, 2114
. : milestone, 2061,
section candidate
no_agent (1.477 ms) : 1465, 1489
. : milestone, 1477,
appsec (3.797 ms) : 3576, 4017
. : milestone, 3797,
iast (2.261 ms) : 2192, 2330
. : milestone, 2261,
iast_GLOBAL (2.299 ms) : 2230, 2368
. : milestone, 2299,
profiling (2.093 ms) : 2037, 2149
. : milestone, 2093,
tracing (2.068 ms) : 2014, 2121
. : milestone, 2068,
|
ygree
force-pushed
the
ygree/llmobs-systest-fixes
branch
from
5cd257e to
cbd6226
Compare
ygree
changed the title
…wthTestOpenAiLlmInteractions::test_completion
…teractions::test_chat_completion_tool_call
…d with python openai instrumentation and system-tests
… with variables + chat_template, longest-first overlap handling) and support map-based LLM input serialization (messages + prompt) in LLMObs mapper. Also filter empty instruction messages to match system-test expectations.
…st and return [image] (not empty) when stripped input_image URLs are missing, aligning mixed-input chat_template output with expected behavior.
…output.messages from request params so existing error-span tests pass.
…ol_definitions tags
…JSON argument parsing and remove duplicate manual parsing logic from ResponseDecorator.
ygree
changed the title
ygree
added
tag: ai generated
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c879ba692
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
...enai-java-3.0/src/main/java/datadog/trace/instrumentation/openai_java/ResponseDecorator.java
Outdated
Show resolved
Hide resolved
...enai-java-3.0/src/main/java/datadog/trace/instrumentation/openai_java/ResponseDecorator.java
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Aligns OpenAI Java LLMObs span payloads with expected intake/system-test schema by:
_ml_obs_tag.integration_ml_obs_tag.source_ml_obs_tag.ddtrace.version_ml_obs_tag.error_ml_obs_tag.error_typemodel_name(and stable placeholder output where applicable) is set on error paths forchat/completions/embeddings/responses.
input.prompt,variables,chat_template)tool_definitions)stream,tool_choice,text.verbosity, etc.)JsonValueUtils._ddmap with span/trace idsmeta.errorinputserialization (messages+prompt)tool_definitionsintometa.Motivation
OpenAI/LLMObs system tests exposed schema and tag mismatches in Java payloads (especially response spans, tool metadata, error mapping, and prompt tracking structure). This change brings Java output in line with expected LLMObs intake contract and behavior.
Additional Notes
openai-java-3.0min version updated from3.0.0to3.0.1.DataDog/dd-apm-test-agent#280
DataDog/system-tests#6364
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.