-
Notifications
You must be signed in to change notification settings - Fork 324
Add support for content parts and image URLs in AI Guard #10449
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
smola
wants to merge
1
commit into
master
Choose a base branch
from
smola/ai-guard-content-parts
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+847
−31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1052760
Total [baseline] (8.702 s) : 0, 8701558
Agent [candidate] (1.056 s) : 0, 1055645
Total [candidate] (8.665 s) : 0, 8665125
section iast
Agent [baseline] (1.221 s) : 0, 1221349
Total [baseline] (9.334 s) : 0, 9334324
Agent [candidate] (1.221 s) : 0, 1220552
Total [candidate] (9.338 s) : 0, 9338012
gantt
title insecure-bank - break down per module: candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.17 ms) : 0, 1170
crashtracking [candidate] (1.173 ms) : 0, 1173
BytebuddyAgent [baseline] (621.337 ms) : 0, 621337
BytebuddyAgent [candidate] (621.529 ms) : 0, 621529
AgentMeter [baseline] (28.61 ms) : 0, 28610
AgentMeter [candidate] (28.637 ms) : 0, 28637
GlobalTracer [baseline] (256.451 ms) : 0, 256451
GlobalTracer [candidate] (256.634 ms) : 0, 256634
AppSec [baseline] (32.794 ms) : 0, 32794
AppSec [candidate] (32.598 ms) : 0, 32598
Debugger [baseline] (60.651 ms) : 0, 60651
Debugger [candidate] (60.106 ms) : 0, 60106
Remote Config [baseline] (616.215 µs) : 0, 616
Remote Config [candidate] (620.274 µs) : 0, 620
Telemetry [baseline] (9.776 ms) : 0, 9776
Telemetry [candidate] (9.06 ms) : 0, 9060
Flare Poller [baseline] (5.951 ms) : 0, 5951
Flare Poller [candidate] (9.952 ms) : 0, 9952
section iast
crashtracking [baseline] (1.176 ms) : 0, 1176
crashtracking [candidate] (1.183 ms) : 0, 1183
BytebuddyAgent [baseline] (787.433 ms) : 0, 787433
BytebuddyAgent [candidate] (787.614 ms) : 0, 787614
AgentMeter [baseline] (11.1 ms) : 0, 11100
AgentMeter [candidate] (11.119 ms) : 0, 11119
GlobalTracer [baseline] (247.229 ms) : 0, 247229
GlobalTracer [candidate] (247.151 ms) : 0, 247151
IAST [baseline] (26.825 ms) : 0, 26825
IAST [candidate] (26.775 ms) : 0, 26775
AppSec [baseline] (33.95 ms) : 0, 33950
AppSec [candidate] (33.77 ms) : 0, 33770
Debugger [baseline] (65.931 ms) : 0, 65931
Debugger [candidate] (65.419 ms) : 0, 65419
Remote Config [baseline] (548.437 µs) : 0, 548
Remote Config [candidate] (534.991 µs) : 0, 535
Telemetry [baseline] (8.552 ms) : 0, 8552
Telemetry [candidate] (8.422 ms) : 0, 8422
Flare Poller [baseline] (3.407 ms) : 0, 3407
Flare Poller [candidate] (3.408 ms) : 0, 3408
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1059028
Total [baseline] (11.014 s) : 0, 11014492
Agent [candidate] (1.061 s) : 0, 1060511
Total [candidate] (11.055 s) : 0, 11054684
section appsec
Agent [baseline] (1.228 s) : 0, 1228487
Total [baseline] (10.95 s) : 0, 10950175
Agent [candidate] (1.231 s) : 0, 1230943
Total [candidate] (10.915 s) : 0, 10914541
section iast
Agent [baseline] (1.237 s) : 0, 1236809
Total [baseline] (11.225 s) : 0, 11225100
Agent [candidate] (1.226 s) : 0, 1226127
Total [candidate] (11.141 s) : 0, 11141028
section profiling
Agent [baseline] (1.179 s) : 0, 1179362
Total [baseline] (10.862 s) : 0, 10861803
Agent [candidate] (1.186 s) : 0, 1185924
Total [candidate] (10.87 s) : 0, 10870244
gantt
title petclinic - break down per module: candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.177 ms) : 0, 1177
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (621.791 ms) : 0, 621791
BytebuddyAgent [candidate] (623.225 ms) : 0, 623225
AgentMeter [baseline] (28.719 ms) : 0, 28719
AgentMeter [candidate] (28.798 ms) : 0, 28798
GlobalTracer [baseline] (256.77 ms) : 0, 256770
GlobalTracer [candidate] (257.347 ms) : 0, 257347
AppSec [baseline] (32.924 ms) : 0, 32924
AppSec [candidate] (33.111 ms) : 0, 33111
Debugger [baseline] (61.567 ms) : 0, 61567
Debugger [candidate] (63.363 ms) : 0, 63363
Remote Config [baseline] (638.545 µs) : 0, 639
Remote Config [candidate] (611.705 µs) : 0, 612
Telemetry [baseline] (11.423 ms) : 0, 11423
Telemetry [candidate] (9.844 ms) : 0, 9844
Flare Poller [baseline] (8.536 ms) : 0, 8536
Flare Poller [candidate] (7.693 ms) : 0, 7693
section appsec
crashtracking [baseline] (1.171 ms) : 0, 1171
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (650.459 ms) : 0, 650459
BytebuddyAgent [candidate] (651.434 ms) : 0, 651434
AgentMeter [baseline] (11.803 ms) : 0, 11803
AgentMeter [candidate] (11.792 ms) : 0, 11792
GlobalTracer [baseline] (256.708 ms) : 0, 256708
GlobalTracer [candidate] (257.376 ms) : 0, 257376
IAST [baseline] (25.484 ms) : 0, 25484
IAST [candidate] (25.298 ms) : 0, 25298
AppSec [baseline] (167.134 ms) : 0, 167134
AppSec [candidate] (167.618 ms) : 0, 167618
Debugger [baseline] (67.137 ms) : 0, 67137
Debugger [candidate] (67.211 ms) : 0, 67211
Remote Config [baseline] (660.573 µs) : 0, 661
Remote Config [candidate] (671.357 µs) : 0, 671
Telemetry [baseline] (9.151 ms) : 0, 9151
Telemetry [candidate] (9.363 ms) : 0, 9363
Flare Poller [baseline] (3.625 ms) : 0, 3625
Flare Poller [candidate] (3.714 ms) : 0, 3714
section iast
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.177 ms) : 0, 1177
BytebuddyAgent [baseline] (797.713 ms) : 0, 797713
BytebuddyAgent [candidate] (789.803 ms) : 0, 789803
AgentMeter [baseline] (11.222 ms) : 0, 11222
AgentMeter [candidate] (11.116 ms) : 0, 11116
GlobalTracer [baseline] (250.271 ms) : 0, 250271
GlobalTracer [candidate] (247.72 ms) : 0, 247720
IAST [baseline] (27.046 ms) : 0, 27046
IAST [candidate] (27.009 ms) : 0, 27009
AppSec [baseline] (35.924 ms) : 0, 35924
AppSec [candidate] (34.546 ms) : 0, 34546
Debugger [baseline] (65.354 ms) : 0, 65354
Debugger [candidate] (66.828 ms) : 0, 66828
Remote Config [baseline] (550.806 µs) : 0, 551
Remote Config [candidate] (554.083 µs) : 0, 554
Telemetry [baseline] (8.607 ms) : 0, 8607
Telemetry [candidate] (8.601 ms) : 0, 8601
Flare Poller [baseline] (3.483 ms) : 0, 3483
Flare Poller [candidate] (3.48 ms) : 0, 3480
section profiling
crashtracking [baseline] (1.208 ms) : 0, 1208
crashtracking [candidate] (1.2 ms) : 0, 1200
BytebuddyAgent [baseline] (673.635 ms) : 0, 673635
BytebuddyAgent [candidate] (677.127 ms) : 0, 677127
AgentMeter [baseline] (8.603 ms) : 0, 8603
AgentMeter [candidate] (8.667 ms) : 0, 8667
GlobalTracer [baseline] (215.463 ms) : 0, 215463
GlobalTracer [candidate] (216.908 ms) : 0, 216908
AppSec [baseline] (32.214 ms) : 0, 32214
AppSec [candidate] (32.577 ms) : 0, 32577
Debugger [baseline] (67.128 ms) : 0, 67128
Debugger [candidate] (67.665 ms) : 0, 67665
Remote Config [baseline] (585.301 µs) : 0, 585
Remote Config [candidate] (587.636 µs) : 0, 588
Telemetry [baseline] (8.675 ms) : 0, 8675
Telemetry [candidate] (8.724 ms) : 0, 8724
Flare Poller [baseline] (3.685 ms) : 0, 3685
Flare Poller [candidate] (3.741 ms) : 0, 3741
ProfilingAgent [baseline] (98.597 ms) : 0, 98597
ProfilingAgent [candidate] (98.908 ms) : 0, 98908
Profiling [baseline] (99.161 ms) : 0, 99161
Profiling [candidate] (99.473 ms) : 0, 99473
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 20 metrics, 15 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section baseline
no_agent (17.068 ms) : 16897, 17238
. : milestone, 17068,
appsec (18.477 ms) : 18288, 18666
. : milestone, 18477,
code_origins (17.511 ms) : 17337, 17685
. : milestone, 17511,
iast (17.772 ms) : 17598, 17947
. : milestone, 17772,
profiling (18.402 ms) : 18218, 18586
. : milestone, 18402,
tracing (17.417 ms) : 17246, 17587
. : milestone, 17417,
section candidate
no_agent (16.947 ms) : 16782, 17112
. : milestone, 16947,
appsec (18.957 ms) : 18759, 19154
. : milestone, 18957,
code_origins (17.579 ms) : 17403, 17755
. : milestone, 17579,
iast (17.816 ms) : 17640, 17993
. : milestone, 17816,
profiling (19.149 ms) : 18960, 19337
. : milestone, 19149,
tracing (17.737 ms) : 17560, 17913
. : milestone, 17737,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section baseline
no_agent (1.192 ms) : 1180, 1204
. : milestone, 1192,
iast (3.144 ms) : 3105, 3184
. : milestone, 3144,
iast_FULL (5.596 ms) : 5540, 5652
. : milestone, 5596,
iast_GLOBAL (3.623 ms) : 3566, 3680
. : milestone, 3623,
profiling (1.923 ms) : 1907, 1939
. : milestone, 1923,
tracing (1.838 ms) : 1823, 1853
. : milestone, 1838,
section candidate
no_agent (1.188 ms) : 1176, 1200
. : milestone, 1188,
iast (3.201 ms) : 3164, 3239
. : milestone, 3201,
iast_FULL (5.685 ms) : 5629, 5742
. : milestone, 5685,
iast_GLOBAL (3.596 ms) : 3544, 3648
. : milestone, 3596,
profiling (1.931 ms) : 1916, 1947
. : milestone, 1931,
tracing (1.792 ms) : 1778, 1805
. : milestone, 1792,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
. : milestone, 1471,
appsec (3.739 ms) : 3518, 3959
. : milestone, 3739,
iast (2.254 ms) : 2185, 2323
. : milestone, 2254,
iast_GLOBAL (2.295 ms) : 2225, 2364
. : milestone, 2295,
profiling (2.067 ms) : 2013, 2121
. : milestone, 2067,
tracing (2.074 ms) : 2021, 2128
. : milestone, 2074,
section candidate
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (2.541 ms) : 2484, 2598
. : milestone, 2541,
iast (2.253 ms) : 2184, 2322
. : milestone, 2253,
iast_GLOBAL (2.29 ms) : 2220, 2359
. : milestone, 2290,
profiling (2.095 ms) : 2038, 2151
. : milestone, 2095,
tracing (2.06 ms) : 2006, 2113
. : milestone, 2060,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~940f8b5d1e, baseline=1.59.0-SNAPSHOT~5a54a81cb0
dateFormat X
axisFormat %s
section baseline
no_agent (15.007 s) : 15007000, 15007000
. : milestone, 15007000,
appsec (14.665 s) : 14665000, 14665000
. : milestone, 14665000,
iast (18.225 s) : 18225000, 18225000
. : milestone, 18225000,
iast_GLOBAL (17.831 s) : 17831000, 17831000
. : milestone, 17831000,
profiling (14.789 s) : 14789000, 14789000
. : milestone, 14789000,
tracing (14.782 s) : 14782000, 14782000
. : milestone, 14782000,
section candidate
no_agent (15.334 s) : 15334000, 15334000
. : milestone, 15334000,
appsec (14.706 s) : 14706000, 14706000
. : milestone, 14706000,
iast (18.367 s) : 18367000, 18367000
. : milestone, 18367000,
iast_GLOBAL (17.894 s) : 17894000, 17894000
. : milestone, 17894000,
profiling (15.114 s) : 15114000, 15114000
. : milestone, 15114000,
tracing (14.651 s) : 14651000, 14651000
. : milestone, 14651000,
|
5 tasks
940f8b5 to
8c3688e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Support multi-modal prompts (text + images) in AI Guard.
Additional Notes
Validated with system tests at DataDog/system-tests#6131
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APPSEC-60264