Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API Security sampling #6923

Draft
wants to merge 15 commits into
base: master
Choose a base branch
from
Draft

API Security sampling #6923

wants to merge 15 commits into from

Conversation

ValentinZakharov
Copy link
Contributor

What Does This Do

Introduced API Security sampling mechanism

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

@ValentinZakharov ValentinZakharov added the comp: asm waf Application Security Management (WAF) label Apr 16, 2024
@ValentinZakharov ValentinZakharov self-assigned this Apr 16, 2024
@pr-commenter
Copy link

pr-commenter bot commented Apr 16, 2024 •
edited

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master vzakharov/api_sec_sampling
git_commit_date 1713861708 1713913003
git_commit_sha f34dfb5 1cf51e2
release_version 1.34.0-SNAPSHOT~f34dfb52a1 1.33.0-SNAPSHOT~1cf51e2d2c
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713915653 1713915653
ci_job_id 495028221 495028221
ci_pipeline_id 32766847 32766847
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 12 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:profiling:Remote Config better
[-67.396µs; -17.973µs] or [-9.373%; -2.500%]		 676.358µs 719.043µs
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.075 s) : 0, 1075443
Total [baseline] (8.545 s) : 0, 8545079
Agent [candidate] (1.083 s) : 0, 1082951
Total [candidate] (8.578 s) : 0, 8578386
section iast
Agent [baseline] (1.198 s) : 0, 1197709
Total [baseline] (9.007 s) : 0, 9006846
Agent [candidate] (1.199 s) : 0, 1198921
Total [candidate] (8.998 s) : 0, 8998033
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.197 s) : 0, 1197137
Total [baseline] (8.978 s) : 0, 8977894
Agent [candidate] (1.202 s) : 0, 1201630
Total [candidate] (8.974 s) : 0, 8974189
section iast_TELEMETRY_OFF
Agent [baseline] (1.196 s) : 0, 1196258
Total [baseline] (8.998 s) : 0, 8997555
Agent [candidate] (1.204 s) : 0, 1204267
Total [candidate] (9.011 s) : 0, 9010679
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent iast 1.198 s 122.266 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.197 s 121.694 ms (11.3%)
Agent iast_TELEMETRY_OFF 1.196 s 120.815 ms (11.2%)
Total tracing 8.545 s -
Total iast 9.007 s 461.767 ms (5.4%)
Total iast_HARDCODED_SECRET_DISABLED 8.978 s 432.815 ms (5.1%)
Total iast_TELEMETRY_OFF 8.998 s 452.476 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.083 s -
Agent iast 1.199 s 115.97 ms (10.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.202 s 118.679 ms (11.0%)
Agent iast_TELEMETRY_OFF 1.204 s 121.317 ms (11.2%)
Total tracing 8.578 s -
Total iast 8.998 s 419.648 ms (4.9%)
Total iast_HARDCODED_SECRET_DISABLED 8.974 s 395.804 ms (4.6%)
Total iast_TELEMETRY_OFF 9.011 s 432.293 ms (5.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.321 ms) : 0, 673321
BytebuddyAgent [candidate] (678.281 ms) : 0, 678281
GlobalTracer [baseline] (310.021 ms) : 0, 310021
GlobalTracer [candidate] (312.13 ms) : 0, 312130
AppSec [baseline] (49.477 ms) : 0, 49477
AppSec [candidate] (49.592 ms) : 0, 49592
Remote Config [baseline] (655.995 µs) : 0, 656
Remote Config [candidate] (666.146 µs) : 0, 666
Telemetry [baseline] (7.579 ms) : 0, 7579
Telemetry [candidate] (7.681 ms) : 0, 7681
section iast
BytebuddyAgent [baseline] (793.964 ms) : 0, 793964
BytebuddyAgent [candidate] (795.018 ms) : 0, 795018
GlobalTracer [baseline] (287.989 ms) : 0, 287989
GlobalTracer [candidate] (288.458 ms) : 0, 288458
AppSec [baseline] (50.258 ms) : 0, 50258
AppSec [candidate] (50.137 ms) : 0, 50137
Remote Config [baseline] (579.191 µs) : 0, 579
Remote Config [candidate] (573.342 µs) : 0, 573
Telemetry [baseline] (8.107 ms) : 0, 8107
Telemetry [candidate] (6.632 ms) : 0, 6632
IAST [baseline] (22.582 ms) : 0, 22582
IAST [candidate] (23.653 ms) : 0, 23653
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (793.507 ms) : 0, 793507
BytebuddyAgent [candidate] (796.203 ms) : 0, 796203
GlobalTracer [baseline] (287.413 ms) : 0, 287413
GlobalTracer [candidate] (289.123 ms) : 0, 289123
AppSec [baseline] (51.556 ms) : 0, 51556
AppSec [candidate] (49.077 ms) : 0, 49077
Remote Config [baseline] (563.449 µs) : 0, 563
Remote Config [candidate] (583.68 µs) : 0, 584
Telemetry [baseline] (6.548 ms) : 0, 6548
Telemetry [candidate] (6.571 ms) : 0, 6571
IAST [baseline] (23.185 ms) : 0, 23185
IAST [candidate] (25.51 ms) : 0, 25510
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (792.551 ms) : 0, 792551
BytebuddyAgent [candidate] (797.729 ms) : 0, 797729
GlobalTracer [baseline] (288.425 ms) : 0, 288425
GlobalTracer [candidate] (290.589 ms) : 0, 290589
AppSec [baseline] (48.797 ms) : 0, 48797
AppSec [candidate] (47.75 ms) : 0, 47750
Remote Config [baseline] (581.322 µs) : 0, 581
Remote Config [candidate] (576.048 µs) : 0, 576
Telemetry [baseline] (8.808 ms) : 0, 8808
Telemetry [candidate] (7.264 ms) : 0, 7264
IAST [baseline] (22.789 ms) : 0, 22789
IAST [candidate] (25.777 ms) : 0, 25777
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.092 s) : 0, 1092001
Total [baseline] (10.498 s) : 0, 10497704
Agent [candidate] (1.075 s) : 0, 1074608
Total [candidate] (10.26 s) : 0, 10260141
section appsec
Agent [baseline] (1.198 s) : 0, 1198435
Total [baseline] (10.427 s) : 0, 10426703
Agent [candidate] (1.205 s) : 0, 1204981
Total [candidate] (10.557 s) : 0, 10556580
section iast
Agent [baseline] (1.199 s) : 0, 1199112
Total [baseline] (10.706 s) : 0, 10706410
Agent [candidate] (1.198 s) : 0, 1197909
Total [candidate] (10.698 s) : 0, 10697882
section profiling
Agent [baseline] (1.276 s) : 0, 1275887
Total [baseline] (10.642 s) : 0, 10641717
Agent [candidate] (1.264 s) : 0, 1264305
Total [candidate] (10.59 s) : 0, 10590490
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.092 s -
Agent appsec 1.198 s 106.434 ms (9.7%)
Agent iast 1.199 s 107.11 ms (9.8%)
Agent profiling 1.276 s 183.886 ms (16.8%)
Total tracing 10.498 s -
Total appsec 10.427 s -71.001 ms (-0.7%)
Total iast 10.706 s 208.706 ms (2.0%)
Total profiling 10.642 s 144.013 ms (1.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent appsec 1.205 s 130.373 ms (12.1%)
Agent iast 1.198 s 123.3 ms (11.5%)
Agent profiling 1.264 s 189.697 ms (17.7%)
Total tracing 10.26 s -
Total appsec 10.557 s 296.439 ms (2.9%)
Total iast 10.698 s 437.741 ms (4.3%)
Total profiling 10.59 s 330.349 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (683.927 ms) : 0, 683927
BytebuddyAgent [candidate] (673.467 ms) : 0, 673467
GlobalTracer [baseline] (314.401 ms) : 0, 314401
GlobalTracer [candidate] (309.624 ms) : 0, 309624
AppSec [baseline] (50.414 ms) : 0, 50414
AppSec [candidate] (48.879 ms) : 0, 48879
Remote Config [baseline] (682.211 µs) : 0, 682
Remote Config [candidate] (653.935 µs) : 0, 654
Telemetry [baseline] (7.714 ms) : 0, 7714
Telemetry [candidate] (7.575 ms) : 0, 7575
section appsec
BytebuddyAgent [baseline] (694.195 ms) : 0, 694195
BytebuddyAgent [candidate] (700.269 ms) : 0, 700269
GlobalTracer [baseline] (290.607 ms) : 0, 290607
GlobalTracer [candidate] (294.066 ms) : 0, 294066
AppSec [baseline] (149.529 ms) : 0, 149529
AppSec [candidate] (148.794 ms) : 0, 148794
IAST [baseline] (19.198 ms) : 0, 19198
IAST [candidate] (19.094 ms) : 0, 19094
Remote Config [baseline] (606.369 µs) : 0, 606
Remote Config [candidate] (615.704 µs) : 0, 616
Telemetry [baseline] (9.965 ms) : 0, 9965
Telemetry [candidate] (7.473 ms) : 0, 7473
section iast
BytebuddyAgent [baseline] (794.244 ms) : 0, 794244
BytebuddyAgent [candidate] (794.011 ms) : 0, 794011
GlobalTracer [baseline] (288.404 ms) : 0, 288404
GlobalTracer [candidate] (288.139 ms) : 0, 288139
AppSec [baseline] (50.154 ms) : 0, 50154
AppSec [candidate] (49.803 ms) : 0, 49803
IAST [baseline] (23.339 ms) : 0, 23339
IAST [candidate] (24.492 ms) : 0, 24492
Remote Config [baseline] (571.624 µs) : 0, 572
Remote Config [candidate] (563.887 µs) : 0, 564
Telemetry [baseline] (8.144 ms) : 0, 8144
Telemetry [candidate] (6.567 ms) : 0, 6567
section profiling
BytebuddyAgent [baseline] (682.572 ms) : 0, 682572
BytebuddyAgent [candidate] (675.942 ms) : 0, 675942
GlobalTracer [baseline] (381.756 ms) : 0, 381756
GlobalTracer [candidate] (379.104 ms) : 0, 379104
AppSec [baseline] (50.427 ms) : 0, 50427
AppSec [candidate] (49.495 ms) : 0, 49495
Remote Config [baseline] (719.043 µs) : 0, 719
Remote Config [candidate] (676.358 µs) : 0, 676
Telemetry [baseline] (7.502 ms) : 0, 7502
Telemetry [candidate] (7.401 ms) : 0, 7401
ProfilingAgent [baseline] (96.113 ms) : 0, 96113
ProfilingAgent [candidate] (95.591 ms) : 0, 95591
Profiling [baseline] (96.136 ms) : 0, 96136
Profiling [candidate] (95.615 ms) : 0, 95615

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-23T23:10:40 2024-04-23T23:17:28
git_branch master vzakharov/api_sec_sampling
git_commit_date 1713861708 1713913003
git_commit_sha f34dfb5 1cf51e2
release_version 1.34.0-SNAPSHOT~f34dfb52a1 1.33.0-SNAPSHOT~1cf51e2d2c
start_time 2024-04-23T23:10:27 2024-04-23T23:17:15
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713914592 1713914592
ci_job_id 495028222 495028222
ci_pipeline_id 32766847 32766847
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.331 ms) : 1311, 1350
.   : milestone, 1331,
appsec (1.715 ms) : 1690, 1739
.   : milestone, 1715,
appsec_no_iast (1.727 ms) : 1702, 1751
.   : milestone, 1727,
iast (1.501 ms) : 1479, 1523
.   : milestone, 1501,
profiling (1.486 ms) : 1462, 1510
.   : milestone, 1486,
tracing (1.483 ms) : 1459, 1507
.   : milestone, 1483,
section candidate
no_agent (1.336 ms) : 1317, 1355
.   : milestone, 1336,
appsec (1.722 ms) : 1697, 1746
.   : milestone, 1722,
appsec_no_iast (1.722 ms) : 1698, 1745
.   : milestone, 1722,
iast (1.479 ms) : 1456, 1502
.   : milestone, 1479,
profiling (1.524 ms) : 1498, 1551
.   : milestone, 1524,
tracing (1.487 ms) : 1463, 1511
.   : milestone, 1487,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.311 ms, 1.35 ms] -
appsec 1.715 ms [1.69 ms, 1.739 ms] 383.799 µs (28.8%)
appsec_no_iast 1.727 ms [1.702 ms, 1.751 ms] 395.806 µs (29.7%)
iast 1.501 ms [1.479 ms, 1.523 ms] 170.358 µs (12.8%)
profiling 1.486 ms [1.462 ms, 1.51 ms] 155.384 µs (11.7%)
tracing 1.483 ms [1.459 ms, 1.507 ms] 152.101 µs (11.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.336 ms [1.317 ms, 1.355 ms] -
appsec 1.722 ms [1.697 ms, 1.746 ms] 385.371 µs (28.8%)
appsec_no_iast 1.722 ms [1.698 ms, 1.745 ms] 385.389 µs (28.8%)
iast 1.479 ms [1.456 ms, 1.502 ms] 142.781 µs (10.7%)
profiling 1.524 ms [1.498 ms, 1.551 ms] 188.23 µs (14.1%)
tracing 1.487 ms [1.463 ms, 1.511 ms] 151.048 µs (11.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.387 µs) : 344, 385
.   : milestone, 364,
iast (474.487 µs) : 453, 496
.   : milestone, 474,
iast_FULL (535.456 µs) : 515, 556
.   : milestone, 535,
iast_GLOBAL (495.907 µs) : 474, 518
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (469.974 µs) : 448, 492
.   : milestone, 470,
iast_INACTIVE (450.148 µs) : 429, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (469.62 µs) : 448, 492
.   : milestone, 470,
tracing (442.657 µs) : 422, 463
.   : milestone, 443,
section candidate
no_agent (368.789 µs) : 349, 388
.   : milestone, 369,
iast (477.915 µs) : 456, 499
.   : milestone, 478,
iast_FULL (538.443 µs) : 517, 559
.   : milestone, 538,
iast_GLOBAL (497.838 µs) : 477, 519
.   : milestone, 498,
iast_HARDCODED_SECRET_DISABLED (475.286 µs) : 453, 498
.   : milestone, 475,
iast_INACTIVE (446.095 µs) : 425, 467
.   : milestone, 446,
iast_TELEMETRY_OFF (466.673 µs) : 446, 487
.   : milestone, 467,
tracing (443.308 µs) : 422, 464
.   : milestone, 443,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.387 µs [344.017 µs, 384.757 µs] -
iast 474.487 µs [453.213 µs, 495.761 µs] 110.101 µs (30.2%)
iast_FULL 535.456 µs [514.577 µs, 556.336 µs] 171.07 µs (46.9%)
iast_GLOBAL 495.907 µs [474.271 µs, 517.544 µs] 131.521 µs (36.1%)
iast_HARDCODED_SECRET_DISABLED 469.974 µs [448.301 µs, 491.647 µs] 105.587 µs (29.0%)
iast_INACTIVE 450.148 µs [429.128 µs, 471.169 µs] 85.762 µs (23.5%)
iast_TELEMETRY_OFF 469.62 µs [447.566 µs, 491.674 µs] 105.233 µs (28.9%)
tracing 442.657 µs [422.059 µs, 463.255 µs] 78.27 µs (21.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.789 µs [349.257 µs, 388.322 µs] -
iast 477.915 µs [456.363 µs, 499.466 µs] 109.126 µs (29.6%)
iast_FULL 538.443 µs [517.436 µs, 559.45 µs] 169.654 µs (46.0%)
iast_GLOBAL 497.838 µs [476.768 µs, 518.907 µs] 129.048 µs (35.0%)
iast_HARDCODED_SECRET_DISABLED 475.286 µs [453.01 µs, 497.561 µs] 106.497 µs (28.9%)
iast_INACTIVE 446.095 µs [425.317 µs, 466.872 µs] 77.305 µs (21.0%)
iast_TELEMETRY_OFF 466.673 µs [445.977 µs, 487.368 µs] 97.884 µs (26.5%)
tracing 443.308 µs [422.321 µs, 464.295 µs] 74.519 µs (20.2%)

Dacapo

@ValentinZakharov ValentinZakharov force-pushed the vzakharov/api_sec_sampling branch 2 times, most recently from 700ae57 to 640b30d Compare April 19, 2024 13:59
@ValentinZakharov ValentinZakharov force-pushed the vzakharov/api_sec_sampling branch 5 times, most recently from 7078c4a to 82f629a Compare April 23, 2024 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ValentinZakharov ValentinZakharov

Labels
comp: asm waf Application Security Management (WAF)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@ValentinZakharov