feat(testing-cli): implement benchmark coverage script

[LouisTsai-Csie]

🗒️ Description

Introduces a new CLI tool benchmark_coverage that generates a markdown/JSON/terminal report showing which opcodes have benchmark test coverage for different benchmark modes (--gas-benchmark-values & --fixed-opcode-count mode).

• Scans benchmark tests and generates coverage report for each opcode
  • Distinguishes between two benchmark modes:
    • worst-case-benchmark (--gas-benchmark-values): Tests using benchmark_test wrapper
    • fixed-opcode-count (--fixed-opcode-count): Tests using benchmark_test + target_opcode= + code_generator=
  • Imports opcode definitions from ethereum.forks.osaka.vm.instructions.Ops for accurate sorting and completeness
  • Handles opcode aliases (SHA3 -> KECCAK256, DIFFICULTY -> PREVRANDAO).
  • Outputs markdown table sorted by opcode hex value

Detection logic:

Scenario	worst-case-benchmark	fixed-opcode-count
target_opcode= + code_generator=	✓	✓
target_opcode= without code_generator=	✓	✗
No target_opcode=, has @pytest.mark.parametrize("opcode", [...])	✓	✗

Usage:

  uv run benchmark_coverage           # Generate markdown report
  uv run benchmark_coverage --json    # Output as JSON
  uv run benchmark_coverage -o out.md # Write to file

Current Coverage Report

Opcode	Name	worst-case-benchmark	fixed-opcode-count
0x00	STOP	-	-
0x01	ADD	test_arithmetic	test_arithmetic
0x02	MUL	test_arithmetic	test_arithmetic
0x03	SUB	test_arithmetic	test_arithmetic
0x04	DIV	test_arithmetic	test_arithmetic
0x05	SDIV	test_arithmetic	test_arithmetic
0x06	MOD	test_arithmetic, test_mod	test_arithmetic
0x07	SMOD	test_arithmetic, test_mod	test_arithmetic
0x08	ADDMOD	test_mod_arithmetic	-
0x09	MULMOD	test_mod_arithmetic	-
0x0A	EXP	test_arithmetic	test_arithmetic
0x0B	SIGNEXTEND	test_arithmetic	test_arithmetic
0x10	LT	test_comparison	test_comparison
0x11	GT	test_comparison	test_comparison
0x12	SLT	test_comparison	test_comparison
0x13	SGT	test_comparison	test_comparison
0x14	EQ	test_comparison	test_comparison
0x15	ISZERO	test_iszero	test_iszero
0x16	AND	test_bitwise	test_bitwise
0x17	OR	test_bitwise	test_bitwise
0x18	XOR	test_bitwise	test_bitwise
0x19	NOT	test_not_op	test_not_op
0x1A	BYTE	test_bitwise	test_bitwise
0x1B	SHL	test_bitwise	test_bitwise
0x1C	SHR	test_bitwise, test_shifts	test_bitwise
0x1D	SAR	test_bitwise, test_shifts	test_bitwise
0x1E	CLZ	test_clz_diff, test_clz_same	test_clz_same
0x20	KECCAK	test_keccak, test_keccak_max_permutations	test_keccak, test_keccak_max_permutations
0x30	ADDRESS	test_call_frame_context_ops	test_call_frame_context_ops
0x31	BALANCE	test_ext_account_query_cold, test_ext_account_query_warm	test_ext_account_query_warm
0x32	ORIGIN	test_call_frame_context_ops	test_call_frame_context_ops
0x33	CALLER	test_call_frame_context_ops	test_call_frame_context_ops
0x34	CALLVALUE	test_callvalue_from_origin	test_callvalue_from_origin
0x35	CALLDATALOAD	test_calldataload	test_calldataload
0x36	CALLDATASIZE	test_calldatasize	test_calldatasize
0x37	CALLDATACOPY	test_calldatacopy_from_call, test_calldatacopy_from_origin	test_calldatacopy_from_call, test_calldatacopy_from_origin
0x38	CODESIZE	test_codesize	test_codesize
0x39	CODECOPY	test_codecopy	test_codecopy
0x3A	GASPRICE	test_call_frame_context_ops	test_call_frame_context_ops
0x3B	EXTCODESIZE	test_ext_account_query_warm, test_extcode_ops	test_ext_account_query_warm
0x3C	EXTCODECOPY	test_extcode_ops, test_extcodecopy_warm	test_extcodecopy_warm
0x3D	RETURNDATASIZE	test_returndatasize_nonzero, test_returndatasize_zero	test_returndatasize_nonzero, test_returndatasize_zero
0x3E	RETURNDATACOPY	test_returndatacopy	test_returndatacopy
0x3F	EXTCODEHASH	test_ext_account_query_warm, test_extcode_ops	test_ext_account_query_warm
0x40	BLOCKHASH	test_blockhash	test_blockhash
0x41	COINBASE	test_block_context_ops	test_block_context_ops
0x42	TIMESTAMP	test_block_context_ops	test_block_context_ops
0x43	NUMBER	test_block_context_ops	test_block_context_ops
0x44	PREVRANDAO	test_block_context_ops	test_block_context_ops
0x45	GASLIMIT	test_block_context_ops	test_block_context_ops
0x46	CHAINID	test_block_context_ops	test_block_context_ops
0x47	SELFBALANCE	test_selfbalance	test_selfbalance
0x48	BASEFEE	test_block_context_ops	test_block_context_ops
0x49	BLOBHASH	test_blobhash	test_blobhash
0x4A	BLOBBASEFEE	test_block_context_ops	test_block_context_ops
0x50	POP	-	-
0x51	MLOAD	test_memory_access	test_memory_access
0x52	MSTORE	test_memory_access	test_memory_access
0x53	MSTORE8	test_memory_access	test_memory_access
0x54	SLOAD	test_storage_access_cold, test_storage_access_warm	-
0x55	SSTORE	test_storage_access_cold, test_storage_access_warm	-
0x56	JUMP	test_jumps	-
0x57	JUMPI	test_jumpi_fallthrough, test_jumpis	test_jumpi_fallthrough
0x58	PC	test_pc_op	test_pc_op
0x59	MSIZE	test_msize	test_msize
0x5A	GAS	test_gas_op	test_gas_op
0x5B	JUMPDEST	test_jumpdests	test_jumpdests
0x5C	TLOAD	test_tload	test_tload
0x5D	TSTORE	test_tstore	test_tstore
0x5E	MCOPY	test_mcopy	test_mcopy
0x5F	PUSH0	test_push	test_push
0x60	PUSH1	test_push	test_push
0x61	PUSH2	test_push	test_push
0x62	PUSH3	test_push	test_push
0x63	PUSH4	test_push	test_push
0x64	PUSH5	test_push	test_push
0x65	PUSH6	test_push	test_push
0x66	PUSH7	test_push	test_push
0x67	PUSH8	test_push	test_push
0x68	PUSH9	test_push	test_push
0x69	PUSH10	test_push	test_push
0x6A	PUSH11	test_push	test_push
0x6B	PUSH12	test_push	test_push
0x6C	PUSH13	test_push	test_push
0x6D	PUSH14	test_push	test_push
0x6E	PUSH15	test_push	test_push
0x6F	PUSH16	test_push	test_push
0x70	PUSH17	test_push	test_push
0x71	PUSH18	test_push	test_push
0x72	PUSH19	test_push	test_push
0x73	PUSH20	test_push	test_push
0x74	PUSH21	test_push	test_push
0x75	PUSH22	test_push	test_push
0x76	PUSH23	test_push	test_push
0x77	PUSH24	test_push	test_push
0x78	PUSH25	test_push	test_push
0x79	PUSH26	test_push	test_push
0x7A	PUSH27	test_push	test_push
0x7B	PUSH28	test_push	test_push
0x7C	PUSH29	test_push	test_push
0x7D	PUSH30	test_push	test_push
0x7E	PUSH31	test_push	test_push
0x7F	PUSH32	test_push	test_push
0x80	DUP1	test_dup	test_dup
0x81	DUP2	test_dup	test_dup
0x82	DUP3	test_dup	test_dup
0x83	DUP4	test_dup	test_dup
0x84	DUP5	test_dup	test_dup
0x85	DUP6	test_dup	test_dup
0x86	DUP7	test_dup	test_dup
0x87	DUP8	test_dup	test_dup
0x88	DUP9	test_dup	test_dup
0x89	DUP10	test_dup	test_dup
0x8A	DUP11	test_dup	test_dup
0x8B	DUP12	test_dup	test_dup
0x8C	DUP13	test_dup	test_dup
0x8D	DUP14	test_dup	test_dup
0x8E	DUP15	test_dup	test_dup
0x8F	DUP16	test_dup	test_dup
0x90	SWAP1	test_swap	test_swap
0x91	SWAP2	test_swap	test_swap
0x92	SWAP3	test_swap	test_swap
0x93	SWAP4	test_swap	test_swap
0x94	SWAP5	test_swap	test_swap
0x95	SWAP6	test_swap	test_swap
0x96	SWAP7	test_swap	test_swap
0x97	SWAP8	test_swap	test_swap
0x98	SWAP9	test_swap	test_swap
0x99	SWAP10	test_swap	test_swap
0x9A	SWAP11	test_swap	test_swap
0x9B	SWAP12	test_swap	test_swap
0x9C	SWAP13	test_swap	test_swap
0x9D	SWAP14	test_swap	test_swap
0x9E	SWAP15	test_swap	test_swap
0x9F	SWAP16	test_swap	test_swap
0xA0	LOG0	test_log	test_log
0xA1	LOG1	test_log	test_log
0xA2	LOG2	test_log	test_log
0xA3	LOG3	test_log	test_log
0xA4	LOG4	test_log	test_log
0xF0	CREATE	test_create, test_creates_collisions	test_create, test_creates_collisions
0xF1	CALL	test_ext_account_query_warm, test_xcall	test_ext_account_query_warm
0xF2	CALLCODE	test_ext_account_query_warm, test_xcall	test_ext_account_query_warm
0xF3	RETURN	test_return_revert	test_return_revert
0xF4	DELEGATECALL	test_ext_account_query_warm, test_xcall	test_ext_account_query_warm
0xF5	CREATE2	test_create, test_creates_collisions	test_create, test_creates_collisions
0xFA	STATICCALL	test_ext_account_query_warm, test_xcall	test_ext_account_query_warm
0xFD	REVERT	test_return_revert	test_return_revert
0xFF	SELFDESTRUCT	test_selfdestruct_existing	-

🔗 Related Issues or PRs

Require #1869

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx tox -e static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
• Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
• Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.87%. Comparing base (cc82819) to head (35d6d67).

Additional details and impacted files

@@             Coverage Diff              @@
##           forks/osaka    #1898   +/-   ##
============================================
  Coverage        83.87%   83.87%           
============================================
  Files              402      402           
  Lines            25101    25101           
  Branches          2285     2285           
============================================
  Hits             21053    21053           
  Misses            3609     3609           
  Partials           439      439           

Flag	Coverage Δ
unittests	83.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.