Reduce token scan overhead

[dduan]

Share lexToken across parser paths and load byte-class tables only when needed during scanString.

[github-actions]

Comparing results between 'main' and 'pull_request'

Host 'runnervmh13bl' with 4 'x86_64' processors with 15 GB memory, running:
#18~24.04.1-Ubuntu SMP Sat Jun 28 04:46:03 UTC 2025

TOMLDecoderBenchmarks

Decode toml.io example metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (μs) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	69	69	70	70	76	92	272	7092
pull_request	69	70	70	71	75	89	131	7098
Δ	0	1	0	1	-1	-3	-141	6
Improvement %	0	-1	0	-1	1	3	52	6

Retains: results within specified thresholds, fold down for details.

Retains *	p0	p25	p50	p75	p90	p99	p100	Samples
main	624	624	624	624	624	624	624	7092
pull_request	624	624	624	624	624	624	624	7098
Δ	0	0	0	0	0	0	0	6
Improvement %	0	0	0	0	0	0	0	6

Parse toml.io example metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (ns) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	6442	6543	6583	6635	6715	9671	58489	10000
pull_request	6211	6343	6383	6435	6515	10455	43792	10000
Δ	-231	-200	-200	-200	-200	784	-14697	0
Improvement %	4	3	3	3	3	-8	25	0

Retains: results within specified thresholds, fold down for details.

Retains *	p0	p25	p50	p75	p90	p99	p100	Samples
main	8	9	9	9	9	9	9	10000
pull_request	8	9	9	9	9	9	9	10000
Δ	0	0	0	0	0	0	0	0
Improvement %	0	0	0	0	0	0	0	0

decode canada.toml metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (ms) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	268	268	268	269	270	270	270	4
pull_request	264	264	265	265	266	266	266	4
Δ	-4	-4	-3	-4	-4	-4	-4	0
Improvement %	1	1	1	1	1	1	1	0

Retains: results within specified thresholds, fold down for details.

Retains (K) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	1674	1674	1674	1674	1674	1674	1674	4
pull_request	1674	1674	1674	1674	1674	1674	1674	4
Δ	0	0	0	0	0	0	0	0
Improvement %	0	0	0	0	0	0	0	0

decode twitter.toml metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (μs) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	9413	9470	9486	9511	9552	9789	10020	105
pull_request	9091	9118	9134	9159	9208	9486	9680	109
Δ	-322	-352	-352	-352	-344	-303	-340	4
Improvement %	3	4	4	4	4	3	3	4

Retains: results within specified thresholds, fold down for details.

Retains (K) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	66	66	66	66	66	66	66	105
pull_request	66	66	66	66	66	66	66	109
Δ	0	0	0	0	0	0	0	4
Improvement %	0	0	0	0	0	0	0	4

parse canada.toml metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (ms) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	17	17	17	18	18	19	19	58
pull_request	15	15	16	16	16	18	18	64
Δ	-2	-2	-1	-2	-2	-1	-1	6
Improvement %	12	12	6	11	11	5	5	6

Retains: results within specified thresholds, fold down for details.

Retains *	p0	p25	p50	p75	p90	p99	p100	Samples
main	132	133	133	133	133	133	133	58
pull_request	132	133	133	133	133	133	133	64
Δ	0	0	0	0	0	0	0	6
Improvement %	0	0	0	0	0	0	0	6

parse twitter.toml metrics

Time (wall clock): results within specified thresholds, fold down for details.

Time (wall clock) (μs) *	p0	p25	p50	p75	p90	p99	p100	Samples
main	3045	3066	3084	3099	3119	3287	3594	319
pull_request	2894	2916	2927	2941	2953	3127	3372	335
Δ	-151	-150	-157	-158	-166	-160	-222	16
Improvement %	5	5	5	5	5	5	6	16

Retains: results within specified thresholds, fold down for details.

Retains *	p0	p25	p50	p75	p90	p99	p100	Samples
main	1292	1293	1293	1293	1293	1293	1293	319
pull_request	1292	1293	1293	1293	1293	1293	1293	335
Δ	0	0	0	0	0	0	0	16
Improvement %	0	0	0	0	0	0	0	16