Autoresearch optimization run for validation pipeline

.claude/settings.local.json

@@ -0,0 +1,12 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(ruby benchmark_regexp.rb:*)",
+      "Bash(bundle exec ruby:*)",
+      "Bash(git fetch:*)",
+      "Bash(ruby -e:*)",
+      "Bash(bundle install:*)"
+    ]
+  },
+  "enableAllProjectMcpServers": false
+}

autoresearch.checks.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+set -euo pipefail
+./test_fast.sh 2>&1 | tail -50

autoresearch.ideas.md

@@ -0,0 +1,25 @@
+# Autoresearch Ideas — Static Validation Performance
+
+## Tried and rejected
+- Local field def cache in collect_fields_inner — double-caching over Profile
+- Replace parents.dup+<< with [*parents, frag_type].freeze — slower due to splat
+- DefinitionDependencies Set→Array — within noise
+- Cache `referenced?` results per type_defn — within noise
+- Replace all_references Set with Array — regression
+- Bulk while-loop conversion — YJIT handles .each blocks well, no clear win
+- Eager return_type in Field struct — pays cost for all fields, no clear win
+- Memoize Schema::Field#type for resolver_class path — breaks tests (resolver class mutable)
+- Linked list / index-based @path (before stack→variable refactor) — same perf under YJIT
+- Sentinel-based fetch in required_args_cache — within noise
+- Reuse visited_fragments hash via clear — within noise
+- Cache single-element parent arrays per type + freeze — extra branching outweighs savings
+- Cache selections_may_conflict? per selections array — hash overhead exceeds savings
+- PendingField deferred definition lookup — extra is_a? branching costs more than saved
+- Replace Forwardable in NodeWithPath — within noise
+- Flatten Profile#field to single hash — array key hashing 5x slower than nested identity hash
+
+## Remaining opportunities (diminishing returns)
+- **collect_fields_inner** — 12.8% self-time, dominated by hash read/write (7.5%) and Field.new (1.4%). Fundamental to the algorithm.
+- **GC pressure** — 4-5%, mostly from Field objects and response_keys hashes. Would need object pooling.
+- **Merge on_field handler chain** — 7 method dispatches × 2247 fields = 15K super calls. Would require combining rule logic into single method, very invasive.
+- **Reduce Field object count** — 2050 per validation, 85% never compared. But deferred creation adds branching cost that offsets savings.

autoresearch.md

@@ -0,0 +1,67 @@
+# Autoresearch: Static Validation Pipeline Performance
+
+## Objective
+Optimize the entire static validation pipeline in graphql-ruby. The pipeline validates GraphQL queries against a schema before execution, checking field existence, argument compatibility, fragment usage, type correctness, and field mergeability.
+
+The benchmark runs 5 different validation workloads:
+- **introspection**: Introspection query against a card schema (~550µs)
+- **abstract_frags**: Fragment-heavy query with abstract types (~270µs)
+- **abstract_frags2**: Another abstract fragment query (~450µs)
+- **big_query**: Large query against a big schema (~4ms)
+- **fields_merge**: 5000 identical `hello` fields — tests FieldsWillMerge rule (~3.1s, 99.8% of total!)
+
+## Metrics
+- **Primary**: total_us (µs, lower is better) — sum of all 5 workload times
+- **Secondary**: introspection_us, abstract_frags_us, abstract_frags2_us, big_query_us, fields_merge_us
+
+## How to Run
+`./autoresearch.sh` — outputs `METRIC name=number` lines.
+
+## Files in Scope
+- `lib/graphql/static_validation/rules/fields_will_merge.rb` — THE bottleneck (O(n²) field comparison)
+- `lib/graphql/static_validation/base_visitor.rb` — AST visitor with validation hooks
+- `lib/graphql/static_validation/interpreter_visitor.rb` — Includes all rules into visitor
+- `lib/graphql/static_validation/validator.rb` — Entry point for validation
+- `lib/graphql/static_validation/validation_context.rb` — Context passed to validators
+- `lib/graphql/static_validation/definition_dependencies.rb` — Fragment dependency tracking
+- `lib/graphql/static_validation/literal_validator.rb` — Input literal validation
+- `lib/graphql/static_validation/all_rules.rb` — Rule ordering
+- `lib/graphql/static_validation/rules/*.rb` — All individual validation rules
+- `lib/graphql/language/static_visitor.rb` — Base AST visitor
+
+## Off Limits
+- Schema definition files (unless refactor needed to unlock perf)
+- Benchmark files (except autoresearch.sh)
+- Test files
+- Parser/lexer
+
+## Constraints
+- Tests must pass: `./test_fast.sh`
+- No new gem dependencies
+- Validation must produce identical results (same errors for same inputs)
+- Don't break public API
+
+## What's Been Tried
+1. **Deduplicate fields by signature in find_conflicts_within** — For fields with same response key, group by (name+definition+args) signature. Only compare across groups + one pair within. 143x speedup on pathological 5000-field case.
+2. **Replace Array#& with intersect?** in mutually_exclusive? — avoids allocating intersection array.
+3. **Eliminate redundant field lookups** — FieldsAreDefinedOnType was re-looking up field via types.field() when it was already in @field_definitions.last. RequiredArgumentsArePresent was calling types.arguments() twice.
+4. **Flatten fragment spreads into single field map** (XING article approach) — Instead of 3-phase (fields-within, fields-vs-fragments, fragments-vs-fragments) with exponential recursive fragment cross-comparison, expand all fragment spreads inline into one flat field list. Eliminates find_conflicts_between_fragments/find_conflicts_between_fields_and_fragment entirely. 23% improvement on big_query.
+5. **Caching fields_and_fragments_from_selection** — FAILED, `parents` parameter differs across calls for same node, affects mutually_exclusive? checks. Would need to separate raw field collection from parent tracking.
+
+## Key Insights (Updated)
+- Benchmark uses YJIT + visibility profiles + did_you_mean(nil) + allow_legacy_invalid_return_type_conflicts(false) to match production
+- Primary metric now includes large_query (36KB query against 31K-line schema) — most representative
+- `Visibility::Profile#field` is ~9% of large_query time. Profile caches internally, double-caching doesn't help.
+- `Visibility::Profile#initialize` creates ~12 Hash.new per validation (~5%)
+- `collect_fields_inner` is ~7% — Field struct creation + field lookups + parents array copies
+- `FieldsWillMergeError#add_conflict` used expensive `AbstractNode#==` for dedup — fixed with identity check
+- `FragmentTypesExist` was loading ALL types just to build did_you_mean dictionary even when did_you_mean disabled — fixed
+- Small benchmarks (abstract_frags ~550µs) have high variance, not useful for detecting small improvements
+
+## Key Insights
+- `fields_will_merge` is 99.8% of total time due to O(n²) `find_conflicts_within`
+- With 5000 identical `hello` fields grouped under same response key, it does ~12.5M comparisons
+- Each comparison in `find_conflict` checks field name equality, argument equality, return type conflicts, and recurses into sub-selections
+- The `same_arguments?` method uses `Array#find` (O(n)) for each argument comparison
+- `fields_and_fragments_from_selection` allocates Struct instances and arrays on every call
+- `compared_fragments_key` builds strings with interpolation for cache keys

autoresearch.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+set -euo pipefail
+
+# NOTE: benchmark/large_schema.graphql and benchmark/large_query.graphql
+# are private files not included in the repo. Provide your own large schema
+# and query files at those paths to run benchmarks.
+
+# Quick syntax check on key files
+ruby -c lib/graphql/static_validation/rules/fields_will_merge.rb > /dev/null 2>&1
+ruby -c lib/graphql/static_validation/base_visitor.rb > /dev/null 2>&1
+ruby -c lib/graphql/static_validation/validator.rb > /dev/null 2>&1
+
+exec bundle exec ruby -Ispec/support -e '
+RubyVM::YJIT.enable
+ADD_WARDEN = false
+TESTING_EXEC_NEXT = false
+TESTING_METHOD = false
+require "graphql"
+require "jazz"
+
+# Build schemas with visibility profiles enabled (production-like)
+CARD_SCHEMA = GraphQL::Schema.from_definition(File.read("benchmark/schema.graphql"))
+CARD_SCHEMA.use(GraphQL::Schema::Visibility)
+
+BIG_SCHEMA = GraphQL::Schema.from_definition(File.join("benchmark", "big_schema.graphql"))
+BIG_SCHEMA.use(GraphQL::Schema::Visibility)
+
+# Large real-world schema + query
+LARGE_SCHEMA_DEF = GraphQL::Schema.from_definition(File.read("benchmark/large_schema.graphql"))
+LARGE_SCHEMA_DEF.use(GraphQL::Schema::Visibility)
+LARGE_SCHEMA_DEF.allow_legacy_invalid_return_type_conflicts(false)
+LARGE_SCHEMA_DEF.did_you_mean(nil)
+
+FIELDS_WILL_MERGE_SCHEMA = GraphQL::Schema.from_definition("type Query { hello: String }")
+FIELDS_WILL_MERGE_SCHEMA.use(GraphQL::Schema::Visibility)
+
+# Parse documents
+ABSTRACT_FRAGMENTS = GraphQL.parse(File.read("benchmark/abstract_fragments.graphql"))
+ABSTRACT_FRAGMENTS_2 = GraphQL.parse(File.read("benchmark/abstract_fragments_2.graphql"))
+BIG_QUERY = GraphQL.parse(File.read("benchmark/big_query.graphql"))
+LARGE_QUERY = GraphQL.parse(File.read("benchmark/large_query.graphql"))
+FIELDS_WILL_MERGE_QUERY = GraphQL.parse("{ #{Array.new(5000, "hello").join(" ")} }")
+
+# Suppress warnings during benchmark
+$VERBOSE = nil
+
+# Pre-create Query objects so we only benchmark static validation itself
+# (not Query initialization, Profile creation, etc.)
+def make_query(schema, doc)
+  schema.query_class.new(schema, document: doc)
+end
+
+def make_validator(schema)
+  GraphQL::StaticValidation::Validator.new(schema: schema)
+end
+
+Q_AF  = make_query(CARD_SCHEMA, ABSTRACT_FRAGMENTS)
+Q_AF2 = make_query(CARD_SCHEMA, ABSTRACT_FRAGMENTS_2)
+Q_BQ  = make_query(BIG_SCHEMA, BIG_QUERY)
+Q_LQ  = make_query(LARGE_SCHEMA_DEF, LARGE_QUERY)
+Q_FM  = make_query(FIELDS_WILL_MERGE_SCHEMA, FIELDS_WILL_MERGE_QUERY)
+
+V_CARD     = make_validator(CARD_SCHEMA)
+V_BIG      = make_validator(BIG_SCHEMA)
+V_LARGE = make_validator(LARGE_SCHEMA_DEF)
+V_FM       = make_validator(FIELDS_WILL_MERGE_SCHEMA)
+
+max_errors = nil  # no limit
+
+# Warmup
+5.times do
+  V_CARD.validate(Q_AF, max_errors: max_errors)
+  V_CARD.validate(Q_AF2, max_errors: max_errors)
+  V_BIG.validate(Q_BQ, max_errors: max_errors)
+  V_LARGE.validate(Q_LQ, max_errors: max_errors)
+  V_FM.validate(Q_FM, max_errors: max_errors)
+end
+
+n_small = 80
+n_big = 50
+n_large = 20
+n_merge = 5
+times = {}
+
+t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+n_small.times { V_CARD.validate(Q_AF, max_errors: max_errors) }
+times[:abstract_frags] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n_small * 1_000_000).round(1)
+
+t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+n_small.times { V_CARD.validate(Q_AF2, max_errors: max_errors) }
+times[:abstract_frags2] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n_small * 1_000_000).round(1)
+
+t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+n_big.times { V_BIG.validate(Q_BQ, max_errors: max_errors) }
+times[:big_query] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n_big * 1_000_000).round(1)
+
+t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+n_large.times { V_LARGE.validate(Q_LQ, max_errors: max_errors) }
+times[:large_query] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n_large * 1_000_000).round(1)
+
+t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+n_merge.times { V_FM.validate(Q_FM, max_errors: max_errors) }
+times[:fields_merge] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n_merge * 1_000_000).round(1)
+
+# Primary metric: realistic workloads (abstract_frags + abstract_frags2 + big_query + large_query)
+realistic = (times[:abstract_frags] + times[:abstract_frags2] + times[:big_query] + times[:large_query]).round(1)
+puts "METRIC total_us=#{realistic}"
+times.each { |k,v| puts "METRIC #{k}_us=#{v}" }
+' 2>/dev/null

benchmark/compare_validation.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+# Before/after comparison of static validation performance.
+# Pre-creates Query+Profile objects, measures only validation itself.
+# Run with: bundle exec ruby -Ispec/support benchmark/compare_validation.rb
+#
+# NOTE: Requires private benchmark/large_schema.graphql and benchmark/large_query.graphql
+# files not included in the repo.
+
+RubyVM::YJIT.enable
+ADD_WARDEN = false
+TESTING_EXEC_NEXT = false
+TESTING_METHOD = false
+require "graphql"
+require "jazz"
+$VERBOSE = nil
+
+# Build schemas with visibility profiles enabled (production-like)
+CARD_SCHEMA = GraphQL::Schema.from_definition(File.read("benchmark/schema.graphql"))
+CARD_SCHEMA.use(GraphQL::Schema::Visibility)
+
+BIG_SCHEMA = GraphQL::Schema.from_definition(File.join("benchmark", "big_schema.graphql"))
+BIG_SCHEMA.use(GraphQL::Schema::Visibility)
+
+LARGE_SCHEMA_DEF = GraphQL::Schema.from_definition(File.read("benchmark/large_schema.graphql"))
+LARGE_SCHEMA_DEF.use(GraphQL::Schema::Visibility)
+LARGE_SCHEMA_DEF.allow_legacy_invalid_return_type_conflicts(false)
+LARGE_SCHEMA_DEF.did_you_mean(nil)
+
+FIELDS_WILL_MERGE_SCHEMA = GraphQL::Schema.from_definition("type Query { hello: String }")
+FIELDS_WILL_MERGE_SCHEMA.use(GraphQL::Schema::Visibility)
+
+# Parse documents
+ABSTRACT_FRAGMENTS = GraphQL.parse(File.read("benchmark/abstract_fragments.graphql"))
+ABSTRACT_FRAGMENTS_2 = GraphQL.parse(File.read("benchmark/abstract_fragments_2.graphql"))
+BIG_QUERY = GraphQL.parse(File.read("benchmark/big_query.graphql"))
+LARGE_QUERY = GraphQL.parse(File.read("benchmark/large_query.graphql"))
+FIELDS_WILL_MERGE_QUERY = GraphQL.parse("{ #{Array.new(5000, "hello").join(" ")} }")
+
+def make_query(schema, doc) = schema.query_class.new(schema, document: doc)
+def make_validator(schema) = GraphQL::StaticValidation::Validator.new(schema: schema)
+
+Q_AF  = make_query(CARD_SCHEMA, ABSTRACT_FRAGMENTS)
+Q_AF2 = make_query(CARD_SCHEMA, ABSTRACT_FRAGMENTS_2)
+Q_BQ  = make_query(BIG_SCHEMA, BIG_QUERY)
+Q_LQ  = make_query(LARGE_SCHEMA_DEF, LARGE_QUERY)
+Q_FM  = make_query(FIELDS_WILL_MERGE_SCHEMA, FIELDS_WILL_MERGE_QUERY)
+
+V_CARD     = make_validator(CARD_SCHEMA)
+V_BIG      = make_validator(BIG_SCHEMA)
+V_LARGE = make_validator(LARGE_SCHEMA_DEF)
+V_FM       = make_validator(FIELDS_WILL_MERGE_SCHEMA)
+
+# Warmup
+8.times do
+  V_CARD.validate(Q_AF)
+  V_CARD.validate(Q_AF2)
+  V_BIG.validate(Q_BQ)
+  V_LARGE.validate(Q_LQ)
+  V_FM.validate(Q_FM)
+end
+
+def bench(label, validator, query, n)
+  # 3 trials, take the median
+  results = 3.times.map do
+    t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    n.times { validator.validate(query) }
+    ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t) / n * 1_000_000).round(1)
+  end.sort
+  median = results[1]
+  puts "  %-20s %8.1f µs  (trials: %s)" % [label, median, results.map { |r| "%.1f" % r }.join(", ")]
+  median
+end
+
+puts "Static Validation Benchmark (YJIT, Visibility Profiles, pure validation)"
+puts "=" * 72
+times = {}
+times[:abstract_frags]  = bench("abstract_frags",  V_CARD, Q_AF, 100)
+times[:abstract_frags2] = bench("abstract_frags2", V_CARD, Q_AF2, 100)
+times[:big_query]       = bench("big_query",       V_BIG, Q_BQ, 60)
+times[:large_query]     = bench("large_query",     V_LARGE, Q_LQ, 30)
+times[:fields_merge]    = bench("fields_merge",    V_FM, Q_FM, 8)
+
+realistic = (times[:abstract_frags] + times[:abstract_frags2] + times[:big_query] + times[:large_query]).round(1)
+puts "-" * 72
+puts "  %-20s %8.1f µs" % ["TOTAL (realistic)", realistic]
+puts "  %-20s %8.1f µs" % ["fields_merge", times[:fields_merge]]

lib/graphql/language/static_visitor.rb

@@ -30,11 +30,16 @@ def on_document_children(document_node)
       end
 
       def on_field_children(new_node)
-        new_node.arguments.each do |arg_node| # rubocop:disable Development/ContextIsPassedCop
-          on_argument(arg_node, new_node)
+        args = new_node.arguments
+        if !args.empty?
+          args.each do |arg_node| # rubocop:disable Development/ContextIsPassedCop
+            on_argument(arg_node, new_node)
+          end
         end
-        visit_directives(new_node)
-        visit_selections(new_node)
+        dirs = new_node.directives
+        visit_directives(new_node) if !dirs.empty?
+        sels = new_node.selections
+        visit_selections(new_node) if !sels.empty?
       end
 
       def visit_directives(new_node)
@@ -59,17 +64,20 @@ def visit_selections(new_node)
       end
 
       def on_fragment_definition_children(new_node)
-        visit_directives(new_node)
+        visit_directives(new_node) if !new_node.directives.empty?
         visit_selections(new_node)
       end
 
       alias :on_inline_fragment_children :on_fragment_definition_children
 
       def on_operation_definition_children(new_node)
-        new_node.variables.each do |arg_node|
-          on_variable_definition(arg_node, new_node)
+        vars = new_node.variables
+        if !vars.empty?
+          vars.each do |arg_node|
+            on_variable_definition(arg_node, new_node)
+          end
         end
-        visit_directives(new_node)
+        visit_directives(new_node) if !new_node.directives.empty?
         visit_selections(new_node)
       end
 

lib/graphql/schema/list.rb

@@ -19,7 +19,7 @@ def list?
       end
 
       def to_type_signature
-        "[#{@of_type.to_type_signature}]"
+        @type_signature ||= -"[#{@of_type.to_type_signature}]"
       end
 
       # This is for introspection, where it's expected the name will be `null`

lib/graphql/schema/non_null.rb

@@ -24,7 +24,7 @@ def list?
       end
 
       def to_type_signature
-        "#{@of_type.to_type_signature}!"
+        @type_signature ||= -"#{@of_type.to_type_signature}!"
       end
 
       def inspect