Feat/new parsing layer

.claude/CLAUDE.md

@@ -36,26 +36,33 @@ to builders without direct coupling.
 | `scan_context.go` | `ScanCtx` / `NewScanCtx` — loads Go packages via `golang.org/x/tools/go/packages` |
 | `index.go` | `TypeIndex` — node classification (meta/route/operation/model/parameters/response) |
 | `declaration.go` | `EntityDecl` — wraps a type/value declaration with its enclosing file/package |
+| `classify/` | Classification predicates usable from both scanner and builders (e.g. `IsAllowedExtension`) |
 
-### `internal/parsers/` — comment-block parsing engine
+### `internal/parsers/` — scanner classification + helpers
+
+Post grammar-migration (P6.3), `parsers/` is intentionally scanner-only. The
+old regex-based comment-block parsing engine is gone; what remains are
+classification helpers used by the scanner and builders, plus subpackages
+for the grammar parser and its satellite helpers.
+
+**Root — scanner classification**
 
 | File | Contents |
 |------|----------|
-| `sectioned_parser.go` | The section-driven parser that walks title/description/annotation blocks |
-| `parsers.go`, `parsers_helpers.go` | Dispatch + helpers for tag/package filtering, value extraction |
-| `tag_parsers.go`, `matchers.go` | Tag recognisers (`TypeName`, `Model`, etc.) |
-| `regexprs.go` | Shared regular expressions for annotation parsing |
-| `meta.go` | Swagger info-block parsing (title, version, license, contact) |
-| `responses.go`, `route_params.go` | Response / route-parameter annotation parsing |
-| `validations.go`, `extensions.go` | Validation directives, `x-*` extensions |
-| `enum.go`, `security.go` | Enum extraction from Go constants, security-definition blocks |
-| `yaml_parser.go`, `yaml_spec_parser.go` | Embedded-YAML parsing for `swagger:operation` bodies |
-| `lines.go`, `parsed_path_content.go` | Comment-line and path-content helpers |
-| `errors.go` | Sentinel errors |
+| `matchers.go` | `ExtractAnnotation`, `ModelOverride`, `ResponseOverride`, `ParametersOverride` — the scanner-level annotation classifiers |
+| `regexprs.go` | Regex definitions backing the matchers + `rxRoute` / `rxOperation` for the path-annotation parsers |
+| `parsed_path_content.go` | `ParsedPathContent` + `ParseOperationPathAnnotation` / `ParseRoutePathAnnotation` |
+
+**Subpackages**
+
+| Package | Role |
+|---------|------|
+| `grammar/` | The grammar parser — `NewParser`, `Block`, `Property`, keyword tables |
+| `yaml/` | YAML sub-parser used by grammar's typed-extensions surface and by operation / meta body unmarshal |
 
 ### `internal/builders/` — Swagger object construction
 
-Each sub-package owns one concern and a `taggers.go` file wiring parsers to its targets.
+Each sub-package owns one concern; `walker.go` carries the per-block grammar dispatch.
 
 | Package | Contents |
 |---------|----------|
@@ -64,9 +71,11 @@ Each sub-package owns one concern and a `taggers.go` file wiring parsers to its
 | `operations` | Operation (route handler) annotation parsing |
 | `parameters` | Parameter annotation parsing |
 | `responses` | Response annotation parsing |
-| `routes` | Route/path discovery and matching |
-| `items` | Array-item targets (typable + validations, no own annotations) |
-| `resolvers` | `SwaggerSchemaForType`, identity/assertion helpers shared by builders |
+| `routes` | Route/path discovery + body parsers (`body_params.go`, `body_responses.go`) |
+| `common` | `*common.Builder` embedded by every per-decl builder; `SchemesList` + `SecurityRequirements` shared by routes/spec |
+| `handlers` | Walker callback factories shared across schema/parameters/responses (`Number`, `Integer`, `UniqueBool`, `PatternString`, …) |
+| `resolvers` | `SwaggerSchemaForType`, identity/assertion helpers, items-chain ifaces adapters (`ItemsTypable` / `ItemsValidations`) shared by builders |
+| `validations` | Type-aware coercion / shape-check primitives (`CoerceEnum`, `ParseDefault`, `IsLegalForType`) |
 
 ### `internal/ifaces/` — cross-package interfaces
 

.gitattributes

@@ -0,0 +1,6 @@
+# Golden test fixtures must keep LF line endings on every platform
+# so byte-level comparison doesn't trip on Windows checkouts that
+# default to core.autocrlf=true.
+*.json text eol=lf
+internal/parsers/grammar/grammar_test/testdata/golden/* text eol=lf
+fixtures/integration/golden/* text eol=lf

.github/copilot-instructions.md

@@ -8,7 +8,10 @@ and extracts API metadata — routes, parameters, responses, schemas, and more 
 `spec.Swagger` document. Supports Go modules (go1.11+).
 
 Single module: `github.com/go-openapi/codescan`. Public API is a thin facade at the root; the
-implementation lives under `internal/`.
+implementation lives under `internal/` and is split into three layers: **scanner** (package /
+AST ingestion), **parsers** (annotation grammar + sub-language parsers), and **builders**
+(emitting Swagger objects). A thin `ifaces` package glues parsers to builders without direct
+coupling.
 
 ### Layout
 
@@ -25,12 +28,20 @@ Internal tree:
 | Package | Role |
 |---------|------|
 | `internal/scanner` | Package loading via `golang.org/x/tools/go/packages`, entity discovery, `ScanCtx`, `TypeIndex`, `Options` |
-| `internal/parsers` | Comment-block parsing engine: sectioned parser, meta, responses, route params, validations, extensions, YAML body parser, enum extraction, security definitions |
-| `internal/builders/spec` | Top-level `Builder` orchestrating the final `*spec.Swagger` |
-| `internal/builders/schema` | Go type → Swagger schema conversion (largest builder) |
-| `internal/builders/{operations,parameters,responses,routes,items}` | Per-concern builders |
-| `internal/builders/resolvers` | `SwaggerSchemaForType` and shared assertion helpers |
-| `internal/ifaces` | `SwaggerTypable`, `ValidationBuilder`, `ValueParser`, `Objecter` — decouples parsers from builders |
+| `internal/scanner/classify` | Classification predicates shared with builders (e.g. `IsAllowedExtension`) |
+| `internal/parsers` | Scanner-level annotation classifiers (`ExtractAnnotation`, `ModelOverride`, …) and route/operation path-annotation parsing |
+| `internal/parsers/grammar` | Annotation grammar parser: preprocessor, lexer, recursive-descent parser, typed `Block` family, `Walker` visitor, diagnostics |
+| `internal/parsers/yaml` | YAML sub-parser used by the grammar's typed-extensions surface and by operation / meta body unmarshal |
+| `internal/parsers/routebody` | Sub-parser for the multi-line body grammar nested under `swagger:route` |
+| `internal/parsers/security` | Inline security-requirement line parser shared by routes and meta |
+| `internal/builders/spec` | Top-level orchestrator producing the final `*spec.Swagger` |
+| `internal/builders/schema` | Go type → Swagger schema conversion (largest builder); supports full Schema and SimpleSchema modes |
+| `internal/builders/{operations,parameters,responses,routes}` | Per-annotation builders |
+| `internal/builders/common` | `*common.Builder` embedded by every per-decl builder; parsed-block cache, post-decl queue, diagnostic accumulator, `MakeRef` |
+| `internal/builders/handlers` | Reusable Walker callback factories (`Number`, `Integer`, `UniqueBool`, `Extension`, parameter/schema dispatch) |
+| `internal/builders/validations` | Type-aware coercion (`CoerceEnum`, `ParseDefault`) + shape checks (`IsLegalForType`) |
+| `internal/builders/resolvers` | `SwaggerSchemaForType`, identity / assertion helpers, items-chain ifaces adapters |
+| `internal/ifaces` | `SwaggerTypable`, `ValidationBuilder`, `OperationValidationBuilder`, `ValueParser`, `Objecter` — decouples parsers from builders |
 | `internal/logger` | Debug logging (gated on `Options.Debug`) |
 | `internal/scantest` | Test utilities: golden compare, fixture loading, mocks, classification helpers |
 | `internal/integration` | Black-box integration tests against `fixtures/integration/golden/*.json` |
@@ -47,7 +58,7 @@ Fixtures:
 - `Run(*Options) (*spec.Swagger, error)` — scan Go packages and produce a Swagger spec.
 - `Options` — packages, work dir, build tags, include/exclude filters, `ScanModels`, `InputSpec`
   overlay, plus feature flags (`RefAliases`, `TransparentAliases`, `DescWithRef`,
-  `SetXNullableForPointers`, `SkipExtensions`, `Debug`).
+  `SetXNullableForPointers`, `SkipExtensions`, `OnDiagnostic`, `Debug`).
 
 ### Dependencies
 
@@ -82,7 +93,8 @@ Coding conventions are found beneath `.github/copilot` (symlinked to `.claude/ru
 - Tests: `go test ./...`. CI runs on `{ubuntu, macos, windows} x {stable, oldstable}` with `-race`.
 - Test framework: `github.com/go-openapi/testify/v2` (not `stretchr/testify`; `testifylint` does not work).
 - Parsers never import builders — write into the interfaces in `internal/ifaces`. When adding a new
-  annotation, extend the relevant builder's `taggers.go` rather than reaching into parser internals.
+  annotation keyword, extend the grammar keyword table and wire a matching Walker handler in the
+  relevant builder rather than reaching across the parser/builder boundary.
 - Test helpers live in `internal/scantest`; do not widen production API to satisfy a test.
 
 See `.github/copilot/` for detailed rules on Go conventions, linting, testing, and contributions.

.golangci.yml

@@ -4,8 +4,12 @@ linters:
   disable:
     - depguard
     - funlen
+    - goconst # disabled, perhaps temporarily as this linter has become way too pick and noisy
     - godox
+    - gomodguard
+    - gomodguard_v2
     - exhaustruct
+    - ireturn # not suited to the design of this repo
     - nlreturn
     - nonamedreturns
     - noinlineerr
@@ -55,6 +59,7 @@ formatters:
   exclusions:
     generated: lax
     paths:
+      - fixtures
       - third_party$
       - builtin$
       - examples$