feat(engine): persist Python definitions into fqn_index (PR-02)#720
Open
shivasurya wants to merge 1 commit into
Open
feat(engine): persist Python definitions into fqn_index (PR-02)#720shivasurya wants to merge 1 commit into
shivasurya wants to merge 1 commit into
Conversation
SafeDep Report SummaryNo dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
Pathfinder Report✅ No security findings on the changed files. This pull request is clean. Powered by Code Pathfinder. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## shiva/rauth-pr-01-index-schema #720 +/- ##
==================================================================
+ Coverage 85.49% 85.54% +0.05%
==================================================================
Files 195 198 +3
Lines 27605 27756 +151
==================================================================
+ Hits 23600 23745 +145
- Misses 3113 3116 +3
- Partials 892 895 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
db99dae to
7926bf4
Compare
Populate the persisted FQN index with every Python function, method, and class the engine discovers during a scan, plus per-file mtime stamps so a later query can detect stale files. The index is a side effect of the same call-graph build, not a second analysis; the in-memory call graph is byte-identical with or without a cache. - BuildCallGraphWithCache: new entry point that takes an *AnalysisCache and, after the existing passes, writes the Python FQN slice. BuildCallGraph stays as a nil-cache wrapper so serve/integration/tests are untouched (same pattern as PR-01's OpenAnalysisCache). - fqn_collect.go: enumerates functions/methods (from callGraph.Functions) and classes (from the code graph), producing fqn_index rows with kind, parent FQN, signature, start line, and source="project". Every entry is project source: these are definitions in the scanned project's own .py files. Module-level variables and import aliases are deferred (see below). - fqn_writer.go: ReplacePythonFqnIndex atomically swaps the python slice of fqn_index and stamps indexed_files, all in one transaction; LookupFqn reader for PR-04. Go's caches are never touched. - file_mtime.go: indexed_files table + IsStale/GetIndexedFileMtime for per-file staleness, mtime-based per the spec (no checksum). - scan/ci: the analysis cache is now opened once before the call-graph build and shared by the Python and Go builders (was Go-only). Adds --index-path and --rebuild-index to ci for parity with scan. Deferred with reason: module-level variables and import aliases depend on internal scope/import-map iteration whose stable contract is not established; emitting uncertain rows would be worse than indexing the callable definitions rules actually match. Tracked as a Phase 0 follow-up. No global schema bump: indexed_files is additive and fqn_index was empty in PR-01 and is fully repopulated each run, so a wipe would only cold-start the warm Go cache. Verified: unit tests (collection + mtime at 100%, writer ~88% incl. error injection), an in-process integration test over a parsed Python project, and an end-to-end binary scan whose fqn_index output shows correct kinds, lines, parent FQNs, and signatures for functions/methods/classes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0d8a827 to
119cca7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Stack: PR-02 of 14 (Rule Authoring Toolchain, Phase 0). Stacked on #719 (PR-01) — review/merge that first; this PR's base is the PR-01 branch so the diff is scoped to PR-02.
What
Populate the persisted FQN index (added empty in PR-01) with every Python function, method, and class the engine discovers during a scan, plus per-file mtime stamps so a later query can detect stale files. The index is a side effect of the same call-graph build, not a second analysis. The in-memory call graph is identical with or without a cache.
Changes
BuildCallGraphWithCache(builder.go): new entry point taking an*AnalysisCache; after the existing passes it writes the Python FQN slice.BuildCallGraphstays a nil-cache wrapper soserve/integration/tests are untouched (the PR-01OpenAnalysisCachepattern).fqn_collect.go: enumerates functions/methods (fromcallGraph.Functions) and classes (from the code graph) intofqn_indexrows with kind, parent FQN, signature, start line. Every entry issource="project": these are definitions in the scanned project's own.pyfiles.fqn_writer.go:ReplacePythonFqnIndexatomically swaps the python slice offqn_indexand stampsindexed_files, in one transaction;LookupFqnreader for PR-04. Go'sfunction_index/pass4caches are never touched.file_mtime.go:indexed_filestable +IsStale/GetIndexedFileMtime, mtime-based per the spec (no checksum).scan/ci: the analysis cache is now opened once before the call-graph build and shared by the Python and Go builders (was Go-only).--index-path/--rebuild-indexadded tocifor parity withscan.Decisions (documented)
indexed_filesis additive (CREATE TABLE IF NOT EXISTS) andfqn_indexwas empty in PR-01 and is fully repopulated each run, so bumping would only gratuitously cold-start the warm Go cache. Matches the documented per-table policy. A per-tableindexed_files_versionguards that one table.Verification
go build ./...clean;go test ./...green;golangci-lint run->0 issues.BuildCallGraphWithCache, assertmodels.sanitize/models.User/models.User.save/__init__land infqn_index, allsource=project, and the file is stamped (not stale)..pyproject):Note: the repo
lintGotask chainslintPython, red onmainfrom pre-existing black formatting in threepython-sdk/files unrelated to this PR.🤖 Generated with Claude Code