Add llms.txt generation for AI-friendly article extraction#486
Merged
Add llms.txt generation for AI-friendly article extraction#486
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Implements the llms.txt standard (https://llmstxt.org/) to make PolicyEngine research articles more token-efficient for AI consumption. - Add scripts/generate-llms-txt.ts that generates: - /llms.txt - Index with links to sections - /llms-full.txt - All articles combined - /llms-research-us.txt - US articles only - /llms-research-uk.txt - UK articles only - Replace verbose Plotly JSON charts with text summaries using figure captions - Transform iframes to [Interactive: description] placeholders - Support optional ai_summary field in posts.json for custom summaries - Integrate into build process (runs before vite build) - Add tsx as dev dependency for running TypeScript scripts Closes #485 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Build scripts appropriately use console.log for progress output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
… test - Explicitly sort posts by date descending before slicing for Recent Research - Add llms-recent.txt (last 50 articles) as a lighter alternative to the 2.6MB full archive - Remove generated llms*.txt files from git tracking (already in .gitignore) - Add smoke test verifying all 5 output files, expected sections, and sort order Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MaxGhenis
force-pushed
the
feature/llms-txt
branch
from
4f5dc44 to
369ef21
Compare
- Extract slugFromFilename helper to deduplicate regex - Replace nested ternary with if/else for header selection - Simplify single-expression arrow callback - Consolidate stats printing into data-driven loop Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🎉 This PR is included in version 0.3.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
|
Sorry @github-actions[bot], only members of the PolicyEngine/core-developers team can invoke Claude Code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the llms.txt standard to make PolicyEngine research articles more token-efficient for AI consumption. This follows the pattern used by Bun, Svelte, and other projects.
Changes
Add
scripts/generate-llms-txt.tsthat generates:/llms.txt- Index with links to sections (~3KB)/llms-full.txt- All articles combined (~2.5MB, down from ~10MB+ with raw Plotly JSON)/llms-research-us.txt- US articles only (~1.3MB)/llms-research-uk.txt- UK articles only (~1.1MB)Replace verbose Plotly JSON charts with text summaries using figure captions
Transform iframes to
[Interactive: description]placeholdersSupport optional
ai_summaryfield in posts.json for custom summariesIntegrate into build process (runs before vite build)
Generated files are gitignored (built on deploy)
Token savings
[Interactive: ...]Workflow for PRs adding new articles
ai_summaryfield to your post inposts.jsonfor a custom AI-friendly summary**Figure 1: Winners by income decile**) - the caption becomes the chart summaryTest plan
npm run generate-llms-txt- generates all 4 files[Chart: caption][Interactive: ...]npm run build- llms.txt generated before vite buildCloses #485
🤖 Generated with Claude Code