Add test coverage check script and CI job#77
Closed
barrettruth wants to merge 55 commits intonvimdev:mainfrom
Closed
Add test coverage check script and CI job#77barrettruth wants to merge 55 commits intonvimdev:mainfrom
barrettruth wants to merge 55 commits intonvimdev:mainfrom
Conversation
# Conflicts: # test/linter/cpplint_spec.lua
replaces monolithic test job with parallel jobs grouped by package manager. each job installs only what it needs and runs its specific tests. - uses nvim-neorocks/nvim-busted-action for neovim + vusted setup - updates all actions to latest versions (checkout@v4, setup-python@v5, etc) - adds coverage job to warn when tools are missing tests - removes setup.sh and env.sh (no longer needed)
- Strip _extmark_id in linter helper (internal nvim field) - Update tests to use separate code field instead of message suffix - Update end_col values to match guard.nvim behavior - Update rustfmt expected output for new import formatting style - Update buf formatter expected output (compact style) - Add missing W292 diagnostic to flake8 test - Remove buf linter test (requires buf.yaml config)
Move tests from test/{formatter,linter}/ to test/{pip,npm,go,rust,lua,binary,clang,haskell}/
to allow CI to use glob patterns (busted test/pip/*_spec.lua) instead of
listing each test file explicitly.
This means adding new tools no longer requires workflow changes.
Run each clang test file in its own busted process to prevent test state from leaking between clang-format and clang-tidy tests.
Revert checkmake linter to original fn-based approach. Update test to match actual diagnostic output order.
Original test was named swiftformat.lua (missing _spec suffix) so busted never ran it. Restored with correct naming and added swiftformat installation to CI.
This reverts commit 51d94b1.
This reverts commit 3b5c3af.
# Conflicts: # test/lua/cpplint_spec.lua # test/lua/zsh_spec.lua
Old test helpers used guard.nvim's async pipeline with a hardcoded vim.wait(3000), causing flaky CI failures. New approach runs tools directly via vim.system():wait() — zero timing issues, tests real tool output, no dependency on guard.nvim's format/filetype modules. This commit strips all existing tests and CI jobs down to a single MVP (black formatter + flake8 linter) to validate the approach. Remaining tools will be re-added in a follow-up PR. BREAKING CHANGE: test helpers renamed from fmt_helper/lint_helper to a unified test/helper.lua with run_fmt(), run_lint(), get_linter()
Re-adds all previously covered tools using the new synchronous test infrastructure from the prior commit. All 8 CI job categories restored (pip, npm, go, rust, lua, binary, clang, haskell). Linter assertions now use structural checks (has diagnostics, correct source/bufnr/types) instead of exact field matching, preventing breakage when tool versions change output formatting.
…x, shfmt, jq, xmllint, hadolint)
…, typstyle, pg_format, ruff_fix, ruff linter, ormolu)
buf internally accesses /tmp/snap-private-tmp on CI runners, causing permission errors. Added opts.tmpdir and opts.cwd params to run_fmt helper so buf test uses a workspace-relative directory instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Problem: 11 tools (detekt, ktfmt, dprint, fnlfmt, csharpier, rubocop, cljfmt, npm_groovy_lint, mixformat, nixfmt, swift-format) had no test coverage, and CI lacked jobs for dotnet, ruby, clojure, elixir, nix, and swift ecosystems. Solution: add test specs for all 11 tools plus the npm_groovy_lint_fix variant. Add 6 new CI jobs (test-dotnet, test-ruby, test-clojure, test-elixir, test-nix, test-swift) and extend test-binary (detekt, ktfmt, dprint), test-lua (fnlfmt), and test-npm (npm-groovy-lint + java) with the required tool installs.
Problem: test-nix failed because cachix/install-nix-action does not set up channels, so nix-env -iA nixpkgs.* cannot resolve. test-dotnet failed because actions/setup-dotnet does not add ~/.dotnet/tools to PATH. Solution: use nix profile install with flake ref instead of nix-env, and export $HOME/.dotnet/tools onto PATH in the test step.
Problem: dprint test config lacked a plugins array so dprint found no formatting plugins. dotnet-csharpier was still not on PATH because inline export does not persist across GitHub Actions steps the way GITHUB_PATH does. Solution: add the typescript WASM plugin URL to dprint.json config. Use GITHUB_PATH instead of export for the dotnet tools directory so it takes effect in subsequent steps.
Problem: csharpier >= 1.0.0 renamed the binary from dotnet-csharpier to csharpier and made the format subcommand required. The old definition pointed at a binary that no longer exists. Solution: change cmd to csharpier and prepend format to the args. Remove the now-unnecessary GITHUB_PATH workaround since the tools directory is already on PATH on hosted runners.
Problem: npm-groovy-lint requires Java and a slow CodeNarc warm-up, dragging down the entire test-npm job for all lightweight JS tools. Solution: move npm-groovy-lint install and pre-warm to test-binary which already has Java 21. Move the test file to test/binary/. Drop setup-java from test-npm.
Problem: npm-groovy-lint requires a slow CodeNarc server warm-up that causes the CI job to hang. The pre-warm approach is unreliable. Solution: remove the test and CI install for now. Tracked as a skipped item in the coverage check.
barrettruth
force-pushed
the
test/coverage
branch
from
2fc4357 to
466c64e
Compare
Problem: mypy linter had no test coverage despite being testable with standard cmd/args/fname/parse pattern. Solution: add test that runs mypy on a file with a type error and asserts diagnostics are produced with correct structure. Install mypy in test-pip CI job.
barrettruth
force-pushed
the
test/coverage
branch
from
466c64e to
6a0c385
Compare
Problem: test-haskell job spent ~5 minutes setting up GHC, cabal, and compiling hlint + ormolu from source just to test two tools. Solution: download pre-built hlint and ormolu binaries in test-binary, which already follows this pattern. Delete the test-haskell job entirely. Move test files to test/binary/.
barrettruth
force-pushed
the
test/coverage
branch
from
6a0c385 to
9257839
Compare
Problem: mypyc was skipped in coverage despite sharing mypy's base config and shipping in the same pip package. Solution: add identical test to test/pip, reusing the same type-error input and structural assertions.
barrettruth
force-pushed
the
test/coverage
branch
from
9257839 to
9ba3fdb
Compare
Problem: prettierd was skipped in coverage as untestable despite being installable via npm. Solution: add test that mirrors the custom fn behavior (filename arg + stdin pipe) and install @fsouza/prettierd in the npm CI job.
barrettruth
force-pushed
the
test/coverage
branch
from
9ba3fdb to
3b0e61a
Compare
Problem: no way to verify that all formatters and linters have corresponding test files, making it easy to add a tool without tests. Solution: add scripts/coverage.lua that loads both modules, scans test files for references to each tool, and exits non-zero if any testable tool lacks coverage. Add a lightweight CI job and Makefile target.
Problem: npm-groovy-lint requires a slow CodeNarc server warm-up that causes CI hangs, so its test was removed from the test branch. Solution: add npm_groovy_lint and npm_groovy_lint_fix to the skip list.
Problem: mypyc was marked untestable but now has a test on test/all-tools. Solution: remove it from the skip list so coverage counts it.
Problem: the skip list was a plain string array with no explanation for why each tool is excluded. Solution: change to a name=reason table so justifications live next to the data. Update lookups from vim.tbl_contains to direct key access.
barrettruth
force-pushed
the
test/coverage
branch
from
c9c6a7d to
45e08fa
Compare
Member
|
Uh... I don't really mind if people bring prs without tests. I've never had issues with guard-collection before. There used to be tick boxes in the readme but I got that removed cause that confused people. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
There's no automated way to verify that all formatters and linters defined in the collection have corresponding test files. New tools can be added without tests and nobody would notice.
Solution
Add a lightweight coverage check that loads both modules, scans all test files for references to each tool name, and fails if any tool lacks coverage.
The coverage script lists the few skipped tools and why.
Add a
coverageCI job runs this on every push/PR — it only needs neovim and guard.nvim, no tool installs, so it completes in seconds. Amake coveragetarget is also added for local use.Depends on #76