feat: Add per-column metrics to summary#34
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #34 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 10 11 +1
Lines 930 1014 +84
=========================================
+ Hits 930 1014 +84 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds optional per-column numeric aggregation metrics to DataFrameComparison.summary() output (and JSON), with presets exposed via a new diffly.metrics module and a repeatable CLI flag. This addresses requests for “how much” numeric columns differ, not just “where”.
Changes:
- Introduce
diffly.metricswith preset metric callables and aquantile(q)factory. - Thread a new optional
metricsmapping throughsummary(), testing helpers, and the CLI (--metric). - Add unit + golden/fixture tests and update docs to describe/auto-document metrics.
Reviewed changes
Copilot reviewed 142 out of 142 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_metrics.py | New unit tests validating metric preset computations (mean/std/quantile/etc.). |
| tests/summary/test_summary.py | Extends summary JSON-shape assertions to include per-column metrics and adds a metrics-enabled scenario. |
| tests/summary/fixtures/metrics_presets_many/test_metrics_presets_many.py | Fixture generator for many preset metrics in summary output. |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “many presets” rendering when perfect matches hidden (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering when perfect matches hidden (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering when perfect matches hidden (plain). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain, no top changes). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain, no top changes). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain, full header). |
| tests/summary/fixtures/metrics_presets_many/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “many presets” rendering (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/test_metrics_presets_few.py | Fixture generator for a smaller set of preset metrics. |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI, no top changes). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI, no top changes). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI, full header, no top changes). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (pretty ANSI, full header, no top changes). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering without Top Changes (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns and no top changes (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns and no top changes (plain). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns and no top changes (plain, full header). |
| tests/summary/fixtures/metrics_presets_few/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for “few presets” rendering w/ imperfect-only columns and no top changes (plain, full header). |
| tests/summary/fixtures/metrics_long_labels/test_metrics_long_labels.py | Fixture generator to stress-test rendering with long metric labels. |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for long-label metrics rendering (plain). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for long-label metrics rendering (plain). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for long-label metrics rendering (plain, full header). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for long-label metrics rendering (plain, full header). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for long-label metrics rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for long-label metrics rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for long-label metrics rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_long_labels/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for long-label metrics rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_custom/test_metrics_custom.py | Fixture generator demonstrating a quantile metric + a user lambda metric. |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (pretty ANSI). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (pretty ANSI, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering without Top Changes (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (plain). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (plain, full header). |
| tests/summary/fixtures/metrics_custom/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt | Updated golden output for custom metrics rendering w/ imperfect-only columns and no Top Changes (plain, full header). |
| docs/index.md | Adds a “Per-column metrics” bullet to the landing page feature list. |
| docs/api/summary.rst | Documents the metrics API and auto-documents diffly.metrics presets/factory. |
| diffly/testing.py | Adds metrics passthrough to assert_frame_equal / assert_collection_equal and documents it. |
| diffly/metrics.py | New metrics module: metric type alias, presets, and quantile() factory. |
| diffly/comparison.py | Adds metrics parameter to DataFrameComparison.summary() and passes it into Summary. |
| diffly/cli.py | Adds repeatable --metric option and wires presets into summary rendering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Hi EgeKaraismailogluQC , looks very nice, I'd like to try it out!
Eric |
|
Hi Eric Brahmann (@EED85)! You can find the development guide here. |
Co-authored-by: Marius Merkle <122545105+MariusMerkleQC@users.noreply.github.com>
Oliver Borchert (borchero)
left a comment
There was a problem hiding this comment.
Thanks for the PR, this will be useful :)
Oliver Borchert (borchero)
left a comment
There was a problem hiding this comment.
Just one more open thread from my end, otherwise good to go, thank you! :)
Motivation
Closes #15. Users comparing data frames with numerical columns have asked for aggregate statistics (mean, quantiles, deviations) alongside the existing "Match Rate" and "Top Changes" per column. Until now the summary only showed where columns differ; this adds by how much.
The feature is optional: passing no
metricsargument preserves today's output.Changes
diffly.metricsmodule: preset callablesmean,median,min,max,std,mean_absolute_deviation,mean_relative_deviation, and aquantile(q)factory. Each computes an aggregation overleft, rightacross all joined rows.metricsargument onsummary(): acceptsMapping[str, Callable[[pl.Expr, pl.Expr], pl.Expr]]. Dict keys become column headers; values can be presets,quantile(q), or user lambdas.assert_frame_equal,assert_collection_equal, and exposed in the CLI via repeatable--metric <preset-name>..4gformatting. Non-numerical columns get blank cells.diffly.metrics, and a Key Features bullet on the landing page.Example
Specifying metrics:
Output (Columns section):
Custom metrics are equally easy: