Skip to content

feat: add preview review reference and update interactive iterate step #441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
johnnygreco wants to merge 13 commits into
base: main
Choose a base branch
from
+38 −5
Open

feat: add preview review reference and update interactive iterate step #441

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
6d7f39f
feat: add preview review reference for Data Designer skill
johnnygreco Mar 20, 2026
95ff92b
feat: add preview review offer to interactive workflow iterate step
johnnygreco Mar 20, 2026
e5f7b17
fix: remove stale "and serve again" from iterate step
johnnygreco Mar 20, 2026
a897cb7
fix: reframe design choices as general feature-fit guidance
johnnygreco Mar 20, 2026
b4bfdb4
fix: move judge calibration from design choices to data quality
johnnygreco Mar 20, 2026
0c916c1
fix: make review offer more prominent in iterate step
johnnygreco Mar 20, 2026
bcc94c1
fix: clarify "the user" in iterate step
johnnygreco Mar 20, 2026
205e647
fix: generalize generation time warning across workflows
johnnygreco Mar 20, 2026
b5922dd
fix: soften generation time warning wording
johnnygreco Mar 20, 2026
834f822
fix: rephrase generation time warning
johnnygreco Mar 20, 2026
d4f5556
feat: make preview review a dedicated workflow step
johnnygreco Mar 20, 2026
71eaa3a
fix: revert preview review to an offer within the iterate step
johnnygreco Mar 20, 2026
ee6fc7b
Merge branch 'main' into feat/skill-preview-review-reference
johnnygreco Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions skills/data-designer/references/preview-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Preview Review Guide

## Mindset

Quality is statistical, not per-record. Fix systemic issues that affect many records; don't chase cosmetic flaws in individual ones. But don't stop early — clear patterns of broken data or ignored instructions are worth fixing.

## Reading Sample Records

Load `dataset.parquet` from the preview results directory (printed as `Results path:` by the preview command, or the most recent `artifacts/preview_results_*/` directory). Use pandas to load the parquet file and print the records in a compact, reviewable format.
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 dataset.parquet existence not guaranteed by workflow

The guide instructs the agent to load dataset.parquet from the preview results directory, but the Preview step in both workflow files only documents that data-designer preview --save-results produces HTML files (specifically sample_records_browser.html). There is no mention anywhere in the workflow documentation that a dataset.parquet file is written to that directory.

If --save-results does not produce a parquet file, an agent following this guide will hit a missing-file error when trying to load it for self-review. Either:

  1. Confirm and document that --save-results always produces a dataset.parquet alongside the HTML output (and note this in the Preview steps of both workflow files), or
  2. Update this guide to use the HTML browser or another artefact that is known to exist.
Prompt To Fix With AI 
This is a comment left during a code review.
Path: skills/data-designer/references/preview-review.md
Line: 9

Comment:
**`dataset.parquet` existence not guaranteed by workflow**

The guide instructs the agent to load `dataset.parquet` from the preview results directory, but the `Preview` step in both workflow files only documents that `data-designer preview --save-results` produces HTML files (specifically `sample_records_browser.html`). There is no mention anywhere in the workflow documentation that a `dataset.parquet` file is written to that directory.

If `--save-results` does not produce a parquet file, an agent following this guide will hit a missing-file error when trying to load it for self-review. Either:
1. Confirm and document that `--save-results` always produces a `dataset.parquet` alongside the HTML output (and note this in the `Preview` steps of both workflow files), or
2. Update this guide to use the HTML browser or another artefact that is known to exist.

How can I resolve this? If you propose a fix, please make it concise.


## What to Look For

The specifics depend on the dataset and its intended use. The categories below are common starting points — adapt based on what matters for this dataset.

### Diversity
- **Mode collapse**: are records clustering around the same patterns, topics, or phrasings?
- **Sampler effectiveness**: are samplers being used effectively to steer diversity in the dataset?
- **Structural monotony**: do LLM-generated columns follow the same template across records?

### Data Quality
- **Instruction compliance**: does generated content follow prompt constraints (step counts, format requirements, allowed values)?
- **Internal consistency**: does data within a record agree with itself?
- **Encoding integrity**: no garbled encoding, mojibake, or broken unicode.
- **Plausibility**: do examples look like they could come from the real domain, or are they obviously synthetic?
- **Judge calibration** (if applicable): are scores consistent across similar-quality records? Does the judge catch visible problems?

### Design Choices
Are the right Data Designer features being used? For example:
- A text column that consistently produces structured data or code might be better as a specialized column type.
- Values drawn from a fixed set or known distribution could use a sampler instead of an LLM column.
4 changes: 2 additions & 2 deletions skills/data-designer/workflows/autopilot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ In this mode, make reasonable design decisions autonomously based on the dataset
- Note the sample records directory printed by the `data-designer preview` command
- Give the user a clickable link: `file://<sample-records-dir>/sample_records_browser.html`
7. **Create** — If the user specified a record count:
- 50 or fewer: run `data-designer create <path> --num-records <N> --dataset-name <name>` directly.
- More than 50: warn that generation can take a long time and ask for confirmation before running.
- Run `data-designer create <path> --num-records <N> --dataset-name <name>`.
- Generation speed depends heavily on the dataset configuration and the user's inference setup. For larger datasets, warn the user and ask for confirmation before running.
Comment on lines +23 to +24
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Contradictory instruction order in Create step

The first bullet instructs the agent to run the create command unconditionally, but the second bullet says to warn and ask for confirmation before running for larger datasets. An agent following these bullets in order would execute the long-running command first, then warn the user — making the confirmation meaningless.

The warning/confirmation check should come before the run instruction. Consider restructuring so the guard comes first:

Suggested change
- Run `data-designer create <path> --num-records <N> --dataset-name <name>`.
- Generation speed depends heavily on the dataset configuration and the user's inference setup. For larger datasets, warn the user and ask for confirmation before running.
- Generation speed depends heavily on the dataset configuration and the user's inference setup. For larger datasets, warn the user and ask for confirmation before running.
- Run `data-designer create <path> --num-records <N> --dataset-name <name>`.
Prompt To Fix With AI 
This is a comment left during a code review.
Path: skills/data-designer/workflows/autopilot.md
Line: 23-24

Comment:
**Contradictory instruction order in Create step**

The first bullet instructs the agent to run the create command unconditionally, but the second bullet says to warn and ask for confirmation before running for larger datasets. An agent following these bullets in order would execute the long-running command first, then warn the user — making the confirmation meaningless.

The warning/confirmation check should come before the run instruction. Consider restructuring so the guard comes first:

```suggestion
  - Generation speed depends heavily on the dataset configuration and the user's inference setup. For larger datasets, warn the user and ask for confirmation before running.
  - Run `data-designer create <path> --num-records <N> --dataset-name <name>`.
```

How can I resolve this? If you propose a fix, please make it concise.

- If no record count was specified, skip this step.
8. **Present** — Summarize what was built: columns, samplers used, key design choices. If the create command was run, share the results. Ask the user if they want any changes. If so, edit the script, re-validate, re-preview, and iterate.
9 changes: 6 additions & 3 deletions skills/data-designer/workflows/interactive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,11 @@ This is an interactive, iterative design process. Do not disengage from the loop
6. **Preview** — Run `data-designer preview <path> --save-results` to generate sample records as HTML files.
- Note the sample records directory printed by the `data-designer preview` command
- Give the user a clickable link: `file://<sample-records-dir>/sample_records_browser.html`
7. **Iterate** — Ask the user for feedback. Edit the script, re-validate, re-preview, and serve again. Repeat until they are satisfied.
7. **Iterate**
- Ask the user for feedback.
- Offer to review the records yourself and suggest improvements. If the user accepts, read `references/preview-review.md` for guidance.
- Apply changes, re-validate, and re-preview. Repeat until the user is satisfied.
8. **Finalize** — Once the user is happy, tell them they can run the following command to create the dataset:
- `data-designer create <path> --num-records <N> --dataset-name <name>`.
- Warn the user that generation can take a long time for large record counts (50+).
- Do not run this command yourself — it can take a long time for large datasets and the user should control when it runs.
- Caution the user that generation speed depends heavily on the dataset configuration and their inference setup.
- Do not run this command yourself — the user should control when it runs.
Loading