Skip to content

Commit b7198f3

Browse files
tdhopperclaude
tdhopper
and
claude
committed
Add migration and testing documentation
MIGRATION_NOTES.md: - Comprehensive overview of all changes made - Detailed dependency update table - Next steps and setup instructions - Potential issues and rollback plan - Success criteria checklist TESTING_CHECKLIST.md: - Environment setup verification steps - Build testing procedures - Known migration issues to watch for - CI/CD testing guide - Performance check recommendations These documents provide guidance for the first build with modernized dependencies and help identify compatibility issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 2497c2a commit b7198f3

File tree

2 files changed

+393
-0
lines changed

2 files changed

+393
-0
lines changed

‎MIGRATION_NOTES.md‎

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
# Migration Notes: Modernization to 2025
2+
3+
This document summarizes the modernization work completed on pythonplot.com.
4+
5+
## Overview
6+
7+
The project has been modernized from 2020-era dependencies and infrastructure to 2025 standards:
8+
- Python 3.6 → 3.11
9+
- Conda → uv package manager
10+
- Travis CI → GitHub Actions
11+
- Outdated plotting libraries → Latest versions
12+
- Deprecated packages removed
13+
14+
## Changes Made
15+
16+
### 1. Python Environment Modernization
17+
18+
**New Files:**
19+
- `pyproject.toml`: Modern Python project configuration with all dependencies
20+
- `setup_r.sh`: Script to install R packages (ggplot2, mgcv)
21+
22+
**Updated Files:**
23+
- `runtime.txt`: 3.6 → 3.11
24+
- `requirements.txt`: All packages updated to latest versions with minimum version constraints
25+
26+
**Removed Dependencies:**
27+
- `altair-saver` → Deprecated, using native altair rendering
28+
- `chart-studio` → Not needed with plotly v5 local rendering
29+
- `plotly-orca` → Replaced by Kaleido
30+
- Duplicate `selenium` entry removed
31+
32+
**Updated Dependencies:**
33+
| Package | Old Version | New Version | Notes |
34+
|---------|-------------|-------------|-------|
35+
| plotly | 4.10.0 | 5.24+ | No authentication needed for local rendering |
36+
| plotnine | <0.7.0 | 0.13+ | Better ggplot2 compatibility |
37+
| statsmodels | 0.11.1 | 0.14+ | Security fixes |
38+
| seaborn | (latest) | 0.13+ | Updated API |
39+
| altair | (latest) | 5.0+ | Major version bump |
40+
| rpy2 | (unversioned) | 3.5+ | Python 3.11+ compatible |
41+
| selenium | (duplicated) | 4.15+ | API changes in v4 |
42+
| pandas | (latest) | 2.0+ | Major improvements |
43+
| matplotlib | (latest) | 3.7+ | Modern features |
44+
45+
### 2. R Integration Updates
46+
47+
**Approach:** Separate system R installation (no longer using conda)
48+
49+
**Changes:**
50+
- Created `setup_r.sh` to install R packages
51+
- R must be installed separately (via Homebrew, apt, or CRAN)
52+
- rpy2 updated to work with system R and Python 3.11+
53+
- Documentation updated to explain R setup
54+
55+
**Environment Variable:** `R_HOME` must be set correctly for rpy2 to work
56+
57+
### 3. Build System Updates
58+
59+
**Makefile Changes:**
60+
- Removed `plotly_auth` target (no longer needed)
61+
- Added `setup` target for uv-based dependency installation
62+
- Updated `dev_environment` to use uv instead of conda
63+
- Updated `.PHONY` declarations
64+
65+
**What Didn't Change:**
66+
- Core build targets (`render`, `qrender`, `test`, `run_nb`)
67+
- render.py script (compatible with new libraries)
68+
- Notebook structure and tagging system
69+
70+
### 4. CI/CD Migration
71+
72+
**From:** Travis CI
73+
**To:** GitHub Actions
74+
75+
**New File:** `.github/workflows/deploy.yml`
76+
77+
**Workflow Features:**
78+
- Uses r-lib/setup-r for R installation
79+
- Uses uv for Python dependencies
80+
- Runs on every push and pull request
81+
- Automatic Netlify deployment:
82+
- Production: master branch
83+
- Preview: all other branches
84+
85+
**Required GitHub Secrets:**
86+
- `NETLIFY_AUTH_TOKEN`
87+
- `NETLIFY_SITE_ID`
88+
89+
**Legacy Files (Now Deprecated):**
90+
- `.travis.yml`
91+
- `.travis/authenticate_plotly.py` (marked deprecated with explanation)
92+
- `.travis/install_conda.sh` (wasn't being used)
93+
- `.travis/invalidate_cloudfront.py` (not needed with Netlify)
94+
95+
### 5. Documentation Updates
96+
97+
**README.md:**
98+
- Added modern development setup instructions
99+
- Updated to reflect uv usage
100+
- Added build and CI/CD sections
101+
- Clarified plotly no longer needs authentication
102+
103+
**CLAUDE.md:**
104+
- Updated all dependency versions
105+
- Documented new setup process
106+
- Added CI/CD section
107+
- Updated technical constraints
108+
109+
**New Documentation:**
110+
- `TESTING_CHECKLIST.md`: Comprehensive testing guide for first build
111+
- `MIGRATION_NOTES.md`: This file
112+
113+
## Next Steps
114+
115+
### Before First Build
116+
117+
1. **Install System Dependencies:**
118+
```bash
119+
# macOS
120+
brew install r
121+
122+
# Ubuntu/Debian
123+
sudo apt-get install r-base r-base-dev
124+
```
125+
126+
2. **Set Up Development Environment:**
127+
```bash
128+
make dev_environment
129+
```
130+
131+
3. **Verify Setup:**
132+
```bash
133+
# Test Python packages
134+
python -c "import pandas, plotly, seaborn, plotnine, altair, rpy2; print('✓ Python packages OK')"
135+
136+
# Test R integration
137+
python -c "import rpy2.robjects as ro; ro.r('library(ggplot2)'); print('✓ R integration OK')"
138+
```
139+
140+
### Testing the Build
141+
142+
4. **Run Tests:**
143+
```bash
144+
make test
145+
```
146+
147+
5. **Try Quick Render (no notebook execution):**
148+
```bash
149+
make qrender
150+
```
151+
152+
6. **Full Build (executes notebook):**
153+
```bash
154+
make render
155+
```
156+
- This will take several minutes
157+
- Watch for any library compatibility issues
158+
- Check `TESTING_CHECKLIST.md` for specific things to verify
159+
160+
### Setting Up CI/CD
161+
162+
7. **Configure GitHub Secrets:**
163+
- Go to your GitHub repo → Settings → Secrets and variables → Actions
164+
- Add `NETLIFY_AUTH_TOKEN` (from Netlify account settings)
165+
- Add `NETLIFY_SITE_ID` (from Netlify site settings)
166+
167+
8. **Test GitHub Actions:**
168+
- Push to a test branch
169+
- Check Actions tab to see workflow run
170+
- Verify Netlify preview deployment
171+
172+
9. **Deploy to Production:**
173+
- Merge to master branch
174+
- GitHub Actions will automatically deploy to production
175+
176+
## Potential Issues to Watch For
177+
178+
### Library Compatibility
179+
180+
1. **plotly v5 Changes:**
181+
- API changes from v4 to v5
182+
- Different image rendering (Kaleido vs Orca)
183+
- No authentication needed
184+
185+
2. **altair v5 Changes:**
186+
- May need updates to save PNG images
187+
- Selenium/geckodriver integration
188+
189+
3. **plotnine 0.13:**
190+
- Better ggplot2 compatibility but some API changes
191+
192+
4. **statsmodels 0.14:**
193+
- Check regression plots and confidence intervals
194+
195+
### Environment Issues
196+
197+
1. **R_HOME:**
198+
- Must be set correctly for rpy2
199+
- GitHub Actions sets this automatically
200+
- Locally: `export R_HOME=$(R RHOME)`
201+
202+
2. **Headless Rendering:**
203+
- Selenium needs xvfb on Linux
204+
- GitHub Actions handles this automatically
205+
- Locally on Linux: `xvfb-run make render`
206+
207+
3. **geckodriver:**
208+
- Must be in PATH for selenium
209+
- GitHub Actions installs this automatically
210+
- Locally: Install via package manager or download
211+
212+
## Rollback Plan
213+
214+
If you encounter critical issues:
215+
216+
1. **Keep Travis CI temporarily:**
217+
- Don't delete `.travis.yml` until GitHub Actions is proven
218+
- Both can run simultaneously
219+
220+
2. **Pin problematic packages:**
221+
- If specific library versions cause issues, pin to older versions in requirements.txt
222+
- Example: `plotly==4.14.3` if v5 has problems
223+
224+
3. **Gradual migration:**
225+
- Update libraries one at a time
226+
- Test each change thoroughly
227+
228+
## Files Changed Summary
229+
230+
**New Files:**
231+
- `pyproject.toml`
232+
- `setup_r.sh`
233+
- `.github/workflows/deploy.yml`
234+
- `TESTING_CHECKLIST.md`
235+
- `MIGRATION_NOTES.md`
236+
237+
**Modified Files:**
238+
- `runtime.txt`
239+
- `requirements.txt`
240+
- `Makefile`
241+
- `README.md`
242+
- `CLAUDE.md`
243+
- `.travis/authenticate_plotly.py` (deprecated with explanation)
244+
245+
**Deprecated (Not Deleted):**
246+
- `environment.yml` (conda environment - no longer used)
247+
- `.travis.yml` (Travis CI config)
248+
- `.travis/` directory (old CI scripts)
249+
250+
**No Changes:**
251+
- `render.py` (compatible with new libraries)
252+
- `Examples.ipynb` (may need updates if libraries break examples)
253+
- `templates/t_index.html`
254+
- `tests/test_plots.py`
255+
- `INTRO.md`
256+
257+
## Success Criteria
258+
259+
The migration is successful when:
260+
261+
- [ ] `make dev_environment` completes without errors
262+
- [ ] `make test` passes all tests
263+
- [ ] `make render` executes notebook and generates all plots
264+
- [ ] GitHub Actions workflow runs successfully
265+
- [ ] Site deploys correctly to Netlify
266+
- [ ] All plotting libraries render correctly
267+
- [ ] R ggplot2 examples work via rpy2
268+
269+
## Support
270+
271+
If you encounter issues:
272+
273+
1. Check `TESTING_CHECKLIST.md` for common problems
274+
2. Review GitHub Actions logs for CI issues
275+
3. Check package compatibility in individual notebook cells
276+
4. File issues in the GitHub repository
277+
278+
---
279+
280+
**Migration completed:** 2025-11-04
281+
**Python version:** 3.11+
282+
**Primary changes:** uv adoption, library updates, CI/CD modernization

‎TESTING_CHECKLIST.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Testing Checklist for Modernized Build
2+
3+
This checklist should be completed after setting up the modernized development environment.
4+
5+
## Environment Setup Testing
6+
7+
- [ ] **Install R** (if not already installed)
8+
```bash
9+
# macOS
10+
brew install r
11+
12+
# Ubuntu/Debian
13+
sudo apt-get install r-base r-base-dev
14+
```
15+
16+
- [ ] **Run R setup script**
17+
```bash
18+
./setup_r.sh
19+
```
20+
Expected: ggplot2 and mgcv packages installed successfully
21+
22+
- [ ] **Install Python dependencies with uv**
23+
```bash
24+
# Install uv if needed
25+
curl -LsSf https://astral.sh/uv/install.sh | sh
26+
27+
# Install dependencies
28+
uv pip install -r requirements.txt
29+
```
30+
Expected: All packages install without errors
31+
32+
- [ ] **Verify R integration**
33+
```bash
34+
python -c "import rpy2.robjects as ro; ro.r('library(ggplot2)'); print('R integration works!')"
35+
```
36+
Expected: No errors, prints "R integration works!"
37+
38+
## Build Testing
39+
40+
- [ ] **Run tests**
41+
```bash
42+
make test
43+
```
44+
Expected: All tests pass
45+
46+
- [ ] **Quick render test (without notebook execution)**
47+
```bash
48+
make qrender
49+
```
50+
Expected: Generates web/index.html from existing Examples.ipynb
51+
52+
- [ ] **Full build test (with notebook execution)**
53+
```bash
54+
make render
55+
```
56+
Expected:
57+
- Executes notebook successfully
58+
- Generates all plot images in web/img/plots/
59+
- Creates web/index.html
60+
- No errors from any plotting library
61+
62+
## Known Migration Issues to Watch For
63+
64+
### Plotly v5 Changes
65+
- [ ] Verify plotly plots render without authentication
66+
- [ ] Check that Kaleido is used for static image export (not Orca)
67+
- [ ] Ensure all plotly examples in notebook still work
68+
69+
### Altair v5 Changes
70+
- [ ] Verify altair plots render to PNG
71+
- [ ] Confirm altair-saver is not needed (using native rendering)
72+
- [ ] Check selenium/geckodriver works for image export
73+
74+
### plotnine 0.13+ Changes
75+
- [ ] Verify all ggplot-style plots render correctly
76+
- [ ] Check for any deprecation warnings
77+
78+
### statsmodels 0.14+ Changes
79+
- [ ] Verify regression line plots work
80+
- [ ] Check confidence interval calculations
81+
82+
### rpy2 3.5+ Changes
83+
- [ ] Verify %%R magic commands work
84+
- [ ] Check data passing between Python and R
85+
- [ ] Ensure all R ggplot2 examples render
86+
87+
### Selenium 4+ Changes
88+
- [ ] Verify geckodriver compatibility
89+
- [ ] Check headless browser operations
90+
- [ ] Test with xvfb if on Linux
91+
92+
## CI/CD Testing
93+
94+
- [ ] **GitHub Actions workflow**
95+
- Push to a test branch
96+
- Verify all workflow steps complete
97+
- Check Netlify preview deploy works
98+
- Merge to master and verify production deploy
99+
100+
## Performance Checks
101+
102+
- [ ] Note build time for comparison (old vs new)
103+
- [ ] Check generated image file sizes
104+
- [ ] Verify web/index.html loads correctly in browser
105+
106+
## Rollback Plan
107+
108+
If critical issues are found:
109+
1. Keep Travis CI config (.travis.yml) until GitHub Actions is confirmed working
110+
2. Document any notebook cells that fail with new library versions
111+
3. Consider pinning problematic packages to older versions temporarily

0 commit comments

Comments
 (0)