Raw view: Data Science Notebooks CLAUDE.md
No invisible, zero-width, or bidi-control characters detected.
# {{PROJECT_NAME}}
Python data project. Reusable code lives in the package `src/{{PACKAGE_NAME}}/`; notebooks in `notebooks/` call it. Environment managed by **uv** (`uv sync`, `uv run ...`); never `pip install` or `conda`.
## Commands
| Task | Command |
|------|---------|
| Install | `uv sync` |
| Lab | `uv run jupyter lab` |
| Run a notebook headless | `uv run jupyter nbconvert --to notebook --execute notebooks/<name>.ipynb --output /tmp/<name>.ipynb` |
| Tests | `uv run pytest` (tests cover `src/`, not notebooks) |
| Lint / format | `uv run ruff check . && uv run ruff format .` (notebooks included via `extend-include = ["*.ipynb"]`) |
| Strip outputs | `uv run nbstripout notebooks/*.ipynb` |
| Pipeline (DVC) | `uv run dvc repro` (pull inputs first: `uv run dvc pull`) |
Definition of done: `uv run ruff check .`, `uv run pytest` exit 0; any notebook you touched re-executes top to bottom without error via the nbconvert command above; `git diff --stat` shows no output-cell churn.
## Notebooks
- Never edit, add, or delete `outputs` or `execution_count` in a `.ipynb`. A pre-commit `nbstripout` hook clears them; if you must change a notebook, edit `source` cells only and run `uv run nbstripout <file>` before committing.
- Prefer editing notebooks through the `NotebookEdit` tool or by changing the paired `.py` file when `jupytext` pairing is enabled (`notebooks/*.py` with `# %%` cells). Do not hand-edit the JSON.
- Keep notebooks thin: a notebook loads data, calls functions from `{{PACKAGE_NAME}}`, and displays results. Any cell longer than ~15 lines or any function `def` is a signal to move code into `src/` with a test.
- One notebook per question. Name `NN-<topic>.ipynb` with a two-digit prefix for ordering (`01-load.ipynb`, `02-clean.ipynb`). The first markdown cell states the question and the data version used.
- Notebooks must run top to bottom from a fresh kernel. No reliance on out-of-order execution, no `%store`.
- Exploratory scratch work goes in `notebooks/scratch/` which is gitignored.
## Data paths
- All paths come from `{{PACKAGE_NAME}}.paths`: `RAW = DATA_ROOT / "raw"`, `INTERIM`, `PROCESSED`, `MODELS`, `REPORTS`. `DATA_ROOT` reads the `{{DATA_ROOT}}` env var and defaults to `./data`. Never hardcode a path or use `~`.
- `data/raw/` is read-only. Code never writes there; cleaning writes to `data/interim/`, features to `data/processed/`.
- `data/` is gitignored except `data/.gitkeep` and any `*.dvc` pointer files. Never `git add` a `.csv`, `.parquet`, or model binary.
- Prefer Parquet over CSV for anything written by code. Read CSV only for raw inputs.
## Reproducibility
- Every random operation takes an explicit seed. Use `rng = np.random.default_rng({{SEED}})` and pass `rng` down; for libraries that need an int, pass `random_state={{SEED}}`. `torch.manual_seed({{SEED}})` where applicable. The constant lives in `{{PACKAGE_NAME}}.config.SEED`.
- `train_test_split`, shuffles, samplers, and model constructors all receive the seed. A diff that changes a metric without a seed change is a bug.
- Pin everything: commit `uv.lock`. Record the data version (DVC hash or file checksum) in the notebook's first cell and in `reports/*.md`.
- Long-running steps are DVC stages in `dvc.yaml` with declared `deps` and `outs`, not notebook cells. Run with `uv run dvc repro`; add a stage with `uv run dvc stage add -n <name> -d <dep> -o <out> <cmd>`.
- Figures saved to `reports/figures/` with `fig.savefig(path, dpi=150, bbox_inches="tight")` from code, not screenshotted.
## Code in `src/`
- Pure functions over DataFrames in, DataFrames out. Validate schemas at module boundaries with `pandera` (or a plain column-set assertion) so silent column drift fails fast.
- No notebook-only imports (`IPython.display`, `%magic`) inside `src/`.
- Type hints on all public functions. Docstring states input columns and output columns.
- Tests in `tests/` with small synthetic frames built in fixtures; never read from `data/` in a test.
## Do not
- Do not commit notebook outputs, checkpoints (`.ipynb_checkpoints/`), or `__pycache__`.
- Do not `pd.set_option("display.max_rows", None)` or print whole DataFrames in committed notebooks; use `.head()` and `.describe()`.
- Do not download data in a notebook; add a DVC stage or a script under `scripts/` and document the source URL in `data/README.md`.
- Do not change `SEED` to make a result look better.