stat_pairs_backtest/AGENTS.md

# Agent Instructions

## Repository purpose

This repository analyzes test results with Jupyter notebooks and Python or
Bash scripts. Inputs are commonly SQLite databases containing time-series data
and JSON columns, but analyses may use other test-result formats.

Ignore `__SAV__/`. It is unrelated legacy material, is not part of the active
project, and must not be read, edited, moved, or used as a source of conventions
unless the user explicitly requests it.

## Active layout

- `notebooks/`: exploratory and report-oriented Jupyter notebooks.
- `scripts/`: reusable Python and Bash analysis utilities.
- `data/`: local input data. Contents are ignored except for `.gitkeep`.
- `results/`: generated tables, figures, exports, and reports. Contents are
  ignored except for `.gitkeep`.
- `requirements.txt`: Python dependencies needed to reproduce repository work.

Keep reusable logic in `scripts/` and use notebooks to orchestrate analysis,
explain decisions, and present results. Do not create a separate `analysis/`
tree.

## Python environment

The intended virtual environment is `~/.pyenv/python3.12-venv`.

```bash
source ~/.pyenv/python3.12-venv/bin/activate
python -m pip install -r requirements.txt
```

Agents may install packages in this environment when needed. Whenever a package
is installed for repository work, update `requirements.txt` in the same change
with a suitable direct dependency declaration. Use `python -m pip`, not bare
`pip`, in documented commands.

Do not create an in-repository virtual environment unless the user asks for
one.

## Data handling

- Treat files in `data/` as local, potentially large, and potentially
  sensitive.
- Do not commit SQLite databases, raw test results, or generated results.
- Do not modify source data in place. Write transformed data and exports under
  `results/`.
- Use parameterized SQL for values. Do not construct SQL by interpolating
  untrusted data.
- Parse JSON columns defensively and preserve missing, malformed, and unexpected
  values unless the analysis explicitly defines another policy.
- State assumptions about timestamps, time zones, ordering, units, and duplicate
  observations in the notebook or script that relies on them.
- Avoid loading entire databases into memory when a filtered query or chunked
  read is practical.

## Notebook conventions

- A notebook must run from a fresh kernel, top to bottom, without relying on
  hidden interactive state.
- Set random seeds where nondeterminism affects results.
- Keep data paths relative to the repository root and avoid machine-specific
  absolute paths.
- Move logic that is reused or substantial enough to test into `scripts/`.
- Clear cell outputs before committing notebooks. Never commit embedded source
  data, credentials, or bulky generated output.
- Keep concise Markdown context near analyses: purpose, input assumptions,
  method, and interpretation.

## Scripts

- Python scripts should expose reusable functions and use a guarded CLI entry
  point when executable.
- Bash scripts must start with `#!/usr/bin/env bash` and use
  `set -euo pipefail`.
- Prefer explicit CLI arguments over hard-coded paths or parameters.
- Fail with actionable error messages when required data, tables, columns, or
  configuration are missing.

## Verification

Verification should be proportional to the change. At minimum:

- Run `pytest` for Python script changes.
- Add or update tests for reusable parsing, transformation, query, and
  calculation logic.
- Execute changed notebooks from a fresh kernel with `nbmake`.
- Run changed Bash scripts against a safe fixture or exercise their
  non-destructive validation/help path.
- Clear notebook outputs after execution and before committing.

Useful commands:

```bash
python -m pytest
python -m pytest --nbmake notebooks
jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace path/to/notebook.ipynb
```

If verification cannot be run, report exactly what was skipped and why.

## Release rules

- Update `CHANGELOG.md` for every release with the release version, release
  date, Git tag, and a concise summary of notable changes.
- Keep an `Unreleased` section at the top of `CHANGELOG.md` for changes that
  have not been included in a tagged release yet.
- Move relevant entries from `Unreleased` into the dated release section when
  creating a release, and leave `Unreleased` present for future changes.
- Use release headers in `YYYY-MM-DD vMAJOR.MINOR.PATCH` form.
- Use version numbers in `MAJOR.MINOR.PATCH` form. Start this repository at
  `0.0.1`.
- Use Git tags in `vMAJOR.MINOR.PATCH` form, matching the changelog version
  exactly. For example, version `0.0.1` must be tagged as `v0.0.1`.
- Create the Git tag only after the changelog and any release-related version
  changes are complete.
- When the user requests creating a release, treat that as explicit permission
  to commit the release changes, create the matching Git tag, and push both the
  branch and tag.
- Do not push release commits or tags unless the user explicitly requests it.

## Mandatory background review

Changes to Python scripts, Bash scripts, or notebook code cells require approval
from a separate background reviewer agent before the implementing agent may
declare the work complete.

The implementing agent must:

1. Finish the implementation and run the relevant verification.
2. Ask a separate background agent to review the diff for correctness,
   reproducibility, data safety, and test coverage.
3. Address every material finding, rerun affected checks, and request follow-up
   review when the fix materially changes the code.
4. Report the reviewer outcome in the final response.

The reviewer must inspect the actual diff and relevant surrounding files; a
self-review does not satisfy this requirement. Documentation-only,
configuration-only, dependency-only, and ignore-rule-only changes do not
require background approval unless they also alter Python, Bash, or notebook
code cells.

If no background reviewer is available, complete all other work but do not
claim reviewer approval. End the handoff with the exact status:

`review pending`

## Change discipline

- Preserve user changes and avoid unrelated cleanup.
- Do not edit or commit generated files from `data/` or `results/`.
- Do not push or commit unless the user explicitly requests it. The `master`
  branch being unprotected does not imply permission to push directly.
- Keep changes focused and explain any new assumptions or dependencies.