disaster recovery
This commit is contained in:
yasha
@@ -0,0 +1,71 @@
|
||||
### Streamlined Calibration for Baseline LR Gates
|
||||
|
||||
If you’re mainly struggling with mis‑calibrated confidence on your edge‑filtered checks, here’s a **minimal** integration to fix it without heavy lifting.
|
||||
|
||||
---
|
||||
|
||||
#### 1. Add a toggle and holdout in `config.yaml`
|
||||
```yaml
|
||||
baseline:
|
||||
calibration_enabled: true # turn on/off easily
|
||||
calibration_method: "isotonic" # handles multiclass
|
||||
calibration_holdout: 0.2 # 20% of your train split
|
||||
random_state: 42
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 2. Quick split for calibration
|
||||
In your `run_baseline_checks` (before any CI gates):
|
||||
```python
|
||||
# original train/val split
|
||||
X_main, X_val, y_main, y_val = train_test_split(
|
||||
X_pruned, y_labels, test_size=0.2, random_state=seed
|
||||
)
|
||||
|
||||
if self.config['baseline']['calibration_enabled']:
|
||||
X_train, X_cal, y_train, y_cal = train_test_split(
|
||||
X_main, y_main,
|
||||
test_size=self.config['baseline']['calibration_holdout'],
|
||||
random_state=self.config['baseline']['random_state']
|
||||
)
|
||||
else:
|
||||
X_train, y_train = X_main, y_main
|
||||
X_cal, y_cal = None, None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 3. Fit an isotonic calibrator only when needed
|
||||
```python
|
||||
# train raw LR
|
||||
lr = LogisticRegression(...).fit(X_train, y_train)
|
||||
|
||||
if X_cal is not None:
|
||||
from sklearn.calibration import CalibratedClassifierCV
|
||||
calibrator = CalibratedClassifierCV(lr, method='isotonic', cv='prefit')
|
||||
calibrator.fit(X_cal, y_cal)
|
||||
else:
|
||||
calibrator = lr
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 4. Use calibrated probabilities in your gates
|
||||
Replace all `lr.predict_proba(X)` calls with:
|
||||
```python
|
||||
probs = calibrator.predict_proba(X)
|
||||
# binary: edge = |probs[:,1] - 0.5|
|
||||
# ternary: edge = max(probs, axis=1) - 1/3
|
||||
```
|
||||
Then run your existing CI lower‑bound checks as usual.
|
||||
|
||||
---
|
||||
|
||||
#### 5. (Optional) Skip persistence
|
||||
For a quick fix you can skip saving/loading the calibrator—just build and use it in the same process.
|
||||
|
||||
---
|
||||
|
||||
With these five steps, you’ll correct your edge‑confidence estimates with minimal code and configuration. If your gates then pass, proceed to GRU training; if they still fail, the issue is likely weak features rather than calibration.
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
Below is a consolidated set of revision instructions — and the key code snippets you’ll need — to switch your GRU/SAC pipeline to supervise **log‑returns** (so your “ret” and “gauss_params” heads are targeting log‑return), and to wire everything end‑to‑end so you get a clean 55 %+ edge before SAC.
|
||||
|
||||
---
|
||||
|
||||
## 🛠 Revision Playbook
|
||||
|
||||
1. **Compute forward log‐returns in your data pipeline**
|
||||
In `TradingPipeline.define_labels_and_align` (or wherever you set up `df`):
|
||||
|
||||
```python
|
||||
# Replace any raw-return calculation with log‐return
|
||||
N = config['gru']['prediction_horizon']
|
||||
df['fwd_log_ret'] = np.log(df['close'].shift(-N) / df['close'])
|
||||
df['direction_label'] = (df['fwd_log_ret'] > 0).astype(int)
|
||||
# If you do ternary:
|
||||
flat_thr = config['gru']['flat_sigma_multiplier'] * df['fwd_log_ret'].rolling(…)
|
||||
df['dir3_label'] = pd.cut(df['fwd_log_ret'],
|
||||
bins=[-np.inf, -flat_thr, flat_thr, np.inf],
|
||||
labels=[0,1,2]).astype(int)
|
||||
```
|
||||
|
||||
2. **Align your targets**
|
||||
Drop the last N rows so `fwd_log_ret` has no NaNs:
|
||||
|
||||
```python
|
||||
df = df.iloc[:-N]
|
||||
```
|
||||
|
||||
3. **Pass log‐return into both heads**
|
||||
When you build your sequences and target dicts:
|
||||
|
||||
```python
|
||||
y_ret_seq = ... # shape (n_seq, 1) from fwd_log_ret
|
||||
y_dir3_seq = ... # one‑hot from dir3_label
|
||||
|
||||
y_train = {'mu': y_ret_seq, # Huber head
|
||||
'gauss_params': y_ret_seq, # NLL head uses same target
|
||||
'dir3': y_dir3_seq} # classification head
|
||||
```
|
||||
|
||||
4. **Update your GRU builder to match**
|
||||
Make sure your v3 model has exactly three outputs:
|
||||
```python
|
||||
model = Model(inputs, outputs=[mu_output, gauss_params_output, dir3_output])
|
||||
model.compile(
|
||||
optimizer=Adam(lr),
|
||||
loss={
|
||||
'mu': Huber(delta),
|
||||
'gauss_params': gaussian_nll,
|
||||
'dir3': categorical_focal_loss
|
||||
},
|
||||
loss_weights={'mu':1.0, 'gauss_params':0.2, 'dir3':0.4},
|
||||
metrics={'dir3':'accuracy'}
|
||||
)
|
||||
```
|
||||
|
||||
5. **Train with the new targets dict**
|
||||
In `GRUModelHandler.train(...)`, replace your fit call with:
|
||||
|
||||
```python
|
||||
history = model.fit(
|
||||
X_train_seq,
|
||||
y_train_dict,
|
||||
validation_data=(X_val_seq, y_val_dict),
|
||||
…callbacks…
|
||||
)
|
||||
```
|
||||
|
||||
6. **Calibrate on the “dir3” softmax outputs**
|
||||
Your calibrator (Temp/Vector) must now consume the 3‑class logits or probabilities:
|
||||
|
||||
```python
|
||||
raw_logits = handler.predict_logits(X_val_seq)
|
||||
calibrator.fit(raw_logits, y_val_dir3)
|
||||
```
|
||||
|
||||
7. **Feed SAC the log‐return μ and σ**
|
||||
In your `TradingEnv`, when you construct the state:
|
||||
|
||||
```python
|
||||
mu, log_sigma, probs = gru_handler.predict(X_step)
|
||||
sigma = np.exp(log_sigma)
|
||||
edge = 2 * calibrated_p_up - 1 # if binary
|
||||
z_score = np.abs(mu) / sigma
|
||||
state = [mu, sigma, edge, z_score, prev_position]
|
||||
```
|
||||
|
||||
8. **Re‐run baseline check on log‐returns**
|
||||
Your logistic baseline in `run_baseline_checks` should now be trained on `X_train_pruned` vs `y_dir3_label` (or binary), ensuring the CI ≥ 0.52 before you even build your GRU.
|
||||
|
||||
9. **Validate end‑to‑end edge**
|
||||
After these changes, you should see:
|
||||
- Baseline logistic CI LB ≥ 0.52
|
||||
- GRU “edge” hit‑rate ≥ 0.55 on validation
|
||||
- SAC backtest hitting meaningful Sharpe/Win‑rate gates
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Example Code Snippets
|
||||
|
||||
### 1) Gaussian NLL stays the same:
|
||||
|
||||
```python
|
||||
@saving.register_keras_serializable(package='GRU')
|
||||
def gaussian_nll(y_true, y_pred):
|
||||
mu, log_sigma = tf.split(y_pred, 2, axis=-1)
|
||||
y_true = tf.reshape(y_true, tf.shape(mu))
|
||||
inv_var = tf.exp(-2*log_sigma)
|
||||
return tf.reduce_mean(0.5 * inv_var * tf.square(y_true-mu) + log_sigma)
|
||||
```
|
||||
|
||||
### 2) Build & compile v3 model:
|
||||
|
||||
```python
|
||||
def build_gru_model_v3(...):
|
||||
inp = layers.Input((lookback, n_features))
|
||||
x = layers.GRU(gru_units, return_sequences=True)(inp)
|
||||
x = layers.LayerNormalization()(x)
|
||||
if attention_units>0:
|
||||
x = layers.MultiHeadAttention(...)(x,x)
|
||||
x = layers.GlobalAveragePooling1D()(x)
|
||||
|
||||
gauss = layers.Dense(2, name='gauss_params')(x)
|
||||
mu = layers.Lambda(lambda z: z[:,0:1], name='mu')(gauss)
|
||||
dir3_logits = layers.Dense(3, name='dir3_logits')(x)
|
||||
dir3 = layers.Activation('softmax', name='dir3')(dir3_logits)
|
||||
|
||||
model = Model(inp, [mu, gauss, dir3])
|
||||
model.compile(
|
||||
optimizer=Adam(lr),
|
||||
loss={'mu':Huber(delta),
|
||||
'gauss_params':gaussian_nll,
|
||||
'dir3':categorical_focal_loss},
|
||||
loss_weights={'mu':1.0,'gauss_params':0.2,'dir3':0.4},
|
||||
metrics={'dir3':'accuracy'}
|
||||
)
|
||||
return model
|
||||
```
|
||||
|
||||
### 3) Fitting in your handler:
|
||||
|
||||
```python
|
||||
history = self.model.fit(
|
||||
X_train_seq, y_train_dict,
|
||||
validation_data=(X_val_seq, y_val_dict),
|
||||
epochs=max_epochs,
|
||||
batch_size=batch_size,
|
||||
callbacks=[early_stop, csv_logger, TqdmCallback()]
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Why these changes boost edge?
|
||||
|
||||
- **Log‐returns** stabilize variance & symmetrize up/down moves.
|
||||
- **NLL + Huber on log‐return** gives the model both distributional uncertainty (σ) and a robust error measure.
|
||||
- **Proper softmax head** on three classes (up/flat/down) cleans up classification.
|
||||
- **Calibration + optimized edge threshold** ensures your SAC agent only sees high‐confidence signals (edge≥thr).
|
||||
|
||||
Together, this gets your baseline GRU above 55 % “edge” on validation, so the SAC agent can then learn a meaningful sizing policy rather than fight noise.
|
||||
|
||||
Let me know if you need any further refinements!
|
||||
@@ -0,0 +1,187 @@
|
||||
## Revision Instructions for AI Dev‑Agent
|
||||
Implement end‑to‑end missing‑bar handling in GRU and SAC. Apply the steps below in sequence, with small PRs and CI green at each stage.
|
||||
|
||||
---
|
||||
|
||||
### 1 | Config updates
|
||||
|
||||
**File:** `config.yaml`
|
||||
Add under `data` and create new sections for `gru` and `sac`:
|
||||
|
||||
```yaml
|
||||
data:
|
||||
bar_frequency: "1T"
|
||||
missing:
|
||||
strategy: "neutral" # drop | neutral | ffill | interpolate
|
||||
max_gap: 5 # max consecutive missing bars allowed
|
||||
interpolate:
|
||||
method: "linear"
|
||||
limit: 10
|
||||
|
||||
gru:
|
||||
drop_imputed_sequences: true # drop any sequence containing imputed bars
|
||||
|
||||
sac:
|
||||
imputed_handling: "hold" # hold | skip | penalty
|
||||
action_penalty: 0.05 # used if imputed_handling=penalty
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2 | Detect & fill missing bars
|
||||
|
||||
**File:** `src/data_loader.py`
|
||||
|
||||
1. **Import** at top:
|
||||
```python
|
||||
import pandas as pd
|
||||
from .io_manager import IOManager
|
||||
```
|
||||
|
||||
2. **Implement** `find_missing_bars(df, freq)` and `_consecutive_gaps` helpers.
|
||||
|
||||
3. **Implement** `report_missing(missing, cfg, io, logger)` as described.
|
||||
|
||||
4. **Implement** `fill_missing_bars(df, cfg, io, logger)`:
|
||||
- Detect missing timestamps.
|
||||
- Call `report_missing`.
|
||||
- Reindex to full date_range.
|
||||
- Apply `strategy`:
|
||||
- `drop`: return original df.
|
||||
- `neutral`: ffill close, set open=high=low=close, volume=0.
|
||||
- `ffill`: `df_full.ffill().bfill()`.
|
||||
- `interpolate`: use `df_full.interpolate(...)`.
|
||||
- **After filling**, add column:
|
||||
```python
|
||||
df['bar_imputed'] = df.index.isin(missing)
|
||||
```
|
||||
- **Error** if longest gap > `cfg.data.missing.max_gap`.
|
||||
|
||||
5. **Integrate** in `TradingPipeline.load_and_preprocess_data` **before** feature engineering:
|
||||
```python
|
||||
df = fill_missing_bars(df, self.cfg, io, logger)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3 | Sequence creation respects imputed bars
|
||||
|
||||
**File:** `src/trading_pipeline.py`
|
||||
|
||||
1. In `create_sequences`, after building `X_seq` and `y_seq`, **build** `mask_seq` of shape `(n, lookback)` from `df['bar_imputed']`.
|
||||
|
||||
2. **Conditionally drop** sequences:
|
||||
```python
|
||||
if self.cfg.gru.drop_imputed_sequences:
|
||||
valid = ~mask_seq.any(axis=1)
|
||||
X_seq = X_seq[valid]; y_seq = y_seq[valid]
|
||||
```
|
||||
3. **Log**:
|
||||
```python
|
||||
logger.info(f"Generated {orig_n} sequences, dropped {orig_n - X_seq.shape[0]} with imputed bars")
|
||||
```
|
||||
4. **Include** `bar_imputed` as a feature column in `minimal_whitelist`.
|
||||
|
||||
---
|
||||
|
||||
### 4 | GRU model input channel
|
||||
|
||||
**File:** `src/model_gru_v3.py` (or `model_gru.py` if v3)
|
||||
|
||||
1. **Update input shape**: increase `n_features` by 1 to include `bar_imputed`.
|
||||
|
||||
2. **No further architectural change**; the model now sees the imputed‑flag channel.
|
||||
|
||||
---
|
||||
|
||||
### 5 | SAC environment handles imputed bars
|
||||
|
||||
**File:** `src/trading_env.py`
|
||||
|
||||
1. **Read** `bar_imputed` into `self.bar_imputed` aligned with your sequences.
|
||||
|
||||
2. **In `step(action)`**, at the top:
|
||||
```python
|
||||
imputed = self.bar_imputed[self.current_step]
|
||||
if imputed:
|
||||
mode = self.cfg.sac.imputed_handling
|
||||
if mode == "skip":
|
||||
self.current_step += 1
|
||||
return next_state, 0.0, False, {}
|
||||
if mode == "hold":
|
||||
action = self.position
|
||||
if mode == "penalty":
|
||||
reward = - self.cfg.sac.action_penalty * (action - self.position)**2
|
||||
self._update_position(action)
|
||||
self.current_step += 1
|
||||
return next_state, reward, False, {}
|
||||
# existing normal step follows
|
||||
```
|
||||
|
||||
3. **Ensure** imputed transitions are added to replay buffer only when `mode` ≠ `skip`.
|
||||
|
||||
4. **Log**:
|
||||
```python
|
||||
logger.debug(f"SAC step {self.current_step} on imputed bar: handling={mode}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6 | Logging & artefacts
|
||||
|
||||
1. **Data load** warning:
|
||||
```
|
||||
WARNING Detected {total} missing bars, longest gap {longest}; applied strategy={strategy}
|
||||
```
|
||||
|
||||
2. **Sequence creation** info:
|
||||
```
|
||||
INFO Generated {orig_n} sequences, dropped {dropped} with imputed bars
|
||||
```
|
||||
|
||||
3. **SAC training** debug:
|
||||
```
|
||||
DEBUG SAC on imputed bar at step {step}: handling={mode}
|
||||
```
|
||||
|
||||
4. **Report** saved under `results/<run_id>/`:
|
||||
- `missing_bars_summary.json`
|
||||
- `imputed_sequence_summary.json` with counts.
|
||||
- `sac_imputed_transitions.csv` (optional detailed log).
|
||||
|
||||
---
|
||||
|
||||
### 7 | Unit tests
|
||||
|
||||
**Files:** `tests/test_data_loader.py`, `tests/test_sequence_creation.py`, `tests/test_trading_env.py`
|
||||
|
||||
1. **`test_data_loader.py`**:
|
||||
- Synthetic gappy DataFrame → assert `bar_imputed` flags and each strategy’s output.
|
||||
|
||||
2. **`test_sequence_creation.py`**:
|
||||
- Build toy DataFrame with `bar_imputed`; assert sequences dropped when `drop_imputed_sequences=True`.
|
||||
|
||||
3. **`test_trading_env.py`**:
|
||||
- Create `TradingEnv` with known imputed steps; for each `imputed_handling` mode assert `step()` behavior:
|
||||
- `skip` moves without adding to buffer;
|
||||
- `hold` returns same position;
|
||||
- `penalty` returns negative reward equal to penalty formula.
|
||||
|
||||
---
|
||||
|
||||
### 8 | Documentation
|
||||
|
||||
1. **README.md** → add **Data Quality** section describing missing‑bar handling, config keys, and recommended defaults.
|
||||
|
||||
2. **docs/v3_changelog.md** → note new missing‑bar feature and cfg flags.
|
||||
|
||||
---
|
||||
|
||||
**Roll‑out Plan:**
|
||||
|
||||
- **PR 1:** Config + data_loader missing‑bar detection & fill + tests.
|
||||
- **PR 2:** Sequence creation & GRU channel update + tests.
|
||||
- **PR 3:** SAC env updates + tests.
|
||||
- **PR 4:** Logging/artefacts + docs.
|
||||
|
||||
Merge each after CI passes.
|
||||
@@ -0,0 +1,140 @@
|
||||
## **Revision Document – v3 Output Contract & Figure Specifications**
|
||||
This single guide merges **I/O plumbing**, **logging**, **CI hooks**, **artefact paths**, and **figure design** into one actionable playbook.
|
||||
Apply the steps **in order**, submitting small PRs so CI remains green throughout.
|
||||
|
||||
---
|
||||
|
||||
### 0 ▪ Foundations
|
||||
|
||||
| Step | File(s) | Action |
|
||||
|------|---------|--------|
|
||||
| 0.1 | **`config.yaml`** | Add: ```yaml base_dirs: {results: results, models: models, logs: logs} output: {figure_dpi: 150, figure_size: [16, 9], log_level: INFO}``` |
|
||||
| 0.2 | `src/utils/run_id.py` | `make_run_id()` → `"20250418_152310_ab12cd"` (timestamp + short git‑hash). |
|
||||
| 0.3 | `src/__init__.py` | Expose `__version__`, `GIT_SHA`, `BUILD_DATE`. |
|
||||
|
||||
---
|
||||
|
||||
### 1 ▪ Core I/O & Logging
|
||||
|
||||
| File | Content |
|
||||
|------|---------|
|
||||
| **`src/io_manager.py`** | `IOManager(cfg, run_id)` <br>• `path(section, name)`: returns full path under `results|models|logs|figures`.<br>• `save_json`, `save_df` (CSV ≤ 100 MB else Parquet), `save_figure` (uses cfg dpi/size). |
|
||||
| **`src/logger_setup.py`** | `setup_logger(cfg, run_id, io)` with colourised console (INFO) + rotating file handler (DEBUG) in `logs/<run_id>/`. |
|
||||
|
||||
**`run.py` entry banner**
|
||||
|
||||
```python
|
||||
run_id = make_run_id()
|
||||
cfg = load_config(args.config)
|
||||
io = IOManager(cfg, run_id)
|
||||
logger = setup_logger(cfg, run_id, io)
|
||||
logger.info(f"GRU‑SAC v{__version__} | commit {GIT_SHA} | run {run_id}")
|
||||
logger.info(f"Loaded config file: {args.config}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2 ▪ Stage Outputs
|
||||
|
||||
| Stage | Implementation notes | Artefacts |
|
||||
|-------|---------------------|-----------|
|
||||
| **Data load & preprocess** | After sampling/NaN purge save: <br>`io.save_json(summary, "preprocess_summary")`<br>`io.save_df(df.head(20), "head_preprocessed")` | `results/<run_id>/preprocess_summary.txt`<br>`head_preprocessed.csv` |
|
||||
| **Feature engineering** | Generate correlation heat‑map (see figure table) → `io.save_figure(...,"feature_corr_heatmap")` | 〃 |
|
||||
| **Label generation** | Log distribution; produce histogram figure. | `label_histogram.png` |
|
||||
| **Baseline 1 & 2** | Consolidate in `baseline_checker.py`; each returns dict with accuracy, CI etc. <br>`io.save_json(report,"baseline1_report")` (and 2). | `baseline1_report.txt / baseline2_report.txt` |
|
||||
| **Feature whitelist** | Save JSON to `models/<run_id>/final_whitelist_<run_id>.json`. | — |
|
||||
| **GRU training** | Use Keras CSVLogger to `logs/<run_id>/gru_history.csv`; after training plot learning curve. | `gru_learning_curve.png` + `.keras` model |
|
||||
| **Calibration (Vector)** | Save `calibrator_vec_<run_id>.npy`; plot reliability curve. | `reliability_curve_val_<run_id>.png` |
|
||||
| **SAC training** | Write `episode_rewards.csv`, plot reward curve, save final agent under `models/sac_train_<run_id>/`. | `sac_reward_plot.png` |
|
||||
| **Back‑test** | Save step‑level CSV, metrics JSON, summary figure. | `backtest_results_<run_id>.csv`<br>`performance_metrics_<run_id>.txt`<br>`backtest_summary_<run_id>.png` |
|
||||
|
||||
---
|
||||
|
||||
### 3 ▪ Figure Specifications
|
||||
|
||||
| File | Visualises | Layout / Details |
|
||||
|------|-------------|------------------|
|
||||
| **feature_corr_heatmap.png** | Pearson correlation of engineered features (pre‑prune). | Square heat‑map, features sorted by |ρ| vs target; diverging palette centred at 0; annotate |ρ| > 0.5; colour‑bar. |
|
||||
| **label_histogram.png** | Direction‑label class mix (train split). | Bar chart: Down / Flat / Up (binary shows two). Percentages on bar tops; title shows ε value. |
|
||||
| **gru_learning_curve.png** | GRU training progress. | 3 stacked panes: total loss (log‑y), val dir3 accuracy, vertical dashed “early‑stop”; share epoch‑axis. |
|
||||
| **reliability_curve_val_*.png** | Calibration quality post‑Vector scaling. | Left 70 %: reliability diagram (10 equal‑freq bins). Right 30 %: histogram of predicted p_up. Title shows ECE & Brier. |
|
||||
| **sac_reward_plot.png** | Offline SAC learning curve. | Smoothed episode reward (EMA 0.2) vs steps; action‑variance on twin y‑axis; checkpoint ticks. |
|
||||
| **backtest_summary_*.png** | Live back‑test overview. | 3 stacked plots:<br>1) Price line + blue/red background for edge ≥ 0.1.<br>2) Position size step‑graph.<br>3) Equity curve with shaded draw‑downs; textbox shows Sharpe & Max DD. |
|
||||
|
||||
_All figs_: 16 × 9 in, 150 DPI, `plt.tight_layout()`, footer `"© GRU‑SAC v3"` right‑bottom.
|
||||
|
||||
---
|
||||
|
||||
### 4 ▪ Unit Tests
|
||||
|
||||
* `tests/test_output_contract.py`
|
||||
* Run mini‑pipeline (`tests/smoke.yaml`), assert each required file exists > 2 KB.
|
||||
* Validate JSON keys (`accuracy`, `ci_lower` etc.).
|
||||
* `assert_any_close(softmax(logits), probs)` for logits view.
|
||||
|
||||
---
|
||||
|
||||
### 5 ▪ CI Workflow (`.github/workflows/pipeline.yml`)
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
build-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with: {python-version: "3.10"}
|
||||
- run: pip install -r requirements.txt
|
||||
- run: black --check .
|
||||
- run: ruff .
|
||||
- run: pytest -q
|
||||
- name: Smoke e2e
|
||||
run: python run.py --config tests/smoke.yaml
|
||||
- name: Upload artefacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: run-${{ github.sha }}
|
||||
path: |
|
||||
results/*/*
|
||||
logs/*/*
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6 ▪ Documentation Updates
|
||||
|
||||
* **`README.md`** → new *Outputs* section reproducing the artefact table.
|
||||
* **`docs/v3_changelog.md`** → one‑pager summarising v3 versus v2 differences (labels, calibration, outputs).
|
||||
|
||||
---
|
||||
|
||||
### 7 ▪ Roll‑out Plan (5‑PR cadence)
|
||||
|
||||
1. **PR #1** – run‑id, IOManager, logger, CI log upload.
|
||||
2. **PR #2** – data & feature stage outputs + tests.
|
||||
3. **PR #3** – GRU training outputs + calibration figure.
|
||||
4. **PR #4** – SAC & back‑test outputs, reward & summary figs.
|
||||
5. **PR #5** – docs & README refresh.
|
||||
|
||||
Tag `v3.0.0` after PR #5 passes.
|
||||
|
||||
---
|
||||
|
||||
### 8 ▪ Success Criteria for CI
|
||||
|
||||
Fail the pipeline when **any** occurs:
|
||||
|
||||
* `baseline1_report.txt` CI‑LB < 0.52
|
||||
* `edge_filtered_accuracy` (val) < 0.60
|
||||
* Back‑test Sharpe < 1.2 or Max DD > 15 %
|
||||
|
||||
---
|
||||
|
||||
Implementing this **single integrated revision** provides:
|
||||
|
||||
* **Deterministic artefact paths** for every run.
|
||||
* **Rich, shareable figures** for quick diagnostics.
|
||||
* **Audit‑ready logs/reports** for research traceability.
|
||||
|
||||
Merge each step once CI is green; you’ll have a reproducible, fully instrumented pipeline ready for iterative accuracy pushes toward the 65 % target.
|
||||
@@ -0,0 +1,55 @@
|
||||
Refactoring Plan for trading_pipeline.py
|
||||
=======================================
|
||||
|
||||
Goal: Break down the large TradingPipeline class into smaller, dedicated modules for better maintainability and readability, while minimizing disruption to the existing E2E system.
|
||||
|
||||
Strategy:
|
||||
|
||||
1. **Keep `TradingPipeline` as the Orchestrator:**
|
||||
* The main `TradingPipeline` class in `trading_pipeline.py` remains.
|
||||
* Responsibilities:
|
||||
* Load configuration (`__init__`).
|
||||
* Initialize core components (`DataLoader`, `FeatureEngineer`, etc.).
|
||||
* Manage overall state (instance variables like `df_raw`, `gru_model`, etc.).
|
||||
* Run the main execution flow (`execute`), including the walk-forward loop.
|
||||
* Call functions from new stage-specific modules.
|
||||
|
||||
2. **Create Stage-Specific Modules:**
|
||||
* Create a new sub-directory: `src/pipeline_stages`.
|
||||
* Move stage-specific logic from `TradingPipeline` methods into functions within these modules.
|
||||
* Proposed Modules:
|
||||
* `src/pipeline_stages/data_processing.py`: Loading, feature engineering, label generation, alignment, splitting.
|
||||
* `src/pipeline_stages/feature_processing.py`: Scaling, feature selection, pruning.
|
||||
* `src/pipeline_stages/sequence_creation.py`: GRU sequence creation.
|
||||
* `src/pipeline_stages/modelling.py`: GRU/SAC training/loading, calibration, SAC aggregation.
|
||||
* `src/pipeline_stages/evaluation.py`: Baseline checks, backtesting, results saving, metric aggregation, final decision.
|
||||
|
||||
3. **Refactor `TradingPipeline` Methods:**
|
||||
* Simplify existing methods in `TradingPipeline` (e.g., `load_and_preprocess_data`, `engineer_features`).
|
||||
* These methods will now primarily:
|
||||
* Import the corresponding function from the `pipeline_stages` module.
|
||||
* Call the imported function, passing necessary data and components (`config`, `data_loader`, `io`, state variables).
|
||||
* Receive results and update the `TradingPipeline` instance's state.
|
||||
|
||||
4. **Data Flow:**
|
||||
* Emphasize explicit data passing via function arguments and return values between stages.
|
||||
* The `TradingPipeline.execute` method orchestrates this flow.
|
||||
* State required across multiple stages/folds remains as `TradingPipeline` instance attributes.
|
||||
|
||||
5. **Dependencies:**
|
||||
* Pass `config`, `io`, and component instances (`DataLoader`, `FeatureEngineer`, etc.) as arguments to the stage functions that need them.
|
||||
|
||||
Benefits:
|
||||
|
||||
* **Readability:** `trading_pipeline.py` becomes a clearer orchestrator.
|
||||
* **Maintainability:** Easier to isolate and modify specific stages.
|
||||
* **Testability:** Stage functions are potentially easier to unit test.
|
||||
* **Reduced Risk:** Focuses on moving logic, minimizing E2E breakage compared to a full rewrite.
|
||||
|
||||
Implementation Steps:
|
||||
|
||||
1. Create the `src/pipeline_stages` directory and module files.
|
||||
2. Incrementally move logic for each stage into the corresponding module's functions.
|
||||
3. Update `TradingPipeline` methods to import and call these new functions.
|
||||
4. Adjust imports and function signatures as needed.
|
||||
5. Proceed stage by stage, verifying structure and data flow.
|
||||
@@ -0,0 +1,55 @@
|
||||
Below is a consolidated list of every “validation gate” we enforce in the pipeline—split by the **GRU** (prediction) stage and the **SAC** (position‑sizing) stage. Each check either **aborts** the run (hard‐fail) or **warns** you that a non‑critical gate didn’t clear.
|
||||
|
||||
---
|
||||
|
||||
## GRU‑stage Validation Gates
|
||||
|
||||
| Gate # | Check | Data used | Threshold | Action on Fail |
|
||||
|--------|----------------------------------------------------------------|-----------------------|------------------|--------------------|
|
||||
| **G1** | **Raw binary LR** (Internal split) | train 80/20 split | CI LB ≥ `binary_ci_lb` (0.52) | **Abort** |
|
||||
| **G2** | **Raw binary RF** (Internal split, optional) | train 80/20 split | CI LB ≥ `binary_rf_ci_lb` (0.54) | **Abort** (if enabled) |
|
||||
| **G3** | **Raw ternary LR** (Internal split, if `use_ternary`) | train 80/20 split | CI LB ≥ `ternary_ci_lb` (0.40) | **Warn** |
|
||||
| **G4** | **Raw ternary RF** (Internal split, optional) | train 80/20 split | CI LB ≥ `ternary_rf_ci_lb` (0.42) | **Warn** |
|
||||
| **G5** | **Forward‑fold binary LR** (True OOS, test fold t+1) | fold’s test set | CI LB ≥ `forward_ci_lb` (0.52) | **Abort** |
|
||||
| **G6** | **Feature‑selection re‑baseline** (post‑prune binary LR) | pruned train 80/20 | CI LB ≥ `binary_ci_lb` (0.52) | **Abort** |
|
||||
| **G7** | **Calibration check** (edge‑filtered p_cal on val split) | val split | CI LB ≥ `calibration_ci_lb` (0.55) | **Abort** |
|
||||
|
||||
> **Notes:**
|
||||
> • G1 catches “no predictive signal” cheaply.
|
||||
> • G5 ensures it actually *generalises* forward.
|
||||
> • G7 makes sure your calibrated probabilities have real edge before SAC ever sees them.
|
||||
|
||||
---
|
||||
|
||||
## SAC‑stage Validation Gates
|
||||
|
||||
| Gate # | Check | Data used | Threshold | Action on Fail |
|
||||
|---------|-------------------------------------------------------------|-----------------------|---------------------|----------------------|
|
||||
| **G8** | **Edge‑filtered binary LR** | val split probabilities | CI LB ≥ `edge_binary_ci_lb` (0.60) | **Abort** |
|
||||
| **G9** | **Edge‑filtered binary RF** | val split probabilities | CI LB ≥ `edge_binary_rf_ci_lb` (0.62) | **Abort** |
|
||||
| **G10** | **Edge‑filtered ternary LR** (if `use_ternary`) | val split p_cat[:,2]–p_cat[:,0] | CI LB ≥ `edge_ternary_ci_lb` (0.57) | **Warn** |
|
||||
| **G11** | **Edge‑filtered ternary RF** (if `use_ternary`) | val split p_cat[:,2]–p_cat[:,0] | CI LB ≥ `edge_ternary_rf_ci_lb` (0.58) | **Warn** |
|
||||
| **G12** | **Backtest performance** (Sharpe / Max DD on test fold) | aggregated test folds | Sharpe ≥ `backtest.sharpe_lb` (1.2)<br>Max DD ≤ `backtest.max_dd_ub` (15 %) | **Abort** if violated|
|
||||
|
||||
> **Notes:**
|
||||
> • G8–G9 gate the **high‐confidence edge** you feed into SAC. If they fail, SAC will only ever go all‑in/flat, so we abort.
|
||||
> • G10–G11 warn you if your flat/no‑move sizing is shaky—SAC can still run, but you’ll get a console warning suggesting you tweak your flat thresholds or features.
|
||||
> • G12 validates final **live‐like** performance; if you can’t hit the Sharpe/Max DD targets on the unseen test folds, the entire run is considered a no‑go.
|
||||
|
||||
---
|
||||
|
||||
### What to do on each pattern
|
||||
|
||||
1. **Any GRU‑abort gate (G1, G2, G5, G6, G7) fails** →
|
||||
**Stop** before training. Improve features, horizon, calibration settings, or prune strategy.
|
||||
|
||||
2. **GRU passes but SAC‑binary edge gates (G8/G9) fail** →
|
||||
**Stop** before SAC training. Means your probabilities have no reliable high‑confidence edge—tweak calibration threshold or retrain GRU.
|
||||
|
||||
3. **GRU & SAC‑binary gates pass, but SAC‑ternary edge gates (G10/G11) warn** →
|
||||
**Proceed** with a warning: consider adding flat‑specific features or raising the `edge_threshold`.
|
||||
|
||||
4. **All gates pass** →
|
||||
Full pipeline runs to completion: GRU training, SAC training, backtest, resulting in models, logs, and performance reports.
|
||||
|
||||
By strictly enforcing these gates, you ensure every GRU and SAC model you train has demonstrable, forward‑tested edge—maximizing your chances of hitting that 65 % directional target in live trading.
|
||||
@@ -0,0 +1,124 @@
|
||||
1. Nested Cross-Validation (for GRU Hyperparameter Tuning)
|
||||
Goal: To tune GRU hyperparameters (like gru_units, learning_rate, etc.) robustly for each main walk-forward fold, using only the training data allocated to that fold. This prevents hyperparameters from being influenced by data that will later appear in the fold's validation or test set.
|
||||
Current Implementation: The hyperparameter_tuning.gru.sweep_enabled flag exists, but the tuning logic isn't currently nested within the fold processing loop in train_or_load_gru_fold.
|
||||
Implementation Strategy:
|
||||
Modify train_or_load_gru_fold (in gru_sac_predictor/src/pipeline_stages/modelling.py): This is the function responsible for training or loading the GRU for a specific outer walk-forward fold.
|
||||
Check sweep_enabled: Inside this function, right before the actual GRU training would normally occur (i.e., if config['gru']['train_gru'] is true and a model isn't being loaded), check if config['hyperparameter_tuning']['gru']['sweep_enabled'] is also true.
|
||||
Inner CV Loop: If sweep is enabled:
|
||||
Data: Use the X_train_seq and y_train_seq_dict passed into this function (these represent the training data for the current outer fold).
|
||||
Inner Splits: Use a time-series-appropriate splitter (like sklearn.model_selection.TimeSeriesSplit) on the sequence indices (train_indices_new if returned, otherwise derive from X_train_seq) to create, say, 3 or 5 inner train/validation splits within the outer fold's training data.
|
||||
Optuna Study: Create a new Optuna study (or similar hyperparameter optimization framework) specific to this outer fold.
|
||||
Objective Function: Define an Optuna objective function that takes a trial object:
|
||||
It suggests hyperparameters based on config['hyperparameter_tuning']['gru']['search_space'].
|
||||
It iterates through the inner CV splits. For each inner split:
|
||||
Instantiate a temporary GRUModelHandler (or just the model) with the trial's hyperparameters.
|
||||
Train the model on the inner training data slice.
|
||||
Evaluate it on the inner validation data slice (e.g., calculate val_loss).
|
||||
Return the average performance (e.g., average val_loss) across the inner splits.
|
||||
Run Study: Execute study.optimize with the objective function and n_trials from the config.
|
||||
Best Parameters: Retrieve the study.best_params after optimization.
|
||||
Final Fold Training: Instantiate the GRUModelHandler (gru_handler passed into the function) or build the GRU model using these best_params. Train this single, final model for the outer fold on the entire X_train_seq and y_train_seq_dict.
|
||||
Return: Return this optimally tuned GRU model and handler for the outer fold to proceed.
|
||||
Configuration:
|
||||
The existing hyperparameter_tuning.gru section is mostly sufficient.
|
||||
You might add a key like inner_cv_splits: 3 to control the inner loop.
|
||||
Considerations: This significantly increases computation time, as n_trials * inner_cv_splits models are trained per outer fold.
|
||||
|
||||
2. Gap and Regime-Aware Folds
|
||||
Here’s a minimal “wrapper” you can drop around your existing `_generate_walk_forward_folds` to get both gap‑aware **and** regime‑aware filtering, without rewriting your core logic:
|
||||
|
||||
```python
|
||||
def generate_filtered_folds(df, config):
|
||||
# 1) Tag regimes once, right after loading & feature‐engineering the full dataset
|
||||
if config['walk_forward']['regime']['enabled']:
|
||||
df = add_regime_tags(
|
||||
df,
|
||||
indicator=config['walk_forward']['regime']['indicator'],
|
||||
window=config['walk_forward']['regime']['indicator_params']['window'],
|
||||
quantiles=config['walk_forward']['regime']['quantiles']
|
||||
)
|
||||
min_pct = config['walk_forward']['regime']['min_regime_representation_pct']
|
||||
|
||||
# 2) Split into contiguous chunks on data gaps
|
||||
chunks = split_into_contiguous_chunks(
|
||||
df,
|
||||
config['walk_forward']['gap_threshold_minutes']
|
||||
)
|
||||
|
||||
# 3) For each chunk, run your normal fold‐generator, then filter by regime
|
||||
for chunk_start, chunk_end in chunks:
|
||||
df_chunk = df.loc[chunk_start:chunk_end]
|
||||
# skip tiny chunks
|
||||
if (chunk_end - chunk_start).days < config['walk_forward'].get('min_chunk_days', 1):
|
||||
continue
|
||||
|
||||
# your existing generator (rolling or block)—
|
||||
# it yields tuples of (train_start, train_end, val_start, val_end, test_start, test_end)
|
||||
for (t0, t1, v0, v1, e0, e1) in self._original_fold_generator(df_chunk, config):
|
||||
# if regime gating is off, just yield
|
||||
if not config['walk_forward']['regime']['enabled']:
|
||||
yield (t0, t1, v0, v1, e0, e1)
|
||||
continue
|
||||
|
||||
# 4) Check regime balance in each period
|
||||
periods = {
|
||||
'train': df_chunk.loc[t0:t1],
|
||||
'val': df_chunk.loc[v0:v1],
|
||||
'test': df_chunk.loc[e0:e1],
|
||||
}
|
||||
bad = False
|
||||
for name, subdf in periods.items():
|
||||
counts = subdf['regime_tag'].value_counts(normalize=True) * 100
|
||||
# ensure every regime appears ≥ min_pct
|
||||
for regime in sorted(df['regime_tag'].unique()):
|
||||
pct = counts.get(regime, 0.0)
|
||||
if pct < min_pct:
|
||||
bad = True
|
||||
break
|
||||
if bad:
|
||||
break
|
||||
|
||||
if bad:
|
||||
# you can log which period/regime failed here
|
||||
continue
|
||||
# otherwise it’s a valid fold
|
||||
yield (t0, t1, v0, v1, e0, e1)
|
||||
```
|
||||
|
||||
### Explanation of the steps
|
||||
|
||||
1. **Regime Tagging**
|
||||
- Run once, up‑front: compute your volatility or trend indicator over the full series, cut it into quantile bins, and assign each row a `regime_tag` of 0/1/2.
|
||||
|
||||
2. **Gap Partitioning**
|
||||
- Split the DataFrame into contiguous “chunks” wherever index gaps exceed your `gap_threshold_minutes`.
|
||||
- This avoids forcing folds that straddle a hole in the data.
|
||||
|
||||
3. **Fold Generation (Unchanged)**
|
||||
- Call your existing `_generate_walk_forward_folds` (rolling or block) on each contiguous chunk.
|
||||
|
||||
4. **Regime‐Balance Filter**
|
||||
- For each candidate fold, slice out the train/val/test segments, compute the fraction of each regime tag, and **skip** any fold where any regime appears below your `min_regime_representation_pct`.
|
||||
|
||||
---
|
||||
|
||||
#### Configuration sketch
|
||||
|
||||
```yaml
|
||||
walk_forward:
|
||||
# existing fields…
|
||||
gap_threshold_minutes: 5
|
||||
regime:
|
||||
enabled: true
|
||||
indicator: volatility
|
||||
indicator_params:
|
||||
window: 20
|
||||
quantiles: [0.33, 0.66]
|
||||
min_regime_representation_pct: 10
|
||||
```
|
||||
|
||||
With this wrapper, you get:
|
||||
|
||||
- **Automatic split** at data outages > 5 min
|
||||
- **Dynamic skip** of any time‐slice folds that would be blind to a market regime (e.g. all high‑vol or all low‑vol)
|
||||
- **No changes** to your core split logic—just filter its outputs.
|
||||
Reference in New Issue
Block a user