# GRU-SAC Trading Predictor

This project implements a multi-stage machine learning pipeline for predicting financial market movements (specifically cryptocurrency price direction) and generating trading signals. It combines a Gated Recurrent Unit (GRU) network for initial prediction with a Soft Actor-Critic (SAC) reinforcement learning agent for refining trading decisions. The pipeline features advanced walk-forward validation, hyperparameter tuning, probability calibration, baseline model comparisons, and configurable validation gates.

## Key Features

*   **Data Handling:** Loads tick/minute-level data, preprocesses (resampling, missing value imputation), and handles large datasets efficiently.
*   **Feature Engineering:** Generates a comprehensive set of technical indicators (ATR, RSI, MACD, etc.) and allows for feature selection.
*   **Advanced Walk-Forward Validation:** Implements a flexible `FoldGenerator` supporting:
    *   Contiguous data chunk splitting based on time gaps.
    *   Regime awareness (e.g., volatility regimes) ensuring folds are representative.
    *   N-Fold Block Splitting.
    *   Rolling/Expanding Window validation.
    *   Minimum fold duration checks.
    *   Purging and embargo periods.
*   **GRU Model (v3):**
    *   Predicts future price movement direction (Binary or Ternary) and expected return magnitude (`mu`).
    *   Architecture includes GRU layer, optional Multi-Head Attention with causal masking, Layer Normalization, and separate output heads.
    *   Uses Huber loss for `mu` and Categorical Focal Loss for direction prediction.
    *   Supports optional Nested Cross-Validation using Optuna for hyperparameter tuning within each walk-forward fold.
*   **Soft Actor-Critic (SAC) Agent:**
    *   Trains offline using the GRU predictions as part of the state.
    *   Aims to learn an optimal trading policy (position sizing/timing).
    *   Features automatic entropy tuning (alpha), optional Prioritized Experience Replay (PER), and state normalization.
*   **Probability Calibration:** Calibrates GRU output probabilities using Vector Scaling (for ternary) or Temperature Scaling (for binary - *currently assumes ternary/vector*). Includes optimization for the decision edge threshold.
*   **Baseline Models:** Compares GRU performance against standard baselines (Logistic Regression, Random Forest) for both binary and ternary classification tasks.
*   **Validation Gates:** Implements configurable checks at multiple pipeline stages (Baseline performance, GRU performance, Backtest performance, Final Release) to ensure model quality and potentially halt execution.
*   **Incremental Fold Reporting:** Generates detailed JSON reports for each fold, logging the status and results of each validation gate step, facilitating debugging even if a fold fails mid-way.
*   **Modularity & Configuration:** Highly configurable via `config.yaml`. Pipeline stages are modular functions.
*   **Artifact Management:** Uses an `IOManager` (assumed) to handle saving/loading of data, models, results, logs, and figures in a structured run-specific manner.

## Pipeline Flow Diagram

```mermaid
graph TD
    subgraph Global_Setup ["Global Setup"]
        direction LR
        A[Load Config] --> B["Init IOManager"];
        B --> C["Init Pipeline Components: DataLoader, FeatureEng, GRUHandler, etc."];
        C --> D[Load Full Raw Data];
        D --> E["Init FoldGenerator w/ Data"];
    end

    subgraph Fold_Loop ["Fold Loop"]
        direction TB
        F[Generate Folds via FoldGenerator] --> G{Iterate Folds};
        G -- Fold Data --> H[Start Fold Processing];

        subgraph Fold_Processing ["Fold N Processing"]
            direction TB
            I[Engineer Features] --> J[Define Labels & Align];
            J --> K[Split Data: Train/Val/Test];
            K --> L[Scale Features];
            L --> M["Coarse Filter Features (Optional)"];
            M --> N[Run Initial Baseline Checks];
            N --> N_Report["Record Baseline 1 Report"];
            N_Report --> N_Gate{Baseline Gate 1 Passed?};

            N_Gate -- Yes --> O[Select & Prune Features];
            O --> P[Run Post-Pruning Baseline Checks];
            P --> P_Report["Record Baseline 2 Report"];
            P_Report --> P_Gate{Baseline Gate 2 Passed?};

            P_Gate -- Yes --> Q[Update Scaled Data = Pruned];
            Q --> R[Create Sequences];
            R --> S["Train/Load GRU w/ Nested CV?"];
            S --> T[Calibrate Probabilities];
            T --> U[Run GRU Validation Checks];
            U --> U_Report["Record GRU Validation Report"];
            U_Report --> U_Gate{GRU Gate Passed?};

            U_Gate -- Yes --> V[Train/Load SAC Agent];
            V --> V_Report["Record SAC Status Report"];
            V_Report --> W[Run Backtest Simulation];
            W --> W_Report["Record Backtest Report & Perf Gate Check"];
            %% Checks Perf Gates Internally
            W_Report --> X[Store Fold Metrics];
            X --> Y[Run Forward Baseline Check];
            Y --> Y_Gate{Forward Baseline Gate Passed?};
            Y_Gate -- Yes --> Z[Fold Success];

            %% Failure Paths within Fold (Leading to AA: Fold Failed)
            N_Gate -- No --> AA[Fold Failed];
            P_Gate -- No --> AA;
            U_Gate -- No --> AA;
            Y_Gate -- No --> AA;
            %% Note: Backtest Perf Gate Fail doesn't halt fold here, just logged in report
            W_Report -- Error during Backtest --> AA;
            %% Handle explicit errors

        end

        H --> I;
        %% Start processing for the fold
        %% Save Report in Finally Block (Implied - happens before moving to AB)
        Z -- Save Report --> AB{More Folds?};
        AA -- Save Report --> AB;
        AB -- Yes --> G;
        G -- No More Folds --> AC[End Fold Loop];

    end

    subgraph Final_Aggregation ["Final Aggregation"]
        direction TB
        AC --> AD["Aggregate Fold Metrics"];
        AC --> AE["Aggregate SAC Agents (Optional)"];
        AD --> AF["Final Release Decision"];
    end

    E --> F;
    %% Fold Generation Starts after Global Setup
    AF --> AG[End Pipeline];

    style Fold_Processing fill:#f9f,stroke:#333,stroke-width:2px
```

## Installation / Setup

*(Assuming standard Python environment management)*

1.  **Clone the repository:**
    ```bash
    git clone <repository_url>
    cd gru_sac_predictor
    ```
2.  **Create and activate a virtual environment:** (Recommended)
    ```bash
    python -m venv venv
    source venv/bin/activate # or venv\Scripts\activate on Windows
    ```
3.  **Install dependencies:**
    ```bash
    pip install -r requirements.txt
    ```
    *(Ensure `requirements.txt` includes `tensorflow`, `pandas`, `numpy`, `scikit-learn`, `pyyaml`, `optuna`, `joblib`, `matplotlib`, `seaborn`, `tqdm` etc.)*
4.  **Data Setup:** Ensure the crypto market database exists at the location specified by `data.db_dir` in `config.yaml`. The `DataLoader` expects a specific database structure (details should be added based on `DataLoader` implementation).

## Configuration (`config.yaml`)

The pipeline's behavior is primarily controlled by `config.yaml`. Below is a detailed breakdown of key sections and parameters:

**`pipeline`:**
*   `description`: Textual description of the configuration.
*   `stages_to_run`: (Optional) List of stages to execute (e.g., `["data", "features", "gru"]`) for partial runs. Defaults to all stages if omitted.

**`data`:**
*   `ticker`, `exchange`, `interval`, `start_date`, `end_date`: Parameters for data loading via `DataLoader`.
*   `db_dir`: **Crucial**. Path to the database directory (relative to project root or absolute). Default: '../data/crypto_market_data'.
*   `bar_frequency`: Target frequency for resampling (e.g., "1T" for 1 minute). Used in preprocessing.
*   `missing`: Configuration for handling missing data points after resampling.
    *   `strategy`: Method ('drop', 'neutral', 'ffill', 'interpolate'). Default: 'neutral'.
    *   `max_gap`: Maximum consecutive missing bars allowed before applying `strategy` or potentially dropping (depends on strategy). Default: 60.
    *   `interpolate`: Settings if `strategy: "interpolate"`. Includes `method` and `limit`.
*   `volatility_sampling`: (Optional) Configuration for volatility-based downsampling.
    *   `enabled`: Boolean flag. Default: `False`.
    *   `window`: Window for volatility calculation. Default: 30.
    *   `quantile`: Quantile threshold for sampling. Default: 0.5.

**`features`:**
*   Parameters for base feature calculation in `FeatureEngineer.add_base_features` (e.g., `atr_window`, `rsi_window`, `macd_fast`, etc.). Defaults shown in `config.yaml`.
*   Parameters for feature selection (`FeatureEngineer.select_features`).
    *   `selection_method`: Method used (e.g., "correlation"). Default: "correlation".
    *   `correlation_threshold`: Threshold for correlation-based selection. Default: 0.02.
    *   `min_features_after_selection`: Minimum features to retain. Default: 10.
    *   `coarse_univariate_quantile`: (Used in `TradingPipeline` before baseline) Quantile threshold for a coarse univariate filter based on correlation with the target before main feature selection. Keeps features *above* this quantile. Default: 0.70 (i.e., keep top 30%).

**`walk_forward`:** Configuration for data splitting and fold generation (`FoldGenerator`).
*   `enabled`: Master switch for walk-forward vs. single split. Default: `True`.
*   `split_ratios`: Used if `enabled: False`. Defines train/validation ratios for a single split. Test ratio is inferred. Defaults: `train: 0.6`, `validation: 0.2`.
    **(Walk-Forward Settings - used if `enabled: True`)**
    *   `num_folds`: Number of folds. If > 1, uses N-Fold Block splitting. If <= 1, uses Rolling Window mode. Default: 5.
    **(Rolling Window Specific - used if `num_folds <= 1`)**
        *   `train_period_days`, `validation_period_days`, `test_period_days`: Duration of each period in days. Defaults: 25, 10, 10.
        *   `step_days`: How many days to slide the window forward. Default: 7.
        *   `expanding_window`: If `True`, training period grows; otherwise, it slides. Default: `false`.
    **(General Walk-Forward Settings)**
        *   `initial_offset_days`: Skip days at the beginning before the first fold starts. Default: 14.
        *   `purge_window_minutes`: Drop training samples potentially overlapping with validation/test based on lookahead/lookback (implementation details needed). Default: 0.
        *   `embargo_minutes`: Gap added after train/val periods before starting the next. Default: 0.
        *   `final_holdout_period_days`: Reserve days at the very end, excluded from all folds. Default: 0.
        *   `min_fold_duration_days`: Minimum *total* duration (train_start to test_end) for a fold to be considered valid. Default: 1.
    **(Gap and Regime Settings)**
        *   `gap_threshold_minutes`: Split data into contiguous chunks if time gap exceeds this. Default: 5.
        *   `min_chunk_days`: Minimum duration for a chunk to be used for fold generation. Default: 1.
        *   `regime`: Configuration for regime-aware fold generation.
            *   `enabled`: Enable regime filtering. Default: `true`.
            *   `indicator`: Indicator used for regimes (e.g., 'volatility'). Default: 'volatility'.
            *   `indicator_params`: Parameters for the chosen indicator (e.g., `window`). Default: `window: 20`.
            *   `quantiles`: Quantiles to define regime boundaries. Default: `[0.33, 0.66]` (for 3 regimes).
            *   `min_regime_representation_pct`: Minimum percentage each defined regime must occupy in *each* period (train, val, test) of a candidate fold for it to be yielded. Default: 10.

**`gru`:** Configuration for the GRU model (`GRUModelHandler`, label generation).
*   `train_gru`: Master switch to train a new GRU model vs. loading one. Default: `true`.
*   `use_ternary`: Use ternary (Up/Flat/Down) labels instead of binary (Up/Down). Default: `True`.
*   `prediction_horizon`: Lookahead period (in units of `data.interval`) for target calculation. Default: 5.
*   `flat_sigma_multiplier`: 'k' factor for ternary flat label threshold (ε = k * rolling_std(fwd_ret)). Used if `use_ternary: True`. Default: 0.25.
*   `label_smoothing`: Alpha for *binary* label smoothing (0.0 disables). Default: 0.0. (Note: Focal loss has its own smoothing).
*   `drop_imputed_sequences`: If `True`, sequences containing any bar marked as imputed (`bar_imputed` feature) are dropped. Default: `true`.
    **(Model Architecture v3 - `build_gru_model_v3`)**
    *   `gru_units`: GRU layer units. Default: 96.
    *   `attention_units`: Attention layer units (set <= 0 to disable). Default: 16.
    *   `dropout_rate`: Dropout rate for GRU/Attention. Default: 0.1.
    *   `learning_rate`: Adam optimizer learning rate. Default: 1e-4.
    *   `l2_reg`: L2 regularization factor for Dense layers. Default: 1e-4.
*   **(Loss Function v3 - `build_gru_model_v3`)**
    *   `focal_gamma`: Gamma for categorical focal loss (if `use_ternary: True`). Default: 2.0.
    *   `focal_label_smoothing`: Label smoothing within focal loss (if `use_ternary: True`). Default: 0.1.
    *   `huber_delta`: Delta for Huber loss (mu/return prediction). Default: 1.0.
    *   `loss_weight_mu`: Weight for the mu/return loss component. Default: 0.3.
    *   `loss_weight_dir3`: Weight for the direction loss component. Default: 1.0.
*   **(Training Parameters - `GRUModelHandler.train`)**
    *   `lookback`: Sequence length (timesteps) for GRU input. Default: 60.
    *   `epochs`: Maximum training epochs. Default: 25.
    *   `batch_size`: Training batch size. Default: 128.
    *   `patience`: Early stopping patience (epochs). Default: 5.
*   **(Loading Control - `train_or_load_gru_fold`)**
    *   `load_gru_model`: Section to specify loading a pre-trained model.
        *   `run_id`: Specific GRU pipeline *run ID* to load from. If `null`, training occurs (if `train_gru: True`). Default: `null`.
        *   `fold_num`: Specific fold number to load. If `null`, loader might try to find 'best'/last based on internal logic. Default: `null`.

**`hyperparameter_tuning.gru`:** Configuration for Optuna Nested CV (`train_or_load_gru_fold`).
*   `sweep_enabled`: Master switch to enable/disable Optuna sweep. Default: `False`.
*   `study_name`: Name for the Optuna study. Default: "gru_optimization".
*   `direction`: "minimize" (val_loss) or "maximize" (val_accuracy). Default: "minimize".
*   `n_trials`: Number of hyperparameter combinations to try. Default: 50.
*   `inner_cv_splits`: Number of inner folds for TimeSeriesSplit within the training data. Default: 3.
*   `pruner`: Optuna pruner ('median', 'hyperband', etc.). Default: "median".
*   `sampler`: Optuna sampler ('tpe', 'random', etc.). Default: "tpe".
*   `search_space`: Dictionary defining parameters to tune, their types (`int`, `float`, `categorical`, `loguniform`), and ranges/choices. Defaults shown in `config.yaml`.

**`calibration`:** Configuration for probability calibration (`calibrate_probabilities_fold`).
*   `method`: Calibration method ('vector', 'temperature', etc.). Default: 'vector'. (Note: Implementation assumes 'vector' for ternary, 'temperature' for binary).
*   `optimize_edge_threshold`: Whether to optimize the edge threshold using Youden's J on validation predictions. Default: `true`.
*   `edge_threshold`: Initial/fixed edge threshold if not optimizing. Default: 0.5.
    **(Rolling Calibration - Used if implemented within Calibrator classes)**
    *   `rolling_window_size`, `rolling_min_samples`, `rolling_step`: Parameters for rolling calibration methods. Defaults: 250, 50, 50.
*   `reliability_plot_bins`: Number of bins for reliability plot generation. Default: 10.

**`sac`:** Configuration for the SAC agent (`SACTradingAgent`) and trainer (`SACTrainer`).
*   `imputed_handling`: How SAC environment handles steps with imputed features ('hold', 'skip', etc. - requires env implementation). Default: "hold".
*   `action_penalty`: Penalty applied during SAC training for large/frequent actions (requires env implementation). Default: 0.05.
    **(Agent Hyperparameters - `SACTradingAgent`)**
    *   `gamma`, `tau`, `actor_lr`, `critic_lr`, `alpha`, `alpha_auto_tune`, `target_entropy`: Standard SAC hyperparameters. Defaults shown in `config.yaml`.
    *   `lr_decay_rate`, `decay_steps`: Optional learning rate decay parameters.
    *   `ou_noise_stddev`: Optional Ornstein-Uhlenbeck noise parameter (if used).
*   **(Training Loop - `SACTrainer`)**
    *   `total_training_steps`, `buffer_capacity`, `batch_size`, `start_steps`, `update_after`, `update_every`, `save_freq`, `log_freq`, `eval_freq`: Parameters controlling the SAC training loop. Defaults shown in `config.yaml`.
*   **(Alpha Annealing - `SACTrainer`)**
    *   `alpha_anneal_start_step`, `alpha_anneal_end_step`, `initial_alpha`, `final_alpha`: Parameters for annealing the entropy temperature `alpha` (if `alpha_auto_tune: True`). Defaults shown in `config.yaml`.
*   **(Prioritized Experience Replay (PER) - `SACTrainer`/Buffer)**
    *   `use_per`: Enable PER. Default: `False`.
    *   `per_alpha`, `per_beta_start`, `per_beta_frames`: PER parameters (used if `use_per: True`). Defaults shown in `config.yaml`.
    *   `per_alpha_anneal_enabled`, `per_alpha_start`, `per_alpha_end`, `per_alpha_anneal_steps`: Optional annealing for PER alpha.
*   `oracle_seeding_pct`: (Experimental) Percentage of buffer to pre-fill using heuristic. Default: 0.0.
    **(State Normalization - `SACTrainer`)**
    *   `use_state_filter`: Use MeanStdFilter for state normalization. Default: `True`.
    *   `state_dim_fallback`, `action_dim_fallback`: Fallback dimensions if they cannot be inferred (e.g., from loaded agent). Defaults: 5, 1.
*   `train_sac`: Master switch to train a new SAC agent vs. loading one for the fold. Default: `True`.

**`sac_aggregation`:** Configuration for aggregating SAC agents after walk-forward (`aggregate_sac_agents`).
*   `enabled`: Enable aggregation. Default: `True`.
*   `method`: Aggregation method ('average_weights' currently supported). Default: "average_weights".

**`environment`:** Parameters passed to the `TradingEnv` (used by `SACTrainer` and `Backtester`).
*   `initial_capital`: Starting capital. Default: 10000.0.
*   `transaction_cost`: Fractional cost per trade. Default: 0.0005 (0.05%).
*   `reward_scale`: Multiplier applied to raw PnL reward. Default: 100.0.
*   `action_penalty_lambda`: Penalty factor for action magnitude/changes in reward calculation (0 disables). Default: 0.0.

**`baselines`:** Configuration for `BaselineChecker`.
*   `run_baseline1`, `run_baseline2`, `run_baseline3`, `run_baseline4`: Enable specific baselines (Logistic Regression, Random Forest for binary/ternary). Defaults: True/False/True/False.
    Sections for each baseline (`logistic_regression`, `random_forest`, `multinomial_logistic_regression`, `ternary_random_forest`) containing model hyperparameters (e.g., `max_iter`, `n_estimators`), internal validation split (`val_subset_split_ratio`, `val_subset_shuffle`), and CI confidence level (`ci_confidence_level`). Defaults shown in `config.yaml`.

**`validation_gates`:** Thresholds checked at different stages to potentially halt the pipeline fold.
*   `run_baseline_check`: Master switch for *all* baseline checks. Default: `True`.
    Baseline Gates (`baseline<N>_min_ci_lb`, `baseline<N>_edge_min_ci_lb`): Minimum acceptable Confidence Interval Lower Bound for baseline accuracy (raw and edge-filtered) for both binary and ternary models. Defaults shown in `config.yaml`.
*   `gru_performance`: Gates applied after GRU calibration (`run_gru_validation_checks_fold`).
    *   `enabled`: Enable these GRU checks. Default: `True`.
    *   `min_edge_accuracy`: Minimum edge-filtered accuracy on validation set. Default: 0.60.
    *   `max_brier_score`: Maximum acceptable Brier score on validation set. Default: 0.24.
*   `backtest`: Master switch for *all* backtest performance gates. Default: `True`.
*   `backtest_performance`: Specific gates applied after backtesting (`run_backtest_fold`).
    *   `enabled`: Enable Sharpe and Max Drawdown checks. Default: `True`.
    *   `min_sharpe_ratio`: Minimum acceptable annualized Sharpe ratio. Default: 1.2.
    *   `max_drawdown_pct`: Maximum acceptable drawdown percentage. Default: 15.0.
*   `final_release`: (Used in `TradingPipeline.final_release_decision`) Criteria applied to *aggregated* metrics across all folds.
    *   `min_successful_folds_pct`: Minimum required percentage of fully successful folds. Default: 0.75.
    *   `median_sharpe_threshold`: Minimum median Sharpe ratio across successful folds. Default: 1.3.
    *   `max_drawdown_max_threshold`: Maximum allowable *worst-case* drawdown across all successful folds. Default: 20.0.

**`control`:** Flags controlling pipeline execution flow.
*   `generate_plots`: Generate and save plots (learning curves, backtest summary, etc.). Default: `True`.
*   `sac_load_run_id`: Specify SAC Training *Run ID* to load for backtesting (used if `sac.train_sac: False`). Default: `null`.
*   `sac_load_step`: Step/checkpoint to load ('final' or step number). Default: 'final'.
*   `sac_resume_run_id`, `sac_resume_step`: Parameters for resuming a previous SAC training run (implementation status needs verification). Defaults: `null`, 'final'.

**`logging`:** Configuration for Python's `logging` module.
*   `console_level`, `file_level`: Logging levels (DEBUG, INFO, WARNING, ERROR, CRITICAL). Defaults: INFO, DEBUG.
*   `log_to_file`: Enable file logging. Default: `True`. Log file path determined by `IOManager`.
*   `log_format`, `log_date_format`: Formatting strings.
*   `log_file_max_bytes`, `log_file_backup_count`: Parameters for `RotatingFileHandler`. Defaults: 10MB, 5 backups.

**`output`:** Configuration for saving artifacts and plots (`IOManager`).
*   `base_dirs`: Base directories for `results`, `models`, `logs`. Defaults: "results", "models", "logs".
*   `figure_dpi`, `figure_size`, `figure_footer`, `plot_style`: Matplotlib settings for saved figures. Defaults shown in `config.yaml`.
*   `reward_plot_smoothing_alpha`: EMA alpha for smoothing SAC reward plot. Default: 0.2.
*   `dataframe_save_format`: Format for saving DataFrames ('csv', 'parquet', 'parquet_if_large'). Default: "parquet_if_large".
*   `dataframe_max_csv_mb`: Size threshold (MB) for using Parquet if format is 'parquet_if_large'. Default: 100.

## Pipeline Stages

The pipeline executes the following stages, primarily coordinated by `TradingPipeline.execute`:

1.  **Load & Preprocess Data (`load_and_preprocess_data` -> `pipeline_stages.data_processing.load_and_preprocess`)**
    *   **Purpose:** Load raw market data, clean, resample, and handle missing values.
    *   **Logic:** Uses `DataLoader` to fetch data based on `config['data']`. Applies resampling to `data.bar_frequency`. Imputes missing values based on `data.missing` config. Optionally performs volatility sampling (`data.volatility_sampling`). Calculates basic returns (`return_1m`). Adds `bar_imputed` feature.
    *   **Inputs:** `config['data']`.
    *   **Outputs:** `self.df_raw` (DataFrame with preprocessed data and DatetimeIndex).

2.  **Initialize Fold Generator (`TradingPipeline.__init__`)**
    *   **Purpose:** Prepare the fold generation mechanism.
    *   **Logic:** Instantiates `FoldGenerator` with the full `self.df_raw`. If `walk_forward.regime.enabled` is true, it calls internal helper `_add_regime_tags` to calculate and add the `regime_tag` column to an internal copy (`self.df_raw_tagged`) based on `walk_forward.regime` parameters. Stores `self.regime_enabled` status.
    *   **Inputs:** `config['walk_forward']`, `self.df_raw`.
    *   **Outputs:** `self.fold_generator` (initialized instance), `self.regime_enabled`.

3.  **Generate Folds & Loop (`TradingPipeline.execute` -> `FoldGenerator.generate_folds`)**
    *   **Purpose:** Iterate through walk-forward fold definitions.
    *   **Logic:** Calls `self.fold_generator.generate_folds()`. This generator internally:
        *   Splits `self.df_raw_tagged` into contiguous chunks based on `walk_forward.gap_threshold_minutes`.
        *   Skips chunks shorter than `walk_forward.min_chunk_days`.
        *   For each valid chunk, calls `FoldGenerator._generate_folds_for_chunk` which implements single split, N-block, or rolling window logic based on `walk_forward.enabled` and `walk_forward.num_folds`.
        *   Applies `walk_forward.min_fold_duration_days` check to each candidate fold.
        *   If `self.regime_enabled`, checks regime balance using `walk_forward.regime.min_regime_representation_pct` for each period (train, val, test) in the candidate fold before yielding.
    *   **Inputs:** `config['walk_forward']`, `self.df_raw_tagged` (internal to generator).
    *   **Outputs:** Iterator yielding tuples of fold dates `(train_start, train_end, val_start, val_end, test_start, test_end)`.

4.  **--- Inside Fold Loop (`TradingPipeline.execute`) ---**

    a.  **Engineer Features (`engineer_features` -> `pipeline_stages.data_processing.engineer_features_for_fold`)**
        *   **Purpose:** Calculate technical indicators for the current fold's raw data slice.
        *   **Logic:** Takes the `current_fold_data_raw` slice. Calls `FeatureEngineer.add_all_features` (or similar) based on `config['features']` parameters (windows, etc.).
        *   **Inputs:** `current_fold_data_raw`, `config['features']`.
        *   **Outputs:** `df_engineered_fold` (DataFrame with features for the fold).

    b.  **Define Labels & Align (`define_labels_and_align` -> `pipeline_stages.data_processing.define_labels_and_align_fold`)**
        *   **Purpose:** Generate target labels (return, direction) and align features/targets.
        *   **Logic:** Calculates forward returns (`fwd_log_ret_N`) based on `gru.prediction_horizon`. Generates direction labels (binary or ternary based on `gru.use_ternary`, using `gru.flat_sigma_multiplier` for ternary). Aligns features (X) and targets (y) by dropping NaNs introduced by lookaheads/lookbacks. Stores required target column names. Calculates volatility (`eps`) used for ternary labels.
        *   **Inputs:** `df_engineered_fold`, `config['gru']`.
        *   **Outputs:** `df_labeled_aligned_fold`, `self.target_dir_col`, `self.target_columns`, `self.fwd_returns_aligned`, `self.eps_aligned`.

    c.  **Split Data (`split_data` -> `pipeline_stages.data_processing.split_data_fold`)**
        *   **Purpose:** Split the aligned data for the current fold into train, validation, and test sets based on the dates yielded by `FoldGenerator`.
        *   **Logic:** Slices `df_labeled_aligned_fold` using the `fold_dates` tuple. Extracts corresponding X (features) and y (targets). Stores original unsplit data slices for reference. Extracts ordinal direction labels. Extracts corresponding slices of forward returns and epsilon values needed for baseline checks.
        *   **Inputs:** `df_labeled_aligned_fold`, `fold_dates`, `self.fwd_returns_aligned`, `self.eps_aligned`, `self.target_columns`, `self.target_dir_col`.
        *   **Outputs:** `self.X_train_raw`, `self.X_val_raw`, `self.X_test_raw`, `self.y_train`, `self.y_val`, `self.y_test`, `self.df_train_original`, `self.df_val_original`, `self.df_test_original`, `self.y_dir_train_ordinal`, `self.y_dir_val_ordinal`, `self.fwd_ret_train`, `self.eps_train`, `self.fwd_ret_val`, `self.eps_val`.

    d.  **Scale Features (`scale_features` -> `pipeline_stages.feature_processing.scale_features_fold`)**
        *   **Purpose:** Apply standardization (StandardScaler) to features.
        *   **Logic:** Fits `StandardScaler` on `self.X_train_raw`, then transforms train, validation, and test sets.
        *   **Inputs:** `self.X_train_raw`, `self.X_val_raw`, `self.X_test_raw`.
        *   **Outputs:** `self.scaler` (fitted scaler object), `self.X_train_scaled`, `self.X_val_scaled`, `self.X_test_scaled`.

    e.  **Coarse Univariate Filter (`TradingPipeline.execute`)**
        *   **Purpose:** Optional initial filtering step before baseline checks.
        *   **Logic:** Calculates absolute correlation between each feature in `self.X_train_scaled` and the aligned `self.y_dir_train_ordinal`. Keeps features with correlation >= the quantile specified by `features.coarse_univariate_quantile`. Applies this filter to create `X_train_coarse` and `X_val_coarse`.
        *   **Inputs:** `self.X_train_scaled`, `self.X_val_scaled`, `self.y_dir_train_ordinal`, `config['features']`.
        *   **Outputs:** `X_train_coarse`, `X_val_coarse` (or `None` if filter fails/removes all features).

    f.  **Initial Baseline Check (`run_baseline_checks` -> `pipeline_stages.evaluation.run_baseline_checks_fold`)**
        *   **Purpose:** Run baseline models on scaled (or coarsely filtered) data as an early validation gate.
        *   **Logic:** Calls the stage function, passing scaled data (or `X_*_coarse` if available), ordinal labels, forward returns, and epsilon values. The stage function uses `BaselineChecker` to train and evaluate baselines specified in `config['baselines']`. Compares performance (CI lower bounds) against thresholds in `config['validation_gates']`. Generates label histogram plot.
        *   **Inputs:** Scaled features (`X_train_scaled`