# GRU + Simplified SAC Trading Agent

This project implements a cryptocurrency trading system using a GRU model for price prediction and a **Simplified SAC (Soft Actor-Critic)** agent for position sizing.

The system predicts future *price* using a GRU model adapted from the V6 architecture. It calculates the *predicted percentage return* from this price prediction and estimates prediction *uncertainty* based on the standard deviation of Monte Carlo dropout predictions. It also extracts recent *momentum* and *volatility* features. These values, along with a risk proxy (`z_proxy`), form the **5-dimensional state** input (`[predicted_return, mc_unscaled_std_dev, z_proxy, momentum_5, volatility_20]`) to the SAC reinforcement learning agent, which determines optimal position sizing (-1 to +1) using a **squashed Gaussian policy** and **automatic entropy tuning**.

The system incorporates efficiency improvements by pre-computing GRU predictions and uncertainties before generating SAC experiences or running the backtest. It includes detailed backtesting, performance reporting, and visualization capabilities, including **SAC training loss plots**.

## System Design

The system integrates a GRU predictor and a Simplified SAC agent within a backtesting framework.

### 1. Data Flow & Processing

1.  **Loading:** Raw 1-minute OHLCV data is loaded from the SQLite database directory specified in `main.py` (e.g., `downloaded_data/`) using `src.data_pipeline.load_data_from_db` which utilizes `src.crypto_db_fetcher.CryptoDBFetcher`.
2.  **Splitting:** Data is chronologically split into training (60%), validation (20%), and test (20%) sets using `src.data_pipeline.create_data_pipeline`.
3.  **GRU Training / Loading (on Train/Validation Sets):**
    *   If `TRAIN_GRU_MODEL` is `True`:
        *   *Preprocessing*: `TradingSystem._preprocess_data_for_gru_training` calculates V6 features plus basic return features (`calculate_v6_features`) on the raw train/val data. It determines the future *price* target (`prediction_horizon` steps ahead) and aligns features, targets (prices), and the *unscaled* starting close prices needed for return calculation.
        *   *Scaling*: Within `TradingSystem.train_gru`, a `StandardScaler` is fitted *only* on the training features. A `MinMaxScaler` is fitted *only* on the training future *price* targets. Train and validation features/targets are scaled using these fitted scalers.
        *   *Sequence Creation*: `src.data_pipeline.create_sequences_v2` creates input sequences `(batch, sequence_length, num_features)` and corresponding scaled target prices using the scaled features/targets and the unscaled start prices.
        *   *Model Training*: `CryptoGRUModel.train` builds the V6-style GRU model (if not already built) and trains it using Mean Squared Error (MSE) loss on the scaled sequences. Callbacks monitor `val_rmse` for early stopping and model checkpointing. The best model (`best_model_reg.keras`) and the fitted scalers (`feature_scaler.joblib`, `y_scaler.joblib`) are saved.
    *   If `LOAD_EXISTING_SYSTEM` is `True` and `TRAIN_GRU_MODEL` is `False`:
        *   Attempts to load a pre-trained GRU model and scalers. If `GRU_MODEL_LOAD_RUN_ID` is set in `main.py`, it loads the GRU from that specific run ID's directory (`gru_sac_predictor/models/run_<run_id>`); otherwise, it attempts to load from the default `MODEL_SAVE_PATH` (expecting the model and scalers to be directly in that path).
        *   **Note:** SAC model loading is handled *separately* based on the `LOAD_SAC_AGENT` flag and the `GRU_MODEL_LOAD_RUN_ID` setting (see Model Loading/Training section in `main.py` for details).
4.  **SAC Training (on Validation Set):**
    *   **Training Loop:** The training process runs for a fixed number of epochs (`SAC_EPOCHS`).
    *   **Experience Generation** (`TradingSystem.generate_trading_experiences`):
        *   **Efficiency:** Pre-computes all required GRU outputs (predicted returns, uncertainties) for the entire validation set by calling `CryptoGRUModel.evaluate` *once*.
        *   **State Extraction:** Extracts pre-computed GRU outputs and relevant features (`momentum_5`, `volatility_20`) from the validation features dataframe.
        *   **Experience Format:** Iterates through the pre-computed results. Forms the 5D state `s_t = [pred_return_t, uncertainty_t, z_proxy_t, momentum_5_t, volatility_20_t]` (where `z_proxy` uses the position *before* the action). The SAC agent (`SimplifiedSACTradingAgent.get_action`) provides a *non-deterministic* action `a_t` and `log_prob`. The next state `s_{t+1}` is constructed similarly (using `action` for `z_proxy`). A reward `r_t = action * actual_return - cost` is calculated. The transition `(s_t, a_t, r_t, s_{t+1}, done)` is stored.
        *   **Note:** Experience sampling strategies (recency bias, stratification) defined in `experience_config` are currently *not* implemented in `generate_trading_experiences` but the configuration remains.
    *   **Agent Training** (`TradingSystem.train_sac` calls `SimplifiedSACTradingAgent.train`): Iterates for `SAC_EPOCHS`. In each epoch, the agent performs one training step. Batches are sampled from the replay buffer. Actor and Critic networks are updated using the SAC algorithm with automatic alpha tuning. Agent uses `store_transition` to add experiences to its internal NumPy buffer.
    *   **History Plotting:** After successful training, `plot_sac_training_history` is called to generate and save a plot of actor and critic losses.
5.  **Backtesting (on Test Set):**
    *   *Pre-computation* (`ExtendedBacktester.backtest`): Preprocesses test data, scales, creates sequences, calls `CryptoGRUModel.evaluate` once for GRU outputs, and extracts required features (`momentum_5`, `volatility_20`).
    *   *State Generation*: Constructs the 5D state `s_t = [pred_return, uncertainty, z_proxy, momentum_5, volatility_20]` using pre-computed results and the current position.
    *   *Action Selection*: The trained `SimplifiedSACTradingAgent` selects a *deterministic* action `a_t` (unpacking the tuple returned by `get_action`).
    *   *Portfolio Simulation*: Calculates PnL based on the previous position, actual return, and transaction costs.
    *   *Logging*: Records detailed metrics, trade history, and timestamps.
6.  **Evaluation:**
    *   *Performance Metrics*: `ExtendedBacktester._calculate_performance_metrics` computes overall portfolio metrics (Sharpe, Sortino, Drawdown, correlations, etc.) and Buy & Hold benchmark metrics.
    *   *Visualization*: `ExtendedBacktester.plot_results` generates a 3-panel plot: GRU Predictions vs Actual Price (with uncertainty), SAC Actions (Position Size), and Portfolio Value vs Buy & Hold (with trade markers).
    *   *Reporting*: `ExtendedBacktester.generate_performance_report` creates a detailed Markdown report.

### 2. Core Components & Inputs/Outputs

*   **`src.crypto_db_fetcher.CryptoDBFetcher`**: Loads and resamples data from SQLite DBs.
*   **`src.data_pipeline`**: Functions for DB loading, data splitting, sequence creation.
*   **`src.trading_system.calculate_v6_features`**: Calculates features (TA-Lib based V6 set + past returns).
*   **`src.trading_system._preprocess_data_for_gru_training`**: Prepares features, future price targets, and start prices.
*   **`src.gru_predictor.CryptoGRUModel`**: (V6 Adaptation)
    *   `train()`: Trains the GRU price prediction model. Saves model (`.keras`) and scalers (`.joblib`).
    *   `evaluate()`: Performs standard prediction and MC dropout inference. Returns dict including `pred_percent_change`, `mc_unscaled_std_dev`, `predicted_unscaled_prices`, `true_unscaled_prices`.
*   **`src.sac_agent_simplified.SimplifiedSACTradingAgent`**: (V7 Simplified)
    *   **Goal:** Learns a policy mapping state to optimal position size (-1.0 to +1.0). Optimized for faster training.
    *   **State Input:** 5-element array `[predicted_return, mc_unscaled_std_dev, z_proxy, momentum_5, volatility_20]`.
    *   **Action Output:** Float between -1.0 and +1.0.
    *   `get_action()`: Selects action (stochastic or deterministic). Adds uncertainty-scaled noise during exploration.
    *   `store_transition()`: Adds experience to internal NumPy buffer.
    *   `train()`: Updates agent using buffer samples (internally handles batch size). Uses `@tf.function` for performance.
    *   `save()` / `load()`: Handles Actor/Critic weights (`.weights.h5`), potentially `alpha.npy`.
    *   **Note:** Models and optimizers are built explicitly during `__init__` using dummy inputs to prevent TensorFlow graph mode issues.
*   **`src.trading_system.TradingSystem`**: Integrates GRU and SAC. Manages training pipelines, feature calculation, experience generation.
*   **`src.trading_system.ExtendedBacktester`**: Performs efficient backtesting using pre-computed GRU outputs, calculates metrics, plots results, generates reports.
*   **`src.trading_system.plot_sac_training_history`**: Generates plot for SAC actor/critic losses during training.

### 3. Model Architectures

*   **GRU (`src.gru_predictor.CryptoGRUModel._build_model`)**: V6 Architecture.
    *   Input -> GRU(100) -> Dropout(0.2) -> Dense(1, linear).
    *   Compiled with Adam (LR=0.001), MSE loss.
*   **Simplified SAC (`src.sac_agent_simplified.SimplifiedSACTradingAgent`)**:
    *   **Actor Network**: MLP `(state_dim=5)` -> Dense(64, relu) -> [BN] -> Dense(64, relu) -> [BN] -> [Residual] -> Dense(1, name='mu'), Dense(1, name='log_std'). Output is `mu` and `log_std` for a **Gaussian policy**. `log_std` is clipped.
    *   **Critic Network (x2)**: MLP `(state_dim=5 + action_dim=1)` -> Dense(64, relu) -> [BN] -> Dense(64, relu) -> [BN] -> [Residual] -> Dense(1, linear).
    *   **Algorithm**: Implements SAC with Clipped Double-Q, **automatic entropy tuning** (optimizing `alpha` based on `target_entropy`), squashed actions (`tanh`), faster learning rates, smaller networks/buffer, optional Batch Normalization / Residual connections. Uses Huber loss for critics. `@tf.function` used for update steps (`_update_critics`, `_update_actor_and_alpha`).

### 4. Features & State Representation

*   **GRU Features:** Uses the V6 feature set plus basic past returns (see `calculate_v6_features`). Cyclical time features (`hour_sin`, `hour_cos`) are added *before* data splitting.
*   **SAC State (`state_dim=5`):**
    1.  `predicted_return`: GRU predicted percentage return for the next period.
    2.  `uncertainty`: GRU MC dropout standard deviation (unscaled).
    3.  `z_proxy`: Risk proxy, calculated as `current_position * volatility_20`.
    4.  `momentum_5`: 5-minute return (`return_5m` feature).
    5.  `volatility_20`: 20-day volatility (`volatility_14d` feature, name mismatch intended).
*   **Scaling:** Features for GRU scaled with `StandardScaler`. Target price for GRU scaled with `MinMaxScaler`. SAC state components are used directly without separate scaling.

### 5. Evaluation

*   **GRU Model:** Evaluated using RMSE loss on validation set. Callbacks monitor `val_rmse`. Plots compare predicted vs actual price.
*   **SAC Agent & Overall System:** Evaluated via the `ExtendedBacktester` metrics (Sharpe, Sortino, Max Drawdown, correlations, etc.), plots (Portfolio vs B&H, Actions), and a final Markdown report. SAC training progress monitored via saved loss plots (`sac_training_history_<run_id>.png`).

## File Structure

- `downloaded_data/`: **Place your SQLite database files here.** (Or update `DB_DIR` in `main.py`).
- `gru_sac_predictor/`: Project root directory.
  - `models/`: Trained models saved here under `run_<run_id>/` directories.
  - `results/`: Backtest results saved here under `<run_id>/` directories.
  - `logs/`: Log files saved here under `<run_id>/` directories.
  - `src/`: Core Python modules.
    - `crypto_db_fetcher.py`
    - `data_pipeline.py`
    - `gru_predictor.py`
    - `sac_agent_simplified.py`
    - `trading_system.py`
  - `main.py`: Main script.
  - `requirements.txt`
  - `README.md`

## Setup

1.  **Data:** Place your V6 `downloaded_data` directory containing the SQLite files relative to the `gru_sac_predictor` project root, or update the `DB_DIR` variable in `main.py` to point to the correct location.
2.  **Dependencies:** Install required packages:
    ```bash
    pip install -r requirements.txt
    ```
    *Strongly Recommended:* Install TA-Lib for the full feature set. See TA-Lib installation guides for your OS.
3.  **Configuration:** Review and adjust parameters in `main.py`. Key parameters include:
    *   `DB_DIR`, `TICKER`, `EXCHANGE`, `START_DATE`, `END_DATE`, `INTERVAL`
    *   Model hyperparameters (GRU and SAC sections)
    *   Control Flags: `LOAD_EXISTING_SYSTEM`, `TRAIN_GRU_MODEL`, `TRAIN_SAC_AGENT`, `LOAD_SAC_AGENT`
    *   Loading Specific Models: `GRU_MODEL_LOAD_RUN_ID` (set to a specific run ID string like `'YYYYMMDD_HHMMSS'` to load *only* the GRU model from `gru_sac_predictor/models/run_<run_id>/`). SAC loading depends on `LOAD_SAC_AGENT` flag.
    *   SAC Training: `SAC_EPOCHS` defines the number of training epochs.
    *   Experience Generation: `experience_config` dictionary (sampling strategies currently not implemented).
    *   Backtesting: `INITIAL_CAPITAL`, `TRANSACTION_COST`.
4.  **Run:** Execute from the project root directory (the one *containing* `gru_sac_predictor`):
    ```bash
    python -m gru_sac_predictor.main
    ```
    Output files (logs, models, plots, report) will be generated in `gru_sac_predictor/logs/`, `gru_sac_predictor/models/`, and `gru_sac_predictor/results/` within run-specific subdirectories.

## Reporting

The report generated by the `ExtendedBacktester` includes performance metrics, correlation analyses, and configuration details. Key metrics include:

*   Total/Annualized Return
*   Sharpe & Sortino Ratios
*   Volatility & Max Drawdown
*   Buy & Hold Comparison
*   Position/Prediction Accuracy
*   Prediction/Position/Uncertainty Correlations
*   Total Trades 