2025-04-18 03:31:11,814 - root - INFO - Using Base Models Directory: /home/yasha/develop/gru_sac_predictor/models 2025-04-18 03:31:11,814 - root - INFO - Using results directory: /home/yasha/develop/gru_sac_predictor/results/20250418_033111 2025-04-18 03:31:11,814 - root - INFO - Using logs directory: /home/yasha/develop/gru_sac_predictor/logs/20250418_033111 2025-04-18 03:31:11,814 - root - INFO - Using models directory: /home/yasha/develop/gru_sac_predictor/models/20250418_033111 2025-04-18 03:31:11,814 - root - INFO - Logging setup complete. Log file: /home/yasha/develop/gru_sac_predictor/logs/20250418_033111/pipeline_20250418_033111.log 2025-04-18 03:31:11,814 - root - INFO - --- Starting Pipeline Run: 20250418_033111 --- 2025-04-18 03:31:11,814 - root - INFO - Using config: /home/yasha/develop/gru_sac_predictor/gru_sac_predictor/config.yaml 2025-04-18 03:31:11,814 - root - INFO - Resolved relative db_dir '../data/crypto_market_data' to absolute path: /home/yasha/develop/data/crypto_market_data 2025-04-18 03:31:11,814 - gru_sac_predictor.src.data_loader - INFO - Initialized DataLoader with db_dir='/home/yasha/develop/data/crypto_market_data' 2025-04-18 03:31:11,815 - gru_sac_predictor.src.feature_engineer - INFO - FeatureEngineer initialized with minimal whitelist: ['return_1m', 'return_15m', 'return_60m', 'ATR_14', 'volatility_14d', 'chaikin_AD_10', 'svi_10', 'EMA_10', 'EMA_50', 'MACD', 'MACD_signal', 'hour_sin', 'hour_cos'] 2025-04-18 03:31:11,815 - gru_sac_predictor.src.gru_model_handler - INFO - GRUModelHandler initialized for run 20250418_033111 in /home/yasha/develop/gru_sac_predictor/models/20250418_033111 2025-04-18 03:31:11,815 - gru_sac_predictor.src.calibrator - INFO - Calibrator initialized with edge threshold: 0.55 2025-04-18 03:31:11,815 - gru_sac_predictor.src.backtester - INFO - Backtester initialized. 2025-04-18 03:31:11,815 - gru_sac_predictor.src.backtester - INFO - Initial Capital: 10000.00 2025-04-18 03:31:11,815 - gru_sac_predictor.src.backtester - INFO - Transaction Cost: 0.0500% 2025-04-18 03:31:11,815 - gru_sac_predictor.src.backtester - INFO - Edge Threshold: 0.550 2025-04-18 03:31:11,819 - root - INFO - Saved run configuration to /home/yasha/develop/gru_sac_predictor/results/20250418_033111/run_config.yaml 2025-04-18 03:31:11,819 - root - INFO - === Starting Pipeline Execution === 2025-04-18 03:31:11,819 - root - INFO - --- Stage: Loading and Preprocessing Data --- 2025-04-18 03:31:11,819 - gru_sac_predictor.src.data_loader - INFO - Loading data for SOL-USDT (bnbspot) from 2025-03-01 to 2025-03-10, interval 1min 2025-04-18 03:31:11,822 - gru_sac_predictor.src.data_loader - INFO - Scanning for DB files recursively in: /home/yasha/develop/data/crypto_market_data 2025-04-18 03:31:11,869 - gru_sac_predictor.src.data_loader - INFO - Found 316 DB files. Using newest: 20250416.mktdata.ohlcv.db 2025-04-18 03:31:11,973 - gru_sac_predictor.src.data_loader - INFO - Identified 8 potential DB files: ['20250228.mktdata.ohlcv.db', '20250301.mktdata.ohlcv.db', '20250302.mktdata.ohlcv.db', '20250303.mktdata.ohlcv.db', '20250304.mktdata.ohlcv.db', '20250305.mktdata.ohlcv.db', '20250306.mktdata.ohlcv.db', '20250307.mktdata.ohlcv.db'] 2025-04-18 03:31:12,037 - gru_sac_predictor.src.data_loader - INFO - Combined data shape before final filtering/resampling: (9827, 5) 2025-04-18 03:31:12,038 - gru_sac_predictor.src.data_loader - INFO - Shape after final date filtering: (9827, 5) 2025-04-18 03:31:12,039 - gru_sac_predictor.src.data_loader - INFO - Successfully loaded and processed data for SOL-USDT. Final shape: (9827, 5) 2025-04-18 03:31:12,040 - root - INFO - Raw data loaded successfully: 9827 rows from 2025-03-01 00:00:00+00:00 to 2025-03-07 23:59:00+00:00 2025-04-18 03:31:12,040 - root - INFO - --- Stage: Engineering Features --- 2025-04-18 03:31:12,040 - gru_sac_predictor.src.feature_engineer - INFO - --- Adding Base Features --- 2025-04-18 03:31:12,040 - gru_sac_predictor.src.feature_engineer - INFO - Adding cyclical hour features (sin/cos)... 2025-04-18 03:31:12,042 - gru_sac_predictor.src.feature_engineer - INFO - Adding imbalance features... 2025-04-18 03:31:12,051 - gru_sac_predictor.src.feature_engineer - INFO - Successfully added imbalance features. 2025-04-18 03:31:12,051 - gru_sac_predictor.src.feature_engineer - INFO - Adding TA features... 2025-04-18 03:31:12,140 - gru_sac_predictor.src.feature_engineer - INFO - Successfully added TA features. 2025-04-18 03:31:12,141 - gru_sac_predictor.src.feature_engineer - INFO - Base feature engineering complete. DataFrame shape: (9827, 20) 2025-04-18 03:31:12,143 - root - INFO - Feature engineering complete. Shape: (9827, 20) 2025-04-18 03:31:12,143 - root - INFO - --- Stage: Defining Labels and Aligning --- 2025-04-18 03:31:12,146 - root - INFO - Dropped 5 rows due to NaN targets (horizon=5). 2025-04-18 03:31:12,147 - root - INFO - Labels (horizon=5) defined and aligned. Features shape: (9822, 20), Targets shape: (9822, 2) 2025-04-18 03:31:12,147 - root - INFO - --- Stage: Splitting Data --- 2025-04-18 03:31:12,147 - __main__ - INFO - Using split ratios: Train=0.60, Val=0.20, Test=0.20 2025-04-18 03:31:12,148 - root - INFO - Data split complete: 2025-04-18 03:31:12,148 - root - INFO - Train: X=(5893, 20), y=(5893, 2) (2025-03-01 00:00:00+00:00 to 2025-03-05 06:01:00+00:00) 2025-04-18 03:31:12,148 - root - INFO - Val: X=(1964, 20), y=(1964, 2) (2025-03-05 06:02:00+00:00 to 2025-03-06 14:55:00+00:00) 2025-04-18 03:31:12,148 - root - INFO - Test: X=(1965, 20), y=(1965, 2) (2025-03-06 14:56:00+00:00 to 2025-03-07 23:53:00+00:00) 2025-04-18 03:31:12,148 - root - INFO - --- Stage: Selecting and Pruning Features --- 2025-04-18 03:31:12,148 - gru_sac_predictor.src.feature_engineer - INFO - --- Selecting Features (LogReg L1 + VIF) --- 2025-04-18 03:31:12,148 - gru_sac_predictor.src.feature_engineer - INFO - Starting selection from 20 raw features. 2025-04-18 03:31:12,149 - gru_sac_predictor.src.feature_engineer - INFO - Performing Logistic Regression (L1, C=0.1) selection... 2025-04-18 03:31:13,975 - gru_sac_predictor.src.feature_engineer - INFO - Features selected by LogReg L1: ['open', 'high', 'low', 'close', 'volume', 'hour_sin', 'hour_cos', 'chaikin_AD_10', 'svi_10', 'gap_imbalance', 'return_1m', 'return_15m', 'return_60m', 'ATR_14', 'volatility_14d', 'EMA_10', 'EMA_50', 'MACD', 'MACD_signal', 'RSI_14'] 2025-04-18 03:31:13,976 - gru_sac_predictor.src.feature_engineer - INFO - Candidate whitelist after LogReg (20 features): ['ATR_14', 'EMA_10', 'EMA_50', 'MACD', 'MACD_signal', 'RSI_14', 'chaikin_AD_10', 'close', 'gap_imbalance', 'high', 'hour_cos', 'hour_sin', 'low', 'open', 'return_15m', 'return_1m', 'return_60m', 'svi_10', 'volatility_14d', 'volume'] 2025-04-18 03:31:13,976 - gru_sac_predictor.src.feature_engineer - INFO - Performing VIF filtering (threshold=10.0) on candidate features... 2025-04-18 03:31:14,095 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'low' due to high VIF (13101.36 > 10.0)... 2025-04-18 03:31:14,206 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'high' due to high VIF (12437.61 > 10.0)... 2025-04-18 03:31:14,308 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'EMA_10' due to high VIF (8173.76 > 10.0)... 2025-04-18 03:31:14,401 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'open' due to high VIF (2292.25 > 10.0)... 2025-04-18 03:31:14,483 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'close' due to high VIF (482.83 > 10.0)... 2025-04-18 03:31:14,556 - gru_sac_predictor.src.feature_engineer - INFO - Removing feature 'MACD' due to high VIF (71.12 > 10.0)... 2025-04-18 03:31:14,623 - gru_sac_predictor.src.feature_engineer - INFO - VIF filtering complete. Max VIF = 3.97 <= 10.0. 2025-04-18 03:31:14,624 - gru_sac_predictor.src.feature_engineer - INFO - Final whitelist after VIF filtering (14 features): ['ATR_14', 'EMA_50', 'MACD_signal', 'RSI_14', 'chaikin_AD_10', 'gap_imbalance', 'hour_cos', 'hour_sin', 'return_15m', 'return_1m', 'return_60m', 'svi_10', 'volatility_14d', 'volume'] 2025-04-18 03:31:14,625 - root - INFO - Saved final feature whitelist (14 features) to /home/yasha/develop/gru_sac_predictor/models/20250418_033111/final_whitelist_20250418_033111.json 2025-04-18 03:31:14,626 - root - INFO - Pruning feature sets using final whitelist: ['ATR_14', 'EMA_50', 'MACD_signal', 'RSI_14', 'chaikin_AD_10', 'gap_imbalance', 'hour_cos', 'hour_sin', 'return_15m', 'return_1m', 'return_60m', 'svi_10', 'volatility_14d', 'volume'] 2025-04-18 03:31:14,631 - root - INFO - Feature shapes after pruning: Train=(5893, 14), Val=(1964, 14), Test=(1965, 14) 2025-04-18 03:31:14,632 - root - INFO - --- Stage: Scaling Features --- 2025-04-18 03:31:14,633 - root - INFO - Fitting StandardScaler on training data (numeric columns only)... 2025-04-18 03:31:14,640 - root - INFO - Feature scaler saved to /home/yasha/develop/gru_sac_predictor/models/20250418_033111/feature_scaler_20250418_033111.joblib 2025-04-18 03:31:14,660 - root - INFO - Features scaled successfully. 2025-04-18 03:31:14,660 - root - INFO - --- Stage: Creating Sequences --- 2025-04-18 03:31:14,661 - root - INFO - Creating sequences with lookback=60 2025-04-18 03:31:14,949 - root - INFO - Sequence shapes created: 2025-04-18 03:31:14,949 - root - INFO - Train: X=(5833, 60, 14), y_ret=(5833,), y_dir=(5833,) 2025-04-18 03:31:14,949 - root - INFO - Val: X=(1904, 60, 14), y_ret=(1904,), y_dir=(1904,) 2025-04-18 03:31:14,950 - root - INFO - Test: X=(1905, 60, 14), y_ret=(1905,), y_dir=(1905,) 2025-04-18 03:31:14,950 - root - INFO - --- Stage: Training or Loading GRU Model --- 2025-04-18 03:31:14,950 - root - INFO - Attempting to train a new GRU model for run 20250418_033111... 2025-04-18 03:31:14,950 - gru_sac_predictor.src.gru_model_handler - INFO - Building GRU model: lookback=60, n_features=14 2025-04-18 03:31:16,165 - gru_sac_predictor.src.gru_model_handler - INFO - Model built successfully. 2025-04-18 03:31:16,172 - gru_sac_predictor.src.gru_model_handler - INFO - Model: "crypto_gru" ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ input (InputLayer) │ (None, 60, 14) │ 0 │ - │ ├──────────────────────────┼───────────────────────┼───────────────┼───────────────────────┤ │ gru (GRU) │ (None, 64) │ 15,360 │ input[0][0] │ ├──────────────────────────┼───────────────────────┼───────────────┼───────────────────────┤ │ gauss_params (Dense) │ (None, 2) │ 130 │ gru[0][0] │ ├──────────────────────────┼───────────────────────┼───────────────┼───────────────────────┤ │ ret (Lambda) │ (None, 1) │ 0 │ gauss_params[0][0] │ ├──────────────────────────┼───────────────────────┼───────────────┼───────────────────────┤ │ dir (Dense) │ (None, 1) │ 65 │ gru[0][0] │ └──────────────────────────┴───────────────────────┴───────────────┴───────────────────────┘ Total params: 15,555 (60.76 KB) Trainable params: 15,555 (60.76 KB) Non-trainable params: 0 (0.00 B) 2025-04-18 03:31:16,173 - gru_sac_predictor.src.gru_model_handler - INFO - Starting GRU training: epochs=25, batch=256, patience=5 2025-04-18 03:31:16,173 - gru_sac_predictor.src.gru_model_handler - INFO - Train X shape: (5833, 60, 14) 2025-04-18 03:31:16,173 - gru_sac_predictor.src.gru_model_handler - INFO - Val X shape: (1904, 60, 14) 2025-04-18 03:31:16,173 - gru_sac_predictor.src.gru_model_handler - INFO - Train y keys: ['ret', 'gauss_params', 'dir'] 2025-04-18 03:31:16,173 - gru_sac_predictor.src.gru_model_handler - INFO - Val y keys: ['ret', 'gauss_params', 'dir'] 2025-04-18 03:31:25,608 - gru_sac_predictor.src.gru_model_handler - INFO - GRU training finished. 2025-04-18 03:31:25,608 - gru_sac_predictor.src.gru_model_handler - INFO - Best validation loss: -0.7041 2025-04-18 03:31:25,647 - gru_sac_predictor.src.gru_model_handler - INFO - GRU model saved successfully to: /home/yasha/develop/gru_sac_predictor/models/20250418_033111/gru_model_20250418_033111.keras 2025-04-18 03:31:25,647 - root - INFO - Newly trained GRU model saved to /home/yasha/develop/gru_sac_predictor/models/20250418_033111/gru_model_20250418_033111.keras 2025-04-18 03:31:25,647 - root - INFO - Using GRU model trained in current run: 20250418_033111 2025-04-18 03:31:25,647 - root - INFO - --- Stage: Baseline Checks (Placeholder) --- 2025-04-18 03:31:25,647 - root - WARNING - Baseline checks stage not implemented. 2025-04-18 03:31:25,647 - root - INFO - --- Stage: Calibrating Probabilities --- 2025-04-18 03:31:25,648 - root - INFO - No existing calibration temperature found for run 20250418_033111 at /home/yasha/develop/gru_sac_predictor/models/run_20250418_033111/calibration_temp_20250418_033111.npy. 2025-04-18 03:31:25,648 - root - INFO - Calculating optimal temperature on validation set... 2025-04-18 03:31:25,648 - gru_sac_predictor.src.gru_model_handler - INFO - Generating predictions for 1904 samples... 2025-04-18 03:31:25,916 - gru_sac_predictor.src.gru_model_handler - INFO - Predictions generated successfully. 2025-04-18 03:31:25,916 - gru_sac_predictor.src.calibrator - INFO - Optimizing calibration temperature using 1904 samples... 2025-04-18 03:31:25,921 - gru_sac_predictor.src.calibrator - INFO - Optimal temperature found: T = 10.0000 2025-04-18 03:31:25,922 - root - INFO - Saved newly calculated calibration temperature T=10.0000 to /home/yasha/develop/gru_sac_predictor/models/20250418_033111/calibration_temp_20250418_033111.npy 2025-04-18 03:31:25,922 - root - INFO - Generating validation reliability curve plot... 2025-04-18 03:31:26,138 - gru_sac_predictor.src.calibrator - INFO - Reliability curve saved to /home/yasha/develop/gru_sac_predictor/results/20250418_033111/reliability_curve_val_20250418_033111.png 2025-04-18 03:31:26,138 - root - INFO - --- Stage: Training or Loading SAC Agent --- 2025-04-18 03:31:26,138 - root - INFO - SAC training is enabled. Instantiating SACTrainer... 2025-04-18 03:31:26,138 - gru_sac_predictor.src.sac_trainer - INFO - Initializing SACTrainer with Run ID: sac_train_20250418_033126 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - SAC Models Dir: /home/yasha/develop/gru_sac_predictor/models/sac_train_20250418_033126 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - SAC Logs Dir: /home/yasha/develop/gru_sac_predictor/logs/20250418_033111/sac_train_20250418_033126 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - SAC Results Dir:/home/yasha/develop/gru_sac_predictor/results/20250418_033111/sac_train_20250418_033126 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - SAC TB Dir: /home/yasha/develop/gru_sac_predictor/logs/20250418_033111/sac_train_20250418_033126/tensorboard 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - === Starting SAC Training Process (SAC Run ID: sac_train_20250418_033126) === 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - Using artifacts from GRU Run ID: 20250418_033111 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - INFO - --- Loading Dependencies from GRU Run ID: 20250418_033111 --- 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - ERROR - Models directory for GRU run 20250418_033111 not found at: /home/yasha/develop/gru_sac_predictor/models/run_20250418_033111 2025-04-18 03:31:26,139 - gru_sac_predictor.src.sac_trainer - ERROR - Failed to load GRU dependencies. Aborting SAC training. 2025-04-18 03:31:26,139 - __main__ - ERROR - SAC training failed. Proceeding without a newly trained agent. 2025-04-18 03:31:26,139 - __main__ - ERROR - SAC training failed and no load path specified in config. Cannot proceed with backtesting. 2025-04-18 03:31:26,139 - root - INFO - --- Stage: Running Backtest --- 2025-04-18 03:31:26,139 - root - ERROR - Pipeline execution failed: 'NoneType' object has no attribute 'columns' Traceback (most recent call last): File "/home/yasha/develop/gru_sac_predictor/gru_sac_predictor/src/trading_pipeline.py", line 1053, in execute self.run_backtest() File "/home/yasha/develop/gru_sac_predictor/gru_sac_predictor/src/trading_pipeline.py", line 910, in run_backtest if 'close' in self.df_test.columns: # df_test has original columns before feature selection AttributeError: 'NoneType' object has no attribute 'columns' 2025-04-18 03:31:26,140 - root - ERROR - === Pipeline Execution Terminated Due to Error ===