research/stat_pairs_backtest
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

progress

Browse Source
This commit is contained in:
Oleg Sheynin
2025-07-10 18:14:37 +00:00
parent 46072e03a2
commit 85c9d2ab93
15 changed files with 578 additions and 227 deletions
Download Patch File Download Diff File Expand all files Collapse all files
research/notebooks/pt_pair_backtest.ipynb
+1390
View File
File diff suppressed because one or more lines are too long
research/notebooks/pt_sliding.ipynb
+858
View File
@@ -0,0 +1,858 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Sliding Fit Strategy Visualization Notebook\n",
"\n",
"This notebook is specifically designed for the SlidingFitStrategy, which uses a sliding window approach.\n",
"It re-trains the model every minute and shows how cointegration, model parameters, and trading signals evolve over time.\n",
"You can visualize the dynamic nature of the sliding window and how the relationship between instruments changes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎯 Key Features:\n",
"\n",
"1. **Interactive Configuration**: \n",
" - Easy switching between CRYPTO and EQUITY configurations\n",
" - Simple parameter adjustment for thresholds and training periods\n",
"\n",
"2. **Single Pair Focus**: \n",
" - Instead of running multiple pairs, focuses on one pair at a time\n",
" - Allows deep analysis of the relationship between two instruments\n",
"\n",
"3. **Step-by-Step Visualization**:\n",
" - **Raw price data**: Individual prices, normalized comparison, and price ratios\n",
" - **Training analysis**: Cointegration testing and VECM model fitting\n",
" - **Dis-equilibrium visualization**: Both raw and scaled dis-equilibrium with threshold lines\n",
" - **Strategy execution**: Trading signal generation and visualization\n",
" - **Prediction analysis**: Actual vs predicted prices with trading signals overlaid\n",
"\n",
"4. **Rich Analytics**:\n",
" - Cointegration status and VECM model details\n",
" - Statistical summaries for all stages\n",
" - Threshold crossing analysis\n",
" - Trading signal breakdown\n",
"\n",
"5. **Interactive Experimentation**:\n",
" - Easy parameter modification\n",
" - Re-run capabilities for different configurations\n",
" - Support for both StaticFitStrategy and SlidingFitStrategy\n",
"\n",
"### 🚀 How to Use:\n",
"\n",
"1. **Start Jupyter**:\n",
" ```bash\n",
" cd src/notebooks\n",
" jupyter notebook pairs_trading_visualization.ipynb\n",
" ```\n",
"\n",
"2. **Customize Your Analysis**:\n",
" - Change `SYMBOL_A` and `SYMBOL_B` to your desired trading pair\n",
" - Switch between `CRYPTO_CONFIG` and `EQT_CONFIG`\n",
" - Choose your strategy (StaticFitStrategy or SlidingFitStrategy)\n",
" - Adjust thresholds and parameters as needed\n",
"\n",
"3. **Run and Visualize**:\n",
" - Execute cells step by step to see the analysis unfold\n",
" - Rich matplotlib visualizations show relationships and signals\n",
" - Comprehensive summary at the end\n",
"\n",
"The notebook provides exactly what you requested - a way to visualize the relationship between two instruments and their scaled dis-equilibrium, with all the stages of your pairs trading strategy clearly displayed and analyzed.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup and Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Trading Parameters Configuration\n",
"# Specify your configuration file, trading symbols and date here\n",
"\n",
"# Configuration file selection\n",
"CONFIG_FILE = \"equity\" # Options: \"equity\", \"crypto\", or custom filename (without .cfg extension)\n",
"\n",
"# Trading pair symbols\n",
"SYMBOL_A = \"COIN\" # Change this to your desired symbol A\n",
"SYMBOL_B = \"MSTR\" # Change this to your desired symbol B\n",
"\n",
"# Date for data file selection (format: YYYYMMDD)\n",
"TRADING_DATE = \"20250605\" # Change this to your desired date\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import os\n",
"sys.path.append('..')\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"from typing import Dict, List, Optional\n",
"from IPython.display import clear_output\n",
"\n",
"# Import our modules\n",
"from pt_trading.fit_methods import SlidingFit, PairState\n",
"from tools.data_loader import load_market_data\n",
"from pt_trading.trading_pair import TradingPair\n",
"from pt_trading.results import BacktestResult\n",
"\n",
"# Set plotting style\n",
"plt.style.use('seaborn-v0_8')\n",
"sns.set_palette(\"husl\")\n",
"plt.rcParams['figure.figsize'] = (15, 10)\n",
"\n",
"print(\"Setup complete!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Load Configuration from Configuration Files using HJSON\n",
"import hjson\n",
"import os\n",
"\n",
"def load_config_from_file(config_type=\"equity\"):\n",
" \"\"\"Load configuration from configuration files using HJSON\"\"\"\n",
" config_file = f\"../../configuration/{config_type}.cfg\"\n",
" \n",
" try:\n",
" with open(config_file, 'r') as f:\n",
" # HJSON handles comments, trailing commas, and other human-friendly features\n",
" config = hjson.load(f)\n",
" \n",
" # Convert relative paths to absolute paths from notebook perspective\n",
" if 'data_directory' in config:\n",
" data_dir = config['data_directory']\n",
" if data_dir.startswith('./'):\n",
" # Convert relative path to absolute path from notebook's perspective\n",
" config['data_directory'] = os.path.abspath(f\"../../{data_dir[2:]}\")\n",
" \n",
" return config\n",
" \n",
" except FileNotFoundError:\n",
" print(f\"Configuration file not found: {config_file}\")\n",
" return None\n",
" except hjson.HjsonDecodeError as e:\n",
" print(f\"HJSON parsing error in {config_file}: {e}\")\n",
" return None\n",
" except Exception as e:\n",
" print(f\"Unexpected error loading config from {config_file}: {e}\")\n",
" return None\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(f\"Trading Parameters:\")\n",
"print(f\" Configuration: {CONFIG_FILE}\")\n",
"print(f\" Symbol A: {SYMBOL_A}\")\n",
"print(f\" Symbol B: {SYMBOL_B}\")\n",
"print(f\" Trading Date: {TRADING_DATE}\")\n",
"\n",
"# Load the specified configuration\n",
"print(f\"\\nLoading {CONFIG_FILE} configuration using HJSON...\")\n",
"test_config = load_config_from_file(CONFIG_FILE)\n",
"assert test_config is not None\n",
"BT_TEST_CONFIG = test_config\n",
"\n",
"if BT_TEST_CONFIG:\n",
" print(f\"✓ Successfully loaded {BT_TEST_CONFIG['security_type']} configuration\")\n",
" print(f\" Data directory: {BT_TEST_CONFIG['data_directory']}\")\n",
" print(f\" Database table: {BT_TEST_CONFIG['db_table_name']}\")\n",
" print(f\" Exchange: {BT_TEST_CONFIG['exchange_id']}\")\n",
" print(f\" Training window: {BT_TEST_CONFIG['training_minutes']} minutes\")\n",
" print(f\" Open threshold: {BT_TEST_CONFIG['dis-equilibrium_open_trshld']}\")\n",
" print(f\" Close threshold: {BT_TEST_CONFIG['dis-equilibrium_close_trshld']}\")\n",
" \n",
" # Automatically construct data file name based on date and config type\n",
" # if CONFIG['security_type'] == \"CRYPTO\":\n",
" DATA_FILE = f\"{TRADING_DATE}.mktdata.ohlcv.db\"\n",
" # elif CONFIG['security_type'] == \"EQUITY\":\n",
" # DATA_FILE = f\"{TRADING_DATE}.alpaca_sim_md.db\"\n",
" # else:\n",
" # DATA_FILE = f\"{TRADING_DATE}.mktdata.db\" # Default fallback\n",
"\n",
" # Update CONFIG with the specific data file and instruments\n",
" BT_TEST_CONFIG[\"datafiles\"] = [DATA_FILE]\n",
" BT_TEST_CONFIG[\"instruments\"] = [SYMBOL_A, SYMBOL_B]\n",
" \n",
" print(f\"\\nData Configuration:\")\n",
" print(f\" Data File: {DATA_FILE}\")\n",
" print(f\" Security Type: {BT_TEST_CONFIG['security_type']}\")\n",
" \n",
" # Verify data file exists\n",
" import os\n",
" data_file_path = f\"{BT_TEST_CONFIG['data_directory']}/{DATA_FILE}\"\n",
" if os.path.exists(data_file_path):\n",
" print(f\" ✓ Data file found: {data_file_path}\")\n",
" else:\n",
" print(f\" ⚠ Data file not found: {data_file_path}\")\n",
" print(f\" Please check if the date and file exist in the data directory\")\n",
" \n",
" # List available files in the data directory\n",
" try:\n",
" data_dir = BT_TEST_CONFIG['data_directory']\n",
" if os.path.exists(data_dir):\n",
" available_files = [f for f in os.listdir(data_dir) if f.endswith('.db')]\n",
" print(f\" Available files in {data_dir}:\")\n",
" for file in sorted(available_files)[:5]: # Show first 5 files\n",
" print(f\" - {file}\")\n",
" if len(available_files) > 5:\n",
" print(f\" ... and {len(available_files)-5} more files\")\n",
" except Exception as e:\n",
" print(f\" Could not list files in data directory: {e}\")\n",
"else:\n",
" print(\"⚠ Failed to load configuration. Please check the configuration file.\")\n",
" print(\"Available configuration files:\")\n",
" config_dir = \"../../configuration\"\n",
" if os.path.exists(config_dir):\n",
" config_files = [f for f in os.listdir(config_dir) if f.endswith('.cfg')]\n",
" for file in config_files:\n",
" print(f\" - {file}\")\n",
" else:\n",
" print(f\" Configuration directory not found: {config_dir}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Select Trading Pair and Initialize Strategy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Initialize Strategy\n",
"# Trading pair and data file are now defined in the previous cell\n",
"\n",
"# Initialize SlidingFitStrategy\n",
"STRATEGY = SlidingFitStrategy()\n",
"\n",
"print(f\"Strategy Initialization:\")\n",
"print(f\" Selected pair: {SYMBOL_A} & {SYMBOL_B}\")\n",
"print(f\" Data file: {DATA_FILE}\")\n",
"print(f\" Strategy: {type(STRATEGY).__name__}\")\n",
"print(f\"\\nStrategy characteristics:\")\n",
"print(f\" - Sliding window training every minute\")\n",
"print(f\" - Dynamic cointegration testing\")\n",
"print(f\" - State-based position management\")\n",
"print(f\" - Continuous model re-training\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load and Prepare Market Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load market data\n",
"datafile_path = f\"{BT_TEST_CONFIG['data_directory']}/{DATA_FILE}\"\n",
"print(f\"Loading data from: {datafile_path}\")\n",
"\n",
"market_data_df = load_market_data(datafile_path, config=BT_TEST_CONFIG)\n",
"\n",
"print(f\"Loaded {len(market_data_df)} rows of market data\")\n",
"print(f\"Symbols in data: {market_data_df['symbol'].unique()}\")\n",
"print(f\"Time range: {market_data_df['tstamp'].min()} to {market_data_df['tstamp'].max()}\")\n",
"\n",
"# Create trading pair\n",
"pair = TradingPair(\n",
" market_data=market_data_df,\n",
" symbol_a=SYMBOL_A,\n",
" symbol_b=SYMBOL_B,\n",
" price_column=BT_TEST_CONFIG[\"price_column\"]\n",
")\n",
"\n",
"print(f\"\\nCreated trading pair: {pair}\")\n",
"print(f\"Market data shape: {pair.market_data_.shape}\")\n",
"print(f\"Column names: {pair.colnames()}\")\n",
"\n",
"# Calculate maximum possible iterations for sliding window\n",
"training_minutes = BT_TEST_CONFIG[\"training_minutes\"]\n",
"max_iterations = len(pair.market_data_) - training_minutes\n",
"print(f\"\\nSliding window analysis:\")\n",
"print(f\" Training window size: {training_minutes} minutes\")\n",
"print(f\" Maximum iterations: {max_iterations}\")\n",
"print(f\" Total analysis time: ~{max_iterations} minutes\")\n",
"\n",
"# Display sample data\n",
"print(f\"\\nSample data:\")\n",
"display(pair.market_data_.head())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run SlidingFitStrategy with Real-Time Visualization"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Run the sliding strategy with detailed tracking\n",
"print(f\"Running SlidingFitStrategy on {pair}...\")\n",
"print(f\"This will process {max_iterations} minutes of data with sliding training windows.\\n\")\n",
"\n",
"# Initialize tracking variables\n",
"iteration_data = []\n",
"cointegration_history = []\n",
"beta_history = []\n",
"alpha_history = []\n",
"state_history = []\n",
"disequilibrium_history = []\n",
"scaled_disequilibrium_history = []\n",
"timestamp_history = []\n",
"training_mu_history = []\n",
"training_std_history = []\n",
"\n",
"# Initialize the strategy state\n",
"pair.user_data_['state'] = PairState.INITIAL\n",
"pair.user_data_[\"trades\"] = pd.DataFrame(columns=pd.Index(STRATEGY.TRADES_COLUMNS, dtype=str))\n",
"pair.user_data_[\"is_cointegrated\"] = False\n",
"\n",
"bt_result = BacktestResult(config=BT_TEST_CONFIG)\n",
"training_minutes = BT_TEST_CONFIG[\"training_minutes\"]\n",
"open_threshold = BT_TEST_CONFIG[\"dis-equilibrium_open_trshld\"]\n",
"close_threshold = BT_TEST_CONFIG[\"dis-equilibrium_close_trshld\"]\n",
"\n",
"# Limit iterations for demonstration (change this to max_iterations for full run)\n",
"max_demo_iterations = min(200, max_iterations) # Process first 200 minutes\n",
"print(f\"Processing first {max_demo_iterations} iterations for demonstration...\\n\")\n",
"\n",
"for curr_training_start_idx in range(max_demo_iterations):\n",
" if curr_training_start_idx % 20 == 0:\n",
" print(f\"Processing iteration {curr_training_start_idx}/{max_demo_iterations}...\")\n",
"\n",
" # Get datasets for this iteration\n",
" pair.get_datasets(\n",
" training_minutes=training_minutes,\n",
" training_start_index=curr_training_start_idx,\n",
" testing_size=1\n",
" )\n",
"\n",
" if len(pair.training_df_) < training_minutes:\n",
" print(f\"Iteration {curr_training_start_idx}: Not enough training data. Stopping.\")\n",
" break\n",
"\n",
" # Record timestamp for this iteration\n",
" current_timestamp = pair.testing_df_['tstamp'].iloc[0] if len(pair.testing_df_) > 0 else None\n",
" timestamp_history.append(current_timestamp)\n",
"\n",
" # Train and test cointegration\n",
" try:\n",
" is_cointegrated = pair.train_pair()\n",
" cointegration_history.append(is_cointegrated)\n",
"\n",
" if is_cointegrated:\n",
" # Record model parameters\n",
" beta_history.append(pair.vecm_fit_.beta.flatten())\n",
" alpha_history.append(pair.vecm_fit_.alpha.flatten())\n",
" training_mu_history.append(pair.training_mu_)\n",
" training_std_history.append(pair.training_std_)\n",
"\n",
" # Generate prediction for current minute\n",
" pair.predict()\n",
"\n",
" if len(pair.predicted_df_) > 0:\n",
" current_disequilibrium = pair.predicted_df_['disequilibrium'].iloc[0]\n",
" current_scaled_disequilibrium = pair.predicted_df_['scaled_disequilibrium'].iloc[0]\n",
" disequilibrium_history.append(current_disequilibrium)\n",
" scaled_disequilibrium_history.append(current_scaled_disequilibrium)\n",
" else:\n",
" disequilibrium_history.append(np.nan)\n",
" scaled_disequilibrium_history.append(np.nan)\n",
" else:\n",
" # No cointegration\n",
" beta_history.append(None)\n",
" alpha_history.append(None)\n",
" training_mu_history.append(np.nan)\n",
" training_std_history.append(np.nan)\n",
" disequilibrium_history.append(np.nan)\n",
" scaled_disequilibrium_history.append(np.nan)\n",
"\n",
" except Exception as e:\n",
" print(f\"Iteration {curr_training_start_idx}: Training failed: {str(e)}\")\n",
" cointegration_history.append(False)\n",
" beta_history.append(None)\n",
" alpha_history.append(None)\n",
" training_mu_history.append(np.nan)\n",
" training_std_history.append(np.nan)\n",
" disequilibrium_history.append(np.nan)\n",
" scaled_disequilibrium_history.append(np.nan)\n",
"\n",
" # Record current state\n",
" current_state = pair.user_data_.get('state', PairState.INITIAL)\n",
" state_history.append(current_state)\n",
"\n",
"print(f\"\\nCompleted {len(cointegration_history)} iterations\")\n",
"print(f\"Cointegration rate: {sum(cointegration_history)/len(cointegration_history)*100:.1f}%\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize Sliding Window Results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create comprehensive visualization of sliding window results\n",
"fig, axes = plt.subplots(6, 1, figsize=(18, 24))\n",
"\n",
"# Filter valid timestamps\n",
"valid_timestamps = [ts for ts in timestamp_history if ts is not None]\n",
"n_points = len(valid_timestamps)\n",
"\n",
"if n_points == 0:\n",
" print(\"No valid data points to visualize\")\n",
"else:\n",
" # 1. Cointegration Status Over Time\n",
" cointegration_values = [1 if coint else 0 for coint in cointegration_history[:n_points]]\n",
" axes[0].plot(valid_timestamps, cointegration_values, 'o-', alpha=0.7, markersize=3)\n",
" axes[0].fill_between(valid_timestamps, cointegration_values, alpha=0.3)\n",
" axes[0].set_title('Cointegration Status Over Time (1=Cointegrated, 0=Not Cointegrated)')\n",
" axes[0].set_ylabel('Cointegrated')\n",
" axes[0].set_ylim(-0.1, 1.1)\n",
" axes[0].grid(True)\n",
"\n",
" # 2. Beta Coefficients Evolution\n",
" valid_betas = []\n",
" beta_timestamps = []\n",
" for i, beta in enumerate(beta_history[:n_points]):\n",
" if beta is not None and i < len(valid_timestamps):\n",
" valid_betas.append(beta)\n",
" beta_timestamps.append(valid_timestamps[i])\n",
"\n",
" if valid_betas:\n",
" beta_array = np.array(valid_betas)\n",
" axes[1].plot(beta_timestamps, beta_array[:, 1], 'o-', alpha=0.7, markersize=2,\n",
" label='Beta[1]', color='red')\n",
" axes[1].set_title('VECM Beta[1] Coefficient Evolution (Beta[0] = 1.0 by normalization)')\n",
" axes[1].set_ylabel('Beta[1] Value')\n",
" axes[1].legend()\n",
" axes[1].grid(True)\n",
"\n",
" # 3. Training Mean and Std Evolution\n",
" valid_mu = [mu for mu in training_mu_history[:n_points] if not np.isnan(mu)]\n",
" valid_std = [std for std in training_std_history[:n_points] if not np.isnan(std)]\n",
" mu_timestamps = [valid_timestamps[i] for i, mu in enumerate(training_mu_history[:n_points]) if not np.isnan(mu)]\n",
"\n",
" if valid_mu:\n",
" axes[2].plot(mu_timestamps, valid_mu, 'b-', alpha=0.7, label='Training Mean', linewidth=1)\n",
" ax2_twin = axes[2].twinx()\n",
" ax2_twin.plot(mu_timestamps, valid_std, 'r-', alpha=0.7, label='Training Std', linewidth=1)\n",
" axes[2].set_title('Training Dis-equilibrium Statistics Evolution')\n",
" axes[2].set_ylabel('Mean', color='b')\n",
" ax2_twin.set_ylabel('Std', color='r')\n",
" axes[2].grid(True)\n",
" axes[2].legend(loc='upper left')\n",
" ax2_twin.legend(loc='upper right')\n",
"\n",
" # 4. Raw Dis-equilibrium Over Time\n",
" valid_diseq = [diseq for diseq in disequilibrium_history[:n_points] if not np.isnan(diseq)]\n",
" diseq_timestamps = [valid_timestamps[i] for i, diseq in enumerate(disequilibrium_history[:n_points]) if not np.isnan(diseq)]\n",
"\n",
" if valid_diseq:\n",
" axes[3].plot(diseq_timestamps, valid_diseq, 'g-', alpha=0.7, linewidth=1)\n",
" # Add rolling mean\n",
" if len(valid_diseq) > 10:\n",
" rolling_mean = pd.Series(valid_diseq).rolling(window=10, min_periods=1).mean()\n",
" axes[3].plot(diseq_timestamps, rolling_mean, 'r-', alpha=0.8, linewidth=2, label='10-period MA')\n",
" axes[3].legend()\n",
" axes[3].set_title('Raw Dis-equilibrium Over Time')\n",
" axes[3].set_ylabel('Dis-equilibrium')\n",
" axes[3].grid(True)\n",
"\n",
" # 5. Scaled Dis-equilibrium with Thresholds\n",
" valid_scaled_diseq = [diseq for diseq in scaled_disequilibrium_history[:n_points] if not np.isnan(diseq)]\n",
" scaled_diseq_timestamps = [valid_timestamps[i] for i, diseq in enumerate(scaled_disequilibrium_history[:n_points]) if not np.isnan(diseq)]\n",
"\n",
" if valid_scaled_diseq:\n",
" axes[4].plot(scaled_diseq_timestamps, valid_scaled_diseq, 'purple', alpha=0.7, linewidth=1)\n",
" axes[4].axhline(y=open_threshold, color='red', linestyle='--', alpha=0.8,\n",
" label=f'Open Threshold ({open_threshold})')\n",
" axes[4].axhline(y=close_threshold, color='blue', linestyle='--', alpha=0.8,\n",
" label=f'Close Threshold ({close_threshold})')\n",
" axes[4].axhline(y=0, color='black', linestyle='-', alpha=0.5, linewidth=0.5)\n",
" axes[4].set_title('Scaled Dis-equilibrium with Trading Thresholds')\n",
" axes[4].set_ylabel('Scaled Dis-equilibrium')\n",
" axes[4].legend()\n",
" axes[4].grid(True)\n",
"\n",
" # 6. Price Data with Training Windows\n",
" # Show original price data with indication of training windows\n",
" colname_a, colname_b = pair.colnames()\n",
" price_data = pair.market_data_[:n_points + training_minutes].copy()\n",
"\n",
" axes[5].plot(price_data['tstamp'], price_data[colname_a], alpha=0.7, label=f'{SYMBOL_A}', linewidth=1)\n",
" axes[5].plot(price_data['tstamp'], price_data[colname_b], alpha=0.7, label=f'{SYMBOL_B}', linewidth=1)\n",
"\n",
" # Highlight training windows\n",
" for i in range(0, min(n_points, 10), max(1, n_points//20)): # Show every 20th window\n",
" start_idx = i\n",
" end_idx = i + training_minutes\n",
" if end_idx < len(price_data):\n",
" window_data = price_data.iloc[start_idx:end_idx]\n",
" axes[5].axvspan(window_data['tstamp'].iloc[0], window_data['tstamp'].iloc[-1],\n",
" alpha=0.1, color='gray')\n",
"\n",
" axes[5].set_title(f'Price Data with Training Windows (Gray bands show some training windows)')\n",
" axes[5].set_ylabel('Price')\n",
" axes[5].set_xlabel('Time')\n",
" axes[5].legend()\n",
" axes[5].grid(True)\n",
"\n",
"plt.tight_layout()\n",
"plt.show()\n",
"\n",
"# Print summary statistics\n",
"print(f\"\\n\" + \"=\"*80)\n",
"print(f\"SLIDING WINDOW ANALYSIS SUMMARY\")\n",
"print(f\"=\"*80)\n",
"print(f\"Total iterations processed: {n_points}\")\n",
"print(f\"Cointegration episodes: {sum(cointegration_history[:n_points])}\")\n",
"print(f\"Cointegration rate: {sum(cointegration_history[:n_points])/n_points*100:.1f}%\")\n",
"if valid_betas:\n",
" print(f\"Beta coefficient stability: Std = {np.std(beta_array, axis=0)}\")\n",
"if valid_scaled_diseq:\n",
" threshold_breaches = sum(1 for x in valid_scaled_diseq if abs(x) > open_threshold)\n",
" print(f\"Open threshold breaches: {threshold_breaches} ({threshold_breaches/len(valid_scaled_diseq)*100:.1f}%)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Analyze Training Window Evolution"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Detailed analysis of how training windows evolve\n",
"print(\"TRAINING WINDOW EVOLUTION ANALYSIS\")\n",
"print(\"=\" * 50)\n",
"\n",
"# Analyze cointegration stability\n",
"if len(cointegration_history) > 1:\n",
" # Find cointegration change points\n",
" change_points = []\n",
" for i in range(1, len(cointegration_history)):\n",
" if cointegration_history[i] != cointegration_history[i - 1]:\n",
" change_points.append(\n",
" (\n",
" i,\n",
" cointegration_history[i],\n",
" valid_timestamps[i] if i < len(valid_timestamps) else None,\n",
" )\n",
" )\n",
"\n",
" print(f\"\\nCointegration Change Points:\")\n",
" if change_points:\n",
" for idx, status, timestamp in change_points[:10]: # Show first 10\n",
" status_str = \"GAINED\" if status else \"LOST\"\n",
" print(f\" Iteration {idx}: {status_str} cointegration at {timestamp}\")\n",
" if len(change_points) > 10:\n",
" print(f\" ... and {len(change_points)-10} more changes\")\n",
" else:\n",
" print(f\" No cointegration changes detected\")\n",
"\n",
"# Analyze beta stability when cointegrated\n",
"if valid_betas and len(valid_betas) > 10:\n",
" beta_df = pd.DataFrame(\n",
" valid_betas,\n",
" columns=pd.Index([f\"Beta_{i}\" for i in range(len(valid_betas[0]))], dtype=str),\n",
" )\n",
" beta_df[\"timestamp\"] = beta_timestamps\n",
"\n",
" print(f\"\\nBeta Coefficient Analysis:\")\n",
" print(f\" Number of valid beta estimates: {len(valid_betas)}\")\n",
" print(f\" Beta statistics:\")\n",
" for col in beta_df.columns[:-1]: # Exclude timestamp\n",
" print(\n",
" f\" {col}: Mean={beta_df[col].mean():.4f}, Std={beta_df[col].std():.4f}\"\n",
" )\n",
"\n",
" # Check for beta regime changes\n",
" beta_changes = []\n",
" threshold = 0.1 # 10% change threshold\n",
" for i in range(1, len(valid_betas)):\n",
" if np.any(\n",
" np.abs(np.array(valid_betas[i]) - np.array(valid_betas[i - 1])) > threshold\n",
" ):\n",
" beta_changes.append(i)\n",
"\n",
" print(f\" Significant beta changes (>{threshold*100}%): {len(beta_changes)}\")\n",
" if beta_changes:\n",
" print(\n",
" f\" Change frequency: {len(beta_changes)/len(valid_betas)*100:.1f}% of cointegrated periods\"\n",
" )\n",
"\n",
"# Analyze dis-equilibrium characteristics\n",
"if valid_scaled_diseq:\n",
" scaled_diseq_series = pd.Series(valid_scaled_diseq)\n",
"\n",
" print(f\"\\nDis-equilibrium Analysis:\")\n",
" print(f\" Mean: {scaled_diseq_series.mean():.4f}\")\n",
" print(f\" Std: {scaled_diseq_series.std():.4f}\")\n",
" print(f\" Min: {scaled_diseq_series.min():.4f}\")\n",
" print(f\" Max: {scaled_diseq_series.max():.4f}\")\n",
"\n",
" # Threshold analysis\n",
" open_breaches = sum(1 for x in valid_scaled_diseq if abs(x) >= open_threshold)\n",
" close_opportunities = sum(\n",
" 1 for x in valid_scaled_diseq if abs(x) <= close_threshold\n",
" )\n",
"\n",
" print(\n",
" f\" Open threshold breaches: {open_breaches} ({open_breaches/len(valid_scaled_diseq)*100:.1f}%)\"\n",
" )\n",
" print(\n",
" f\" Close opportunities: {close_opportunities} ({close_opportunities/len(valid_scaled_diseq)*100:.1f}%)\"\n",
" )\n",
"\n",
" # Mean reversion analysis\n",
" zero_crossings = 0\n",
" for i in range(1, len(valid_scaled_diseq)):\n",
" if (valid_scaled_diseq[i - 1] * valid_scaled_diseq[i]) < 0: # Sign change\n",
" zero_crossings += 1\n",
"\n",
" print(f\" Zero crossings (mean reversion events): {zero_crossings}\")\n",
" if zero_crossings > 0:\n",
" print(\n",
" f\" Average time between mean reversions: {len(valid_scaled_diseq)/zero_crossings:.1f} minutes\"\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Run Complete Strategy (Optional)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Optional: Run the complete strategy to generate actual trades\n",
"# Warning: This may take several minutes depending on data size\n",
"\n",
"RUN_COMPLETE_STRATEGY = False # Set to True to run full strategy\n",
"\n",
"if RUN_COMPLETE_STRATEGY:\n",
" print(\"Running complete SlidingFitStrategy...\")\n",
" print(\"This may take several minutes...\")\n",
"\n",
" # Reset strategy state\n",
" STRATEGY.curr_training_start_idx_ = 0\n",
"\n",
" # Create new pair and result objects\n",
" pair_full = TradingPair(\n",
" market_data=market_data_df,\n",
" symbol_a=SYMBOL_A,\n",
" symbol_b=SYMBOL_B,\n",
" price_column=BT_TEST_CONFIG[\"price_column\"]\n",
" )\n",
"\n",
" bt_result_full = BacktestResult(config=BT_TEST_CONFIG)\n",
"\n",
" # Run strategy\n",
" pair_trades = STRATEGY.run_pair(config=BT_TEST_CONFIG, pair=pair_full, bt_result=bt_result_full)\n",
"\n",
" if pair_trades is not None and len(pair_trades) > 0:\n",
" print(f\"\\nGenerated {len(pair_trades)} trading signals:\")\n",
" display(pair_trades)\n",
"\n",
" # Analyze trades\n",
" trade_times = pair_trades['time'].unique()\n",
" print(f\"\\nTrade Analysis:\")\n",
" print(f\" Unique trade times: {len(trade_times)}\")\n",
" print(f\" Trade frequency: {len(trade_times)/max_iterations*100:.2f}% of total periods\")\n",
"\n",
" # Group trades by time\n",
" for trade_time in trade_times[:5]: # Show first 5 trade times\n",
" trades_at_time = pair_trades[pair_trades['time'] == trade_time]\n",
" print(f\"\\n Trade at {trade_time}:\")\n",
" for _, trade in trades_at_time.iterrows():\n",
" print(f\" {trade['action']} {trade['symbol']} @ ${trade['price']:.2f} \"\n",
" f\"(dis-eq: {trade['scaled_disequilibrium']:.2f})\")\n",
" else:\n",
" print(\"\\nNo trading signals generated\")\n",
" print(\"Possible reasons:\")\n",
" print(\" - Insufficient cointegration periods\")\n",
" print(\" - Dis-equilibrium never exceeded thresholds\")\n",
" print(\" - Strategy-specific conditions not met\")\n",
"else:\n",
" print(\"Complete strategy execution is disabled.\")\n",
" print(\"Set RUN_COMPLETE_STRATEGY = True to run the full strategy.\")\n",
" print(\"Note: This may take several minutes depending on your data size.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interactive Parameter Analysis"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Interactive analysis for parameter optimization\n",
"print(\"PARAMETER SENSITIVITY ANALYSIS\")\n",
"print(\"=\"*40)\n",
"\n",
"print(f\"Current parameters:\")\n",
"print(f\" Training window: {BT_TEST_CONFIG['training_minutes']} minutes\")\n",
"print(f\" Open threshold: {BT_TEST_CONFIG['dis-equilibrium_open_trshld']}\")\n",
"print(f\" Close threshold: {BT_TEST_CONFIG['dis-equilibrium_close_trshld']}\")\n",
"\n",
"# Recommendations based on observed data\n",
"if valid_scaled_diseq:\n",
" diseq_stats = pd.Series(valid_scaled_diseq).describe()\n",
" print(f\"\\nObserved scaled dis-equilibrium statistics:\")\n",
" print(f\" 75th percentile: {diseq_stats['75%']:.2f}\")\n",
" print(f\" 95th percentile: {np.percentile(valid_scaled_diseq, 95):.2f}\")\n",
" print(f\" 99th percentile: {np.percentile(valid_scaled_diseq, 99):.2f}\")\n",
"\n",
" # Suggest optimal thresholds\n",
" suggested_open = np.percentile(np.abs(valid_scaled_diseq), 85)\n",
" suggested_close = np.percentile(np.abs(valid_scaled_diseq), 30)\n",
"\n",
" print(f\"\\nSuggested threshold optimization:\")\n",
" print(f\" Suggested open threshold: {suggested_open:.2f} (85th percentile)\")\n",
" print(f\" Suggested close threshold: {suggested_close:.2f} (30th percentile)\")\n",
"\n",
" if suggested_open != open_threshold or suggested_close != close_threshold:\n",
" print(f\"\\nTo test these parameters, modify the CONFIG dictionary:\")\n",
" print(f\" CONFIG['dis-equilibrium_open_trshld'] = {suggested_open:.2f}\")\n",
" print(f\" CONFIG['dis-equilibrium_close_trshld'] = {suggested_close:.2f}\")\n",
"\n",
"# Training window recommendations\n",
"if len(cointegration_history) > 0:\n",
" cointegration_rate = sum(cointegration_history)/len(cointegration_history)\n",
" print(f\"\\nTraining window analysis:\")\n",
" print(f\" Current cointegration rate: {cointegration_rate*100:.1f}%\")\n",
"\n",
" if cointegration_rate < 0.3:\n",
" print(f\" Recommendation: Consider increasing training window (current: {training_minutes})\")\n",
" print(f\" Suggested: {int(training_minutes * 1.5)} minutes\")\n",
" elif cointegration_rate > 0.8:\n",
" print(f\" Recommendation: Consider decreasing training window for more responsive model\")\n",
" print(f\" Suggested: {int(training_minutes * 0.75)} minutes\")\n",
" else:\n",
" print(f\" Current training window appears appropriate\")\n",
"\n",
"print(f\"\\nTo re-run analysis with different parameters:\")\n",
"print(f\"1. Modify the CONFIG dictionary above\")\n",
"print(f\"2. Re-run from the 'Run SlidingFitStrategy' cell\")\n",
"print(f\"3. Compare results with current analysis\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary and Conclusions\n",
"\n",
"This notebook demonstrates the SlidingFitStrategy's dynamic approach to pairs trading.\n",
"Key insights from the sliding window analysis:\n",
"\n",
"1. **Cointegration Stability**: How often the pair maintains cointegration\n",
"2. **Model Parameter Evolution**: How VECM coefficients change over time\n",
"3. **Threshold Effectiveness**: How well current thresholds capture trading opportunities\n",
"4. **Mean Reversion Patterns**: Frequency and timing of dis-equilibrium corrections\n",
"\n",
"The sliding approach allows for:\n",
"- **Adaptive modeling**: Responds to changing market conditions\n",
"- **Dynamic thresholding**: Can be optimized based on observed patterns\n",
"- **Real-time monitoring**: Provides continuous assessment of pair relationships\n",
"- **Risk management**: Early detection of cointegration breakdown"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "python3.12-venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
research/notebooks/pt_static.ipynb
+771
View File
@@ -0,0 +1,771 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pairs Trading Visualization Notebook\n",
"\n",
"This notebook allows you to visualize pairs trading strategies on individual instrument pairs.\n",
"You can examine the relationship between two instruments, their dis-equilibrium, and trading signals."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎯 Key Features:\n",
"\n",
"1. **Interactive Configuration**: \n",
" - Easy switching between CRYPTO and EQUITY configurations\n",
" - Simple parameter adjustment for thresholds and training periods\n",
"\n",
"2. **Single Pair Focus**: \n",
" - Instead of running multiple pairs, focuses on one pair at a time\n",
" - Allows deep analysis of the relationship between two instruments\n",
"\n",
"3. **Step-by-Step Visualization**:\n",
" - **Raw price data**: Individual prices, normalized comparison, and price ratios\n",
" - **Training analysis**: Cointegration testing and VECM model fitting\n",
" - **Dis-equilibrium visualization**: Both raw and scaled dis-equilibrium with threshold lines\n",
" - **Strategy execution**: Trading signal generation and visualization\n",
" - **Prediction analysis**: Actual vs predicted prices with trading signals overlaid\n",
"\n",
"4. **Rich Analytics**:\n",
" - Cointegration status and VECM model details\n",
" - Statistical summaries for all stages\n",
" - Threshold crossing analysis\n",
" - Trading signal breakdown\n",
"\n",
"5. **Interactive Experimentation**:\n",
" - Easy parameter modification\n",
" - Re-run capabilities for different configurations\n",
" - Support for both StaticFitStrategy and SlidingFitStrategy\n",
"\n",
"### 🚀 How to Use:\n",
"\n",
"1. **Start Jupyter**:\n",
" ```bash\n",
" cd src/notebooks\n",
" jupyter notebook pairs_trading_visualization.ipynb\n",
" ```\n",
"\n",
"2. **Customize Your Analysis**:\n",
" - Change `SYMBOL_A` and `SYMBOL_B` to your desired trading pair\n",
" - Switch between `CRYPTO_CONFIG` and `EQT_CONFIG`\n",
" - Only **StaticFitStrategy** is supported. \n",
" - Adjust thresholds and parameters as needed\n",
"\n",
"3. **Run and Visualize**:\n",
" - Execute cells step by step to see the analysis unfold\n",
" - Rich matplotlib visualizations show relationships and signals\n",
" - Comprehensive summary at the end\n",
"\n",
"The notebook provides exactly what you requested - a way to visualize the relationship between two instruments and their scaled dis-equilibrium, with all the stages of your pairs trading strategy clearly displayed and analyzed.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup and Imports"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Setup complete!\n"
]
}
],
"source": [
"import sys\n",
"import os\n",
"sys.path.append('..')\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"from typing import Dict, List, Optional\n",
"\n",
"# Import our modules\n",
"from pt_trading.fit_methods import StaticFit, SlidingFit\n",
"from tools.data_loader import load_market_data\n",
"from pt_trading.trading_pair import TradingPair\n",
"from pt_trading.results import BacktestResult\n",
"\n",
"# Set plotting style\n",
"plt.style.use('seaborn-v0_8')\n",
"sns.set_palette(\"husl\")\n",
"plt.rcParams['figure.figsize'] = (12, 8)\n",
"\n",
"print(\"Setup complete!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Configuration"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Using EQUITY configuration\n",
"Available instruments: ['COIN', 'GBTC', 'HOOD', 'MSTR', 'PYPL']\n"
]
}
],
"source": [
"# Configuration - Choose between CRYPTO_CONFIG or EQT_CONFIG\n",
"\n",
"CRYPTO_CONFIG = {\n",
" \"security_type\": \"CRYPTO\",\n",
" \"data_directory\": \"../../data/crypto\",\n",
" \"datafiles\": [\n",
" \"20250519.mktdata.ohlcv.db\",\n",
" ],\n",
" \"db_table_name\": \"bnbspot_ohlcv_1min\",\n",
" \"exchange_id\": \"BNBSPOT\",\n",
" \"instrument_id_pfx\": \"PAIR-\",\n",
" \"instruments\": [\n",
" \"BTC-USDT\",\n",
" \"BCH-USDT\",\n",
" \"ETH-USDT\",\n",
" \"LTC-USDT\",\n",
" \"XRP-USDT\",\n",
" \"ADA-USDT\",\n",
" \"SOL-USDT\",\n",
" \"DOT-USDT\",\n",
" ],\n",
" \"trading_hours\": {\n",
" \"begin_session\": \"00:00:00\",\n",
" \"end_session\": \"23:59:00\",\n",
" \"timezone\": \"UTC\",\n",
" },\n",
" \"price_column\": \"close\",\n",
" \"min_required_points\": 30,\n",
" \"zero_threshold\": 1e-10,\n",
" \"dis-equilibrium_open_trshld\": 2.0,\n",
" \"dis-equilibrium_close_trshld\": 0.5,\n",
" \"training_minutes\": 120,\n",
" \"funding_per_pair\": 2000.0,\n",
"}\n",
"\n",
"EQT_CONFIG = {\n",
" \"security_type\": \"EQUITY\",\n",
" \"data_directory\": \"../../data/equity\",\n",
" \"datafiles\": {\n",
" \"0508\": \"20250508.alpaca_sim_md.db\",\n",
" \"0509\": \"20250509.alpaca_sim_md.db\",\n",
" \"0510\": \"20250510.alpaca_sim_md.db\",\n",
" \"0511\": \"20250511.alpaca_sim_md.db\",\n",
" \"0512\": \"20250512.alpaca_sim_md.db\",\n",
" \"0513\": \"20250513.alpaca_sim_md.db\",\n",
" \"0514\": \"20250514.alpaca_sim_md.db\",\n",
" \"0515\": \"20250515.alpaca_sim_md.db\",\n",
" \"0516\": \"20250516.alpaca_sim_md.db\",\n",
" \"0517\": \"20250517.alpaca_sim_md.db\",\n",
" \"0518\": \"20250518.alpaca_sim_md.db\",\n",
" \"0519\": \"20250519.alpaca_sim_md.db\",\n",
" \"0520\": \"20250520.alpaca_sim_md.db\",\n",
" \"0521\": \"20250521.alpaca_sim_md.db\",\n",
" \"0522\": \"20250522.alpaca_sim_md.db\",\n",
" },\n",
" \"db_table_name\": \"md_1min_bars\",\n",
" \"exchange_id\": \"ALPACA\",\n",
" \"instrument_id_pfx\": \"STOCK-\",\n",
" \"instruments\": [\n",
" \"COIN\",\n",
" \"GBTC\",\n",
" \"HOOD\",\n",
" \"MSTR\",\n",
" \"PYPL\",\n",
" ],\n",
" \"trading_hours\": {\n",
" \"begin_session\": \"9:30:00\",\n",
" \"end_session\": \"16:00:00\",\n",
" \"timezone\": \"America/New_York\",\n",
" },\n",
" \"price_column\": \"close\",\n",
" \"min_required_points\": 30,\n",
" \"zero_threshold\": 1e-10,\n",
" \"dis-equilibrium_open_trshld\": 2.0,\n",
" \"dis-equilibrium_close_trshld\": 1.0, #0.5,\n",
" \"training_minutes\": 120,\n",
" \"funding_per_pair\": 2000.0,\n",
"}\n",
"\n",
"# Choose your configuration\n",
"CONFIG = EQT_CONFIG # Change to CRYPTO_CONFIG if you want to use crypto data\n",
"\n",
"print(f\"Using {CONFIG['security_type']} configuration\")\n",
"print(f\"Available instruments: {CONFIG['instruments']}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Select Trading Pair and Data File"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Selected pair: COIN & GBTC\n",
"Data file: 20250509.alpaca_sim_md.db\n",
"Strategy: StaticFitStrategy\n"
]
}
],
"source": [
"# Select your trading pair and strategy\n",
"SYMBOL_A = \"COIN\" # Change these to your desired symbols\n",
"SYMBOL_B = \"GBTC\"\n",
"DATA_FILE = CONFIG[\"datafiles\"][\"0509\"]\n",
"\n",
"# Choose strategy\n",
"FIT_METHOD = StaticFit()\n",
"\n",
"print(f\"Selected pair: {SYMBOL_A} & {SYMBOL_B}\")\n",
"print(f\"Data file: {DATA_FILE}\")\n",
"print(f\"Strategy: {type(FIT_METHOD).__name__}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Market Data"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Current working directory: /home/oleg/devel/pairs_trading/src/notebooks\n",
"Loading data from: ../../data/equity/20250509.alpaca_sim_md.db\n",
"Error: Execution failed on sql 'select tstamp, tstamp_ns as time_ns, substr(instrument_id, 7) as symbol, open, high, low, close, volume, num_trades, vwap from md_1min_bars where exchange_id ='ALPACA' and instrument_id in (\"STOCK-COIN\",\"STOCK-GBTC\",\"STOCK-HOOD\",\"STOCK-MSTR\",\"STOCK-PYPL\")': no such table: md_1min_bars\n"
]
},
{
"ename": "Exception",
"evalue": "",
"output_type": "error",
"traceback": [
"\u001b[31m---------------------------------------------------------------------------\u001b[39m",
"\u001b[31mOperationalError\u001b[39m Traceback (most recent call last)",
"\u001b[36mFile \u001b[39m\u001b[32m~/.pyenv/python3.12-venv/lib/python3.12/site-packages/pandas/io/sql.py:2664\u001b[39m, in \u001b[36mSQLiteDatabase.execute\u001b[39m\u001b[34m(self, sql, params)\u001b[39m\n\u001b[32m 2663\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m2664\u001b[39m \u001b[43mcur\u001b[49m\u001b[43m.\u001b[49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\u001b[43msql\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 2665\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m cur\n",
"\u001b[31mOperationalError\u001b[39m: no such table: md_1min_bars",
"\nThe above exception was the direct cause of the following exception:\n",
"\u001b[31mDatabaseError\u001b[39m Traceback (most recent call last)",
"\u001b[36mFile \u001b[39m\u001b[32m~/devel/pairs_trading/src/notebooks/../tools/data_loader.py:11\u001b[39m, in \u001b[36mload_sqlite_to_dataframe\u001b[39m\u001b[34m(db_path, query)\u001b[39m\n\u001b[32m 9\u001b[39m conn = sqlite3.connect(db_path)\n\u001b[32m---> \u001b[39m\u001b[32m11\u001b[39m df = \u001b[43mpd\u001b[49m\u001b[43m.\u001b[49m\u001b[43mread_sql_query\u001b[49m\u001b[43m(\u001b[49m\u001b[43mquery\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mconn\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 12\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m df\n",
"\u001b[36mFile \u001b[39m\u001b[32m~/.pyenv/python3.12-venv/lib/python3.12/site-packages/pandas/io/sql.py:528\u001b[39m, in \u001b[36mread_sql_query\u001b[39m\u001b[34m(sql, con, index_col, coerce_float, params, parse_dates, chunksize, dtype, dtype_backend)\u001b[39m\n\u001b[32m 527\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m pandasSQL_builder(con) \u001b[38;5;28;01mas\u001b[39;00m pandas_sql:\n\u001b[32m--> \u001b[39m\u001b[32m528\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mpandas_sql\u001b[49m\u001b[43m.\u001b[49m\u001b[43mread_query\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m 529\u001b[39m \u001b[43m \u001b[49m\u001b[43msql\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 530\u001b[39m \u001b[43m \u001b[49m\u001b[43mindex_col\u001b[49m\u001b[43m=\u001b[49m\u001b[43mindex_col\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 531\u001b[39m \u001b[43m \u001b[49m\u001b[43mparams\u001b[49m\u001b[43m=\u001b[49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 532\u001b[39m \u001b[43m \u001b[49m\u001b[43mcoerce_float\u001b[49m\u001b[43m=\u001b[49m\u001b[43mcoerce_float\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 533\u001b[39m \u001b[43m \u001b[49m\u001b[43mparse_dates\u001b[49m\u001b[43m=\u001b[49m\u001b[43mparse_dates\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 534\u001b[39m \u001b[43m \u001b[49m\u001b[43mchunksize\u001b[49m\u001b[43m=\u001b[49m\u001b[43mchunksize\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 535\u001b[39m \u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m=\u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 536\u001b[39m \u001b[43m \u001b[49m\u001b[43mdtype_backend\u001b[49m\u001b[43m=\u001b[49m\u001b[43mdtype_backend\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m 537\u001b[39m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[36mFile \u001b[39m\u001b[32m~/.pyenv/python3.12-venv/lib/python3.12/site-packages/pandas/io/sql.py:2728\u001b[39m, in \u001b[36mSQLiteDatabase.read_query\u001b[39m\u001b[34m(self, sql, index_col, coerce_float, parse_dates, params, chunksize, dtype, dtype_backend)\u001b[39m\n\u001b[32m 2717\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mread_query\u001b[39m(\n\u001b[32m 2718\u001b[39m \u001b[38;5;28mself\u001b[39m,\n\u001b[32m 2719\u001b[39m sql,\n\u001b[32m (...)\u001b[39m\u001b[32m 2726\u001b[39m dtype_backend: DtypeBackend | Literal[\u001b[33m\"\u001b[39m\u001b[33mnumpy\u001b[39m\u001b[33m\"\u001b[39m] = \u001b[33m\"\u001b[39m\u001b[33mnumpy\u001b[39m\u001b[33m\"\u001b[39m,\n\u001b[32m 2727\u001b[39m ) -> DataFrame | Iterator[DataFrame]:\n\u001b[32m-> \u001b[39m\u001b[32m2728\u001b[39m cursor = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\u001b[43msql\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mparams\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 2729\u001b[39m columns = [col_desc[\u001b[32m0\u001b[39m] \u001b[38;5;28;01mfor\u001b[39;00m col_desc \u001b[38;5;129;01min\u001b[39;00m cursor.description]\n",
"\u001b[36mFile \u001b[39m\u001b[32m~/.pyenv/python3.12-venv/lib/python3.12/site-packages/pandas/io/sql.py:2676\u001b[39m, in \u001b[36mSQLiteDatabase.execute\u001b[39m\u001b[34m(self, sql, params)\u001b[39m\n\u001b[32m 2675\u001b[39m ex = DatabaseError(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mExecution failed on sql \u001b[39m\u001b[33m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00msql\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m'\u001b[39m\u001b[33m: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mexc\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m\"\u001b[39m)\n\u001b[32m-> \u001b[39m\u001b[32m2676\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m ex \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mexc\u001b[39;00m\n",
"\u001b[31mDatabaseError\u001b[39m: Execution failed on sql 'select tstamp, tstamp_ns as time_ns, substr(instrument_id, 7) as symbol, open, high, low, close, volume, num_trades, vwap from md_1min_bars where exchange_id ='ALPACA' and instrument_id in (\"STOCK-COIN\",\"STOCK-GBTC\",\"STOCK-HOOD\",\"STOCK-MSTR\",\"STOCK-PYPL\")': no such table: md_1min_bars",
"\nThe above exception was the direct cause of the following exception:\n",
"\u001b[31mException\u001b[39m Traceback (most recent call last)",
"\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[5]\u001b[39m\u001b[32m, line 6\u001b[39m\n\u001b[32m 3\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mCurrent working directory: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mos.getcwd()\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m\"\u001b[39m)\n\u001b[32m 4\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mLoading data from: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdatafile_path\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m\"\u001b[39m)\n\u001b[32m----> \u001b[39m\u001b[32m6\u001b[39m market_data_df = \u001b[43mload_market_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdatafile_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mconfig\u001b[49m\u001b[43m=\u001b[49m\u001b[43mCONFIG\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 8\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mLoaded \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(market_data_df)\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m rows of market data\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m 9\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mSymbols in data: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mmarket_data_df[\u001b[33m'\u001b[39m\u001b[33msymbol\u001b[39m\u001b[33m'\u001b[39m].unique()\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m\"\u001b[39m)\n",
"\u001b[36mFile \u001b[39m\u001b[32m~/devel/pairs_trading/src/notebooks/../tools/data_loader.py:69\u001b[39m, in \u001b[36mload_market_data\u001b[39m\u001b[34m(datafile, config)\u001b[39m\n\u001b[32m 66\u001b[39m query += \u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33m where exchange_id =\u001b[39m\u001b[33m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mexchange_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m'\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m 67\u001b[39m query += \u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33m and instrument_id in (\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[33m'\u001b[39m\u001b[33m,\u001b[39m\u001b[33m'\u001b[39m.join(instrument_ids)\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m)\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m---> \u001b[39m\u001b[32m69\u001b[39m df = \u001b[43mload_sqlite_to_dataframe\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdb_path\u001b[49m\u001b[43m=\u001b[49m\u001b[43mdatafile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mquery\u001b[49m\u001b[43m=\u001b[49m\u001b[43mquery\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 71\u001b[39m \u001b[38;5;66;03m# Trading Hours\u001b[39;00m\n\u001b[32m 72\u001b[39m date_str = df[\u001b[33m\"\u001b[39m\u001b[33mtstamp\u001b[39m\u001b[33m\"\u001b[39m][\u001b[32m0\u001b[39m][\u001b[32m0\u001b[39m:\u001b[32m10\u001b[39m]\n",
"\u001b[36mFile \u001b[39m\u001b[32m~/devel/pairs_trading/src/notebooks/../tools/data_loader.py:18\u001b[39m, in \u001b[36mload_sqlite_to_dataframe\u001b[39m\u001b[34m(db_path, query)\u001b[39m\n\u001b[32m 16\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m excpt:\n\u001b[32m 17\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mError: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mexcpt\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m\"\u001b[39m)\n\u001b[32m---> \u001b[39m\u001b[32m18\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m() \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mexcpt\u001b[39;00m\n\u001b[32m 19\u001b[39m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[32m 20\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[33m\"\u001b[39m\u001b[33mconn\u001b[39m\u001b[33m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mlocals\u001b[39m():\n",
"\u001b[31mException\u001b[39m: "
]
}
],
"source": [
"# Load market data\n",
"datafile_path = f\"{CONFIG['data_directory']}/{DATA_FILE}\"\n",
"print(f\"Current working directory: {os.getcwd()}\")\n",
"print(f\"Loading data from: {datafile_path}\")\n",
"\n",
"market_data_df = load_market_data(datafile_path, config=CONFIG)\n",
"\n",
"print(f\"Loaded {len(market_data_df)} rows of market data\")\n",
"print(f\"Symbols in data: {market_data_df['symbol'].unique()}\")\n",
"print(f\"Time range: {market_data_df['tstamp'].min()} to {market_data_df['tstamp'].max()}\")\n",
"\n",
"# Display first few rows\n",
"market_data_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Create Trading Pair and Analyze"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Create trading pair\n",
"pair = TradingPair(\n",
" market_data=market_data_df,\n",
" symbol_a=SYMBOL_A,\n",
" symbol_b=SYMBOL_B,\n",
" price_column=CONFIG[\"price_column\"]\n",
")\n",
"\n",
"print(f\"Created trading pair: {pair}\")\n",
"print(f\"Market data shape: {pair.market_data_.shape}\")\n",
"print(f\"Column names: {pair.colnames()}\")\n",
"\n",
"# Display first few rows of pair data\n",
"pair.market_data_.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Split Data into Training and Testing"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Get training and testing datasets\n",
"training_minutes = CONFIG[\"training_minutes\"]\n",
"pair.get_datasets(training_minutes=training_minutes)\n",
"\n",
"print(f\"Training data: {len(pair.training_df_)} rows\")\n",
"print(f\"Testing data: {len(pair.testing_df_)} rows\")\n",
"print(f\"Training period: {pair.training_df_['tstamp'].iloc[0]} to {pair.training_df_['tstamp'].iloc[-1]}\")\n",
"print(f\"Testing period: {pair.testing_df_['tstamp'].iloc[0]} to {pair.testing_df_['tstamp'].iloc[-1]}\")\n",
"\n",
"# Check for any missing data\n",
"print(f\"Training data null values: {pair.training_df_.isnull().sum().sum()}\")\n",
"print(f\"Testing data null values: {pair.testing_df_.isnull().sum().sum()}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize Raw Price Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Plot raw price data\n",
"fig, axes = plt.subplots(3, 1, figsize=(15, 12))\n",
"\n",
"# Combined price plot\n",
"colname_a, colname_b = pair.colnames()\n",
"all_data = pd.concat([pair.training_df_, pair.testing_df_]).reset_index(drop=True)\n",
"\n",
"# Plot individual prices\n",
"axes[0].plot(all_data['tstamp'], all_data[colname_a], label=f'{SYMBOL_A}', alpha=0.8)\n",
"axes[0].plot(all_data['tstamp'], all_data[colname_b], label=f'{SYMBOL_B}', alpha=0.8)\n",
"axes[0].axvline(x=pair.training_df_['tstamp'].iloc[-1], color='red', linestyle='--', alpha=0.7, label='Train/Test Split')\n",
"axes[0].set_title(f'Price Comparison: {SYMBOL_A} vs {SYMBOL_B}')\n",
"axes[0].set_ylabel('Price')\n",
"axes[0].legend()\n",
"axes[0].grid(True)\n",
"\n",
"# Normalized prices for comparison\n",
"norm_a = all_data[colname_a] / all_data[colname_a].iloc[0]\n",
"norm_b = all_data[colname_b] / all_data[colname_b].iloc[0]\n",
"\n",
"axes[1].plot(all_data['tstamp'], norm_a, label=f'{SYMBOL_A} (normalized)', alpha=0.8)\n",
"axes[1].plot(all_data['tstamp'], norm_b, label=f'{SYMBOL_B} (normalized)', alpha=0.8)\n",
"axes[1].axvline(x=pair.training_df_['tstamp'].iloc[-1], color='red', linestyle='--', alpha=0.7, label='Train/Test Split')\n",
"axes[1].set_title('Normalized Price Comparison')\n",
"axes[1].set_ylabel('Normalized Price')\n",
"axes[1].legend()\n",
"axes[1].grid(True)\n",
"\n",
"# Price ratio\n",
"price_ratio = all_data[colname_a] / all_data[colname_b]\n",
"axes[2].plot(all_data['tstamp'], price_ratio, label=f'{SYMBOL_A}/{SYMBOL_B} Ratio', color='green', alpha=0.8)\n",
"axes[2].axvline(x=pair.training_df_['tstamp'].iloc[-1], color='red', linestyle='--', alpha=0.7, label='Train/Test Split')\n",
"axes[2].set_title('Price Ratio')\n",
"axes[2].set_ylabel('Ratio')\n",
"axes[2].set_xlabel('Time')\n",
"axes[2].legend()\n",
"axes[2].grid(True)\n",
"\n",
"plt.tight_layout()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Train the Pair and Check Cointegration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Train the pair and check cointegration\n",
"try:\n",
" is_cointegrated = pair.train_pair()\n",
" print(f\"Pair {pair} cointegration status: {is_cointegrated}\")\n",
"\n",
" if is_cointegrated:\n",
" print(f\"VECM Beta coefficients: {pair.vecm_fit_.beta.flatten()}\")\n",
" print(f\"Training dis-equilibrium mean: {pair.training_mu_:.6f}\")\n",
" print(f\"Training dis-equilibrium std: {pair.training_std_:.6f}\")\n",
"\n",
" # Display VECM summary\n",
" print(\"\\nVECM Model Summary:\")\n",
" print(pair.vecm_fit_.summary())\n",
" else:\n",
" print(\"Pair is not cointegrated. Cannot proceed with strategy.\")\n",
"\n",
"except Exception as e:\n",
" print(f\"Training failed: {str(e)}\")\n",
" is_cointegrated = False"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize Training Period Dis-equilibrium"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if is_cointegrated:\n",
" # fig, axes = plt.subplots(, 1, figsize=(15, 10))\n",
"\n",
" # # Raw dis-equilibrium\n",
" # axes[0].plot(pair.training_df_['tstamp'], pair.training_df_['dis-equilibrium'],\n",
" # color='blue', alpha=0.8, label='Raw Dis-equilibrium')\n",
" # axes[0].axhline(y=pair.training_mu_, color='red', linestyle='--', alpha=0.7, label='Mean')\n",
" # axes[0].axhline(y=pair.training_mu_ + pair.training_std_, color='orange', linestyle='--', alpha=0.5, label='+1 Std')\n",
" # axes[0].axhline(y=pair.training_mu_ - pair.training_std_, color='orange', linestyle='--', alpha=0.5, label='-1 Std')\n",
" # axes[0].set_title('Training Period: Raw Dis-equilibrium')\n",
" # axes[0].set_ylabel('Dis-equilibrium')\n",
" # axes[0].legend()\n",
" # axes[0].grid(True)\n",
"\n",
" # Scaled dis-equilibrium\n",
" fig, axes = plt.subplots(1, 1, figsize=(15, 5))\n",
" axes.plot(pair.training_df_['tstamp'], pair.training_df_['scaled_dis-equilibrium'],\n",
" color='green', alpha=0.8, label='Scaled Dis-equilibrium')\n",
" axes.axhline(y=0, color='red', linestyle='--', alpha=0.7, label='Mean (0)')\n",
" axes.axhline(y=1, color='orange', linestyle='--', alpha=0.5, label='+1 Std')\n",
" axes.axhline(y=-1, color='orange', linestyle='--', alpha=0.5, label='-1 Std')\n",
" axes.axhline(y=CONFIG['dis-equilibrium_open_trshld'], color='purple',\n",
" linestyle=':', alpha=0.7, label=f\"Open Threshold ({CONFIG['dis-equilibrium_open_trshld']})\")\n",
" axes.axhline(y=CONFIG['dis-equilibrium_close_trshld'], color='brown',\n",
" linestyle=':', alpha=0.7, label=f\"Close Threshold ({CONFIG['dis-equilibrium_close_trshld']})\")\n",
" axes.set_title('Training Period: Scaled Dis-equilibrium')\n",
" axes.set_ylabel('Scaled Dis-equilibrium')\n",
" axes.set_xlabel('Time')\n",
" axes.legend()\n",
" axes.grid(True)\n",
"\n",
" plt.tight_layout()\n",
" plt.show()\n",
"\n",
" # Print statistics\n",
" print(f\"Training dis-equilibrium statistics:\")\n",
" print(f\" Mean: {pair.training_df_['dis-equilibrium'].mean():.6f}\")\n",
" print(f\" Std: {pair.training_df_['dis-equilibrium'].std():.6f}\")\n",
" print(f\" Min: {pair.training_df_['dis-equilibrium'].min():.6f}\")\n",
" print(f\" Max: {pair.training_df_['dis-equilibrium'].max():.6f}\")\n",
"\n",
" print(f\"\\nScaled dis-equilibrium statistics:\")\n",
" print(f\" Mean: {pair.training_df_['scaled_dis-equilibrium'].mean():.6f}\")\n",
" print(f\" Std: {pair.training_df_['scaled_dis-equilibrium'].std():.6f}\")\n",
" print(f\" Min: {pair.training_df_['scaled_dis-equilibrium'].min():.6f}\")\n",
" print(f\" Max: {pair.training_df_['scaled_dis-equilibrium'].max():.6f}\")\n",
"else:\n",
" print(\"The pair is not cointegrated\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate Predictions and Run Strategy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if is_cointegrated:\n",
" try:\n",
" # Generate predictions\n",
" pair.predict()\n",
" print(f\"Generated predictions for {len(pair.predicted_df_)} rows\")\n",
"\n",
" # Display prediction data structure\n",
" print(f\"Prediction columns: {list(pair.predicted_df_.columns)}\")\n",
" print(f\"Prediction period: {pair.predicted_df_['tstamp'].iloc[0]} to {pair.predicted_df_['tstamp'].iloc[-1]}\")\n",
"\n",
" # Run strategy\n",
" bt_result = BacktestResult(config=CONFIG)\n",
" pair_trades = FIT_METHOD.run_pair(config=CONFIG, pair=pair, bt_result=bt_result)\n",
"\n",
" if pair_trades is not None and len(pair_trades) > 0:\n",
" print(f\"\\nGenerated {len(pair_trades)} trading signals:\")\n",
" print(pair_trades)\n",
" else:\n",
" print(\"\\nNo trading signals generated\")\n",
"\n",
" except Exception as e:\n",
" print(f\"Prediction/Strategy failed: {str(e)}\")\n",
" pair_trades = None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Visualize Predictions and Dis-equilibrium"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if is_cointegrated and hasattr(pair, 'predicted_df_'):\n",
" fig, axes = plt.subplots(4, 1, figsize=(16, 16))\n",
"\n",
" # Actual vs Predicted Prices\n",
" colname_a, colname_b = pair.colnames()\n",
"\n",
" axes[0].plot(pair.predicted_df_['tstamp'], pair.predicted_df_[colname_a],\n",
" label=f'{SYMBOL_A} Actual', alpha=0.8)\n",
" axes[0].plot(pair.predicted_df_['tstamp'], pair.predicted_df_[f'{colname_a}_pred'],\n",
" label=f'{SYMBOL_A} Predicted', alpha=0.8, linestyle='--')\n",
" axes[0].set_title('Actual vs Predicted Prices - Symbol A')\n",
" axes[0].set_ylabel('Price')\n",
" axes[0].legend()\n",
" axes[0].grid(True)\n",
"\n",
" axes[1].plot(pair.predicted_df_['tstamp'], pair.predicted_df_[colname_b],\n",
" label=f'{SYMBOL_B} Actual', alpha=0.8)\n",
" axes[1].plot(pair.predicted_df_['tstamp'], pair.predicted_df_[f'{colname_b}_pred'],\n",
" label=f'{SYMBOL_B} Predicted', alpha=0.8, linestyle='--')\n",
" axes[1].set_title('Actual vs Predicted Prices - Symbol B')\n",
" axes[1].set_ylabel('Price')\n",
" axes[1].legend()\n",
" axes[1].grid(True)\n",
"\n",
" # Raw dis-equilibrium\n",
" axes[2].plot(pair.predicted_df_['tstamp'], pair.predicted_df_['disequilibrium'],\n",
" color='blue', alpha=0.8, label='Dis-equilibrium')\n",
" axes[2].axhline(y=pair.training_mu_, color='red', linestyle='--', alpha=0.7, label='Training Mean')\n",
" axes[2].set_title('Testing Period: Raw Dis-equilibrium')\n",
" axes[2].set_ylabel('Dis-equilibrium')\n",
" axes[2].legend()\n",
" axes[2].grid(True)\n",
"\n",
" # Scaled dis-equilibrium with trading signals\n",
" axes[3].plot(pair.predicted_df_['tstamp'], pair.predicted_df_['scaled_disequilibrium'],\n",
" color='green', alpha=0.8, label='Scaled Dis-equilibrium')\n",
"\n",
" # Add threshold lines\n",
" axes[3].axhline(y=CONFIG['dis-equilibrium_open_trshld'], color='purple',\n",
" linestyle=':', alpha=0.7, label=f\"Open Threshold ({CONFIG['dis-equilibrium_open_trshld']})\")\n",
" axes[3].axhline(y=CONFIG['dis-equilibrium_close_trshld'], color='brown',\n",
" linestyle=':', alpha=0.7, label=f\"Close Threshold ({CONFIG['dis-equilibrium_close_trshld']})\")\n",
"\n",
" # Add trading signals if they exist\n",
" if pair_trades is not None and len(pair_trades) > 0:\n",
" for _, trade in pair_trades.iterrows():\n",
" color = 'red' if 'BUY' in trade['action'] else 'blue'\n",
" marker = '^' if 'BUY' in trade['action'] else 'v'\n",
" axes[3].scatter(trade['time'], trade['scaled_disequilibrium'],\n",
" color=color, marker=marker, s=100, alpha=0.8,\n",
" label=f\"{trade['action']} {trade['symbol']}\" if _ < 2 else \"\")\n",
"\n",
" axes[3].set_title('Testing Period: Scaled Dis-equilibrium with Trading Signals')\n",
" axes[3].set_ylabel('Scaled Dis-equilibrium')\n",
" axes[3].set_xlabel('Time')\n",
" axes[3].legend()\n",
" axes[3].grid(True)\n",
"\n",
" plt.tight_layout()\n",
" plt.show()\n",
"\n",
" # Print prediction statistics\n",
" print(f\"\\nTesting dis-equilibrium statistics:\")\n",
" print(f\" Mean: {pair.predicted_df_['disequilibrium'].mean():.6f}\")\n",
" print(f\" Std: {pair.predicted_df_['disequilibrium'].std():.6f}\")\n",
" print(f\" Min: {pair.predicted_df_['disequilibrium'].min():.6f}\")\n",
" print(f\" Max: {pair.predicted_df_['disequilibrium'].max():.6f}\")\n",
"\n",
" print(f\"\\nTesting scaled dis-equilibrium statistics:\")\n",
" print(f\" Mean: {pair.predicted_df_['scaled_disequilibrium'].mean():.6f}\")\n",
" print(f\" Std: {pair.predicted_df_['scaled_disequilibrium'].std():.6f}\")\n",
" print(f\" Min: {pair.predicted_df_['scaled_disequilibrium'].min():.6f}\")\n",
" print(f\" Max: {pair.predicted_df_['scaled_disequilibrium'].max():.6f}\")\n",
"\n",
" # Count threshold crossings\n",
" open_crossings = (pair.predicted_df_['scaled_disequilibrium'] >= CONFIG['dis-equilibrium_open_trshld']).sum()\n",
" close_crossings = (pair.predicted_df_['scaled_disequilibrium'] <= CONFIG['dis-equilibrium_close_trshld']).sum()\n",
" print(f\"\\nThreshold crossings:\")\n",
" print(f\" Open threshold ({CONFIG['dis-equilibrium_open_trshld']}): {open_crossings} times\")\n",
" print(f\" Close threshold ({CONFIG['dis-equilibrium_close_trshld']}): {close_crossings} times\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary and Analysis"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"=\" * 60)\n",
"print(\"PAIRS TRADING ANALYSIS SUMMARY\")\n",
"print(\"=\" * 60)\n",
"\n",
"print(f\"\\nPair: {SYMBOL_A} & {SYMBOL_B}\")\n",
"print(f\"Strategy: {type(FIT_METHOD).__name__}\")\n",
"print(f\"Data file: {DATA_FILE}\")\n",
"print(f\"Training period: {training_minutes} minutes\")\n",
"\n",
"print(f\"\\nCointegration Status: {'✓ COINTEGRATED' if is_cointegrated else '✗ NOT COINTEGRATED'}\")\n",
"\n",
"if is_cointegrated:\n",
" print(f\"\\nVECM Model:\")\n",
" print(f\" Beta coefficients: {pair.vecm_fit_.beta.flatten()}\")\n",
" print(f\" Training mean: {pair.training_mu_:.6f}\")\n",
" print(f\" Training std: {pair.training_std_:.6f}\")\n",
"\n",
" if pair_trades is not None and len(pair_trades) > 0:\n",
" print(f\"\\nTrading Signals: {len(pair_trades)} generated\")\n",
" unique_times = pair_trades['time'].unique()\n",
" print(f\" Unique trade times: {len(unique_times)}\")\n",
"\n",
" # Group by time to see paired trades\n",
" for trade_time in unique_times:\n",
" trades_at_time = pair_trades[pair_trades['time'] == trade_time]\n",
" print(f\"\\n Trade at {trade_time}:\")\n",
" for _, trade in trades_at_time.iterrows():\n",
" print(f\" {trade['action']} {trade['symbol']} @ ${trade['price']:.2f} (dis-eq: {trade['scaled_disequilibrium']:.2f})\")\n",
" else:\n",
" print(f\"\\nTrading Signals: None generated\")\n",
" print(\" Possible reasons:\")\n",
" print(\" - Dis-equilibrium never exceeded open threshold\")\n",
" print(\" - Insufficient testing data\")\n",
" print(\" - Strategy-specific conditions not met\")\n",
"\n",
"else:\n",
" print(\"\\nCannot proceed with trading strategy - pair is not cointegrated\")\n",
" print(\"Consider:\")\n",
" print(\" - Trying different symbol pairs\")\n",
" print(\" - Adjusting training period length\")\n",
" print(\" - Using different data timeframe\")\n",
"\n",
"print(\"\\n\" + \"=\" * 60)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Interactive Analysis (Optional)\n",
"\n",
"You can modify the parameters below and re-run the analysis:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Interactive parameter adjustment\n",
"print(\"Current parameters:\")\n",
"print(f\" Open threshold: {CONFIG['dis-equilibrium_open_trshld']}\")\n",
"print(f\" Close threshold: {CONFIG['dis-equilibrium_close_trshld']}\")\n",
"print(f\" Training minutes: {CONFIG['training_minutes']}\")\n",
"\n",
"# Uncomment and modify these to experiment:\n",
"# CONFIG['dis-equilibrium_open_trshld'] = 1.5\n",
"# CONFIG['dis-equilibrium_close_trshld'] = 0.3\n",
"# CONFIG['training_minutes'] = 180\n",
"\n",
"print(\"\\nTo re-run with different parameters:\")\n",
"print(\"1. Modify the parameters above\")\n",
"print(\"2. Re-run from the 'Split Data into Training and Testing' cell\")\n",
"print(\"3. Or try different symbol pairs by changing SYMBOL_A and SYMBOL_B\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "python3.12-venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
research/pt_backtest.py
+257
View File
@@ -0,0 +1,257 @@
import argparse
import glob
import importlib
import os
from datetime import date, datetime
from typing import Any, Dict, List, Optional
import pandas as pd
from tools.config import expand_filename, load_config
from tools.data_loader import get_available_instruments_from_db, load_market_data
from pt_trading.results import (
BacktestResult,
create_result_database,
store_config_in_database,
store_results_in_database,
)
from pt_trading.fit_methods import PairsTradingFitMethod
from pt_trading.trading_pair import TradingPair
def resolve_datafiles(config: Dict, cli_datafiles: Optional[str] = None) -> List[str]:
"""
Resolve the list of data files to process.
CLI datafiles take priority over config datafiles.
Supports wildcards in config but not in CLI.
"""
if cli_datafiles:
# CLI override - comma-separated list, no wildcards
datafiles = [f.strip() for f in cli_datafiles.split(",")]
# Make paths absolute relative to data directory
data_dir = config.get("data_directory", "./data")
resolved_files = []
for df in datafiles:
if not os.path.isabs(df):
df = os.path.join(data_dir, df)
resolved_files.append(df)
return resolved_files
# Use config datafiles with wildcard support
config_datafiles = config.get("datafiles", [])
data_dir = config.get("data_directory", "./data")
resolved_files = []
for pattern in config_datafiles:
if "*" in pattern or "?" in pattern:
# Handle wildcards
if not os.path.isabs(pattern):
pattern = os.path.join(data_dir, pattern)
matched_files = glob.glob(pattern)
resolved_files.extend(matched_files)
else:
# Handle explicit file path
if not os.path.isabs(pattern):
pattern = os.path.join(data_dir, pattern)
resolved_files.append(pattern)
return sorted(list(set(resolved_files))) # Remove duplicates and sort
def run_backtest(
config: Dict,
datafile: str,
price_column: str,
fit_method: PairsTradingFitMethod,
instruments: List[str],
) -> BacktestResult:
"""
Run backtest for all pairs using the specified instruments.
"""
bt_result: BacktestResult = BacktestResult(config=config)
def _create_pairs(config: Dict, instruments: List[str]) -> List[TradingPair]:
nonlocal datafile
all_indexes = range(len(instruments))
unique_index_pairs = [(i, j) for i in all_indexes for j in all_indexes if i < j]
pairs = []
# Update config to use the specified instruments
config_copy = config.copy()
config_copy["instruments"] = instruments
market_data_df = load_market_data(datafile, config=config_copy)
for a_index, b_index in unique_index_pairs:
pair = TradingPair(
market_data=market_data_df,
symbol_a=instruments[a_index],
symbol_b=instruments[b_index],
price_column=price_column,
)
pairs.append(pair)
return pairs
pairs_trades = []
for pair in _create_pairs(config, instruments):
single_pair_trades = fit_method.run_pair(
pair=pair, config=config, bt_result=bt_result
)
if single_pair_trades is not None and len(single_pair_trades) > 0:
pairs_trades.append(single_pair_trades)
# Check if result_list has any data before concatenating
if len(pairs_trades) == 0:
print("No trading signals found for any pairs")
return bt_result
result = pd.concat(pairs_trades, ignore_index=True)
result["time"] = pd.to_datetime(result["time"])
result = result.set_index("time").sort_index()
bt_result.collect_single_day_results(result)
return bt_result
def main() -> None:
parser = argparse.ArgumentParser(description="Run pairs trading backtest.")
parser.add_argument(
"--config", type=str, required=True, help="Path to the configuration file."
)
parser.add_argument(
"--datafiles",
type=str,
required=False,
help="Comma-separated list of data files (overrides config). No wildcards supported.",
)
parser.add_argument(
"--instruments",
type=str,
required=False,
help="Comma-separated list of instrument symbols (e.g., COIN,GBTC). If not provided, auto-detects from database.",
)
parser.add_argument(
"--result_db",
type=str,
required=True,
help="Path to SQLite database for storing results. Use 'NONE' to disable database output.",
)
args = parser.parse_args()
config: Dict = load_config(args.config)
# Dynamically instantiate fit method class
fit_method_class_name = config.get("fit_method_class", None)
assert fit_method_class_name is not None
module_name, class_name = fit_method_class_name.rsplit(".", 1)
module = importlib.import_module(module_name)
fit_method = getattr(module, class_name)()
# Resolve data files (CLI takes priority over config)
datafiles = resolve_datafiles(config, args.datafiles)
if not datafiles:
print("No data files found to process.")
return
print(f"Found {len(datafiles)} data files to process:")
for df in datafiles:
print(f" - {df}")
# Create result database if needed
if args.result_db.upper() != "NONE":
args.result_db = expand_filename(args.result_db)
create_result_database(args.result_db)
# Initialize a dictionary to store all trade results
all_results: Dict[str, Dict[str, Any]] = {}
# Store configuration in database for reference
if args.result_db.upper() != "NONE":
# Get list of all instruments for storage
all_instruments = []
for datafile in datafiles:
if args.instruments:
file_instruments = [
inst.strip() for inst in args.instruments.split(",")
]
else:
file_instruments = get_available_instruments_from_db(datafile, config)
all_instruments.extend(file_instruments)
# Remove duplicates while preserving order
unique_instruments = list(dict.fromkeys(all_instruments))
store_config_in_database(
db_path=args.result_db,
config_file_path=args.config,
config=config,
fit_method_class=fit_method_class_name,
datafiles=datafiles,
instruments=unique_instruments,
)
# Process each data file
price_column = config["price_column"]
for datafile in datafiles:
print(f"\n====== Processing {os.path.basename(datafile)} ======")
# Determine instruments to use
if args.instruments:
# Use CLI-specified instruments
instruments = [inst.strip() for inst in args.instruments.split(",")]
print(f"Using CLI-specified instruments: {instruments}")
else:
# Auto-detect instruments from database
instruments = get_available_instruments_from_db(datafile, config)
print(f"Auto-detected instruments: {instruments}")
if not instruments:
print(f"No instruments found for {datafile}, skipping...")
continue
# Process data for this file
try:
fit_method.reset()
bt_results = run_backtest(
config=config,
datafile=datafile,
price_column=price_column,
fit_method=fit_method,
instruments=instruments,
)
# Store results with file name as key
filename = os.path.basename(datafile)
all_results[filename] = {"trades": bt_results.trades.copy()}
# Store results in database
if args.result_db.upper() != "NONE":
store_results_in_database(args.result_db, datafile, bt_results)
print(f"Successfully processed {filename}")
except Exception as err:
print(f"Error processing {datafile}: {str(err)}")
import traceback
traceback.print_exc()
# Calculate and print results using a new BacktestResult instance for aggregation
if all_results:
aggregate_bt_results = BacktestResult(config=config)
aggregate_bt_results.calculate_returns(all_results)
aggregate_bt_results.print_grand_totals()
aggregate_bt_results.print_outstanding_positions()
if args.result_db.upper() != "NONE":
print(f"\nResults stored in database: {args.result_db}")
else:
print("No results to display.")
if __name__ == "__main__":
main()