Release v1.0.4

.envrc

@@ -0,0 +1 @@
+source /home/oleg/.pyenv/python3.12-venv/bin/activate

.gitignore

@@ -3,9 +3,22 @@ __pycache__/
 __OLD__/
 .specstory/
 .history/
-.cursorindexingignore
-data
 .vscode/
+*.py[cod]
+.ipynb_checkpoints/
+.pytest_cache/
+
+# Local environments
+.venv/
+venv/
+
+# Local test data and generated analysis results
+data/*
+!data/.gitkeep
+results/*
+!results/.gitkeep
+
+data
+
 cvttpy
-# SpecStory explanation file
-.specstory/.what-is-this.md
+tmp/

AGENTS.md

@@ -0,0 +1,156 @@
+# Agent Instructions
+
+## Repository purpose
+
+This repository analyzes test results with Jupyter notebooks and Python or
+Bash scripts. Inputs are commonly SQLite databases containing time-series data
+and JSON columns, but analyses may use other test-result formats.
+
+Ignore `__SAV__/`. It is unrelated legacy material, is not part of the active
+project, and must not be read, edited, moved, or used as a source of conventions
+unless the user explicitly requests it.
+
+## Active layout
+
+- `notebooks/`: exploratory and report-oriented Jupyter notebooks.
+- `scripts/`: reusable Python and Bash analysis utilities.
+- `data/`: local input data. Contents are ignored except for `.gitkeep`.
+- `results/`: generated tables, figures, exports, and reports. Contents are
+  ignored except for `.gitkeep`.
+- `requirements.txt`: Python dependencies needed to reproduce repository work.
+
+Keep reusable logic in `scripts/` and use notebooks to orchestrate analysis,
+explain decisions, and present results. Do not create a separate `analysis/`
+tree.
+
+## Python environment
+
+The intended virtual environment is `~/.pyenv/python3.12-venv`.
+
+```bash
+source ~/.pyenv/python3.12-venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+Agents may install packages in this environment when needed. Whenever a package
+is installed for repository work, update `requirements.txt` in the same change
+with a suitable direct dependency declaration. Use `python -m pip`, not bare
+`pip`, in documented commands.
+
+Do not create an in-repository virtual environment unless the user asks for
+one.
+
+## Data handling
+
+- Treat files in `data/` as local, potentially large, and potentially
+  sensitive.
+- Do not commit SQLite databases, raw test results, or generated results.
+- Do not modify source data in place. Write transformed data and exports under
+  `results/`.
+- Use parameterized SQL for values. Do not construct SQL by interpolating
+  untrusted data.
+- Parse JSON columns defensively and preserve missing, malformed, and unexpected
+  values unless the analysis explicitly defines another policy.
+- State assumptions about timestamps, time zones, ordering, units, and duplicate
+  observations in the notebook or script that relies on them.
+- Avoid loading entire databases into memory when a filtered query or chunked
+  read is practical.
+
+## Notebook conventions
+
+- A notebook must run from a fresh kernel, top to bottom, without relying on
+  hidden interactive state.
+- Set random seeds where nondeterminism affects results.
+- Keep data paths relative to the repository root and avoid machine-specific
+  absolute paths.
+- Move logic that is reused or substantial enough to test into `scripts/`.
+- Clear cell outputs before committing notebooks. Never commit embedded source
+  data, credentials, or bulky generated output.
+- Keep concise Markdown context near analyses: purpose, input assumptions,
+  method, and interpretation.
+
+## Scripts
+
+- Python scripts should expose reusable functions and use a guarded CLI entry
+  point when executable.
+- Bash scripts must start with `#!/usr/bin/env bash` and use
+  `set -euo pipefail`.
+- Prefer explicit CLI arguments over hard-coded paths or parameters.
+- Fail with actionable error messages when required data, tables, columns, or
+  configuration are missing.
+
+## Verification
+
+Verification should be proportional to the change. At minimum:
+
+- Run `pytest` for Python script changes.
+- Add or update tests for reusable parsing, transformation, query, and
+  calculation logic.
+- Execute changed notebooks from a fresh kernel with `nbmake`.
+- Run changed Bash scripts against a safe fixture or exercise their
+  non-destructive validation/help path.
+- Clear notebook outputs after execution and before committing.
+
+Useful commands:
+
+```bash
+python -m pytest
+python -m pytest --nbmake notebooks
+jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace path/to/notebook.ipynb
+```
+
+If verification cannot be run, report exactly what was skipped and why.
+
+## Release rules
+
+- Update `CHANGELOG.md` for every release with the release version, release
+  date, Git tag, and a concise summary of notable changes.
+- Keep an `Unreleased` section at the top of `CHANGELOG.md` for changes that
+  have not been included in a tagged release yet.
+- Move relevant entries from `Unreleased` into the dated release section when
+  creating a release, and leave `Unreleased` present for future changes.
+- Use release headers in `YYYY-MM-DD vMAJOR.MINOR.PATCH` form.
+- Use version numbers in `MAJOR.MINOR.PATCH` form. Start this repository at
+  `0.0.1`.
+- Use Git tags in `vMAJOR.MINOR.PATCH` form, matching the changelog version
+  exactly. For example, version `0.0.1` must be tagged as `v0.0.1`.
+- Create the Git tag only after the changelog and any release-related version
+  changes are complete.
+- When the user requests creating a release, treat that as explicit permission
+  to commit the release changes, create the matching Git tag, and push both the
+  branch and tag.
+- Do not push release commits or tags unless the user explicitly requests it.
+
+## Mandatory background review
+
+Changes to Python scripts, Bash scripts, or notebook code cells require approval
+from a separate background reviewer agent before the implementing agent may
+declare the work complete.
+
+The implementing agent must:
+
+1. Finish the implementation and run the relevant verification.
+2. Ask a separate background agent to review the diff for correctness,
+   reproducibility, data safety, and test coverage.
+3. Address every material finding, rerun affected checks, and request follow-up
+   review when the fix materially changes the code.
+4. Report the reviewer outcome in the final response.
+
+The reviewer must inspect the actual diff and relevant surrounding files; a
+self-review does not satisfy this requirement. Documentation-only,
+configuration-only, dependency-only, and ignore-rule-only changes do not
+require background approval unless they also alter Python, Bash, or notebook
+code cells.
+
+If no background reviewer is available, complete all other work but do not
+claim reviewer approval. End the handoff with the exact status:
+
+`review pending`
+
+## Change discipline
+
+- Preserve user changes and avoid unrelated cleanup.
+- Do not edit or commit generated files from `data/` or `results/`.
+- Do not push or commit unless the user explicitly requests it. The `master`
+  branch being unprotected does not imply permission to push directly.
+- Keep changes focused and explain any new assumptions or dependencies.

CHANGELOG.md

@@ -0,0 +1,95 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## Unreleased
+
+No unreleased changes yet.
+
+## 2026-07-30 v1.0.4
+
+- Updated notebook and Panel analysis for the SP Quant result database schema,
+  including explicit `trading_instructions` columns for action, assets,
+  scaled disequilibrium, and beta.
+- Changed selected-pair market charts to read from the `market` table and kept
+  legacy packed instruction JSON support for older result databases.
+- Added `scaled_disequilibrium` and `beta` to selected-pair theoretical
+  execution displays.
+- Improved VS Code notebook usability with the `jupyter_bokeh` dependency,
+  direct Plotly figure rendering, and a dropdown Analyze control for individual
+  pair selection.
+- Made the Panel app use the dark theme by default and reduced the sidebar
+  width from 430 px to 215 px with responsive sidebar controls.
+- Expanded tests and notebook verification coverage for the new database schema
+  and Panel layout defaults.
+
+## 2026-07-29 v1.0.3
+
+- Removed invalid fixed sizing mode from Panel Tabulator grids to avoid Bokeh
+  layout warnings while preserving compact table layout.
+- Changed the Panel Calculate action to refresh the result-file list before
+  loading data and removed the standalone Panel Refresh button.
+
+## 2026-07-29 v1.0.2
+
+- Added a Panel application for single-day SPBT result analysis with result-file
+  selection, minimum TARGET-change input, pair TheoRet table, pair selector,
+  selected-pair execution table, and market/trade chart.
+- Added a launcher script for the Panel application.
+- Changed notebook and Panel pair analysis to use per-row Analyze actions from
+  the Pair TheoRet grid, deferring selected-pair calculations until clicked.
+- Adjusted Panel sizing so key controls use compact widths and Pair TheoRet uses
+  content width with vertical scrolling instead of full-width paginated layout.
+- Added a FastListTemplate shell to the Panel application for sidebar controls
+  and configurable app color accents.
+- Made Plotly chart panes use all available horizontal space.
+
+## 2026-07-28 v1.0.1
+
+- Added the `spbt_day` notebook for interactive single-day backtest result
+  analysis, including SQLite result file selection from the local data
+  directory.
+- Added selector-pair loading and dense ranking by `mr_score.final`, preserving
+  rows with invalid score JSON for inspection.
+- Added theoretical return calculation for ranked pairs from
+  `trading_instructions`, including reusable helper functions and tests.
+- Added a Plotly histogram for visual analysis of total theoretical return by
+  pair.
+- Moved notebook support code into reusable `scripts/spbt_day.py` helpers.
+- Adjusted notebook table outputs to show all relevant rows and reduce
+  redundant intermediate displays.
+- Added an alphabetically sorted pair selector for individual pair analysis.
+- Added selected-pair theoretical execution tables and aligned TheoRet
+  calculations with target-delta trade generation.
+- Added per-asset `strength` values to selected-pair theoretical execution
+  tables.
+- Corrected theoretical execution size to use
+  `10000 * strength / reference_price`.
+- Removed `:USD` quote suffixes from displayed pair names in notebook tables,
+  chart hovers, and the pair selector dropdown while preserving full internal
+  pair keys for calculations.
+- Added `num_trades` to pair TheoRet summaries, counting asset-level theoretical
+  trades from effective `TARGET` and `CLOSE` instructions.
+- Added sortable interactive grids for the pair TheoRet and selected-pair
+  theoretical execution tables.
+- Styled interactive dataframe grids with black text on white backgrounds for
+  readability across notebook themes.
+- Added a selected-pair Plotly chart that overlays theoretical BUY/SELL
+  executions on relative 1-minute market close data for both instruments.
+- Anchored the selected-pair market chart at trading-day midnight and normalized
+  relative prices to each instrument's close at that timestamp.
+- Added a `min_pctg_change` threshold for ranked pair TheoRet calculations to
+  skip small target-strength changes after a position is acquired.
+- Added a notebook input field for the minimum TARGET strength-change threshold.
+
+## 2026-07-25 v0.0.9
+
+- Added contributing guidance and Python dependency declarations.
+- Added placeholder files for active project directories.
+- Updated ignore rules for local data, generated results, caches, and local
+  environments.
+- Documented unreleased changelog handling and release push behavior.
+
+## 2026-07-25 v0.0.1
+
+- Established the initial repository structure and project guidance.

CONTRIBUTING.md

@@ -0,0 +1,54 @@
+# Contributing
+
+## Setup
+
+Use the shared Python 3.12 virtual environment:
+
+```bash
+source ~/.pyenv/python3.12-venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+If you install another package for repository work, add its direct dependency
+to `requirements.txt`.
+
+## Repository layout
+
+- Put notebooks in `notebooks/`.
+- Put reusable Python and Bash utilities in `scripts/`.
+- Put local input files in `data/`.
+- Put generated artifacts in `results/`.
+
+The contents of `data/` and `results/` are ignored. Do not force-add test
+databases, raw test results, generated exports, or notebook outputs.
+
+`__SAV__/` is unrelated legacy material and is outside the active project.
+
+## Working with notebooks
+
+Notebooks must execute from top to bottom in a fresh kernel. Use relative paths,
+document data assumptions, and move reusable logic into tested scripts.
+
+Before handing off a change:
+
+```bash
+python -m pytest
+python -m pytest --nbmake notebooks
+jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace path/to/notebook.ipynb
+```
+
+Run only the checks relevant to the files present in the repository, and report
+anything that could not be run.
+
+## Review requirement
+
+Python scripts, Bash scripts, and notebook code-cell changes require review and
+approval by a separate background agent. Address material findings and rerun
+affected checks before completion. If a reviewer is unavailable, the change may
+be handed off only with the status `review pending`.
+
+Documentation, dependency declarations, and ignore rules do not require this
+background review when no Python, Bash, or notebook code cells changed.
+
+The `master` branch is not protected. That does not remove the review
+requirement or authorize an agent to commit or push without an explicit request.

TODO.md

@@ -0,0 +1,9 @@
+- [ ] Add disequilibrium chart
+- [ ] Add scatter chart for `mr-rank <-->  realized pnl`
+
+# DONE
+
+## 2026-07-29
+
+- [x] Change notebook and panel (stat_pairs_backtest) to use sp_quant's database tables `trading_instructions`  and `market`, to have *disequilibrium* and *beta*
+

notebooks/.gitkeep

@@ -0,0 +1 @@
+

panel/spbt_day_panel.py

@@ -0,0 +1,303 @@
+"""Panel application for single-day SPBT result analysis."""
+
+from __future__ import annotations
+
+from pathlib import Path
+import sys
+from typing import Any
+
+import pandas as pd
+import panel as pn
+
+
+APP_DIR = Path(__file__).resolve().parent
+REPO_ROOT = APP_DIR.parent
+if str(REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(REPO_ROOT))
+
+from scripts import spbt_day
+
+
+pn.extension("tabulator", "plotly")
+
+
+PAIR_THEO_RET_SORT_COLUMNS = ["total_pnl", "pair_name"]
+PAIR_THEO_RET_DISPLAY_DROP_COLUMNS = ["total_pnl"]
+APP_TITLE = "SPBT Day Analysis"
+APP_ACCENT_COLOR = "#226c67"
+APP_HEADER_COLOR = "#184c47"
+APP_SIDEBAR_WIDTH = 215
+APP_SIDEBAR_CONTROL_WIDTH = 200
+
+
+class SpbtDayPanelApp:
+    """Stateful Panel UI for single-day SPBT analysis."""
+
+    def __init__(self, repo_root: Path | None = None) -> None:
+        self.repo_root = (repo_root or spbt_day.find_repo_root(REPO_ROOT)).resolve()
+        self.selector_pair_rankings = pd.DataFrame()
+        self.trading_instructions = pd.DataFrame()
+        self.pair_theo_ret = pd.DataFrame()
+        self.selected_pair_theo_executions = pd.DataFrame()
+        self.selected_pair_name: str | None = None
+        self.min_pctg_change = 0.0
+
+        self.directory_input = pn.widgets.TextInput(
+            label="Directory",
+            value=str(self.repo_root / "data"),
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.show_all_files = pn.widgets.Checkbox(label="Show all files", value=False)
+        self.file_select = pn.widgets.Select(
+            label="SQLite result file",
+            options={},
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.min_pctg_change_input = pn.widgets.FloatInput(
+            label="Mininal TARGET change (%)",
+            value=0.0,
+            step=1.0,
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.calculate_button = pn.widgets.Button(
+            label="Calculate",
+            color="primary",
+            width=110,
+        )
+
+        self.status = pn.pane.Markdown("")
+        self.pair_theo_ret_table = spbt_day.create_pair_theo_ret_analyze_grid(
+            pd.DataFrame(),
+            height=420,
+        )
+        self.total_pnl_histogram = pn.pane.Plotly(
+            None,
+            height=360,
+            sizing_mode="stretch_width",
+        )
+        self.selected_pair_message = pn.pane.Markdown(
+            "Click Analyze in the Pair TheoRet grid to load individual-pair details."
+        )
+        self.selected_pair_executions_table = spbt_day.create_selected_pair_executions_grid(
+            height=320,
+        )
+        self.selected_pair_market_plot = pn.pane.Plotly(
+            None,
+            height=520,
+            sizing_mode="stretch_width",
+        )
+
+        self.calculate_button.on_click(self.calculate)
+        self.directory_input.param.watch(self.refresh_files, "value")
+        self.show_all_files.param.watch(self.refresh_files, "value")
+        self.pair_theo_ret_table.on_click(
+            self.analyze_pair_click,
+            column=spbt_day.ANALYZE_BUTTON_COLUMN,
+        )
+        self.refresh_files()
+
+    def set_status(self, message: str, *, error: bool = False) -> None:
+        """Update visible status text."""
+        prefix = "**Error:** " if error else ""
+        self.status.object = f"{prefix}{message}" if message else ""
+
+    def selected_database_path(self) -> Path:
+        """Return the selected result database path."""
+        if not self.file_select.value:
+            raise ValueError("Select a SQLite result file before calculating.")
+        db_path = Path(str(self.file_select.value)).resolve()
+        if not db_path.exists():
+            raise FileNotFoundError(f"Selected database does not exist: {db_path}")
+        if not db_path.is_file():
+            raise ValueError(f"Selected database path is not a file: {db_path}")
+        return db_path
+
+    def refresh_files(self, *_events: Any) -> bool:
+        """Refresh selectable SQLite files from the configured directory."""
+        try:
+            directory = spbt_day.normalize_directory(
+                self.directory_input.value,
+                self.repo_root,
+            )
+            candidates = spbt_day.list_candidate_files(
+                directory,
+                show_all=self.show_all_files.value,
+            )
+        except Exception as exc:
+            self.file_select.options = {}
+            self.file_select.value = None
+            self.set_status(str(exc), error=True)
+            return False
+
+        options = {path.name: str(path) for path in candidates}
+        previous_value = self.file_select.value
+        self.file_select.options = options
+        if previous_value in options.values():
+            self.file_select.value = previous_value
+        elif options:
+            self.file_select.value = next(iter(options.values()))
+        else:
+            self.file_select.value = None
+
+        if options:
+            self.set_status(f"Found {len(options):,} file(s) in {directory}.")
+        else:
+            self.set_status(f"No selectable files found in {directory}.")
+        return True
+
+    def calculate(self, *_events: Any) -> None:
+        """Load selected data and calculate all-pair TheoRet."""
+        self.calculate_button.loading = True
+        try:
+            if not self.refresh_files():
+                return
+            db_path = self.selected_database_path()
+            self.min_pctg_change = float(self.min_pctg_change_input.value)
+
+            conn = spbt_day.connect_sqlite_read_only(db_path)
+            try:
+                self.selector_pair_rankings = spbt_day.load_selector_pair_rankings(conn)
+                self.trading_instructions = spbt_day.load_trading_instructions(conn)
+            finally:
+                conn.close()
+
+            self.pair_theo_ret = (
+                spbt_day.add_total_pnl(
+                    spbt_day.calculate_ranked_pairs_theo_ret(
+                        self.selector_pair_rankings,
+                        self.trading_instructions,
+                        min_pctg_change=self.min_pctg_change,
+                    )
+                )
+                .sort_values(
+                    PAIR_THEO_RET_SORT_COLUMNS,
+                    ascending=[True, True],
+                    kind="mergesort",
+                )
+                .drop(columns=PAIR_THEO_RET_DISPLAY_DROP_COLUMNS)
+                .reset_index(drop=True)
+            )
+            self.pair_theo_ret_table.value = spbt_day.format_pair_theo_ret_for_analyze_grid(
+                self.pair_theo_ret
+            )
+            self.total_pnl_histogram.object = spbt_day.create_total_pnl_histogram(
+                self.pair_theo_ret
+            )
+            self.clear_selected_pair_analysis()
+
+            self.set_status(
+                f"Calculated {len(self.pair_theo_ret):,} pair row(s) from {db_path.name}."
+            )
+        except Exception as exc:
+            self.set_status(str(exc), error=True)
+        finally:
+            self.calculate_button.loading = False
+
+    def clear_selected_pair_analysis(self) -> None:
+        """Clear individual-pair outputs until a row Analyze button is clicked."""
+        self.selected_pair_name = None
+        self.selected_pair_theo_executions = pd.DataFrame()
+        self.selected_pair_message.object = (
+            "Click Analyze in the Pair TheoRet grid to load individual-pair details."
+        )
+        self.selected_pair_executions_table.value = pd.DataFrame(
+            columns=spbt_day.SELECTED_PAIR_EXECUTION_DISPLAY_COLUMNS
+        )
+        self.selected_pair_market_plot.object = None
+
+    def analyze_pair_click(self, event: Any) -> None:
+        """Run selected-pair analysis from a Pair TheoRet Analyze button click."""
+        self.update_selected_pair(
+            spbt_day.pair_name_from_analyze_event(self.pair_theo_ret_table, event)
+        )
+
+    def analyze_pair_row(self, row: int) -> None:
+        """Run selected-pair analysis for a Pair TheoRet table row."""
+        event = type("AnalyzeEvent", (), {"row": row})()
+        self.analyze_pair_click(event)
+
+    def update_selected_pair(self, pair_name: str) -> None:
+        """Calculate selected-pair executions and market plot."""
+        if self.trading_instructions.empty:
+            self.clear_selected_pair_analysis()
+            return
+
+        self.selected_pair_name = pair_name
+        self.selected_pair_message.object = (
+            f"Selected pair: **{spbt_day.format_pair_name_for_display(pair_name)}**"
+        )
+        self.selected_pair_theo_executions = spbt_day.calculate_pair_theo_executions(
+            pair_name,
+            self.trading_instructions,
+            min_pctg_change=self.min_pctg_change,
+        )
+        self.selected_pair_executions_table.value = (
+            self.selected_pair_theo_executions.reindex(
+                columns=spbt_day.SELECTED_PAIR_EXECUTION_DISPLAY_COLUMNS
+            )
+        )
+
+        try:
+            trading_day_start_ns = spbt_day.infer_trading_day_start_ns(
+                self.trading_instructions
+            )
+            conn = spbt_day.connect_sqlite_read_only(self.selected_database_path())
+            try:
+                selected_pair_market_data = spbt_day.load_pair_market_data(
+                    conn,
+                    pair_name,
+                    trading_day_start_ns=trading_day_start_ns,
+                )
+            finally:
+                conn.close()
+            self.selected_pair_market_plot.object = spbt_day.create_pair_trades_market_plot(
+                pair_name,
+                selected_pair_market_data,
+                self.selected_pair_theo_executions,
+            )
+        except Exception as exc:
+            self.selected_pair_market_plot.object = None
+            self.set_status(str(exc), error=True)
+
+    @property
+    def view(self) -> pn.template.FastListTemplate:
+        """Return the app layout."""
+        controls = pn.Column(
+            "## Inputs",
+            self.directory_input,
+            self.show_all_files,
+            self.file_select,
+            self.min_pctg_change_input,
+            self.calculate_button,
+            self.status,
+            width=APP_SIDEBAR_CONTROL_WIDTH,
+        )
+        main = pn.Column(
+            "## Pair TheoRet",
+            self.pair_theo_ret_table,
+            self.total_pnl_histogram,
+            "## Individual Pair",
+            self.selected_pair_message,
+            "### Theoretical Executions",
+            self.selected_pair_executions_table,
+            "### Trades on Market Data",
+            self.selected_pair_market_plot,
+        )
+        return pn.template.FastListTemplate(
+            title=APP_TITLE,
+            sidebar=[controls],
+            main=[main],
+            sidebar_width=APP_SIDEBAR_WIDTH,
+            accent_base_color=APP_ACCENT_COLOR,
+            header_background=APP_HEADER_COLOR,
+            main_layout=None,
+            theme=pn.template.DarkTheme,
+        )
+
+
+app_controller = SpbtDayPanelApp()
+app = app_controller.view
+app.servable(title=APP_TITLE)

requirements.txt

@@ -0,0 +1,14 @@
+# Interactive analysis
+ipykernel>=6.29,<7
+ipywidgets>=8.1,<9
+itables>=2.2,<3
+jupyter>=1.1,<2
+jupyter_bokeh>=4.0,<5
+nbformat>=5.10,<6
+pandas>=2.2,<3
+panel>=1.5,<2
+plotly>=5.24,<7
+
+# Verification
+nbmake>=1.5,<2
+pytest>=8,<9

results/.gitkeep

@@ -0,0 +1 @@
+

scripts/.gitkeep

@@ -0,0 +1 @@
+

scripts/run_spbt_day_panel.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$repo_root"
+
+panel serve panel/spbt_day_panel.py --show "$@"

src/pt_backtest.py

@@ -1,731 +0,0 @@
-import datetime
-import sys
-import json
-
-from typing import Any, Dict, List, Tuple, Optional
-
-import pandas as pd
-import numpy as np
-
-# ============= statsmodels ===================
-from statsmodels.tsa.vector_ar.vecm import VECM
-
-NanoPerMin = 1e9
-UNSET_FLOAT: float = sys.float_info.max
-UNSET_INT: int = sys.maxsize
-
-# ------------------------ Configuration ------------------------
-# Default configuration
-CRYPTO_CONFIG: Dict = {
-    # --- Data retrieval
-    "data_directory": "./data/crypto",
-    "datafiles": [
-        "20250519.mktdata.ohlcv.db",
-    ],
-    "db_table_name": "bnbspot_ohlcv_1min",
- 
-    # ----- Instruments
-    "exchange_id": "BNBSPOT",
-    "instrument_id_pfx": "PAIR-",
-
-    "instruments": [
-        "BTC-USDT",
-        "ETH-USDT",
-        "LTC-USDT",
-    ],    
-    
-    "trading_hours": {
-        "begin_session": "00:00:00", 
-        "end_session": "23:59:00", 
-        "timezone": "UTC"
-    },
-
-    # ----- Model Settings
-    "price_column": "close",
-    "min_required_points": 30,
-    "zero_threshold": 1e-10,
-    "equilibrium_threshold_open": 5.0,
-    "equilibrium_threshold_close": 1.0,
-    "training_minutes": 120,
-
-    # ----- Validation
-    "funding_per_pair": 2000.0,  # USD
-}
-# ========================== EQUITIES
-EQT_CONFIG: Dict = {
-    # --- Data retrieval
-    "data_directory": "./data/equity",
-    "datafiles": [
-        "20250508.alpaca_sim_md.db",
-        # "20250509.alpaca_sim_md.db",
-        # "20250512.alpaca_sim_md.db",
-        # "20250513.alpaca_sim_md.db",
-        # "20250514.alpaca_sim_md.db",
-        # "20250515.alpaca_sim_md.db",
-        # "20250516.alpaca_sim_md.db",
-        # "20250519.alpaca_sim_md.db",
-        # "20250520.alpaca_sim_md.db"
-    ],
-    "db_table_name": "md_1min_bars",
-
-     # ----- Instruments
-   "exchange_id": "ALPACA",
-    "instrument_id_pfx": "STOCK-",
-
-    "instruments": [
-        "COIN",
-        "GBTC",
-        "HOOD",
-        "MSTR",
-        "PYPL",
-    ],
-    
-    "trading_hours": {
-        "begin_session": "9:30:00", 
-        "end_session": "16:00:00", 
-        "timezone": "America/New_York"
-    },
-    
-    # ----- Model Settings
-    "price_column": "close",
-    "min_required_points": 30,
-    "zero_threshold": 1e-10,
-    "equilibrium_threshold_open": 5.0,
-    "equilibrium_threshold_close": 1.0,
-    "training_minutes": 120,
-
-    # ----- Validation
-    "funding_per_pair": 2000.0,
-}
-
-# ==========================================================================
-CONFIG = EQT_CONFIG
-TRADES = {}
-TOTAL_UNREALIZED_PNL = 0.0  # Global variable to track total unrealized PnL
-TOTAL_REALIZED_PNL = 0.0    # Global variable to track total realized PnL
-OUTSTANDING_POSITIONS = []  # Global list to track outstanding positions with share quantities
-
-class TradingPair:
-    symbol_a_: str
-    symbol_b_: str
-    price_column_: str
-
-    def __init__(self, symbol_a: str, symbol_b: str, price_column: str):
-        self.symbol_a_ = symbol_a
-        self.symbol_b_ = symbol_b
-        self.price_column_ = price_column
-
-    def colnames(self) -> List[str]:
-        return [f"{self.price_column_}_{self.symbol_a_}", f"{self.price_column_}_{self.symbol_b_}"]
-
-    def __repr__(self) ->str:
-        return f"{self.symbol_a_} & {self.symbol_b_}"
-
-def convert_time_to_UTC(value: str, timezone: str):
-
-    from zoneinfo import ZoneInfo
-    from datetime import datetime
-
-    # Parse it to naive datetime object
-    local_dt = datetime.strptime(value, '%Y-%m-%d %H:%M:%S')
-
-    zinfo = ZoneInfo(timezone)
-    result = local_dt.replace(tzinfo=zinfo)
-    
-    result = result.astimezone(ZoneInfo('UTC'))
-    result = result.strftime('%Y-%m-%d %H:%M:%S')
-
-    return result
-
-
-    pass
-    
-def load_market_data(datafile: str, config: Dict) -> pd.DataFrame:
-    from tools.data_loader import load_sqlite_to_dataframe
-
-    instrument_ids = ["\"" + config["instrument_id_pfx"] + instrument + "\"" for instrument in config["instruments"]]
-    exchange_id = config["exchange_id"]
-
-    query = "select tstamp"
-    query += ", tstamp_ns as time_ns"
-    query += ", substr(instrument_id, 7) as symbol"
-    query += ", open"
-    query += ", high"
-    query += ", low"
-    query += ", close"
-    query += ", volume"
-    query += ", num_trades"
-    query += ", vwap"
-
-    query += f" from {config['db_table_name']}"
-    query += f" where exchange_id ='{exchange_id}'"
-    query += f" and instrument_id in ({','.join(instrument_ids)})"
-
-    df = load_sqlite_to_dataframe(db_path=datafile, query=query)
-
-    # Trading Hours
-    date_str = df["tstamp"][0][0:10]
-    trading_hours = CONFIG['trading_hours']
-    start_time = f"{date_str} {trading_hours['begin_session']}"
-    end_time = f"{date_str} {trading_hours['end_session']}"
-    
-    start_time = convert_time_to_UTC(start_time, trading_hours["timezone"])
-    end_time = convert_time_to_UTC(end_time, trading_hours["timezone"])
-
-    # Perform boolean selection
-    df = df[(df["tstamp"] >= start_time) & (df["tstamp"] <= end_time)]
-    df["tstamp"] = pd.to_datetime(df["tstamp"])
-
-    return df
-
-def transform_dataframe(df: pd.DataFrame, price_column: str):
-    # Select only the columns we need
-    df_selected = df[["tstamp", "symbol", price_column]]
-
-    # Start with unique timestamps
-    result_df: pd.DataFrame = pd.DataFrame(df_selected["tstamp"]).drop_duplicates().reset_index(drop=True)
-
-    # For each unique symbol, add a corresponding close price column
-    for symbol in df_selected["symbol"].unique():
-        # Filter rows for this symbol
-        df_symbol = df_selected[df_selected["symbol"] == symbol].reset_index(drop=True)
-
-        # Create column name like "close-COIN"
-        new_price_column = f"{price_column}_{symbol}"
-
-        # Create temporary dataframe with timestamp and price
-        temp_df = pd.DataFrame({
-            "tstamp": df_symbol["tstamp"],
-            new_price_column: df_symbol[price_column]
-        })
-
-        # Join with our result dataframe
-        result_df = pd.merge(result_df, temp_df, on="tstamp", how="left")
-        result_df = result_df.reset_index(drop=True) # do not dropna() since irrelevant symbol would affect dataset
-
-    return result_df
-
-def get_datasets(df: pd.DataFrame, training_minutes: int, pair: TradingPair) -> Tuple[pd.DataFrame, pd.DataFrame]:
-    # Training dataset
-    colname_a, colname_b = pair.colnames()
-    df = df[["tstamp", colname_a, colname_b]]
-    df = df.dropna()
-
-    training_df = df.iloc[:training_minutes - 1, :].copy()
-    training_df.reset_index(drop=True).dropna().reset_index(drop=True)
-
-    # Testing dataset
-    testing_df = df.iloc[training_minutes:, :].copy()
-    testing_df.reset_index(drop=True).dropna().reset_index(drop=True)
-
-    return (training_df, testing_df)
-
-def fit_VECM(training_pair_df, pair: TradingPair):
-    vecm_model = VECM(training_pair_df[pair.colnames()].reset_index(drop=True), coint_rank=1)
-    vecm_fit = vecm_model.fit()
-
-    # Check if the model converged properly
-    if not hasattr(vecm_fit, "beta") or vecm_fit.beta is None:
-        print(f"{pair}: VECM model failed to converge properly")
-
-    return vecm_fit
-
-def create_trading_signals(vecm_fit, testing_pair_df, pair: TradingPair) -> pd.DataFrame:
-    result_columns = [
-        "time",
-        "action",
-        "symbol",
-        "price",
-        "equilibrium",
-        "pair",
-    ]
-
-    next_values = vecm_fit.predict(steps=len(testing_pair_df))
-    colname_a, colname_b = pair.colnames()
-
-    # Convert prediction to a DataFrame for readability
-    predicted_df = pd.DataFrame(next_values, columns=[colname_a, colname_b])
-
-    beta = vecm_fit.beta
-
-    predicted_df["equilibrium_term"] = (
-        beta[0] * predicted_df[colname_a]
-        + beta[1] * predicted_df[colname_b]
-    )
-
-    pair_result_df = pd.merge(
-        testing_pair_df.reset_index(drop=True), predicted_df, left_index=True, right_index=True, suffixes=('', '_pred')
-    ).dropna()
-
-    pair_result_df["equilibrium"] = (
-        beta[0] * pair_result_df[colname_a]
-        + beta[1] * pair_result_df[colname_b]
-    )
-
-    pair_result_df["abs_equilibrium"] = np.abs(pair_result_df["equilibrium"])
-
-    # Reset index to ensure proper indexing
-    pair_result_df = pair_result_df.reset_index()
-
-    # Iterate through the testing dataset to find the first trading opportunity
-    open_row_index = None
-    initial_abs_term = None
-
-    for row_idx in range(len(pair_result_df)):
-        current_abs_term = pair_result_df["abs_equilibrium"][row_idx]
-
-        # Check if current row has sufficient equilibrium (not near-zero)
-        if current_abs_term >= CONFIG["equilibrium_threshold_open"]:
-            open_row_index = row_idx
-            initial_abs_term = current_abs_term
-            break
-
-    # If no row with sufficient equilibrium found, skip this pair
-    if open_row_index is None:
-        print(f"{pair}: Insufficient divergence in testing dataset. Skipping.")
-        return pd.DataFrame()
-
-    # Look for close signal starting from the open position
-    trading_signals_df = (
-        pair_result_df["abs_equilibrium"][open_row_index:]
-        # < initial_abs_term / CONFIG["equilibrium_threshold_close"]
-        < CONFIG["equilibrium_threshold_close"]
-    )
-
-    # Adjust indices to account for the offset from open_row_index
-    close_row_index = None
-    for idx, value in trading_signals_df.items():
-        if value:
-            close_row_index = idx
-            break
-
-    open_row = pair_result_df.loc[open_row_index]
-    open_tstamp = open_row["tstamp"]
-    open_eqlbrm = open_row["equilibrium"]
-    open_px_a = open_row[f"{colname_a}"]
-    open_px_b = open_row[f"{colname_b}"]
-
-    abs_beta = abs(beta[1])
-    pred_px_b = pair_result_df.loc[open_row_index][f"{colname_b}_pred"]
-    pred_px_a = pair_result_df.loc[open_row_index][f"{colname_a}_pred"]
-
-    if pred_px_b * abs_beta - pred_px_a > 0:
-        open_side_a = "BUY"
-        open_side_b = "SELL"
-        close_side_a = "SELL"
-        close_side_b = "BUY"
-    else:
-        open_side_b = "BUY"
-        open_side_a = "SELL"
-        close_side_b = "SELL"
-        close_side_a = "BUY"
-
-    # If no close signal found, print position and unrealized PnL
-    if close_row_index is None:
-        global TOTAL_UNREALIZED_PNL, OUTSTANDING_POSITIONS
-
-        last_row_index = len(pair_result_df) - 1
-        last_row = pair_result_df.loc[last_row_index]
-        last_tstamp = last_row["tstamp"]
-        last_px_a = last_row[f"{colname_a}"]
-        last_px_b = last_row[f"{colname_b}"]
-
-        # Calculate share quantities based on $1000 funding per pair
-        # Split $1000 equally between the two positions ($500 each)
-        funding_per_position = CONFIG["funding_per_pair"] / 2
-        shares_a = funding_per_position / open_px_a
-        shares_b = funding_per_position / open_px_b
-
-        # Calculate unrealized PnL for each position
-        if open_side_a == "BUY":
-            unrealized_pnl_a = (last_px_a - open_px_a) / open_px_a * 100
-            unrealized_dollar_a = shares_a * (last_px_a - open_px_a)
-        else:  # SELL
-            unrealized_pnl_a = (open_px_a - last_px_a) / open_px_a * 100
-            unrealized_dollar_a = shares_a * (open_px_a - last_px_a)
-
-        if open_side_b == "BUY":
-            unrealized_pnl_b = (last_px_b - open_px_b) / open_px_b * 100
-            unrealized_dollar_b = shares_b * (last_px_b - open_px_b)
-        else:  # SELL
-            unrealized_pnl_b = (open_px_b - last_px_b) / open_px_b * 100
-            unrealized_dollar_b = shares_b * (open_px_b - last_px_b)
-
-        total_unrealized_pnl = unrealized_pnl_a + unrealized_pnl_b
-        total_unrealized_dollar = unrealized_dollar_a + unrealized_dollar_b
-
-        # Add to global total
-        TOTAL_UNREALIZED_PNL += total_unrealized_pnl
-
-        # Store outstanding positions
-        OUTSTANDING_POSITIONS.append({
-            'pair': str(pair),
-            'symbol_a': pair.symbol_a_,
-            'symbol_b': pair.symbol_b_,
-            'side_a': open_side_a,
-            'side_b': open_side_b,
-            'shares_a': shares_a,
-            'shares_b': shares_b,
-            'open_px_a': open_px_a,
-            'open_px_b': open_px_b,
-            'current_px_a': last_px_a,
-            'current_px_b': last_px_b,
-            'unrealized_dollar_a': unrealized_dollar_a,
-            'unrealized_dollar_b': unrealized_dollar_b,
-            'total_unrealized_dollar': total_unrealized_dollar,
-            'open_time': open_tstamp,
-            'last_time': last_tstamp,
-            'initial_abs_term': initial_abs_term,
-            'current_abs_term': pair_result_df.loc[last_row_index, "abs_equilibrium"],
-            'closing_threshold': initial_abs_term / CONFIG["equilibrium_threshold_close"],
-            'equilibrium_ratio': pair_result_df.loc[last_row_index, "abs_equilibrium"] / (initial_abs_term / CONFIG["equilibrium_threshold_close"])
-        })
-
-        print(f"{pair}: NO CLOSE SIGNAL FOUND - Position held until end of session")
-        print(f"  Open: {open_tstamp} | Last: {last_tstamp}")
-        print(f"  {pair.symbol_a_}: {open_side_a} {shares_a:.2f} shares @ ${open_px_a:.2f} -> ${last_px_a:.2f} | Unrealized: ${unrealized_dollar_a:.2f} ({unrealized_pnl_a:.2f}%)")
-        print(f"  {pair.symbol_b_}: {open_side_b} {shares_b:.2f} shares @ ${open_px_b:.2f} -> ${last_px_b:.2f} | Unrealized: ${unrealized_dollar_b:.2f} ({unrealized_pnl_b:.2f}%)")
-        print(f"  Total Unrealized: ${total_unrealized_dollar:.2f} ({total_unrealized_pnl:.2f}%)")
-
-        # Return only open trades (no close trades)
-        trd_signal_tuples = [
-            (
-                open_tstamp,
-                open_side_a,
-                pair.symbol_a_,
-                open_px_a,
-                open_eqlbrm,
-                pair,
-            ),
-            (
-                open_tstamp,
-                open_side_b,
-                pair.symbol_b_,
-                open_px_b,
-                open_eqlbrm,
-                pair,
-            ),
-        ]
-    else:
-        # Close signal found - create complete trade
-        close_row = pair_result_df.loc[close_row_index]
-        close_tstamp = close_row["tstamp"]
-        close_eqlbrm = close_row["equilibrium"]
-        close_px_a = close_row[f"{colname_a}"]
-        close_px_b = close_row[f"{colname_b}"]
-
-        print(f"{pair}: Close signal found at index {close_row_index}")
-
-        trd_signal_tuples = [
-            (
-                open_tstamp,
-                open_side_a,
-                pair.symbol_a_,
-                open_px_a,
-                open_eqlbrm,
-                pair,
-            ),
-            (
-                open_tstamp,
-                open_side_b,
-                pair.symbol_b_,
-                open_px_b,
-                open_eqlbrm,
-                pair,
-            ),
-            (
-                close_tstamp,
-                close_side_a,
-                pair.symbol_a_,
-                close_px_a,
-                close_eqlbrm,
-                pair,
-            ),
-            (
-                close_tstamp,
-                close_side_b,
-                pair.symbol_b_,
-                close_px_b,
-                close_eqlbrm,
-                pair,
-            ),
-        ]
-
-    # Add tuples to data frame
-    return pd.DataFrame(
-        trd_signal_tuples,
-        columns=result_columns,
-    )
-
-def run_single_pair(market_data: pd.DataFrame, price_column:str, pair: TradingPair) -> Optional[pd.DataFrame]:
-    colname_a = f"{price_column}_{pair.symbol_a_}"
-    colname_b = f"{price_column}_{pair.symbol_b_}"
-    training_pair_df, testing_pair_df = get_datasets(df=market_data, training_minutes=CONFIG["training_minutes"], pair=pair)
-
-    # Check if we have enough data points for a meaningful analysis
-    min_required_points = CONFIG[
-        "min_required_points"
-    ]  # Minimum number of points for a reasonable VECM model
-    if len(training_pair_df) < min_required_points:
-        print(
-            f"{pair}: Not enough data points for analysis. Found {len(training_pair_df)}, need at least {min_required_points}"
-        )
-        return None
-
-    # Check for non-finite values
-    if not np.isfinite(training_pair_df).all().all():
-        print(f"{pair}: Data contains non-finite values (NaN or inf)")
-        return None
-
-    # Fit the VECM
-    try:
-        vecm_fit = fit_VECM(training_pair_df, pair=pair)
-    except Exception as e:
-        print(f"{pair}: VECM fitting failed: {str(e)}")
-        return None
-
-    # Add safeguard against division by zero
-    if (
-        abs(vecm_fit.beta[1]) < CONFIG["zero_threshold"]
-    ):  # Small threshold to avoid division by very small numbers
-        print(f"{pair}: Skipping due to near-zero beta[1] value: {vecm_fit.beta[1]}")
-        return None
-
-    try:
-        pair_trades = create_trading_signals(
-                vecm_fit=vecm_fit,
-                testing_pair_df=testing_pair_df,
-                pair=pair,
-        )
-    except Exception as e:
-        print(f"{pair}: Prediction failed: {str(e)}")
-        return None
-
-    return pair_trades
-
-def add_trade(pair_nm, symbol, action, price):
-    pair_nm = str(pair_nm)
-
-
-    if pair_nm not in TRADES:
-        TRADES[pair_nm] = {symbol: []}
-    if symbol not in TRADES[pair_nm]:
-        TRADES[pair_nm][symbol] = []
-    TRADES[pair_nm][symbol].append((action, price))
-
-def collect_single_day_results(result):
-    if result is None:
-        return
-
-    print("\n --------------  Suggested Trades ")
-    print(result)
-
-    for row in result.itertuples():
-        action = row.action
-        symbol = row.symbol
-        price = row.price
-        add_trade(pair_nm=row.pair, action=action, symbol=symbol, price=price)
-
-def print_single_day_results(result):
-    for pair, symbols in TRADES.items():
-        print(f"\n--- {pair} ---")
-        for symbol, trades in symbols.items():
-            for side, price in trades:
-                print(f"{symbol} {side} at ${price}")
-
-def print_results_suummary(all_results):
-    # Summary of all processed files
-    print("\n====== Summary of All Processed Files ======")
-    for filename, data in all_results.items():
-        trade_count = sum(
-            len(trades)
-            for symbol_trades in data["trades"].values()
-            for trades in symbol_trades.values()
-        )
-        print(f"{filename}: {trade_count} trades")
-
-
-def calculate_returns(all_results: Dict):
-    global TOTAL_REALIZED_PNL
-    print("\n====== Returns By Day and Pair ======")
-
-    for filename, data in all_results.items():
-        day_return = 0
-        print(f"\n--- {filename} ---")
-
-        # Process each pair
-        for pair, symbols in data["trades"].items():
-            pair_return = 0
-            pair_trades = []
-
-            # Calculate individual symbol returns in the pair
-            for symbol, trades in symbols.items():
-                if len(trades) >= 2:  # Need at least entry and exit
-                    # Get entry and exit trades
-                    entry_action, entry_price = trades[0]
-                    exit_action, exit_price = trades[1]
-
-                    # Calculate return based on action
-                    symbol_return = 0
-                    if entry_action == "BUY" and exit_action == "SELL":
-                        # Long position
-                        symbol_return = (exit_price - entry_price) / entry_price * 100
-                    elif entry_action == "SELL" and exit_action == "BUY":
-                        # Short position
-                        symbol_return = (entry_price - exit_price) / entry_price * 100
-
-                    pair_trades.append(
-                        (
-                            symbol,
-                            entry_action,
-                            entry_price,
-                            exit_action,
-                            exit_price,
-                            symbol_return,
-                        )
-                    )
-                    pair_return += symbol_return
-
-            # Print pair returns
-            if pair_trades:
-                print(f"  {pair}:")
-                for (
-                    symbol,
-                    entry_action,
-                    entry_price,
-                    exit_action,
-                    exit_price,
-                    symbol_return,
-                ) in pair_trades:
-                    print(
-                        f"    {symbol}: {entry_action} @ ${entry_price:.2f}, {exit_action} @ ${exit_price:.2f}, Return: {symbol_return:.2f}%"
-                    )
-                print(f"    Pair Total Return: {pair_return:.2f}%")
-                day_return += pair_return
-
-        # Print day total return and add to global realized PnL
-        if day_return != 0:
-            print(f"  Day Total Return: {day_return:.2f}%")
-            TOTAL_REALIZED_PNL += day_return
-
-def run_pairs(summaries_df: pd.DataFrame, price_column: str) -> None:
-
-    result_df = transform_dataframe(df=summaries_df, price_column=price_column)
-
-    stock_price_columns = [
-        column
-        for column in result_df.columns
-        if column.startswith(f"{price_column}_")
-    ]
-
-    # Find the starting indices for A and B
-    all_indexes = range(len(stock_price_columns))
-    unique_index_pairs = [(i, j) for i in all_indexes for j in all_indexes if i < j]
-
-    pairs_trades = []
-    for a_index, b_index in unique_index_pairs:
-        # Get the actual variable names
-        colname_a = stock_price_columns[a_index]
-        colname_b = stock_price_columns[b_index]
-
-        symbol_a = colname_a[len(f"{price_column}-") :]
-        symbol_b = colname_b[len(f"{price_column}-") :] 
-        pair = TradingPair(symbol_a, symbol_b, price_column)
-
-        single_pair_trades = run_single_pair(market_data=result_df, price_column=price_column, pair=pair)
-        if len(single_pair_trades) > 0:
-            pairs_trades.append(single_pair_trades)
-    # Check if result_list has any data before concatenating
-    if len(pairs_trades) == 0:
-        print("No trading signals found for any pairs")
-        return None
-
-    result = pd.concat(pairs_trades, ignore_index=True)
-    result["time"] = pd.to_datetime(result["time"])
-    result = result.set_index("time").sort_index()
-
-    collect_single_day_results(result)
-    # print_single_day_results(result)
-
-def print_outstanding_positions():
-    """Print all outstanding positions with share quantities and unrealized PnL"""
-    if not OUTSTANDING_POSITIONS:
-        print("\n====== NO OUTSTANDING POSITIONS ======")
-        return
-
-    print(f"\n====== OUTSTANDING POSITIONS ======")
-    print(f"{'Pair':<15} {'Symbol':<6} {'Side':<4} {'Shares':<10} {'Open $':<8} {'Current $':<10} {'Unrealized $':<12} {'%':<8} {'Close Eq':<10}")
-    print("-" * 105)
-
-    total_unrealized_dollar = 0.0
-
-    for pos in OUTSTANDING_POSITIONS:
-        # Print position A
-        print(f"{pos['pair']:<15} {pos['symbol_a']:<6} {pos['side_a']:<4} {pos['shares_a']:<10.2f} {pos['open_px_a']:<8.2f} {pos['current_px_a']:<10.2f} {pos['unrealized_dollar_a']:<12.2f} {pos['unrealized_dollar_a']/500*100:<8.2f} {'':<10}")
-
-        # Print position B
-        print(f"{'':<15} {pos['symbol_b']:<6} {pos['side_b']:<4} {pos['shares_b']:<10.2f} {pos['open_px_b']:<8.2f} {pos['current_px_b']:<10.2f} {pos['unrealized_dollar_b']:<12.2f} {pos['unrealized_dollar_b']/500*100:<8.2f} {'':<10}")
-
-        # Print pair totals with equilibrium info
-        equilibrium_status = "CLOSE" if pos['current_abs_term'] < pos['closing_threshold'] else f"{pos['equilibrium_ratio']:.2f}x"
-        print(f"{'':<15} {'PAIR':<6} {'TOT':<4} {'':<10} {'':<8} {'':<10} {pos['total_unrealized_dollar']:<12.2f} {pos['total_unrealized_dollar']/1000*100:<8.2f} {equilibrium_status:<10}")
-
-        # Print equilibrium details
-        print(f"{'':<15} {'EQ':<6} {'INFO':<4} {'':<10} {'':<8} {'':<10} {'Curr:':<6}{pos['current_abs_term']:<6.4f} {'Thresh:':<7}{pos['closing_threshold']:<6.4f} {'':<10}")
-        print("-" * 105)
-
-        total_unrealized_dollar += pos['total_unrealized_dollar']
-
-    print(f"{'TOTAL OUTSTANDING':<80} ${total_unrealized_dollar:<12.2f}")
-
-if __name__ == "__main__":
-    # Initialize a dictionary to store all trade results
-    all_results = {}
-
-    # Initialize global PnL tracking variables
-    TOTAL_REALIZED_PNL = 0.0
-    TOTAL_UNREALIZED_PNL = 0.0
-    OUTSTANDING_POSITIONS = []
-
-    # Process each data file
-    price_column = CONFIG["price_column"]
-    for datafile in CONFIG["datafiles"]:
-        print(f"\n====== Processing {datafile} ======")
-
-        # Clear the TRADES global dictionary and reset unrealized PnL for the new file
-        TRADES.clear()
-        TOTAL_UNREALIZED_PNL = 0.0
-        TOTAL_REALIZED_PNL = 0.0
-
-        # Process data for this file
-        try:
-            run_pairs(
-                summaries_df=load_market_data(f'{CONFIG["data_directory"]}/{datafile}', config=CONFIG),
-                price_column=price_column
-            )
-
-            # Store results with file name as key
-            filename = datafile.split("/")[-1]
-            all_results[filename] = {"trades": TRADES.copy()}
-
-            print(f"Successfully processed {filename}")
-
-            # Print total unrealized PnL for this file
-            if TOTAL_UNREALIZED_PNL != 0:
-                print(f"\n====== TOTAL UNREALIZED PnL for {filename}: {TOTAL_UNREALIZED_PNL:.2f}% ======")
-            else:
-                print(f"\n====== No unrealized positions for {filename} ======")
-
-        except Exception as e:
-            print(f"Error processing {datafile}: {str(e)}")
-
-    # print_results_suummary(all_results)
-    calculate_returns(all_results)
-
-    # Print grand totals
-    print(f"\n====== GRAND TOTALS ACROSS ALL PAIRS ======")
-    print(f"Total Realized PnL: {TOTAL_REALIZED_PNL:.2f}%")
-    print(f"Total Unrealized PnL: {TOTAL_UNREALIZED_PNL:.2f}%")
-    print(f"Combined Total PnL: {TOTAL_REALIZED_PNL + TOTAL_UNREALIZED_PNL:.2f}%")
-
-    print_outstanding_positions()

src/tools/data_loader.py

@@ -1,25 +0,0 @@
-
-import sys
-import sqlite3
-import pandas as pd
-
-def load_sqlite_to_dataframe(db_path, query):
-    try:
-        conn = sqlite3.connect(db_path)
-
-        df = pd.read_sql_query(query, conn)
-        return df
-    except sqlite3.Error as excpt:
-        print(f"SQLite error: {excpt}")
-        raise
-    except Exception as e:
-        print(f"Error: {excpt}")
-        raise
-    finally:
-        if 'conn' in locals():
-            conn.close()
-
-if __name__ == "__main__":
-    df1 = load_sqlite_to_dataframe(sys.argv[1], table_name='md_1min_bars')
-
-    print(df1)

tests/test_spbt_day_panel.py

@@ -0,0 +1,243 @@
+import importlib.util
+import sqlite3
+from pathlib import Path
+
+import pandas as pd
+
+
+def load_panel_app_module():
+    module_path = Path("panel/spbt_day_panel.py").resolve()
+    spec = importlib.util.spec_from_file_location("spbt_day_panel_app", module_path)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+def create_panel_fixture_db(db_path: Path) -> None:
+    trading_day_start_ns = pd.Timestamp("2026-06-17T00:00:00Z").value
+    conn = sqlite3.connect(db_path)
+    try:
+        conn.execute(
+            """
+            CREATE TABLE selector_pairs (
+                time_ns INTEGER,
+                tstamp TEXT,
+                pair_name TEXT,
+                instrument_a TEXT,
+                instrument_b TEXT,
+                mr_score TEXT
+            )
+            """
+        )
+        conn.execute(
+            """
+            CREATE TABLE trading_instructions (
+                tstamp TEXT,
+                tstamp_ns INTEGER,
+                type TEXT,
+                book_id TEXT,
+                strategy_id TEXT,
+                action TEXT,
+                quote_asset TEXT,
+                assets TEXT,
+                scaled_disequilibrium REAL,
+                beta REAL
+            )
+            """
+        )
+        conn.execute(
+            """
+            CREATE TABLE market (
+                tstamp TEXT,
+                tstamp_ns INTEGER,
+                exch_acct TEXT,
+                instrument_id TEXT,
+                open REAL,
+                high REAL,
+                low REAL,
+                close REAL,
+                volume REAL,
+                vwap REAL,
+                num_trades INTEGER
+            )
+            """
+        )
+        conn.execute(
+            "INSERT INTO selector_pairs VALUES (?, ?, ?, ?, ?, ?)",
+            (
+                10,
+                "2026-06-17T00:00:00Z",
+                "AAA:USD-BBB:USD",
+                "EXCH:PAIR-AAA-USD",
+                "EXCH:PAIR-BBB-USD",
+                '{"final":"0.5"}',
+            ),
+        )
+        conn.executemany(
+            "INSERT INTO trading_instructions VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+            [
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "TARGET_POSITION",
+                    "book",
+                    "strategy-AAA:USD-BBB:USD",
+                    "TARGET",
+                    "USD",
+                    '{"AAA":{"reference_price":"100","strength":"0.5"},'
+                    '"BBB":{"reference_price":"50","strength":"-0.5"}}',
+                    -1.25,
+                    0.75,
+                ),
+                (
+                    "2026-06-17T00:01:00Z",
+                    trading_day_start_ns + 60_000_000_000,
+                    "CLOSE_POSITION",
+                    "book",
+                    "strategy-AAA:USD-BBB:USD",
+                    "CLOSE",
+                    "USD",
+                    '{"AAA":{"reference_price":"110"},'
+                    '"BBB":{"reference_price":"45"}}',
+                    -0.5,
+                    0.75,
+                ),
+            ],
+        )
+        conn.executemany(
+            "INSERT INTO market VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+            [
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "EXCH",
+                    "PAIR-AAA-USD",
+                    100.0,
+                    100.0,
+                    100.0,
+                    100.0,
+                    1.0,
+                    100.0,
+                    1,
+                ),
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "EXCH",
+                    "PAIR-BBB-USD",
+                    50.0,
+                    50.0,
+                    50.0,
+                    50.0,
+                    1.0,
+                    50.0,
+                    1,
+                ),
+            ],
+        )
+        conn.commit()
+    finally:
+        conn.close()
+
+
+def test_pair_analyze_grid_keeps_clean_labels_and_full_pair_values():
+    module = load_panel_app_module()
+    pair_theo_ret = pd.DataFrame(
+        {
+            "pair_name": ["BTC:USD-ETH:USD", "ADA:USD-BTC:USD"],
+            "mr_ranking": [2, 1],
+            "realized_pnl": [0.0, 0.0],
+            "unrealized_pnl": [0.0, 0.0],
+        }
+    )
+
+    formatted = module.spbt_day.format_pair_theo_ret_for_analyze_grid(pair_theo_ret)
+
+    assert formatted["pair_name"].tolist() == ["BTC-ETH", "ADA-BTC"]
+    assert formatted[module.spbt_day.PAIR_NAME_VALUE_COLUMN].tolist() == [
+        "BTC:USD-ETH:USD",
+        "ADA:USD-BTC:USD",
+    ]
+
+
+def test_panel_app_uses_fast_list_template(tmp_path):
+    module = load_panel_app_module()
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    view = app.view
+
+    assert not hasattr(app, "refresh_button")
+    assert isinstance(view, module.pn.template.FastListTemplate)
+    assert view.title == module.APP_TITLE
+    assert view.theme is module.pn.template.DarkTheme
+    assert view.sidebar_width == module.APP_SIDEBAR_WIDTH
+    assert view.accent_base_color == module.APP_ACCENT_COLOR
+    assert view.header_background == module.APP_HEADER_COLOR
+    assert len(view.sidebar) == 1
+    assert len(view.main) == 1
+
+
+def test_panel_app_calculates_pairs_and_selected_pair_outputs(tmp_path):
+    module = load_panel_app_module()
+    data_dir = tmp_path / "data"
+    data_dir.mkdir()
+    db_path = data_dir / "20260617.spbt_results.db"
+    create_panel_fixture_db(db_path)
+
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    app.directory_input.value = str(data_dir)
+    app.refresh_files()
+    app.min_pctg_change_input.value = 0.0
+
+    app.calculate()
+
+    assert app.file_select.value == str(db_path)
+    assert app.directory_input.sizing_mode == "stretch_width"
+    assert app.directory_input.width is None
+    assert app.file_select.sizing_mode == "stretch_width"
+    assert app.file_select.width is None
+    assert app.min_pctg_change_input.sizing_mode == "stretch_width"
+    assert app.min_pctg_change_input.width is None
+    assert app.calculate_button.width == 110
+    assert app.total_pnl_histogram.sizing_mode == "stretch_width"
+    assert app.selected_pair_market_plot.sizing_mode == "stretch_width"
+    assert app.pair_theo_ret_table.pagination is None
+    assert app.pair_theo_ret_table.layout == "fit_data_table"
+    assert app.pair_theo_ret_table.value["pair_name"].tolist() == ["AAA-BBB"]
+    assert (
+        app.pair_theo_ret_table.value[module.spbt_day.PAIR_NAME_VALUE_COLUMN].tolist()
+        == ["AAA:USD-BBB:USD"]
+    )
+    assert app.selected_pair_name is None
+    assert app.selected_pair_executions_table.value.empty
+    assert app.selected_pair_market_plot.object is None
+
+    app.analyze_pair_row(0)
+
+    assert app.selected_pair_name == "AAA:USD-BBB:USD"
+    assert app.selected_pair_executions_table.value["action"].tolist() == [
+        "TARGET",
+        "TARGET",
+        "CLOSE",
+        "CLOSE",
+    ]
+    assert app.selected_pair_market_plot.object is not None
+
+
+def test_calculate_refreshes_file_list_before_loading(tmp_path):
+    module = load_panel_app_module()
+    data_dir = tmp_path / "data"
+    data_dir.mkdir()
+
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    app.directory_input.value = str(data_dir)
+    app.refresh_files()
+    assert app.file_select.value is None
+
+    db_path = data_dir / "20260617.spbt_results.db"
+    create_panel_fixture_db(db_path)
+
+    app.calculate()
+
+    assert app.file_select.value == str(db_path)
+    assert app.pair_theo_ret_table.value["pair_name"].tolist() == ["AAA-BBB"]