Release v1.0.4

.envrc

@@ -0,0 +1 @@
+source /home/oleg/.pyenv/python3.12-venv/bin/activate

.gitignore

@@ -3,9 +3,22 @@ __pycache__/
 __OLD__/
 .specstory/
 .history/
-.cursorindexingignore
-data
 .vscode/
+*.py[cod]
+.ipynb_checkpoints/
+.pytest_cache/
+
+# Local environments
+.venv/
+venv/
+
+# Local test data and generated analysis results
+data/*
+!data/.gitkeep
+results/*
+!results/.gitkeep
+
+data
+
 cvttpy
-# SpecStory explanation file
+tmp/
-.specstory/.what-is-this.md

AGENTS.md

@@ -0,0 +1,156 @@
+# Agent Instructions
+
+## Repository purpose
+
+This repository analyzes test results with Jupyter notebooks and Python or
+Bash scripts. Inputs are commonly SQLite databases containing time-series data
+and JSON columns, but analyses may use other test-result formats.
+
+Ignore `__SAV__/`. It is unrelated legacy material, is not part of the active
+project, and must not be read, edited, moved, or used as a source of conventions
+unless the user explicitly requests it.
+
+## Active layout
+
+- `notebooks/`: exploratory and report-oriented Jupyter notebooks.
+- `scripts/`: reusable Python and Bash analysis utilities.
+- `data/`: local input data. Contents are ignored except for `.gitkeep`.
+- `results/`: generated tables, figures, exports, and reports. Contents are
+  ignored except for `.gitkeep`.
+- `requirements.txt`: Python dependencies needed to reproduce repository work.
+
+Keep reusable logic in `scripts/` and use notebooks to orchestrate analysis,
+explain decisions, and present results. Do not create a separate `analysis/`
+tree.
+
+## Python environment
+
+The intended virtual environment is `~/.pyenv/python3.12-venv`.
+
+```bash
+source ~/.pyenv/python3.12-venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+Agents may install packages in this environment when needed. Whenever a package
+is installed for repository work, update `requirements.txt` in the same change
+with a suitable direct dependency declaration. Use `python -m pip`, not bare
+`pip`, in documented commands.
+
+Do not create an in-repository virtual environment unless the user asks for
+one.
+
+## Data handling
+
+- Treat files in `data/` as local, potentially large, and potentially
+  sensitive.
+- Do not commit SQLite databases, raw test results, or generated results.
+- Do not modify source data in place. Write transformed data and exports under
+  `results/`.
+- Use parameterized SQL for values. Do not construct SQL by interpolating
+  untrusted data.
+- Parse JSON columns defensively and preserve missing, malformed, and unexpected
+  values unless the analysis explicitly defines another policy.
+- State assumptions about timestamps, time zones, ordering, units, and duplicate
+  observations in the notebook or script that relies on them.
+- Avoid loading entire databases into memory when a filtered query or chunked
+  read is practical.
+
+## Notebook conventions
+
+- A notebook must run from a fresh kernel, top to bottom, without relying on
+  hidden interactive state.
+- Set random seeds where nondeterminism affects results.
+- Keep data paths relative to the repository root and avoid machine-specific
+  absolute paths.
+- Move logic that is reused or substantial enough to test into `scripts/`.
+- Clear cell outputs before committing notebooks. Never commit embedded source
+  data, credentials, or bulky generated output.
+- Keep concise Markdown context near analyses: purpose, input assumptions,
+  method, and interpretation.
+
+## Scripts
+
+- Python scripts should expose reusable functions and use a guarded CLI entry
+  point when executable.
+- Bash scripts must start with `#!/usr/bin/env bash` and use
+  `set -euo pipefail`.
+- Prefer explicit CLI arguments over hard-coded paths or parameters.
+- Fail with actionable error messages when required data, tables, columns, or
+  configuration are missing.
+
+## Verification
+
+Verification should be proportional to the change. At minimum:
+
+- Run `pytest` for Python script changes.
+- Add or update tests for reusable parsing, transformation, query, and
+  calculation logic.
+- Execute changed notebooks from a fresh kernel with `nbmake`.
+- Run changed Bash scripts against a safe fixture or exercise their
+  non-destructive validation/help path.
+- Clear notebook outputs after execution and before committing.
+
+Useful commands:
+
+```bash
+python -m pytest
+python -m pytest --nbmake notebooks
+jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace path/to/notebook.ipynb
+```
+
+If verification cannot be run, report exactly what was skipped and why.
+
+## Release rules
+
+- Update `CHANGELOG.md` for every release with the release version, release
+  date, Git tag, and a concise summary of notable changes.
+- Keep an `Unreleased` section at the top of `CHANGELOG.md` for changes that
+  have not been included in a tagged release yet.
+- Move relevant entries from `Unreleased` into the dated release section when
+  creating a release, and leave `Unreleased` present for future changes.
+- Use release headers in `YYYY-MM-DD vMAJOR.MINOR.PATCH` form.
+- Use version numbers in `MAJOR.MINOR.PATCH` form. Start this repository at
+  `0.0.1`.
+- Use Git tags in `vMAJOR.MINOR.PATCH` form, matching the changelog version
+  exactly. For example, version `0.0.1` must be tagged as `v0.0.1`.
+- Create the Git tag only after the changelog and any release-related version
+  changes are complete.
+- When the user requests creating a release, treat that as explicit permission
+  to commit the release changes, create the matching Git tag, and push both the
+  branch and tag.
+- Do not push release commits or tags unless the user explicitly requests it.
+
+## Mandatory background review
+
+Changes to Python scripts, Bash scripts, or notebook code cells require approval
+from a separate background reviewer agent before the implementing agent may
+declare the work complete.
+
+The implementing agent must:
+
+1. Finish the implementation and run the relevant verification.
+2. Ask a separate background agent to review the diff for correctness,
+   reproducibility, data safety, and test coverage.
+3. Address every material finding, rerun affected checks, and request follow-up
+   review when the fix materially changes the code.
+4. Report the reviewer outcome in the final response.
+
+The reviewer must inspect the actual diff and relevant surrounding files; a
+self-review does not satisfy this requirement. Documentation-only,
+configuration-only, dependency-only, and ignore-rule-only changes do not
+require background approval unless they also alter Python, Bash, or notebook
+code cells.
+
+If no background reviewer is available, complete all other work but do not
+claim reviewer approval. End the handoff with the exact status:
+
+`review pending`
+
+## Change discipline
+
+- Preserve user changes and avoid unrelated cleanup.
+- Do not edit or commit generated files from `data/` or `results/`.
+- Do not push or commit unless the user explicitly requests it. The `master`
+  branch being unprotected does not imply permission to push directly.
+- Keep changes focused and explain any new assumptions or dependencies.

CHANGELOG.md

@@ -0,0 +1,95 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## Unreleased
+
+No unreleased changes yet.
+
+## 2026-07-30 v1.0.4
+
+- Updated notebook and Panel analysis for the SP Quant result database schema,
+  including explicit `trading_instructions` columns for action, assets,
+  scaled disequilibrium, and beta.
+- Changed selected-pair market charts to read from the `market` table and kept
+  legacy packed instruction JSON support for older result databases.
+- Added `scaled_disequilibrium` and `beta` to selected-pair theoretical
+  execution displays.
+- Improved VS Code notebook usability with the `jupyter_bokeh` dependency,
+  direct Plotly figure rendering, and a dropdown Analyze control for individual
+  pair selection.
+- Made the Panel app use the dark theme by default and reduced the sidebar
+  width from 430 px to 215 px with responsive sidebar controls.
+- Expanded tests and notebook verification coverage for the new database schema
+  and Panel layout defaults.
+
+## 2026-07-29 v1.0.3
+
+- Removed invalid fixed sizing mode from Panel Tabulator grids to avoid Bokeh
+  layout warnings while preserving compact table layout.
+- Changed the Panel Calculate action to refresh the result-file list before
+  loading data and removed the standalone Panel Refresh button.
+
+## 2026-07-29 v1.0.2
+
+- Added a Panel application for single-day SPBT result analysis with result-file
+  selection, minimum TARGET-change input, pair TheoRet table, pair selector,
+  selected-pair execution table, and market/trade chart.
+- Added a launcher script for the Panel application.
+- Changed notebook and Panel pair analysis to use per-row Analyze actions from
+  the Pair TheoRet grid, deferring selected-pair calculations until clicked.
+- Adjusted Panel sizing so key controls use compact widths and Pair TheoRet uses
+  content width with vertical scrolling instead of full-width paginated layout.
+- Added a FastListTemplate shell to the Panel application for sidebar controls
+  and configurable app color accents.
+- Made Plotly chart panes use all available horizontal space.
+
+## 2026-07-28 v1.0.1
+
+- Added the `spbt_day` notebook for interactive single-day backtest result
+  analysis, including SQLite result file selection from the local data
+  directory.
+- Added selector-pair loading and dense ranking by `mr_score.final`, preserving
+  rows with invalid score JSON for inspection.
+- Added theoretical return calculation for ranked pairs from
+  `trading_instructions`, including reusable helper functions and tests.
+- Added a Plotly histogram for visual analysis of total theoretical return by
+  pair.
+- Moved notebook support code into reusable `scripts/spbt_day.py` helpers.
+- Adjusted notebook table outputs to show all relevant rows and reduce
+  redundant intermediate displays.
+- Added an alphabetically sorted pair selector for individual pair analysis.
+- Added selected-pair theoretical execution tables and aligned TheoRet
+  calculations with target-delta trade generation.
+- Added per-asset `strength` values to selected-pair theoretical execution
+  tables.
+- Corrected theoretical execution size to use
+  `10000 * strength / reference_price`.
+- Removed `:USD` quote suffixes from displayed pair names in notebook tables,
+  chart hovers, and the pair selector dropdown while preserving full internal
+  pair keys for calculations.
+- Added `num_trades` to pair TheoRet summaries, counting asset-level theoretical
+  trades from effective `TARGET` and `CLOSE` instructions.
+- Added sortable interactive grids for the pair TheoRet and selected-pair
+  theoretical execution tables.
+- Styled interactive dataframe grids with black text on white backgrounds for
+  readability across notebook themes.
+- Added a selected-pair Plotly chart that overlays theoretical BUY/SELL
+  executions on relative 1-minute market close data for both instruments.
+- Anchored the selected-pair market chart at trading-day midnight and normalized
+  relative prices to each instrument's close at that timestamp.
+- Added a `min_pctg_change` threshold for ranked pair TheoRet calculations to
+  skip small target-strength changes after a position is acquired.
+- Added a notebook input field for the minimum TARGET strength-change threshold.
+
+## 2026-07-25 v0.0.9
+
+- Added contributing guidance and Python dependency declarations.
+- Added placeholder files for active project directories.
+- Updated ignore rules for local data, generated results, caches, and local
+  environments.
+- Documented unreleased changelog handling and release push behavior.
+
+## 2026-07-25 v0.0.1
+
+- Established the initial repository structure and project guidance.

CONTRIBUTING.md

@@ -0,0 +1,54 @@
+# Contributing
+
+## Setup
+
+Use the shared Python 3.12 virtual environment:
+
+```bash
+source ~/.pyenv/python3.12-venv/bin/activate
+python -m pip install -r requirements.txt
+```
+
+If you install another package for repository work, add its direct dependency
+to `requirements.txt`.
+
+## Repository layout
+
+- Put notebooks in `notebooks/`.
+- Put reusable Python and Bash utilities in `scripts/`.
+- Put local input files in `data/`.
+- Put generated artifacts in `results/`.
+
+The contents of `data/` and `results/` are ignored. Do not force-add test
+databases, raw test results, generated exports, or notebook outputs.
+
+`__SAV__/` is unrelated legacy material and is outside the active project.
+
+## Working with notebooks
+
+Notebooks must execute from top to bottom in a fresh kernel. Use relative paths,
+document data assumptions, and move reusable logic into tested scripts.
+
+Before handing off a change:
+
+```bash
+python -m pytest
+python -m pytest --nbmake notebooks
+jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace path/to/notebook.ipynb
+```
+
+Run only the checks relevant to the files present in the repository, and report
+anything that could not be run.
+
+## Review requirement
+
+Python scripts, Bash scripts, and notebook code-cell changes require review and
+approval by a separate background agent. Address material findings and rerun
+affected checks before completion. If a reviewer is unavailable, the change may
+be handed off only with the status `review pending`.
+
+Documentation, dependency declarations, and ignore rules do not require this
+background review when no Python, Bash, or notebook code cells changed.
+
+The `master` branch is not protected. That does not remove the review
+requirement or authorize an agent to commit or push without an explicit request.

TODO.md

@@ -0,0 +1,9 @@
+- [ ] Add disequilibrium chart
+- [ ] Add scatter chart for `mr-rank <-->  realized pnl`
+
+# DONE
+
+## 2026-07-29
+
+- [x] Change notebook and panel (stat_pairs_backtest) to use sp_quant's database tables `trading_instructions`  and `market`, to have *disequilibrium* and *beta*
+

notebooks/.gitkeep

@@ -0,0 +1 @@
+

panel/spbt_day_panel.py

@@ -0,0 +1,303 @@
+"""Panel application for single-day SPBT result analysis."""
+
+from __future__ import annotations
+
+from pathlib import Path
+import sys
+from typing import Any
+
+import pandas as pd
+import panel as pn
+
+
+APP_DIR = Path(__file__).resolve().parent
+REPO_ROOT = APP_DIR.parent
+if str(REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(REPO_ROOT))
+
+from scripts import spbt_day
+
+
+pn.extension("tabulator", "plotly")
+
+
+PAIR_THEO_RET_SORT_COLUMNS = ["total_pnl", "pair_name"]
+PAIR_THEO_RET_DISPLAY_DROP_COLUMNS = ["total_pnl"]
+APP_TITLE = "SPBT Day Analysis"
+APP_ACCENT_COLOR = "#226c67"
+APP_HEADER_COLOR = "#184c47"
+APP_SIDEBAR_WIDTH = 215
+APP_SIDEBAR_CONTROL_WIDTH = 200
+
+
+class SpbtDayPanelApp:
+    """Stateful Panel UI for single-day SPBT analysis."""
+
+    def __init__(self, repo_root: Path | None = None) -> None:
+        self.repo_root = (repo_root or spbt_day.find_repo_root(REPO_ROOT)).resolve()
+        self.selector_pair_rankings = pd.DataFrame()
+        self.trading_instructions = pd.DataFrame()
+        self.pair_theo_ret = pd.DataFrame()
+        self.selected_pair_theo_executions = pd.DataFrame()
+        self.selected_pair_name: str | None = None
+        self.min_pctg_change = 0.0
+
+        self.directory_input = pn.widgets.TextInput(
+            label="Directory",
+            value=str(self.repo_root / "data"),
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.show_all_files = pn.widgets.Checkbox(label="Show all files", value=False)
+        self.file_select = pn.widgets.Select(
+            label="SQLite result file",
+            options={},
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.min_pctg_change_input = pn.widgets.FloatInput(
+            label="Mininal TARGET change (%)",
+            value=0.0,
+            step=1.0,
+            sizing_mode="stretch_width",
+            width=None,
+        )
+        self.calculate_button = pn.widgets.Button(
+            label="Calculate",
+            color="primary",
+            width=110,
+        )
+
+        self.status = pn.pane.Markdown("")
+        self.pair_theo_ret_table = spbt_day.create_pair_theo_ret_analyze_grid(
+            pd.DataFrame(),
+            height=420,
+        )
+        self.total_pnl_histogram = pn.pane.Plotly(
+            None,
+            height=360,
+            sizing_mode="stretch_width",
+        )
+        self.selected_pair_message = pn.pane.Markdown(
+            "Click Analyze in the Pair TheoRet grid to load individual-pair details."
+        )
+        self.selected_pair_executions_table = spbt_day.create_selected_pair_executions_grid(
+            height=320,
+        )
+        self.selected_pair_market_plot = pn.pane.Plotly(
+            None,
+            height=520,
+            sizing_mode="stretch_width",
+        )
+
+        self.calculate_button.on_click(self.calculate)
+        self.directory_input.param.watch(self.refresh_files, "value")
+        self.show_all_files.param.watch(self.refresh_files, "value")
+        self.pair_theo_ret_table.on_click(
+            self.analyze_pair_click,
+            column=spbt_day.ANALYZE_BUTTON_COLUMN,
+        )
+        self.refresh_files()
+
+    def set_status(self, message: str, *, error: bool = False) -> None:
+        """Update visible status text."""
+        prefix = "**Error:** " if error else ""
+        self.status.object = f"{prefix}{message}" if message else ""
+
+    def selected_database_path(self) -> Path:
+        """Return the selected result database path."""
+        if not self.file_select.value:
+            raise ValueError("Select a SQLite result file before calculating.")
+        db_path = Path(str(self.file_select.value)).resolve()
+        if not db_path.exists():
+            raise FileNotFoundError(f"Selected database does not exist: {db_path}")
+        if not db_path.is_file():
+            raise ValueError(f"Selected database path is not a file: {db_path}")
+        return db_path
+
+    def refresh_files(self, *_events: Any) -> bool:
+        """Refresh selectable SQLite files from the configured directory."""
+        try:
+            directory = spbt_day.normalize_directory(
+                self.directory_input.value,
+                self.repo_root,
+            )
+            candidates = spbt_day.list_candidate_files(
+                directory,
+                show_all=self.show_all_files.value,
+            )
+        except Exception as exc:
+            self.file_select.options = {}
+            self.file_select.value = None
+            self.set_status(str(exc), error=True)
+            return False
+
+        options = {path.name: str(path) for path in candidates}
+        previous_value = self.file_select.value
+        self.file_select.options = options
+        if previous_value in options.values():
+            self.file_select.value = previous_value
+        elif options:
+            self.file_select.value = next(iter(options.values()))
+        else:
+            self.file_select.value = None
+
+        if options:
+            self.set_status(f"Found {len(options):,} file(s) in {directory}.")
+        else:
+            self.set_status(f"No selectable files found in {directory}.")
+        return True
+
+    def calculate(self, *_events: Any) -> None:
+        """Load selected data and calculate all-pair TheoRet."""
+        self.calculate_button.loading = True
+        try:
+            if not self.refresh_files():
+                return
+            db_path = self.selected_database_path()
+            self.min_pctg_change = float(self.min_pctg_change_input.value)
+
+            conn = spbt_day.connect_sqlite_read_only(db_path)
+            try:
+                self.selector_pair_rankings = spbt_day.load_selector_pair_rankings(conn)
+                self.trading_instructions = spbt_day.load_trading_instructions(conn)
+            finally:
+                conn.close()
+
+            self.pair_theo_ret = (
+                spbt_day.add_total_pnl(
+                    spbt_day.calculate_ranked_pairs_theo_ret(
+                        self.selector_pair_rankings,
+                        self.trading_instructions,
+                        min_pctg_change=self.min_pctg_change,
+                    )
+                )
+                .sort_values(
+                    PAIR_THEO_RET_SORT_COLUMNS,
+                    ascending=[True, True],
+                    kind="mergesort",
+                )
+                .drop(columns=PAIR_THEO_RET_DISPLAY_DROP_COLUMNS)
+                .reset_index(drop=True)
+            )
+            self.pair_theo_ret_table.value = spbt_day.format_pair_theo_ret_for_analyze_grid(
+                self.pair_theo_ret
+            )
+            self.total_pnl_histogram.object = spbt_day.create_total_pnl_histogram(
+                self.pair_theo_ret
+            )
+            self.clear_selected_pair_analysis()
+
+            self.set_status(
+                f"Calculated {len(self.pair_theo_ret):,} pair row(s) from {db_path.name}."
+            )
+        except Exception as exc:
+            self.set_status(str(exc), error=True)
+        finally:
+            self.calculate_button.loading = False
+
+    def clear_selected_pair_analysis(self) -> None:
+        """Clear individual-pair outputs until a row Analyze button is clicked."""
+        self.selected_pair_name = None
+        self.selected_pair_theo_executions = pd.DataFrame()
+        self.selected_pair_message.object = (
+            "Click Analyze in the Pair TheoRet grid to load individual-pair details."
+        )
+        self.selected_pair_executions_table.value = pd.DataFrame(
+            columns=spbt_day.SELECTED_PAIR_EXECUTION_DISPLAY_COLUMNS
+        )
+        self.selected_pair_market_plot.object = None
+
+    def analyze_pair_click(self, event: Any) -> None:
+        """Run selected-pair analysis from a Pair TheoRet Analyze button click."""
+        self.update_selected_pair(
+            spbt_day.pair_name_from_analyze_event(self.pair_theo_ret_table, event)
+        )
+
+    def analyze_pair_row(self, row: int) -> None:
+        """Run selected-pair analysis for a Pair TheoRet table row."""
+        event = type("AnalyzeEvent", (), {"row": row})()
+        self.analyze_pair_click(event)
+
+    def update_selected_pair(self, pair_name: str) -> None:
+        """Calculate selected-pair executions and market plot."""
+        if self.trading_instructions.empty:
+            self.clear_selected_pair_analysis()
+            return
+
+        self.selected_pair_name = pair_name
+        self.selected_pair_message.object = (
+            f"Selected pair: **{spbt_day.format_pair_name_for_display(pair_name)}**"
+        )
+        self.selected_pair_theo_executions = spbt_day.calculate_pair_theo_executions(
+            pair_name,
+            self.trading_instructions,
+            min_pctg_change=self.min_pctg_change,
+        )
+        self.selected_pair_executions_table.value = (
+            self.selected_pair_theo_executions.reindex(
+                columns=spbt_day.SELECTED_PAIR_EXECUTION_DISPLAY_COLUMNS
+            )
+        )
+
+        try:
+            trading_day_start_ns = spbt_day.infer_trading_day_start_ns(
+                self.trading_instructions
+            )
+            conn = spbt_day.connect_sqlite_read_only(self.selected_database_path())
+            try:
+                selected_pair_market_data = spbt_day.load_pair_market_data(
+                    conn,
+                    pair_name,
+                    trading_day_start_ns=trading_day_start_ns,
+                )
+            finally:
+                conn.close()
+            self.selected_pair_market_plot.object = spbt_day.create_pair_trades_market_plot(
+                pair_name,
+                selected_pair_market_data,
+                self.selected_pair_theo_executions,
+            )
+        except Exception as exc:
+            self.selected_pair_market_plot.object = None
+            self.set_status(str(exc), error=True)
+
+    @property
+    def view(self) -> pn.template.FastListTemplate:
+        """Return the app layout."""
+        controls = pn.Column(
+            "## Inputs",
+            self.directory_input,
+            self.show_all_files,
+            self.file_select,
+            self.min_pctg_change_input,
+            self.calculate_button,
+            self.status,
+            width=APP_SIDEBAR_CONTROL_WIDTH,
+        )
+        main = pn.Column(
+            "## Pair TheoRet",
+            self.pair_theo_ret_table,
+            self.total_pnl_histogram,
+            "## Individual Pair",
+            self.selected_pair_message,
+            "### Theoretical Executions",
+            self.selected_pair_executions_table,
+            "### Trades on Market Data",
+            self.selected_pair_market_plot,
+        )
+        return pn.template.FastListTemplate(
+            title=APP_TITLE,
+            sidebar=[controls],
+            main=[main],
+            sidebar_width=APP_SIDEBAR_WIDTH,
+            accent_base_color=APP_ACCENT_COLOR,
+            header_background=APP_HEADER_COLOR,
+            main_layout=None,
+            theme=pn.template.DarkTheme,
+        )
+
+
+app_controller = SpbtDayPanelApp()
+app = app_controller.view
+app.servable(title=APP_TITLE)

requirements.txt

@@ -0,0 +1,14 @@
+# Interactive analysis
+ipykernel>=6.29,<7
+ipywidgets>=8.1,<9
+itables>=2.2,<3
+jupyter>=1.1,<2
+jupyter_bokeh>=4.0,<5
+nbformat>=5.10,<6
+pandas>=2.2,<3
+panel>=1.5,<2
+plotly>=5.24,<7
+
+# Verification
+nbmake>=1.5,<2
+pytest>=8,<9

results/.gitkeep

@@ -0,0 +1 @@
+

scripts/.gitkeep

@@ -0,0 +1 @@
+

scripts/run_spbt_day_panel.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$repo_root"
+
+panel serve panel/spbt_day_panel.py --show "$@"

src/backtest_configs.py

@@ -1,95 +0,0 @@
-from typing import Any, Dict, List, Optional
-
-
-# ------------------------ Configuration ------------------------
-# Default configuration
-CRYPTO_CONFIG: Dict = {
-    "security_type": "CRYPTO",
-    # --- Data retrieval
-    "data_directory": "./data/crypto",
-    "datafiles": [
-        # "20250519.mktdata.ohlcv.db",
-        # "20250520.mktdata.ohlcv.db",
-        # "20250521.mktdata.ohlcv.db",
-        # "20250522.mktdata.ohlcv.db",
-        # "20250523.mktdata.ohlcv.db",
-        # "20250524.mktdata.ohlcv.db",
-        "20250525.mktdata.ohlcv.db",
-    ],
-    "db_table_name": "bnbspot_ohlcv_1min",
-    # ----- Instruments
-    "exchange_id": "BNBSPOT",
-    "instrument_id_pfx": "PAIR-",
-    "instruments": [
-        "BTC-USDT",
-        "BCH-USDT",
-        "ETH-USDT",
-        "LTC-USDT",
-        "XRP-USDT",
-        "ADA-USDT",
-        "SOL-USDT",
-        "DOT-USDT",
-    ],
-    "trading_hours": {
-        "begin_session": "00:00:00",
-        "end_session": "23:59:00",
-        "timezone": "UTC",
-    },
-    # ----- Model Settings
-    "price_column": "close",
-    "min_required_points": 30,
-    "zero_threshold": 1e-10,
-
-    "dis-equilibrium_open_trshld": 2.0,
-    "dis-equilibrium_close_trshld": 0.5,
-
-    # "training_minutes": 120,
-    "training_minutes": 60,
-    # ----- Validation
-    "funding_per_pair": 2000.0,  # USD
-}
-
-# ========================== EQUITIES
-EQT_CONFIG: Dict = {
-    # --- Data retrieval
-    "security_type": "EQUITY",
-    "data_directory": "./data/equity",
-    "datafiles": [
-        # "20250508.alpaca_sim_md.db",
-        # "20250509.alpaca_sim_md.db",
-        "20250512.alpaca_sim_md.db",
-        # "20250513.alpaca_sim_md.db",
-        # "20250514.alpaca_sim_md.db",
-        # "20250515.alpaca_sim_md.db",
-        # "20250516.alpaca_sim_md.db",
-        # "20250519.alpaca_sim_md.db",
-        # "20250520.alpaca_sim_md.db"
-    ],
-    "db_table_name": "md_1min_bars",
-    # ----- Instruments
-    "exchange_id": "ALPACA",
-    "instrument_id_pfx": "STOCK-",
-    "instruments": [
-        "COIN",
-        "GBTC",
-        "HOOD",
-        "MSTR",
-        "PYPL",
-    ],
-    "trading_hours": {
-        "begin_session": "9:30:00",
-        "end_session": "16:00:00",
-        "timezone": "America/New_York",
-    },
-    # ----- Model Settings
-    "price_column": "close",
-    "min_required_points": 30,
-    "zero_threshold": 1e-10,
-    "dis-equilibrium_open_trshld": 2.0,
-    "dis-equilibrium_close_trshld": 0.5,
-    "training_minutes": 120,
-    # ----- Validation
-    "funding_per_pair": 2000.0,
-}
-
-

src/pt_backtest.py

@@ -1,300 +0,0 @@
-from abc import ABC, abstractmethod
-import sys
-
-from typing import Any, Dict, List, Optional
-
-import pandas as pd
-import numpy as np
-
-# ============= statsmodels ===================
-from statsmodels.tsa.vector_ar.vecm import VECM
-
-from backtest_configs import CRYPTO_CONFIG
-from tools.data_loader import load_market_data
-from tools.trading_pair import TradingPair
-from results import BacktestResult
-
-NanoPerMin = 1e9
-UNSET_FLOAT: float = sys.float_info.max
-UNSET_INT: int = sys.maxsize
-
-
-CONFIG = CRYPTO_CONFIG
-# CONFIG = EQT_CONFIG
-
-
-trades_columns = [
-    "time",
-    "action",
-    "symbol",
-    "price",
-    "disequilibrium",
-    "scaled_disequilibrium",
-    "pair",
-]
-
-BacktestResults = BacktestResult(config=CONFIG)
-
-class PairTradingStrategy(ABC):
-    @abstractmethod
-    def create_trading_signals(pair: TradingPair, config: Dict) -> pd.DataFrame:
-        ... 
-    @abstractmethod
-    def run_pair(pair: TradingPair) -> Optional[pd.DataFrame]:
-        ... 
-
-
-def run_pair(pair: TradingPair) -> Optional[pd.DataFrame]:
-    pair.get_datasets(training_minutes=CONFIG["training_minutes"])
-    try:
-        is_cointegrated = pair.train_pair()
-        if not is_cointegrated:
-            print(f"{pair} IS NOT COINTEGRATED")
-            return None
-    except Exception as e:
-        print(f"{pair}: Training failed: {str(e)}")
-        return None
-
-    try:
-        pair.predict()
-    except Exception as e:
-        print(f"{pair}: Prediction failed: {str(e)}")
-        return None
-        
-    pair_trades = create_trading_signals(pair=pair, config=CONFIG)
-
-    return pair_trades
-
-
-def create_trading_signals(pair: TradingPair, config: Dict) -> pd.DataFrame:
-    beta = pair.vecm_fit_.beta
-    colname_a, colname_b = pair.colnames()
-
-    predicted_df = pair.predicted_df_
-
-    open_threshold = config["dis-equilibrium_open_trshld"]
-    close_threshold = config["dis-equilibrium_close_trshld"]
-
-    # Iterate through the testing dataset to find the first trading opportunity
-    open_row_index = None
-    for row_idx in range(len(predicted_df)):
-        curr_disequilibrium = predicted_df["scaled_disequilibrium"][row_idx]
-
-        # Check if current row has sufficient disequilibrium (not near-zero)
-        if curr_disequilibrium >= open_threshold:
-            open_row_index = row_idx
-            break
-
-    # If no row with sufficient disequilibrium found, skip this pair
-    if open_row_index is None:
-        print(f"{pair}: Insufficient disequilibrium in testing dataset. Skipping.")
-        return pd.DataFrame()
-
-    # Look for close signal starting from the open position
-    trading_signals_df = (
-        predicted_df["scaled_disequilibrium"][open_row_index:] < close_threshold
-    )
-
-    # Adjust indices to account for the offset from open_row_index
-    close_row_index = None
-    for idx, value in trading_signals_df.items():
-        if value:
-            close_row_index = idx
-            break
-
-    open_row = predicted_df.loc[open_row_index]
-    open_tstamp = open_row["tstamp"]
-    open_disequilibrium = open_row["disequilibrium"]
-    open_scaled_disequilibrium = open_row["scaled_disequilibrium"]
-    open_px_a = open_row[f"{colname_a}"]
-    open_px_b = open_row[f"{colname_b}"]
-
-    abs_beta = abs(beta[1])
-    pred_px_b = predicted_df.loc[open_row_index][f"{colname_b}_pred"]
-    pred_px_a = predicted_df.loc[open_row_index][f"{colname_a}_pred"]
-
-    if pred_px_b * abs_beta - pred_px_a > 0:
-        open_side_a = "BUY"
-        open_side_b = "SELL"
-        close_side_a = "SELL"
-        close_side_b = "BUY"
-    else:
-        open_side_b = "BUY"
-        open_side_a = "SELL"
-        close_side_b = "SELL"
-        close_side_a = "BUY"
-
-    # If no close signal found, print position and unrealized PnL
-    if close_row_index is None:
-
-        last_row_index = len(predicted_df) - 1
-
-        # Use the new method from BacktestResult to handle outstanding positions
-        BacktestResults.handle_outstanding_position(
-            pair=pair,
-            pair_result_df=predicted_df,
-            last_row_index=last_row_index,
-            open_side_a=open_side_a,
-            open_side_b=open_side_b,
-            open_px_a=open_px_a,
-            open_px_b=open_px_b,
-            open_tstamp=open_tstamp,
-        )
-
-        # Return only open trades (no close trades)
-        trd_signal_tuples = [
-            (
-                open_tstamp,
-                open_side_a,
-                pair.symbol_a_,
-                open_px_a,
-                open_disequilibrium,
-                open_scaled_disequilibrium,
-                pair,
-            ),
-            (
-                open_tstamp,
-                open_side_b,
-                pair.symbol_b_,
-                open_px_b,
-                open_disequilibrium,
-                open_scaled_disequilibrium,
-                pair,
-            ),
-        ]
-    else:
-        # Close signal found - create complete trade
-        close_row = predicted_df.loc[close_row_index]
-        close_tstamp = close_row["tstamp"]
-        close_disequilibrium = close_row["disequilibrium"]
-        close_scaled_disequilibrium = close_row["scaled_disequilibrium"]
-        close_px_a = close_row[f"{colname_a}"]
-        close_px_b = close_row[f"{colname_b}"]
-
-        print(f"{pair}: Close signal found at index {close_row_index}")
-
-        trd_signal_tuples = [
-            (
-                open_tstamp,
-                open_side_a,
-                pair.symbol_a_,
-                open_px_a,
-                open_disequilibrium,
-                open_scaled_disequilibrium,
-                pair,
-            ),
-            (
-                open_tstamp,
-                open_side_b,
-                pair.symbol_b_,
-                open_px_b,
-                open_disequilibrium,
-                open_scaled_disequilibrium,
-                pair,
-            ),
-            (
-                close_tstamp,
-                close_side_a,
-                pair.symbol_a_,
-                close_px_a,
-                close_disequilibrium,
-                close_scaled_disequilibrium,
-                pair,
-            ),
-            (
-                close_tstamp,
-                close_side_b,
-                pair.symbol_b_,
-                close_px_b,
-                close_disequilibrium,
-                close_scaled_disequilibrium,
-                pair,
-            ),
-        ]
-
-    # Add tuples to data frame
-    return pd.DataFrame(
-        trd_signal_tuples,
-        columns=trades_columns,
-    )
-
-
-def run_all_pairs(config: Dict, datafile: str, price_column: str) -> None:
-
-    def _create_pairs(config: Dict) -> List[TradingPair]:
-        nonlocal datafile
-        instruments = config["instruments"]
-        all_indexes = range(len(instruments))
-        unique_index_pairs = [(i, j) for i in all_indexes for j in all_indexes if i < j]
-        pairs = []
-        market_data_df = load_market_data(
-            f'{config["data_directory"]}/{datafile}', config=CONFIG
-        )
-        for a_index, b_index in unique_index_pairs:
-            pair = TradingPair(
-                market_data=market_data_df,
-                symbol_a=instruments[a_index],
-                symbol_b=instruments[b_index],
-                price_column=price_column,
-            )
-            pairs.append(pair)
-        return pairs
- 
-
-    pairs_trades = []
-    for pair in _create_pairs(config):
-        single_pair_trades = run_pair(pair=pair)
-        if single_pair_trades is not None and len(single_pair_trades) > 0:
-            pairs_trades.append(single_pair_trades)
-    # Check if result_list has any data before concatenating
-    if len(pairs_trades) == 0:
-        print("No trading signals found for any pairs")
-        return None
-
-    result = pd.concat(pairs_trades, ignore_index=True)
-    result["time"] = pd.to_datetime(result["time"])
-    result = result.set_index("time").sort_index()
-
-    BacktestResults.collect_single_day_results(result)
-    # BacktestResults.print_single_day_results()
-
-
-def main() -> None:
-    # Initialize a dictionary to store all trade results
-    all_results: Dict[str, Dict[str, Any]] = {}
-
-    # Initialize global PnL tracking variables
-
-    # Process each data file
-    price_column = CONFIG["price_column"]
-    for datafile in CONFIG["datafiles"]:
-        print(f"\n====== Processing {datafile} ======")
-
-        # Clear the TRADES global dictionary and reset unrealized PnL for the new file
-        BacktestResults.clear_trades()
-
-        # Process data for this file
-        try:
-            run_all_pairs(
-                config=CONFIG, datafile=datafile, price_column=price_column
-            )
-
-            # Store results with file name as key
-            filename = datafile.split("/")[-1]
-            all_results[filename] = {"trades": BacktestResults.trades.copy()}
-
-            print(f"Successfully processed {filename}")
-
-            # No longer printing unrealized PnL since we removed that functionality
-
-        except Exception as e:
-            print(f"Error processing {datafile}: {str(e)}")
-
-    # BacktestResults.print_results_summary(all_results)
-    BacktestResults.calculate_returns(all_results)
-    # Print grand totals
-    BacktestResults.print_grand_totals()
-    BacktestResults.print_outstanding_positions()
-    
-if __name__ == "__main__":
-    main()

src/results.py

@@ -1,295 +0,0 @@
-from typing import Any, Dict, List
-import pandas as pd
-
-
-class BacktestResult:
-    """
-    Class to handle backtest results, trades tracking, PnL calculations, and reporting.
-    """
-
-    def __init__(self, config: Dict[str, Any]):
-        self.config = config
-        self.trades: Dict[str, Dict[str, Any]] = {}
-        self.total_realized_pnl = 0.0
-        self.outstanding_positions: List[Dict[str, Any]] = []
-
-    def add_trade(self, pair_nm, symbol, action, price):
-        """Add a trade to the results tracking."""
-        pair_nm = str(pair_nm)
-
-        if pair_nm not in self.trades:
-            self.trades[pair_nm] = {symbol: []}
-        if symbol not in self.trades[pair_nm]:
-            self.trades[pair_nm][symbol] = []
-        self.trades[pair_nm][symbol].append((action, price))
-
-    def add_outstanding_position(self, position: Dict[str, Any]):
-        """Add an outstanding position to tracking."""
-        self.outstanding_positions.append(position)
-
-    def add_realized_pnl(self, realized_pnl: float):
-        """Add realized PnL to the total."""
-        self.total_realized_pnl += realized_pnl
-
-    def get_total_realized_pnl(self) -> float:
-        """Get total realized PnL."""
-        return self.total_realized_pnl
-
-    def get_outstanding_positions(self) -> List[Dict[str, Any]]:
-        """Get all outstanding positions."""
-        return self.outstanding_positions
-
-    def get_trades(self) -> Dict[str, Dict[str, Any]]:
-        """Get all trades."""
-        return self.trades
-
-    def clear_trades(self):
-        """Clear all trades (used when processing new files)."""
-        self.trades.clear()
-
-    def collect_single_day_results(self, result):
-        """Collect and process single day trading results."""
-        if result is None:
-            return
-
-        print("\n --------------  Suggested Trades ")
-        print(result)
-
-        for row in result.itertuples():
-            action = row.action
-            symbol = row.symbol
-            price = row.price
-            self.add_trade(
-                pair_nm=row.pair, action=action, symbol=symbol, price=price
-            )
-
-    def print_single_day_results(self):
-        """Print single day results summary."""
-        for pair, symbols in self.trades.items():
-            print(f"\n--- {pair} ---")
-            for symbol, trades in symbols.items():
-                for side, price in trades:
-                    print(f"{symbol} {side} at ${price}")
-
-    def print_results_summary(self, all_results):
-        """Print summary of all processed files."""
-        print("\n====== Summary of All Processed Files ======")
-        for filename, data in all_results.items():
-            trade_count = sum(
-                len(trades)
-                for symbol_trades in data["trades"].values()
-                for trades in symbol_trades.values()
-            )
-            print(f"{filename}: {trade_count} trades")
-
-    def calculate_returns(self, all_results: Dict):
-        """Calculate and print returns by day and pair."""
-        print("\n====== Returns By Day and Pair ======")
-
-        for filename, data in all_results.items():
-            day_return = 0
-            print(f"\n--- {filename} ---")
-
-            # Process each pair
-            for pair, symbols in data["trades"].items():
-                pair_return = 0
-                pair_trades = []
-
-                # Calculate individual symbol returns in the pair
-                for symbol, trades in symbols.items():
-                    if len(trades) >= 2:  # Need at least entry and exit
-                        # Get entry and exit trades
-                        entry_action, entry_price = trades[0]
-                        exit_action, exit_price = trades[1]
-
-                        # Calculate return based on action
-                        symbol_return = 0
-                        if entry_action == "BUY" and exit_action == "SELL":
-                            # Long position
-                            symbol_return = (exit_price - entry_price) / entry_price * 100
-                        elif entry_action == "SELL" and exit_action == "BUY":
-                            # Short position
-                            symbol_return = (entry_price - exit_price) / entry_price * 100
-
-                        pair_trades.append(
-                            (
-                                symbol,
-                                entry_action,
-                                entry_price,
-                                exit_action,
-                                exit_price,
-                                symbol_return,
-                            )
-                        )
-                        pair_return += symbol_return
-
-                # Print pair returns
-                if pair_trades:
-                    print(f"  {pair}:")
-                    for (
-                        symbol,
-                        entry_action,
-                        entry_price,
-                        exit_action,
-                        exit_price,
-                        symbol_return,
-                    ) in pair_trades:
-                        print(
-                            f"    {symbol}: {entry_action} @ ${entry_price:.2f}, {exit_action} @ ${exit_price:.2f}, Return: {symbol_return:.2f}%"
-                        )
-                    print(f"    Pair Total Return: {pair_return:.2f}%")
-                    day_return += pair_return
-
-            # Print day total return and add to global realized PnL
-            if day_return != 0:
-                print(f"  Day Total Return: {day_return:.2f}%")
-                self.add_realized_pnl(day_return)
-
-    def print_outstanding_positions(self):
-        """Print all outstanding positions with share quantities and current values."""
-        if not self.get_outstanding_positions():
-            print("\n====== NO OUTSTANDING POSITIONS ======")
-            return
-
-        print(f"\n====== OUTSTANDING POSITIONS ======")
-        print(
-            f"{'Pair':<15}"
-            f" {'Symbol':<10}"
-            f" {'Side':<4}"
-            f" {'Shares':<10}"
-            f" {'Open $':<8}"
-            f" {'Current $':<10}"
-            f" {'Value $':<12}"
-            f" {'Disequilibrium':<15}"
-        )
-        print("-" * 100)
-
-        total_value = 0.0
-
-        for pos in self.get_outstanding_positions():
-            # Print position A
-            print(
-                f"{pos['pair']:<15}"
-                f" {pos['symbol_a']:<10}"
-                f" {pos['side_a']:<4}"
-                f" {pos['shares_a']:<10.2f}"
-                f" {pos['open_px_a']:<8.2f}"
-                f" {pos['current_px_a']:<10.2f}"
-                f" {pos['current_value_a']:<12.2f}"
-                f" {'':<15}"
-            )
-
-            # Print position B
-            print(
-                f"{'':<15}"
-                f" {pos['symbol_b']:<10}"
-                f" {pos['side_b']:<4}"
-                f" {pos['shares_b']:<10.2f}"
-                f" {pos['open_px_b']:<8.2f}"
-                f" {pos['current_px_b']:<10.2f}"
-                f" {pos['current_value_b']:<12.2f}"
-            )
-
-            # Print pair totals with disequilibrium info
-            print(
-                f"{'':<15}"
-                f" {'PAIR TOTAL':<10}"
-                f" {'':<4}"
-                f" {'':<10}"
-                f" {'':<8}"
-                f" {'':<10}"
-                f" {pos['total_current_value']:<12.2f}"
-            )
-
-            # Print disequilibrium details
-            print(
-                f"{'':<15}"
-                f" {'DISEQUIL':<10}"
-                f" {'':<4}"
-                f" {'':<10}"
-                f" {'':<8}"
-                f" {'':<10}"
-                f" Raw: {pos['current_disequilibrium']:<6.4f}"
-                f" Scaled: {pos['current_scaled_disequilibrium']:<6.4f}"
-            )
-
-            print("-" * 100)
-
-            total_value += pos["total_current_value"]
-
-        print(f"{'TOTAL OUTSTANDING VALUE':<80} ${total_value:<12.2f}")
-
-    def print_grand_totals(self):
-        """Print grand totals across all pairs."""
-        print(f"\n====== GRAND TOTALS ACROSS ALL PAIRS ======")
-        print(f"Total Realized PnL: {self.get_total_realized_pnl():.2f}%")
-
-    def handle_outstanding_position(self, pair, pair_result_df, last_row_index,
-                                  open_side_a, open_side_b, open_px_a, open_px_b,
-                                  open_tstamp):
-        """
-        Handle calculation and tracking of outstanding positions when no close signal is found.
-
-        Args:
-            pair: TradingPair object
-            pair_result_df: DataFrame with pair results
-            last_row_index: Index of the last row in the data
-            open_side_a, open_side_b: Trading sides for symbols A and B
-            open_px_a, open_px_b: Opening prices for symbols A and B
-            open_tstamp: Opening timestamp
-        """
-        last_row = pair_result_df.loc[last_row_index]
-        last_tstamp = last_row["tstamp"]
-        colname_a, colname_b = pair.colnames()
-        last_px_a = last_row[colname_a]
-        last_px_b = last_row[colname_b]
-
-        # Calculate share quantities based on funding per pair
-        # Split funding equally between the two positions
-        funding_per_position = self.config["funding_per_pair"] / 2
-        shares_a = funding_per_position / open_px_a
-        shares_b = funding_per_position / open_px_b
-
-        # Calculate current position values (shares * current price)
-        current_value_a = shares_a * last_px_a
-        current_value_b = shares_b * last_px_b
-        total_current_value = current_value_a + current_value_b
-
-        # Get disequilibrium information
-        current_disequilibrium = last_row["disequilibrium"]
-        current_scaled_disequilibrium = last_row["scaled_disequilibrium"]
-
-        # Store outstanding positions
-        self.add_outstanding_position(
-            {
-                "pair": str(pair),
-                "symbol_a": pair.symbol_a_,
-                "symbol_b": pair.symbol_b_,
-                "side_a": open_side_a,
-                "side_b": open_side_b,
-                "shares_a": shares_a,
-                "shares_b": shares_b,
-                "open_px_a": open_px_a,
-                "open_px_b": open_px_b,
-                "current_px_a": last_px_a,
-                "current_px_b": last_px_b,
-                "current_value_a": current_value_a,
-                "current_value_b": current_value_b,
-                "total_current_value": total_current_value,
-                "open_time": open_tstamp,
-                "last_time": last_tstamp,
-                "current_abs_term": current_scaled_disequilibrium,
-                "current_disequilibrium": current_disequilibrium,
-                "current_scaled_disequilibrium": current_scaled_disequilibrium,
-            }
-        )
-
-        # Print position details
-        print(f"{pair}: NO CLOSE SIGNAL FOUND - Position held until end of session")
-        print(f"  Open: {open_tstamp} | Last: {last_tstamp}")
-        print(f"  {pair.symbol_a_}: {open_side_a} {shares_a:.2f} shares @ ${open_px_a:.2f} -> ${last_px_a:.2f} | Value: ${current_value_a:.2f}")
-        print(f"  {pair.symbol_b_}: {open_side_b} {shares_b:.2f} shares @ ${open_px_b:.2f} -> ${last_px_b:.2f} | Value: ${current_value_b:.2f}")
-        print(f"  Total Value: ${total_current_value:.2f}")
-        print(f"  Disequilibrium: {current_disequilibrium:.4f} | Scaled: {current_scaled_disequilibrium:.4f}")
-
-        return current_value_a, current_value_b, total_current_value

src/tools/data_loader.py

@@ -1,99 +0,0 @@
-import sys
-import sqlite3
-from typing import Dict, Tuple
-import pandas as pd
-
-from tools.trading_pair import TradingPair
-
-
-def load_sqlite_to_dataframe(db_path, query):
-    try:
-        conn = sqlite3.connect(db_path)
-
-        df = pd.read_sql_query(query, conn)
-        return df
-    except sqlite3.Error as excpt:
-        print(f"SQLite error: {excpt}")
-        raise
-    except Exception as e:
-        print(f"Error: {excpt}")
-        raise
-    finally:
-        if "conn" in locals():
-            conn.close()
-
-
-def convert_time_to_UTC(value: str, timezone: str):
-
-    from zoneinfo import ZoneInfo
-    from datetime import datetime
-
-    # Parse it to naive datetime object
-    local_dt = datetime.strptime(value, "%Y-%m-%d %H:%M:%S")
-
-    zinfo = ZoneInfo(timezone)
-    result = local_dt.replace(tzinfo=zinfo)
-
-    result = result.astimezone(ZoneInfo("UTC"))
-    result = result.strftime("%Y-%m-%d %H:%M:%S")
-
-    return result
-
-
-def load_market_data(datafile: str, config: Dict) -> pd.DataFrame:
-    from tools.data_loader import load_sqlite_to_dataframe
-
-    instrument_ids = [
-        '"' + config["instrument_id_pfx"] + instrument + '"'
-        for instrument in config["instruments"]
-    ]
-    security_type = config["security_type"]
-    exchange_id = config["exchange_id"]
-
-    query = "select"
-    if security_type == "CRYPTO":
-        query += " strftime('%Y-%m-%d %H:%M:%S', tstamp/1000000000, 'unixepoch') as tstamp"
-        query += ", tstamp as time_ns"
-    else:
-        query += " tstamp"
-        query += ", tstamp_ns as time_ns"
-
-    query += f", substr(instrument_id, {len(config['instrument_id_pfx']) + 1}) as symbol"
-    query += ", open"
-    query += ", high"
-    query += ", low"
-    query += ", close"
-    query += ", volume"
-    query += ", num_trades"
-    query += ", vwap"
-
-    query += f" from {config['db_table_name']}"
-    query += f" where exchange_id ='{exchange_id}'"
-    query += f" and instrument_id in ({','.join(instrument_ids)})"
-
-    df = load_sqlite_to_dataframe(db_path=datafile, query=query)
-
-    # Trading Hours
-    date_str = df["tstamp"][0][0:10]
-    trading_hours = config["trading_hours"]
-
-    start_time = convert_time_to_UTC(
-        f"{date_str} {trading_hours['begin_session']}", trading_hours["timezone"]
-    )
-    end_time = convert_time_to_UTC(
-        f"{date_str} {trading_hours['end_session']}", trading_hours["timezone"]
-    )
-
-    # Perform boolean selection
-    df = df[(df["tstamp"] >= start_time) & (df["tstamp"] <= end_time)]
-    df["tstamp"] = pd.to_datetime(df["tstamp"])
-
-    return df
-
-
-
-
-# if __name__ == "__main__":
-#     df1 = load_sqlite_to_dataframe(sys.argv[1], table_name="md_1min_bars")
-
-#     print(df1)

src/tools/trading_pair.py

@@ -1,145 +0,0 @@
-
-from typing import List, Optional
-import pandas as pd
-from statsmodels.tsa.vector_ar.vecm import VECM
-
-class TradingPair:
-    market_data_: pd.DataFrame
-    symbol_a_: str
-    symbol_b_: str
-    price_column_: str
-
-    training_mu_: Optional[float]
-    training_std_: Optional[float]
-
-    training_df_: Optional[pd.DataFrame]
-    testing_df_: Optional[pd.DataFrame]
-
-    vecm_fit_: Optional[VECM]
-
-    def __init__(self, market_data: pd.DataFrame, symbol_a: str, symbol_b: str, price_column: str):
-        self.symbol_a_ = symbol_a
-        self.symbol_b_ = symbol_b
-        self.price_column_ = price_column
-        self.market_data_ = self._transform_dataframe(market_data)[["tstamp"] + self.colnames()]
-        
-        
-        self.training_mu_ = None
-        self.training_std_ = None
-        self.training_df_ = None
-        self.testing_df_ = None
-        self.vecm_fit_ = None
-
-    def _transform_dataframe(self, df: pd.DataFrame):
-        # Select only the columns we need
-        df_selected = df[["tstamp", "symbol", self.price_column_]]
-
-        # Start with unique timestamps
-        result_df: pd.DataFrame = pd.DataFrame(df_selected["tstamp"]).drop_duplicates().reset_index(drop=True)
-
-        # For each unique symbol, add a corresponding close price column
-        for symbol in df_selected["symbol"].unique():
-            # Filter rows for this symbol
-            df_symbol = df_selected[df_selected["symbol"] == symbol].reset_index(drop=True)
-
-            # Create column name like "close-COIN"
-            new_price_column = f"{self.price_column_}_{symbol}"
-
-            # Create temporary dataframe with timestamp and price
-            temp_df = pd.DataFrame({
-                "tstamp": df_symbol["tstamp"],
-                new_price_column: df_symbol[self.price_column_]
-            })
-
-            # Join with our result dataframe
-            result_df = pd.merge(result_df, temp_df, on="tstamp", how="left")
-            result_df = result_df.reset_index(drop=True) # do not dropna() since irrelevant symbol would affect dataset
-
-        return result_df
-    def get_datasets(self, training_minutes: int, training_start_index: int = 0, testing_size: Optional[int] = None) -> None:
-        self.training_df_ = self.market_data_.iloc[training_start_index:training_minutes - 1, :].copy()
-        self.training_df_ = self.training_df_.dropna().reset_index(drop=True)
-
-        testing_start_index = training_start_index + training_minutes
-        if testing_size is None:
-            self.testing_df_ = self.market_data_.iloc[testing_start_index:, :].copy()
-        else:
-            self.testing_df_ = self.market_data_.iloc[testing_start_index:testing_start_index + testing_size, :].copy()
-        self.testing_df_ = self.testing_df_.dropna().reset_index(drop=True)
-
-    def colnames(self) -> List[str]:
-        return [f"{self.price_column_}_{self.symbol_a_}", f"{self.price_column_}_{self.symbol_b_}"]
-
-    def fit_VECM(self):
-        vecm_df = self.training_df_[self.colnames()].reset_index(drop=True)
-        vecm_model = VECM(vecm_df, coint_rank=1)
-        vecm_fit = vecm_model.fit()
-
-        # URGENT check beta and alpha
-
-        # Check if the model converged properly
-        if not hasattr(vecm_fit, "beta") or vecm_fit.beta is None:
-            print(f"{self}: VECM model failed to converge properly")
-
-        self.vecm_fit_ = vecm_fit
-        # print(f"{self}: beta={self.vecm_fit_.beta} alpha={self.vecm_fit_.alpha}"  )
-        # print(f"{self}: {self.vecm_fit_.summary()}")
-        pass
-
-    def check_cointegration(self):
-        from statsmodels.tsa.vector_ar.vecm import coint_johansen
-        df = self.training_df_[self.colnames()].reset_index(drop=True)
-        result = coint_johansen(df, det_order=0, k_ar_diff=1)
-        # print(f"{self}: lr1={result.lr1[0]} cvt={result.cvt[0, 1]}.")
-        is_cointegrated = result.lr1[0] > result.cvt[0, 1]
-
-        return is_cointegrated
-
-    def train_pair(self) -> bool:
-        is_cointegrated = self.check_cointegration()
-        if not is_cointegrated:
-            return False
-            pass
-
-        print('*' * 80 + '\n' + f"**************** {self} IS COINTEGRATED ****************\n" + '*' * 80)
-        self.fit_VECM()
-        diseq_series = self.training_df_[self.colnames()] @ self.vecm_fit_.beta
-        self.training_mu_ = diseq_series.mean().iloc[0]
-        self.training_std_ = diseq_series.std().iloc[0]
-
-        self.training_df_["dis-equilibrium"] = self.training_df_[self.colnames()] @ self.vecm_fit_.beta
-        # Normalize the dis-equilibrium
-        self.training_df_["scaled_dis-equilibrium"] = (
-            diseq_series - self.training_mu_
-        ) / self.training_std_
-
-        return True
-
-    def predict(self) -> None:
-        predicted_prices = self.vecm_fit_.predict(steps=len(self.testing_df_))
-
-        # Convert prediction to a DataFrame for readability
-        # predicted_df = 
-
-        self.predicted_df_ = pd.merge(
-            self.testing_df_.reset_index(drop=True),
-            pd.DataFrame(predicted_prices, columns=self.colnames()),
-            left_index=True,
-            right_index=True,
-            suffixes=("", "_pred"),
-        ).dropna()
-
-        self.predicted_df_["disequilibrium"] = self.predicted_df_[self.colnames()] @ self.vecm_fit_.beta
-
-        self.predicted_df_["scaled_disequilibrium"] = (
-            abs(self.predicted_df_["disequilibrium"] - self.training_mu_) / self.training_std_
-        )
-
-        # Reset index to ensure proper indexing
-        self.predicted_df_ = self.predicted_df_.reset_index()
-        return self.predicted_df_
-
-        
-    def __repr__(self) ->str:
-        return f"{self.symbol_a_} & {self.symbol_b_}"
-

tests/test_spbt_day_panel.py

@@ -0,0 +1,243 @@
+import importlib.util
+import sqlite3
+from pathlib import Path
+
+import pandas as pd
+
+
+def load_panel_app_module():
+    module_path = Path("panel/spbt_day_panel.py").resolve()
+    spec = importlib.util.spec_from_file_location("spbt_day_panel_app", module_path)
+    module = importlib.util.module_from_spec(spec)
+    assert spec.loader is not None
+    spec.loader.exec_module(module)
+    return module
+
+
+def create_panel_fixture_db(db_path: Path) -> None:
+    trading_day_start_ns = pd.Timestamp("2026-06-17T00:00:00Z").value
+    conn = sqlite3.connect(db_path)
+    try:
+        conn.execute(
+            """
+            CREATE TABLE selector_pairs (
+                time_ns INTEGER,
+                tstamp TEXT,
+                pair_name TEXT,
+                instrument_a TEXT,
+                instrument_b TEXT,
+                mr_score TEXT
+            )
+            """
+        )
+        conn.execute(
+            """
+            CREATE TABLE trading_instructions (
+                tstamp TEXT,
+                tstamp_ns INTEGER,
+                type TEXT,
+                book_id TEXT,
+                strategy_id TEXT,
+                action TEXT,
+                quote_asset TEXT,
+                assets TEXT,
+                scaled_disequilibrium REAL,
+                beta REAL
+            )
+            """
+        )
+        conn.execute(
+            """
+            CREATE TABLE market (
+                tstamp TEXT,
+                tstamp_ns INTEGER,
+                exch_acct TEXT,
+                instrument_id TEXT,
+                open REAL,
+                high REAL,
+                low REAL,
+                close REAL,
+                volume REAL,
+                vwap REAL,
+                num_trades INTEGER
+            )
+            """
+        )
+        conn.execute(
+            "INSERT INTO selector_pairs VALUES (?, ?, ?, ?, ?, ?)",
+            (
+                10,
+                "2026-06-17T00:00:00Z",
+                "AAA:USD-BBB:USD",
+                "EXCH:PAIR-AAA-USD",
+                "EXCH:PAIR-BBB-USD",
+                '{"final":"0.5"}',
+            ),
+        )
+        conn.executemany(
+            "INSERT INTO trading_instructions VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+            [
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "TARGET_POSITION",
+                    "book",
+                    "strategy-AAA:USD-BBB:USD",
+                    "TARGET",
+                    "USD",
+                    '{"AAA":{"reference_price":"100","strength":"0.5"},'
+                    '"BBB":{"reference_price":"50","strength":"-0.5"}}',
+                    -1.25,
+                    0.75,
+                ),
+                (
+                    "2026-06-17T00:01:00Z",
+                    trading_day_start_ns + 60_000_000_000,
+                    "CLOSE_POSITION",
+                    "book",
+                    "strategy-AAA:USD-BBB:USD",
+                    "CLOSE",
+                    "USD",
+                    '{"AAA":{"reference_price":"110"},'
+                    '"BBB":{"reference_price":"45"}}',
+                    -0.5,
+                    0.75,
+                ),
+            ],
+        )
+        conn.executemany(
+            "INSERT INTO market VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+            [
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "EXCH",
+                    "PAIR-AAA-USD",
+                    100.0,
+                    100.0,
+                    100.0,
+                    100.0,
+                    1.0,
+                    100.0,
+                    1,
+                ),
+                (
+                    "2026-06-17T00:00:00Z",
+                    trading_day_start_ns,
+                    "EXCH",
+                    "PAIR-BBB-USD",
+                    50.0,
+                    50.0,
+                    50.0,
+                    50.0,
+                    1.0,
+                    50.0,
+                    1,
+                ),
+            ],
+        )
+        conn.commit()
+    finally:
+        conn.close()
+
+
+def test_pair_analyze_grid_keeps_clean_labels_and_full_pair_values():
+    module = load_panel_app_module()
+    pair_theo_ret = pd.DataFrame(
+        {
+            "pair_name": ["BTC:USD-ETH:USD", "ADA:USD-BTC:USD"],
+            "mr_ranking": [2, 1],
+            "realized_pnl": [0.0, 0.0],
+            "unrealized_pnl": [0.0, 0.0],
+        }
+    )
+
+    formatted = module.spbt_day.format_pair_theo_ret_for_analyze_grid(pair_theo_ret)
+
+    assert formatted["pair_name"].tolist() == ["BTC-ETH", "ADA-BTC"]
+    assert formatted[module.spbt_day.PAIR_NAME_VALUE_COLUMN].tolist() == [
+        "BTC:USD-ETH:USD",
+        "ADA:USD-BTC:USD",
+    ]
+
+
+def test_panel_app_uses_fast_list_template(tmp_path):
+    module = load_panel_app_module()
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    view = app.view
+
+    assert not hasattr(app, "refresh_button")
+    assert isinstance(view, module.pn.template.FastListTemplate)
+    assert view.title == module.APP_TITLE
+    assert view.theme is module.pn.template.DarkTheme
+    assert view.sidebar_width == module.APP_SIDEBAR_WIDTH
+    assert view.accent_base_color == module.APP_ACCENT_COLOR
+    assert view.header_background == module.APP_HEADER_COLOR
+    assert len(view.sidebar) == 1
+    assert len(view.main) == 1
+
+
+def test_panel_app_calculates_pairs_and_selected_pair_outputs(tmp_path):
+    module = load_panel_app_module()
+    data_dir = tmp_path / "data"
+    data_dir.mkdir()
+    db_path = data_dir / "20260617.spbt_results.db"
+    create_panel_fixture_db(db_path)
+
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    app.directory_input.value = str(data_dir)
+    app.refresh_files()
+    app.min_pctg_change_input.value = 0.0
+
+    app.calculate()
+
+    assert app.file_select.value == str(db_path)
+    assert app.directory_input.sizing_mode == "stretch_width"
+    assert app.directory_input.width is None
+    assert app.file_select.sizing_mode == "stretch_width"
+    assert app.file_select.width is None
+    assert app.min_pctg_change_input.sizing_mode == "stretch_width"
+    assert app.min_pctg_change_input.width is None
+    assert app.calculate_button.width == 110
+    assert app.total_pnl_histogram.sizing_mode == "stretch_width"
+    assert app.selected_pair_market_plot.sizing_mode == "stretch_width"
+    assert app.pair_theo_ret_table.pagination is None
+    assert app.pair_theo_ret_table.layout == "fit_data_table"
+    assert app.pair_theo_ret_table.value["pair_name"].tolist() == ["AAA-BBB"]
+    assert (
+        app.pair_theo_ret_table.value[module.spbt_day.PAIR_NAME_VALUE_COLUMN].tolist()
+        == ["AAA:USD-BBB:USD"]
+    )
+    assert app.selected_pair_name is None
+    assert app.selected_pair_executions_table.value.empty
+    assert app.selected_pair_market_plot.object is None
+
+    app.analyze_pair_row(0)
+
+    assert app.selected_pair_name == "AAA:USD-BBB:USD"
+    assert app.selected_pair_executions_table.value["action"].tolist() == [
+        "TARGET",
+        "TARGET",
+        "CLOSE",
+        "CLOSE",
+    ]
+    assert app.selected_pair_market_plot.object is not None
+
+
+def test_calculate_refreshes_file_list_before_loading(tmp_path):
+    module = load_panel_app_module()
+    data_dir = tmp_path / "data"
+    data_dir.mkdir()
+
+    app = module.SpbtDayPanelApp(repo_root=tmp_path)
+    app.directory_input.value = str(data_dir)
+    app.refresh_files()
+    assert app.file_select.value is None
+
+    db_path = data_dir / "20260617.spbt_results.db"
+    create_panel_fixture_db(db_path)
+
+    app.calculate()
+
+    assert app.file_select.value == str(db_path)
+    assert app.pair_theo_ret_table.value["pair_name"].tolist() == ["AAA-BBB"]