Market Matching Algorithm Improvements

This document describes the implementation of three critical improvements to Pam’s market matching algorithm, as identified in ANALYSIS_matching_algorithm.md.

Summary of Changes

Improvement #1: Semantic Subject Extraction

Problem

The old implementation used brittle keyword matching:

# BEFORE
subj = "BTC" if ("bitcoin" in title or "btc" in title) else \
       ("ETH" if ("ethereum" in title or "eth" in title) else "GEN")

Issues: - Missed variants: “BTCUSD”, “XBT”, “Bitcoin Reference Time Index” - Falls back to “GEN” which guarantees false matches - No ticker parsing for Kalshi’s BRTI/ERTI tickers

Solution

Added alias mapping with ticker parsing in matcher/ontology.py:

# AFTER
ASSET_ALIASES = {
    "BTC": ["bitcoin", "btc", "xbt", "brti", "btcusd", "btc/usd", "btc-usd",
            "kxbtc", "kbtc", "bitcoin reference time", "₿"],
    "ETH": ["ethereum", "eth", "ether", "erti", "ethusd", "eth/usd", "eth-usd",
            "kxeth", "keth", "ethereum reference time"],
    # ... additional assets
}

def extract_subject(title: str, ticker: str = "") -> str:
    # Priority 1: Parse Kalshi-style tickers (KXBTCD, BRTI-24AUG)
    # Priority 2: Check all aliases with word boundary matching
    # Returns "UNKNOWN" for fail-fast behavior instead of "GEN"

Before/After Examples

Improvement #2: Context-Aware Threshold Extraction

Problem

The old implementation used loose regex with no validation:

# BEFORE
m = re.search(r"(\d{2,3}(?:[.,]\d{3})*(?:\.\d+)?)k", title) or \
    re.search(r"(\d{5,7}(?:\.\d+)?)", title)

Issues: - Could extract dates instead of prices (“August 28” → 28) - No context awareness (price vs percentage vs date) - No validation against reasonable asset price ranges

Solution

Added ordered pattern matching with bounds validation in matcher/ontology.py:

# AFTER
THRESHOLD_PATTERNS = [
    # Dollar amounts (highest priority)
    (r"\$\s*(\d{1,3}(?:,\d{3})+)", lambda m: float(m.group(1).replace(",", ""))),
    # K suffix
    (r"(\d{2,3})\s*[kK](?:\s|$|[^a-zA-Z0-9])", lambda m: float(m.group(1)) * 1000),
    # Contextual extraction (above/below/exceed)
    (r"(?:above|below|exceed|reach)\s+(\d{5,7})", lambda m: float(m.group(1))),
]

THRESHOLD_BOUNDS = {
    "BTC": (10_000, 500_000),
    "ETH": (500, 20_000),
    # ...
}

def extract_threshold(title: str, subject: str) -> Optional[float]:
    # Ordered pattern matching with bounds validation

Before/After Examples

Improvement #3: Oracle Risk Scoring

Problem

The old implementation used a binary -0.1 penalty:

# BEFORE (in matcher/align.py)
if (L.resolution_source or "").lower() != (R.resolution_source or "").lower():
    score -= 0.1

Issues: - Underweights significant oracle divergence risk - CF Benchmarks RTI vs Binance can differ by 0.5-2% at volatile moments - Unknown oracles treated the same as known mismatches

Solution

Created new matcher/oracle_risk.py module with quantified risk scoring:

# AFTER
ORACLE_HIERARCHY = {
    "CFB RTI 60s avg": {"type": "index", "lag_sec": 60, "provider": "cf_benchmarks"},
    "Binance 1m close": {"type": "spot", "lag_sec": 60, "provider": "binance"},
    # ...
}

def oracle_divergence_risk(source_a: str, source_b: str) -> float:
    """Returns risk score from 0.0 to 1.0"""
    # Considers: provider mismatch, type mismatch (index vs spot), unknown sources

Updated matcher/align.py to use quantified risk:

# AFTER (in matcher/align.py)
oracle_risk = oracle_divergence_risk(source_l, source_r)
score -= oracle_risk * 0.4  # Up to -0.4 penalty (was -0.1)

Also enhanced sources/parsers.py with ordered pattern matching for better oracle detection.

Before/After Examples

Files Changed

New Files

• matcher/oracle_risk.py - Oracle divergence risk scoring module

Modified Files

• matcher/ontology.py - Added extract_subject() with alias mapping, extract_threshold() with bounds validation
• matcher/align.py - Added equivalence_score_detailed(), integrated oracle risk scoring
• sources/parsers.py - Added ordered pattern matching for resolution source detection
• arbitrage/crypto_scanner.py - Updated _extract_event_info() to use improved extraction

Test Files

• tests/test_matching_improvements.py - 35 unit tests covering all improvements
• tests/demo_matching_improvements.py - Demo script with before/after comparison

Testing

Unit Tests

python3 -m pytest tests/test_matching_improvements.py -v

Results: 35 tests passed

Test coverage: - TestSemanticSubjectExtraction - 8 tests - TestContextAwareThresholdExtraction - 8 tests - TestOracleRiskScoring - 6 tests - TestEnhancedParsers - 5 tests - TestEquivalenceScoring - 4 tests - TestRealWorldMarketPatterns - 4 tests

Demo Script

PYTHONPATH=. python3 tests/demo_matching_improvements.py

Shows before/after comparison with real-world market title patterns.

Impact Analysis

Combined Effect on Matching Quality

Key Improvements

1. False Negatives Reduced - Semantic extraction catches more valid matches (XBT, BRTI tickers, compound titles)

2. False Positives Reduced - “UNKNOWN” subject returns 0 score instead of potentially matching “GEN”

3. Risk-Adjusted Scoring - Oracle mismatch is now properly weighted, preventing execution of opportunities with high settlement risk

Configuration

No new configuration required. The improvements are backwards-compatible and use reasonable defaults:

• Asset bounds are based on current market price ranges (can be updated in THRESHOLD_BOUNDS)
• Oracle risk weights can be adjusted in ORACLE_HIERARCHY and PROVIDER_DIVERGENCE_RISK
• Maximum oracle penalty (0.4) can be changed in equivalence_score() call