Celes Renata
301 lines
21 KiB
Markdown
301 lines
21 KiB
Markdown
# Implementation Plan: Competitive Intelligence & Historical Pattern Matching Layer
|
||
|
||
## Overview
|
||
|
||
This plan implements a third signal layer for the Stonks Oracle aggregation engine: competitive intelligence and historical pattern matching. The layer mines existing PostgreSQL data (document_impact_records, trend_windows, document_company_mentions) to identify how similar catalyst types resolved historically for a company and its competitors, then feeds pattern-based signals into the aggregation engine alongside company-specific (layer 1) and macro (layer 2) signals. All modules extend existing services — no new Kubernetes deployments required. Tasks are ordered so each step builds on the previous, with property-based tests validating core logic early.
|
||
|
||
## Tasks
|
||
|
||
- [x] 1. Database migration and shared schemas
|
||
- [x] 1.1 Create PostgreSQL migration `infra/migrations/017_competitive_historical_patterns.sql`
|
||
- Add `competitor_relationships` table with id (UUID PK), company_a_id (FK companies), company_b_id (FK companies), relationship_type (VARCHAR CHECK direct_rival|same_sector|overlapping_products|supply_chain_adjacent), strength (FLOAT CHECK [0,1]), bidirectional (BOOLEAN), source (VARCHAR CHECK manual|inferred), active (BOOLEAN), created_at, updated_at
|
||
- Add `competitive_signal_records` table with id (UUID PK), source_document_id (FK documents), source_ticker, target_ticker, catalyst_type, pattern_confidence, signal_direction, signal_strength, relationship_strength, computed_at
|
||
- Add CHECK constraint preventing self-referencing relationships (company_a_id != company_b_id)
|
||
- Add unique index on (LEAST(company_a_id, company_b_id), GREATEST(company_a_id, company_b_id)) WHERE active = TRUE to prevent duplicate active pairs
|
||
- Add indexes: idx_competitor_rel_company_a, idx_competitor_rel_company_b (both WHERE active = TRUE), idx_competitive_signals_target (target_ticker, computed_at DESC), idx_competitive_signals_source (source_ticker, computed_at DESC)
|
||
- _Requirements: 7.1, 7.2_
|
||
|
||
- [x] 1.2 Add new Pydantic schemas and enums to `services/shared/schemas.py`
|
||
- Add `RelationshipType` enum (direct_rival, same_sector, overlapping_products, supply_chain_adjacent)
|
||
- Add `CatalystTier` enum (major_corporate_decision, routine_signal)
|
||
- Add `MAJOR_DECISION_CATALYSTS` frozenset (m_and_a, legal, restructuring, leadership_change, strategic_pivot, buyback, dividend_change)
|
||
- Add `CompetitorRelationshipSchema`, `CompetitiveSignalRecordSchema`, `HistoricalPatternSchema` Pydantic models
|
||
- _Requirements: 1.1, 4.4, 7.1, 7.2, 11.1_
|
||
|
||
- [x] 1.3 Add competitive configuration fields to `services/shared/config.py`
|
||
- Add `CompetitiveConfig` dataclass with fields: competitive_signal_weight (0.2), competitive_enabled (True), pattern_confidence_threshold (0.3), propagation_strength_threshold (0.2), routine_lookback_days (180), major_decision_lookback_days (365), major_decision_weight_multiplier (1.3), staleness_window_days (180), staleness_recent_days (90), staleness_decay_penalty (0.5), min_pattern_samples (3)
|
||
- Add `competitive: CompetitiveConfig` to `AppConfig` with env var loading in `load_config()`
|
||
- _Requirements: 5.6, 6.1, 9.1, 9.2, 11.2, 11.3_
|
||
|
||
- [x] 2. Checkpoint — Ensure migration and schemas are consistent
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 3. Competitor Registry and auto-inference
|
||
- [x] 3.1 Implement `services/symbol_registry/competitors.py`
|
||
- Implement `CompetitorRelationshipCreate` and `CompetitorRelationship` Pydantic models for API request/response
|
||
- Implement `POST /companies/{company_id}/competitors` — create relationship with audit event
|
||
- Implement `GET /companies/{company_id}/competitors` — list active relationships ordered by strength descending
|
||
- Implement `PUT /companies/{company_id}/competitors/{relationship_id}` — update relationship with audit event recording previous state
|
||
- Implement `DELETE /companies/{company_id}/competitors/{relationship_id}` — soft-delete (set active=False), preserve row
|
||
- Register routes as a FastAPI router on the Symbol Registry app
|
||
- Handle error cases: self-referencing (400), duplicate active pair (409), non-existent company (404)
|
||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
|
||
|
||
- [x] 3.2 Write property test for competitor relationship persistence round-trip
|
||
- **Property 1: Competitor relationship persistence round-trip**
|
||
- **Validates: Requirements 1.1, 7.1**
|
||
|
||
- [x] 3.3 Write property test for competitor query completeness and ordering
|
||
- **Property 2: Competitor query completeness and ordering**
|
||
- **Validates: Requirements 1.2**
|
||
|
||
- [x] 3.4 Write property test for soft-delete preserves row
|
||
- **Property 3: Soft-delete preserves row**
|
||
- **Validates: Requirements 1.3**
|
||
|
||
- [x] 3.5 Implement `services/symbol_registry/competitor_inference.py`
|
||
- Implement `infer_competitors(pool, company_id) -> list[CompetitorRelationship]`
|
||
- Query companies sharing the same sector and industry
|
||
- Rank candidates by co-mention frequency in `document_company_mentions`
|
||
- Compute strength = `0.3 * sector_match + 0.7 * normalized_co_mention_count`
|
||
- Upsert relationships with `source='inferred'`, refreshing strength on re-inference (no duplicates)
|
||
- Implement `POST /companies/{company_id}/competitors/infer` endpoint returning candidate relationships
|
||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5_
|
||
|
||
- [x] 3.6 Write property test for auto-inference produces valid candidates
|
||
- **Property 4: Auto-inference produces valid candidates**
|
||
- **Validates: Requirements 2.1, 2.3**
|
||
|
||
- [x] 3.7 Write property test for auto-inference ranks by co-mention frequency
|
||
- **Property 5: Auto-inference ranks by co-mention frequency**
|
||
- **Validates: Requirements 2.2**
|
||
|
||
- [x] 3.8 Write property test for auto-inference idempotence
|
||
- **Property 6: Auto-inference idempotence**
|
||
- **Validates: Requirements 2.4**
|
||
|
||
- [x] 4. Checkpoint — Ensure competitor registry and inference work correctly
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 5. Pattern Matcher — core historical pattern mining
|
||
- [x] 5.1 Implement `services/aggregation/pattern_matcher.py`
|
||
- Implement `HistoricalPattern` dataclass matching the design specification
|
||
- Implement `classify_catalyst_tier(catalyst_type) -> str` — deterministic mapping of major_corporate_decision vs routine_signal catalyst types
|
||
- Implement `compute_pattern_confidence(sample_count, outcome_consistency, data_recency_days, tier) -> float` using the formula: `sample_factor * 0.4 + consistency * 0.4 + recency_factor * 0.2`, with 1.3× multiplier for major decisions
|
||
- Implement `find_self_patterns(pool, ticker, catalyst_type, horizons) -> list[HistoricalPattern]` — query document_impact_records joined with trend_windows for same company-catalyst pair across configurable time horizons (1d, 7d, 30d)
|
||
- Implement `find_cross_company_patterns(pool, source_ticker, target_ticker, catalyst_type, horizons) -> list[HistoricalPattern]` — query cross-company historical patterns
|
||
- Only consider records linked to document_intelligence with validation_status='valid' and documents with status != 'rejected'
|
||
- Apply insufficient data threshold: when sample_count < 3, cap confidence at 0.25 and set insufficient_data=True
|
||
- Apply staleness decay: when no instances in last 90 days and all data older than 180 days, apply 0.5 decay penalty
|
||
- Use 365-day lookback for major_corporate_decision catalysts, 180-day for routine_signal
|
||
- Compute separate HistoricalPatterns for each catalyst tier
|
||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 11.1, 11.2, 11.3, 11.5_
|
||
|
||
- [x] 5.2 Write property test for pattern computation correctness
|
||
- **Property 7: Pattern computation correctness**
|
||
- **Validates: Requirements 3.1, 3.2, 4.2**
|
||
|
||
- [x] 5.3 Write property test for pattern confidence monotonicity
|
||
- **Property 8: Pattern confidence monotonicity**
|
||
- **Validates: Requirements 3.3, 11.2**
|
||
|
||
- [x] 5.4 Write property test for insufficient data threshold
|
||
- **Property 9: Insufficient data threshold**
|
||
- **Validates: Requirements 3.4**
|
||
|
||
- [x] 5.5 Write property test for valid-only data filtering
|
||
- **Property 10: Valid-only data filtering**
|
||
- **Validates: Requirements 3.5**
|
||
|
||
- [x] 5.6 Write property test for catalyst tier classification determinism
|
||
- **Property 19: Catalyst tier classification determinism**
|
||
- **Validates: Requirements 11.1**
|
||
|
||
- [x] 5.7 Write property test for major decision extended lookback
|
||
- **Property 20: Major decision extended lookback**
|
||
- **Validates: Requirements 11.3, 11.5**
|
||
|
||
- [x] 6. Checkpoint — Ensure pattern matcher and property tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 7. Signal Propagation Engine
|
||
- [x] 7.1 Implement `services/aggregation/signal_propagation.py`
|
||
- Implement `CompetitiveSignalRecord` dataclass matching the design specification
|
||
- Implement `propagate_signals(pool, ticker, catalyst_type, impact_score, document_id, config) -> list[CompetitiveSignalRecord]` — look up competitors, query cross-company patterns, produce weighted competitive signals
|
||
- Signal weighting: `signal_strength = pattern.avg_strength * relationship.strength * pattern.pattern_confidence * source_impact_score`
|
||
- Signal direction: bullish if pattern.bullish_pct > bearish_pct, else bearish
|
||
- Skip propagation when relationship.strength < propagation_strength_threshold (default 0.2), log skip reason
|
||
- Exclude patterns with pattern_confidence < pattern_confidence_threshold (default 0.3), log exclusion reason
|
||
- Persist CompetitiveSignalRecord objects to the competitive_signal_records PostgreSQL table
|
||
- Implement `build_pattern_weighted_signals(patterns, competitive_signals, reference_time, window, config) -> list[WeightedSignal]` — convert pattern/competitive signals to WeightedSignal objects for aggregation
|
||
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 9.1_
|
||
|
||
- [x] 7.2 Write property test for competitive signal strength monotonicity
|
||
- **Property 11: Competitive signal strength monotonicity**
|
||
- **Validates: Requirements 4.3**
|
||
|
||
- [x] 7.3 Write property test for signal propagation threshold gating
|
||
- **Property 12: Signal propagation threshold gating**
|
||
- **Validates: Requirements 4.5, 9.1**
|
||
|
||
- [x] 7.4 Write property test for pattern signal to WeightedSignal conversion
|
||
- **Property 13: Pattern signal to WeightedSignal conversion**
|
||
- **Validates: Requirements 5.2**
|
||
|
||
- [x] 7.5 Write property test for competitive signal persistence round-trip
|
||
- **Property 21: Competitive signal persistence round-trip**
|
||
- **Validates: Requirements 4.4, 7.2**
|
||
|
||
- [x] 8. Checkpoint — Ensure signal propagation and property tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 9. Aggregation engine integration
|
||
- [x] 9.1 Extend `services/aggregation/worker.py` to incorporate pattern-based and competitive signals
|
||
- Add `competitive_signal_weight` and `competitive_enabled` fields to `AggregationConfig`
|
||
- In `aggregate_company_window`, check competitive toggle state from `risk_configs` table (same pattern as macro toggle)
|
||
- When competitive layer is enabled: query self-company historical patterns for active catalyst types in the window, query competitive signals targeting this ticker
|
||
- Convert each pattern signal to a `WeightedSignal` using: document_id = source document, sentiment_value = +1.0 (bullish) or -1.0 (bearish), impact_score = signal_strength × competitive_signal_weight, recency decay from source document publication time, confidence gating from pattern_confidence
|
||
- Merge pattern/competitive signals with company-specific and macro signals before computing trend direction, strength, confidence, and contradiction score
|
||
- Include contributing source_document_ids in evidence references for traceability
|
||
- When competitive layer is disabled or no pattern data exists, produce identical output to company+macro-only aggregation
|
||
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6_
|
||
|
||
- [x] 9.2 Write property test for pattern-company contradiction detection
|
||
- **Property 14: Pattern-company contradiction detection**
|
||
- **Validates: Requirements 5.3**
|
||
|
||
- [x] 9.3 Write property test for pattern evidence traceability
|
||
- **Property 15: Pattern evidence traceability**
|
||
- **Validates: Requirements 5.4**
|
||
|
||
- [x] 9.4 Write property test for no-degradation and disabled-layer equivalence
|
||
- **Property 16: No-degradation and disabled-layer equivalence**
|
||
- **Validates: Requirements 5.5, 6.2**
|
||
|
||
- [x] 9.5 Write property test for staleness decay penalty
|
||
- **Property 17: Staleness decay penalty**
|
||
- **Validates: Requirements 9.2**
|
||
|
||
- [x] 10. Checkpoint — Ensure aggregation integration works correctly
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 11. Pattern-only suppression and safety
|
||
- [x] 11.1 Extend `services/recommendation/suppression.py` with pattern-only suppression
|
||
- Add `PATTERN_ONLY_SIGNAL = "pattern_only_signal"` to `SuppressionReason` enum
|
||
- Implement `evaluate_pattern_only_suppression(summary, pattern_signal_count, company_signal_count, macro_signal_count) -> bool`
|
||
- When pattern-based signals are the sole basis for a trend direction change, force recommendation to `mode='informational'` and append pattern-only caveat to thesis
|
||
- _Requirements: 9.3_
|
||
|
||
- [x] 11.2 Write property test for pattern-only suppression
|
||
- **Property 18: Pattern-only suppression**
|
||
- **Validates: Requirements 9.3**
|
||
|
||
- [x] 12. Competitive layer toggle and API endpoints
|
||
- [x] 12.1 Implement competitive toggle and status endpoints in `services/api/app.py`
|
||
- Add `GET /api/admin/competitive/status` returning current enabled/disabled state from `risk_configs` table
|
||
- Add `PUT /api/admin/competitive/toggle` to switch competitive layer on/off, persisting to `risk_configs` and recording an audit event with previous state, new state, and operator
|
||
- Toggle state is read from PostgreSQL at the start of each aggregation cycle (no caching)
|
||
- When disabled, pattern mining remains queryable via API but signal propagation is skipped during aggregation
|
||
- When re-enabled, resume computing signals using latest historical data including intelligence ingested while disabled
|
||
- _Requirements: 6.1, 6.2, 6.3, 6.4, 6.5, 6.7_
|
||
|
||
- [x] 12.2 Implement pattern and competitive signal query endpoints in `services/api/app.py`
|
||
- Add `GET /api/patterns/{ticker}` — historical patterns for a company, filterable by catalyst_type and time_horizon
|
||
- Add `GET /api/patterns/{ticker}/competitors` — cross-company patterns showing how this company's catalysts affected competitors
|
||
- Add `GET /api/patterns/{ticker}/competitive-signals` — recent competitive signals targeting this company
|
||
- Add `GET /api/patterns/{ticker}/decisions` — major corporate decision history with trend outcomes and pattern statistics
|
||
- Include sample_count, outcome distribution, pattern_confidence, and date range in all responses
|
||
- _Requirements: 10.1, 10.2, 10.3, 10.4, 11.4, 11.6_
|
||
|
||
- [x] 13. Checkpoint — Ensure API endpoints and toggle logic work correctly
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 14. Lake publisher extensions
|
||
- [x] 14.1 Add competitive fact publishers to the lake publisher service
|
||
- Implement `publish_competitor_relationship_fact` writing partitioned Parquet datasets to `stonks-lakehouse/warehouse/competitor_relationships/dt={date}/`
|
||
- Implement `publish_competitive_signal_fact` writing partitioned Parquet datasets to `stonks-lakehouse/warehouse/competitive_signals/dt={date}/target_ticker={ticker}/`
|
||
- Register new fact types in the lake publisher's job processing loop
|
||
- _Requirements: 7.3, 7.4_
|
||
|
||
- [x] 15. Signal propagation wiring into aggregation pipeline
|
||
- [x] 15.1 Wire signal propagation into the aggregation worker
|
||
- After document intelligence is produced for a company, trigger signal propagation for the company's competitors
|
||
- In the aggregation cycle, call `propagate_signals` for each new document intelligence record when competitive layer is enabled
|
||
- Handle sustained propagation errors: after configurable threshold (default 5 consecutive failures), alert operators and continue with company-specific + macro signals only
|
||
- _Requirements: 4.1, 9.4_
|
||
|
||
- [x] 15.2 Wire pattern mining into the aggregation cycle
|
||
- During `aggregate_company_window`, call pattern matcher for self-company patterns and collect competitive signals for the ticker
|
||
- Merge resulting WeightedSignals into the signal list before trend computation
|
||
- Ensure evidence references include pattern signal source document IDs
|
||
- _Requirements: 5.1, 5.4_
|
||
|
||
- [x] 16. Checkpoint — Ensure full backend pipeline works end-to-end
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 17. Dashboard — Competitors panel and historical patterns
|
||
- [x] 17.1 Add competitors panel to Company Detail page
|
||
- On `frontend/src/pages/CompanyDetail.tsx`, add a Competitors tab showing active competitor relationships with ticker, relationship_type, strength score, source (manual/inferred)
|
||
- Add API hooks for `GET /companies/{company_id}/competitors` in `frontend/src/api/hooks.ts`
|
||
- Add infer button triggering `POST /companies/{company_id}/competitors/infer`
|
||
- _Requirements: 8.1_
|
||
|
||
- [x] 17.2 Add historical patterns panel to Company Detail page
|
||
- On `frontend/src/pages/CompanyDetail.tsx`, add a Historical Patterns tab showing recent patterns: catalyst_type, outcome distribution (bullish_pct, bearish_pct), sample_count, pattern_confidence
|
||
- Add API hook for `GET /api/patterns/{ticker}`
|
||
- _Requirements: 8.2_
|
||
|
||
- [x] 17.3 Add competitive signals panel to Company Detail page
|
||
- On `frontend/src/pages/CompanyDetail.tsx`, add a Competitive Signals tab showing incoming signals: source ticker, catalyst_type, signal_direction, signal_strength
|
||
- Add API hook for `GET /api/patterns/{ticker}/competitive-signals`
|
||
- Click-through on a signal shows full detail: source company, source document, catalyst_type, historical pattern statistics, competitor relationship
|
||
- _Requirements: 8.5, 8.4_
|
||
|
||
- [x] 17.4 Add corporate decision timeline to Company Detail page
|
||
- On `frontend/src/pages/CompanyDetail.tsx`, add a Decisions tab showing major_corporate_decision events: catalyst type, date, summary, trend outcome that followed
|
||
- Add API hook for `GET /api/patterns/{ticker}/decisions`
|
||
- _Requirements: 11.4_
|
||
|
||
- [x] 17.5 Add pattern-based evidence indicators to Trend detail page
|
||
- On `frontend/src/pages/TrendDetail.tsx`, visually distinguish pattern-based and competitive signal evidence from company-specific and macro evidence (badge/icon differentiation)
|
||
- _Requirements: 8.3_
|
||
|
||
- [x] 17.6 Add competitive toggle to Trading Controls page
|
||
- On `frontend/src/pages/Trading.tsx`, add competitive signal layer enable/disable switch alongside existing macro toggle, with confirmation dialog
|
||
- Add API hooks for `GET /api/admin/competitive/status` and `PUT /api/admin/competitive/toggle`
|
||
- _Requirements: 6.6_
|
||
|
||
- [x] 18. Checkpoint — Ensure frontend pages render and integrate with API
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
- [x] 19. Integration wiring and final validation
|
||
- [x] 19.1 Write integration tests for competitive pipeline end-to-end
|
||
- Test document intelligence → pattern mining → signal propagation → aggregation flow
|
||
- Test lake publisher writes correct Parquet partitions for competitor relationships and competitive signals
|
||
- Test competitive toggle state change propagates to next aggregation cycle
|
||
- Test toggle disable/re-enable cycle preserves data integrity
|
||
- _Requirements: 4.1, 5.1, 6.1, 6.4, 7.3_
|
||
|
||
- [x] 19.2 Write unit tests for API endpoints and dashboard components
|
||
- Test competitor CRUD endpoints return correct data and error codes (400, 404, 409)
|
||
- Test pattern query endpoints return correct data with filtering
|
||
- Test competitive toggle endpoint persists state and records audit event
|
||
- Test auto-inference endpoint with empty data, single company, no co-mentions
|
||
- Add MSW handlers for competitive endpoints in `frontend/src/test/mocks/handlers.ts`
|
||
- Test competitors panel, historical patterns panel, competitive signals panel, and decision timeline render correctly
|
||
- _Requirements: 1.4, 2.5, 6.5, 8.1, 8.2, 8.5, 10.1, 10.4_
|
||
|
||
- [x] 20. Final checkpoint — Ensure all tests pass
|
||
- Ensure all tests pass, ask the user if questions arise.
|
||
|
||
## Notes
|
||
|
||
- Tasks marked with `*` are optional and can be skipped for faster MVP
|
||
- Each task references specific requirements for traceability
|
||
- Checkpoints ensure incremental validation after each major phase
|
||
- Property tests validate the 21 correctness properties from the design using Hypothesis
|
||
- The design uses Python throughout — no language selection needed
|
||
- No new Kubernetes deployments required; all modules extend existing services
|
||
- Next migration number is 017 (016 is global-news-interpolation)
|
||
- Competitive layer follows the same toggle/suppression/aggregation pattern as the macro layer for consistency
|