diff --git a/.kiro/specs/stonks-oracle/tasks.md b/.kiro/specs/stonks-oracle/tasks.md index 559035b..a4db06b 100644 --- a/.kiro/specs/stonks-oracle/tasks.md +++ b/.kiro/specs/stonks-oracle/tasks.md @@ -172,15 +172,75 @@ - Document how to toggle trading modes and approve live execution - _Requirements: 8.2, 12.1_ -## Recommended First Vertical Slice -- [ ] Track 5 to 10 symbols -- [ ] Ingest one market data API, one news API, and one filings source per symbol group -- [ ] Persist raw artifacts to MinIO and metadata to PostgreSQL -- [ ] Extract structured document intelligence through Ollama -- [ ] Generate 7-day company trend summaries with market context -- [ ] Produce paper-trade recommendations only -- [ ] Publish analytical facts for bars, signals, and paper trades into MinIO -- [ ] Expose a simple dashboard with evidence, trend cards, and prediction-vs-outcome views +## Phase 17 - First Vertical Slice: Live Pipeline End-to-End + +- [ ] 17. Activate the full data pipeline for a set of tracked symbols +- [ ] 17.1 Seed initial symbols and configure sources via the dashboard + - Use the dashboard Companies page to add 5-10 symbols (e.g. AAPL, MSFT, GOOGL, AMZN, TSLA, NVDA, META, JPM, V, UNH) + - For each company, add sources via the Company Detail → Sources tab: one `market_api` source (Polygon), one `news_api` source, one `filings_api` source + - Configure source `config` JSON with the correct Polygon endpoint patterns per ticker + - Verify companies and sources appear in the dashboard and via `curl https://stonks-registry.celestium.life/companies` + - _Requirements: 1.1, 1.2, 1.3, 2.1_ + +- [ ] 17.2 Wire the scheduler to enqueue ingestion jobs for active sources + - Verify the scheduler service reads active companies and sources from PostgreSQL + - Verify it enqueues Redis jobs for each source on its polling interval + - Check scheduler logs: `kubectl logs -n stonks-oracle deployment/scheduler --tail=50` + - Confirm ingestion queue has items: check Redis keys via `kubectl exec` + - Fix any issues with the scheduler → source → Redis queue flow + - _Requirements: 3.1, 3.2_ + +- [ ] 17.3 Validate ingestion workers pull data from Polygon and persist to MinIO/PostgreSQL + - Check ingestion worker logs for successful API calls to Polygon + - Verify raw market data artifacts land in MinIO `stonks-raw-market` bucket + - Verify document metadata rows appear in PostgreSQL `documents` table + - Verify `ingestion_runs` table shows completed runs with `items_fetched > 0` + - Debug and fix any adapter errors (auth, rate limits, response parsing) + - _Requirements: 4.1, 4.2, 4.3_ + +- [ ] 17.4 Validate parser normalizes documents and extractor produces intelligence + - Check parser worker logs for document processing + - Verify normalized text appears in MinIO `stonks-normalized` bucket + - Verify `parse_quality_score` and `parse_confidence` are set on documents + - Check extractor worker logs for Ollama calls + - Verify `document_intelligence` rows appear with summaries, sentiments, and impact records + - Verify `document_impact_records` link intelligence to companies + - Debug any Ollama connection or schema validation issues + - _Requirements: 5.1, 5.2, 6.1, 6.2, 6.3_ + +- [ ] 17.5 Validate aggregation produces trend summaries + - Check aggregation worker logs for trend window generation + - Verify `trend_windows` table has entries with direction, strength, confidence + - Verify `trend_evidence` table links trends to contributing documents + - Verify contradiction scores are computed + - Check the dashboard Trends page shows trend cards with real data + - _Requirements: 7.1, 7.2, 7.3_ + +- [ ] 17.6 Validate recommendation engine produces paper-trade recommendations + - Check recommendation worker logs for recommendation generation + - Verify `recommendations` table has entries with action, confidence, thesis + - Verify `recommendation_evidence` links recommendations to documents + - Verify risk evaluation runs and `risk_evaluations` table has entries + - Check the dashboard Recommendations page shows real recommendations + - _Requirements: 8.1, 8.2, 8.3_ + +- [ ] 17.7 Validate lake publisher writes analytical facts to MinIO/Trino + - Check lake-publisher worker logs for Parquet writes + - Verify MinIO `stonks-lakehouse` bucket has partitioned Parquet files + - Verify Trino can query the lakehouse tables via the SQL Explorer page + - Test a sample query: `SELECT * FROM lakehouse.stonks.documents LIMIT 10` + - _Requirements: 10.1, 10.2, 10.3_ + +- [ ] 17.8 Validate dashboard shows live data across all pages + - Verify Home page shows non-zero metrics (active companies, documents, recommendations) + - Verify Companies page lists seeded companies with source counts + - Verify Documents page shows ingested documents with parse quality badges + - Verify Trends page shows trend cards with real direction/strength + - Verify Recommendations page shows generated recommendations + - Verify Ops Pipeline page shows document stage counts + - Verify Ops Ingestion page shows throughput data + - Take screenshots or curl key endpoints to confirm + - _Requirements: 13.1, 13.2, 13.3, 13.4, 13.6_ ## Phase 16 - Web Dashboard Frontend