phase 17: add vertical slice tasks for live pipeline activation

This commit is contained in:
Celes Renata
2026-04-11 20:43:47 -07:00
parent 37d5f9b01c
commit 1410d324b7
+69 -9
View File
@@ -172,15 +172,75 @@
- Document how to toggle trading modes and approve live execution
- _Requirements: 8.2, 12.1_
## Recommended First Vertical Slice
- [ ] Track 5 to 10 symbols
- [ ] Ingest one market data API, one news API, and one filings source per symbol group
- [ ] Persist raw artifacts to MinIO and metadata to PostgreSQL
- [ ] Extract structured document intelligence through Ollama
- [ ] Generate 7-day company trend summaries with market context
- [ ] Produce paper-trade recommendations only
- [ ] Publish analytical facts for bars, signals, and paper trades into MinIO
- [ ] Expose a simple dashboard with evidence, trend cards, and prediction-vs-outcome views
## Phase 17 - First Vertical Slice: Live Pipeline End-to-End
- [ ] 17. Activate the full data pipeline for a set of tracked symbols
- [ ] 17.1 Seed initial symbols and configure sources via the dashboard
- Use the dashboard Companies page to add 5-10 symbols (e.g. AAPL, MSFT, GOOGL, AMZN, TSLA, NVDA, META, JPM, V, UNH)
- For each company, add sources via the Company Detail → Sources tab: one `market_api` source (Polygon), one `news_api` source, one `filings_api` source
- Configure source `config` JSON with the correct Polygon endpoint patterns per ticker
- Verify companies and sources appear in the dashboard and via `curl https://stonks-registry.celestium.life/companies`
- _Requirements: 1.1, 1.2, 1.3, 2.1_
- [ ] 17.2 Wire the scheduler to enqueue ingestion jobs for active sources
- Verify the scheduler service reads active companies and sources from PostgreSQL
- Verify it enqueues Redis jobs for each source on its polling interval
- Check scheduler logs: `kubectl logs -n stonks-oracle deployment/scheduler --tail=50`
- Confirm ingestion queue has items: check Redis keys via `kubectl exec`
- Fix any issues with the scheduler → source → Redis queue flow
- _Requirements: 3.1, 3.2_
- [ ] 17.3 Validate ingestion workers pull data from Polygon and persist to MinIO/PostgreSQL
- Check ingestion worker logs for successful API calls to Polygon
- Verify raw market data artifacts land in MinIO `stonks-raw-market` bucket
- Verify document metadata rows appear in PostgreSQL `documents` table
- Verify `ingestion_runs` table shows completed runs with `items_fetched > 0`
- Debug and fix any adapter errors (auth, rate limits, response parsing)
- _Requirements: 4.1, 4.2, 4.3_
- [ ] 17.4 Validate parser normalizes documents and extractor produces intelligence
- Check parser worker logs for document processing
- Verify normalized text appears in MinIO `stonks-normalized` bucket
- Verify `parse_quality_score` and `parse_confidence` are set on documents
- Check extractor worker logs for Ollama calls
- Verify `document_intelligence` rows appear with summaries, sentiments, and impact records
- Verify `document_impact_records` link intelligence to companies
- Debug any Ollama connection or schema validation issues
- _Requirements: 5.1, 5.2, 6.1, 6.2, 6.3_
- [ ] 17.5 Validate aggregation produces trend summaries
- Check aggregation worker logs for trend window generation
- Verify `trend_windows` table has entries with direction, strength, confidence
- Verify `trend_evidence` table links trends to contributing documents
- Verify contradiction scores are computed
- Check the dashboard Trends page shows trend cards with real data
- _Requirements: 7.1, 7.2, 7.3_
- [ ] 17.6 Validate recommendation engine produces paper-trade recommendations
- Check recommendation worker logs for recommendation generation
- Verify `recommendations` table has entries with action, confidence, thesis
- Verify `recommendation_evidence` links recommendations to documents
- Verify risk evaluation runs and `risk_evaluations` table has entries
- Check the dashboard Recommendations page shows real recommendations
- _Requirements: 8.1, 8.2, 8.3_
- [ ] 17.7 Validate lake publisher writes analytical facts to MinIO/Trino
- Check lake-publisher worker logs for Parquet writes
- Verify MinIO `stonks-lakehouse` bucket has partitioned Parquet files
- Verify Trino can query the lakehouse tables via the SQL Explorer page
- Test a sample query: `SELECT * FROM lakehouse.stonks.documents LIMIT 10`
- _Requirements: 10.1, 10.2, 10.3_
- [ ] 17.8 Validate dashboard shows live data across all pages
- Verify Home page shows non-zero metrics (active companies, documents, recommendations)
- Verify Companies page lists seeded companies with source counts
- Verify Documents page shows ingested documents with parse quality badges
- Verify Trends page shows trend cards with real direction/strength
- Verify Recommendations page shows generated recommendations
- Verify Ops Pipeline page shows document stage counts
- Verify Ops Ingestion page shows throughput data
- Take screenshots or curl key endpoints to confirm
- _Requirements: 13.1, 13.2, 13.3, 13.4, 13.6_
## Phase 16 - Web Dashboard Frontend