admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
cfe4a92987a90cf53e7056d4e7ebee29b88d56ab
stonks-oracle/.kiro/specs/beta-api-test-suite/requirements.md
T
Celes Renata 898f89926d feat: beta API integration test suite — 85 new tests across 6 modules 
Extends integration test coverage from 108 to 193 tests for the beta gate.

New test modules:
- test_query_api_extended.py (33 tests): documents, evidence, macro/competitive, ops/admin, agents, analytics
- test_registry_write_paths.py (16 tests): write paths, validation, duplicates, competitor/exposure CRUD
- test_risk_approval_lifecycle.py (8 tests): evaluation edge cases, full approval lifecycle
- test_trading_extended.py (12 tests): config round-trips, decision filtering, override validation
- test_cross_service_roundtrip.py (4 tests): cross-service data consistency
- test_error_handling.py (12 tests): 404s, 422s, empty states, health checks

Seed script extended with watchlists, approvals, lockouts, notifications,
ingestion runs, saved queries, and daily risk snapshots.
2026-04-20 02:34:19 +00:00

21 KiB
Raw Blame History

Requirements Document

Introduction

Comprehensive API integration test suite for the Stonks Oracle beta gate. The suite validates every HTTP endpoint across all four API services (query-api, symbol-registry, risk-engine, trading-engine) that can be tested without external broker or news API dependencies. Tests run against a seeded PostgreSQL database with deterministic data, exercising read paths, write paths, edge cases, empty-state behavior, error handling, pagination/filtering, and round-trip fidelity. The suite extends the existing ~41 integration tests to provide full endpoint coverage suitable for blocking promotion from beta to paper-trading.

Glossary

  • Test_Suite: The collection of pytest-asyncio integration test modules under tests/integration/ that validate API behavior against live sandbox services
  • Seed_Script: The tests/integration/seed_sandbox.py module that populates the database with deterministic UUIDs, timestamps, and relationships for reproducible assertions
  • Query_API: The FastAPI service at services/api/app.py exposing ~50 read/write endpoints for analytics, evidence drill-down, admin controls, macro/competitive layers, agent management, and operational dashboards
  • Symbol_Registry: The FastAPI service at services/symbol_registry/app.py exposing CRUD endpoints for companies, aliases, watchlists, sources, competitor relationships, and exposure profiles
  • Risk_Engine: The FastAPI service at services/risk/app.py exposing order risk evaluation, approval workflow, and approval expiry endpoints
  • Trading_Engine: The FastAPI service at services/trading/app.py exposing engine control, configuration, decisions, metrics, notifications, backtest, and override order endpoints
  • Beta_Gate: The promotion pipeline (infra/inttest/promote.sh) that deploys to beta namespace, seeds data, runs the Test_Suite, and promotes to paper-trading only if all tests pass
  • Sandbox: The ephemeral Kubernetes namespace with PostgreSQL, Redis, and MinIO where integration tests execute
  • Deterministic_UUID: A hardcoded UUID in the Seed_Script that enables exact equality assertions in tests
  • Round_Trip: A test pattern where data is written via a POST/PUT endpoint and then read back via a GET endpoint to verify fidelity

Requirements

Requirement 1: Expanded Seed Data for Full Coverage

User Story: As a test engineer, I want the seed script to populate all database tables with enough variety to exercise every API code path, so that tests can validate filtering, pagination, edge cases, and empty-state behavior.

Acceptance Criteria

  1. THE Seed_Script SHALL insert at least 2 watchlists with at least 3 watchlist members across the watchlists
  2. THE Seed_Script SHALL insert at least 2 operator approval records: one with status "pending" and one with status "approved"
  3. THE Seed_Script SHALL insert at least 2 symbol lockout records: one expired and one still active
  4. THE Seed_Script SHALL insert at least 1 notification record with delivery_status "delivered"
  5. THE Seed_Script SHALL insert at least 2 ingestion run records with different statuses ("completed" and "failed")
  6. THE Seed_Script SHALL insert at least 1 saved query record
  7. THE Seed_Script SHALL insert at least 1 daily risk snapshot record
  8. THE Seed_Script SHALL export all new Deterministic_UUIDs as named constants and include them in the SEED lookup dictionaries for test assertion use
  9. WHEN the Seed_Script is run against an already-seeded database, THE Seed_Script SHALL be idempotent and not raise errors due to duplicate key violations

Requirement 2: Query API — Document and Evidence Endpoints

User Story: As a test engineer, I want comprehensive tests for the Query API document timeline, evidence drill-down, and trend projection endpoints, so that schema drift between services is caught before promotion.

Acceptance Criteria

  1. WHEN a GET request is made to /api/documents with ticker filter parameter, THE Query_API SHALL return only documents mentioning that ticker
  2. WHEN a GET request is made to /api/documents with doc_type filter parameter, THE Query_API SHALL return only documents of that type
  3. WHEN a GET request is made to /api/documents/{id} for a seeded document, THE Query_API SHALL return the document with intelligence, company_mentions, and title fields populated
  4. WHEN a GET request is made to /api/documents/{id} with a non-existent UUID, THE Query_API SHALL return HTTP 404
  5. WHEN a GET request is made to /api/recommendations/{id}/evidence for a seeded recommendation, THE Query_API SHALL return an evidence drill-down with document and intelligence references
  6. WHEN a GET request is made to /api/trends/{id}/evidence for a seeded trend, THE Query_API SHALL return an evidence drill-down with supporting and opposing document references
  7. WHEN a GET request is made to /api/trends/{id}/projection for a seeded trend with a projection, THE Query_API SHALL return the trend projection with projected_direction, projected_strength, projected_confidence, and macro_contribution_pct fields

Requirement 3: Query API — Macro and Competitive Layer Endpoints

User Story: As a test engineer, I want tests for the macro event, competitive signal, and pattern endpoints, so that the three-layer signal aggregation engine is validated end-to-end.

Acceptance Criteria

  1. WHEN a GET request is made to /api/macro/status, THE Query_API SHALL return the macro layer toggle status with enabled field
  2. WHEN a GET request is made to /api/macro/events, THE Query_API SHALL return at least 2 seeded global events with event_types, severity, affected_regions, and summary fields
  3. WHEN a GET request is made to /api/macro/events/{id} for a seeded event, THE Query_API SHALL return the event detail with macro_impacts containing per-company impact records
  4. WHEN a GET request is made to /api/macro/impacts/{ticker} for "AAPL", THE Query_API SHALL return at least 1 macro impact record with macro_impact_score, impact_direction, and contributing_factors
  5. WHEN a GET request is made to /api/competitive/status, THE Query_API SHALL return the competitive layer toggle status with enabled field
  6. WHEN a GET request is made to /api/competitive/signals/{ticker} for "AAPL", THE Query_API SHALL return competitive signal records with source_ticker, target_ticker, signal_direction, and signal_strength fields
  7. WHEN a GET request is made to /api/competitive/patterns/{ticker} for "AAPL", THE Query_API SHALL return historical pattern data (may be empty if no patterns computed)

Requirement 4: Query API — Operational and Admin Endpoints

User Story: As a test engineer, I want tests for the operational dashboard, source management, trading config, approval workflow, and lockout endpoints, so that admin functionality is validated before promotion.

Acceptance Criteria

  1. WHEN a GET request is made to /api/ops/pipeline/health, THE Query_API SHALL return pipeline health with document_stages, parsing, extraction, aggregation, and queue_depths fields
  2. WHEN a GET request is made to /api/ops/ingestion/summary, THE Query_API SHALL return ingestion statistics with total_runs and by_source_type fields
  3. WHEN a GET request is made to /api/ops/sources/coverage-gaps, THE Query_API SHALL return gap analysis with missing_source_types and stale_sources lists
  4. WHEN a PUT request is made to /api/sources/{id}/toggle?active=false for a seeded source, THE Query_API SHALL return HTTP 200 and the source active status SHALL be updated
  5. WHEN a GET request is made to /api/trading/config, THE Query_API SHALL return the risk configuration with trading_mode and config fields
  6. WHEN a GET request is made to /api/approvals/pending, THE Query_API SHALL return a list containing the seeded pending approval record
  7. WHEN a GET request is made to /api/lockouts/active, THE Query_API SHALL return a list containing the seeded active lockout record
  8. WHEN a POST request is made to /api/lockouts with a valid ticker and expiry, THE Query_API SHALL create a lockout and return HTTP 200 with the lockout details
  9. WHEN a DELETE request is made to /api/lockouts/{id} for the created lockout, THE Query_API SHALL remove the lockout and return HTTP 200

Requirement 5: Query API — Agent and Variant Management Endpoints

User Story: As a test engineer, I want tests for the AI agent CRUD, variant lifecycle, and performance endpoints, so that the agent management system is validated.

Acceptance Criteria

  1. WHEN a GET request is made to /api/agents, THE Query_API SHALL return at least 3 seeded agents with id, name, slug, model_name, and active fields
  2. WHEN a GET request is made to /api/agents/{id} for a seeded agent, THE Query_API SHALL return the agent detail with system_prompt, temperature, and max_tokens fields
  3. WHEN a POST request is made to /api/agents with a valid agent body, THE Query_API SHALL create the agent and return HTTP 201 with the new agent including a generated slug
  4. WHEN a PUT request is made to /api/agents/{id} with updated fields, THE Query_API SHALL update the agent and return the modified record
  5. WHEN a GET request is made to /api/agents/{id}/variants for a seeded agent, THE Query_API SHALL return at least 1 variant with variant_name, model_name, and is_active fields
  6. WHEN a POST request is made to /api/agents/{id}/variants with a valid variant body, THE Query_API SHALL create the variant and return HTTP 201
  7. WHEN a POST request is made to /api/agents/{id}/variants/{vid}/activate, THE Query_API SHALL set the variant as active and deactivate any previously active variant for that agent
  8. WHEN a GET request is made to /api/agents/{id}/performance for a seeded agent, THE Query_API SHALL return performance metrics derived from the agent performance log
  9. WHEN a GET request is made to /api/agents/{id}/variants/{vid}/performance for a seeded variant, THE Query_API SHALL return variant-level performance metrics

Requirement 6: Symbol Registry — Write Path and Edge Case Endpoints

User Story: As a test engineer, I want tests for the Symbol Registry write paths (create, update, delete) and edge cases (duplicates, not-found, validation), so that data integrity is validated.

Acceptance Criteria

  1. WHEN a POST request is made to /companies with a duplicate ticker and exchange, THE Symbol_Registry SHALL return HTTP 409
  2. WHEN a GET request is made to /companies/{id} with a non-existent UUID, THE Symbol_Registry SHALL return HTTP 404
  3. WHEN a POST request is made to /companies/{id}/aliases with a valid alias, THE Symbol_Registry SHALL create the alias and return HTTP 201 with id, alias, and alias_type
  4. WHEN a POST request is made to /companies/{id}/sources with a valid source body, THE Symbol_Registry SHALL create the source and return HTTP 201 with id, source_type, and source_name
  5. WHEN a POST request is made to /companies/{id}/sources with an invalid source_type, THE Symbol_Registry SHALL return HTTP 422
  6. WHEN a POST request is made to /watchlists with a valid name, THE Symbol_Registry SHALL create the watchlist and return HTTP 201
  7. WHEN a POST request is made to /watchlists/{id}/members/{company_id} for a seeded company, THE Symbol_Registry SHALL add the member and return HTTP 201
  8. WHEN a GET request is made to /watchlists/{id}/members after adding members, THE Symbol_Registry SHALL return the member companies with ticker and legal_name fields
  9. WHEN a POST request is made to /watchlists with a duplicate name, THE Symbol_Registry SHALL return HTTP 409

Requirement 7: Symbol Registry — Competitor and Exposure Write Paths

User Story: As a test engineer, I want tests for competitor relationship CRUD and exposure profile upsert, so that the competitive intelligence data layer is validated.

Acceptance Criteria

  1. WHEN a POST request is made to /companies/{id}/competitors with a valid competitor body, THE Symbol_Registry SHALL create the relationship and return HTTP 201 with relationship_type, strength, and bidirectional fields
  2. WHEN a POST request is made to /companies/{id}/competitors where company_id equals company_b_id, THE Symbol_Registry SHALL return HTTP 400 with a self-referencing error
  3. WHEN a PUT request is made to /companies/{id}/competitors/{rel_id} with updated strength, THE Symbol_Registry SHALL update the relationship and return the modified record
  4. WHEN a DELETE request is made to /companies/{id}/competitors/{rel_id}, THE Symbol_Registry SHALL soft-delete the relationship (set active=false) and return HTTP 200
  5. WHEN a PUT request is made to /companies/{id}/exposure with a valid exposure profile, THE Symbol_Registry SHALL create or update the profile and return the new version with incremented version number
  6. WHEN a GET request is made to /companies/{id}/exposure/history for a company with multiple profile versions, THE Symbol_Registry SHALL return all versions ordered by version descending
  7. WHEN a GET request is made to /companies/{id}/exposure for a company with no exposure profile, THE Symbol_Registry SHALL return HTTP 404

Requirement 8: Risk Engine — Evaluation Edge Cases and Approval Workflow

User Story: As a test engineer, I want tests for risk evaluation edge cases, the full approval lifecycle, and approval expiry, so that the risk gate is validated.

Acceptance Criteria

  1. WHEN a POST request is made to /evaluate with a minimal order (only ticker), THE Risk_Engine SHALL return a valid evaluation with evaluation_id, eligible, and rejection_reasons
  2. WHEN a POST request is made to /evaluate with an order exceeding the absolute position cap, THE Risk_Engine SHALL return eligible: false with at least one rejection reason
  3. WHEN a POST request is made to /evaluate with a custom config overriding risk parameters, THE Risk_Engine SHALL use the provided config for evaluation
  4. WHEN a GET request is made to /approvals/pending, THE Risk_Engine SHALL return a list of pending approval records from the seeded data
  5. WHEN a GET request is made to /approvals/{id} for a seeded pending approval, THE Risk_Engine SHALL return the approval detail with ticker, side, quantity, status, and expires_at fields
  6. WHEN a POST request is made to /approvals/{id}/review with approved: true, THE Risk_Engine SHALL update the approval status to "approved" and return the new status
  7. WHEN a POST request is made to /approvals/{id}/review for a non-existent approval, THE Risk_Engine SHALL return HTTP 404
  8. WHEN a POST request is made to /approvals/expire, THE Risk_Engine SHALL expire stale pending approvals and return the count of expired records

Requirement 9: Trading Engine — Configuration, Metrics, and Notification Endpoints

User Story: As a test engineer, I want tests for trading engine configuration round-trips, metrics consistency, notification config CRUD, and notification history, so that the autonomous trading control plane is validated.

Acceptance Criteria

  1. WHEN a PUT request is made to /api/trading/config with risk_tier: "aggressive" followed by a GET to /api/trading/status, THE Trading_Engine SHALL reflect the updated risk tier in the status response (Round_Trip)
  2. WHEN a POST request is made to /api/trading/pause followed by a GET to /api/trading/status, THE Trading_Engine SHALL report paused: true in the status response
  3. WHEN a POST request is made to /api/trading/resume followed by a GET to /api/trading/status, THE Trading_Engine SHALL report paused: false in the status response
  4. WHEN a GET request is made to /api/trading/metrics, THE Trading_Engine SHALL return all numeric portfolio metrics where total_portfolio_value approximately equals active_pool + reserve_pool + unrealized_pnl
  5. WHEN a GET request is made to /api/trading/metrics/history, THE Trading_Engine SHALL return a list of portfolio snapshots with portfolio_value and snapshot_date fields
  6. WHEN a PUT request is made to /api/trading/notifications/config with sms_enabled: true, THE Trading_Engine SHALL update the notification config and a subsequent GET SHALL reflect the change (Round_Trip)
  7. WHEN a GET request is made to /api/trading/notifications/history, THE Trading_Engine SHALL return a list of notification records (at least 1 from seed data)
  8. WHEN a GET request is made to /api/trading/decisions with ticker filter, THE Trading_Engine SHALL return only decisions for that ticker

Requirement 10: Trading Engine — Decision Filtering, Backtest, and Override Validation

User Story: As a test engineer, I want tests for decision audit trail filtering, backtest submission, and override order validation, so that the trading engine's write paths and input validation are verified.

Acceptance Criteria

  1. WHEN a GET request is made to /api/trading/decisions with limit=1, THE Trading_Engine SHALL return at most 1 decision record
  2. WHEN a GET request is made to /api/trading/decisions with decision=act, THE Trading_Engine SHALL return only decisions with decision: "act"
  3. WHEN a POST request is made to /api/trading/override/order with an invalid ticker (lowercase or special characters), THE Trading_Engine SHALL return HTTP 422
  4. WHEN a POST request is made to /api/trading/override/order with quantity of 0 or negative, THE Trading_Engine SHALL return HTTP 422
  5. WHEN a POST request is made to /api/trading/override/order with a valid market order, THE Trading_Engine SHALL return HTTP 202 with job_id, status: "queued", ticker, side, and quantity fields, or a structured error if Redis is unavailable
  6. IF the Trading_Engine returns HTTP 503 for an override order due to Redis unavailability, THEN THE Trading_Engine SHALL return a JSON error body (not an unhandled exception)

Requirement 11: Query API — Analytics, Saved Queries, and Schema Introspection

User Story: As a test engineer, I want tests for the analytics query engine, saved query CRUD, and schema introspection endpoints, so that the data exploration features are validated.

Acceptance Criteria

  1. WHEN a GET request is made to /api/analytics/schema, THE Query_API SHALL return the Trino schema information (or a structured error if Trino is unavailable)
  2. WHEN a GET request is made to /api/pg/schema, THE Query_API SHALL return the PostgreSQL schema with table and column information
  3. WHEN a GET request is made to /api/saved-queries, THE Query_API SHALL return a list of saved queries (at least 1 from seed data or migrations)
  4. WHEN a POST request is made to /api/saved-queries with a valid query body, THE Query_API SHALL create the saved query and return the record
  5. WHEN a DELETE request is made to /api/saved-queries/{id} for the created query, THE Query_API SHALL delete the query and return HTTP 200
  6. WHEN a POST request is made to /api/pg/query with a valid read-only SQL query, THE Query_API SHALL execute the query and return results

Requirement 12: Cross-Service Round-Trip and Contract Validation

User Story: As a test engineer, I want round-trip tests that write data through one service and read it through another, so that cross-service data consistency is validated.

Acceptance Criteria

  1. WHEN a company is created via POST to Symbol_Registry /companies and then queried via GET to Query_API /api/companies, THE company SHALL appear in the Query_API response with matching ticker and legal_name
  2. WHEN an exposure profile is created via PUT to Symbol_Registry /companies/{id}/exposure and then queried via GET to Symbol_Registry /companies/{id}/exposure, THE profile SHALL match the submitted data with geographic_revenue_mix, supply_chain_regions, and market_position_tier fields preserved
  3. WHEN a competitor relationship is created via POST to Symbol_Registry /companies/{id}/competitors and then queried from both sides, THE relationship SHALL be visible from both company_a and company_b perspectives
  4. WHEN a risk evaluation is performed via POST to Risk_Engine /evaluate and then the same ticker's recommendation is queried via Query_API /api/recommendations, THE recommendation SHALL include a risk_evaluation field with matching evaluation structure

Requirement 13: Error Handling and Empty-State Behavior

User Story: As a test engineer, I want tests that verify all endpoints return structured JSON errors (not HTML or stack traces) and handle empty-state gracefully, so that the frontend never receives unexpected response formats.

Acceptance Criteria

  1. WHEN a GET request is made to any list endpoint with no matching data, THE service SHALL return HTTP 200 with an empty list (not HTTP 404 or an error)
  2. WHEN a GET request is made to any detail endpoint with a non-existent UUID, THE service SHALL return HTTP 404 with a JSON body containing an error message
  3. WHEN a POST request is made with an invalid JSON body (missing required fields), THE service SHALL return HTTP 422 with a JSON body containing validation error details
  4. WHEN a GET request is made to /api/trends/{id}/projection for a trend with no projection, THE Query_API SHALL return HTTP 404 with a JSON error (not HTTP 500)
  5. WHEN a GET request is made to /api/macro/impacts/{ticker} for a ticker with no macro impacts, THE Query_API SHALL return HTTP 200 with an empty list
  6. THE Test_Suite SHALL verify that all health endpoints (/health) across all four services return {"status": "ok"} with HTTP 200
Reference in New Issue View Git Blame Copy Permalink