feat: beta API integration test suite — 85 new tests across 6 modules
Extends integration test coverage from 108 to 193 tests for the beta gate. New test modules: - test_query_api_extended.py (33 tests): documents, evidence, macro/competitive, ops/admin, agents, analytics - test_registry_write_paths.py (16 tests): write paths, validation, duplicates, competitor/exposure CRUD - test_risk_approval_lifecycle.py (8 tests): evaluation edge cases, full approval lifecycle - test_trading_extended.py (12 tests): config round-trips, decision filtering, override validation - test_cross_service_roundtrip.py (4 tests): cross-service data consistency - test_error_handling.py (12 tests): 404s, 422s, empty states, health checks Seed script extended with watchlists, approvals, lockouts, notifications, ingestion runs, saved queries, and daily risk snapshots.
This commit is contained in:
Celes Renata
@@ -0,0 +1,216 @@
|
||||
# Design Document: Beta API Test Suite
|
||||
|
||||
## Overview
|
||||
|
||||
This design describes a comprehensive integration test suite that validates every HTTP endpoint across the four Stonks Oracle API services (query-api, symbol-registry, risk-engine, trading-engine) that can be tested without external broker or news API dependencies. The suite extends the existing ~56 integration tests to achieve full endpoint coverage, serving as the beta gate that blocks promotion to paper-trading if any test fails.
|
||||
|
||||
The test suite operates against an ephemeral Kubernetes sandbox with seeded PostgreSQL data, using deterministic UUIDs for exact equality assertions. It validates read paths, write paths, round-trips, edge cases, error handling, pagination/filtering, and empty-state behavior.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "K8s Sandbox Namespace"
|
||||
PG[(PostgreSQL)]
|
||||
Redis[(Redis)]
|
||||
MinIO[(MinIO)]
|
||||
QA[query-api :8000]
|
||||
SR[symbol-registry :8000]
|
||||
RE[risk-engine :8000]
|
||||
TE[trading-engine :8000]
|
||||
end
|
||||
|
||||
subgraph "Test Runner Job"
|
||||
SEED[seed_sandbox.py]
|
||||
CONF[conftest.py]
|
||||
T1[test_query_api.py]
|
||||
T2[test_registry_api.py]
|
||||
T3[test_risk_api.py]
|
||||
T4[test_trading_api.py]
|
||||
T5[test_signal_flow.py]
|
||||
T6[test_frontend_data_deps.py]
|
||||
T7[test_query_api_extended.py]
|
||||
T8[test_registry_write_paths.py]
|
||||
T9[test_risk_approval_lifecycle.py]
|
||||
T10[test_trading_extended.py]
|
||||
T11[test_cross_service_roundtrip.py]
|
||||
T12[test_error_handling.py]
|
||||
end
|
||||
|
||||
SEED -->|INSERT| PG
|
||||
CONF -->|httpx.AsyncClient| QA
|
||||
CONF -->|httpx.AsyncClient| SR
|
||||
CONF -->|httpx.AsyncClient| RE
|
||||
CONF -->|httpx.AsyncClient| TE
|
||||
QA --> PG
|
||||
SR --> PG
|
||||
RE --> PG
|
||||
TE --> PG
|
||||
TE --> Redis
|
||||
```
|
||||
|
||||
The architecture preserves the existing test infrastructure:
|
||||
- **Runner Job**: K8s Job (`infra/inttest/runner.yaml`) executes pytest in the sandbox namespace
|
||||
- **Seed Script**: `tests/integration/seed_sandbox.py` populates deterministic data before tests run
|
||||
- **Fixtures**: `tests/integration/conftest.py` provides `ProfiledAsyncClient` wrappers per service
|
||||
- **New test files** extend coverage without modifying existing passing tests
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Extended Seed Script (`seed_sandbox.py`)
|
||||
|
||||
New seed functions added to populate tables required by untested endpoints:
|
||||
|
||||
| Table | New Records | Purpose |
|
||||
|-------|-------------|---------|
|
||||
| `watchlists` | 2 watchlists | Watchlist CRUD tests |
|
||||
| `watchlist_members` | 3 members | Watchlist membership tests |
|
||||
| `operator_approvals` | 2 (pending + approved) | Approval lifecycle tests |
|
||||
| `symbol_lockouts` | 2 (active + expired) | Lockout CRUD tests |
|
||||
| `notifications` | 1 delivered | Notification history tests |
|
||||
| `ingestion_runs` | 2 (completed + failed) | Ingestion summary tests |
|
||||
| `saved_queries` | 1 | Saved query CRUD tests |
|
||||
| `daily_risk_snapshots` | 1 | Risk snapshot tests |
|
||||
|
||||
All new records use deterministic UUIDs exported as named constants.
|
||||
|
||||
### 2. New Test Modules
|
||||
|
||||
| Module | Service | Coverage |
|
||||
|--------|---------|----------|
|
||||
| `test_query_api_extended.py` | query-api | Documents filtering, evidence drill-down, trend projections, macro/competitive endpoints, ops dashboard, agents CRUD, analytics |
|
||||
| `test_registry_write_paths.py` | symbol-registry | Alias/source/watchlist/competitor/exposure write paths, validation errors, duplicates |
|
||||
| `test_risk_approval_lifecycle.py` | risk-engine | Full approval lifecycle, evaluation edge cases, expiry |
|
||||
| `test_trading_extended.py` | trading-engine | Config round-trips, decision filtering, notification CRUD, override validation |
|
||||
| `test_cross_service_roundtrip.py` | cross-service | Write via one service, read via another |
|
||||
| `test_error_handling.py` | all services | 404s, 422s, empty lists, structured JSON errors |
|
||||
|
||||
### 3. Conftest Extensions
|
||||
|
||||
New fixtures added to `conftest.py`:
|
||||
- Export new seed IDs (watchlists, approvals, lockouts, notifications, ingestion runs, saved queries)
|
||||
- No changes to existing client fixtures
|
||||
|
||||
## Data Models
|
||||
|
||||
### New Seed ID Constants
|
||||
|
||||
```python
|
||||
# Watchlists
|
||||
WATCHLIST_01 = UUID("00000000-0000-4000-ac00-000000000001")
|
||||
WATCHLIST_02 = UUID("00000000-0000-4000-ac00-000000000002")
|
||||
|
||||
# Operator Approvals
|
||||
APPROVAL_PENDING = UUID("00000000-0000-4000-ad00-000000000001")
|
||||
APPROVAL_APPROVED = UUID("00000000-0000-4000-ad00-000000000002")
|
||||
|
||||
# Symbol Lockouts
|
||||
LOCKOUT_ACTIVE = UUID("00000000-0000-4000-ae00-000000000001")
|
||||
LOCKOUT_EXPIRED = UUID("00000000-0000-4000-ae00-000000000002")
|
||||
|
||||
# Notifications
|
||||
NOTIFICATION_01 = UUID("00000000-0000-4000-af00-000000000001")
|
||||
|
||||
# Ingestion Runs
|
||||
INGESTION_RUN_01 = UUID("00000000-0000-4000-b300-000000000001")
|
||||
INGESTION_RUN_02 = UUID("00000000-0000-4000-b300-000000000002")
|
||||
|
||||
# Saved Queries
|
||||
SAVED_QUERY_01 = UUID("00000000-0000-4000-b400-000000000001")
|
||||
|
||||
# Daily Risk Snapshot
|
||||
RISK_SNAPSHOT_01 = UUID("00000000-0000-4000-b500-000000000001")
|
||||
```
|
||||
|
||||
### Seed ID Export Dictionary Extension
|
||||
|
||||
```python
|
||||
SEED_WATCHLIST_IDS = {
|
||||
"WL_01": str(WATCHLIST_01),
|
||||
"WL_02": str(WATCHLIST_02),
|
||||
}
|
||||
SEED_APPROVAL_IDS = {
|
||||
"PENDING": str(APPROVAL_PENDING),
|
||||
"APPROVED": str(APPROVAL_APPROVED),
|
||||
}
|
||||
SEED_LOCKOUT_IDS = {
|
||||
"ACTIVE": str(LOCKOUT_ACTIVE),
|
||||
"EXPIRED": str(LOCKOUT_EXPIRED),
|
||||
}
|
||||
SEED_NOTIFICATION_IDS = {"NOTIF_01": str(NOTIFICATION_01)}
|
||||
SEED_INGESTION_RUN_IDS = {
|
||||
"RUN_01": str(INGESTION_RUN_01),
|
||||
"RUN_02": str(INGESTION_RUN_02),
|
||||
}
|
||||
SEED_SAVED_QUERY_IDS = {"SQ_01": str(SAVED_QUERY_01)}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The test suite validates error handling across all services:
|
||||
|
||||
1. **404 Not Found**: All detail endpoints return JSON `{"detail": "..."}` for non-existent UUIDs
|
||||
2. **422 Validation Error**: POST/PUT endpoints return structured validation errors for invalid bodies
|
||||
3. **409 Conflict**: Duplicate creation attempts return conflict errors
|
||||
4. **400 Bad Request**: Business logic violations (self-referencing competitors, etc.)
|
||||
5. **503 Service Unavailable**: Graceful degradation when Redis is unavailable (trading engine override)
|
||||
6. **Empty Lists**: List endpoints return `[]` with HTTP 200, never 404
|
||||
|
||||
Each error response is verified to be valid JSON (not HTML stack traces).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Approach: Integration Testing with Seeded Data
|
||||
|
||||
**Why PBT does not apply**: This is an integration test suite that validates HTTP API contracts against live services with a seeded database. The tests verify:
|
||||
- Correct HTTP status codes
|
||||
- Response schema conformance
|
||||
- Data consistency across services
|
||||
- Error handling behavior
|
||||
|
||||
These are integration tests with concrete, deterministic assertions — not pure functions with universal properties. The input space is fixed (seeded data), and behavior depends on external service state (PostgreSQL, Redis). Property-based testing would require mocking the entire service layer, defeating the purpose of integration testing.
|
||||
|
||||
### Test Organization
|
||||
|
||||
**Existing tests (preserved as-is)**:
|
||||
- `test_query_api.py` — 17 tests covering core query API reads
|
||||
- `test_registry_api.py` — 8 tests covering registry CRUD
|
||||
- `test_risk_api.py` — 4 tests covering risk evaluation and approvals
|
||||
- `test_trading_api.py` — 12 tests covering trading engine endpoints
|
||||
- `test_signal_flow.py` — 11 tests covering cross-service contracts
|
||||
- `test_frontend_data_deps.py` — 21 tests covering frontend page dependencies
|
||||
|
||||
**New tests (added by this spec)**:
|
||||
- `test_query_api_extended.py` — ~25 tests for uncovered query API endpoints
|
||||
- `test_registry_write_paths.py` — ~15 tests for registry write paths and edge cases
|
||||
- `test_risk_approval_lifecycle.py` — ~10 tests for approval workflow and evaluation edge cases
|
||||
- `test_trading_extended.py` — ~12 tests for trading config round-trips, filtering, notifications
|
||||
- `test_cross_service_roundtrip.py` — ~6 tests for cross-service data consistency
|
||||
- `test_error_handling.py` — ~10 tests for structured error responses
|
||||
|
||||
**Total target**: ~130+ integration tests (existing ~56 + ~75 new)
|
||||
|
||||
### Test Execution
|
||||
|
||||
- Framework: pytest + pytest-asyncio
|
||||
- HTTP client: httpx.AsyncClient wrapped in ProfiledAsyncClient
|
||||
- Timeout: 30s per request, 600s total Job deadline
|
||||
- Parallelism: Sequential within each file (shared state from writes), files can run in parallel
|
||||
- Idempotency: Seed script uses `ON CONFLICT DO NOTHING` for re-runnability
|
||||
|
||||
### Naming Convention
|
||||
|
||||
Test classes follow the pattern `Test{Service}{Feature}` with descriptive method names:
|
||||
```python
|
||||
class TestQueryAPIDocumentFiltering:
|
||||
async def test_filter_documents_by_ticker(self, query_client, seed_ids): ...
|
||||
async def test_filter_documents_by_doc_type(self, query_client, seed_ids): ...
|
||||
```
|
||||
|
||||
### Assertions
|
||||
|
||||
- Status code assertions first (`assert resp.status_code == 200`)
|
||||
- Schema assertions verify required fields exist
|
||||
- Value assertions use deterministic seed data for exact equality
|
||||
- Numeric assertions use approximate comparison where appropriate (portfolio math)
|
||||
- List assertions verify minimum counts from seed data
|
||||
Reference in New Issue
Block a user