# Integration Test Pipeline — Design ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ Kubernetes Job: inttest-runner │ │ │ │ Stage 1: Create namespace stonks-inttest-{id} │ │ Stage 2: Deploy infra (postgres, redis, minio) │ │ Stage 3: Run migrations + seed data │ │ Stage 4: Deploy services (query-api, registry, risk, trading) │ │ Stage 5: Wait for readiness │ │ Stage 6: Run API integration tests (pytest) │ │ Stage 7: Run frontend render tests (vitest + fetch) │ │ Stage 8: Collect profiling data │ │ Stage 9: Teardown namespace │ └─────────────────────────────────────────────────────────────────┘ ``` ## Implementation Approach ### Option A: Kubernetes Job with Python test runner (chosen) A single Kubernetes Job that: 1. Creates ephemeral infra via `kubectl apply` (postgres, redis, minio manifests) 2. Runs a Python seed script directly against the DB 3. Deploys service pods pointing at the ephemeral infra 4. Runs pytest-based integration tests against the live services 5. Collects timing metrics 6. Tears down everything ### Why not Helm? The sandbox doesn't need Helm's complexity. Plain manifests are simpler, faster, and easier to debug. The infra is ephemeral — no upgrades, no rollbacks. ## Components ### 1. Sandbox Manifests (`infra/inttest/`) ``` infra/inttest/ ├── namespace.yaml # Namespace template ├── postgres.yaml # PostgreSQL 16 StatefulSet (no PV) ├── redis.yaml # Redis 7 Deployment ├── minio.yaml # MinIO Deployment + bucket init Job ├── services.yaml # All 4 API services (query-api, registry, risk, trading) └── runner.yaml # The test runner Job itself ``` Each manifest uses `${NAMESPACE}` placeholder, substituted at runtime. ### 2. Seed Script (`tests/integration/seed_sandbox.py`) Pure SQL + MinIO operations. No external API calls. Inserts: - Companies, sources, aliases, competitor relationships - Documents with intelligence records - Trend windows with evidence and projections - Recommendations with evidence citations - Orders, positions, trading decisions - Global events with macro impacts - AI agents with variants and performance logs - Trading engine config, portfolio snapshots - MinIO objects (normalized text files) All UUIDs are deterministic (hardcoded) for reproducible assertions. ### 3. Integration Tests (`tests/integration/test_api_endpoints.py`) pytest-based tests that call every API endpoint the frontend depends on: ```python class TestQueryAPI: async def test_companies_list(self): resp = await client.get("/api/companies") assert resp.status_code == 200 data = resp.json() assert len(data) >= 5 assert all("ticker" in c for c in data) async def test_trend_detail(self): resp = await client.get(f"/api/trends/{SEED_TREND_ID}") assert resp.status_code == 200 data = resp.json() assert data["trend_direction"] in ("bullish", "bearish", "mixed") assert 0 <= data["confidence"] <= 1 # ... 40+ test functions ``` ### 4. Frontend Render Tests (`tests/integration/test_frontend_renders.py`) Uses the live sandbox APIs (not MSW mocks) to verify each page's data dependencies: ```python class TestFrontendDataDeps: """Verify every API call each frontend page makes returns valid data.""" async def test_home_page_deps(self): # Home page calls: companies, pipeline health, ingestion summary, recommendations for endpoint in ["/api/companies", "/api/ops/pipeline/health", "/api/ops/ingestion/summary"]: resp = await client.get(endpoint) assert resp.status_code == 200 async def test_company_detail_deps(self): # CompanyDetail calls 12 different endpoints company_id = SEED_COMPANY_IDS["AAPL"] for endpoint in [ f"/api/companies/{company_id}", f"/api/companies/{company_id}/sources", f"/api/companies/AAPL/macro-impacts", f"/api/companies/{company_id}/competitors", ]: resp = await client.get(endpoint) assert resp.status_code == 200 ``` ### 5. Profiler (`tests/integration/profiler.py`) Wraps each test with timing: - Records wall-clock time per API call - Computes P50/P95/P99 across all calls - Outputs a summary table at the end - Flags any endpoint > 500ms as "slow" ### 6. Runner Script (`infra/inttest/run_pipeline.sh`) Standalone orchestration script with a well-defined CLI contract so any CI/CD system (or a human) can invoke it. The future CI/CD pipeline spec will call this script as a stage. **CLI interface:** ``` Usage: bash infra/inttest/run_pipeline.sh [OPTIONS] Options: --image-tag TAG Docker image tag to deploy (default: latest) --namespace NAME Override namespace name (default: stonks-inttest-) --skip-teardown Leave namespace running after tests (for debugging) --results-file PATH Path for JSON results output (default: inttest-results.json) Exit codes: 0 All tests passed 1 One or more test failures 2 Infrastructure setup failure (postgres/redis/minio/services didn't start) ``` **JSON result contract** (`inttest-results.json`): ```json { "run_id": "stonks-inttest-1705312800", "image_tag": "abc123", "started_at": "2025-01-15T12:00:00Z", "completed_at": "2025-01-15T12:07:30Z", "exit_code": 0, "stages": { "infra_deploy": {"duration_s": 45.2, "status": "ok"}, "seed_data": {"duration_s": 8.1, "status": "ok"}, "service_deploy": {"duration_s": 32.5, "status": "ok"}, "integration_tests": {"duration_s": 28.3, "status": "ok"}, "teardown": {"duration_s": 5.0, "status": "ok"} }, "tests": { "total": 41, "passed": 41, "failed": 0, "errors": 0 }, "profiling": { "endpoints": { "/api/companies": {"p50_ms": 12, "p95_ms": 25, "p99_ms": 45}, ... }, "slow_endpoints": [] } } ``` This contract is designed so the future CI/CD pipeline can: 1. Parse `exit_code` to decide whether to promote to the next stage 2. Parse `profiling.slow_endpoints` to flag performance regressions 3. Archive the full JSON as a build artifact 4. Display `tests.passed`/`tests.failed` in a dashboard ```bash #!/bin/bash set -euo pipefail # Parse CLI args IMAGE_TAG="latest" NAMESPACE="stonks-inttest-$(date +%s)" SKIP_TEARDOWN=false RESULTS_FILE="inttest-results.json" while [[ $# -gt 0 ]]; do case $1 in --image-tag) IMAGE_TAG="$2"; shift 2 ;; --namespace) NAMESPACE="$2"; shift 2 ;; --skip-teardown) SKIP_TEARDOWN=true; shift ;; --results-file) RESULTS_FILE="$2"; shift 2 ;; *) echo "Unknown option: $1"; exit 2 ;; esac done # Cleanup function (always runs, even on failure) cleanup() { if [ "$SKIP_TEARDOWN" = false ]; then kubectl delete namespace "$NAMESPACE" --wait=false 2>/dev/null || true fi } trap cleanup EXIT # Stage 1: Create namespace kubectl create namespace "$NAMESPACE" # Stage 2: Deploy infra kubectl create configmap postgres-migrations --from-file=infra/migrations/ -n "$NAMESPACE" export NAMESPACE envsubst < infra/inttest/postgres.yaml | kubectl apply -n "$NAMESPACE" -f - envsubst < infra/inttest/redis.yaml | kubectl apply -n "$NAMESPACE" -f - envsubst < infra/inttest/minio.yaml | kubectl apply -n "$NAMESPACE" -f - kubectl wait --for=condition=ready pod -l app=postgres -n "$NAMESPACE" --timeout=120s kubectl wait --for=condition=ready pod -l app=redis -n "$NAMESPACE" --timeout=60s kubectl wait --for=condition=ready pod -l app=minio -n "$NAMESPACE" --timeout=60s # Stage 3: Seed data (run from a pod with DB access) # ... seed runner pod ... # Stage 4: Deploy services (using specified image tag) envsubst < infra/inttest/services.yaml | sed "s/:latest/:${IMAGE_TAG}/g" | kubectl apply -n "$NAMESPACE" -f - kubectl wait --for=condition=ready pod -l tier=api -n "$NAMESPACE" --timeout=120s # Stage 5: Run integration tests envsubst < infra/inttest/runner.yaml | sed "s/:latest/:${IMAGE_TAG}/g" | kubectl apply -n "$NAMESPACE" -f - kubectl wait --for=condition=complete job/inttest-runner -n "$NAMESPACE" --timeout=600s # Stage 6: Collect results kubectl logs job/inttest-runner -n "$NAMESPACE" > "$RESULTS_FILE" # Stage 7: Teardown (handled by trap) ``` ## Profiling Strategy ### What to measure 1. **Seed insertion time** — how long to populate all tables 2. **Service startup time** — time from pod creation to readiness 3. **API response times** — per-endpoint P50/P95/P99 4. **Memory usage** — `kubectl top pods` snapshot during tests ### Performance targets | Metric | Target | Action if exceeded | |--------|--------|--------------------| | Seed insertion | < 30s | Batch INSERT optimization | | Service startup | < 30s each | Reduce import time, lazy loading | | API P95 | < 200ms | Query optimization, indexes | | API P99 | < 500ms | Connection pooling, caching | | Total pipeline | < 10 min | Parallelize stages | ### Optimization opportunities to discover - Slow SQL queries (missing indexes, N+1 patterns) - Heavy service startup (import chains) - Inefficient aggregation math - Unnecessary serialization overhead - Connection pool sizing ## Data Flow ``` Seed Script ├── PostgreSQL: companies, documents, trends, recommendations, orders, ... ├── MinIO: normalized text files, audit artifacts └── Redis: (empty — no queue state needed for API tests) Integration Tests ├── Query API ← PostgreSQL (read-only queries) ├── Symbol Registry ← PostgreSQL (CRUD operations) ├── Risk Engine ← PostgreSQL (evaluation + approvals) └── Trading Engine ← PostgreSQL + Redis (status, decisions, backtest) ``` ## Namespace Lifecycle ``` CREATE namespace → Deploy postgres, redis, minio → Wait for healthy → Run migrations (init container) → Run seed script → Deploy services → Wait for ready → Run tests → Collect results → DELETE namespace (always, even on failure) ``` ## Integration Contract for Future CI/CD Pipeline This spec produces a standalone runner (`infra/inttest/run_pipeline.sh`) with a well-defined contract. A future spec ("CI/CD Deployment Pipeline") will consume it as one stage in a larger pipeline: ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Future CI/CD Pipeline (separate spec) │ │ │ │ 1. Git push → webhook to self-hosted runner on gremlin nodes │ │ 2. Lint + Unit Tests (ruff, pytest, vitest) │ │ 3. Docker Build → push to GHCR (self-hosted, no GH Actions compute) │ │ 4. ┌──────────────────────────────────────────────────────────┐ │ │ │ Integration Tests (THIS SPEC) │ │ │ │ bash infra/inttest/run_pipeline.sh --image-tag $SHA │ │ │ │ → reads inttest-results.json │ │ │ │ → exit code 0 = promote, 1 = block │ │ │ └──────────────────────────────────────────────────────────┘ │ │ 5. Promote to beta namespace (if tests pass) │ │ 6. Promote to paper namespace (manual gate or auto) │ │ 7. Promote to live namespace (market-hours blocker + break-glass) │ │ │ │ Each stage has enable/disable toggle. │ │ Promotions blocked during market hours (9:30–16:00 ET) unless │ │ break-glass is activated. │ └─────────────────────────────────────────────────────────────────────────┘ ``` **What this spec provides to the future pipeline:** - `infra/inttest/run_pipeline.sh` — callable with `--image-tag` to test any build - `inttest-results.json` — machine-readable results for promotion decisions - Exit codes for pass/fail gating - `--skip-teardown` for debugging failed runs - All K8s manifests in `infra/inttest/` for sandbox lifecycle - Deterministic seed data and comprehensive API test coverage