feat: AI Agents management page with per-agent performance tracking
New Agents tab in the sidebar (Ops group) for viewing, editing, and
creating AI agent configurations:
Database (migration 026):
- ai_agents table: editable configs for each LLM agent (model, prompts,
temperature, tokens, retries). source='system' for built-in,
source='user' for custom. Seeds 3 system agents (Document Extractor,
Event Classifier, Thesis Rewriter) using WHERE NOT EXISTS to never
overwrite user edits across reinstalls.
- agent_performance_log table: per-invocation metrics (duration,
confidence, retries, tokens, errors) linked to agent config.
API endpoints:
- GET/POST /api/agents — list and create agents
- GET/PUT/DELETE /api/agents/{id} — view, edit, delete (system agents
can be edited but not deleted)
- GET /api/agents/{id}/performance — aggregated metrics (success rate,
avg/p95 latency, confidence, token usage)
- GET /api/agents/{id}/performance/history — hourly time series
Frontend:
- AgentsPage with sidebar list + detail panel
- Agent detail: config display, system prompt viewer, performance
dashboard with metrics cards and time-series chart
- Edit form: all config fields editable including system prompt,
model, temperature, tokens, retries
- Create form: new user-defined agents with auto-slug generation
- System agents show blue badge, user agents show green badge
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
-- AI Agent configurations: user-editable agent profiles.
|
||||
-- Seed rows have source='system' and are re-inserted on migration only if
|
||||
-- missing, so user edits (source='user') are never overwritten.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ai_agents (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name VARCHAR(100) NOT NULL UNIQUE,
|
||||
slug VARCHAR(100) NOT NULL UNIQUE,
|
||||
purpose TEXT NOT NULL DEFAULT '',
|
||||
model_provider VARCHAR(50) NOT NULL DEFAULT 'ollama',
|
||||
model_name VARCHAR(200) NOT NULL DEFAULT 'llama3.1:8b',
|
||||
system_prompt TEXT NOT NULL DEFAULT '',
|
||||
user_prompt_template TEXT NOT NULL DEFAULT '',
|
||||
prompt_version VARCHAR(100) NOT NULL DEFAULT '',
|
||||
schema_version VARCHAR(50) NOT NULL DEFAULT '1.0.0',
|
||||
temperature FLOAT DEFAULT 0.0,
|
||||
max_tokens INTEGER DEFAULT 32768,
|
||||
timeout_seconds INTEGER DEFAULT 120,
|
||||
max_retries INTEGER DEFAULT 2,
|
||||
active BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
source VARCHAR(20) NOT NULL DEFAULT 'system', -- 'system' or 'user'
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_agents_slug ON ai_agents(slug);
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_agents_active ON ai_agents(active);
|
||||
|
||||
-- Seed the three built-in agents (only if they don't already exist)
|
||||
INSERT INTO ai_agents (name, slug, purpose, model_provider, model_name, system_prompt, prompt_version, schema_version, source)
|
||||
SELECT * FROM (VALUES
|
||||
(
|
||||
'Document Intelligence Extractor',
|
||||
'document-extractor',
|
||||
'Extracts structured intelligence (sentiment, catalysts, impact scores, key facts, risks) from company news, SEC filings, earnings transcripts, and press releases.',
|
||||
'ollama',
|
||||
'llama3.1:8b',
|
||||
'You are a financial document analyst. Extract structured data as JSON. Return ONLY a single JSON object. No markdown fences, no explanation, no text before or after the JSON. Every field in the schema is required. Use "other" for catalyst_type if unsure. Keep evidence_spans short (under 20 words each). Keep key_facts to 3-5 items max.',
|
||||
'document-intel-v2',
|
||||
'2.0.0',
|
||||
'system'
|
||||
),
|
||||
(
|
||||
'Global Event Classifier',
|
||||
'event-classifier',
|
||||
'Classifies global/geopolitical news into structured macro events with impact type, severity, affected regions/sectors/commodities, and estimated duration.',
|
||||
'ollama',
|
||||
'llama3.1:8b',
|
||||
'Classify this global news article as a macro event. Fill every field. RULES: - Only extract facts EXPLICITLY stated in the article - Do NOT infer geopolitical implications not stated - Distinguish between announced policy and rumored policy - If severity is unclear, default to "low" - confidence: 0.0-1.0 your confidence in this classification',
|
||||
'event-classification-v1',
|
||||
'1.0.0',
|
||||
'system'
|
||||
),
|
||||
(
|
||||
'Thesis Rewriter',
|
||||
'thesis-rewriter',
|
||||
'Rewrites deterministic trade thesis summaries into clear, professional analyst prose. Optional layer — system falls back to deterministic thesis if this fails.',
|
||||
'ollama',
|
||||
'llama3.1:8b',
|
||||
'You are a concise financial analyst. You rewrite structured trade thesis summaries into clear, professional prose suitable for an internal research note. STRICT RULES: 1. Do NOT add any information not present in the input. 2. Do NOT fabricate numbers, dates, company names. 3. Keep under 150 words. 4. Preserve all factual claims, risk notes, evidence counts. 5. Neutral, professional tone. 6. Return ONLY the rewritten thesis text.',
|
||||
'thesis-rewrite-v1',
|
||||
'1.0.0',
|
||||
'system'
|
||||
)
|
||||
) AS v(name, slug, purpose, model_provider, model_name, system_prompt, prompt_version, schema_version, source)
|
||||
WHERE NOT EXISTS (SELECT 1 FROM ai_agents WHERE ai_agents.slug = v.slug);
|
||||
|
||||
-- Agent performance log: per-invocation metrics linked to agent config.
|
||||
-- This supplements model_performance_metrics with agent-level attribution.
|
||||
CREATE TABLE IF NOT EXISTS agent_performance_log (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
agent_id UUID NOT NULL REFERENCES ai_agents(id) ON DELETE CASCADE,
|
||||
document_id UUID REFERENCES documents(id) ON DELETE SET NULL,
|
||||
ticker VARCHAR(20),
|
||||
success BOOLEAN NOT NULL,
|
||||
duration_ms INTEGER NOT NULL DEFAULT 0,
|
||||
confidence FLOAT DEFAULT 0.0,
|
||||
retry_count INTEGER DEFAULT 0,
|
||||
input_tokens INTEGER DEFAULT 0,
|
||||
output_tokens INTEGER DEFAULT 0,
|
||||
error_message TEXT,
|
||||
recorded_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_perf_agent ON agent_performance_log(agent_id, recorded_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_perf_time ON agent_performance_log(recorded_at DESC);
|
||||
Reference in New Issue
Block a user