Evidence-grounded AI for software incident investigation.
TracePilot investigates software incidents by combining read-only engineering tools, hybrid retrieval, persistent evidence, and LLM reasoning. The system is designed around a simple constraint: a model should not be able to cite evidence that the application cannot establish.
The Hallucination Danger in Operational Incidents
LLMs can produce plausible incident diagnoses while inventing supporting details. In an operational investigation, that creates a dangerous failure mode.
- Nonexistent commits cited as root causes
- Fabricated file paths that misdirect on-call engineers
- Unsupported diagnoses created from superficial pattern matching
- Fake citations that bypass verification
- Uncontrolled tool usage risking mutating production infrastructure
How can an LLM investigate an incident while keeping its evidence chain verifiable?
System Overview & Capabilities
TracePilot is an AI incident investigation system built around deterministic verification boundaries.
- Creates an investigation workspace bound to incident context
- Executes guarded, strictly read-only diagnostic tools across GitHub, Sentry, and Kubernetes
- Retrieves relevant technical knowledge via hybrid dense + lexical retrieval
- Persists raw diagnostic evidence into PostgreSQL with immutable UUIDs before model synthesis
- Directs LLM reasoning over persisted records with required citation references
- Deterministically validates every citation against persisted database records
- Stores structured investigation timelines and diagnoses for post-incident review
Data Flow & Infrastructure
Incident
↓
Investigation
↓
Durable Job Queue (FOR UPDATE SKIP LOCKED)
↓
Read-only Tools + Retrieval (GitHub, Sentry, K8s)
↓
Evidence Persistence (Pydantic → PostgreSQL UUID)
↓
LLM Reasoning (DeepSeek Chat)
↓
Citation Validation (Deterministic DB Check)
↓
Hypothesis / ReviewThe Persist-Before-Cite Evidence Contract
Shifting trust from probabilistic model output to deterministic database logic
Tool / Retrieval Result
↓
Pydantic Validation
↓
PostgreSQL Persistence
↓
Immutable Evidence UUID
↓
LLM Hypothesis Generation
↓
Deterministic Citation ValidationA model cannot simply invent an evidence identifier.
Before a citation is accepted by TracePilot, application logic verifies that the evidence exists, belongs to the current investigation, and that the cited UUID was actually persisted by an allowlisted tool or retrieval run.
The system does not claim to eliminate hallucination probabilistically. Instead, it moves the critical trust boundary into deterministic application and database logic.
It moves a critical trust boundary from probabilistic model output into deterministic application and database logic.
Engineering Details
Three-Stage Hybrid Retrieval
To provide accurate technical documentation context, TracePilot combines dense vector search with lexical full-text search and structured reranking.
- Dense retrieval: pgvector cosine similarity over Gemini embeddings.
- Lexical retrieval: PostgreSQL tsvector and tsquery full-text search.
- Reciprocal Rank Fusion (RRF): Dense and lexical candidate lists are fused using RRF with k=60.
- Structured LLM Reranking: Top candidate chunks undergo structured scoring before context injection.
Durable Job Queue via PostgreSQL
TracePilot uses PostgreSQL itself for investigation job coordination, eliminating external queue dependencies like Celery or RabbitMQ.
- Jobs are claimed atomically using SELECT ... FOR UPDATE SKIP LOCKED.
- Workers receive 240-second heartbeat leases.
- Automatic failure recovery: If a worker crashes, the lease expires and the job becomes reclaimable.
queued
↓
worker claims job (FOR UPDATE SKIP LOCKED)
↓
running (240s lease heartbeat)
↓
completed
worker crash → lease expires → job reclaimableGuarded Read-Only Tooling
The system deliberately exposes only allowlisted read-only tools to prevent accidental mutation of operational infrastructure.
- GitHub: Recent commits, commit details, PR diffs, file contents, code identifier search.
- Sentry: Issue details, issue events, error stack traces.
- Kubernetes: Pod logs, pod status inspection.
- Zero destructive authority: No shell execution, arbitrary SQL, git push, or kubectl apply.
System Centerpiece & Inspection
Inspect the live execution state, benchmarks, security boundaries, and architectural guarantees.
04. Hybrid RAG Retrieval
Dense vector search + full-text search combined via RRF (k=60), then LLM reranked to top-1.
RRF_Score: 0.958 | Doc: 'runbook-webhooks-sla.md#L45' (Hit@1)Only allowlisted read-only diagnostics executed. Zero shell/mutation access.
Rigorous Verification Evidence
Malicious tool calls & prompt injections blocked
Zero unverified or fabricated citations on holdout
Diagnostic scenarios correctly solved
Pytest unit & integration suites passing
Across 77 Python backend files
Webhook Deliveries Timeout After Client Cleanup
Success rate dropped from 99% to 61% with failures clustering at exactly two seconds, while external provider dashboards showed healthy endpoints.
Worker claims investigation via FOR UPDATE SKIP LOCKED
Queries GitHub tool for recent commits touching HTTP client configuration
Retrieves technical runbook on third-party webhook latency profiles
Inspects diff of commit removing custom timeout parameters
Persists commit diff and runbook section as immutable PostgreSQL evidence
DeepSeek generates diagnosis linking the shared client refactor with the 2-second default limit
Deterministic citation validator confirms all cited UUIDs match persisted records
A shared HTTP client refactor removed the webhook-specific timeout override and imposed an insufficient 2-second default.
Engineering Capabilities Proven
Disciplined Technical Claims
- TracePilot is an engineering prototype and evaluation system, not a claim of production-scale incident-management infrastructure.
- The public deployment currently verifies the frontend. The FastAPI backend is configured and locally/Docker verified; public backend deployment is not presented as an independently hosted enterprise service.
- Retrieval benchmarks are derived from a fixed evaluation set (12 operational queries over 10 curated documents) rather than open-web retrieval claims.
Engineering Retrospective
The hardest part of an AI system is often not the model call. It is deciding what the model is allowed to know, what it is allowed to do, and what the surrounding software must guarantee independently.
Guarantees Belong in Code
Probabilistic models should not be responsible for enforcing security boundaries or citing truthfully. Application code and database constraints must enforce those guarantees.
Reranking Closes the Gap
Hybrid retrieval (dense + lexical) provides broad recall (100% Hit@3), but structured reranking is what drives top-1 precision from 75% to 91.7%.