← Projects

Personal project

Kubernetes Incident Context & Investigation Assistant

Python · Kubernetes · Elasticsearch · PydanticAI · Grafana

The problem

When a pod crash-loops or a service degrades, on-call engineers lose time manually correlating Kubernetes resource changes, logs, and runbooks before they can even start fixing anything.

The approach

I built a PydanticAI investigation agent with five read-only tools that pulls Kubernetes resource history and deduplicated logs against Elasticsearch-indexed evidence, forces every claim to cite a concrete source, and separates confirmed observations from hypotheses so it never asserts something it didn't actually see.

The outcome

Validated with 22 automated tests across three failure scenarios and a 30-case benchmark tracking retrieval quality, latency, and cost.

More detail on this project — the agent's tool design, example investigations, and what I'd change next time — is coming soon.