Skip to content

Azure AIOps Incident RCA

This project follows an incident from a FastAPI request through its dependency telemetry, then uses a Microsoft Foundry agent to explain what the evidence shows. An offline evaluator checks every report against labeled cases.

Built
October 2026
Status
API deployed on Azure Container Apps
Checks
93 automated tests and an offline RCA gate
Stack

Python · FastAPI · Microsoft Foundry · Azure Container Apps · OpenTelemetry · KQL · Prometheus · Grafana

  1. FastAPI + OpenTelemetry
  2. Application Insights
  3. KQL extraction
  4. Foundry RCA agent
  5. Offline evaluator

How it works

  • The FastAPI service on Azure Container Apps sends OpenTelemetry request and dependency traces to Application Insights.
  • A KQL query joins each request to its dependency spans by operation ID, so a 504 on the API lines up with the failed downstream call.
  • A Pandas ETL step turns the selected incident into a sanitized JSON record. Operation IDs and expected answers are removed before the model sees it.
  • A managed Foundry agent (gpt-5-mini) returns a JSON report: a status, a root-cause category, the root cause, a confidence level and the evidence.
  • Locally, Prometheus and Grafana watch the development service and its incident simulator.

Keeping the agent grounded

Every piece of evidence in a report must match an exact path=value token from the input, such as dependency.success=false. Reports with unsupported evidence or a malformed response are rejected. When the telemetry isn't enough, for example a 404 with no named dependency, the agent must answer insufficient_evidence instead of guessing. It suggests checks to investigate but never performs remediation.

A sanitized report for a dependency timeout
{
  "status": "diagnosed",
  "root_cause_category": "dependency_timeout",
  "root_cause": "Timeout of downstream dependency search_service correlated with request HTTP 504 (dependency.success=false). Underlying infrastructure cause is uncertain.",
  "confidence": "high",
  "evidence": [
    "request.status_code=504",
    "request.duration_ms=1007.0",
    "dependency.name=\"search_service\"",
    "dependency.duration_ms=1004.0",
    "dependency.success=false"
  ]
}

Evaluation

The benchmark has 24 labeled cases: six dependency timeouts, six application exceptions, six latency regressions measured against a baseline, and six cases where the evidence is not enough to diagnose.

Foundry RCA agent v2 on the 24-case benchmark
MeasureResult
Diagnosis accuracy18/18
Correct abstentions6/6
Evidence claims grounded in telemetry153/153
Schema validity24/24
False diagnoses on insufficient evidence0/6

These are familiar development cases, so the scores don't show how the agent would do on new incidents or in production. An LLM judge is implemented but has not been run yet.

Deployment and security

  • Runs on Azure Container Apps with 0.25 vCPU and 0.5 GiB of memory, scaling to zero when idle.
  • The /ask endpoint requires an API key of at least 32 characters. Production turns off the docs, metrics and simulator endpoints.
  • Prompts are limited to 2,000 characters and model responses to 1,024 tokens.
  • Credentials go through Container Apps secrets and are never baked into the Docker image.

All projects