Support investigations with AI agents

After working through enough support investigations, I stopped thinking of a CLI as just a developer convenience. It can be the way an AI agent sees an internal system.

The first support message is rarely the whole problem. It is usually a symptom, while the useful evidence is spread across logs, traces, events, workload state, and code. An AI agent can move through that evidence, but it needs an interface into each system.

That is what PUP and kubectl-readonly represent here. PUP gives an AI agent a view into Datadog. kubectl-readonly gives it a view into Kubernetes. The names are examples. The pattern applies to any internal system where the team already has a CLI. To the AI agent, these CLIs are its eyes.

The AI agent does the reasoning across those views. The CLI exposes the information it needs and limits what it can do. A read-only flag or wrapper means the AI agent can inspect logs, follow an identifier, compare a trace with the workload that produced it, and read the code behind a customer-facing message without changing the system.

Authentication is one consequence of putting that boundary in the CLI, not the reason to start there. If the CLI is already configured to authenticate, the AI agent can use it without being given the API key or access token. It gets the information it needs without carrying the system's credentials through the investigation.

There is an engineering path here as well. Someone new to the organization can learn an existing internal CLI, understand the system it exposes, and add the read-only boundary. A small piece of learning becomes a tool the team and its AI agents can reuse.

Support tickets are where the pattern becomes concrete. They rarely arrive with the answer attached. Usually, they come as one sentence:

"Stream Ended."

"It fails when DNS is enabled."

"The pod is running, so why can't the customer connect?"

Each one is a useful clue, but it is still only a symptom. The answer may be in an application log, a trace, a restart event, a malformed destination, or the code that turns a specific failure into a generic message.

The examples below are redacted. They show how I move from that first support message to an answer backed by evidence without publishing request IDs, account details, hostnames, destinations, or exact customer timestamps.

Start with the first useful signal

The AI agent can start with a symptom. The investigation starts when it follows an identifier.

First, ask Datadog

My first stop is Datadog. I keep the time window narrow and search for the service and the failure class. PUP returns JSON, so the AI agent can filter records directly instead of copying values out of a dashboard by hand.

pup --read-only --output json logs search \
  --query='service:stream-engine status:error' \
  --from='15m' --limit=100

A redacted result might look like this:

{
  "timestamp": "2026-08-29T10:15:02Z",
  "service": "stream-engine",
  "level": "error",
  "message": "No URI set before starting",
  "request_id": "request_id_redacted",
  "pod_name": "stream_engine_pod_redacted"
}

One result is enough to change the question. We are no longer asking why the console failed in general. We can ask which request reached which worker, what input the worker received, and why the media pipeline rejected it.

From there, I check traces and deploy events without leaving the read-only path:

pup --read-only --output json traces search \
  --query='service:stream-engine status:error' --from='15m'
pup --read-only --output json events search \
  --query='deploy' --from='2h'

I keep the first window small. If the evidence points outside it, I expand the search.

Then follow the identifier into Kubernetes

Once the Datadog result gives us a pod and a service, we know which workload to inspect in Kubernetes.

kubectl readonly get pods -n production -l app=stream-engine -o wide
kubectl readonly describe pod stream_engine_pod_redacted -n production
kubectl readonly logs stream_engine_pod_redacted -n production \
  --since=15m --tail=300
kubectl readonly get events -n production --sort-by=.lastTimestamp

A redacted pod summary might look like this:

NAME                         READY   STATUS    RESTARTS   AGE
stream_engine_pod_redacted   1/1     Running   1          3h

Last State:  Terminated
Reason:     Error
Restarts:   1

A pod in Running state is only a snapshot. It tells us the current state, not whether the process stayed healthy during the incident. The restart count, last termination reason, events, and logs bounded to the incident window fill in that history.

The two tools answer different questions. PUP tells us what the application and observability data saw. kubectl-readonly tells us what happened to the workload around the same time.

Test the explanation everyone wants to believe

Support reports often arrive with a theory attached. Here, the visible difference is the DNS path. It is tempting to call it a DNS problem and stop there. The AI agent should test that theory.

I compare the path that works with the one that fails:

path:              dns_disabled
worker_selected:   worker_path_redacted_a
pipeline_error:    none

path:              dns_enabled
worker_selected:   worker_path_redacted_b
pipeline_error:    no_uri_before_starting
destination:       rtmp_destination_redacted

The comparison gives us a more precise answer. DNS may be selecting a different product path, but the malformed destination is the input that fails once that path is selected. I would report those separately: one is observed, and the other is inferred from the evidence.

The correlation shows where the paths split. The logs and code show what failed after the split.

Read the code after the logs

Logs tell us what happened in production. The repository tells us why the customer saw that failure as the message they reported.

The AI agent searches the relevant code path using the error signature and the user-facing error:

rg -n 'No URI set before starting|Stream Ended|broadcast' .

The result is redacted, but the sequence is enough to show the path:

broadcast_start
  -> destination_resolution
  -> pipeline_start
  -> low_level_destination_error
  -> generic_stream_ended_message

Now we can trace the failure from the support report to the code. The report tells us which condition changes the outcome. PUP finds the first application error. kubectl-readonly ties it to a workload and shows what happened around the same time. The code explains how a specific destination failure became the generic "Stream Ended" message.

The customer does not need every internal detail. The useful handoff is the one that explains the symptom and points Engineering at the failure path.

Leave a trail someone else can check

A useful answer is one someone else can check.

I keep one rule for these investigations: each step must produce a concrete signal or record why the signal is unavailable. In the example above, the signals are:

The table is a small test harness. It turns "the AI agent looked around" into a sequence of checks that shows what ran. If permissions prevent a query, logs are missing, or an endpoint returns an ambiguous result, the report records UNTESTED instead of quietly moving on.

Structured output helps during the investigation. JSON works well while the AI agent correlates records. A short evidence table works better for the human handoff. The format can change, but the signal should not disappear in a paragraph of prose.

The tools the AI agent uses

PUP gives the AI agent command-line access to Datadog data. It covers logs, traces, metrics, monitors, events, incidents, and other operational data. Its structured output gives the AI agent records it can search and compare directly.

kubectl-readonly is a wrapper around kubectl. It allows inspection commands such as get, describe, logs, top, events, explain, and rollout status. It blocks commands such as delete, apply, scale, exec, and edit. It also blocks common attempts to print Secret values.

In both cases, the AI agent invokes the CLI. The CLI controls which operations are available. Neither tool diagnoses the incident. They put the evidence in the same session where the AI agent can read the support report and the code.

I use the same sequence each time:

Start with the customer's exact condition.

Establish a narrow timeline.

Carry identifiers between systems.

Check the runtime state.

Compare the working and failing paths.

Read the code that maps the internal error to the message the customer sees.

State facts, inferences, confidence, and open questions separately.

Install the tools

Install and authenticate PUP

On macOS and Linux, the Homebrew installation is:

brew tap datadog-labs/pack
brew install datadog-labs/pack/pup

pup auth login
pup auth status
pup --version

For a headless AI agent, configure the credentials outside the prompt through the environment or a secret manager. The CLI reads them. The AI agent does not need to. API keys should never end up in prompts, tickets, logs, or AGENTS.md.

Installing PUP does not make an investigation read-only by itself. Commands from the AI agent still need to begin with:

pup --read-only ...

PUP's read-only flag blocks create, update, and delete operations while keeping the query surfaces available. PUP's --agent flag is separate. It controls PUP's output mode for AI-agent use. It does not make commands read-only or grant the AI agent additional permissions. If the environment does not enable that mode automatically, pass --agent.

pup --agent --read-only --output json logs search \\
  --query='service:stream-engine status:error' --from='15m' --limit=100

Install kubectl-readonly through Krew

Krew manages kubectl plugins. Install the read-only plugin once on the machine that runs the AI agent:

kubectl krew install readonly
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

The command installs a local plugin. It does not modify a cluster. After installation, use the guarded command form for every cluster query:

kubectl readonly get pods -A -o wide
kubectl readonly config current-context
kubectl readonly auth can-i get pods --all-namespaces

Before handing the tool to an AI agent, test both sides of the boundary:

kubectl-readonly --readonly-check-ok get pods
kubectl-readonly --readonly-check-ok delete pod pod_name_redacted

The allowed query should pass. The delete query should be reported as blocked without reaching the cluster.

The wrapper is a client-side guardrail. Pair it with a Kubernetes identity that has read-only RBAC permissions. If someone bypasses the wrapper or it fails, the Kubernetes identity still cannot write to the cluster.

Give the AI agent some defaults

Writing "use read-only" in AGENTS.md is not enough. The file should tell the AI agent how to investigate, what to do when a command is blocked, and what a finished answer must contain.

Keep this policy near the repository or in the AI agent's project instructions:

## Production investigation policy

When investigating a support report or production incident:

1. Use PUP for Datadog evidence. Every PUP command must include `--read-only`.
2. Use `kubectl readonly ...` for Kubernetes evidence. Never use bare `kubectl get`, `kubectl logs`, `kubectl describe`, or another unguarded cluster command.
3. Never bypass a blocked command. Do not retry it through another binary or alternate spelling.
4. Use bounded time windows, small result limits, and structured output when available.
5. Do not expose Secret values, customer credentials, stream keys, tokens, or unrelated personal data.
6. Treat the support report as a hypothesis, not as the root cause.
7. Correlate Datadog evidence, Kubernetes state, and repository code before concluding.
8. Report the exact commands, time windows, identifiers, and error messages that support the conclusion.
9. Separate facts, inferences, confidence, open questions, and next steps.
10. Do not make production changes. Escalate proposed remediation for human approval.

The instruction file and the tools have different jobs. AGENTS.md tells the AI agent how to behave. The CLI flag, wrapper, and RBAC permissions decide what the AI agent is allowed to do. That separation makes the boundary easier to review.

A skill is a playbook, not a permission

A reusable skill can hold this sequence for the AI agent: normalize the report, choose the first query, carry identifiers across systems, and format the handoff. That keeps the AI agent from having to rediscover the workflow on every ticket.

But a skill is still an instruction. It does not grant access or make a command safe. PUP's --read-only flag, kubectl-readonly, and Kubernetes RBAC remain the enforcement layers. The AI agent follows the playbook; the tools enforce the boundary.

A complete run has a bounded time window, at least one concrete signal, the identifier linking it to the workload, facts separated from inferences, explicit UNTESTED or blocked gaps, and a next question or safe next step.

A run that returns "nothing found" is still useful if it says what was searched, why the result is inconclusive, and where the AI agent should look next.

The support workflow

The workflow looks like this:

flowchart LR
    A[Support report] --> B[Normalize intake]
    B --> C[PUP Datadog timeline]
    C --> D[kubectl readonly runtime context]
    D --> E[Code and ownership mapping]
    E --> F[Compare hypotheses]
    F --> G[Human reviewed conclusion]
    G --> H[Support response or engineering follow up]

Support does not need to know the right dashboard or namespace. It needs to provide the condition, the time, and the customer impact. The AI agent handles the first evidence pass.

Before I call the investigation done, I want the handoff to answer six questions:

What is the most likely cause, in one or two sentences?

What logs, traces, pod state, events, and code path support it?

How confident are we, and why?

Under what conditions are customers affected?

What is the smallest safe remediation or the owning team?

What still needs confirmation?

If the handoff cannot answer these questions, the investigation may be fast, but it is still a guess.

Where does the time go?

The time savings come from removing repeated setup work. They do not come from pretending every incident has a five-minute answer.

Without a defined path, the work tends to bounce between people and systems:

Support sends a symptom with a partial timestamp.

An engineer asks for an identifier.

Someone searches a dashboard.

Another person checks a namespace.

The two people compare notes.

The original hypothesis gets mixed with later guesses.

With PUP, kubectl-readonly, and an instruction file, the AI agent starts from a known sequence. It can query Datadog, carry the request or pod identifier into Kubernetes, inspect the relevant logs, and return the evidence in one response.

To see whether the workflow helps, I track four timestamps:

support intake to first relevant evidence

support intake to first plausible hypothesis

support intake to a conclusion backed by evidence

conclusion to a response Support can use

I also track repeated commands, handoffs, reopened conclusions, and the percentage of reports that contain explicit evidence. Missing evidence is marked UNTESTED, while blocked write attempts are tracked separately. These measurements show whether the workflow is improving investigations or only producing faster text. A short answer without a signal is still just a faster guess.

The exact minutes vary. Missing identifiers, incomplete logs, and unfamiliar services still require human work. The useful part is that the AI agent starts with the right tools and the right questions.

The guardrails I won't trade away

Read-only access is necessary, but it is only part of the design:

Give the AI agent the smallest Datadog scope that supports investigation.

Give its Kubernetes identity read-only RBAC permissions.

Keep credentials out of instruction files and chat transcripts.

Use bounded log windows and avoid following logs indefinitely.

Never print Secret values, stream keys, tokens, or unrelated customer data.

Redact account IDs, request IDs, hostnames, and destinations before sharing examples.

Treat a blocked command as evidence that the boundary is working. Do not bypass it.

Require human approval for restarts, rollouts, configuration changes, or remediation that affects customers.

The AI agent only needs enough access to gather evidence for the next human decision.

A practical place to start

I would not start by building a large platform. I would start with the tools the team already uses.

Install PUP. Add kubectl-readonly. Put the investigation method in AGENTS.md. Give the AI agent evidence, not control over production.

Then follow the same sequence each time: normalize the report, establish the timeline, follow the identifier, inspect the workload, read the code, compare the explanations, and return the evidence with the conclusion.

When the next ticket says "Stream Ended," start with one command that tells you what happened, another that shows where it happened, and a method that explains why.