Runtime Threat Detection and Audit Pipelines — eBPF, Falco, and the Action-Log Discipline

The kernel watches; the rules interpret; the log remembers.

§ IFrame

A multi-agent system that has passed every build-time trust gate and every deploy-time policy check can still be compromised at runtime. The compromise rarely looks like an exploit on day one. It looks like a sequence of legitimate actions performed in an order or at a tempo no operator anticipated. An agent that should read from one bucket reads from a second. An agent that should call a single model API calls three. A sidecar that should write only to standard out opens a connection to a previously unobserved host. Each step, in isolation, passes the policy that admitted the workload. The aggregate is the breach.

The runtime layer is where this aggregate becomes legible. Two questions live here. First — what is each process actually doing right now, at the syscall and network level. Second — what is the cumulative action history of each agent identity, and does that history match the agent's declared role. The first question is observability. The second is audit. Today's lesson takes both.

§ IIFoundations

Three primitives carry the runtime trust discipline. Each has a specific function; each has a failure mode when treated as a substitute for one of the others.

These three compose. eBPF watches; Falco interprets; the audit log remembers. A multi-agent system without all three has gaps — observation without interpretation drowns the operator; interpretation without history cannot answer the question that matters most after an incident.

§ IIIMechanism

The runtime detection pipeline runs in four stages. Each stage has a contract with the next; each contract is what makes the pipeline trustworthy.

Stage one — kernel-event capture

A Falco daemonset on every node runs an eBPF program that subscribes to the kernel hooks Falco's rules reference. The eBPF program normalizes events into a Falco-internal struct and pushes them to userspace through a ring buffer. The ring buffer is the back-pressure surface — if Falco userspace cannot keep up, events drop and a metric fires. Drop-rate is the first signal the pipeline is failing.

Stage two — rule evaluation

Falco userspace pulls events from the ring buffer and evaluates each against the loaded rule corpus. Rules can join across events using stateful conditions (Falco's macros and lists primitives) but the joining is bounded — Falco does not hold unbounded session state because the kernel does not promise event ordering across CPUs. The rule corpus is divided into bundles: upstream Falco rules, cloud-provider bundles (AWS, GCP, Azure), and the operator's local rules. Local rules carry the named-technique discipline: a rule called agent-egress-to-undeclared-host is read faster at 3 a.m. than one called falco_rule_47.

Stage three — output routing

Falco's output channels go to two destinations in parallel. The high-priority alerts (Falco priorities CRITICAL and EMERGENCY) page the on-call. The full event stream — including INFO and NOTICE — writes to a log aggregator (Loki, OpenSearch, or a cloud-native service). The aggregator is the forensic surface. The page is the present-tense response.

Stage four — audit-log emission

The agent itself — not Falco — emits a structured audit-log entry on every action it takes. The audit log is application-level, not kernel-level, because it carries semantic information eBPF cannot see: which model was called, which prompt was sent, which tool was invoked, what arguments. The audit log entries sign with the agent's SPIFFE identity (the chain from the Wed admission lesson and the Mon workload-identity lesson) and write to an append-only destination — a write-once object store, a Merkle-tree-backed ledger, or a hash-chained log file. Append-only is the trust property; the agent cannot rewrite its own history.

§ IVWorked Example

Consider a three-agent inference pipeline. Agent A reads market data from a permissioned S3 bucket, agent B runs inference against a Bedrock endpoint, agent C writes the inference output to a permissioned PostgreSQL store. Each agent runs in its own pod under SPIFFE identities spiffe://hedronite/agent-a, agent-b, agent-c.

A Falco rule called agent-network-egress-outside-allowlist watches every TCP connect syscall from any container in the agents namespace. The rule joins the destination IP against an allowlist maintained as a Falco list. Any connection outside the list emits a CRITICAL event. The list is generated nightly by a controller that reads the deployment manifest's declared dependencies and resolves them to current IP ranges.

A second rule, agent-spawned-shell, fires on any execve syscall whose process name matches bash, sh, dash, ash, zsh inside the agents namespace. An LLM-driven agent should never spawn a shell. If one does, either the agent has been compromised through prompt injection or the operator has misconfigured the pod. Either way, page.

A third rule, agent-read-credentials-from-disk, fires on any openat syscall against paths matching /var/run/secrets/, /etc/kubernetes/, /home/*/.ssh/ from the agents namespace. ESO injects credentials as environment variables or projected volumes the application reads at startup. A read after startup from these paths is anomalous; a CRITICAL fires.

Meanwhile, agent B emits an audit-log entry every time it calls the Bedrock endpoint. The entry carries the SPIFFE identity, a UTC timestamp, the model ARN, a SHA-256 of the prompt, the input-token count, the response-token count, the HTTP status, and a tool-trace section if the model invoked a tool. The entry signs with the agent's workload key and writes to an OpenSearch index configured with write-once-read-many semantics.

An incident: Falco fires agent-network-egress-outside-allowlist at 02:14:17 UTC for agent B. The destination IP resolves to a known prompt-injection-exfil host. The on-call pulls agent B's audit log for the prior five minutes. The audit log shows agent B received a prompt at 02:14:12 UTC whose SHA-256 matches an entry in the published prompt-injection-corpus feed. Five seconds later the egress happened. The cross-reference is the post-mortem.

§ VConnection to Prior Lessons

Tuesday's lesson (secret management) established the credential lifecycle — issue, rotate, revoke. Wednesday's lesson (admission control + ESO) established the deploy-time gate — credentials reach the pod only if policy permits. Today closes the runtime gate. Together the three days build the β-Trust DevOps spine: a credential exists only if it should, reaches a workload only if the workload is admitted, and is used only if the runtime watches and the audit log remembers.

The Mon 2026-05-20 lesson on SPIFFE workload identity is the identity backbone all three layers reference. Audit-log entries are only meaningful if the identity signing them is non-forgeable; SPIFFE's SVID architecture is what makes the signature trustworthy. The detection layer is only actionable if the operator can attribute a syscall back to an agent identity; Falco's container-metadata enrichment uses the same SPIFFE chain.

The Tuesday-Wednesday-Thursday triad is the operator's daily-bread answer to the question what does it take to run a multi-agent system trustworthily. Three days; three layers; one chain.

§ VIConnection to Today's Dev + Cert Lessons

Today's Go lesson takes the audit-log primitive and shows how to build the emission side in Go. The middleware-chain pattern is Go's idiomatic answer to the log every action, tag with identity, sign at write-time discipline. Where this Ops lesson says append-only audit log keyed by SPIFFE identity, the Dev lesson shows the exact http.Handler decorator composition, the slog configuration, the context-propagation discipline, and the hash-chain primitive that gives append-only its trust property.

Today's Cert lesson takes the same audit-trail discipline and applies it to two vendor-managed agentic surfaces. Bedrock Agents emit trace data the operator never wrote; GitHub Copilot Workflows emit audit events through the GitHub Audit Log API. The cert-prep frame asks: what does it cost the operator when the audit trail is managed by the vendor, and where does the operator-owned audit primitive remain necessary.

§ VIIClosing

Three layers of trust. Build-time, deploy-time, runtime. The first two are gates; the third is the eye. A system that has all three has somewhere to catch failure. A system missing any one of them has failure with nowhere to land.

Examine well. Reflect on this. The β-Trust week closes at the runtime gate today. The next β visit, two weeks forward, will reach for the data primitive — encryption at rest, data-classification controllers, lineage tracking. The chain continues. Each link is the thing the next link assumes.