Systems field guide

AI agents: architecture, controls and evaluation

A source-linked guide to agent loops, tools, memory, permissions, evaluation and production controls.

Start with the smallest workflow that works. Add autonomy only when tests prove it improves the task.

For product, engineering and operations teams deciding when and how to deploy an AI agent.

Inspect live AI trends

6system checkpoints

6official sources

29 Aug 2026Evidence checked

29 Aug 2026Content reviewed

Direct answer

What is an AI agent, and when should a team use one?

An AI agent is a model-led system that selects actions, uses tools, observes results and continues until it reaches a stop condition. Use one when a task needs flexible multi-step decisions that fixed code cannot express well. Keep a deterministic workflow when the steps are known, repeatable and high risk. Give every agent narrow tools, scoped credentials, hard budgets, complete logs and a human approval gate for consequential actions.1 2 3

Decision

Choose a workflow before an agent

A workflow follows a defined code path. An agent lets a model choose the path and tools. That extra freedom can solve variable tasks, but it also adds cost, latency and failure modes.1 2

How it works

Begin with one model call plus retrieval or a single tool. Add routing, parallel work or evaluator loops only when a measured error requires them. Move to open-ended planning only when fixed orchestration cannot handle the task variation.1 2

What to inspect

Build a representative task set before increasing autonomy. Compare completion quality, tool errors, human corrections, time and cost against the simpler workflow. Include ambiguous requests, missing data and tool failures, not only ideal cases.1 2

Decision rule

Use the least autonomous design that meets the success threshold. Reject an agent design when its extra steps do not create a repeatable gain. A fluent trace is not evidence; task outcomes are the evidence.1 2

Known limits

Benchmarks can hide production complexity. Real users omit context, connected systems change, and long tasks accumulate small errors. Recheck the decision after tool, model or task-distribution changes.1 2

Operating sequence

Write the outcome and risk limits. Ship the simplest workflow. Measure failures. Add one capability. Rerun the same task set. Keep the change only if the gain survives review.1 2

Evidence for this checkpoint

Architecture

Make the agent loop explicit

An agent loop receives a goal, inspects context, proposes an action, calls a tool, observes the result and decides whether to continue. Production safety depends on the code around that loop.1 2 3

How it works

Keep planning, tool execution and stopping as separate interfaces. The model may propose the next action, but deterministic code should validate the schema, user scope, tool permission, resource target and remaining budget before execution.1 2 3

What to inspect

Record each observation, proposal, validation result, tool response and stop reason with stable identifiers. Test whether the loop stops on success, repeated failure, low confidence, missing access, user cancellation and exhausted limits.1 2 3

Decision rule

Do not let natural-language instructions serve as the only stop condition. Use hard limits for steps, time, tokens, spend and retries. Return a partial result with a clear state when a limit ends the run.1 2 3

Known limits

A complete trace supports diagnosis, but it does not make a harmful action reversible. Place prevention before execution and use logs to improve tests, policy and recovery.1 2 3

Operating sequence

Parse the goal. Load only allowed context. Ask for one structured action. Validate it. Execute through a narrow adapter. Store the observation. Stop, continue or escalate by explicit rule.1 2 3

Evidence for this checkpoint

Interfaces

Treat tools and context as security boundaries

Tools turn model output into real effects. Context systems expose data and capabilities. A useful interface therefore needs clear ownership, narrow inputs, typed outputs and an authorization check outside the model.1 2 3

How it works

Model Context Protocol separates hosts, clients and servers, but protocol separation is not permission by itself. The host application must decide which server, data and operation a user may access for each request.1 2 3

What to inspect

Test tools with malformed fields, injected instructions, stale identifiers, cross-tenant targets, oversized results and delayed responses. Confirm that validation fails closed and that the model never receives secrets it does not need.1 2 3

Decision rule

Prefer small, purpose-built tools over broad shell, browser or database access. Give read and write operations separate tools. Require a fresh authorization decision for every write, even when an earlier read succeeded.1 2 3

Known limits

A correct schema cannot prove that an action is appropriate. Business rules, legal duties and user intent still need separate checks. Third-party servers can also change after integration.1 2 3

Operating sequence

Inventory capabilities. Split read from write. Define schemas and size limits. Bind access to the current user and resource. Validate every call. Sanitize results. Revoke unused connections.1 2 3

Evidence for this checkpoint

State

Design memory as governed data

Agent memory can include the current task, prior observations, user preferences and retrieved records. More memory can improve continuity, but irrelevant or unsafe memory can redirect later decisions.1 2 3

How it works

Separate short-lived task state from durable user or organization memory. Store structured facts with source, owner, creation time, sensitivity and expiry. Let retrieval select a small relevant set instead of replaying every prior message.1 2 3

What to inspect

Test whether one user can affect another user, whether deleted facts return, whether stale preferences override new instructions and whether untrusted retrieved text can become a durable instruction. Inspect both writes and reads.1 2 3

Decision rule

Store only information with a defined future use and retention rule. Require higher assurance before saving sensitive facts or instructions. Give users a way to inspect, correct and delete durable memory.1 2 3

Known limits

Retrieval quality and access control reduce risk, but neither guarantees relevance or truth. Memory should support a decision, not silently decide policy, identity or permission.1 2 3

Operating sequence

Classify the candidate fact. Check consent and purpose. Attach provenance and expiry. Store it in the correct scope. Filter retrieval by identity. Log use. Honor correction and deletion.1 2 3

Evidence for this checkpoint

Control

Bound permissions and consequential actions

Excessive agency appears when a model can perform more actions than the task requires, with more permission or less supervision than the impact justifies. The control must sit outside the model.1 2 3

How it works

Use short-lived credentials scoped to one user, tenant, tool and resource set. Separate proposal from execution. For high-impact actions, show the exact target and effect, then require a person or policy engine to approve.1 2 3

What to inspect

Try indirect prompt injection, confused-user requests, duplicate actions, changed targets and partial tool failures. Verify that the agent cannot expand its own role, reuse stale approval or hide a different action behind friendly wording.1 2 3

Decision rule

Require approval when an action is external, irreversible, costly, privacy-sensitive or difficult to verify. For lower-risk actions, use allowlists, transaction limits, idempotency keys and reversible staging before automatic execution.1 2 3

Known limits

Human approval can become a ritual when reviewers lack context or face too many prompts. Approval screens must state the effect plainly and remain rare enough to receive attention.1 2 3

Operating sequence

Classify impact. Resolve current identity. Calculate allowed scope. Produce a preview. Ask for approval when required. Execute once. Confirm the result. Preserve a clear rollback or incident path.1 2 3

Evidence for this checkpoint

Operations

Evaluate outcomes, traces and recovery

Agent evaluation must cover the final result and the path taken. A correct answer can still use a forbidden tool, leak data, waste resources or leave a system in an unsafe state.1 2 3

How it works

Score task completion, evidence quality, policy compliance, tool accuracy, step count, latency, cost and recovery. Keep deterministic assertions for hard rules. Use human or model review only for qualities that cannot be checked directly.1 2 3

What to inspect

Run offline cases before release, then monitor sampled production traces and outcome metrics. Add every confirmed incident and difficult human correction to the regression set. Test model, prompt and tool changes against the same baseline.1 2 3

Decision rule

Set release thresholds by impact. A low-risk research helper may tolerate recoverable misses. An agent that changes records or contacts people needs stricter policy, accuracy and recovery evidence before expansion.1 2 3

Known limits

No finite test set proves universal reliability. Production distributions shift and evaluators have blind spots. Keep exposure bounded, monitor leading indicators and preserve a fast way to disable tools or the whole loop.1 2 3

Operating sequence

Define success and forbidden outcomes. Build cases from real work. Add trace assertions. Run the baseline. Release to a small scope. Review failures. Expand only after stable results and tested recovery.1 2 3

Evidence for this checkpoint

Current matched signals

These live trend pages match the guide topic by an exact title rule. General AI stories are excluded.

Official source record

What is an AI agent, and when should a team use one?

Choose a workflow before an agent

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Make the agent loop explicit

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Treat tools and context as security boundaries

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Design memory as governed data

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Bound permissions and consequential actions

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Evaluate outcomes, traces and recovery

How it works

What to inspect

Decision rule

Known limits

Operating sequence

Evidence for this checkpoint

Current matched signals

6 official sources