Workflows vs agents
- Workflow: the LLM is called along a predefined code path (chain, router, parallel, evaluator-optimizer). Predictable, cheap, testable.
- Agent: the LLM decides the next step and which tools to call, in a loop, until done. Flexible, but less predictable.
Start with the simplest thing that works. Most "agents" in production are well-designed workflows.
The agent loop
Architecture for production agents
- Tools are APIs: clear names, strict JSON schemas, idempotent, with good error messages the model can act on.
- Least privilege: read tools freely; write tools behind confirmation or policy checks.
- State and memory: conversation state, task scratchpad, and long-term memory as separate concerns.
- Budgets: max steps, max tokens and timeouts per run.
- Observability: trace every step (prompt, tool call, result, cost).
- Evals: task-success rate on a fixed scenario set, re-run on every prompt or model change.
- Prompt-injection defence: treat tool outputs (web pages, emails) as data, never as instructions.
See it in action: my Jev decision walkthrough shows a tool-call gate as a working guardrail: the model answers typed questions with probabilities, and plain code applies thresholds to allow, confirm or block each call the agent wants to make.
Our team built a Claude-driven developer setup agent: typing "cms setup" in the repo provisions and validates a local environment of 60+ microservices plus MongoDB and Redis, and walks the developer through failures with log interpretation. Now I'm building a CRM/ERP agent platform in Python with a tool registry, guardrails and evals designed first.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
My post on this
Traditional API vs API with AI: an architectural shift
An AI-enabled API adds an intelligence layer between the contract and execution, bringing new responsibilities: context management, model orchestration, tool execution, guardrails, semantic caching and cost observability.
Related topics
RAG (Retrieval-Augmented Generation) Architecture
Ground LLM answers in your own data — ingestion, chunking, embeddings, vector search, reranking, prompting with citations, and evaluation.
Model Context Protocol (MCP)
An open protocol that standardises how AI apps connect to tools and data — build a server once, use it from any MCP-capable client.
LLM Evaluation (Evals)
Test AI features like software — golden datasets, code-based and LLM-as-judge graders, regression gates in CI, and production monitoring.