Newsletter
LLM

LLMOps in 2027: From MLOps to LLM Agents

What LLMOps actually means in 2027 — prompts, evals, tracing, gateways and agent operations — and how it grew out of MLOps. The updated edition of Engatica's LLMops post.

Engatica editors · 2026-10-24 · 10 min read
LLMOpsLLM agentsMLOpsEvals

The original version of this article was the last Engatica post still pulling search traffic in 2023, ranking for "llmops" just as the term took off. The 2027 stack it describes has five layers, a standard protocol for its tools — and an observability specification that, as of mid-2026, was still being rewritten in public. LLMOps did not settle down. It grew up, and it grew a second centre of gravity: the agent.

What LLMOps is (and is not)

MLOps managed a model you trained: versioned data, training pipelines, a registry, deployment and drift monitoring. LLMOps manages a model you mostly rent: the artefacts under your control are prompts, retrieval corpora, tool definitions, routing and guardrails — plus the evaluation harnesses that tell you whether a change helped or quietly broke things. If your team fine-tunes, both disciplines apply; most teams, most of the time, live in the second world.

The five layers of a 2027 LLMOps stack

  1. Versioned prompts and configuration. Prompts are code. They need versions, owners, changelogs and rollback — not a shared doc and hope.
  2. An AI gateway. One OpenAI-compatible entry point in front of every model provider gives you routing, fallbacks, key management and cost control in a single place. Increasingly, that gateway also brokers MCP servers and tool calls, not just model traffic.
  3. Tracing. OpenTelemetry-style traces of every call — prompt, context, tool calls, latency, tokens, cost — are the unit of debugging for LLM systems. Without traces, every incident is anecdote.
  4. Evals as CI. Golden datasets, LLM-as-judge scorers and regression gates that run before a prompt or model change ships. In 2027 this is the difference between a demo and a system.
  5. Agent operations. Agents add loops, tools and side effects: you now monitor task completion, tool-call error rates, runaway loops and per-task cost, not just per-call quality.

You can assemble those five layers from parts, or take them as a platform. The platform end of the market has consolidated around hosted suites and a smaller set of open-source, self-hosted apps, some of which bundle versioned prompts, an OpenAI-compatible gateway, OpenTelemetry tracing, evals and a tool catalog in a single application teams run themselves. The build-versus-buy logic from our agents guide applies unchanged: whichever route you choose, own your traces and evals.

Where teams go wrong

The pattern is consistent: shipping on vibes (no evals), logging nothing sensitive-looking (and therefore nothing useful), hard-coding one provider into every service, and discovering cost only at invoice time. Every one of those is an operations failure, not a modelling failure — which is precisely why LLMOps exists as its own discipline.

What to do on Monday

Going deeper: our LLM hub collects the applied guides, and the practical guide to LLM agents covers the agent layer this stack now has to support.

Layer six: protocols, identity, and a standard still being written

The five layers above describe a stack you control. Layer six is the one the industry now controls jointly — and it has two halves. The first is the tool plane. With the Model Context Protocol donated to the Linux Foundation's Agentic AI Foundation in December 2025, how agents reach tools stopped being a per-vendor decision: MCP servers are now part of the operations surface, to be versioned, traced and permissioned like any other production dependency (Linux Foundation, 2025). The second half is identity. In February 2026, NIST's Center for AI Standards and Innovation launched an AI Agent Standards Initiative, and its National Cybersecurity Center of Excellence opened a project on software and AI agent identity and authorization — a response to the common practice of running agents as generic service accounts that no one can audit after the fact (NIST, 2026; Cloud Security Alliance, 2026). If your agents act, "which agent, authorised by whom, within what limits" is now an LLMOps question, not a security team's afterthought.

One honest caveat belongs in every 2027 stack diagram: the tracing layer is standardising more slowly than the rest. OpenTelemetry's GenAI semantic conventions — the vocabulary behind agent traces, including MCP tool-call spans — were moved into a dedicated repository in 2026 and remain in development status, so attribute names can still change (iBL, 2026). The practical posture: emit OpenTelemetry-shaped telemetry from day one (the direction of travel is not in doubt — Anthropic began shipping Claude Platform traces in the GenAI conventions in October 2026), but pin your instrumentation version, and treat your own trace schema as an interface you version like an API. Standards reward the teams that adopted them early and versioned them honestly; they punish the teams that assumed "standard" meant "finished."

Frequently asked questions

What is the difference between LLMOps and MLOps?

MLOps managed a model you trained: versioned data, training pipelines, a registry, deployment and drift monitoring. LLMOps manages a model you mostly rent — the artefacts under your control are prompts, retrieval corpora, tool definitions, routing and guardrails, plus the evaluation harnesses that show whether a change helped or quietly broke things. If your team fine-tunes, both disciplines apply; most teams live in the second world.

What are the layers of an LLMOps stack?

Five core layers: versioned prompts and configuration; an AI gateway giving one entry point for routing, fallbacks and cost control; tracing of every call; evals run as CI before any prompt or model change ships; and agent operations — monitoring task completion, tool-call errors and per-task cost. A sixth layer is emerging: standard tool protocols such as MCP, plus agent identity and authorization.

Is OpenTelemetry the standard for LLM tracing?

It is the direction of travel, but not a finished standard. OpenTelemetry's GenAI semantic conventions — the vocabulary behind agent traces, including MCP tool-call spans — were moved into a dedicated repository in 2026 and remain in development status, so attribute names can still change. Emit OpenTelemetry-shaped telemetry from day one, pin your instrumentation version, and version your own trace schema like an API.

Why does agent identity belong in LLMOps?

Because agents act — screening, scoring, transacting — often as generic service accounts that nobody can audit after the fact. In February 2026 NIST launched an AI Agent Standards Initiative, and its NCCoE opened a project on software and AI agent identity and authorization. "Which agent, authorised by whom, within what limits" is now an operations question. See also our practical guide to LLM agents.

Sources

  1. Linux Foundation. (2025). "Linux Foundation Announces the Formation of the Agentic AI Foundation (AAIF)". Agentic AI Foundation.
  2. iBL. (2026). "AI Agent Observability Has No Standard to Buy Yet". iBL.ai.
  3. OpenHands. (2026). "What Is AI Agent Observability? How to Trace, Govern, and Control Agents at Scale". OpenHands.
  4. AI Parabellum. (2026). "Claude Adds OpenTelemetry Tracing for AI Agents". AI Parabellum.
  5. NIST / Center for AI Standards and Innovation. (2026). "Announcing the 'AI Agent Standards Initiative' for Interoperable and Secure Innovation" (release of 17 February 2026, reprinted by HPCwire/AIwire, 20 February 2026).
  6. Cloud Security Alliance. (2026). "NIST AI Agent Standards: What It Means for Enterprise Security" (CSA Research Note).
  7. Chowdhry, A. (2025). "LangChain: $125 Million Raised To Advance Agent Engineering Platform". Pulse2.
  8. Gartner. (2026). "Gartner AI Hype Cycle: Why Control Now Drives AI Value". Gartner.