Langfuse vs MLflow: Which One I'd Pick for LLM Observability
- Nikhil Verma
- 16 hours ago
- 8 min read
If you're building anything with LLMs in production, you eventually hit the same wall: you need to know what your app is actually doing — what prompts went out, what came back, how much it cost, and why it broke at 2am. That's observability, and two names keep coming up: Langfuse and MLflow. Before comparing them, it's worth laying out what you're actually trying to observe — because observability means something different at each layer of a modern AI system.
LLM observability in five layers
Logging and tracing are what separate a demo from a production-ready app — the kind of thing nobody thinks about until 2am, when a customer says the bot gave a wrong answer and you have absolutely no way to find out why. Here's the fuller picture, layer by layer.
1. Request-level logging
Every request gets a request_id and trace_id propagated across the stack. Log:
User/session and tenant IDs
Timestamp, environment, deployment version
Model/provider, parameters, fallback route
Token usage, cost, latency, status, error type
This lets you debug failed or costly requests without guessing.
2. Prompt, RAG, and context logging
Most quality failures are context failures, not model failures. Trace:
System prompt and template version
User query and conversation state
Retrieved chunk IDs, scores, sources
Reranker results, tool inputs/outputs, citations
When quality drops, this shows whether retrieval was irrelevant, the prompt changed, a tool failed, or the model reasoned poorly.
3. Agent and tool tracing
The final response isn't enough — trace every span:
Planner/router decisions
Tool calls, inputs, outputs, duration
Retries, fallbacks, loop count
Guardrail checks and blocked actions
This exposes runaway loops, duplicate tool calls, silent retries, and hidden cost spikes.
4. Performance, cost, and reliability metrics
Monitor:
p50/p95/p99 latency and time-to-first-token
Tokens and cost per successful task
Error, timeout, rate-limit, cache-hit, and fallback rates
Retrieval/tool success, safety, user feedback, task success
A 200 OK isn't success if the answer is hallucinated.
5. Distributed tracing and quality evaluation
A request crosses User → API → Orchestrator → Vector DB → Reranker → LLM → Tools → Database. Traces find bottlenecks; sampled traces feed evals for groundedness, citations, safety, and task completion.
Privacy note: redact PII and secrets, enforce tenant isolation and retention limits, and avoid logging raw payloads by default.
A common stack pairing: OpenTelemetry + OTLP for vendor-neutral trace transport, an LLM-specific tool (Langfuse, MLflow, or similar) for prompt/eval-level detail, and Datadog or Grafana for the infra layer underneath.
Traditional observability tells you if the system worked. LLM observability tells you what the model saw, why it acted, and whether the outcome was actually good.
Where your cloud provider's job actually ends
Here's the mental model that made this click for me. Say you're running on AWS — you already get a chunk of this for free. CloudWatch will tell you a pod got OOM-killed. EKS will tell you a container restarted three times overnight. An ALB access log will tell you a request took 4.2 seconds. That's real, useful signal, and it's genuinely AWS's job to give it to you.
But the second that request hits your application code and calls an LLM, AWS goes quiet. It has no idea what prompt you actually sent, what the model reasoned through, whether your RAG pipeline retrieved the wrong chunk, or whether your agent quietly looped through the same tool call four times before giving up. That's not a gap AWS forgot to fill — it was never their layer to begin with. It's yours.
So a realistic production stack ends up looking like this: AWS (or GCP, or Azure) handles infra and pod-level monitoring, OpenTelemetry carries trace context across that boundary, and then — inside your application — you need something purpose-built to pick up where the cloud provider stops: logging the prompt, the retrieval, the tool calls, the tokens, the cost. That's the job Langfuse or MLflow actually does. Pick one; you generally don't need both.
A concrete example: AWS Bedrock + Langfuse
AWS actually does have an official platform for hosting agentic AI apps — it's not just raw EC2/Lambda. Amazon Bedrock AgentCore is AWS's managed runtime for building and deploying AI agents in production, and Bedrock Knowledge Bases is its managed RAG layer (it handles the vector store, chunking, and retrieval for you). So a Bedrock-based stack looks like this:
A user request comes in and hits Bedrock AgentCore, AWS's managed layer for actually running the agent: orchestration, tool calls, and the underlying Claude/Titan/other foundation model call all happen here.
AgentCore calls out to Bedrock Knowledge Base when it needs context — Bedrock's managed vector retrieval (backed by OpenSearch under the hood), so you don't run your own vector DB.
Both of those steps emit OpenTelemetry spans, which Langfuse (or MLflow) picks up via an OTel endpoint: LLM calls and tokens, tool calls and retries, cost per request, latency and errors — all captured with no custom instrumentation beyond pointing the OTel exporter at your tool of choice.
The agent's output goes back to the user.
This is a real, documented integration path, not a hypothetical — Langfuse ships an official Bedrock AgentCore integration guide and a separate cookbook for tracing Bedrock Agents specifically, both OTel-based. MLflow autologs Bedrock calls too, so either tool is a drop-in addition, not a rearchitecture — this exact shape works whether you plug in Langfuse or MLflow at the observability layer.
What Bedrock's own observability already gives you
It's worth being precise here, because Bedrock isn't a blank slate — it has real native observability, and Langfuse isn't replacing it, it's filling in around it:
CloudWatch GenAI Observability gives you out-of-the-box dashboards for invocation count, InvocationLatency, TimeToFirstToken, input/output token counts, throttles, and client/server error counts — all aggregated by model ID.
Model Invocation Logging (opt-in) will dump the full raw input/output payload for every call to CloudWatch Logs or S3, if you turn it on.
AgentCore Observability adds built-in metrics for the runtime, memory, gateway, and tool resources, plus span/trace data once you instrument your code with the AWS ADOT (OpenTelemetry) SDK — viewable through CloudWatch's Transaction Search, which is built on X-Ray.
Bedrock Prompt Management stores prompt versions, diffs versions side by side, and lets you fill in test variables and run a prompt to compare model outputs.
So AWS genuinely covers infra metrics, aggregate token/latency dashboards, distributed traces once instrumented, and prompt versioning/testing. That's a wider native footprint than just infra.
The gap Langfuse still fills, once you account for all of that:
It's four separate consoles, not one. CloudWatch dashboards, CloudWatch Logs/S3 payload dumps, X-Ray Transaction Search, and Prompt Management are four different AWS surfaces with no built-in way to click a trace and see its prompt version, its eval score, and its cost together. That correlation is the actual product Langfuse (and MLflow) sell — a single trace view, not four.
Prompt Management's testing is pre-production. You compare versions against test variables you type in yourself; it isn't wired to run a new prompt version against real captured production traces the way Langfuse's Prompt Experiments are.
No eval loop tied to live traffic. Bedrock's separate Model Evaluation feature benchmarks a model against a dataset you supply — it doesn't continuously score your actual production traces the way Langfuse's or MLflow's evaluators do.
Nothing is on by default. AgentCore Observability requires explicitly setting AGENT_OBSERVABILITY_ENABLED=true plus ADOT instrumentation.
Trace sampling, not full capture. CloudWatch Transaction Search indexes only 1% of spans as full trace summaries by default (you can raise it, at added cost) — Langfuse captures every trace by design.
Aggregation is by model ID only. CloudWatch has no concept of cost for user X or latency for feature Y — Langfuse/MLflow let you tag traces with your own metadata (user, tenant, feature) and slice cost/latency by it.
No cross-provider view. CloudWatch/X-Ray/Prompt Management only know about Bedrock. The moment your stack also calls OpenAI directly, or a self-hosted model, you lose a unified view — Langfuse and MLflow are provider-agnostic by design.
In short: AWS covers more ground natively than most people assume — infra, metrics, traces once instrumented, even prompt versioning. What Langfuse adds isn't a capability AWS is entirely missing; it's unification — one console correlating trace, prompt version, cost, and eval score — plus full-capture-by-default and cross-provider portability, none of which the four separate AWS surfaces give you together.
The short version
So, which one's actually better? It depends less on features and more on where each tool came from.
Langfuse was built from day one for LLM applications. MLflow was built in 2018 for classic ML experiment tracking (think scikit-learn, XGBoost) and later grew GenAI tracing on top of that foundation. That difference in origin shapes almost everything else about how they feel to use.
My take: if your workload is LLM-first — agents, RAG pipelines, chatbots — I prefer Langfuse. If you're running classic ML models and LLMs on the same platform and want one tool for both, MLflow earns its keep.
Where they actually differ
Here is the practical comparison I use when choosing between them:
Area | Langfuse | MLflow |
License | MIT core; some enterprise governance features are commercial. | Apache 2.0, fully open source. |
Self-hosting | Free for core tracing; some advanced self-hosted features are commercial. | Free OSS self-hosting; some production-scale managed features are Databricks-managed. |
LLM tracing | @observe and native OpenTelemetry ingestion. | @mlflow.trace and autologging for major frameworks. |
Evaluation | Evaluation capabilities; OSS vs managed scope differs. | Evaluation capabilities; OSS vs managed scope differs. |
Prompt management | Versioning and comparison with an LLM-focused workflow. | Versioning and comparison with an ML-platform-oriented workflow. |
Classic ML lifecycle | No — focused on LLM/agent observability. | Yes — experiments, model lifecycle, and model registry. |
Best fit | LLM/agent observability as the center of gravity. | Classic ML + GenAI genuinely sharing one platform. |
My practical summary: choose Langfuse when LLM/agent observability is the center of gravity; choose MLflow when classic ML and GenAI genuinely need one platform.
Why I lean Langfuse
1. It was designed around traces, not experiments. LLM apps are chains of calls — an agent step, a tool call, a retrieval, a generation. Langfuse's tracing model maps directly onto that.
2. The prompt playground and versioning are genuinely useful day-to-day. Being able to iterate on a prompt, see cost/latency/quality side by side, and roll back a version is something I use constantly — it's a first-class feature, not an add-on.
3. OpenTelemetry-native. If you already have OTel instrumentation elsewhere in your stack, Langfuse slots in cleanly instead of asking you to adopt a separate logging convention.
4. Self-hosting is free for core tracing, and the license is permissive (MIT core). You're not stuck paying for basic self-hosted tracing — though some advanced self-hosted features sit behind a paid commercial add-on.
Why you might pick MLflow instead
You already run MLflow for classic ML experiment tracking, and adding a second tool just for LLM observability is more operational overhead than it's worth.
You want one pane of glass across traditional models and LLM-based ones.
You're already on Databricks, or plan to be — the OSS eval data model exists, while some polished managed features are available through Databricks Managed MLflow.
Apache 2.0 licensing matters to your organization for compliance reasons.
My honest recommendation
For a pure LLM/agent product, Langfuse is the tool I'd default to — it's built for the problem you actually have, the tracing UX is sharper, and prompt management alone saves real time. MLflow is the better call only when classic ML and GenAI genuinely need to share one platform.
This is one engineer's take based on hands-on differences, not a scored benchmark — worth trying both against your own traces before committing, since retention limits, pricing tiers, and self-hosting overhead can tip the decision depending on your scale.
Sources
Facts in this post — licensing, pricing tiers, which features are OSS vs. gated, and tracing API details — were checked against the following as of September 2026. Inline [n] markers throughout the post point back to this list.
Comments