Most enterprise AI conversations in 2026 begin with a model decision. Which foundation model? Which vendor? Which use case to pilot first? Few begin with a different question, which turns out to matter more once the pilot reaches month six: how will we know, on any given day, whether this AI is still working?

That question is what the industry calls evaluation, or evals. The infrastructure that answers it is the evaluation stack. Whether to build that stack in-house or buy it from a vendor has become one of the most consequential infrastructure decisions an enterprise will make in its AI journey, often without leadership realizing it is a decision at all.

Most organizations make the choice by default. They either inherit a vendor’s eval dashboard with the model they purchased and treat it as sufficient, or they hire two engineers, point them at an open-source library, and call that a strategy. Both defaults tend to fail. The build-versus-buy question deserves the same rigor an enterprise would apply to any other infrastructure decision worth crores per year.

This piece walks through what an eval stack actually is, what the current market looks like, and the framework a CXO can use to make a decision that holds up over the next thirty-six months.

What an Eval Stack Actually Does

An evaluation stack is the system that tells you, continuously and measurably, whether your AI is producing outputs that meet your definition of correct.

It exists because AI outputs are non-deterministic in a way traditional software is not. A payment processing system either confirms a transaction or it does not. The outcome is binary and verifiable. An AI summarizing a credit application produces a paragraph of text, and whether that paragraph is correct depends on whether it captured the right risk indicators, used acceptable language, did not invent figures, and did not omit clauses the underwriter needed to see. None of that can be answered by checking the HTTP response code.

A working eval stack typically consists of four layers, each of which can be built, bought, or assembled from a mix of the two.

The first layer is the test dataset, often called a golden set. This is a curated collection of input-output pairs where the correct or acceptable answer is already known. The dataset is run against the AI on a defined cadence, and the outputs are compared against the expected results. The quality of this layer is determined less by the tooling than by the depth of subject-matter expertise that went into building it. A credit risk eval set is only as useful as the credit risk analyst who decided what a correct summary looks like.

The second layer is the scoring layer. Some scoring is rule-based: did the output contain the loan amount, in the correct format, with the correct currency? Some is semantic and increasingly handled by what the industry calls LLM-as-a-judge, where another model is asked to assess the output against a defined rubric. Reported agreement rates between LLM judges and human raters typically fall in the high-eighties to low-nineties range, depending on task and judge model. That range is high enough to be useful and low enough to require human review on edge cases. Open-source frameworks for this layer include RAGAS for retrieval-augmented generation pipelines, DeepEval for Python-native test suites integrated into existing CI/CD workflows, and Promptfoo, which has become the default for adversarial and red-team testing and was acquired by OpenAI in March 2026 while remaining open source under its MIT license.

The third layer is the regression pipeline. Every time a prompt is updated, the underlying model changes, or a vendor pushes a silent update, the eval suite needs to run again and compare new scores against the previous baseline. Without this layer, an enterprise discovers regressions when customers complain. With it, the regression is caught before deployment. This is where commercial platforms like LangSmith, Braintrust and Arize position themselves, each with a different opinion about whether the center of gravity should be production monitoring, deployment gating, or experiment tracking.

The fourth layer is production monitoring. The test set is static. Production traffic is not. A small percentage of live outputs are sampled, scored, and watched for drift. Vendors in this layer include the platforms named above alongside Langfuse, Arize Phoenix, Helicone and Portkey, with several of these self-hostable, including Langfuse under a permissive MIT license and Phoenix under Elastic License 2.0. Langfuse was acquired by ClickHouse in January 2026 and committed to remaining open source under its existing MIT license following the acquisition.

Together, these four layers form what most teams are referring to when they say eval stack.

Why This Is No Longer Optional Infrastructure

A common reaction from enterprise leaders, particularly outside engineering, is that this sounds like an engineering concern that should sit inside the AI vendor’s responsibility. The vendor sold the model. Surely the vendor is responsible for proving it works.

That assumption holds for narrow use cases on stable models. It breaks for almost everything else. Three forces have shifted evaluation from a vendor concern to an enterprise concern over the past eighteen months.

Independent research and disclosed provider incidents have made clear that production behavior can change without a changelog. Practitioner accounts across production deployments suggest the gap between the onset of degradation and the first user complaint is often measured in days to weeks, which is the window in which the cost of not having an eval stack is paid in customer trust.

Under India’s Digital Personal Data Protection Act, an enterprise deploying AI on customer data carries the obligation of demonstrable purpose limitation and fairness. The Reserve Bank of India’s FREE-AI Framework makes this concrete for regulated entities in financial services, with explicit expectations around board-approved AI governance policies, AI inventories, lifecycle governance covering model approval, testing and change control, and a defined AI incident reporting protocol. The Securities and Exchange Board of India’s emerging expectations around automated decision-making in capital markets, and the broader push from sectoral regulators, point in the same direction. Enterprises will need to show what their AI did, why it did it, and how they knew it was still doing it correctly. Eval logs are the artifact that answers those questions.

The Build-versus-Buy Question, Properly Framed

Once the necessity of an eval stack is accepted, the question becomes one of construction. The temptation is to treat this as a software decision, comparing features and prices across the available platforms. That framing misses what is actually being decided.

The deeper question is which parts of the stack reflect domain knowledge unique to the enterprise, and which parts are commodity infrastructure that the enterprise has no advantage in building.

The scoring infrastructure is closer to commodity. The metrics themselves, faithfulness, answer relevancy, context precision, response coherence, are increasingly standardized. The open-source frameworks named earlier implement them well, and the commercial platforms wrap them in production-grade interfaces. Building this layer from scratch in 2026 is roughly equivalent to building a custom CI/CD system in 2015. Technically possible, occasionally justified, usually not.

The regression and monitoring layers sit between these poles. The mechanics are commodity. The thresholds, alert rules, and escalation paths are not. A platform can tell you that your factual accuracy score dropped from 0.91 to 0.84 overnight. Whether that drop matters depends on whether the use case is internal triage or external customer communication, and that judgment cannot be outsourced to the vendor.

For most enterprises, the honest answer to the build-versus-buy question is therefore a structured composition. Build the parts that encode domain knowledge. Buy the parts that are infrastructure. Assign clear internal ownership of the seams between the two.

What Buying Looks Like, and What It Actually Costs

The eval and observability market has consolidated around a small set of platforms, each with a discernible philosophy.

LangSmith, built by the maintainers of LangChain, is the path of least resistance for teams already standardized on the LangChain or LangGraph framework. Tracing is automatic, dashboards are usable on day one, and the integration is zero-config for those inside the ecosystem. As of mid-2026, its Developer tier is free with one seat and 5,000 base traces per month, and its Plus tier is priced at USD 39 per seat per month with 10,000 base traces included. Enterprise pricing is custom and available on request. The tradeoff is framework coupling. Outside the LangChain stack, much of the value reduces, and the per-trace pricing model can scale faster than expected for RAG pipelines or agentic workflows where a single user interaction can generate multiple traces.

Braintrust has positioned itself as the evaluation-first alternative, with deployment-blocking CI/CD integration and a free tier that includes one million trace spans and 10,000 scores per month, generous enough to support real pilots rather than only experimentation. Its philosophy is that evaluation and observability should be unified, and its trace-to-test pipeline reflects that view. The company raised an USD 80 million Series B in February 2026, led by ICONIQ Capital, at a reported valuation of USD 800 million. For organizations whose primary pain is shipping prompt changes that regress in production, this is the most direct fit. The Pro tier, currently priced at USD 249 per month, includes substantially higher limits but introduces a noticeable jump from the free tier.

Arize, with its open-source Phoenix project and commercial Arize AX platform, brings the deepest heritage in machine learning observability into the LLM era. Drift detection, embedding analysis and OpenTelemetry-native architecture make it the natural fit for enterprises running a mix of traditional machine learning models and generative AI, where unified monitoring across both is required. Phoenix is free and source-available under the Elastic License 2.0, which permits broad internal use but restricts offering Phoenix itself as a hosted service to third parties, a distinction that matters for platform companies and consultancies. The commercial Arize AX platform is enterprise-priced based on span volume and data ingestion.

Langfuse, now part of ClickHouse following the January 2026 acquisition, remains the strongest self-hosted, open-source option under an MIT license. The acquisition came with explicit public commitments from both companies that the open-source license, the self-hosting path, and the cloud pricing structure would all remain unchanged. For organizations whose primary constraint is data residency, where prompts and outputs cannot leave specific infrastructure, Langfuse self-hosted is often the only viable path. The tradeoff is operational. Someone on the team has to maintain the deployment, which now also typically involves running ClickHouse.

Beyond these, a longer tail of platforms (Helicone, Portkey, Datadog’s LLM observability module, Weights and Biases’ Weave, Galileo, and a growing list of newer entrants) addresses specific slices of the problem. Helicone is strong on cost tracking and gateway routing. Portkey overlaps with eval stack functionality but is primarily a gateway. Datadog and New Relic fit best where the AI workload sits alongside an existing observability footprint already on those platforms.

What Building Looks Like, and Where It Goes Wrong

The case for building exists, and is not trivial. An enterprise that builds its eval stack in-house retains complete control of where data flows, can encode bespoke metrics that no commercial platform supports, and avoids per-trace pricing models that scale unpredictably with usage. For organizations operating in regulated sectors or with strict data residency requirements that no commercial cloud vendor can satisfy, building may be the only option. The wider cost-revenue gap in AI infrastructure is itself a reason more enterprises are revisiting the build option in 2026.

The case against building is that the work is harder than it looks. The frameworks named earlier (RAGAS, DeepEval, Promptfoo) are open source and well maintained, which means the metric layer can be assembled in weeks. The harder work is everything around it: the ingestion pipelines, the version-controlled prompt registry, the human annotation workflow, the dashboarding layer, the alerting infrastructure, the integration with existing CI/CD systems and, over the long term, the team that maintains all of it.

A workable in-house eval stack at enterprise scale typically requires sustained engineering attention from people with experience in machine learning operations, plus part-time involvement from subject matter experts who curate the test sets and review edge cases. Whether this is more or less expensive than a commercial platform depends on engineering cost in the relevant market, the scale of inference, and the strategic value of retaining the data inside the enterprise.

The pattern that fails most often is the half-built stack. A team adopts an open-source library, runs evaluations for a few sprints, then loses momentum when the original engineer moves to another project. Dashboards exist but are not watched. Alerts fire but are not triaged. The stack is technically present and operationally absent. This is worse than buying, because it consumes engineering time without delivering the assurance the enterprise needs.

What Is Different for Indian Enterprises

For enterprises in India, the build-versus-buy question is shaped by three factors that do not apply identically in other markets.

The first is data residency. Several Indian banks, insurers and government-adjacent enterprises operate under explicit obligations to keep certain categories of data within domestic infrastructure. The major commercial eval platforms (LangSmith, Braintrust and Arize AX) are predominantly hosted in US or European cloud regions. Enterprise self-hosting is available across most of them, at price points that change the build-versus-buy calculation. For these organizations, a self-hosted open-source stack like Langfuse or Arize Phoenix often becomes the rational choice, because hosted alternatives are not viable.

The second is the engineering cost differential. The cost of building and maintaining an eval stack with engineers based in India is meaningfully lower than the equivalent build cost in markets where the commercial platform pricing models were originally calibrated. This shifts the build-versus-buy ratio compared to the assumptions baked into vendor pricing pages. A platform that prices out the case for building in one geography may not in another.

The third is the regulatory trajectory. The Digital Personal Data Protection Act creates an obligation to demonstrate purpose limitation and data minimization, both of which are easier to enforce when the eval stack is under direct enterprise control. The RBI’s FREE-AI Framework adds further substance for regulated entities: board-approved AI policies, AI inventories, lifecycle governance covering model approval, testing and change control, and a defined AI incident reporting protocol. Each of these expectations rests on the assumption that the regulated entity can produce evidence of what the AI did and how it was monitored, which is the eval stack, by another name. Whether those logs sit inside an enterprise’s own infrastructure or inside a third-party platform is a question the regulator has not yet answered definitively, and the conservative posture for regulated entities is to assume the answer will eventually require local control.

None of these factors closes the question. Each shifts the weights.

The Five Questions to Ask Before Deciding

For CXOs working through this decision, the operational questions are not primarily about features or price. They are about ownership and accountability.

Three Viable Paths for 2026

For most enterprises, the working answer to the build-versus-buy question takes one of three forms.

What none of these three forms tolerates is the implicit default of no eval stack at all, with periodic spot checks substituting for systematic measurement. That posture is becoming indefensible in 2026, both technically and, increasingly, in regulatory terms.