AI coding agents like Cursor, Claude Code, VS Code Copilot, and Windsurf can now access Agenta documentation directly through the Agenta MCP server.
The MCP server implements the Model Context Protocol, allowing AI assistants to search and retrieve Agenta documentation on demand. Instead of manually searching docs, your AI agent can answer questions about Agenta features, APIs, and code examples.
You can now create projects within an organization. This feature helps you organize your work when you're building multiple AI products or managing different teams working on separate initiatives.
Projects provide a way to isolate and organize your AI work within an organization. Each project maintains its own scope for:
This scoping prevents clutter and makes it easy to focus on one product at a time.
You can create a new project directly from the sidebar in the Agenta interface. Once created, you can switch between projects using the sidebar navigation.
Each team member can work in different projects simultaneously. The interface remembers your last active project, making it easy to pick up where you left off.
Projects work well when you need to:
If you're managing complex AI initiatives across multiple products, projects give you the structure to keep everything organized. You can create your first project from the sidebar and start organizing your prompts and evaluations.
For questions about projects or organizational structure, check the FAQ or reach out through our support channels.
The Playground now supports provider built-in tools. You can use web search, code execution, file search, and other native provider tools directly when developing prompts.
Provider built-in tools are capabilities that LLM providers offer natively. Unlike custom tools that you define with JSON schemas, these tools are managed by the provider. When the model needs them, the provider handles execution and returns results automatically.
Common built-in tools include:
Different providers offer different built-in tools:
The tools are saved with your prompt configuration. When you commit changes, the tool configuration is stored with the variant.
When you invoke prompts through Agenta as an LLM gateway, the tools are automatically included in the request. The provider handles tool execution during the call.
Your application receives the final response after all tool calls complete. You don't need to handle tool execution yourself.
Tool definitions follow the LiteLLM format. You can view the exact tool schemas in the Prompt Registry. This helps you understand what parameters each tool accepts and how the provider will use it.
Create a prompt that answers questions using current information:
You are a research assistant. Answer the user's question with accurate,
current information. Use web search when you need recent data.
Question: {{question}}
Enable web search in the tool configuration. When users ask about current events or recent data, the model automatically searches the web for information.
Build a data analysis prompt that performs calculations:
Analyze the following data and provide insights:
{{data}}
Calculate statistics and create visualizations as needed.
Enable code execution for Gemini. The model can run Python code to calculate statistics, process data, and generate visualizations.
Create a prompt that answers questions about uploaded documents:
Answer the user's question based on the uploaded documentation.
Be specific and cite relevant sections.
Question: {{question}}
Enable file search for OpenAI. The model searches through your uploaded files to find relevant information.
Learn more about using the Playground to develop and test prompts with provider built-in tools.
You can now configure reasoning effort for models that support this parameter, such as OpenAI's o1 series and Google's Gemini 2.5 Pro.
Reasoning effort controls how much computational thinking the model applies before generating a response. This is particularly useful for complex reasoning tasks where you want to balance response quality with latency and cost.
The reasoning effort parameter is part of your prompt template configuration. When you fetch prompts via the SDK or invoke them through Agenta as an LLM gateway, the reasoning effort setting is included in the configuration and applied to your requests automatically.
This gives you fine-grained control over model behavior directly from the playground, making it easier to optimize for your specific use case.
We're excited to announce a powerful update to the Agenta playground. You can now use Jinja2 templating in your prompts.
This means you can add sophisticated logic directly into your prompt templates. Use conditional statements, apply filters to variables, and transform data on the fly.
Learn more in our blog post or check the documentation.
Here's a prompt template that uses Jinja2 to adapt based on user expertise level:
You are {% if expertise_level == "beginner" %}a friendly teacher who explains concepts in simple terms{% else %}a technical expert providing detailed analysis{% endif %}.
Explain {{ topic }} {% if include_examples %}with practical examples{% endif %}.
{% if False %} {{expertise_level}} {{include_examples}} {% endif %}
Note: The {% if False %} block makes variables available to the playground without including them in the final prompt.
When you fetch a Jinja2 prompt via the SDK, you get the template format included in the configuration:
{
"prompt": {
"messages": [
{
"role": "user",
"content": "You are {% if expertise_level == \"beginner\" %}a friendly teacher...{% endif %}"
}
],
"llm_config": {
"model": "gpt-4",
"temperature": 0.7
},
"template_format": "jinja2"
}
}
The template_format field tells Agenta how to process your variables. This works both when invoking prompts through Agenta as an LLM gateway and when fetching prompts programmatically via the SDK.
We're open sourcing the core of Agenta under the MIT license. All functional features are now available to the community.
Every feature you need to build, test, and deploy LLM applications is now open source. This includes the evaluation system, prompt playground and management, observability, and all core workflows.
You can run evaluations using LLM-as-a-Judge, custom code evaluators, or any built-in evaluator. Create and manage test sets. Evaluate end-to-end workflows or specific spans in traces.
Experiment with prompts in the playground. Version and commit changes. Deploy to environments. Fetch configurations programmatically.
Trace your LLM applications with OpenTelemetry support. View detailed execution traces. Monitor costs and performance. Filter and search traces.
We've moved development back to the public repository. You can see what we're building, contribute features, and shape the product direction.
Only enterprise collaboration features stay under a separate license. This includes role-based access control (RBAC), single sign-on (SSO), and audit logs. These features support teams with specific compliance and security requirements.
Follow the self-hosting quick start guide to deploy Agenta on your infrastructure. View the source code and contribute on GitHub. Read why we made this decision at agenta.ai/blog/commercial-open-source-is-hard-our-journey.
You can run Agenta on your infrastructure with full access to evaluation, prompting, and observability features. You can modify the code to fit your needs. You can contribute back to the project.
The MIT license gives you freedom to use, modify, and distribute Agenta. We believe open source creates better products through community collaboration.
The Evaluation SDK lets you run evaluations programmatically from code. You get full control over test data and evaluation logic. You can evaluate agents built with any framework and view results in the Agenta dashboard.
Complex AI agents need evaluation that goes beyond UI-based testing. The Evaluation SDK provides code-level control over test data and evaluation logic. You can test agents built with any framework. Run evaluations in your CI/CD pipeline. Debug complex workflows with full trace visibility.
Create test sets directly in your code or fetch existing ones from Agenta. Test sets can include ground truth data for reference-based evaluation or work without it for evaluators that only need the output.
The SDK includes LLM-as-a-Judge, semantic similarity, and regex matching evaluators. You can also write custom Python evaluators for your specific requirements.
Save evaluator configurations in Agenta to reuse them across runs. Configure an evaluator once, then reference it in multiple evaluations.
Evaluate your agent end to end or test specific spans in the execution trace. Test individual components like retrieval steps or tool calls separately.
Evaluations run on your infrastructure. Results appear in the Agenta dashboard with full traces and comparison views.
Install the SDK:
pip install agenta
Here's a minimal example evaluating a simple agent:
import agenta as ag
from agenta.sdk.evaluations import aevaluate
# Initialize
ag.init()
# Define your application
@ag.application(slug="my_agent")
async def my_agent(question: str):
# Your agent logic here
return answer
# Define an evaluator
@ag.evaluator(slug="correctness_check")
async def correctness_check(expected: str, outputs: str):
return {
"score": 1.0 if outputs == expected else 0.0,
"success": outputs == expected,
}
# Create test data
testset = await ag.testsets.acreate(
name="Agent Tests",
data=[
{"question": "What is 2+2?", "expected": "4"},
{"question": "What is the capital of France?", "expected": "Paris"},
],
)
# Run evaluation
result = await aevaluate(
name="Agent Correctness Test",
testsets=[testset.id],
applications=[my_agent],
evaluators=[correctness_check],
)
print(f"View results: {result['dashboard_url']}")
Every evaluation run gets a shareable dashboard link. The dashboard shows full execution traces, comparison views for different versions, aggregated metrics, and individual test case details.
Check out the Quick Start Guide to build your first evaluation.
Online Evaluation automatically evaluates every request to your LLM application in production. Catch quality issues like hallucinations and off-brand responses as they happen.
Online Evaluation runs evaluators on your production traces automatically. Monitor quality in real time instead of discovering issues through user complaints.
Every request to your application gets evaluated automatically. The system runs your configured evaluators on each trace as it arrives.
Configure evaluators like LLM-as-a-Judge with custom prompts tailored to your quality criteria. Use any evaluator that works in regular evaluations.
Create online evaluations with filters for specific spans in your traces. Evaluate just the retrieval step in your RAG pipeline or focus on specific tool calls in your agent.
Set sampling rates to control costs. Evaluate every request during testing, then sample a percentage in production to balance quality monitoring with budget.
View all evaluated requests in one place. Filter traces by evaluation scores to find problematic cases. Jump into detailed traces to understand what went wrong.
Add problematic cases directly to your test sets. Turn production failures into regression tests.
Setting up online evaluation takes a few minutes:
Your application traces will be automatically evaluated as they arrive.
Catch hallucinations by running fact-checking evaluators on every response. Monitor brand compliance using LLM-as-a-Judge evaluators with custom prompts. Track RAG quality by evaluating retrieval in real time. Monitor agent reliability by checking tool calls and reasoning steps. Build better test sets by capturing edge cases from production.
Learn about configuring evaluators for your quality criteria.
The LLM-as-a-Judge evaluator now supports custom output schemas. You can define exactly what feedback structure you need for your evaluations.
Configure the evaluator to return different types of outputs:
Enable the reasoning option to have the LLM explain its evaluation. This improves prediction quality because the model thinks through its assessment before providing a score.
When you include reasoning, the evaluator returns both the score and a detailed explanation of how it arrived at that judgment.
For complete control, provide a raw JSON schema. The evaluator will return responses that match your exact structure.
This lets you capture multiple scores, categorical labels, confidence levels, and custom fields in a single evaluation pass. You can structure the output however your workflow requires.
Once configured, your custom schemas work seamlessly in the evaluation workflow. The results display in the evaluation dashboard with all your custom fields visible.
This makes it easy to analyze multiple dimensions of quality in a single evaluation run.
Binary Score with Reasoning: Return a simple correct/incorrect judgment along with an explanation of why the output succeeded or failed.
Multi-dimensional Feedback: Capture separate scores for accuracy, relevance, completeness, and tone in one evaluation. Include reasoning for each dimension.
Structured Classification: Return categorical labels (excellent/good/fair/poor) along with specific issues found and suggestions for improvement.
To use custom output schemas with LLM-as-a-Judge:
Learn more in the LLM-as-a-Judge documentation.
We've completely rewritten and restructured our documentation with a new architecture. This is one of the largest updates we've made to the documentation, involving a near-complete rewrite of existing content and adding substantial new material.
We've reorganized all documentation using the Diataxis framework.
One of the biggest gaps in our previous documentation was observability. We've added comprehensive documentation covering:
Documentation now includes JavaScript and TypeScript examples alongside Python wherever applicable. This makes it easier for JavaScript developers to integrate Agenta into their applications.
We've added a new "Ask AI" feature that lets you ask questions directly to the documentation. Get instant answers to your questions without searching through pages.
