Gentrace focuses on systematizing the development of large language model (LLM) applications, self-described as facilitating streamlined application development.

Gentrace

Gentrace | 1,665 followers on LinkedIn. Testing and monitoring tailored to your generative AI application. | Testing and monitoring tailored to your AI app. Get from MVP to production. Gentrace systematizes LLM app development.

GTM Engine o11y OG

Jeff Cestra

Co-founder, COO at Gentrace

Daniel C. Liem

Co-founder, CTO at Gentrace

Vivek Nair

Account Executive

Nick Enthoven

Gentrace is expanding our team. We're looking for senior software engineers in NYC and SF who want to build tools to help the most advanced companies in the world make their generative AI systems reliable and predictable. Learn more at https://gentrace.ai/eng

Most engineers approach LLM-as-a-judge all wrong. The usual high-level metrics like hallucination or safety rarely tell you if your app actually works as intended. Even with human evaluators, general metrics won’t help them judge your app’s performance in a useful way. Good evals are built on something specific to your app—a unique “unfair advantage” that gives the LLM clear criteria. LLM-as-a-judge is a widely used approach where an LLM grades another LLM’s output. But without an unfair advantage, it can fall short in quality. LLM-as-a-judge problems: • Circular reasoning: How can an LLM grade what it itself generated? • Poor initial results lead teams back to vibes and manual grading. Imagine an LLM app is tasked with writing emails. A bad eval would ask the LLM to rate itself on how well it followed the prompt—a circular question that adds little signal and won’t help improve the model. Instead, give your LLM unfair advantages. Here’s how: Add tailored asserts: specific criteria your model should meet. For instance, in our email example: • No footer • Directly asks a question • Includes recipient’s email Another approach is comparison to a reference output. You provide a high-quality reference email for comparison, so the LLM can check for: • Missing or extra information • Conciseness vs. verbosity To build reliable AI evaluations, always ask: how can I create an unfair advantage for the LLM These targeted methods ensure your LLM-as-a-judge evals give you actionable insights into the quality of your app. At Gentrace , we’re making it easy for AI engineers to build their own high-quality evals. My unfair advantage blog post shares more examples for using LLM-as-a-judge effectively: https://lnkd.in/gavtDhCY

Doug Safreno

New release, featuring production evaluator graphs and local evaluations / local test data. Production evaluator graphs Production evaluators now automatically create graphs to show how performance is trending over time. For example, you can create a "Safety" evaluator which uses LLM-as-a-judge to score whether an output is compliant with your AI safety policy. Then, you can see how the average output "Safety" trends over time. Local evaluations / local test data Gentrace now allows you to more easily define local evaluations and use completely local data / datasets. This makes Gentrace work better with existing unit testing frameworks and patterns. It also makes Gentrace incrementally adoptable into homegrown testing stacks. More: • User-specific view settings can be saved and overridden from a URL • Filter test runs by their input values • Added explicit compare button • Pinecone v3 (Node) support • o1 support • Fixed 68 bugs

Quizlet takes unstructured text and builds flash cards, syllabi, and other learning tools for students with generative AI. With Gentrace, they increased testing 40x and reduced test duration to 1 minute. Learn more: https://lnkd.in/gF_Zi7Cd

Faire built an AI agent that reviews PRs. They use an AI evaluator in Gentrace to review the reviewer. Learn more about how Faire systematically develops their AI features in their blog post: https://lnkd.in/gf_arpB9

New release, featuring datasets. Datasets Gentrace now supports datasets, which organize test data into separate groups within a pipeline. This is particularly helpful when multiple engineers are working on the same pipeline and want to test different pieces of functionality. Test result settings memory Settings in the test result pages (eg hiding or re-ordering fields) are now saved across results in the same pipeline. This makes it easier to see exactly what you want (and only exactly what you want), without redoing your work every time. More: • Performance improvements on the live view • Tweaked new user experience during pipeline setup • Added bulk archive functionality for test results • gpt-4o-mini supported as an evaluator • Fixed 37 bugs

New release, featuring revamped & realtime test results. Gentrace has three new views to facilitate better testing: 1. Aggregate comparison The new aggregate view shows the statistical differences between different versions of your LLM-based feature. 2. List of test cases with evaluations The new list view shows all of the scores for different test cases. When doing a comparison, it highlights what changed between the different modes. 3. Drilldown per test case The new drilldown view presents a clear picture of the output(s) for a specific test case. It includes the JSON representation, evaluations, and timeline. Other changes • Performance: Gentrace is much faster in many views • gpt-4o supported in playground beta & as a grader • Processor edits show which evaluators will be affected • Improved human notes experience • Fixed 83 bugs

New release, featuring comparative (head-to-head) evaluators, classification evaluators, a better code editor with Python support, creating test data from live runs, and more: Comparative (head-to-head) evaluators Gentrace now supports evaluators that compare two outputs for head-to-head types of analysis. Use them to determine which of two experiments is better using an LLM or heuristics. Classification evaluators Gentrace now supports classification evaluators that do a simple comparison of classification outputs and generate more robust aggregate metrics like precision, recall, and F1. Better code editor and Python support We've upgraded the Gentrace code editor to dramatically improve the code writing experience. Gentrace also now supports Python for writing processors and evaluators. Create test data from live runs Gentrace supports creating test data directly from your live runs. When you create test data from live data, Gentrace allows you to edit the data and choose the pipeline to which it gets saved. More: • Performance work • Revamped / more filters • Prod eval daily AI limit is now per evaluator • Evaluator outputs stream during testing • Save test case data on blur • Boolean metadata type • CSV export of results includes expected values • Fixed 73 bugs

New release, featuring image evaluation, evaluator templates, ISO 27001 compliance, new blog / docs, and more: Image evaluation Gentrace now supports evaluating images using GPT-4 with Vision. Example use cases: • Validate that a generated webpage has certain visual characteristics • Compare a generated image to a baseline for higher quality Evaluator templates Gentrace added evaluator and processor templates. If you create the perfect factualness check or a very useful conciseness evaluator, create a template and share with your organization. ISO 27001 compliance Gentrace has completed our audit and been certified as ISO 27001 compliant. New blog & docs To share our thinking on best practices in generative AI development, we've created a blog. We've also re-organized our docs and added an AI-powered search. More: • Grader role: can only manually grade outputs • During evaluator testing, streaming outputs • Rerun missing, errored, all evaluations • Collapse left bar • Filter live data by metadata • Download live data as CSV • Fixed 34 bugs

We recommend 4 main approaches in detecting AI hallucination: • Automated comparison to an expected value / ground truth • Automated specific hallucination checks • Expert review • End-user feedback Learn more about these strategies on our blog: https://lnkd.in/grWA6tKi

GENT

AI:GENTRACE

Gentrace