Turning Claude, Codex and Cursor Into an AI Eval Agent

How to wire Calibrate into Claude, Codex and Cursor so your coding agent can find where an AI product is failing and drive the fixes.

Resources

• Watch on YouTube (1080p)

TranscriptCopy transcript

0:00Hi everyone, today I will be showing how can you connect Calibrate with your Claude Code0:07or any other AI tool of your choice.0:11So Calibrate comes with its own skill files and we have a CLI as well.0:19So the skill files uses the CLI to interact with the API.0:26So just to get started, let's directly paste this within Claude Code and it will install0:33the skills automatically.0:37In my case, I might already have it installed, but it works anywhere.0:43You can use Claude Code, the terminal, or you can use it within Cursor or Codex.0:48We have commands for each of them.0:53Okay, now once you've installed it, you can just say, login to Calibrate.1:01So what is needed here then is for us to get an API key.1:06So I can go to my workspace, I can generate an API key.1:17I will be deleting this for now, this works.1:20So as you can see here, now Claude Code is trying to install the Calibrate CLI.1:27Now you can see it is asking me where am I trying to sign in.1:31So the hosted version is on ArtPack, but if you have a self-hosted version, you can select1:37this as well.1:38For now, I will just select this because that's what I'm using for the demo.1:45And okay, so it's asking me to run the login command.1:54Now once I have pasted it, I can say that I have done it.1:58Once I have done it, it will first verify by itself if the authentication is really2:04complete.2:05You can see that it is saying that I am logged in now.2:07So Calibrate has been connected to Claude, it's done.2:11Now let's see what can we do with this.2:15Let me open one of my agents and let me ask it to analyze a particular error.2:25Can you analyze the errors and tell me what to do.2:38Now Claude will connect with Calibrate using the Calibrate CLI, it will pull in the data,2:43it will look at the errors, it will try to find if there is any pattern and it will let2:48me know what is going wrong and what can I improve.2:52Let's wait for it to come back.2:53It has analyzed my seven failures here and it says that six of those seven failures are3:00because the agent is saying the same thing twice.3:04So somehow my agent is repeating itself and for the remaining case, actually the agent3:11was right and the LLM judge was wrong and we'll come to this later on and yeah it has3:19given me a very concrete recommendation for what I need to add to my config or to my prompt3:24and the fact that I need to fix my judge.3:28It can actually, as you can see, it is saying that I can do both of these and rerun because3:34it is connected to your code.3:35So it can run this change and for fixing the judge, it can again use the Calibrate CLI3:41to create a new prompt and update it and we'll cover that shortly.3:46Now let's see what else can we do.3:48As you saw that LLM judges could itself be wrong.3:52Here is one example where my LLM judge is only 73% aligned with my human experts.3:59Now I can look at the disagreements, I can look at the reasoning given by the LLM judge4:04by the human for each row and try to figure out why are they not agreeing.4:11But instead of that, I could just have Claude analyze this.4:15So I can just say analyze the disagreements here and tell me what is going wrong.4:24So this will again pull the data.4:26It will read the reasoning and the labels given by the LLM judge and by each of the4:33humans, especially where they are disagreeing.4:37Based on that, it will identify if there is any pattern to their disagreements and suggest4:44fixes.4:45So again, let's wait for it to come back.4:49Claude has completed analyzing it and you can see that it found that there are three4:53disagreements and it has found the root cause that the judge is unnecessarily penalizing5:03the agent response and so the mistake is in the judge and it has already figured out what5:09is the fix and it asks me to update the prompt of the judge itself.5:16So it will automatically update and create a new version of the prompt for the LLM judge5:23based on the disagreement analysis here.5:28Let me ask it to do that, create a new version.5:36Okay now you can see that it created a v2 of this prompt.5:40So let me show you the initial version.5:42This was the initial version of the prompt and the model.5:47It kept the model the same, but just added another line to this and let's see how this5:55performs.5:56When I rerun my LLM judge on the same dataset, now my human agreement is 100 percent right.6:06So this way I was able to use Claude code to analyze errors, get suggestions, analyze6:14disagreement with humans, identify why the prompt needs to be updated or how it needs6:20to be updated, go ahead and do it and rerun and see that my agreement has improved.6:28We can do a lot more with Claude.6:30So if you can come to the docs and see that all the APIs available here are capabilities6:38that Claude has.6:40So for example, creating LLM judges, improving them, creating tests, creating annotators,6:48assigning annotators to jobs, labeling jobs, analyzing labeling jobs, running evaluators6:53on those labeling jobs, creating agents, connecting tests with agents.7:00So you can do a lot of things with Claude.7:03Essentially if your first instinct is I don't want to use the UI, can Claude do it?7:08Most likely it can.7:09We have kept all the core features available for Claude to use directly.7:16And if something doesn't work, let me know, reach out to me and we'll figure it out.7:22But yeah, that's a quick demo of how to use calibrate with Claude.7:25Thank you.