Model Eval for Leaders

An overview of AI model evaluation within the 4-level framework, for leaders who care about real developmental outcomes, not just engagement.

TranscriptCopy transcript

0:00Hello, my name is Aman Dalmia, and this is a recording of the AI evaluation workshop0:11we had a few weeks back.0:13Due to some mistake from my side, we couldn't have the recording, so here we go again.0:20Before that, I work as an AI engineer at Agency Fund.0:24We are a funder-doer.0:26We fund evidence-based interventions and embed our technical and data science teams within0:34our grantees to help them achieve cost-effective impact at scale.0:39I have been working on using AI in the social impact space for the last eight years.0:45I have spent time at Vatwani AI working on agriculture and healthcare, Avanti Fellows0:52building open-source learning products for government school students.0:56I have also spent time building legal AI assistants at Hyperverge and AI tutors for Hyperverge1:03Academy for underprivileged children.1:07And for the last few months, I have been building a co-pilot for nurses at Noora Health.1:14And currently, I split time between the Agency Fund and ArtPark.1:18And at ArtPark, I work on using AI for public health, solving public health issues,1:26and also building evaluation infrastructure like a tool that I've built is called Caliprate,1:32and we'll be looking at that today as well.1:38So we all know that AI is everywhere.1:41We hear that models are constantly growing, costs are reducing, they are doing more things.1:48But at the same time, we are seeing that a lot of AI pilots are not producing the expected impact.1:57So what are the reasons for that?1:59The common challenges that we see, especially in the social impact space,2:03is failing to measure the real impact beyond demos,2:08finding it hard to integrate AI into real workflows,2:12building up the capacity of internal team members, and adapting the organization for this change,2:20ensuring good data quality, and frankly, evaluations,2:26benchmarking how these AI models perform in your context.2:32To help with that, Agency Fund has developed this four-level evaluation framework2:37that helps organizations understand how to perform AI evaluation in the social impact space.2:43The four levels essentially cover different aspects of any AI product in the social impact space.2:50The first level starts off with measuring, is the AI system performing as intended?2:55Level two checks whether the overall product engages the users and retains them.3:01Level three is used to measure whether the product positively impacts users' thoughts3:09and moves them towards desired behaviors.3:13Finally, level four looks at the final development outcome,3:17like learning outcomes or farmer income, etc.3:24For example, we will look at this case study from3:28FarmerChat, which is a product built by Digital Green,3:32essentially, which gives advice to farmers.3:36In this, for this product, level one looks like checking if the AI model gives the correct3:42local farming advice.3:44Level two looks at whether the farmers open the app on a regular basis,3:50and do they keep coming back and asking more questions.3:54Level three looks at whether the farmers adopt those recommendations,3:58and level four looks at whether the recommendations that they adopted finally4:04increased yields or improved incomes.4:09You can use this QR code to go and read the evaluation framework.4:15This particular talk will be focused on the first level,4:19whether the AI system performs as intended.4:22Before we dive in, let's, I, let's discuss if,4:40before we,4:41before we,4:48so what is really model when we,4:52so what do we mean by model in model evaluation?4:55So model can mean anything from single LLM features.4:59For example, if you want the LLM to translate a given sentence from English to Spanish,5:06or if you want it to summarize a given conversation,5:10or given some unstructured data like a document,5:14if you want to extract it in a structured output,5:18all of these are single LLM features.5:21Essentially, you give the LLM a very concrete thing to do.5:25It takes one input and gives one output.5:28The second step is creating a workflow where instead of just one LLM call,5:34you might have a sequence of LLM calls working together.5:38For example, given a query, you might first check if it's a toxic query,5:43or if it's a harmful query that you want to reject.5:46If not, then you might go ahead and translate the query from Spanish to English.5:55Then you might use another LLM to then respond to the query in English.6:00Then you might use another LLM to finally translate it back to Spanish.6:04In this way, you orchestrated multiple LLM calls one after the other6:09to produce the final output.6:11Now, agents are LLMs.6:15LLMs, when they take actions, are what we call agents.6:20They decide their own trajectories.6:22They plan what should be done.6:24They invoke tools.6:25For example, they might conduct web search.6:28They might query a database.6:29They look at the result.6:31They decide what needs to be done next.6:33They take the action.6:34They plan the next steps and decide by themselves when they are done6:38and produce the final output.6:40We call this form of LLMs agents.6:44And when we call model, when we say model evaluation, we mean all of these.6:50We are currently calling all of these model in model evaluation.6:57So, before we go ahead, it is important to consider7:01or important to ask, why is model evaluation even needed?7:06When we had this session online, then we had a lot of interesting points by the audience.7:12And I want to share some of them here.7:16People shared that without evaluation, we can't know if AI works as intended.7:21Different models and their versions behave differently.7:24Even their bias and legitimacy varies.7:28There is a risk of creating perverse incentives.7:33The models may not be tailored to the specific use case or to the user themselves.7:38The model may not have domain-specific context.7:43There could be the quality of the training data.7:52The way that these models are trained and the data used to train them7:56and the way these models are trained.8:02The way these models are trained and the data used to train them8:05can all introduce biases and different failure modes.8:10These models are often trained on high-resource environments8:14or data coming from high-income countries.8:17High-resource environments or data coming from high-income countries8:21and may not be suited well for low- and middle-income countries.8:27An advice given by an AI might be technically correct,8:30but it may not be understandable for the end user, which could be, for example, a farmer.8:36And the user may have given,8:44we might have prompted the model poorly, which could be causing the model to fail.8:49The vocabulary that is being used by the model may not be comprehensible to certain users.8:54And at the end, AI is probabilistic and it can easily lead to hallucinations.9:05So, let's unpack a little bit what we mean.9:15These are all amazing points and all of them are true to certain degree.9:22These are all amazing points and all of them are true.9:28But before we move ahead, I think it can help to understand a little bit9:32what we mean by AI being probabilistic and what do we mean by hallucination.9:37To understand that, it can help a little bit how these AI models really work.9:43So, the way these models are trained is they are trained to predict9:48the next word as they are speaking.9:51What I mean by this is, suppose you take a Wikipedia page of a person9:55or you take a single sentence like the cat chases the rat.9:59The way these models are trained is it would be given a partial sentence like the cat chases the10:07and it would be asked to predict what should be the next word.10:11Now, as humans, we know it should be that.10:13But when the model begins, it has no knowledge of the world.10:17It doesn't understand what is a cat, what is a rat, what is the relation between these.10:22But it has a lot of these internet data, like we have a lot of textual data on the internet10:27where we have embedded the knowledge of the world.10:32And so, these models are trained so that even if it says, for example,10:38the model might start off by saying the cat chases the school.10:42Now, that makes no sense.10:44But that's how the model is at the start.10:46It doesn't know that the cat has no reason to chase a school, for example,10:51or the fact that the cat is an animal.10:54And so, there can be.11:03So, there will be a massive gap in the.11:12But over time, but when model makes this error, we give it the feedback that,11:18hey, this is wrong, that it makes no sense.11:22And we give it an example of what should have been the right answer.11:25Say that it should have been the cat chases the rat.11:29Over time, we give it hundreds and thousands and lakhs of these sentences present on the internet,11:36which reflect our world models.11:39Right.11:39And over time, it starts getting better.11:42For example, in a few iterations, it might start saying the cat chases the human11:48or the cat chases the person, which is a little bit more logical than a cat chasing a school.11:54At least there is one animal chasing another animal.11:58But still, cat chasing an animal hardly makes, cat chasing a person hardly makes sense.12:04So, with more feedback, with more training over time,12:07it will become much better and eventually start saying the cat chases the rat.12:13All to say, the way these models work is they take all of the text present on the internet12:19and they are trained to predict the next word.12:22They are given partial sentences and trained to predict the word that comes after it.12:28And so now you can imagine that majority of the text on the internet is English.12:34And so a big part of these, the training data used to train these models is in English.12:40Right.12:40And so these models are inherently much better and even more efficient at English12:46compared to other languages which are not as represented.12:51So, languages like French and English are considered high-resource languages12:57because they are represented on the internet much more than low-resource languages13:03like Marathi or Konkani, for example.13:08And they do suffer from a lot of these mistakes that everyone else has pointed out as well.13:17And as I was saying, the models also have a knowledge cutoff.13:22What I mean by this is that they only have access to data until a particular date.13:29When these models are trained, all of the data on the internet is scrapped until that date13:35and that is used to train the model.13:37So, these models by themselves don't have access to what happened yesterday13:43or any latest developments.13:45Right.13:45Their knowledge is static.13:48Also, they don't have access to your context.13:50If you are working in a company and you want AI to work to answer queries13:56based on your company documents or your company knowledge,14:00unless you provide that context to the AI separately, it does not have access to it.14:06And it cannot answer it.14:09Also, even though the model may be capable, we may not be giving it the right instructions.14:14And I'll cover an example of that.14:17Similarly, AI may not be the right tool for every given task.14:23For example, if you look at a big mathematical calculation,14:29if you ask the AI to do it because of this nature where it predicts the next word,14:34it is not really calculating that mathematical operation.14:41If you give it an absurdly huge number like 5 5 9 9 9 into 2 2 2 2 2 9 9 9,14:48it will predict what should be the first word of the final output.14:54It is not actually doing any computation.14:57And so, it is very, very likely that it will give you a wrong answer.15:02And this is something that a calculator can do very, very easily.15:06Right?15:07So, I've also seen a pattern where people try to throw LLMs to every single thing out there.15:13But if a query requires you to do a calculation,15:16then asking the LLM to inherently do the calculation15:20is not the right thing because it can make mistakes.15:25Now, for example, some of you may already be aware.15:29So, these models, if you are given a task to compute this massively complicated15:39multiplication like 2 2 2 2 2 9 9 9 into 5 5 4 4 4 3, something like that,15:45you can in principle do it if you are allowed to do the multiplication steps properly.15:53But if I asked you to give me the answer directly without thinking for even a second,15:59if I tell you this problem and I tell you give me the answer without thinking,16:03it is very likely that you will make a mistake as well.16:06It is not because you cannot do multiplication.16:09It is just that you were not given the adequate time to do the multiplication yourself16:17to arrive at the answer.16:19Similarly, a lot of times these models have a lot of capability as well.16:25But we end up asking them to give the answer directly.16:29However, if we give them time to think through step by step and arrive at the answer16:37in a logical step by step manner, they end up performing way better.16:43I have personally seen massive accuracy gains16:46by simply giving the model more quote unquote time to think and letting it16:53follow a step by step process to arrive at the answer16:57instead of forcing it to give me the answer directly in the first go.17:03That is what I mean by us not giving it the right instructions17:06or us not giving it enough time to arrive at the answer.17:11Even our instructions could be very small.17:14They could be vague.17:15They could be confusing.17:16They might be conflicting.17:17We might say something at the start of a prompt,17:20but say something completely different at the end.17:23It is similar to the problems of communicating with another human.17:27If we are not clear, if we are not detailed enough,17:30it is likely that they will follow some other interpretation that we didn't have in mind.17:35So, our instructions could be wrong.17:38And even if the accuracy of AI is as good as we need it to be,17:43it is possible that it may be just too slow.17:46It's possible that given a user query and given adequate time,17:50a model could research all over the internet17:53and come back with the right answer in an hour.17:55But maybe for your context, that might be too slow.17:58And that is also a failure mode.18:01And these models could potentially be very costly as well.18:04Depending on the model that you are using,18:06how you are using it.18:08And so, for a model to succeed, it has to also be cost-effective18:12based on whatever your definition of cost-effective means.18:27Some other questions that others have asked during the call18:32was, does training on our own model?18:37Some other questions that were asked during the session was18:40whether training on your own model, on your own data, reduces these problems.18:46But that is not, number one, that is not trivial to begin with.18:51Even if you get the right data, getting it to work, hosting your model,18:55deploying it in a way that makes sure that the speed of talking to that model is fast,19:01is a very hard problem.19:05A big reason that a lot of people end up finally using off-the-shelf models19:09like OpenAI, Cloud, or Gemini is because they work very well out of the box19:15in terms of speed and cost.19:18And even with the right prompt, they can get to the right answer for your context.19:26And in some cases, it may not.19:28If your use case requires a low resource language or it's a very niche domain,19:35it may not be possible to get these models to work.19:38But I would not start with that assumption.19:41Of course, it's not a default that they will work as easily.19:47But that doesn't mean that there aren't ways to get them to work.19:52So I would not suggest jumping to training your own models, even if you can,20:01because hosting it, maintaining it, continuously improving it is all a challenge.20:08Tomorrow, OpenAI might come up with a much better model with a bigger data set,20:12or Cloud might come up with a new capability, and you can switch it for hardly any cost.20:20Versus if you maintain your own model, you might spend months doing that,20:24and the next update by any of these providers might completely render all of your effort20:29useless.20:31And that risk is real.20:35Yeah.20:35But it's, again, not to say that it's not possible to do it.20:40If you have the technical capability and none of the existing models20:44are close to working for your use case, it may be worth doing it.20:48But the problem of evaluating your AI model is still true.20:53Even if you are maintaining your own model,20:55you still need to evaluate if that trained model works well.21:05OK, and then the next point of our discussion was,21:09why has model evaluation been hard for people?21:12Again, a lot of interesting responses by people.21:16One of them said that it was very hard to operationalize21:21AI evaluation as an ongoing task.21:23The responsibility often fell on CTOs who are overly overloaded.21:29And there's no clear definition of when evaluation can be considered done.21:34And there is even no practical checklist or tool or asset to get started.21:39It's often difficult to determine what good looks like.21:43Once you start building, they find it hard to slow down and ask these hard questions.21:49Often, people struggle with evaluating non-deterministic outputs,21:52which are often the case with these generative models now.21:55Even two same prompts may produce two different outputs22:00with the same model, making the evaluation results unreliable.22:05And once the failure modes are identified, some of them don't know how to do it.22:11It's not clear how to act on them quickly and make fixes.22:17And again, super helpful, very accurate.22:22But the key here is to not aim for perfection.22:26The solution is iteration.22:28We have to start small, quickly iterate, and keep growing,22:32like setting up a process where you are evaluating what you are doing,22:37where you are evaluating what you are doing, even if it is as a slower, if it doesn't.22:58Again, all of these are super helpful.23:00And the only thing I can add is that we don't need to look for perfection23:05or waiting for the evaluation to be done.23:09The key idea is iteration.23:11We need to start small, build a small dataset,23:14evaluate how whatever our current system is works on that dataset,23:19find the failure cases, identify why those failures are happening,23:23and keep improving our prompts or our AI pipeline.23:27And as we deploy it out, we gather more edge cases,23:30newer edge cases, bring it back to our evaluation dataset,23:34keep growing it, and keep evaluating and improving step by step.23:38Iteration really is the key.23:42And another question was whether evaluation is needed at all23:48if one restricts AI to their own dataset, for example, using a RAG setup.23:56Now, if someone can give their own documents,24:00they are asking if evaluation is still needed,24:04if we are restricting AI to those documents only.24:07Now, this assumes a lot of things.24:09Firstly, we are assuming that the model will stay restricted to that context.24:15We have given it some context.24:16We have given it a few documents.24:18But we are assuming that the model will only stay restricted within that context.24:25That needs to be validated.24:26We cannot assume it.24:28Secondly, the right answer for a given query24:32may be somewhere hidden inside a thousand-page PDF.24:35And it is possible that the model may not find it, may not retrieve it.24:41Even if the model retrieves a particular paragraph from that PDF that is relevant,24:46it may not perform the right reasoning required to arrive at the right answer.24:51It may have the right source.24:53It may find the right paragraph.24:56But it may reason itself to a wrong answer.24:59So every step needs validation.25:01Yes, using your own documents reduces the probability that it will make a mistake.25:08But there is still a big scope for error.25:12So evaluation doesn't go away.25:14Now, before we jump further, there is a tool that we have built, that I have built as25:44a hands-on exercise.25:50Now, before we proceed, there's a hands-on exercise that we prepared.26:01Before we move ahead, we have built a hands-on exercise.26:06Before we move next, we have a hands-on exercise that we have made.26:12This is a tool that I have built.26:14It's an open-source, no-code evaluation platform that will help solve a bunch of issues that26:22came up for people, especially being able to do it themselves, being able to identify26:28the errors, and then fix and act on those errors.26:32You can use this QR code or go to this link directly.26:38I will now show how this looks.26:41Holy fuck.