From the course: Agentic AI: Build Your First Agentic AI System
Join today to access over 25,500 courses taught by industry experts.
Implement baseline system calibration (CC)
From the course: Agentic AI: Build Your First Agentic AI System
Implement baseline system calibration (CC)
In the last video, we ran this demo, which seems to be running pretty well. Your agent predicts the right department as well as provides a reasoning. Now, are we ready to go to production? Not really. We need to go through the evaluation process or the continuous development process and continuous calibration. Why do we need to evaluate? Because of three things, right? We need to figure out, does your demo handle edge cases well? What's the accuracy across all departments and where does it fail and why? A simple metric to do this would just be to have something called the routing accuracy. What routing accuracy tells us is whether the agent predicted the right department as humans would have predicted it. In order to do this, we've built ourselves a test dataset. Remember that this dataset should come from historical data or it can also come from production if you've productionized this for a small set of use cases. I've created a dataset for you so that it accurately represents…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.