Why Agent Observability !== Agentic Analytics

Agents were the buzz of 2025 (and probably the hot topic of this year too). As businesses start to contemplate integrating them into their operations, onsite customer-facing agents are poised to be the next interaction layer between businesses and consumers.

While in the past customers may have interacted with your website/mobile app while pondering your business proposition, businesses are now starting to prototype customer-facing agents hosted on their own digital estate to better retain control of their experience, optimise the customer journey, help solve customer frustrations, and potentially nudge customers to spend some money.

These customer-facing agents are now an abstraction layer between your customer and you as a business, becoming a new customer touchpoint and with that, opens a world of new possibilities and challenges.

What are agents?

So let's start off with defining what an agent actually is. Like all definitions in the world, everyone has their own definition of an agent, but there are some common themes.

Google defines AI agents as "software systems that use AI to pursue goals and complete tasks on behalf of users. They show reasoning, planning, and memory and have a level of autonomy to make decisions, learn, and adapt".

AWS on the other hand defines AI agents as "software programs that can interact with their environment, collect data, and use that data to perform self-directed tasks that meet predetermined goals. Humans set goals, but an AI agent independently chooses the best actions it needs to perform to achieve those goals."

Last but not least, Anthropic defines agents as systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

So in short, agents are called agents because they have the agency to decide how best to achieve a predefined goal on behalf of users independently, without having to follow a strict one-directional workflow.

The current challenge

The current biggest challenge for businesses is working out how to move agents from a PoC phase and prototype environment into a real-world environment filled with unexpected edge cases and how to build guardrails to ensure that customer-facing agents perform as expected. The industry is currently flooded with platforms, frameworks and different approaches on how best to build and deploy agents (which are all important topics).

The less talked about discussion however is what happens after you have deployed an agent once it's out there in the wild. You probably want to know if the agent is working as designed, outputs are expected and that it hasn't just gone off the rails and given your customer a massive discount that is now eating into your profit margin. You also want to ensure there is governance and oversight into how an agent performs to ensure there are no compliance risks and that the customer experience is not adversely impacted. This all sounds reasonable right?

What can be used to solve this challenge

There are already quite a few mature and sophisticated agent observability platforms and tools like LangSmith and Langfuse which are designed to help monitor and optimise agent performance. These platforms do what they say on the box, they allow you to observe how your agent is doing.

Now if we go back to the start, agents are becoming an additional middle abstraction layer between the customer and you as the business. So do these Agent Observability tools allow you to understand your customer intent, their current sentiment and what they are actually looking to do right here and right now? Short answer No. These Agent Observability tools are designed to help you monitor the end-to-end behaviours of your agentic system, such as what LLMs or external tools the agents access. 

Fundamentally agent observability is about "tracing" and evaluating the agent's performance by looking at its internal state and telemetry data to better understand how its operating. Common dimensions and metrics returned from these tools include request logs, prompts used, model's responses, token usage, reply latency and any external tools used.

Here is an example of what these structured logs look like from Langfuse:

While these observability tools definitely have their place in the world, they are designed for developers, ML engineers and data scientists to build better agents. What they lack however is providing an avenue to let the business team tell stories about their customers and how wonderfully awesome their agent(s) have been in influencing the customer journey.

Some things change, some things don't…

For those who have been around the neighbourhood for a while, you probably know what digital analytics is. Digital analytics is the process of collecting, measuring, analysing and interpreting data from across your digital estate to understand user behaviour so you can achieve business goals so that ultimately you can call yourselves "data-driven".

Before the explosion of the modern digital analytics platforms today, how did businesses try to understand their customers?

They did this by analysing log files on a web server and this is where the concept of page views and sessions were first introduced. The issue with log analysis is while it tells you how requests reach your website, they don't really tell you much about how a customer is interacting with the website content itself and lacks granularity when it comes to better understanding customer behaviour. In the same vein that you wouldn't use logs to do digital analytics, then agent observability tools are probably not the best-suited for understanding customer interactions with agents and so maybe a new paradigm is needed.

How the new paradigm of "Agentic Analytics" could look like

Let's call this new paradigm "Agentic Analytics" (please don't call it AA) and the purpose of Agentic Analytics is not necessarily to determine how performant an agent is but more about understanding how interacting with the agent is influencing the brand perception by the customer and whether it is helping to improve or deteriorate their current experience. 

And to be honest, no one really knows how Agentic Analytics should be implemented or even work in principle - best practices here are still to be determined. My hypothesis however is that Agentic Analytics will have some of the following core themes (with probably more added as things evolve):

• Same KPIs: Business KPIs will likely stay consistent but the way that business goals are achieved or business models may change. For example, an ecommerce retailer selling sports shirts will probably continue to sell sports shirts but they may do so through new agentic-based channels. But at the end of the day their KPI is probably still going to be related to the business’s revenue as a north-star metric.
• Structured Data: Customers interacting with agents will likely be interacting with the agent through a text-based UI at least for the short-term future. Personally I still think a text-based UI is a horrible and unintuitive interface and do humans even like knowing that they are interacting with a machine? Regardless, agents will produce a lot of rich and qualitative data without bounded scopes of what may be captured in an interaction. Unstructured data is poised to become the new goldmine for analytics but in order to be easily digestible and useful in analysis, the unstructured data captured will need to either be in a predefined schema to provide structured format or alternatively require more advanced qualitative analytics techniques such as NLP analysis.
• Inferred Meaning: Certain metrics such as transactions and revenue will continue to be the same while new metrics may emerge (agent impressions vs agent sessions started perhaps?). Lots of analysis will also probably not be a direct metric but inferred from the data. For example, identifying sentiment and intent based on the given prompts by a customer "I want to cancel my subscription" which means you should probably intervene with some better offer or improve inventory prioritisation and management if a prompt like "when would this product be back in stock" is given by a customer? Lots of creative ways to serve customers better or solve customer challenges and pain points is likely to be inferred from the analytics as opposed to quantitative-based measurement methods.
• Single Customer View: Memory and context is the new king and building a rich profile of a user will be key to helping agents better service customers and personalize the agentic experience to be more relevant. Not that much different to the 360 unified customer view that majority of businesses try to build today. The challenge here is how to effectively pass this customer data quickly and securely to agentic systems, whilst having the capability to constantly refresh with additional context at the same moment a customer is interacting with an agent.
• How to instrument your agentic analytics: It's not likely that a tag management system will work as it's quite hard to manually trigger tags from within an agent so agentic analytics is likely to be natively integrated with the internal workings of the agentic system as a native SDK integration. Similar to the notion of Structured Data, providing strict schemas for the event tracking not only ensures structured conformance but is very coincidentally quite machine readable. Agents are extremely good at understanding machine-readable schemas so this may even allow agents to proactively use the SDK as another external tool to automatically instrument the tracking, without developers needing to think of every use-case and manually instrument every custom event type. This allows the agent to have autonomy to decide when to track what’s needed but still ensure the tracked data is in a predefined format and structure. Autonomous and self-governing tracking anyone?
• Privacy Compliance: Customers providing unsolicited PII data and unstructured qualitative data sounds like a perfect combination for a publicity disaster or pose a serious compliance risk so technical guardrails will still need to apply. Consent to track will very likely moving forward as per the status quo and new legal regulations are likely to emerge in the future posing additional restrictions on how digital analytics should be implemented in agentic systems.

All in all, nothing is set in stone yet and things are moving fast in this space! More to come...