Evaluating Productivity Gains

The AI productivity paradox is real — and the data is in. A new study from the Federal Reserve Banks of Atlanta and Richmond, Duke University, and NBER surveyed nearly 750 CFOs about AI's actual impact on their businesses. The findings challenge both the hype and the doom narratives. Adoption is accelerating: 58% of firms invested in AI in 2025. By end of 2026, that jumps to 85%. But here's the paradox: CFOs perceive 3% productivity improvements, while revenue-based measures show just 1.8%. We feel the impact before it shows up in the numbers. The most striking finding? The strongest productivity gains aren't coming from cutting headcount or costs. They're coming from developing new products and reaching customers more effectively. On jobs: aggregate employment impact is just -0.4%. But the composition is shifting — routine clerical roles down 2 percentage points by 2028, skilled technical roles up 1.4 points. The study's authors — Salomé Baslandze, Zachary Edwards, John Graham, Ty McClure, Brent H. Meyer, Michael Sparks, Sonya R. Waddell, and Daniel Weitz — summarise it well: "The revenue productivity gains associated with AI investment operate largely through innovation, product-market and demand-side channels." If your AI strategy is primarily about cost reduction, you may be leaving the bigger opportunity on the table. What's your experience — where are you seeing AI gains that haven't yet shown up in the metrics? Source: NBER Working Paper 34984 (March 2026) #AIProductivity #FutureOfWork

The AI Productivity Paradox: We're Measuring the Wrong Things Everyone's asking: where are the productivity gains from AI? Companies spend millions on AI tools. Workers use ChatGPT daily. Yet productivity numbers stay flat. Maybe we're looking in the wrong places. Traditional productivity metrics count widgets per hour. Emails sent. Lines of code written. Reports generated. But AI doesn't make us produce more widgets. It changes what work looks like. One Developer writes fewer lines of code now. Is that less productive? Not when those lines solve harder problems. The junior analyst generates reports faster. But she spends saved time on strategic thinking we can't easily measure. Knowledge work productivity was already hard to quantify. AI makes it impossible with old methods. We measure activity, not impact. Speed, not quality. Volume, not insight. A lawyer using AI reviews contracts in half the time. Productivity doubled? Or did she just free up time to handle more complex cases that don't show up in simple metrics? The real productivity gains might be invisible: Faster iteration on creative work Earlier error detection More time for strategic thinking Reduced cognitive load on routine tasks Here's the uncomfortable possibility: AI is working. We just don't know how to measure knowledge work properly. Maybe the productivity paradox says more about our measurement tools than our technology.

The report titled "Estimating AI productivity gains from Claude conversations" by Anthropic, released in November 2025, provides valuable insights into the impact of AI on productivity. Some highlights: AI is applied to substantial work: The median task handled with Claude would take ~1.4 hours without AI, indicating use on meaningful professional tasks rather than trivial micro-work . Time savings are large but uneven: Median estimated savings are ~80–84%, concentrated in reading, synthesis, and writing tasks; tasks requiring physical presence or quick expert judgment see much smaller gains. Higher-wage roles benefit more: Management, legal, and analytical occupations both use AI on longer tasks and capture higher economic value from time saved, amplifying productivity effects. Productivity gains are highly concentrated: Software developers, managers, marketers, customer service reps, and teachers account for most of the estimated economy-wide impact, while sectors like construction, restaurants, and in-person healthcare see little effect. Acceleration creates bottlenecks: Tasks that AI does not speed up, such as supervision, travel, or enforcement, become the dominant constraints within jobs, limiting overall productivity gains. The 1.8% productivity estimate is an upper bound: It assumes universal adoption, static workflows, and no time spent on validation, likely overstating near-term gains even if long-term effects could be larger. Measurement is the key contribution: The report’s main advance is a scalable method for tracking AI productivity using real usage data, enabling longitudinal analysis as tasks, models, and adoption evolve.

What might be the impact of AI on overall productivity? This is an important but challenging question to answer. Important because it has material implications for fiscal and monetary policy, for financial markets, and for the labor market. Challenging because the impact of AI will be broad-based, uneven, and pervasive throughout the economy—making it difficult to extrapolate precise productivity gains from narrow domains to the rest of the economy. We need a scalable framework for measuring the complexity of tasks that AI is actually used to tackle and the associated efficiency gains from its use. Alex Tamkin and I tackle this challenging problem in a research brief published this morning: "Estimating AI productivity gains from Claude conversations" Using privacy-preserving tools, we sample 100k conversations on claude.ai and ask Claude to evaluate how long it would take a skilled professional to complete the tasks that Claude is asked to handle—both with and without AI assistance. We have four key findings: 1) Claude can distinguish between short-horizon and long-horizon tasks. For a set of tasks where we know actual task completion times, Claude systematically produces longer estimates for tasks that actually take humans longer to complete—though in this benchmark forecast exercise Claude is less capable than humans. 2) Across our sample of real world conversations, Claude estimates that AI reduces task completion time by around 80% on average. Some tasks—like evaluating diagnostic images—show smaller time savings of around 20%. Others—like compiling information from reports—show savings of 95%. 3) The tasks that show up in our sample for higher wage occupations tend to be more complex, longer duration tasks. Management related tasks that Claude is asked to handle have the longest estimated human-only duration, with Business and financial operations following closely behind. 4) Aggregating Claude's estimates of task-level efficiency gains, we can assess what current usage of current-generation models might mean for the aggregate economy. Our task-level estimates would imply an increase in U.S. labor productivity growth of 1.8%pt annually if it takes a decade for AI gains to diffuse throughout the economy—roughly doubling the post-2005 pace of U.S. labor productivity growth. There are limitations to this work. Perhaps most notably, Claude's time estimates are imperfect and we lack real-world validation across all the tasks in our sample. Another limitation is that we analyze current-generation models, but capabilities are improving rapidly—which could mean larger productivity impacts ahead. Slower diffusion or bottlenecks could mean smaller. But we think it's important to work in the open and to generate useful signals about how AI is already reshaping the economy. We will continue to track this over time to provide another measure of how capabilities are improving, not just in principle but in practice.

Anthropic just released a fascinating study analyzing 100,000 real Claude conversations to estimate AI's impact on labor productivity. The headline numbers: Tasks that take 90 minutes without AI get done in about 18 minutes with it. Average time savings: 80%. Median task value: $54 in equivalent professional labor. Projected impact: 1.8% annual boost to US labor productivity - double the recent growth rate. Examples of acceleration: Curriculum development that would take teachers 4.5 hours completed in 11 minutes (estimated labor cost: $115). Financial analysts save 80% of time on tasks like interpreting investment data. Executive assistants save 87% of time drafting invoices, memos, and documents. Where the gains are concentrated: Management and legal tasks show the longest time savings (nearly 2 hours per task). Software developers contribute the most to overall productivity gains (19%), followed by operations managers and marketing specialists. The nuance that matters: Time savings vary dramatically: healthcare assistance tasks see 90% speedups while hardware troubleshooting shows only 56%. This creates potential "bottlenecks" where tasks AI can't accelerate become a larger share of the workday. What I appreciate about this research: Anthropic is actually trying to measure what so many of us have felt - that moment when you realize something that used to take 2 hours just took you 2 minutes. https://lnkd.in/ercpDeA7

The UK's Department for Business and Trade just released a 48-page evaluation of MS Copilot. Their conclusion? A generic, off-the-shelf AI chatbot isn't producing significant efficiency gains. Shocker… Here's what they found; 🔹 72% user satisfaction with basic writing and summarizing tasks 🔹 Modest time savings: ~1 hour saved on document drafting, negative time impact on scheduling and presentations 🔹 22% of users encountered hallucinations requiring fact-checking 🔹 Biggest benefits for neurodiverse users and non-native English speakers 🔹 No evidence of broader organizational productivity improvements Basically, it's a decent writing assistant. If we're expecting off-the-shelf LLMs to transform work, we're missing the point. LLMs aren't about optimizing existing workflows - they're about making work conversational. Imagine telling your procurement system: "Flag vendors with unusual pricing patterns from last 18 months" or "Generate an audit response comparing our data practices against our policy frameworks." That requires domain-specific training, system integration, and task-specific capabilities, none of which exist in off-the-shelf LLM driven copilot. Most companies are making the same mistake as the UK government. They're licensing generic AI tools and expecting productivity gains on individual tasks, when the real opportunity is building conversational interfaces to their actual business logic. To hit the nail on productivity gains with AI? 1️⃣ Start with the problem → Look for workflows where people navigate multiple systems, coordinate across functional areas, pass data back and forth, analyze it, and perform well-defined repetitive tasks. 2️⃣ Identify 1-2 specific processes and break them into testable components → Pick process you can decompose into individual tasks. Don't attempt to automate entire workflows until you've proven AI can reliably handle each component. 3️⃣ Invest in clean data, metadata, and integrations → Ensure you have the data infrastructure and system connections needed for AI to execute tasks rather than just generate text. 4️⃣ Measure each task against your hypothesis → Does it help? If all individual tasks were combined, would it provide enough gains to be worth the investment? 4️⃣ Be smart about expectations → This is emerging technology that will improve. Don't expect 100% accuracy out of the gate. The hard truth? Transforming your organization with AI requires an innovation mindset, not digital transformation. It's not about buying a tool, implementing it and seeing immediate ROI. Real transformation requires engineering investment and domain expertise. And that won't come from MS Copilot alone. The organizations that figure this out first won't be asking "Does AI save time on emails?" They'll be asking "What can we make possible when our systems can take orders in plain English?"

Your CEO wants to see AI ROI. Your team is under pressure to “transform.” Everyone’s measuring productivity wins. But here’s the trap: We’re stopping at input metrics. Time saved. Tasks automated. Content velocity. These matter — just like CPC matters in paid ads. But you don’t stop at CPC. You track conversions. You verify the clicks aren’t bots. With AI, we’re declaring victory at “40 hours saved per week” without tracking what those hours produced. Did they create capacity for the campaign that actually moved pipeline? Or did they just get filled with more of the same tactical work? Here’s why this matters beyond just measurement: When you can’t connect AI wins to business outcomes — revenue influenced, deals accelerated, market share gained — you can’t prove impact. And when budget season comes, “we automated a lot of stuff” doesn’t protect headcount. Proving outcomes does. Because it shows you’re increasing revenue per employee, not just doing more with less. The teams that track both productivity inputs AND business outcomes aren’t just measuring better. They’re building the case that keeps their budgets intact. Just check out the chart below. I built this with my CFO. It's our guiding principle for how we want to drive our AI measurement. Our mission, maintain headcount, reduce operating costs and move the resulting savings into revenue driving activities such as paid ads and events. And the outcome we are tracking? Pipeline, the lifeblood of our business. What business outcome are you connecting your AI productivity wins to?

Over the past six months, our team has taken significant steps to integrate coding agents into software engineering workflows. A recurring question I hear from teams is: “What is the impact of coding agents on SWE Productivity?” Impact can be assessed through multiple lenses: Code Quality: Does Copilot-generated code reduce bugs and minimize change-related outages? Velocity: Are engineers able to deliver pull requests at a faster pace? Developer Experience: Are engineers feeling more energized and empowered? Beyond these traditional metrics, I’ve been exploring a personal approach—measuring weekly time savings based on accepted Copilot-generated code. To do this, I built a VS Code plugin that locally tracks sessions with Copilot Chat and evaluates: COMPLEXITY: Based on language and structural patterns QUALITY: Adherence to coding guidelines defined in the Copilot instructions file VOLUME: Lines of code accepted At the end of each week, the plugin generates a report showing how much time Copilot has saved me—often adding several extra hours to my schedule. I’d love to hear what methods others are using to evaluate the productivity impact of coding agents.

When I talk to software developers about agents and ask them what do they want from an agent platform, I hear the same thing over and over. More important than ANY OTHER FEATURE, developers want true ROA (Return on Agents) data to be able to show how agents actually create value. Just showing utilization and engagement is good but its not nearly enough. Modern AI platforms need to get with the times and provide data reflecting productivity gain, business transformation, strategic and societal value. Reliability and accuracy is core of course. But if you want to have a commercially viable product you need these as well: Productivity & Efficiency Impact These begin to measure outcomes for humans and organizations, not just clicks: • Task Automation Rate: percentage of workflows fully handled by the agent without human intervention. • Productivity Lift: time saved per user/role (minutes/hours freed up). • Quality Uplift: accuracy, error reduction, fewer reworks compared to human-only baseline. • Operational Efficiency: cost per task vs. human cost; % reduction in support tickets handled by people. • Sustainability of Use: how long agents continue to deliver value without retraining or human babysitting. Best (Strategic & Societal Value) These measure enduring, systemic impact where agents start being digital extensions of ourselves: • Return on Agent (ROA): value delivered per agent vs. cost of operating it (infra + licensing + supervision). • Innovation Enablement: net new products, features, or processes only possible because of agentic capacity. • Workforce Displacement & Redeployment: percentage of human work displaced, retrained, or elevated into new roles. (This is both a negative and positive indicator of value). • Sustainability Footprint: energy use per task completed vs. human equivalent (carbon, water, compute). • Network Effects: agents collaborating with other agents across orgs/ecosystems to form emergent value chains. ✅ So the progression is: • Good → Can the agent work reliably and get used? • Better → Does the agent actually save time/money and improve output? • Best → Does the agent reshape business models, workforce structures, and ecosystems sustainably? #INSIGHTS #AGENTDATA #ReturnOnAgents

How are you measuring productivity benefits from AI? What’s working—and what’s not? In AI peer groups and leadership exchanges, I hear this question often. While debates over how to measure productivity are nothing new, they’ve resurfaced with urgency in the AI era—especially as we face a generational demographic shift and mounting pressure to deliver more with fewer people. Here are a few examples I’ve heard from others and how they are attempting to measure productivity from AI: - Time savings: Hours saved × average labor cost - Operating cost reduction: % of tasks automated × labor cost - Revenue growth efficiency: Revenue per employee or employees per revenue - Faster revenue recognition: Shorter cycles = faster revenue - AI ROI: (Productivity benefit – AI investment) ÷ labor costs - EBITDA margin: Uplift attributed to labor productivity But the McKinsey Global Institute’s article, The Power of One: How Standout Firms Grow National Productivity, offers a broader view. It suggests tracking Gross Value Added (GVA) per employee—defined as: EBITDA + labor compensation, adjusted for inflation and price distortion. The argument is that this is a better measure of the customer value created by productivity, not just the internal efficiency gains. McKinsey analyzed 8,300 firms using GVA and found: - A small group of standout companies drove the majority of productivity gains. - In sectors like semiconductors, aerospace, and electrical equipment, productivity is a major differentiator. - Standout firms grew productivity more through top-line growth and business shifts, not just cost-cutting. This research analyzed company results from 2011-2019, before the current wave of GenAI, agents, copilots, and LLM-driven automation. But I suspect the signal will become even clearer in the years ahead that AI’s impact on productivity (however you measure it) will increasingly show up in business outcomes, not just process metrics. https://lnkd.in/e6YRfH6t