Performance Benchmarking Systems

AI field note: my word of the year is 𝔼𝕍𝔸𝕃: celebrating the art and science of rigorous measurement of AI performance, progress and purpose. (1 of 3) This year delivered a wealth of new AI models, architectures, and use cases - all united by one thread: evaluation. Model benchmarking, evaluation, or just "eval" has evolved from a simple, singular measure to a more complex blend of stats, metrics, and measurement techniques. Today's evals help discerning practitioners make pragmatic, informed technology decisions and measures improvements as AI systems are tuned. With AI innovation accelerating, staying up to date on evals ensures informed trade-offs when building intelligent systems, agents, and applications. Let's start by looking at measuring "performance"; the best way we know how to compare model behaviors, and find the right fit-for-purpose. Defining 'good performance' now involves a sophisticated suite of metrics across diverse dimensions. ⚙️ Task eval - beyond raw performance numbers. Today's evals measure how models perform across diverse scenarios - from basic comprehension to complex reasoning, reliability, consistency, and nuanced evaluation of reasoning paths, output quality, and edge case handling. 👛 Token economics - balancing cost, efficiency, and operation. Understanding token costs - both input and output - was essential last year, but evals have evolved beyond raw price per token, to understanding efficiency patterns, batching strategies, and the total cost of operation. ⏲️ Time-to-first-token. Speed is a feature, as they say, and while streaming responses have improved user experiences, this metric has become particularly crucial as models are deployed in production environments where user experience directly impacts adoption. 🔥 Inference compute: The amount of compute used for prediction shapes what problems a model can solve. More compute enables greater complexity but increases costs and latency - making it a pivotal benchmark for 2024. For some light holiday reading to explore this further: Service cards (OpenAI, Amazon), Meta's Llama 3 paper, and Anthropic's evaluation sampling research (links below).

🚀 Introducing 𝗳𝗲𝘃-𝗯𝗲𝗻𝗰𝗵: 100 forecasting tasks collected into one large-scale benchmark. We built it to reflect the messiness of real-world problems, with things like covariates that most benchmarks leave out. Why? Current benchmarks are reaching their limits. Many models now achieve similar scores, which makes it hard to tell whether new methods are actually better or just noise. 𝗳𝗲𝘃-𝗯𝗲𝗻𝗰𝗵 addresses this problem by focusing on new capabilities like forecasting with covariates, which are common in real-world applications. • 46 tasks with covariates, 35 multivariate tasks → allowing evaluation of capabilities that most benchmarks do not cover • Aggregation with confidence intervals → making it clear whether performance differences reflect real improvements or random variation • Backed by 𝗳𝗲𝘃, a lightweight Python library for reproducible & extensible benchmarking 🏆 Leaderboard: https://lnkd.in/dZ8nCayy 📝 Paper: https://lnkd.in/dPEMm_an 💻 GitHub: https://lnkd.in/dZ-sw-dD

A new framework offers a clearer picture of what today’s AI can—and can’t—do in real healthcare settings, revealing that top-performing models may ace medical exams but falter at routine hospital tasks. 1️⃣ It maps 121 real-world medical tasks—like diagnosing, documenting, and billing—into five clinician-validated categories. 2️⃣ A benchmark suite of 35 tests, many built on real EHR data, checks model performance across all key areas of healthcare. 3️⃣ Closed tasks are scored with exact answers; open tasks are judged by a panel of AI models rating accuracy, completeness, and clarity. 4️⃣ This jury approach matches doctor evaluations better than standard metrics like ROUGE or BERTScore. 5️⃣ Models generally excel at note writing and patient communication, but struggle with admin tasks like scheduling or coding. 6️⃣ Overall results vary by task, showing that strong exam performance doesn’t guarantee broad clinical readiness. 7️⃣ The system also tracks cost per evaluation, helping users weigh performance against price for deployment decisions. 8️⃣ All tools—taxonomy, benchmarks, scoring methods, and a leaderboard—are open and designed to grow with new models or tasks. ✍🏻 Suhana Bedi, Hejie Cui, Miguel Fuentes, Alyssa Unell, Michael Wornow, Juan M. Banda, Nikesh Kotecha, Timothy Keyes, Yifan Mai, Mert Oez, Hao Qiu, Shrey J., Leonardo Schettini, Mehr Kashyap, Jason Fries, Akshay Swaminathan, Philip Chung, Fateme (Fatima) Nateghi, Asad Ali, Ashwin Nayak, Shivam Vedak MD, MBA, Sneha J., Birju Patel, Oluseyi Fayanju, Shreya Shah, Ethan Goh, MD, Dong-han Yao, MD, Brian Soetikno, MD, PhD, Eduardo Pontes Reis, Sergios Gatidis, Vasu Divi, Robson Capasso, Rachna Saralkar, MD, MS, Chia-Chun Chiang, MD, Jenelle Jindal, Tho Pham, Faraz Ghoddusi, Steven Lin, Albert Chiou, Christy Hong, Mohana Roy, MD, Michael Gensheimer, Hinesh Patel, Kevin Schulman, Dev Dash, Danton Char, N. Lance Downing, MD, François Grolleau, Kameron B., et al. MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks. arXiv. 2025. DOI: 10.48550/arXiv.2505.23802

#AI benchmarking conversations I have often turn into a false choice. #Synthetic vs. #real_world. The truth? Both matter. At Dell Technologies, our TME labs not only analyze benchmarks but also work directly with customer’s production environments to see real-world challenges. Synthetic benchmarks show what the infrastructure can do under ideal conditions. They expose architectural limits, scaling characteristics, peak throughput, and theoretical efficiency. That’s important. It tells you the ceiling. But production environments don’t operate at the ceiling. Real-world benchmarking shows what the system will do under concurrency, mixed workloads, thermal pressure, network contention, and operational overhead. That’s where latency, sustained throughput, and cost per token actually determine business value. One shows maximum capability. The other reveals operational truth. If you’re evaluating AI infrastructure and only looking at peak tokens per second, you’re missing half the story. The best enterprise AI strategies validate both: • Lab performance to understand headroom • Production performance to understand reality Peak numbers sell slides. Sustained performance drives outcomes. #AI #AIBenchmarking #EnterpriseAI #Infrastructure #DataCenter #IWork4Dell

Stop manually benchmarking your PyTorch models. NVIDIA 𝐀𝐈𝐓𝐮𝐧𝐞 is a new library for anyone moving models from research to production. If you've ever spent days wrestling with TensorRT conversions or trying to figure out if 𝘵𝘰𝘳𝘤𝘩.𝘤𝘰𝘮𝘱𝘪𝘭𝘦 is actually faster than 𝘛𝘰𝘳𝘤𝘩𝘐𝘯𝘥𝘶𝘤𝘵𝘰𝘳 for your specific hardware, this library is for you. AITune isn't just another backend, it's an auto-tuner. It benchmarks every available backend (𝘛𝘦𝘯𝘴𝘰𝘳𝘙𝘛, 𝘛𝘰𝘳𝘤𝘩𝘈𝘖, 𝘐𝘯𝘥𝘶𝘤𝘵𝘰𝘳) on your specific hardware and automatically picks the winner. The two paths to performance: 1️⃣ 𝘛𝘩𝘦 𝘗𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯 𝘗𝘢𝘵𝘩 (𝘈𝘖𝘛): Profiles and validates backends, then serializes the best one as an .ait artifact. Result? Zero warmup on redeploy. 2️⃣ 𝘛𝘩𝘦 𝘍𝘢𝘴𝘵 𝘗𝘢𝘵𝘩 (𝘑𝘐𝘛): Set an environment variable and run your existing script. No code changes, no manual setup, it auto-discovers and optimizes on the fly. We often focus on prompt engineering or model architecture, but system-level efficiency is where the real ROI is in production. AITune fills the massive gap for models that don't fit into vLLM (Diffusion, CV, Speech, and Embeddings). It’s effectively "Auto-ML" but for inference optimization. #MachineLearning #AIEngineering #PyTorch #NVIDIA #SystemDesign #LLMOps

More information on [GA4] Benchmarking Overview Benchmarks are key metrics that enable you to compare your business's performance against other businesses in your industry. Google Analytics provides these benchmarks through peer groups—cohorts of similar businesses determined by factors like industry vertical and other relevant details. Key Features Daily Updates: Benchmarks are refreshed every 24 hours to provide the most current data. Eligibility Requirements: To access benchmarking data, your Google Analytics property must have the "Modeling contributions & business insights" setting enabled. Additionally, your property must generate sufficient user data to be included in a peer group. Data Protection Your benchmarking data is encrypted and protected, ensuring privacy and aggregation. There are also thresholds to guarantee that a minimum number of properties are included before benchmarks are available to a peer group. Accessing Benchmarking Metrics To view benchmarking data: Select the desired metric in the overview card on the Home page. Expand the Benchmarking category. Choose from a variety of metrics, such as Acquisition, Engagement, Retention, and Monetization. Using Benchmarking Data When benchmarking data is activated, you'll see: Your property's trendline The median of your peer group The range within your peer group (shaded area) Benchmarking comparisons are available within the 25th to 75th percentile to help you make informed decisions based on your performance relative to your peers. Changing Your Peer Group You can change your peer group to ensure more accurate comparisons. Peer groups are categorized based on industry characteristics, such as Shopping > Apparel or Travel & Transportation. Example Scenarios Acquisition: If your 'New User Rate' is below the 25th percentile, consider boosting user acquisition strategies. Engagement: A high 'Average Engagement Time per Session' could be leveraged by enhancing conversion strategies. Retention: A high 'Bounce Rate' may indicate a need for better user experience and content accessibility. Monetization: Low 'ARPU' suggests exploring strategies like upselling or personalized offers. Conclusion Benchmarking data in GA4 offers actionable insights by comparing your performance with industry peers, helping you identify strengths and areas for improvement to achieve your business goals.

Future of Finance: #3 Culture of Benchmark One of the biggest shifts in Finance today is moving from a self-centered view of performance to a relative, benchmark-based one. Benchmarking forces us to step back — to compare, to question, and to focus where it truly matters. It’s not about being the best everywhere, it’s about understanding where we stand and where we can improve fastest. With operations across six European countries, Orange Europe has a unique advantage: the ability to benchmark at scale. We’ve turned this into a strength, making benchmarking part of our DNA through systematic frameworks applied across multiple domains: 🔸 In Strategy, the Market Value Game, a cross-border and intra-market benchmark of value creation 🔸The Performance Tracker, inspired by Kearney’s GCB and Bain’s North Star, to monitor the impact of efficiency measures 🔸 or in Commercial, a Sales & Distribution Benchmark across channels to identify efficiency levers and best practices 🔸 and even, The Cash Conversion Benchmark, using deep analytics to compare cash generation across countries Building a data-driven performance culture starts with designing financial frameworks that are benchmark-ready — with aligned definitions and standardized data, not custom metrics. Don’t be afraid to standardize — it’s the foundation of comparability and transparency. But benchmarking is above all a (major) cultural transformation: 1️⃣ Start with acknowledging the gap, not challenging the method (this one is according to me the most difficult step) 2️⃣ Take an explanatory approach, not a defensive one (common mistake is to over estimate competitors advantage or own legacy) 3️⃣ Be action- and opportunity-oriented, not focused on justification Benchmarking is not about being judged — it’s about learning faster and continuously improving together. #FutureOfFinance #FinanceTransformation #BenchmarkCulture #PerformanceManagement #DataDrivenFinance #OrangeEurope #FPandA

As we were building SedonaDB, we realized that general-purpose data systems often lack a standardized, comprehensive benchmark for spatial analytics similar to what TPC-H for olap sql workloads, making it difficult to objectively compare performance. So, we decided to develop and open source SpatialBench (as part of the Apache Sedona project); a benchmarking tool that allows developers to generate spatial data at different scales (akin to TPC benchmarks) and run representative queries to test their system's performance. The benchmark queries are designed to reflect the types of operations that are critical for spatial analytics in the enterprise, such as spatial filters, spatial joins, distance queries, and aggregations. In our initial release, we've used it to benchmark SedonaDB, DuckDB, and GeoPandas. In our initial testing with the new benchmark for scale factor 10, we found that performance varies significantly across different systems and queries. For example, while SedonaDB and DuckDB performed similarly on queries 1 through 4, DuckDB was faster on query 5, and SedonaDB was significantly faster on query 9. For more complex queries (10-12), both DuckDB and GeoPandas either crashed or ran out of memory. Overall, SQL-based engines like SedonaDB and DuckDB consistently outperformed pure Pandas-based interfaces for most of the queries. This shows the benchmark's ability to provide a fair and unbiased comparison of spatial analytics systems. Check out the detailed results here: https://lnkd.in/g7ZNK7Ae Want to replicate the tests or benchmark other systems? Learn more and get started in the link below: https://lnkd.in/gd4dAsJM

We Need a Standardized Framework for LLM Performance Reporting We are seeing a new Large Language Model (LLM) drop almost daily, each pushing the frontier on accuracy. But here is the reality: the way we report performance and accuracy is completely fragmented. It's time we talk about a standardized benchmarking framework. Some benchmarks measure latency — TTFT (Time to First Token), TPOT (Time Per Output Token). Others focus on accuracy across tasks like equation solving, summarization, or coding. Some highlight output token throughput. A handful mention model size. Very few — almost none — report power consumption or energy efficiency or size. Sites like LiveBench and Artificial Analysis are doing excellent work, but we still lack a unified standard that practitioners can rely on to make real decisions. Deployment context changes everything: 1. Running inference on-prem at the edge? You care deeply about model size, cost, power draw, and memory footprint. 2. Serving millions of requests from a centralized data center? Throughput, cost-per-token, and latency under load dominate. 3. Building Agentic AI pipelines? Accuracy on multi-step reasoning, tool use reliability, and latency compound across agent calls in ways single-benchmark scores don't capture. So what a standardized benchmark might include? ✅ Accuracy across diverse task categories (reasoning, summarization, code, math) ✅ Latency metrics (TTFT, TPOT, end-to-end) ✅ Throughput (tokens/sec at various concurrency levels) ✅ Model size (parameters, quantization level) ✅ Memory requirements (VRAM / RAM) ✅ Power consumption (Watts, tokens per watt) ✅ Cost efficiency ($ per 1M tokens) ✅ Context window and retrieval performance ✅ Agentic task success rates (multi-step, tool use). We don't need one score to rule them all. We need a profile — a standardized set of dimensions — so engineers can select and compare models based on their specific requirements. Are you seeing this gap? What metrics do you wish were more consistently reported? #AI #LLM #GenAI #MachineLearning #AgenticAI #MLOps #AIInfrastructure #Benchmarking #EnterpriseAI #AIBenchmarking #DataCenter #OnPrem

As computer-use agentic systems increasingly approach — and sometimes even claim to surpass — human-level performance on OSWorld (covering tools like Office, VSCode, Photoshop, etc.), a natural question arises: 👉 Can these systems also handle complex enterprise workflows in platforms such as Salesforce, SAP, Outreach, or EPIC? To explore this, we took the first step and built SCUBA — Salesforce Computer-Use Benchmark. After extensive interviews with Salesforce Admins and Professional Services teams, we curated 300 realistic navigation and automation tasks across Admin, Sales, and Service domains. Each task comes with manual verification and human demonstration trajectories for reproducibility and evaluation. Highlights: 🌊 The first comprehensive and verifiable computer-use agent benchmark for Salesforce tasks 🔍 Empirical insights from ablations across 9 browser and GUI agentic systems (see details in the paper) 🥇 Salesforce proprietary browser agent achieves #1 performance on SCUBA Learn more: 📘 Blog: https://lnkd.in/gwuQMQ2V 💻 Code: https://lnkd.in/gNSbKRGS 📄 Paper: arxiv.org/abs/2509.26506 🏆 Leaderboard: https://lnkd.in/gfUW6_zJ