How to Evaluate Language Models With Domain Benchmarks

#GeoAnalystBench: A #GeoAI benchmark for assessing large language models for spatial analysis workflow and code generation. Recent advances in large language models (LLMs) have fueled growing interest in automating geospatial analysis and GIS workflows, yet their actual capabilities remain uncertain. We call for more rigorous evaluation of LLMs on well-defined geoprocessing tasks before making claims about full GIS automation. To this end, we present GeoAnalystBench, a benchmark of 50 Python-based tasks (with over a hundred sub-tasks) derived from real-world geospatial problems and validated by GIS experts. Each task is paired with a minimum deliverable product, and evaluation covers GIS workflow validity, structural alignment, semantic similarity, and code quality (CodeBLEU). Using this benchmark, we assess both proprietary and open-source models. Results reveal a clear gap: proprietary models such as ChatGPT-4o-mini achieve high validity (95%) and stronger code alignment (CodeBLEU 0.39), while smaller open-source models like DeepSeek-R1-7B often generate incomplete or inconsistent workflows (48.5% validity, 0.272 CodeBLEU). Tasks requiring deeper spatial reasoning, such as spatial relationship detection or optimal site selection, remain the most challenging across all models. These findings demonstrate both the promise and limitations of current LLMs in GIS automation and provide a reproducible framework to advance GeoAI research with human-in-the-loop support. Paper: https://lnkd.in/d53cdfcT Code and Benchmark Datasets: https://lnkd.in/dAriGtcy

New! We’ve published a new set of automated evaluations and benchmarks for RAG - a critical component of Gen AI used by most successful customers today. Sweet. Retrieval-Augmented Generation lets you take general-purpose foundation models - like those from Anthropic, Meta, and Mistral - and “ground” their responses in specific target areas or domains using information which the models haven’t seen before (maybe confidential, private info, new or real-time data, etc). This lets gen AI apps generate responses which are targeted to that domain with better accuracy, context, reasoning, and depth of knowledge than the model provides off the shelf. In this new paper, we describe a way to evaluate task-specific RAG approaches such that they can be benchmarked and compared against real-world uses, automatically. It’s an entirely novel approach, and one we think will help customers tune and improve their AI apps much more quickly, and efficiently. Driving up accuracy, while driving down the time it takes to build a reliable, coherent system. 🔎 The evaluation is tailored to a particular knowledge domain or subject area. For example, the paper describes tasks related to DevOps troubleshooting, scientific research (ArXiv abstracts), technical Q&A (StackExchange), and financial reporting (SEC filings). 📝 Each task is defined by a specific corpus of documents relevant to that domain. The evaluation questions are generated from and grounded in this corpus. 📊 The evaluation assesses the RAG system's ability to perform specific functions within that domain, such as answering questions, solving problems, or providing relevant information based on the given corpus. 🌎 The tasks are designed to mirror real-world scenarios and questions that might be encountered when using a RAG system in practical applications within that domain. 🔬 Unlike general language model benchmarks, these task-specific evaluations focus on the RAG system's performance in retrieving and applying information from the given corpus to answer domain-specific questions. ✍️ The approach allows for creating evaluations for any task that can be defined by a corpus of relevant documents, making it adaptable to a wide range of specific use cases and industries. Really interesting work from the Amazon science team, and a new totem of evaluation for customers choosing and tuning their RAG systems. Very cool. Paper linked below.

The new open-source benchmark, MCP-Universe, is a useful step forward in how we evaluate LLMs. Unlike traditional benchmarks, it tests models on real enterprise tasks, like repository management and financial analysis. The latest results, though, are a wake-up call: as VentureBeat reports, GPT-5 failed in more than half of real work orchestration tasks. Not because the model isn’t powerful, but because raw model strength isn’t the same as enterprise readiness. Two challenges stood out: • Long context windows. Enterprise inputs are sprawling, incomplete, and often contradictory. Expanding the window isn’t enough. You need the right information inside it. Approaches like GraphRAG help by curating authoritative context and enabling multi-hop reasoning across knowledge. • Unfamiliar tools. LLMs struggle to adapt to proprietary formats, workflows, and security protocols. There’s a misconception that adding MCP on top of APIs will magically improve reliability. It won’t. MCP can connect systems, but that doesn’t guarantee value. Reliability comes from agents and tools built for specific jobs, grounded in a company’s own data, rules, and workflows—and from curating the right information, not just more of it. A “universal” layer doesn’t replace the need for domain-specific intelligence.

Evaluating LLMs is not like testing traditional software. Traditional systems are deterministic → pass/fail. LLMs are probabilistic → same input, different outputs, shifting behaviors over time. That makes model selection and monitoring one of the hardest engineering problems today. This is where Eval Protocol (EP) developed by Fireworks AI is so powerful. It’s an open-source framework for building an internal model leaderboard, where you can define, run, and track evals that actually reflect your business needs. → Simulated Users – generate synthetic but realistic user interactions to stress-test models under lifelike conditions. → evaluation_test – pytest-compatible evals (pointwise, groupwise, all) so you can treat model behavior like unit tests in CI/CD. → MCP Extensions – evaluate agents that use tools, multi-step reasoning, or multi-turn dialogue via Model Context Protocol. → UI Review – a dashboard to visualize eval results, compare across models, and catch regressions before they ship. Instead of relying on generic benchmarks, EP lets you encode your own success criteria and continuously measure models against them. If you’re serious about scaling LLMs in production, this is worth a look: evalprotocol.io

There is growing interest in using large language models (LLMs) to retrieve scientific literature and answer medical questions. However, the ability of LLMs to critically assess evidence and reason across multiple documents to provide recommendations at the same proficiency as domain experts remains poorly characterized. Systematic reviews (SRs), in which experts synthesize evidence across studies, are a cornerstone of clinical decision-making, research, and policy. Their rigorous evaluation of study quality and consistency makes them a strong source to evaluate expert reasoning, raising a simple question: Can LLMs match the conclusions of systematic reviews written by clinical experts when given access to the same studies? To explore this question, we present: 🎯 MedEvidence Benchmark: A human-curated benchmark of 284 questions (from 100 open-access SRs) across 10 medical specialties. All questions are manually transformed into closed-form question answering to facilitate evaluation. 📊 Large-scale evaluation on MedEvidence: We analyze 24 LLMs spanning general-domain, medical-finetuned, and reasoning models. Through our systematic evaluation, we find that: 1. Reasoning does not necessarily improve performance 2. Larger models do not consistently yield greater gains 3. Medical fine-tuning degrades accuracy on MedEvidence. Instead, most models show overconfidence, and, contrary to human experts, lack scientific skepticism toward low-quality findings. 😨 These results suggest that more work is still required before LLMs can reliably match the observations from expert-conducted SRs, even though these systems are already deployed and being used by clinicians! 📄Paper: https://lnkd.in/ghTa3pVA 🌐Website: https://lnkd.in/gvCTcsxR Huge shoutout to my incredible first co-authors, Christopher Polzak and Min Woo Sun, and to James Burgess, Yuhui Zhang, and Serena Yeung-Levy for their amazing contributions and collaboration.

new preprint, tl;dr:  • LLMs match or exceed SOTA strategies on chemical reaction optimizations.  • LLMs maintain systematically higher exploration Shannon entropy than BO, yet still find better conditions; BO retains an edge for explicit multi-objective trade-offs.  • we built the Iron Mind platform and we hope that it can serve as a new benchmark for both reaction optimizers and foundation models. Large language models (LLMs) are transforming experimental optimization in physical sciences and engineering. Our new preprint "Pre-trained knowledge elevates large language models beyond traditional chemical reaction optimizers" demonstrates that LLMs consistently match or exceed state-of-the-art Bayesian optimization (BO) across diverse chemical reaction datasets (paper link in comments). This work started with a simple question: “if/when can pre-trained knowledge substitute for traditional exploration-exploitation?” The amazing Robert MacKnight led a systematic benchmarking study across six fully enumerated reaction datasets and found that frontier models excel precisely where BO seems to struggle: complex categorical parameter spaces with scarce high-performing conditions (<5% of space). To deepen our understanding of the relationship between dataset complexity and optimizer performance, we turned to information theory. Shannon entropy analysis revealed something unexpected: LLMs maintain systematically higher exploration entropy than Bayesian methods while achieving superior performance. This suggests pre-trained domain knowledge enables effective parameter space navigation without traditional exploration-exploitation constraints. IMHO, these results warrant a closer look at how we approach experimental design. These findings suggest practical guidance for experimental chemists: LLM-guided optimization excels for high-dimensional categorical problems under tight experimental budgets, while Bayesian methods retain advantages for multi-objective optimization requiring explicit trade-offs. Iron Mind, a no-code platform, was developed to facilitate community engagement and set new benchmarks for optimization strategies and foundation models. It enables direct comparison of human, algorithmic, and LLM optimization campaigns on public leaderboards. Access Iron Mind at https://lnkd.in/eQbfsUex. Excellent work by CMU Ph.D. students Robert MacKnight (Carnegie Mellon University's College of Engineering Carnegie Mellon Chemical Engineering) and Jose Emilio Regio (Carnegie Mellon University Mellon College of Science Chemistry), in collaboration with our colleagues Jeffrey Ethier and Luke A. Baldwin from Air Force Research Laboratory. #ChemicalOptimization #MachineLearning #ExperimentalChemistry #BayesianOptimization #LLMs #AutonomousLabs

We've come to trust LLMs to summarize chemistry papers. Can we trust them to read the reaction scheme? In synthetic chemistry literature and patents, the most important information is rarely contained in the text alone. It is additionally encoded in reaction schemes, tables, and footnotes that only make sense when everything is read together. While today's multimodal LLMs can process images, how well can they actually understand chemical reactions as chemists encounter them in the literature? #RxnBench is a new benchmark designed to evaluate multimodal language models on real chemistry PDFs, rather than pure text or isolated figures. The benchmark is explicitly structured to mirror a realistic reading workflow through two complementary tasks: 🔹Single Figure QA: focuses on fine grained perception and local reasoning from reaction schemes. Models are asked to extract reactants, conditions, yields, stereochemistry, and mechanistic details directly from figures, where the context is limited and visually localized. 🔹Full Document QA: Models must reason across entire papers, integrating information distributed across text, schemes, and tables. With zero to four correct answers per question, an explicit cannot answer option, carefully designed hard distractors, and no partial credit, guessing is not an option. The results: The lead reasoning models perform very well on single figure questions with localized context. However, their performance collapses on full document reasoning. Even the strongest systems fail to reach 50% accuracy when required to synthesize chemistry across text, images, and pages. It will be interesting to see how chemistry foundation models and domain specific visual encoders perform here. If you routinely use LLMs with chemistry literature, this is a reminder that a bit of healthy skepticism is still warranted, particularly for chemical structures and reactions. 📄 RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature, arXiv, December 29, 2025 🔗 https://lnkd.in/eYMUKeiH

How well can LLMs select the right MCP tool for solving real world tasks? Not good. LiveMCPBench is a new benchmark that evaluates agents on a large-scale, dynamic, and realistic set of 527 tools. It shows that most models struggle with tool retrieval and utilization leading to bad performance. LiveMCPBench Creation: 1️⃣ Defined 95 real-world, multi-step tasks that are time-sensitive and require tool use, covering domains like office work, finance, and travel. Each task has "key points" required for successful completion. 2️⃣ Collected thousands of Model Context Protocol (MCP) servers and systematically filter them down to a "plug-and-play" set of 70 servers and 527 tools. 3️⃣ Created a baseline agent based on the ReAct framework that can dynamically plan and interact with the toolset through two core actions: `route` (search for a relevant tool) and `execute` (use the retrieved tool). 4️⃣ Implement an LLM-as-a-Judge. This judge receives the task, the required key points, and the agent's entire trajectory to determine "success" or "failure". 5️⃣ Tested 10 frontier LLMs measuring their task success rates, analyzing their tool usage efficiency, and categorizing their errors. Results:  - 💡Claude Sonnet 4 leads with a 78.95% success rate, many others only achieve 30-50%. - 📈 Most common error (~50%) the inability to find the correct tool, even when the agent formulates a good query. - 🛠️ Most models are "lazy", tend to find a single tool and rely on it exclusively, failing to dynamically leverage or combine multiple tools. - 🔢 Top-performing models are more proactive, using more tool retrieval and execution steps. - ⚖️ Reveals a clear, near-linear trade-off between model cost and performance. - 🤖 LLM-as-a-Judge proves to be a reliable and scalable achieving over 81% agreement with humans. Paper: https://lnkd.in/e4cg7TFa

Everyone loves MCP but has anyone benchmarked AI agents and evaluated LLMs in real-world scenarios to highlight the challenging nature of real-world MCP server interactions? Salesforce AI Research just did. Meet MCP Universe - the first comprehensive benchmark specifically designed to evaluate LLMs in realistic and hard tasks through interaction with real-world MCP servers. Even state-of-the-art models show 𝐬𝐢𝐠𝐧𝐢𝐟𝐢𝐜𝐚𝐧𝐭 𝐥𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐢𝐧 𝐫𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐌𝐂𝐏 𝐢𝐧𝐭𝐞𝐫𝐚𝐜𝐭𝐢𝐨𝐧𝐬: 🥇 GPT-5: 43.72% success rate 🥈 Grok-4: 33.33% success rate 🥉 Claude-4.0-Sonnet: 29.44% success rate Key Findings you need to know: 1️⃣ In the MCP Universe benchmark, 𝐥𝐨𝐧𝐠 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 𝐡𝐚𝐧𝐝𝐥𝐢𝐧𝐠 𝐩𝐨𝐬𝐞𝐬 𝐚 𝐬𝐢𝐠𝐧𝐢𝐟𝐢𝐜𝐚𝐧𝐭 𝐜𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞 𝐟𝐨𝐫 𝐋𝐋𝐌 𝐚𝐠𝐞𝐧𝐭𝐬, particularly in the Location Navigation, Browser Automation, and Financial Analysis domains. These domains frequently require agents to process and reason over lengthy sequences of observations which often exceed the context window limits of many models. 2️⃣ LLMs often struggle to correctly use tools provided by the MCP servers, indicating a 𝐥𝐚𝐜𝐤 𝐨𝐟 𝐟𝐚𝐦𝐢𝐥𝐢𝐚𝐫𝐢𝐭𝐲 𝐰𝐢𝐭𝐡 𝐭𝐡𝐞𝐢𝐫 𝐢𝐧𝐭𝐞𝐫𝐟𝐚𝐜𝐞𝐬 𝐚𝐧𝐝 𝐜𝐨𝐧𝐬𝐭𝐫𝐚𝐢𝐧𝐭𝐬. For example, for the Yahoo Finance MCP server, retrieving a stock price requires specifying a start and end date that differ, yet LLMs frequently set them to be identical, leading to execution errors. 3️⃣ You need 𝐨𝐩𝐭𝐢𝐦𝐚𝐥 𝐚𝐠𝐞𝐧𝐭-𝐦𝐨𝐝𝐞𝐥 𝐩𝐚𝐢𝐫𝐢𝐧𝐠 to maximize performance on complex tasks. You might like enterprise-level agents like Cursor for specific domains, but it doesn't outperform simpler frameworks like ReAct. Cursor demonstrates superior performance in Browser Automation but underperforms in Web Searching. How would we know that without benchmarking? Don't just fall for MCP hype. Start benchmarking and evaluating. Thank you, Salesforce AI Research, for this valuable MCP benchmarking tool. ✅ Start here: https://lnkd.in/ekZEBads ✅ Then go here: https://lnkd.in/ejPWmVbi ✅ Read the Paper: https://lnkd.in/ea9z32au Stephen Pullum Donna Rinck Darryn Tannous Ariel Perez Himanshu Jha Saahil Gupta, Horatio Morgan Uvika Sharma Padmini Soni Noelle Russell Aishwarya Naresh Reganti Sarfaraz Muneer Erik Conn PhD. Sevinj Novruzova Kimberly L. Andreas Horn Barnabas Madda Imran Padshah Ademulegun Blessing James James Beasley

 Massive Multitask Agent Understanding (MMAU) benchmark for testing LLM Agents Capabilities Across Diverse Domains In this paper, authors introduce comprehensive offline tasks that eliminate the need for complex environment setups and help evaluate the performance of large language models (LLMs) as agents across a wide variety of tasks 𝗞𝗲𝘆 𝗰𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀 - Understanding: evaluate an agent’s understanding in different aspects, including: complex instruction following, user intent understanding, statistics parsing, and visual grounding. - Reasoning and Planning: reflect an agent’s thought process and ability to infer logically from complex factors - Problem-solving: focuses on measuring an agent’s ability to successfully implement or execute a task, assuming it has already understood and planned the strategy well - Self-correction: reflects the agent’s ability to identify errors, learn from its environment and past behaviors, and correct itself to eventually overcome obstacles and achieve its task. 𝗞𝗲𝘆 𝗱𝗼𝗺𝗮𝗶𝗻𝘀 𝗳𝗼𝗿 𝗺𝗼𝗱𝗲𝗹 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 - Tool-use : evaluate the model’s response at each assistant turn (i.e., where a function call is expected), conditioning on the ground-truth versions of all previous user or assistant turns - Directed Acyclic Graph (DAG) QA: a user presents a set of requirements to which the LLM must respond by selecting and ordering a sequence of tool invocations from multiple choices provided - Data Science and Machine Learning Coding: leverage the Meta Kaggle Code dataset and curate 28 Python notebook-style conversations, with 123 conversation turns. - Contest-level Programming: select 261 problems from Valid and Test splits of the CodeContests dataset which includes competitive programming problem - Mathematics: derived from DeepMind-Math, consists of 1,000 carefully curated math problems spanning 56 subjects, including calculus, geometry, statistics, and etc. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - for capability of Understanding, GPT-4o significantly outperforms other models, demonstrating its superior capability in handling long contexts, complex user instructions, and capturing user intents. - For capabilities of Reasoning and Planning, GPT-4 family shows the strongest performance - for capability of Problem-solving, performance gap is not significantly large among models - Among open-source models, aside from Mixtral-8x22B, others do not seem to possess the skill to reflect on and correct their own errors effectively 𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻𝘀 - does not encompass all possible domains relevant to LLM agents, such as interactive environments which are also critical yet challenging - current approach to capability decomposition, though insightful, still faces challenges in disentangling compound capabilities 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/dUBXzsfX 𝗖𝗼𝗱𝗲: https://lnkd.in/d8JeUfhX #aiagents