Data-Driven Assessment Models

Incrementality in Retail Media: Key Insights Following up on my previous post about key questions for incrementality models, here’s what strong answers look like: 1.Bias Handling Approach: We use propensity score matching and covariate balancing to ensure test and control groups are comparable. Why It Matters: These methods create fair comparisons between groups exposed and not exposed to marketing, ensuring accurate assessments. 2.Core Assumptions Approach: Our model assumes SUTVA (Stable Unit Treatment Value Assumption) and no hidden confounders, which we rigorously test. Why It Matters: Ensures one customer's behavior doesn't influence another's, enhancing result reliability. 3.Causal Inference Techniques Approach: We apply difference-in-differences, synthetic control methods, and regression discontinuity designs as appropriate. Why It Matters: These techniques isolate the true impact of marketing efforts from other variables. 4.Visual Models Approach: We use Directed Acyclic Graphs (DAGs) to map causal relationships and identify confounders, refining them with domain experts. Why It Matters: DAGs visualize complex factor interactions, clarifying causal pathways. 5.Data Granularity Approach: We leverage transaction-level data with privacy-preserving techniques and apply ecological inference for aggregated data. Why It Matters: Detailed data enables precise incrementality estimates; ecological inference aids insights from group data. 6.Handling Unusual Data Approach: We employ multiple imputation for missing data, robust regression for outliers, and sensitivity analyses for anomalies. Why It Matters: These methods address real-world data issues, ensuring data integrity. 7.Model Validation Approach: We perform A/B tests, backtesting, out-of-sample validation, and compare with traditional marketing mix models. Why It Matters: Validates our model’s accuracy and reliability across different scenarios. 8.Time-Based Adjustments Approach: We incorporate Bayesian structural time series models to account for seasonality, trends, and external events. Why It Matters: Captures temporal patterns like holiday spikes and market shifts. 9.Sample Size Requirements Approach: We conduct power analyses and use adaptive sampling to balance statistical significance and cost-efficiency. Why It Matters: Ensures sufficient data for reliable insights without resource waste. 10.Model Flexibility Approach: Our model utilizes transfer learning to adapt to various campaign types and objectives, from awareness to conversion. Why It Matters: Enables consistent measurement across diverse marketing strategies. #RetailMedia #Incrementality #MarketingAnalytics #DataScience

If you know educators are drowning in a "snowstorm of data" but starved for insight, you should watch this. The harsh truth? The problem isn't a failure of teachers to use data; it's a failure of the assessment infrastructure to provide genuinely useful information. For too long, we’ve relied on systems designed for accountability, not yet for improving the daily interactions within the classroom's "instructional core." In this short explainer, leading researchers Scott Marion and Carla Evans reveal the blueprint for fixing this broken model. Their work defines Instructionally Useful Assessment as one that delivers substantive, actionable insights when it matters—not months later. They detail essential features required for effective assessment design, focusing on three core areas: Representation: The assessment must cohere with the actual curriculum and measure complex, higher-order skills. Just-in-Time Insights: The information must be delivered swiftly and reveal how a student is thinking. Sociocultural/Affective Aspects: It must connect learning to the real world, allowing for collaboration and culturally relevant problems. This new science offers the foundation we need to build the next generation of effective learning tools. Click to view the video and download the full, open-access chapter, "Conceptualizing and Evaluating Instructionally Useful Assessments," to start auditing your assessment systems today. Please share it widely. Tagging colleagues who appreciate these insights. Cedar Rose; Harry Feder; Stanley Schauer; Melissa Johnston; Joe Crawford Sharyn Rosenberg; Barbara J. Helms, Ph.D.; Katrina Roohr; Supraja Narayanaswamy; Chris Woolard; Amy Berman; Ourania Rotou;Christine Cunningham; Brian Reiter; Matthew Ventura PhD; Joyce Zurkowski; Deborah Michele La Torre; Jennifer Torres, EdD; Jason Glass; Andrea Muse; Chad Buckendahl; Doug Mesecar; Matt Gandal; Carla Evans; Adam Rubins; Jennifer Randall; Alka Pateriya; Leslie Nabors Olah; Peter McWalters; Catherine Close; Neal Kingston; Laurie Davis; Rebecca Kockler; Heather Reams; George Mu, Christopher Brandt; Nancy Poon Lue;

Today, I would like to share a recent AI SoTL article entitled, “Toward Fair and Efficient Assessment: Generative AI for Open-Ended Questions in Higher Education” by Pecuchova, Benko and Drlik (2025) (https://lnkd.in/ehfnSAJz  ). Automated assessment of student work has long been a bottleneck in higher ed, especially for open-ended questions that require nuanced evaluation of understanding and reasoning. In a recent study the authors rigorously compare GenAI models and sentence embedding models against human graders using precision, recall, F1-score, and inter-rater agreement metrics (Fleiss’ Kappa, Krippendorff’s Alpha). The dataset comprised 1,885 open-ended responses from 110 students in a software engineering course. The findings show that advanced GenAI models, especially GPTo1, Claude3, and PaLM2 achieve substantial to almost-perfect agreement with human evaluators, outperforming traditional sentence embedding approaches and demonstrating real potential for scalable, reliable grading without excessive manual effort. Models like GPTo1 exhibited the highest fidelity to human judgment, even across diverse answer expressions, while more reference-anchored methods struggled with semantic nuance. From a learning-science perspective, timely, diagnostic feedback is a critical enabler of student learning (Shute, 2008). Delayed grading slows students’ ability to reflect and adjust strategies, undermining self-regulated learning cycles (Zimmerman, 2002). GenAI models that approximate human grading with low error rates can help educators provide faster, context-sensitive feedback as a potential mechanism to strengthen metacognition, conceptual understanding, and motivation. Moreover, the study’s use of inter-rater agreement metrics reflects an assessment validity focus central to educational measurement research (Popham, 2017). Finally, the limitations identified such as domain dependency and cost-based scalability concerns underscore the need for teacher oversight and pedagogically informed integration rather than full automation. Reference Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated grading of open-ended questions in higher education using GenAI models. International Journal of Artificial Intelligence in Education, 35, 3813–3846.

Optimizing RAG Pipelines with Metric-Driven Benchmarking: Benchmarking in Retrieval-Augmented Generation (RAG) ensures pipelines are efficient, reliable, and scalable. The RAG Assessment (RAGA) framework introduces Metric-Driven Development (MDD), a data-centric approach that replaces traditional human-in-the-loop (HITL) methods for ground truth annotations. By leveraging LLMs, RAGAs execute targeted evaluations to refine RAG systems. RAGAs expect datasets with queries, generated answers, retrieved contexts, and ground truth answers to perform evaluations. Typical metrics include Context Precision, Context Recall, Entities Recall, Noise Sensitivity, Response Relevancy, Faithfulness, Multimodal Faithfulness, and Multimodal Relevance. RAGAs help optimize vector retrieval, context quality, and LLM responses by simulating real-world scenarios. Benchmarking aligns pipelines with business goals, making it an essential foundation for every RAG initiative’s success.

Many teams overlook critical data issues and, in turn, waste precious time tweaking hyper-parameters and adjusting model architectures that don't address the root cause. Hidden problems within datasets are often the silent saboteurs, undermining model performance. To counter these inefficiencies, a systematic data-centric approach is needed. By systematically identifying quality issues, you can shift from guessing what's wrong with your data to taking informed, strategic actions. Creating a continuous feedback loop between your dataset and your model performance allows you to spend more time analyzing your data. This proactive approach helps detect and correct problems before they escalate into significant model failures. Here's a comprehensive four-step data quality feedback loop that you can adopt: Step One: Understand Your Model's Struggles Start by identifying where your model encounters challenges. Focus on hard samples in your dataset that consistently lead to errors. Step Two: Interpret Evaluation Results Analyze your evaluation results to discover patterns in errors and weaknesses in model performance. This step is vital for understanding where model improvement is most needed. Step Three: Identify Data Quality Issues Examine your data closely for quality issues such as labeling errors, class imbalances, and other biases influencing model performance. Step Four: Enhance Your Dataset Based on the insights gained from your exploration, begin cleaning, correcting, and enhancing your dataset. This improvement process is crucial for refining your model's accuracy and reliability. Further Learning: Dive Deeper into Data-Centric AI For those eager to delve deeper into this systematic approach, my Coursera course offers an opportunity to get hands-on with data-centric visual AI. You can audit the course for free and learn my process for building and curating better datasets. There's a link in the comments below—check it out and start transforming your data evaluation and improvement processes today. By adopting these steps and focusing on data quality, you can unlock your models' full potential and ensure they perform at their best. Remember, your model's power rests not just in its architecture but also in the quality of the data it learns from. #data #deeplearning #computervision #artificialintelligence

AI and Assessment! Here is another robust research paper on AI and assessment. I would call it foundational for anyone studying assessment through an AI lens. In this paper, Swiecki et al. (2022) start by questioning what they call the Standard Assessment Paradigm (SAP). Their critique is sharp and timely. SAP treats assessment as a snapshot taken at the end of learning, relies heavily on standardized tasks, and reduces complex learning to narrow proxies like scores and grades. It struggles to capture process, context, collaboration, and growth over time. In short, it measures what is easy to score rather than what actually matters for learning. The authors then explore how AI-enabled assessment can respond to these long-standing weaknesses. Some of the key affordances they highlight include: 1. Continuous and process-oriented assessment instead of one-off tests 2. Rich data traces that capture learning as it unfolds 3. Formative feedback loops that support improvement 4. Greater attention to context, patterns, and learning trajectories 5. New possibilities for assessing complex performances and authentic tasks At the same time, the paper is careful not to oversell AI. The authors close by outlining important limitations and cautions, including: 1. Risks of bias embedded in data and algorithms 2. Over-reliance on quantifiable indicators at the expense of professional judgment 3. Transparency and explainability challenges in AI systems 4. Ethical concerns around surveillance, privacy, and student agency 5. The need to keep human educators firmly in the assessment loop Overall, this paper helps reframe AI not as a catalyst for rethinking assessment itself. Definitely worth reading! Link to the paper in the first comment. #AIinEducation #AIAssessment #AssessmentDesign #LearningAnalytics #EdTechResearch #HigherEducation #EducationalResearch

*** Data-driven economic agent-based models *** In the last 10 years, economic agent-based models (ABMs) have become increasingly data-driven. In a book chapter prepared for the forthcoming "The Economy As An Evolving Complex System: Part IV", R. Maria del Rio-Chanona del Rio-Chanona and I discuss what it means for ABMs to be data-driven and why this approach helps overcome traditional limitations of ABMs. We also review methods recently applied to ABMs and highlight a few success stories of data-driven ABMs applied to various parts of the economy. Our chapter serves as both a manifesto and a resource: we propose a clear definition and classification of data-driven ABMs, provide an overview of the field's current state, and offer a guide for newcomers.   In our definition, an ABM qualifies as data-driven if it meets at least one of the following criteria: (i) it initializes at least some agent-level variables using real-world data, or (ii) it aligns at least some of its outputs with empirical time series. While less data-driven models have played a fundamental role in developing ABM theory, our definition of data-driven ABMs constrains models to reproduce a specific economy at a given point in time. It is this idea of reproducing specific economies, rather than abstract economies, that has been gaining traction in the last decade. In our opinion, it is what makes ABMs data-driven.   The shift from theory-driven to data-driven models requires the development of new methods. While we also review traditional approaches for calibrating global parameters, we focus on methods recently applied to ABMs to initialize agent-level variables and make ABM outputs track empirical time series. These methods include population synthesis, network reconstruction, data assimilation, and probabilistic graphical models. They demand a deep understanding of national accounting and how economic data are generated, along with the adoption of new statistical techniques.   Data-driven ABMs have already proven valuable for building reliable counterfactuals, forecasting economic dynamics, and understanding the impact of natural disasters and epidemics on the economy. We review success stories in housing and labor markets as well as in macroeconomics.   Finally, we discuss the challenges and opportunities associated with data-driven ABMs. We argue that data-driven ABMs: (i) push validation standards from stylized facts to time-series tracking and forecasting; (ii) help model behavior in a general way by replacing less critical assumptions with data, allowing for systematic testing of behavioral assumptions on model outputs; and (iii) make counterfactuals more reliable by reproducing empirical dynamics under actual policies. These advantages address current limitations of agent-based models and will promote their adoption in economic research and policymaking.   If you are curious and want to read more, here’s a link to the paper: https://lnkd.in/dT2ZFdVC. Enjoy!