Using Data to Evaluate Project Success

AI projects should be just as measurable as any other business initiative, but many teams struggle to connect AI to financial value. Here’s the approach that works: • Start with the business goal: cost reduction, revenue growth, faster delivery, or better accuracy • Select KPIs that actually reflect impact: fewer refunds, faster response times, more output, fewer errors • Track before-and-after data using dashboards or reports that highlight what’s changed Don’t rely only on rough estimates like “hours saved.” They can be helpful, but they’re not enough on their own. Instead, track what actually improves: • Number of customer requests handled per day • Average time to respond or complete a task • Return or complaint rates • Volume of content produced or leads converted • Number of manual reviews or corrections avoided Even softer benefits like better decision-making or customer experience can be quantified through CSAT scores, survey responses, or processing speed. Bottom line: define success with real KPIs, measure actual changes, and AI ROI becomes visible, credible, and easier to defend.

Most data projects don’t fail because of bad tools. They fail because of bad sequencing. In 2026, building a data project from scratch is less about SQL and more about architectural judgment. 𝐇𝐞𝐫𝐞’𝐬 𝐭𝐡𝐞 𝐫𝐞𝐚𝐥 𝐰𝐨𝐫𝐤𝐟𝐥𝐨𝐰 𝐦𝐨𝐝𝐞𝐫𝐧 𝐝𝐚𝐭𝐚 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 𝐟𝐨𝐥𝐥𝐨𝐰: → 𝐒𝐭𝐚𝐫𝐭 𝐰𝐢𝐭𝐡 𝐭𝐡𝐞 𝐝𝐞𝐜𝐢𝐬𝐢𝐨𝐧 What business outcome are we driving? If there’s no clear stakeholder, stop. → 𝐃𝐞𝐟𝐢𝐧𝐞 𝐦𝐞𝐚𝐬𝐮𝐫𝐚𝐛𝐥𝐞 𝐬𝐮𝐜𝐜𝐞𝐬𝐬 Tie the project to revenue, cost reduction, risk, or velocity. No KPI, no priority. → 𝐌𝐚𝐩 𝐚𝐧𝐝 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐞 𝐬𝐨𝐮𝐫𝐜𝐞𝐬 Reliability and ownership matter more than volume. → 𝐃𝐞𝐬𝐢𝐠𝐧 𝐥𝐚𝐲𝐞𝐫𝐞𝐝 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 Raw → Staging → Curated. Plan for change, not just v1. → 𝐈𝐧𝐠𝐞𝐬𝐭 𝐰𝐢𝐭𝐡 𝐢𝐧𝐭𝐞𝐧𝐭 Batch or streaming based on business latency needs. Not because Kafka looks impressive. → 𝐌𝐨𝐝𝐞𝐥 𝐚𝐫𝐨𝐮𝐧𝐝 𝐛𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐞𝐧𝐭𝐢𝐭𝐢𝐞𝐬 Users. Orders. Revenue. Not tables copied from SaaS tools. → 𝐀𝐝𝐝 𝐝𝐚𝐭𝐚 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐨𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 Freshness, schema changes, anomalies. Trust is engineered, not assumed. → 𝐌𝐚𝐤𝐞 𝐢𝐧𝐬𝐢𝐠𝐡𝐭𝐬 𝐮𝐬𝐚𝐛𝐥𝐞 Dashboards are outputs. Decisions are outcomes. → 𝐆𝐨𝐯𝐞𝐫𝐧 𝐚𝐧𝐝 𝐢𝐭𝐞𝐫𝐚𝐭𝐞 Access control. Cost visibility. Continuous feedback. Strong data engineers don’t just build pipelines. They design systems that survive scale, change, and organizational complexity. P.S. When your team starts a new data initiative, where does it usually break first? Follow Ashish Joshi for more insights

Your project started without a baseline? Welcome to 90% of real-world Monitoring and Evaluation. Most programmes launch with urgency, political pressure, or donor timelines, not perfect data systems. That doesn’t mean you can’t measure change. It just means you need to reconstruct the “before” using the tools seasoned evaluators rely on: 🔹 Start with what already exists Intake forms, early reports, planning documents, grant proposals, even if they weren’t created for MEL, they often contain reference points you can extract. 🔹 Use recall methods strategically Ask participants and staff to describe conditions before the intervention, but anchor their memory to major events: ↳ “Before the school opened…” ↳“Before the water point was installed…” This reduces bias and increases accuracy. 🔹 Pull secondary data to fill the gaps Census tables, ministry surveys, NGO assessments, anything close in geography and timeframe can provide a credible reference. 🔹 Triangulate relentlessly Never rely on one source. Cross-check community recall with government data, staff insights, and documentation. Retrospective baselines aren’t shortcuts. They’re structured, defensible methods for rebuilding the past and they’re what experienced evaluators use when perfection isn’t possible (which is most of the time). 🔥 If you want more practical MEL techniques like this with no jargon, no theory-only talk, join my mailing list for weekly insights that will sharpen your practice. #Baseline

Benchmarking is one of the most direct ways to answer a question every UX team faces at some point: is the design meeting expectations or just looking good by chance? A benchmark might be an industry standard like a System Usability Scale score of 68 or higher, an internal performance target such as a 90 percent task completion rate, or the performance of a previous product version that you are trying to improve upon. The way you compare your data to that benchmark depends on the type of metric you have and the size of your sample. Getting that match right matters because the wrong method can give you either false confidence or unwarranted doubt. If your metric is binary such as pass or fail, yes or no, completed or not completed, and your sample size is small, you should be using an exact binomial test. This calculates the exact probability of seeing your result if the true rate was exactly equal to your benchmark, without relying on large-sample assumptions. For example, if seven out of eight users succeed at a task and your benchmark is 70 percent, the exact binomial test will tell you if that observed 87.5 percent is statistically above your target. When you have binary data with a large sample, you can switch to a z-test for proportions. This uses the normal distribution to compare your observed proportion to the benchmark, and it works well when you expect at least five successes and five failures. In practice, you might have 820 completions out of 1000 attempts and want to know if that 82 percent is higher than an 80 percent target. For continuous measures such as task times, SUS scores, or satisfaction ratings, the right approach is a one-sample t-test. This compares your sample mean to the benchmark mean while taking into account the variation in your data. For example, you might have a SUS score of 75 and want to see if it is significantly higher than the benchmark of 68. Some continuous measures, like task times, come with their own challenge. Time data are often right-skewed: most people finish quickly but a few take much longer, pulling the average up. If you run a t-test on the raw times, these extreme values can distort your conclusion. One fix is to log-transform the times, run the t-test on the transformed data, and then exponentiate the mean to get the geometric mean. This gives a more realistic “typical” time. Another fix is to use the median instead of the mean and compare it to the benchmark using a confidence interval for the median, which is robust to extreme outliers. There are also cases where you start with continuous data but really want to compare proportions. For example, you might collect ratings on a 5-point scale but your reporting goal is to know whether at least 75 percent of users agreed or strongly agreed with a statement. In this case, you set a cut-off score, recode the ratings into agree versus not agree, and then use an exact binomial or z-test for proportions.

I asked a VP of Rev Ops for the specific KPIs they use to evaluate the success or failure of an enrichment project. Here they are, in order from most to least important: 1) Accuracy Is the data accurate? This is first and foremost. If the data isn't correct, nothing else matters 2) Relevance How recent is the data? Can you figure out if someone changed jobs TODAY? Or do you have to wait 6 months for this information? 3) Coverage What % of your TAM can you enrich with your data vendor(s)? 90% is great. 10% is meh. This obviously depends heavily on the property and the industry, but we're usually able to get above 90% coverage, especially with a waterfall. 4) Price Obviously there's a limit here, but I've found most people are willing to pay a pretty penny for accurate, relevant data, especially with high coverage. Data is the fuel for a GTM engine... and folks are willing to invest in premium gasoline if it means their engine runs more efficiently. Curious if you agree with these success metrics or if you're measuring something else?

I wrapped up a pretty eye-opening project that I wanted to share with you. A few months ago, I helped a client untangle their project profitability data, and let me tell you—it was a bit of a beast. 😵💫 Like many digital agencies I see, they had a TON of data... ...but it none of it was woven together. We pulled in data from all over the place: Harvest time tracking, QuickBooks, Gusto Payroll —you name it. It took some serious mapping and visualization work, but we got there. And here’s where it got interesting. 👀 The data revealed something unexpected: 📉 a downward trend in gross margins for one of their key clients. Naturally, that led us to dig deeper. We found out that this client had become increasingly indecisive and unpredictable with their retainer engagement. My client was bending over backward to keep them happy, often without billing for all the extra time. This was a case of good intentions leading to bad margins. 😬 But here’s the good news: armed with this data, my client was able to have a tough but necessary conversation with their client. Expectations were reset, and we’re already seeing those margins start to climb again. Moral of the story? 👉 Data doesn’t lie 👈 When something feels off, there’s usually a reason—be it people, processes, or tech. Having the right data helps you ask the right questions, which leads to better conversations and, ultimately, better results.

Here are some realistic KPIs that project managers can actually track : 1. Schedule Management 🔹 Average Delay Per Milestone – Instead of just tracking whether a project is on time or not, measure how many days/weeks each milestone is getting delayed. 🔹 Number of Change Requests Affecting the Schedule – Count how many changes impacted the original timeline. If the number is high, the planning phase needs improvement. 🔹 Planned vs. Actual Work Hours – Compare how many hours were planned per task vs. actual hours logged. 2. Cost Management 🔹 Budget Creep Per Phase – Instead of just tracking overall budget variance, break it down per phase to catch overruns early. 🔹 Cost to Complete Remaining Work – Forecast how much more is needed to finish the project, based on real-time spending trends. 🔹 % of Work Completed vs. % of Budget Spent – If 50% of the budget is spent but only 30% of work is completed, there's a financial risk. 3. Quality & Delivery 🔹 Number of Rework Cycles – How many times did a deliverable go back for corrections? High numbers indicate poor initial quality. 🔹 Number of Late Defect Reports – If defects are found late in the project (e.g., during UAT instead of development), it increases risk. 🔹 First Pass Acceptance Rate – Measures how often stakeholders approve deliverables on the first submission. 4. Resource & Team Management 🔹 Average Workload per Team Member – Tracks who is overloaded vs. underloaded to ensure fair distribution. 🔹 Unplanned Leaves Per Month – A rise in unplanned leaves might indicate burnout or dissatisfaction. 🔹 Number of Internal Conflicts Logged – Measures how often team members escalate conflicts affecting productivity. 5. Risk & Issue Management 🔹 % of Risks That Turned into Actual Issues – Helps evaluate how well risks are being identified and mitigated. 🔹 Resolution Time for High-Priority Issues – Tracks how quickly critical issues get fixed. 🔹 Escalation Rate to Senior Management – If too many issues are getting escalated, it means the PM or team lacks decision-making authority. 6. Stakeholder & Client Satisfaction 🔹 Number of Unanswered Client Queries – If clients are waiting too long for responses, it could lead to dissatisfaction. 🔹 Client Revisions Per Deliverable – High revision cycles mean expectations were not aligned from the start. 🔹 Frequency of Executive Status Updates – If stakeholders are always asking for updates, the communication process might be weak. 7. Agile Scrum-Specific KPIs 🔹 Story Points Completed vs. Committed – If a team commits to 50 points per sprint but completes only 30, they are overestimating capacity. 🔹 Sprint Goal Success Rate – Tracks how many sprints successfully met their goal without major spillovers. 🔹 Number of Bugs Found in Production – Helps measure the effectiveness of testing. PS: Forget CPI and SPI - I just check time, budget, and happiness. Simple and effective! 😊

What silent metrics decide if your project succeeds... or silently fails? Projects crash quietly. 70% overrun budgets. 50% miss deadlines. The fix? Track these 5 ruthlessly. → 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 𝐏𝐫𝐨𝐠𝐫𝐞𝐬𝐬 • Tasks completed vs total • Milestones hit vs planned dates → 𝐁𝐮𝐝𝐠𝐞𝐭 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 • Actual spend vs budget • Cost variance + burn rate trends → 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 • Defects found + resolved • Client satisfaction scores → 𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐔𝐭𝐢𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 • Team allocation rates • Overtime/underuse percentages → 𝐑𝐢𝐬𝐤 & 𝐈𝐬𝐬𝐮𝐞 𝐌𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 • Open risks + issues count • Risk-triggered actions frequency Data-driven PMs live by these. Teams deliver 40% faster. Stakeholders trust grows. Master these numbers. Transform chaos into control. Follow Carlos Shoji for more insights

Unlocking Success: A Guide to Measure Our Strategic Actions 🎯 In the fast-paced world of business, strategic actions are the driving force behind achieving long-term goals and staying ahead of the competition. But how can we ensure that these actions are not just shots in the dark but effective steps towards our desired outcomes? In this post, will help explore a comprehensive guide to measure the success of our strategic actions by understanding the key pillars, backed by some real-world examples. 1. Define Success Criteria: Imagine you're a tech startup launching a new app. Your strategic action is to enhance user engagement. Success criteria here could be the number of daily active users, average session duration, and the percentage of users who complete a specific in-app action. By setting specific, measurable, achievable, relevant, and time-bound benchmarks, we can monitor progress and make data-driven adjustments as company aims for higher user satisfaction and retention. 2. Choose Appropriate Metrics: Now, let's consider a brick-and-mortar retailer focusing on customer experience. Metrics for their strategic action might involve conducting customer feedback surveys, tracking average time spent in-store, and measuring customer loyalty through repeat purchases. These qualitative and quantitative metrics provide valuable insights into customer sentiments and guide improvements to drive business growth. 3. Collect and Analyze Data: For a global manufacturing company aiming to reduce production costs, data collection becomes a crucial step. They can implement data tracking systems that record resource usage, analyze production line efficiency, and track overall wastage. By employing rigorous data analysis techniques, such as statistical modeling, they can pinpoint areas of inefficiency, optimize processes, and drive cost reductions. 4. Communicate and Report Results: Effective communication is vital for showcasing the impact of your strategic actions. Let's take the example of a marketing agency running a social media campaign for a client. They can present results through engaging infographics, showcasing an increase in website traffic, lead conversions, and brand mentions. Openly discussing campaign successes and challenges with clients fosters trust and collaboration for future strategic initiatives. Conclusion: Measuring the success of our strategic actions is the compass that guides us towards long-term success. By setting success criteria, choosing relevant metrics, collecting and analyzing data, and transparently communicating results, we'll gain valuable insights that power continuous improvement and inspire growth. Remember, each organization's journey is unique, so we need to tailor our approach and leverage the power of data to steer our strategic actions toward remarkable achievements. #StrategicSuccess #DataDrivenDecisions #BusinessStrategy #DataAnalytics #LeadershipInsights #StrategicPlanning #ContinuousImprovement

What criteria should guide the selection of pilot projects for AI integration? (Tip 34/2025) (and What metrics should be considered to measure the success of these pilots)   Criteria for Selecting AI Pilot Projects 1 Alignment with Strategic Goals Rationale: Projects should directly support business objectives (e.g., cost reduction, CX, innovation). Example: A chatbot pilot to improve customer service efficiency aligns with a goal of enhancing user satisfaction.   2 Feasibility and Resource Availability Technical Expertise: Availability of skilled personnel (data scientists, engineers). Infrastructure: Existing tools (cloud platforms, data pipelines) and budget.   3 Data Readiness Quality/Quantity: Sufficient, clean, and relevant data for training models. Accessibility: Data must be legally obtainable and structured for AI use.   4 Scalability Potential Scope: Pilot success should translate to broader applications (e.g., regional → global).   5 Stakeholder Buy-In Leadership Support: Ensures funding and organizational priority. End-User Engagement: Early feedback to drive adoption.   6 Risk Management Technical/Operational Risks: Predictable challenges (e.g., integration complexity). Ethical/Legal Risks: Compliance with regulations (GDPR, bias audits).   7 User Impact Tangible Benefits: Clear improvements in productivity, decision-making, or experience.   8 Cross-Functional Collaboration Team Diversity: Involvement of IT, business units, and domain experts.   Top Metrics to Measure Success 1 Business Impact ROI: Cost savings vs. implementation costs. Revenue Growth: New opportunities generated (e.g., upsell rates).   2 Performance Metrics Model Accuracy: Precision, recall, F1-score, or task-specific KPIs.   3 User Adoption & Satisfaction Usage Rates: Active users and interaction frequency. Feedback Scores: Surveys/NPS to gauge satisfaction.   4 Operational Efficiency Time/Resource Savings: Reduced processing time or manual effort.   5 Scalability Readiness Technical Flexibility: Ease of integration with existing systems. Cost of Expansion: Marginal costs for scaling.   6 Risk Mitigation Error Reduction: Decline in process failures or compliance breaches.   7 Data Quality Improvements Post-Pilot Enhancements: Data cleanliness and availability.   8 Innovation Impact New Use Cases: Additional applications inspired by the pilot.   9 Time-to-Value Speed of Deployment: Duration from launch to measurable results.   10. Ethical Compliance Bias: Audit results for algorithmic fairness.   11. Environmental Impact Sustainability: Reduced energy consumption or carbon footprint.   Note for Leadership Selection Criteria ensure pilots are viable, aligned, and low-risk.  Success Metrics quantify ROI, performance, user impact, and scalability. Prioritize projects with clear strategic value and measurable outcomes to build momentum for broader AI adoption. Image Source: Accenture Transform Partner – Your Digital Transformation Consultancy