Performance Forecasting Models

We are happy to share the results of our exhaustive benchmarking study on forecasting models, where we assessed 87 models across 24 varied datasets. This project aimed to evaluate the performance of univariate forecasting models ranging from naive baselines to sophisticated neural networks, using a comprehensive set of metrics such as RMSE, RMSSE, MAE, MASE, sMAPE, WAPE, and R-squared. The 24 datasets contained a wide range of frequencies, including hourly (4 datasets), daily (5), weekly (2), and monthly (4), quarterly (2), yearly (3). Additionally, there are 4 synthetic datasets without a specific frequency. Some of the datasets also contain covariates (exogenous features) of static, past, and/or future nature. For each model, we aimed to identify hyperparameters that were effective on a global level, across all datasets. Dataset specific hyperpameter tuning for each model was not performed due to budget constraints on this project. We use a simple train/test split along the temporal dimension, ensuring models are trained on historical data and assessed on unseen future data. The attached chart shows a heatmap of the average RMSSE scores for each model, grouped by dataset frequency. The results are filtered to 43 models for brevity, excluding noticeably inferior models and redundant implementations. RMSSE is a scaled version of RMSE, where a model's RMSE score is divided by the RMSE of a naive model. With RMSSE, the lower the score, the better the model's performance. A score of 1.0 indicates performance on par with the naive baseline. Key Findings: - Machine-Learning Dominance: Extra trees and random forest models demonstrate the best overall performance. - Neural Network Success: Variational Encoder, PatchTST, and MLP emerged as top neural network models, with Variational Encoder showing the best results, notably including pretraining on synthetic data. - Efficacy of Simplicity: DLinear and Ridge regression models show strong performance, highlighting efficiency in specific contexts. - Statistical Models' Relevance: TBATS stands out among statistical models for its forecasting accuracy. - Yearly Datasets Insight: On yearly datasets, none of the advanced models surpassed the performance of the naive mean model, highlighting the difficulty of forecasting with datasets that lack conspicuous seasonal patterns. - Pretraining Advantage: The improvement in models like Variational Encoder and NBeats through pretraining on synthetic data suggests a promising avenue for enhancing neural networks' forecasting abilities. All models and datasets are open-source. For a detailed examination of models, datasets, and scores, visit https://lnkd.in/d6mMSudJ. Registration is free, requiring only your email. Our platform is open to anyone interested in benchmarking their models. Any feedback or questions are welcome. Let's raise the state of the art in forecasting!

Are we really delivering the best possible forecasts with state-of-the-art foundation models if our models stop at historical patterns and ignore the external signals shaping the future? In the last few years, we've all seen how foundation models started transforming time series forecasting — unlocking strong zero-shot performance and making high-quality predictions possible without task-specific tuning. But the problem is that most of these models are univariate: they treat time series as isolated signals, leaving out exogenous factors that are often critical for accurate prediction. And that's not how forecasting works outside of a benchmark. Promotions, holidays, weather, pricing — these external influences often explain as much of the future as the past itself. Ignoring them leads to wider prediction intervals and forecasts that are harder to translate into real business decisions. So the real challenge now is: how do we bring that missing context into foundation models? That's the problem Chronos-2 was designed to solve. We built Chronos-2 to handle covariates and multivariate data in a zero-shot manner, and on benchmarks focused on these tasks, it achieves significant reductions in forecast error. But building a foundation model that can handle such diverse, context-dependent signals is not straightforward. Each forecasting task is unique — the number of features, their semantic meaning, and their interactions differ. The solution is a model that can adapt with in-context learning (ICL). Chronos-2 tackles this with two key components: 1. Architecture. In addition to standard temporal attention, we introduce group attention layers that enable information mixing across dimensions, allowing the model to learn from exogenous signals. 2. Training data. Multivariate and covariate time series data are extremely scarce, so we use synthetic data augmentation, adding multivariate structure on top of the univariate series commonly used for pretraining. The result is strong empirical performance across domains. In retail, Chronos-2 captures the impact of promotions on sales. In energy, it learns how weather influences energy consumption. In both cases, incorporating covariates significantly improves forecast accuracy and narrows prediction intervals — making forecasts more actionable. Chronos-2 is available under the Apache 2.0 license and ready to use. Give it a try and let us know what you think! 📄 Technical report: https://lnkd.in/d4RZG8Rq 💻 GitHub: https://lnkd.in/d9mvFT5B 📓 Example notebook: https://lnkd.in/dz69pCyu Abdul Fatir Ansari, Jaris Küken, Andreas Auer, Yuyang (Bernie) Wang, George Karypis, Huzefa Rangwala, Michael Bohlke-Schneider, Nick Erickson, Boran Han, Pedro Mercado, Syama Sundar Rangapuram, Huibin Shen, Lorenzo Stella, Amazon Science

You’re the CFO of a fast-moving consumer goods company. Your CMO just presented their MMM’s forecast for 2026, and you have no idea whether you can trust it. What should you do? Start with the basics by running a backtest. Take the model, hold out a few months of historical outcome data, and ask: can the model forecast what happens in the period it hasn’t seen? If a model can’t explain the past, it has no business guiding your company’s future spend. But you shouldn’t stop there. Backtests are just the first filter. To actually build trust in the model’s forecasting ability, you have to stress test it by making changes to your marketing budgets. This might sound risky, but the reframe I’ve found helpful here is to understand that you’re already making a bet on marketing – your evergreen media plan is one giant untested assumption. By making budget shifts based on a model’s recommendations, you now have the opportunity to restructure that bet in a way that surfaces more signal and can validate the model’s forecasts of those budget changes. Start with something small. Define the expected outcome. And track whether reality matched the forecast. Even if the read is noisy, that’s still valuable information. From here, you can iterate – scale the bet, re-evaluate the model, get tighter confidence intervals – and over time, you’ll want to get into a rhythm of forecasting → placing bets → validating the model → adjusting your media → repeating. That’s the shift I’d love to see more finance and marketing teams make. Don’t wait for a model to be “good enough” to act – use action to make the model better.

Machine learning beats traditional forecasting methods in multi series forecasting. In one of the latest M forecasting competitions, the aim was to advance what we know about time series forecasting methods and strategies. Competitors had to forecast 40k+ time series representing sales for the largest retail company in the world by revenue: Walmart. These are the main findings: ▶️ Performance of ML Methods: Machine learning (ML) models demonstrate superior accuracy compared to simple statistical methods. Hybrid approaches that combine ML techniques with statistical functionalities often yield effective results. Advanced ML methods, such as LightGBM and deep learning techniques, have shown significant forecasting potential. ▶️ Value of Combining Forecasts: Combining forecasts from various methods enhances accuracy. Even simple, equal-weighted combinations of models can outperform more complex approaches, reaffirming the effectiveness of ensemble strategies. ▶️ Cross-Learning Benefits: Utilizing cross-learning from correlated, hierarchical data improves forecasting accuracy. In short, one model to forecast thousands of time series. This approach allows for more efficient training and reduces computational costs, making it a valuable strategy. ▶️ Differences in Performance: Winning methods often outperform traditional benchmarks significantly. However, many teams may not surpass the performance of simpler methods, indicating that straightforward approaches can still be effective. Impact of External Adjustments: Incorporating external adjustments (ie, data based insight) can enhance forecast accuracy. ▶️ Importance of Cross-Validation Strategies: Effective cross-validation (CV) strategies are crucial for accurately assessing forecasting methods. Many teams fail to select the best forecasts due to inadequate CV methods. Utilizing extensive validation techniques can ensure robustness. ▶️ Role of Exogenous Variables: Including exogenous/explanatory variables significantly improves forecasting accuracy. Additional data such as promotions and price changes can lead to substantial improvements over models that rely solely on historical data. Overall, these findings emphasize the effectiveness of ML methods, the value of combining forecasts, and the importance of incorporating external factors and robust validation strategies in forecasting. If you haven’t already, try using machine learning models to forecast your future challenge 🙂 Read the article 👉 https://buff.ly/3O95gQp

The Fab Whisperer: The Next Evolution of Machine Rate Modeling Machine rate models were originally built for one purpose: to estimate the maximum throughput of any tool under ideal conditions. Traditionally, many fabs use static assumptions — average recipe times, theoretical WPH, average batch sizes and cascading factors, and constant overheads. But fabs don’t run in “static” mode. Throughput changes with: recipe sequencing chamber condition tool age batch-size distribution cascading practices drift and stabilization patterns for every factor warm-up effects complex mix interactions This is where AI and machine learning will fundamentally change machine rate modeling. They will evolve from Static → Dynamic → AI-Driven Throughput Models. Static models Use average times and fixed assumptions. Good for early planning — inaccurate in high mix, low volume, or mixed-tech fabs. Dynamic simulation models Incorporate batching logic, recipe time distributions, and simple drift curves. Better — but still limited by linear assumptions. AI-driven throughput models These models learn throughput directly from real tool behavior: Sequence-aware patterns - AI understands that Recipe A after B may run differently than A after D. Condition-based speed - Models capture seasoning curves, lens contamination, target erosion, and chamber age. Batch-size prediction - ML forecasts actual batch sizes based on real incoming lot distributions and arrival rates. Drift signatures - Long-term tool traces reveal nonlinear performance decay — something classical models cannot reproduce. Context-aware WPH forecasting - Given the next several lots and their recipes, AI predicts the maximum throughput achievable in the next hour. This transforms machine rate modeling from a static equation into a machine-scale digital twin of tool physics. AI won't just improve accuracy — it will change how fabs operate: More accurate CAPEX models Smarter dispatching that maximizes intrinsic machine speed Better dedication and recipe clustering More accurate technology cost modeling Predictive planning for new products and new platforms Machine rate modeling will shift from “fixed ideal WPH” to real-time, data-driven throughput intelligence. Fabs that adopt AI-based rate modeling will run faster because they will finally understand how fast their tools can run. #TheFabWhisperer #Semiconductor #FabOperations #MachineRates #ThroughputModeling #AIinManufacturing #HMLV #CapacityModeling #DigitalTwin