Stochastic Optimization Methods

From Gradient Descent to Langevin Dynamics Standard stochastic gradient descent (SGD) takes small steps downhill using noisy gradient estimates ⚡. The randomness in SGD comes from sampling mini-batches of data. Over time this noise vanishes as the learning rate decays, and the algorithm settles into one particular minimum. Langevin dynamics looks similar at first glance but is fundamentally different 🎲. Instead of relying only on minibatch noise, it deliberately injects Gaussian noise at each step, carefully scaled to the step size. This keeps the system exploring even after the learning rate shrinks. The result is a trajectory that does more than just optimize ⛰️. Langevin dynamics explores the landscape, escapes shallow valleys, and converges to a Gibbs distribution that places more weight on low-energy regions 📊. In other words, it bridges optimization and inference: it can act like a noisy optimizer or a sampler depending on how you tune it. Stochastic gradient Langevin dynamics (SGLD) combines the two ideas, mixing minibatch gradients with structured noise. This makes it scalable to large datasets while retaining the ability to balance exploitation with exploration 🌍. The image illustrates this principle: a path driven by both gradient descent and noise, showing how a small modification can transform the behavior of an algorithm.

Stochastic optimization is about acknowledging that uncertainty is a first-class citizen in the decision process, demanding explicit treatment in both modeling and algorithmic design. At its core, stochastic optimization is the disciplined practice of making decisions today that balance immediate rewards with the evolution of the information state, carefully considering how decisions impact future opportunities under uncertainty. Most business problems are framed with deterministic simplifications, followed by a sensitivity analysis after the fact. This approach misses the essence of sequential decision-making under uncertainty, where the timing of decisions, the value of information, and the ability to adapt are often more important than marginal improvements to an objective function evaluated under a single scenario. It is not just about deciding what to do, but when to do it and how much to commit now while preserving flexibility for the future. To operationalize stochastic optimization, we must adopt a clear separation between the state variables (capturing the evolving knowledge of the system), the decision variables (representing the actions we can take now), and the exogenous information (the uncertainties we will observe next). This structured decomposition allows us to move from “solving a scenario” to building policies that can guide decisions across the entire distribution of future possibilities, not just the mean or a handful of edge cases. Stochastic optimization also forces us to rethink the objective function. Instead of maximizing expected profit under an assumed distribution, we must consider the shape of the distribution of outcomes, risk tolerances, and the operational realities of implementing policies that hedge against adverse realizations while capitalizing on favorable ones. This is why effective stochastic optimization frameworks blend forecasting, simulation, and optimization into a unified system, where learning and adapting are built into the policy architecture itself. The promise of stochastic optimization is a structured methodology that makes uncertainty explicit, guides the organization in aligning decision processes with evolving information, and, ultimately, captures value by turning uncertainty into a managed asset rather than a hidden liability.

"Optimization for ML": A lecture series that is exclusively dedicated for gradient descent for ML Every machine learning model learns by doing one thing, again and again: Take a step in the right direction. That step is called gradient descent. And while most people know the name, very few understand the mechanics that make it work. Let us break it down. In its most basic form, the update rule is: θ_new = θ − η⋅∇J(θ) Where: θ is the model parameter J(θ) is the loss function ∇J(θ) is the gradient η is the learning rate This is not just algebra. It is the beating heart of every neural network. One step of gradient descent means: You compute the slope of the loss function at your current position You move in the opposite direction of the slope You repeat this until the loss stops decreasing (or your model becomes good enough) That is it. No magic. Just calculus with a purpose. But here is where it gets interesting. This vanilla version is only the beginning. In practice, we do not compute gradients on the entire dataset every time. We introduce noise with Stochastic Gradient Descent. We add momentum to build acceleration. We adapt learning rates using RMSprop. We correct bias in the early steps using Adam. Each of these changes builds on this one idea - take a better step. To explain this entire journey from basic gradient descent to the most advanced optimizers used in deep learning, I created a 5-part lecture series on Vizuara's YouTube channel: 1) Gradient Descent: https://lnkd.in/gwTHgfNa 2) Stochastic Gradient Descent: https://lnkd.in/gu_hM6BS 3) Momentum: https://lnkd.in/gqs3-6mM 4) RMSprop: https://lnkd.in/g62hurAW 5) Adam: https://lnkd.in/g4jk8Hbt Each lecture covers: -The exact update rule -A visual intuition of how it works -Code-level implementation -When to use and when not to This is not a theoretical exercise. It is what actually drives model performance. Watch the full series here: https://lnkd.in/gn-4JBFn If you understand this one step, you understand how machines learn. And that changes everything.

If you’ve ever calibrated a complex model in Capital Markets and wondered why your optimizer kept working even when derivatives were messy or nonexistent… you were probably relying on Nelder–Mead, one of the most underrated algorithms in quantitative finance. 🔺 What Is Nelder–Mead? Nelder–Mead is a derivative-free optimization algorithm—meaning it doesn't need gradients, Jacobians, or smooth functions. Instead, it uses geometry. Specifically, it works with a shape called a simplex (a triangle in 2D, a tetrahedron in 3D, etc.). At each step, it evaluates the function at the simplex’s vertices and uses four operations: 👉 Reflection – try the opposite direction of the worst point 👉 Expansion – go further if things are improving 👉 Contraction – pull the shape inward if they’re not 👉 Shrinkage – collapse the simplex when completely stuck Through these simple geometric moves, it “crawls” across the surface to find a minimum. 🏦 Why Capital Markets Need Derivative-Free Methods Financial models often produce objective functions that are: Noisy ➡️ Non-smooth ➡️ Simulation-based ➡️ Discontinuous ➡️ Expensive to evaluate Think: ➡️ Calibrating stochastic volatility models ➡️ Minimizing error in Monte Carlo pricing ➡️ Fitting risk models with piecewise or irregular payoffs ➡️ Optimizing execution cost functions with nonlinear penalties In these environments, gradient descent breaks down. But Nelder–Mead thrives because it doesn’t require derivatives at all. 🧠 The Big Idea When calculus-based optimizers can’t help you, Nelder–Mead steps in as the rugged, resilient alternative. It’s not the fastest or the flashiest—but in the noisy reality of markets, it’s often the most reliable way to reach a solution. In quant finance, you don’t just optimize the math—you optimize for the environment. Nelder–Mead was built for exactly that. #QuantFinance #CapitalMarkets #NumericalMethods #Optimization #MachineLearning #QuantitativeAnalysis #FinancialEngineering #DataScience

Having worked in the operations research domain, I thought I had leveraged a variety of methods to formulate optimization problems, and there may not be new approaches to learn. Until I stumbled upon this today. While several approaches to optimizing eCommerce networks exist, this is a new one for me. But then, I have not dabbled in optimization for a few years now. In traditional manufacturing supply chains, demand forecasts are relatively stable and often aggregated (e.g., monthly orders). But in e-commerce, demand: 1. Fluctuates heavily due to flash sales, influencer promotions, or seasonal spikes, 2. Is highly localized (city or micro-region level), and 3. Interacts dynamically with return rates, which themselves are stochastic. Traditional optimization assumes fixed demand values (like deterministic D_i for customer i), while this paper introduces an optimization framework that treats D_i as an uncertain parameter. Treating D_i as uncertain is not new. What is new in this paper is the inclusion of a specific parameter and then leveraging an optimization approach that makes the best use of it. The paper defines uncertain demand (and returns) within interval bounds, forming what’s called a Box uncertainty set: D_i \in [D_i^0 - \Delta_i, \, D_i^0 + \Delta_i] Where a. D_i^0: nominal (forecasted) demand at customer i b. Delta_i: maximum deviation allowed (based on historical volatility or confidence interval) To prevent the optimization from assuming all demands hit their worst-case simultaneously (which would make the solution too conservative), they introduce a budget-of-uncertainty parameter \Gamma_D, following the Bertsimas–Sim robust optimization approach. A must-read for network optimization enthusiasts. https://lnkd.in/gAxGm3df #data #analytics #optimization #supplychain #supplychainoptimization #operationsresearch #networkoptimization

I have a real-world success story of stochastic optimization at United Airlines.   A member of my family booked a flight on United from the west coast to the east coast, changing planes in Chicago, with just one-hour from arrival to departure (!!).   The first flight left 20 minutes late, leaving a very tight connection in a large airport like Chicago. If they missed the connection, there were no other flights to their destination.   But, like a miracle, the flight arrived *on time* to Chicago, and they made their connection. How did this happen?   United uses a powerful stochastic optimization strategy that I call a “parametric cost function approximation.” In this setting, it is more commonly known as inserting schedule slack. United knows from experience that this flight (and most flights) experience various delays due to weather, equipment problems, staffing hiccups, and delayed inbound aircraft. To avoid excessive disruptions, they have to insert schedule slack in each flight.   There are two challenges – designing the structure of the schedule slack, and then tuning how much slack to insert for each leg. In my lab we would have built a simulator and used stochastic search tools. The problem is that a proper model has to not only capture the complex stochastics (the different causes of delays are correlated), but also how the network responds to delays (rerouting crews and aircraft, reassigning passengers). Passenger behavior, which is notorously hard to model, is also important.   I am sure that airlines like United keep careful performance statistics. I suspect that they have sensible, but ad hoc, methods for how they adjust the slack, and this would be a nice area for the academic community to do research that actually matters to industry.   I am virtually positive that United did not use stochastic programming, or Bellman’s equation. For an introduction to parametric cost function approximations, see   https://lnkd.in/eEcpM4Ex (“tinyurl.com/” with “cfapolicy”)   P.S. All major airlines use this approach. American Airlines Delta Air Lines. Full disclosure: I am a million-mile traveler on United.

🚀 Unlock the Engine Room of Machine Learning! 🧠 Ever wonder how massive ML models actually learn and adapt? 🤔 A huge part of the magic lies in Optimization! Dive deep into the core algorithms that power modern AI with these fantastic, comprehensive course notes on Optimization for Machine Learning by Gabriel Peyré (CNRS & ENS). 📚 These notes focus on the crucial first-order methods (think Gradient Descent 📉 and its powerful, scalable variants like SGD) that are essential for training models on large datasets. What's inside? ✅ Fundamentals of Convex Analysis ✅ Gradient Descent Deep Dive (Convergence, Acceleration) ✅ Stochastic Optimization (SGD, SGA, SAG) explained ✅ The magic of Automatic Differentiation (AutoDiff) ⚙️ ✅ Optimization for Shallow & Deep Networks (MLPs) ✅ Regularization Techniques (Ridge, Lasso) ✅ Advanced topics like Mirror Descent & Implicit Bias Perfect for #MachineLearning practitioners, students, and researchers looking to solidify their understanding of how models are trained efficiently and effectively. A must-read for grasping the mathematical foundations! 💡 #Optimization #AI #DataScience #DeepLearning #GradientDescent #Algorithms #Math I'm creating a lot of scientific content which is available on many of media platforms 👇 👇👇 Substack: https://lnkd.in/dTjrF6AP (English) Spotify: https://lnkd.in/dgumrSMR (English) https://lnkd.in/d-gMtCrE (Hebrew) Youtube: https://lnkd.in/dPGJr7WM (English) https://lnkd.in/dydSqeky (Hebrew) Telegram: https://lnkd.in/d_YxVMAR (English) https://lnkd.in/dVVqhNw5 (Hebrew)

Variational Stochastic Gradient Descent (VSGD) is a probabilistic, adaptive optimizer that models true and noisy gradients as random variables and uses stochastic variational inference (SVI) to estimate the true gradient for parameter updates. This framework dynamically adjusts to gradient noise and generalizes methods like Adam, SGDM, and Normalized-SGD under specific noise assumptions. A simplified variant, Constant VSGD, closely resembles Adam but includes uncertainty modeling and long-term memory of gradient behavior. Across benchmark datasets and architectures, VSGD consistently outperforms Adam and SGD in accuracy and convergence speed, with minimal added computational cost, demonstrating strong practical applicability. https://lnkd.in/gBYzKAEV