Dynamic Load Scheduling Algorithms

The Flexible Job Shop Scheduling Problem (FJSP) represents a critical advancement in industrial optimization, extending the classical Job Shop Scheduling Problem (JSSP) by introducing a dual-decision layer. While JSSP requires determining the sequence of operations on pre-assigned machines, FJSP adds the complexity of 'machine assignment', where each operation can be processed by any machine from a compatible set. This flexibility is essential for modern smart manufacturing, as it allows production systems to adapt to machine breakdowns and varying workloads, directly impacting operational efficiency and resource utilization in high-stakes environments. Historically, FJSP has been tackled using traditional exact methods like Integer Programming and meta-heuristics such as Genetic Algorithms (GA) or Taboo Search. More recently, Deep Reinforcement Learning (DRL) has emerged as a dominant approach, utilizing GNNs and Transformers to learn scheduling policies that can generate solutions in real-time. These neural net based methods treat the scheduling environment as a dynamic graph or sequence, attempting to map complex shop floor states to optimal dispatching rules. Despite their potential, current automated solvers face significant bottlenecks. The primary challenge lies in the 'curse of dimensionality' and sequence length. As the number of jobs and machines increases, the scheduling sequence grows quadratically, causing standard Transformers to suffer from extreme computational overhead due to their O(L^2) complexity. Furthermore, GNN-based methods often struggle to capture long-range dependencies between operations scheduled far apart in time, leading to sub-optimal machine assignments and increased makespan. To address the shortcomings highlighted above, the authors of [1] introduce M-CA (Mamba-CrossAttention), a novel architecture that replaces the standard self-attention mechanism with Selective State Space Modeling (Mamba). Mamba offers linear scaling O(L) with respect to sequence length, allowing the model to process much larger scheduling horizons efficiently. The M-CA framework specifically utilizes a 'Mamba-based Encoder' to capture global temporal dependencies and a 'Cross-Attention Decoder' to focus on the immediate machine-operation compatibility. This hybrid approach is superior because it maintains the high-fidelity global context of the entire factory state while drastically reducing the memory footprint and inference time required by traditional Transformers. Experiments show M-CA consistently outperforms state-of-the-art DRL baselines, Transformer-based models, and traditional heuristics across problem scales, achieving lower makespans and up to 5× faster inference. Mamba’s superior 'forgetting and remembering' mechanism drives scalability and robust performance by filtering out irrelevant scheduling noise to focus on critical constraints. The link to the paper [1] is posted in the comments.

Mixtures of Experts have been used in LLMs for quite a few years now, but DeepSeek's MoE implementation introduces several modeling tricks that are worth taking a closer look. 1 - Hybrid routing strategy. DeepSeek uses a hybrid of soft routing and hard routing. In soft routing we compute the weighted sum over all expert outputs, whereas in hard routing we limit the sum to the top k experts with the highest routing scores. In the hybrid version, we have a combination of shared experts and a pool of routed experts, of which only the top k are activated for each input token. The output of the MoE layer is a weighted sum over the shared and routed experts, where the shared experts’ weights are 1 and the routed experts’ weights are the router scores. (Unlike standard MoE implementations, DeepSeek uses a per-expert Sigmoid instead of a Softmax to normalize the router scores. This decouples the expert's router scores from each other, which is important for the next trick, dynamic load balancing.) 2 - Dynamic load balancing. Load balancing — making sure all experts and hence all GPUs inside the training cluster receive the same number of tokens during training — has been one of the most difficult challenges in sparse MoEs. So far, the status quo has been to introduce either load balancing losses (e.g. Switch Transformer) or customized compilers (e.g. MegaBlocks). DeepSeek’s MoE demonstrated for the first time a third solution, namely dynamic load balancing. The trick is to add a bias term b to each expert’s router scores prior to taking the top-k. If an expert is “overloaded” (i.e. receiving more tokens than the total number of tokens divided by the total number of experts), we reduce that expert’s bias by 𝛾, resulting in a smaller chance of the expert being selected by the router. In contrast, if the expert is underloaded, we increase the expert’s bias by 𝛾, increasing the chance of the expert being selected. As training progresses, eventually expert loads are mathematically guaranteed to reach perfect balance. 3 - Sequence-wise balancing. Unlike other MoE models, DeepSeekMoE adds a novel auxiliary loss term that ensures expert balance not just across the entire batch but even more fine-grained across each individual token sequence inside the batch. For example, given a sequence of 100 tokens and a pool of 4 routed experts with k=1, ideally we want each expert to be activated for 25/100 tokens. As usual, more on this in my blog: https://lnkd.in/gwxzX7ud

Load Balancing Algorithms Developers Should Know. Effective load balancing is crucial in system design, providing high availability and optimizing resource utilization. Let's look at how some of the most popular load balancing algorithms work. 𝗦𝘁𝗮𝘁𝗶𝗰 𝗔𝗹𝗴𝗼𝗿𝗶𝘁𝗵𝗺𝘀 𝟭) 𝗥𝗼𝘂𝗻𝗱 𝗿𝗼𝗯𝗶𝗻 It distributes requests sequentially between servers, ensuring equitable distribution. Despite its simplicity, it does not account for server load, which might be a drawback when demand changes significantly. 𝟮) 𝗥𝗮𝗻𝗱𝗼𝗺 Implements a simple way of distributing requests regardless of server load or capability. This form of load distribution is basic, less precise, and suitable for less complicated applications. 𝟯) 𝗜𝗣 𝗵𝗮𝘀𝗵 Uses a consistent hashing method depending on the client's IP address to route requests. This technique is one way to ensure session persistence by consistently directing requests from the same client to the same server. 𝟰) 𝗪𝗲𝗶𝗴𝗵𝘁𝗲𝗱 𝗿𝗼𝘂𝗻𝗱 𝗿𝗼𝗯𝗶𝗻 Improves round robin by assigning requests based on server capacity, aiming to better utilize resources by allocating more requests to higher-capacity servers. This approach seeks to optimize resource use, though actual results can vary with request complexity and system conditions. 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗔𝗹𝗴𝗼𝗿𝗶𝘁𝗵𝗺𝘀 𝟱) 𝗟𝗲𝗮𝘀𝘁 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀 Intelligently sends requests to the server with the fewest active connections, adapting to changing loads. This technique aims to better reflect current server utilization, potentially leading to more efficient resource consumption. 𝟲) 𝗟𝗲𝗮𝘀𝘁 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝘁𝗶𝗺𝗲 Targets performance by routing requests to the server with the quickest response time. By considering both current server load and performance, this technique supports faster processing, potentially reducing response times for users. While these are some of the most popular load-balancing strategies, there are other algorithms that also address specific needs and challenges. Choosing the right algorithm is very important to ensuring your application remains scalable, reliable, and efficient. #codingtips #tips #programming #coding #LoadBalancing

Designing an AI System That Doesn’t Collapse Under Latency Spikes A single user query passes through multiple stages — tokenization → batching → GPU scheduling → model execution → post-processing → response assembly. Now picture this: A few heavy prompts take 5× longer than average. Your batching layer waits to fill the “perfect batch.” Meanwhile, the queue grows. Requests start timing out. Retries stack up. That’s when you realize: You’re not running out of compute. You’re running out of control. Here’s how you design for resilience instead of collapse 👇 1️⃣ Bounded Queues Never let latency scale linearly with load. Bound your input queues and shed load proactively — either by dropping excess requests or serving degraded responses. Unbounded queues are silent killers — they delay backpressure, causing cascading timeouts. Think of it like circuit breakers for inference — graceful denial is better than system-wide collapse. 2️⃣ Adaptive Batching Static batch sizes look great in benchmarks and terrible in production. Instead, make batch sizes dynamic — continuously tuned based on GPU occupancy, queue length, and recent tail latency percentiles (P95/P99). At low load, batch small for lower latency. At high load, batch large for throughput — but with strict timeouts. The goal is elasticity without unpredictability. 3️⃣ Token-Aware Scheduling Batching by request count is naive. In LLM workloads, token length determines cost. A single 10,000-token prompt can stall 15 smaller ones if batched together. Token-aware schedulers measure total token budget per batch and allocate GPU time accordingly. This ensures fairness and consistent latency curves even under mixed workloads. 4️⃣ Partial Caching Most engineers cache final model outputs. That helps little. What actually saves time is pre- and post-compute caching — tokenized inputs, embeddings, and prompt templates. These are deterministic and cheap to reuse, shaving milliseconds off critical paths. Combine that with vector cache lookups to skip redundant reasoning altogether. 5️⃣ Deadline-First Scheduling In multi-tenant inference systems, not all requests are equal. Prioritize requests based on expected completion deadlines instead of FIFO order. This minimizes tail latency and improves QoS across traffic tiers. It’s the same principle airlines use — business class boards first, but everyone still gets there. This is where systems engineering meets AI infrastructure. Because LLM inference at scale isn’t just about throughput — it’s about temporal predictability. Inside my Advanced System Design Cohort, we go deep into these challenges — how to design AI systems that don’t just scale, but stay stable under load. If you’ve been leading distributed systems or AI infra and want to sharpen your architectural depth, there’s a link to a form in the comments — apply, and we’ll check if you’re a great fit.

🚦 Load Balancing Algorithms: How Traffic Is Optimized in Distributed Systems When we say “we use a load balancer”, what we really mean is: An algorithm decides where each request goes. That decision directly impacts latency, stability, and scalability. Here are the most important load-balancing algorithms every system designer should understand: 🔁 Round Robin Requests are distributed sequentially across servers. Works well when servers are identical and requests have similar cost. Simple, but breaks under uneven workloads. ⚖️ Weighted Round Robin Servers receive traffic based on assigned weights. Useful when servers have different capacities or during gradual traffic migrations. 🔌 Least Connections Each request goes to the server with the fewest active connections. Ideal for long-running or unpredictable requests where load varies over time. 🧮 Weighted Least Connections Combines server capacity with current load. One of the most practical algorithms for real production systems. 🔐 Hash-Based Routing Traffic is routed using a hash of client IP, user ID, or headers. Ensures the same client consistently hits the same backend — useful for session affinity and cache locality. 🍪 Sticky Sessions (Session Affinity) Once a user is assigned to a server, they stay there. Easy to implement, but limits scalability and can create uneven load. 🎲 Random Selection A backend is chosen randomly. Surprisingly effective at large scale with stateless services and large server pools. 🧠 Adaptive / Dynamic Algorithms Routing decisions are based on live metrics like latency, error rate, or CPU usage. Traffic naturally flows toward the healthiest servers. Load balancing is not just about splitting traffic evenly, it’s about making smart routing decisions under pressure. Choosing the right algorithm depends on: Request behavior Server capacity Statefulness Failure tolerance Understanding these algorithms is what turns “we scaled it” into “we designed it well.”