Optimizing Quantum Model Performance for Professionals

Quantizing is not enough when fine-tuning a model! Even in the lowest precisions, most of the memory is going to be taken by the optimizer state when training that model! One great strategy that emerged recently is QLoRA. The idea is to apply LoRA adapters to quantized models. When the optimizer state is going to be computed, it is only going to be done on the adapter parameters instead of the whole model, and this will save a large amount of memory! The parameters are converted from BFloat16 / Float16 to 4-bits normal float. This quantization strategy comes from the realization that trained model weights tend to be Normal distributed, and we can create quantization buckets using that fact. This allows the compression of the model parameters without too much information loss. When we quantize a model, we need to capture the quantization constants to be able to dequantize the model. We usually capture them in Float32 to avoid as much dequantization error as possible. To compress further the model, we perform a double quantization to quantize the quantization constants to Float8. During the forward pass, because the input tensors are in BFloat16 / Float16, we need to dequantize the quantized parameters to perform the operations. However, during the backward pass, the original weights do not contribute to the computations, and they can remain quantized.

Quantum computing promises to making LLMs more efficient. And it's already working on real hardware. Efficient fine-tuning of large language models remains a critical bottleneck in AI development, with most researchers focused on purely classical computing approaches. A new paper from Chinese researchers demonstrates how quantum computing principles can dramatically reduce the parameters needed while improving model performance. The team introduces Quantum Weighted Tensor Hybrid Network (QWTHN), which combines quantum neural networks with tensor decomposition techniques to overcome the expressive limitations of traditional Low-Rank Adaptation (LoRA). By leveraging quantum state superposition and entanglement, their approach achieves remarkable efficiency: reducing trainable parameters by 76% while simultaneously improving performance by up to 15% on benchmark datasets. Most importantly, this isn't just theoretical - they've successfully implemented inference on actual quantum computing hardware. This represents a tangible advancement in making quantum computing practical for AI applications, demonstrating that even current-generation quantum devices can enhance the capabilities of billion-parameter language models. The integration of quantum techniques into traditional deep learning frameworks might become standard practice for resource-efficient AI development in the future. More on Quantum Hybrid Networks and other AI highlights in this week's LLM Watch:

Transform-to-Quantum: the problem with QUBO A recent pre-print from the QOBLIB team (“Quantum Optimization Benchmark Library -- The Intractable Decathlon” arXiv:2504.03832) provides a welcome trove of benchmark problems for quantum optimisation. Buried in Table 2 is a lesson every NISQ (or near term fault tolerant) project should pin to the wall: the way you model a problem can make or break any hope of running it on real hardware. Lets use the opportunity to highlight the “Transform to Quantum” part of our ITBQ Framework. Market Split—coefficients blow up • Mixed-Integer formulation: 78 binary variables, coefficient range ≈ 10². • After the routine MIP → QUBO conversion: still <100 variables, but the coefficient range balloons to ~7 × 10⁵. On an NISQ device that range must be encoded in gate angles or penalty weights. Six extra orders of magnitude usually translate into deeper circuits, worse conditioning and larger shot counts—effectively dooming a straightforward variational run. This is still similar on near term fault tolerant devices, in terms of the need for small angle rotations, which consume a substantial amount of T-Gates. LABS—variable count and coefficients explode • Original quadratic model: 81 spins, tiny coefficient range. • QUBO model: ~820 binary variables and a four-order-of-magnitude jump in coefficients. In most qubit-per-variable schemes that is a 10× increase in qubit demand plus the same precision nightmare seen in Market Split. Why it matters These examples are not corner cases—they are exactly the kind of “interesting, small” instances people reach for when chasing near-term quantum advantage. Yet without a smarter Transform-to-Quantum step (re-scaling, alternative encodings, constraint embedding, etc.) the numbers already exceed what today’s noisy processors can represent with meaningful fidelity. Take-away Hardware isn’t the only bottleneck. The modeling choices we make on the classical side determine whether a problem ever fits on quantum silicon. Getting Transform-to-Quantum right is therefore not a detail; it is the path-finder for every credible use case. Read more about our ITBQ Framework here: https://lnkd.in/en2KEjKC

I've been tackling the "barren plateaus" problem in QML, where training stalls inside vast search spaces. My latest experiment in fraud detection revealed a fascinating, counterintuitive solution. I discovered that increasing my quantum circuit's entanglement didn't smooth the path to a solution, but it created a more complex and rugged loss landscape (using a dressed quantum circuit scheme). Taking advantage of the hyvis library, I visualized this effect (thanks to the colleagues of JoS QUANTUM for putting this together), as shown in the first image of the post. The landscape evolves from a simple valley to a rich, expressive terrain (but potentially more complex for an optimizer). But did this complexity hurt performance? Usually that should be the case, but the exact opposite happened. The image shows the model with the most complex landscape (8 CNOTs by layer) not only learned faster (lower loss) but also achieved the highest accuracy (AUC) on the validation set and later in the test set. There is no free lunch on this. We can't generalize from these examples. This added complexity, or "expressivity," is precisely what allowed the model to find a superior solution in this case and avoid getting stuck, but it is not the norm. My biggest conclusion here It seems that for QML, the key to real-world performance isn't avoiding complexity, but leveraging it. To be able to extract permanent benefits, we should follow approaches like what Dra. Eva Andres Nuñez is researching by finding the way to use the extra complexity of entanglement to be able to find the global minima and not get stuck in our quantum optimization procedures using the theory behind SNNs. Here details about the hyvis library in GitHub: https://lnkd.in/dzqcFvDE An insightful paper from Eva about mixing SNNs and quantum: https://lnkd.in/dXDiuCBH Same subject from Jiechen Chen: https://lnkd.in/d-Uyngef #quantumcomputing #machinelearning #ai #datascience #frauddetection #ml #qml

QUANTUM INFORMATION GEOMETRY: QUANTUM NATURAL GRADIENT Review of Quantum Gradient Descent Algorithm and its use in different Quantum Optimization Algorithms Sangram Deshpande https://lnkd.in/eJ9WVNcu Abstract This review explores the quantum gradient descent algorithm, highlighting its distinctive advantages over classical gradient descent methods. Unlike the classical approach, which requires O(n) complexity for gradient computation, the quantum version achieves a remarkable complexity of O(1) through the principles of quantum superposition and entanglement. A particular focus is given to the Variational Quantum Eigensolver (VQE), a hybrid quantum-classical framework that utilizes parameterized quantum circuits combined with classical optimization routines. By replacing the classical optimizer with a quantum gradient descent optimizer, significant performance enhancements are realized, including faster convergence and reduced resource requirements. This work examines the implementation of quantum gradient methods within the VQE algorithm and emphasizes their potential to revolutionize optimization in quantum computing by providing exponential advantages in specific applications. The findings underscore the transformative role of quantum algorithms in advancing computational efficiency for complex optimization tasks. 

Bridging Physics and AI: A Quantum-Inspired Leap in Neural Network Optimization I've mapped a transformer model (SmolLM2-135M) to an Ising spin system and achieved unprecedented optimization using Q*Agents - Heated Ballistic Bifurcation (Quantum Tunneling). What’s New? 🧠 Reimagining AI Optimization: I treated the transformer’s weights as a digital twin of a physical system, governed by quantum principles. Leveraged Q*agents in a quantum-inspired algorithm to reveal emergent structural patterns in neural networks. 📊 Results That Speak for Themselves: Chaotic weight distributions transformed into organized, periodic patterns. Quantum tunneling-like effects drove efficient optimization, even in high-dimensional spaces. Evidence of emergent organization—is this the physics of intelligence manifesting in AI? Why It Matters: 🔋 From Theory to Reality: Imagine training massive AI models with quantum efficiency—on classical hardware. These results pave the way for next-gen optimization techniques that blend physics and machine learning. ⚡ Key Technical Highlights: Model: SmolLM2-135M transformed via Ising spin mapping Method: Heated Ballistic Bifurcation Innovation: Emergent banding in weight matrices suggests fundamental principles shaping model architectures 🌌 This is only the beginning. Are physics-inspired approaches the next frontier in AI? Could this redefine how we design, train, and optimize large models? Let’s collaborate, discuss, and explore these exciting intersections. Share your insights or connect if you’re working in this space!