Veit Heller’s Post

Releasing Raster 0.1 — typed multiple dispatch for Clojure What if you could write math in Clojure and get compiled performance that matches Julia and JAX — without leaving the REPL? Raster brings Julia-style typed multiple dispatch to the JVM. You define functions with `deftm`, annotate parameter types, and the compiler does the rest: devirtualization, automatic differentiation, buffer fusion, SIMD vectorization — all the way down to JVM bytecode. No DSL, no external toolchain. Every optimization is inspectable via `explain-pipeline`. The results surprised us too: * ODE solving (Dormand–Prince 5): 1.4× faster than Julia's DiffEq * LeNet-5 training (compiled AD + SGD): 1.7× faster than JAX on CPU * Forward-mode AD sensitivity: matching Julia's ForwardDiff * Zero heap allocations in compiled hot paths The key idea: don’t build another framework — build a compiler that understands typed dispatch, automatically differentiates, and fuses parallel operations end-to-end. Write generic code with `par/map` and `par/reduce`, get specialized SIMD loops on CPU or OpenCL/Vulkan kernels on GPU from the same source (Futhark inspired). Raster also ships with scientific computing (ODE/PDE solvers, optimization, FFT, special functions), linear algebra (dense + sparse, LAPACK via Panama FFI), deep learning primitives (conv, attention, normalization — all with reverse-mode AD), symbolic computation, and resource-aware compiler optimizations. It's the numerical substrate we are working towards to build collaborative simulation tools at scale. Open source, Clojure-native, JVM-hosted. Try it at the REPL. https://lnkd.in/gBHbi3Dz This is a first release, feedback and contributions are very welcome. #Clojure #JVM #NumericalComputing #MachineLearning #HighPerformanceComputing #Compilers #SIMD #GPU #OpenSource #FunctionalProgramming #Julia #JAX

GPU programming is powerful, but let’s be honest — writing and tuning kernels can get complex very quickly. In our latest Red Hat Developer article, Parshant Sharma and I explore Helion, a Python-embedded DSL that makes GPU kernel development feel much closer to writing PyTorch. The idea is simple but powerful: From: ➡️ 200+ lines of CUDA ➡️ 80-ish lines of Triton ➡️ ~15 lines of PyTorch-like Helion code Helion handles much of the low-level work: tiling, masking, indexing, grid sizing, memory layouts, and autotuning. Developers can focus more on the computation and less on manually managing hardware details. #PyTorch #GPU #Triton #Helion #AI #MachineLearning #Compilers #RedHat #OpenSource Blog : https://lnkd.in/g6K_aFBb Parshant Sharma and I co-authored this blog!

From Python to Rust for embedding inference — here's what we gained. We moved our RAG embedding pipeline to Text Embeddings Inference (TEI) — Hugging Face's Rust-native inference server. Stack: TEI (https://lnkd.in/grvTMJDk) · Rust + Candle ML + Flash Attention v2 + cuBLASLt · NVIDIA A100 80GB Results (500 reqs × batch=10 @ concurrency=10): → Single-req latency: 2.0 ms → Throughput: 1,000 req/s | 10,000 texts/s batched → VRAM footprint: ~2 GB (2.6% of 80GB) → Cold start: 6 seconds → Failures: 0 / 500 ——— Why Python costs you throughput: → GIL serializes tokenization, I/O, and response serialization → No built-in dynamic batching → Every request pays GC pauses + dynamic dispatch overhead → Triton/vLLM solve KV-cache problems embeddings don't have What TEI does differently (87% Rust, 9% CUDA): → No Python in the hot path — HTTP, tokenization, batching, dispatch: all compiled Rust → Flash Attention via Candle: Q·K^T·V fused into one kernel, HBM reads drop O(N²)→O(N) → cuBLASLt for QKV/FFN projections — FP16 tensor cores, zero GEMM dispatch overhead → Token-based dynamic batching — SM occupancy stays high regardless of input length variance → Zero-copy safetensors mmap — no torch.load(), no graph tracing, boot in seconds The model needs ~2 GB VRAM. The bottleneck was never the hardware — it was the runtime between your code and the GPU. Use vLLM for LLMs. Use TEI for embeddings. https://lnkd.in/gHNxzBJy #MachineLearning #Rust #CUDA #Embeddings #InferenceOptimization #RAG #MLOps #FlashAttention #HuggingFace

Proud of what the team built under the hood at TenderGenie. https://lnkd.in/dd2TwD7p   Processing thousands of pages of EPC specs, oil & gas RFPs, and construction annexures in real time means retrieval latency is on the critical path of every bid decision. A slow embedding layer = slower compliance checks, slower scope extraction, slower bid turnaround.   Self-hosted TEI was the right call. Sub-2ms latency, 10,000 texts/s, zero Python in the hot path. But beyond performance, enterprise clients in oil & gas deal with commercially sensitive bid data. Self-hosted means documents are vectorized entirely within their own infrastructure. No third party API call, no data exposure.   And every tender processed, every compliance decision, every scope interpretation compounds into institutional memory. Domain-specific embeddings that understand what a LSTK deviation means or how a P&ID spec maps to a commercial obligation. Generic OpenAI embeddings dont.   In high-stakes, document-heavy industries, infrastructure choices are product decisions. Thanks Microsoft for the Founder Hub's sponsorship of $ 150,000. Many more experimentations are lined up. TenderGenie DataSmith AI

A milestone for us. A faster, sharper experience for you. We've moved our entire embedding infrastructure to a self-hosted Rust-native stack. The numbers: sub-2ms retrieval latency, 1,000 req/s on a single GPU, zero downtime, zero data leaving your environment. What this means for bid & proposals teams: → Faster intelligence Compliance gaps, scope risks, and commercial obligations surface quicker. Every second saved compounds across hundreds of bid decisions. → Your data stays yours Your tender documents, RFPs, and commercial strategies are processed entirely within your own environment. No third party ever sees your data. → Sharper relevance Our models understand LSTK contracts, NEC clauses, P&ID specs, and BOQ structures — not just generic text. The more tenders you process, the sharper the intelligence gets. We're building the intelligence layer that serious bid & proposals teams deserve. This is one step further in that direction. #Procurement #BidManagement #TenderManagement #EPC #DigitalTransformation

Disk-LLM: When I was a PhD student, I dealt with a challenging acceleration sensor data and I and one of my collab friends come up with an idea to use NumPy memmap to handle large size data with a small RAM capability. From this perspective I was think and asking what if we apply NumPy memmap for LLMs and handle large models on disk and relieved the RAM and avoid to use too much RAM. This is approach is not new because there are already similar approaches but NumPy memmap is native numpy and python. It doesn’t require C++, CUDA kernels and massive frameworks. Therefore, I started an open source project on my GitHub, once you have time go and check it and dig into it , let me know your thoughts. https://lnkd.in/gj8cgPhg https://lnkd.in/gkch7CDe #research #opensource #LLMs #GENAI #NumPy

Over the past few months in my PhD, I’ve been exploring different aspects of compilers, especially tensor and sparse tensor compilers. I started by analyzing existing sparse tensor compilers and quickly realized something uncomfortable: there isn’t a complete solution. Most systems feel stitched together, lacking maturity and reliability. To dig deeper, I began building a fuzzer to identify bugs across these infrastructures. This is ongoing work, and I’m gradually expanding it to support more compilers and improve its robustness. While integrating MLIR’s sparse tensor dialect into the fuzzer, one thing stood out, it feels significantly more mature than other approaches, particularly in generating loop structures for sparse iteration. That said, optimization on top of these structures remains challenging. Recently, I’ve been working on a vectorization optimization tailored to MLIR’s sparse loop structures. The speedups are promising, but I’ll be honest, I don’t yet fully understand the hardware-level reasons behind them. That’s something I’m actively trying to close the gap on. In parallel, I’ve been learning polyhedral optimization (even though applying it to sparse data is not straightforward). One broader realization during this process: there’s no clear "getting-started" path in this space. I come from the YouTube era of learning programming language where learning meant: start small -> build something -> understand deeper. But in compiler research, most resources jump straight into first principles, which can make it hard to build momentum early on. To deal with this, I started organizing my notes (previously scattered across Obsidian and Notion) into structured blog posts, focused on building intuition through implementation rather than just theory. So far: - I’ve started an MLIR blog series (https://lnkd.in/eHsG-kMi) - I’m writing an ongoing series on polyhedral optimization and ISL (https://lnkd.in/ewGC2pDT) I plan to finish these series within the coming summer. Next up: a deep dive into SIMD/vectorization and why AVX-512 gather/scatter can bottleneck sparse workloads. I plan to write my entire learning path as different blog series. If you’re getting into compiler research without much prior exposure, I hope this helps you get moving faster. Honestly, I suspect more people will read these posts than my future thesis.

The real engine of modern AI runs on highly optimized C++, CUDA, and vendor libraries under the hood. Why AI infrastructure leans on low-level systems: • Python isn’t built for high-performance parallel compute. Frameworks bypass this by executing heavy ops in C++/CUDA and on GPUs. • At scale (billions of parameters), memory layout, bandwidth, and cache locality matter as much as algorithms. • Performance-critical paths—tensor ops, kernel fusion, distributed communication—live outside Python. Frameworks like PyTorch and TensorFlow aren’t just “wrappers”—they’re layered systems: Python API → execution engine → autograd → graph/compiler → optimized kernels. Python is the control plane. C++/CUDA is the execution layer. And as models scale, the abstraction leaks. If you want to build real AI systems—not just use them—you need to understand: hardware, memory, and distributed systems. #AI #SystemDesign #Backend #CPP #MachineLearning

Conda in HPC 🐍 Conda is an open-source package and environment manager widely used in Python. Each environment acts as an isolated space with its own dependencies, helping reduce version conflicts and allowing multiple projects to be managed independently. At Sanofi, our team creates and maintains Conda environments for researchers. These environments are hosted on a shared filesystem and are directly accessible from any cloud instance launched by a user on our platform, which also helps reduce instance startup time. As a result, the required tools are already available, and users only need to choose their image and instance type. More advanced users, however, remain free to create and customize their own environments. After four months of working with Conda in an HPC context, I’ve noticed a few recurring challenges. Here are three of the most common ones you may face: 🔥 PyTorch is not always available through standard Conda channels for every version of CUDA, NVIDIA’s GPU computing platform. In such cases, installation must be done through pip using --index-url. ✅ Be careful to select the correct version, otherwise the installation may silently result in an incompatible setup. ⚙️ GPU libraries are distributed as builds tied to specific versions of CUDA. When creating a Conda environment, the selected CUDA version defines which package builds are resolved and installed. This determines which binaries are actually installed and can result in either GPU-enabled or CPU-only environments depending on the instance context. ✅ Always create GPU Conda environments from properly configured instances. 📦 Some research projects are not available through standard package channels. Their installation inside a Conda environment is often done in editable mode using pip install -e, which links the Python package inside the environment directly to the local Git repository. ✅ Do not delete the Git repository after installation, as the environment relies on the local source code for execution. Hope this will save you some time and a bit of frustration! #SanofiStory

Most Python GPU code never touches the GPU directly. @vectorize. cuPy. PyTorch ops. You get your speedup, you move on. But at some point, the abstraction starts costing you. Your pipeline has too many unnecessary transfers. Your intermediate results keep bouncing between CPU and GPU when they never needed to leave the device. The bottleneck isn't the kernel , it's the bus. That's when you need to understand what's actually happening underneath. I wrote a deep-dive into GPU-accelerated Python with Numba that goes past the ufunc basics - into the mental model you actually need: → CUDA device functions and why the compiler inlines them for zero overhead → Why your pipeline is slow (spoiler: it's PCIe transfers, not compute) → Device arrays that keep data on-GPU across multiple operations → Warp divergence — what it is and how a single branchless multiply fixes it → Synchronization gotchas that make fast kernels look slow in benchmarks → The memory hierarchy from registers to global memory to host RAM - and the 100-1000x latency gaps between them #CUDA #GPU #HPC #HighPerformanceComputing #DeepLearning #MLEngineering #ParallelComputing