High-Speed Data Transfer Mechanisms

Zero Copy Data Transfer in HPC: A common technique for loading data in high performance applications is called “zero copy” because, well, it doesn’t require a copy. But what does that mean, and why is it useful? As I harp on in many of my posts, data movement is typically one of the largest bottlenecks and biggest challenges in high performance computing today. If we think about a 405B parameter LLM, we are transferring around, at a minimum, 405GB of data in memory. But this is virtually nothing when compared to the petabytes of data required to train that model. Traditional data transfer methods involve multiple copying of data between user space and kernel space, leading to increased CPU usage and reduced throughput. Let’s dive deeper: Problems with traditional data transfer: In a conventional data transfer operation, say from disk to a network interface, the data typically goes through multiple stages: - Reading from disk into kernel buffer - Copy from kernel buffer to user space - transform and copy back to kernel before network send - transmitted to network interface for sending Each requires a copy, requiring cpu cycles and memory bandwidth ultimately becoming rate limiting for large data. How Zero Copy Works: Zero Copy eliminates redundant data copies by using system-level techniques that allow data to be transferred directly between kernel space and the target destination without intermediary copies. Several Zero Copy techniques are implemented in modern operating systems: - Memory Mapping (mmap): mmap allows files to be mapped directly into the address space of a process. This means that the file contents can be accessed as if they were in memory, reducing the need for copying between kernel and user space. - Sendfile(): In networked applications, the sendfile() system call enables data to be sent directly from a file descriptor (such as a file on disk) to a socket, bypassing user space entirely. - Direct I/O: Direct I/O bypasses the kernel’s buffering mechanisms, allowing data to be read or written directly to and from disk. - DMA (Direct Memory Access): hardware-level technique where data is transferred directly between the memory and a device without CPU intervention. Ultimately, zero copy provides reduced CPU utilization, lower latency access, increased throughput, and more efficient memory usage. Several technologies exist that leverage zero copy architecture directly, such as GPU Direct Storage by NVIDIA, RDMA over Converged Ethernet, and even Network Filesystems. Diving into understanding this will help you better understand how to efficiently move data in your HPC applications. If you like my content, feel free to follow or connect! #softwareengineering #hpc

In the March 2024 Issue of IEEE Journal of Solid-State Circuits (JSSC)🏷️https://lnkd.in/gYusghfz, Intel Labs reported a 3D heterogeneously integrated #DWDM optical transmitter (OTX) that simultaneously modulates eight 200GHz-spaced wavelengths (8-λ) at 50Gbps/λ each, delivering a per-fiber bandwidth of 400Gbps. Energy efficiencies of the OTX (measured at 400Gbps across 8-λ) and its EIC portion (including #NRZ serialization and clocking overhead) are 2.5pJ/bit and 1.17pJ/bit, respectively. Excerpts (edited): 📝Two bottlenecks in datacenters are latency between compute nodes and limited per-node resource (e.g., #HBM). One way to improve latency is to flatten network hierarchy by reducing/eliminating network switches while forging direct node-to-node links. Per-node resource can be enhanced by disaggregating/allocating/pooling compute, memory, I/O, etc. 📝Both network flattening and resource pooling require a high-bandwidth, low-latency, energy efficient interconnect solution that can also increase the signal reach for continued #AI or #HPC scale-out. Silicon-Photonic (Si-Ph) Interconnect, with its long-reach ability and high bandwidth density, fits the bill. 📝8 optical carriers driven by an integrated Multi-Wavelength Laser (MWL) are combined to feed 8 cascaded Micro-Ring Modulators (MRMs), whose resonant wavelengths can be modulated electrically. The resonant nature of MRMs enables DWDM without the need for explicit/clear-cut optical multiplexers, a key benefit of MRMs over MZMs (Mach-Zehnder Modulators). A thermal tuning/tracking mechanism is applied to maintain MWL-MRMs alignment with an always-on, closed-loop Thermal Control Unit (TCU). 📝The on-chip DFB (Distributed Feedback) laser can generate a fairly high output power (e.g., 13dBm/λ at 100mA and 80C). 📝Sharing one laser across multiple fibers using splitters maximizes the system energy efficiency. To compensate the optical loss due to splitting, we implement an integrated Semiconductor Optical Amplifier (SOA) such that only 3.7dBm/λ is needed from the MWL for an 8-λ DWDM link. 📝The on-chip laser eliminates the need for dedicating a fiber to an external laser, thereby avoiding the associated coupling loss and power consumption. 📝The OTX contains an EIC fabricated in 28nm CMOS and a PIC in Intel’s 300mm hybrid Si-Ph process, which are flip-chip-bonded. The III-V epitaxial structure that holds the MWL and SOA is wafer-bonded to the rest of the PIC. The whole PIC is attached to a substrate, and wire bonds bring power, clock, and control/observability from the substrate to the OTX die complex. 🔍Observation: Laser integration for CPO is hard but not impossible; one touchstone is a robust MWL-MRMs coherence that is resilient to thermal fluctuations and wavelength/power/process variations. Next, build TSVs into the PIC and replace wires with bumps, making it a true 3D assembly!👍 🏷️Full article: https://lnkd.in/gr295cAF 🏷️CPO (IV): https://lnkd.in/g4TM84Kp ➟To be continued.

SoC interconnect: A System-on-Chip (SoC) contains multiple processing and functional blocks—CPUs, GPUs, DSPs, ISPs, memory controllers, and peripherals—all of which need to communicate efficiently. The interconnect is the communication backbone that links these components together. 🧩 What is a Typical SoC Interconnect? A SoC interconnect is a structured data highway that connects masters (like CPUs, GPUs, DMA engines) and slaves (like DRAM controllers, peripherals, and I/O subsystems). 🔹 Common Interconnect Standards AMBA (ARM) – the most popular: AXI (Advanced eXtensible Interface) – high-performance interconnect for CPUs, GPUs, etc. AHB / APB – used for lower-speed peripheral communication. OCP (Open Core Protocol) – used in older or custom designs. Proprietary fabrics – e.g., Qualcomm’s CoreLink, Apple’s Fabric Interconnect, or NVIDIA’s NVLink internally adapted for SoC. ⚙️ Functions of the SoC Interconnect 1️⃣ Data Routing- Transfers read/write transactions between initiators (masters) and targets (slaves). 2️⃣ Arbitration - Handles multiple concurrent requests and decides priority (QoS). 3️⃣ Address Decoding - Maps address ranges to target devices. 4️⃣ Clock Domain Crossing (CDC) - Synchronises data transfers between components operating at different clock speeds. 5️⃣ Power & Clock Management Hooks - Supports isolation during low-power modes (domain-level power gating). 🔗 What is NoC (Network-on-Chip)? A Network-on-Chip (NoC) is the modern evolution of interconnects — instead of a single shared bus or crossbar, it uses packet-based communication (similar to a data network) between SoC components. 🔹 Why Traditional Buses Fail Old bus-based designs (e.g., AHB) don’t scale well with: Multiple high-bandwidth masters (CPU, GPU, ISP, NPU) Long physical wire delays in large SoCs Power and clock domain segmentation 🔹 NoC Approach Components (called nodes) are connected via routers and links. Each transaction is broken into packets or flits (flow control units). Packets traverse the NoC based on routing algorithms (like mesh, ring, or hierarchical tree). Enables scalable, concurrent, and low-latency communication. 🧠 Role of NoC in SoC 1️⃣ Scalability - Allows dozens of cores, accelerators, and controllers to communicate simultaneously without congestion. 2️⃣ Parallelism - Supports multiple independent data transfers concurrently. 3️⃣ Power Efficiency - Enables localized communication → less global wiring, reduced dynamic power. 4️⃣ Clock/Power Domain Isolation - Easier integration of components running at different frequencies/voltages. 5️⃣ Modularity - Simplifies SoC design reuse; IPs connect via standard NoC interfaces. 6️⃣ Debug & Performance Monitoring - Built-in traffic counters, congestion metrics, and error detection. #architecture #embedded #kernel #learning #linux #system

Scatter-Gather I/O in C for Improved Performance 🚀 We often talk about "zero-copy" in high-performance networking, but the mechanics of how we achieve it at the syscall level are worth understanding. If you are building a server framework or working with high-throughput network protocols, you inevitably run into the "Header + Body" problem: you have a protocol header in one buffer and a data payload in another. The simple approach is to allocate a new, larger buffer, memcpy the header, memcpy the body, write() the larger buffer to the socket. This burns CPU cycles on redundant copying and puts unnecessary pressure on your allocator. Let's be sympathetic to the hardware with readv and writev. These two system calls allow you to pass a vector of buffers (iovec structs) directly to the kernel. You hand the OS a list of pointers and lengths, and it gathers the data atomically during the transmission. This results in fewer system calls by merging multiple calls into one and reduces memory bandwidth by eliminating the intermediate memcpy. The data moves directly from your original buffers to the kernel's network stack. You also get the added benefit of a fully atomic write from n buffers. If you're going to go in C, go fast. #softwareengineering

Problem It Solves Accessing large volumes of data from Amazon S3 Standard can introduce latency and throughput bottlenecks, especially in ML, analytics, and high-performance computing workloads that need repeated or rapid access to the same data. Blog Summary The blog introduces a solution that uses Amazon S3 Express One Zone as a caching layer for S3 Standard. It sets up a data transfer pipeline using AWS Step Functions and AWS DataSync to move frequently accessed data into S3 Express. This reduces access time and boosts performance significantly. In a test, ~2.9 TiB of data was transferred in 4 minutes 25 seconds at a cost of ~$20, enabling faster and lower-latency compute access. https://lnkd.in/e9m4YHmH Pablo Scheri

Looking for novel ways of network service acceleration? This recent article from NTT Network Service Systems Laboratories introduces the In-network Service Acceleration Platform (ISAP), a novel architecture integrating in-network computing (INC) with mobile networks for the 6G era. ISAP accelerates data processing by distributing computing functions across network devices, reducing the burden on user terminals and the cloud. The platform uses event-driven resource deployment and hardware acceleration chaining (GPUs, FPGAs, DPUs) to efficiently handle diverse applications like AI video analysis and metaverse services. The authors detail ISAP's architecture, implementation, evaluation, and demonstration experiments showcasing its capabilities in improving latency, jitter, and resource utilization. Future plans include proposing ISAP elements to international standardization organizations. #BellLabsConsulting

Chinese Scientists Unlock 10,000X Speed Boost in Optical Fiber with Neural Networks Breakthrough in Fiber Optic Bandwidth Researchers at the University of Shanghai have developed a neural network-based technique that can increase fiber optic speeds by a factor of 10,000—potentially reaching up to 125 terabytes per second. This discovery challenges existing assumptions about optical fiber bandwidth limitations and could revolutionize data transmission for high-performance computing, cloud infrastructure, and global internet connectivity. Overcoming Fiber Optic Bottlenecks While fiber optics is already the fastest data transfer medium, it has traditionally been constrained by bandwidth limits due to factors such as signal degradation, interference, and inefficient multiplexing methods. The Chinese research team has bypassed these limitations using neural networks, which optimize signal processing and error correction in ways that classical networking methods cannot. How Neural Networks Enhance Fiber Optics Unlike traditional approaches that manually adjust transmission parameters, neural networks can: • Dynamically optimize signal encoding and decoding to maximize available bandwidth. • Reduce interference and noise, allowing for higher-density data transmission. • Unlock previously untapped potential in fiber optics, making existing infrastructure significantly more efficient. Implications for Global Networking and AI If successfully implemented, this breakthrough could transform: • Cloud computing and data centers, enabling near-instantaneous data transfers. • AI model training, which relies on massive datasets and high-speed networking. • Telecommunications, potentially leading to faster, more efficient 6G and beyond. This discovery redefines the upper limits of optical fiber technology, offering unprecedented speeds that could reshape the future of internet and AI-driven infrastructure.

Ever wondered what makes Kafka and Redis (and other similar systems) powerhouses for high-performance systems? It all boils down to clever use of kernel-level optimizations. ✅ Kafka: Zero Copy for Lightning-Fast Throughput 🚀 Kafka owes much of its speed to the Zero Copy principle. It uses the system call sendfile() to directly transfer data from the OS buffer to the NIC (Network Interface Card) buffer. This eliminates unnecessary data copying and context switching, boosting throughput dramatically. ✅ Redis: Non-Blocking Magic with epoll ⚡ Ever wondered why Redis is so fast even though it's single threaded? Redis leverages the epoll() API to achieve its blazing-fast performance. epoll monitors multiple file descriptors for I/O readiness, making Redis’s single-threaded event loop non-blocking and incredibly efficient. ✅ The Key Takeaway: Optimize Deeply Both Kafka and Redis thrive on deep kernel-level optimizations. They’ve achieved massive popularity by identifying and solving specific bottlenecks at the OS level. Here’s my lesson from this: When building a custom solution, dig deep. Analyze where your CPUs or threads are spending most of their time, and see if kernel-level tweaks or optimizations can unlock game-changing performance. Curious to Learn More? If you're passionate about exploring technical concepts that drive high-throughput, low-latency systems, follow along for more insights! ___ PS: you can refer to the following two articles to learn more about Kafka and Redis Kafka: https://lnkd.in/gJGW8w4y Redis: https://lnkd.in/g9xNeqE5

In the realm of high-bandwidth, low-energy optical connectivity, a groundbreaking advancement is underway. Enter a laser-free optical interconnect leveraging micro-LEDs as the light source. Picture a scenario where numerous blue microLEDs are intricately connected to a photodetector array via multicore imaging fibers, facilitating data transmission speeds exceeding 1.6Tbps, all achieved at an incredibly low power consumption of 1-1.5pJ/bit. The spotlight shines on Avicena Tech, a Sunnyvale, CA startup, as semiconductor manufacturing giant TSMC aligns its support behind this innovative microLED-based connectivity initiative. Explore the full story in the recent IEEE Spectrum article: https://lnkd.in/gDPGUNyh Bardia Pezeshki Chris Pfistner Rob Kalman

Innovations in optical communications are quite literally moving at the speed of light to address high-speed data demands of AI Centers. In my latest substack post I go over the trends scaling optical communications beginning with conventional short reach optics: 🔷 Intensity modulation - Direct detection 🔷 Components (Transmitter, fiber and receiver) 🔷 Specs/Impairments that impact performance (Extinction Ratio, Q, dispersion, loss, etc) Then I talk about scaling solutions for long reach, high BW involving additional complexity and tighter integration: 🔷 Co-packaged optics 🔷 Silicon Photonics 🔷 External modulation 🔷 Coherent Optical I also wrote high-level primers on high-speed SerDes transceivers and selected subsystems (PLL, SAR ADCs, Bandgaps, and Signal integrity) that represent the interfaces that optical communications interacts with. Together, these provide a complete system-level view of high-speed data communications from first principles. For engineers to understand how their blocks impacts the overall system, as well as investors/marketers to get a solid optical communications foundation to understand trends in the space better. Link to the post is in the comments below.