How to Achieve Fast Data Transmission

Why does it take so long to load data from your database? This paper tackles the systems challenges underlying a common data science problem: loading large amounts of data from a database into an in-memory dataframe for data pipelines, ML inference, or exploratory data analysis. What I like the most about the paper is its opening: an in-depth analysis of a popular data loading operation (Pandas.read_sql) and why it takes so long to load data. The way this operation works is simple: it sends a SQL query to a database to request some data, deserializes it into Python objects, then converts them into NumPy arrays backing a Pandas dataframe. Somewhat surprisingly (or not, depending on how cynical you are about Python performance), more than 90% of the load time is spent client-side, deserializing and converting the data after it has been fetched from the server. Moreover, these operations use 4x more memory than the data actually requires. This happens because the underlying implementation is naive: it fetches all the data into an in-memory buffer, then converts it all into Python objects, then converts it all into a dataframe, with no parallelization of any step. Not content with complaining about the problem, the authors propose a solution: ConnectorX, an improved data loading library which is open-source and available on GitHub. The main idea is to stream data from database to dataframe. Before transferring anything, ConnectorX queries the DBMS for metadata about the query result, and then pre-allocates the entire dataframe. It then partitions the data client-side and assigns each partition to a separate worker-thread. Each worker thread streams data directly from the database into the dataframe, fetching small batches of query results from the DBMS, converting them into the proper format, then writing them to the dataframe. To make things really fast, all of this is written in Rust with Python bindings. Overall, this combination of a faster language, more parallelism, and a pipelined streaming approach reduces client-side overhead from 10x to almost nothing.

One of the most challenging and exciting components of #HFT software development is the market data feed handler. The market data feed handler is responsible for receiving, decoding, and processing the market data from various sources, such as exchanges, brokers, and vendors.  The market data feed handler must be able to handle high volumes, high frequencies, and high variabilities of data, as well as deal with issues such as latency, bandwidth, and reliability. Here are some of the most important techniques and tricks that I use to optimize the performance and quality of my market data feed handler: 1/5. I use a dedicated thread or process to receive the market data from the network and store it in a circular buffer. This way, I can avoid blocking or delaying the data reception due to other tasks or operations. 2/5. I use a fast and lightweight protocol, such as FIX, FAST, or ITCH, to encode and decode the market data. This way, I can reduce the size and complexity of the data and improve the parsing speed and efficiency. 3/5. I use a custom data structure, such as a hash table with plain arrays, to store and access the market data in memory. This way, I can optimize the data lookup and retrieval based on the key or symbol of the data. 4/5. I use a profiling or a monitoring tool to measure and analyze the performance and quality of my market data feed handler. This way, I can identify and eliminate any bottlenecks. 5/5. I use a parallel or a distributed technique to process and transmit the market data. This way, I can leverage the power and resources of multiple cores, processors, and improve the data processing and transmission scalability and speed.

File Compression Explained 🔥 Zip reduces data size by encoding it into a more compact form, enabling efficient storage management and optimized data transmission. 𝗘𝘅𝗽𝗹𝗼𝗿𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 𝗧𝘆𝗽𝗲𝘀 • Lossless: Formats like ZIP, PNG, FLAC ensure no data is lost during compression. This type is used for text, executables or any data to guarantee the integrity. • Lossy: Formats like JPEG, MP3 and many video codecs discard less critical data to achieve higher compression ratios. For multimedia where slight degradation is acceptable. 𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗔𝗽𝗽𝗹𝘆 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻? 𝟭. 𝗙𝗶𝗹𝗲 𝗛𝗮𝗻𝗱𝗹𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 If your app handles large data (text files, images, databases, logs), compression algorithms can optimize storage and reduce file transfer times. • File Compression: Use libraries like zlib, gzip or 7-zip to compress files before storing or sending over a network. • Databases: PostgreSQL and MongoDB support data compression, but you can also implement column or record compression using algorithms to save storage. • Compressed Logs: Automate log compression with tools like gzip to save disk space. 𝟮. 𝗠𝗲𝗱𝗶𝗮 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 Compression is key to improving load times and user experience, especially in streaming or real-time content. • Images: Use JPEG or PNG to reduce file sizes without losing too much quality. Libraries like Pillow (Python) or Sharp (Node.js) can compress images efficiently. • Streaming: Implement codecs like H.264 or H.265 using libraries like FFmpeg to compress video while conserving bandwidth. Ideal for video calls, live streaming, or platforms. • Audio: Apply lossy compression with formats like MP3 or AAC to reduce file sizes while maintaining audio quality. 𝟯. 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 𝗶𝗻 𝗔𝗣𝗜𝘀 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮 𝗧𝗿𝗮𝗻𝘀𝗳𝗲𝗿𝘀 When working with APIs that handle large datasets, compression can reduce network traffic and improve performance. • REST APIs: Use GZIP or Brotli to compress API responses, reducing data sent to clients. • WebSockets: Use compression to reduce the amount of data sent over WebSocket connections, optimizing real-time message transmission. • File Transfers: Implement compression for large file transfers between client and server to reduce upload/download times. 𝟰. 𝗖𝗹𝗼𝘂𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗮𝗻𝗱 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 Apply compression to save resources and reduce costs in cloud environments. • Cloud Data Compression: Services like Amazon S3 or Google Cloud Storage allow storing compressed files. Automatically compress files before uploading to save space. • Data Transfer in Microservices: Compress HTTP or gRPC payloads to reduce latency and improve scalability in microservice-based systems. -- 📷 Visualizing Software Engineering, AI and ML concepts through easy-to-understand Sketᵉch. I'm Nina, software engineer & project manager. Sketᵉch now has a LinkedIn Page. Join me! ❤️ #datastorage #devops #softwaredevelopment

Your PLC network is constantly slow. The engineer who knows how to control network traffic is the one who eliminates the lag. You have a big network with 50+ devices and 10 PLCs. Everything is connected. But the network lag is causing minor, unpredictable delays in I/O. So you are stuck. - You are spending time looking for faulty cables or bad switches. - The lag makes your machine feel sluggish and unreliable. - The real problem is a network flood of unnecessary data from your PLCs. This is not a hardware fault. It is an uncontrolled data flow. PLCs send background data that floods the network, slowing down the important I/O traffic. Stop blaming the network card. Start controlling the data flow. The solution is a single checkbox in TIA Portal that puts the brakes on non-critical traffic. Here is the 3-step solution to stabilize your network performance. 1. You find the hidden option. Go to the PROFINET Interface settings in TIA Portal. Look under Advanced Options > Interface Options. 2. You check the box. Select the box: "Limit data infeed into the network." This is your master switch for network control. 3. You stop the data flood. This prevents the PLC from sending non-critical background data that chokes the network. It frees up bandwidth for the fast, real-time I/O data. You stop wasting hours troubleshooting network lag. You start building networks that are clean, fast, and stable. This simple check separates the network engineer from the novice. That is the skill that gets you noticed. ♻️ If you found this useful, repost it. Share your best network stability trick (e.g., specific switch settings, etc.) in the comments. #Siemens #PROFINET #IndustrialAutomation #Engineering #PLC #TIAportal #Troubleshooting #Network #ControlsEngineering #Downtime #Automation #AI #Research #Problemsolving

Ever wondered what makes Kafka and Redis (and other similar systems) powerhouses for high-performance systems? It all boils down to clever use of kernel-level optimizations. ✅ Kafka: Zero Copy for Lightning-Fast Throughput 🚀 Kafka owes much of its speed to the Zero Copy principle. It uses the system call sendfile() to directly transfer data from the OS buffer to the NIC (Network Interface Card) buffer. This eliminates unnecessary data copying and context switching, boosting throughput dramatically. ✅ Redis: Non-Blocking Magic with epoll ⚡ Ever wondered why Redis is so fast even though it's single threaded? Redis leverages the epoll() API to achieve its blazing-fast performance. epoll monitors multiple file descriptors for I/O readiness, making Redis’s single-threaded event loop non-blocking and incredibly efficient. ✅ The Key Takeaway: Optimize Deeply Both Kafka and Redis thrive on deep kernel-level optimizations. They’ve achieved massive popularity by identifying and solving specific bottlenecks at the OS level. Here’s my lesson from this: When building a custom solution, dig deep. Analyze where your CPUs or threads are spending most of their time, and see if kernel-level tweaks or optimizations can unlock game-changing performance. Curious to Learn More? If you're passionate about exploring technical concepts that drive high-throughput, low-latency systems, follow along for more insights! ___ PS: you can refer to the following two articles to learn more about Kafka and Redis Kafka: https://lnkd.in/gJGW8w4y Redis: https://lnkd.in/g9xNeqE5