Will Rust kill Python in data engineering? No. But it has already consumed much of the JavaScript tooling ecosystem. And it's quietly doing the same in data. The pattern: Python remains the interface, Rust becomes the engine. Polars, DataFusion internals - all Rust under the hood, all Python on top. You don't need to learn Rust. But you should know what's happening beneath your Python code. https://lnkd.in/eyhTe44T
DuckDB is Rust? I thought it was C++, no?
The updated list Open-Source Tools in Rust is at https://www.ssp.sh/brain/great-open-source-tools-in-rust/
You don't need to know Rust but it definitely helps. If I knew Rust or C better I would probably instantly have several use cases where I throw out my web socket and use gRPC with a low-level coding language implementation on the server side for performance.
Cool article, I would add Lakekeeper as an interesting and useful open source project in the data engineering space.
I recently re-implemented a crucial step in my pipeline in R replacing Python, and the speed gain was jaw dropping.
Yes, I agree. The only exception is that if performance is really crucial, #Rust might be adopted directly, just as we have seen with #Scala in #Spark.
SSP Data Just to clarify: DuckDB is written in C++, not Rust.