Rust Replaces Python in Data Engineering Ecosystem

View organization page for SSP Data

2,185 followers

Will Rust kill Python in data engineering? No. But it has already consumed much of the JavaScript tooling ecosystem. And it's quietly doing the same in data. The pattern: Python remains the interface, Rust becomes the engine. Polars, DataFusion internals - all Rust under the hood, all Python on top. You don't need to learn Rust. But you should know what's happening beneath your Python code. https://lnkd.in/eyhTe44T

  • No alternative text description for this image

SSP Data Just to clarify: DuckDB is written in C++, not Rust.

DuckDB is Rust? I thought it was C++, no?

You don't need to know Rust but it definitely helps. If I knew Rust or C better I would probably instantly have several use cases where I throw out my web socket and use gRPC with a low-level coding language implementation on the server side for performance.

Cool article, I would add Lakekeeper as an interesting and useful open source project in the data engineering space.

I recently re-implemented a crucial step in my pipeline in R replacing Python, and the speed gain was jaw dropping.

Like
Reply

Yes, I agree. The only exception is that if performance is really crucial, #Rust might be adopted directly, just as we have seen with #Scala in #Spark.

See more comments

To view or add a comment, sign in

Explore content categories