A Stream Processing API for Go (with Generics)
Many years ago I created project Automi to demonstrate the use of Go channels as an underlying pipeline to implement complex stream processing of data. Though the original API was simple to use, its implementation relied heavily on runtime Go reflection. This reflection-based code always felt brittle and sometimes hard to reason about when adding new features.
Since generics support was introduced in Go back in version 1.18, I have always wanted to rewrite the project to use type parameters instead of runtime reflection. Over many recent weekends (and holidays), I was able to reimplement the code to use generics. Now, you can use Automi to compose powerful stream pipelines with compile-time type checks.
Example: streaming from a Go slices
Let’s explore an actual example that uses all of the components of Automi to create a working stream data processing.
The example above explicitly uses Go generic type instantiation to specify a concrete type for each stage of the pipeline.
Read the full post on Medium
There is a more complete blog post on Medium (since code-formatting is frustratingly bad here) where you can read about the project, its components, and how to use it.