Just released erlang_python 1.0.0 I wanted to use Python ML libraries from Erlang/Elixir without the usual headaches - no HTTP APIs, no message queues, no subprocess juggling. So I built this. erlang_python embeds Python directly in the BEAM VM using dirty NIFs. You call Python functions like they're local, and the type conversion just works. Same API from Erlang or Elixir. What's interesting: The GIL problem is solved three ways: - Python 3.12+ sub-interpreters each get their own GIL - Python 3.13 free-threaded builds have no GIL at all - Or just spread work across BEAM processes like you normally would It's async-native. Call Python async functions, stream from generators, run concurrent I/O with asyncio.gather. With streaming, you get tokens as they're generated which is great for LLMs. Good fit for embeddings, semantic search, and RAG pipelines. The docs have examples for all of that. Works on Linux, macOS, FreeBSD. Python 3.11+, OTP 27+. https://lnkd.in/eHh9txfe https://lnkd.in/eX_9cUVH #erlang #elixir #python #ml #ai
Why NIFs and not ports? My usual approach would be to keep ML processing at the python ports and then do whatever comms or coord are required at the beam level 🤔
Krasen Penchev
2mo
For Elixir, there’s already pythonx, which is well established. For Erlang, your tool might be the go to solution.
Karim Semmoud
2mo
I never would have thought in a million years that one day I’d be learning Python because of Erlang 😅 Thanks for the library, will give it a try
Loïc Tosser it may interest you 😉
Peri C.
2mo
This looks like a really solid approach to bridging those two ecosystems. The way it tackles the GIL problem feels like addressing one of the biggest pain points developers face when mixing these technologies.
Two new releases since this post! v1.1.0 - Shared state API to cache data across workers v1.2.0 - Context affinity (to keep ML models loaded), Python thread callbacks, and reentrant workflows I made a small write-up here: https://www.garudax.id/posts/benoitc_github-benoitcerlang-python-execute-python-activity-7428863511889362944-ZtT_?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAAAwLMBCi2PKJlZgIbyHuHO2mT3B9l7A_Q