From Python to C++: Building a Compiler for Performance and Parallelism Xavier Thompson from Nexedi shared a fascinating deep dive into how his team built a Python → C++ compiler, called Typon, designed to unlock true concurrency and performance - without abandoning Python’s simplicity. Python’s flexibility comes with trade-offs - the Global Interpreter Lock (GIL) and interpreter overhead often limit scalability for CPU-bound workloads. Nexedi, whose open-source ERP5 system generates over a million invoices per month, needed a way to scale further without rewriting everything in another language. Enter Typon: A subset of Python that compiles directly to C++, Typon offers: - Built-in concurrency primitives (fork and sync) for structured parallelism. - Automatic memory management and Python-like semantics. - Interoperability with Python libraries (e.g., NumPy) while avoiding GIL constraints. - Native performance through C++ compilation and async I/O handling. Xavier also shared how Typon’s scheduler enables massive parallelism - turning a simple Python web server into one capable of handling tens of thousands of concurrent connections. Under the hood: - The compiler is written in Python itself, leveraging the built-in ast module. - It performs type inference, supports generics, and generates C++ code that mimics Python’s object model. - The team is now building a Typon standard library, with C++ bindings and Python stubs for seamless integration. Typon is now available on PyPI - an exciting milestone for open-source compiler innovation. Learn more or try it out: https://lnkd.in/e9zgeTmt #Python #Cplusplus #CompilerDesign #OpenSource #PerformanceEngineering #Concurrency #ParallelComputing #EuroPython #MLOps #SoftwareEngineering
Alex C.’s Post
More Relevant Posts
-
Your beautiful async Python system just froze. You called one synchronous function—maybe a legacy library, maybe some file I/O—and suddenly your entire event loop is blocked. Every coroutine waits. Every websocket hangs. Everything stops. I see this pattern destroy production systems constantly. The solution? Python's Executor pattern—the bridge between async and sync worlds that most developers never learn properly. In my latest article, I break down: ✅ ThreadPoolExecutor vs ProcessPoolExecutor (and when to use each) ✅ How to wrap blocking libraries with clean async interfaces ✅ The functools.partial trick nobody teaches you ✅ Real production patterns: database pools, connection management ✅ The shutdown behavior that prevents memory leaks This is senior-level Python. No fluff. Just the patterns you need when async meets the real world. 🔗 Read: "The Executor: Running Blocking Code Without Blocking" https://lnkd.in/de9HNqWA #python #coding #programming #softwaredevelopment
To view or add a comment, sign in
-
-
A New Era for Multithreading in Python For years, Python developers hit a hard limit: the Global Interpreter Lock (GIL) prevented true multithreading for CPU-bound tasks. Meanwhile, Go’s goroutines and scheduler delivered effortless concurrency and parallelism — one of the reasons Go became a favorite for high-performance systems. But that gap is starting to close. With Python 3.14, the long-awaited free-threaded (no-GIL) build is officially supported. Here’s what changes: 🔹 True parallelism in threads No-GIL means multiple Python threads can now execute bytecode simultaneously on multiple cores — something previously only Go could do efficiently. 🔹 New concurrency primitives Python adds concurrent.interpreters, enabling multiple isolated interpreters inside one process (similar to goroutines’ lightweight nature, though not as granular). 🔹 Improved thread safety & adaptive optimizations The standard library has been hardened for thread safety, and the adaptive interpreter keeps single-thread performance close to pre-3.14 levels. Takeaway: Python 3.14 doesn’t replace Go for concurrent systems yet — but it finally gives Python developers a native path to real multithreading without multiprocessing hacks. The GIL era is ending.🚀 #Python #GoLang #Concurrency #Multithreading #NoGIL #Python314 #Programming
To view or add a comment, sign in
-
Why Python 3.14 is a Game-Changer for Developers If you’ve ever felt frustrated by Python’s concurrency limits, there’s great news for you. Python 3.14 introduces free-threaded builds (no more global interpreter lock = GIL) and support for parallel sub-interpreters. These changes open the door to true multicore performance for CPU-intensive Python workloads. ✅ What this means: 1. Threads can run Python byte-code truly in parallel when built with GIL disabled, unlocking major gains especially in CPU-bound scenarios. 2. Sub-interpreters: You can isolate workstreams in the same process with less overhead than full processes. 3. Everyday developer quality-of-life upgrades: Better error messages, improved REPL, deferred type annotations, and syntax enhancements. ⚠️ Limitations & things to watch: 1. The GIL is still enabled by default. You’ll need to build or use a “free-threaded” interpreter variant to reap full parallel benefits. 2. Some third-party C-extensions and libraries may not yet be compatible with GIL-free builds, which could slow adoption in production. 3. Single-threaded code might see little or no benefit — or even a small performance dip in some cases. Whether you’re working on backend services, data-processing pipelines, or computational workflows — Python 3.14 is worth keeping an eye on and planning for. As always, test carefully before upgrading mission-critical systems. #python #programming #developers #multithreading #performance #softwareengineering
To view or add a comment, sign in
-
-
There's a fundamental assumption most Python developers make every single day. And it's silently breaking your code. This isn't about complex libraries or advanced architecture. It's a flaw hidden in one of the most basic, seemingly harmless operations we perform. It’s the real reason why code that passes every test on your machine can suddenly crumble when faced with real-world data—from logs, user inputs, or external files. I've seen it cause data corruption, silent failures, and hours of wasted debugging on problems that seem to have no source. The truth is, a hidden chaos exists in the data we process, and most conventional methods are completely blind to it. The solution is deceptively simple, yet most tutorials and even some senior developers overlook it. In my new video, I expose this hidden vulnerability and reveal the one tiny adjustment that bulletproofs your code against this specific type of chaos. This is the missing piece that turns fragile scripts into resilient, professional-grade software that you can trust in any environment. Stop letting "it works on my machine" be your standard. Watch the short video to see the detail that changes everything. https://lnkd.in/eyRUJKvc
Python's `.split()` vs `.splitlines()`: How to Use Them Properly
https://www.youtube.com/
To view or add a comment, sign in
-
🚀 Day 37 — Effective Python Coding Series Asyncio Beyond I/O: Handling CPU-Bound Tasks Efficiently ⚙️ Most people know asyncio as the go-to tool for I/O-bound tasks — like network calls or database requests. But did you know it can also handle CPU-bound workloads effectively? 👀 💡 Here’s the trick: When CPU-heavy tasks block the event loop, your async app can freeze. To avoid that, offload those heavy computations to a ThreadPoolExecutor or ProcessPoolExecutor — while the event loop stays responsive. 🧠 Core Idea: loop.run_in_executor() lets you execute CPU-heavy tasks elsewhere Async code remains non-blocking You get both responsiveness and parallelism ✅ Benefits: Offload CPU-intensive computations Mix I/O + CPU concurrency Keep apps fast and responsive ⚙️ When to use which: ThreadPoolExecutor → lightweight CPU tasks ProcessPoolExecutor → heavy, multi-core computation In short: Asyncio + Executors = True Hybrid Concurrency in Python 💪 #Python #Asyncio #Concurrency #EffectivePython #BackendDevelopment #Developers
To view or add a comment, sign in
-
-
Local computer + ollama pull gpt-oss:20b Topic: Python Code -> C++ Code system_prompt = """ Your task is to convert Python code into high performance C++ code. Respond only with C++ code. Do not provide any explanation other than occasional comments. The C++ response needs to produce an identical output in the fastest possible time. """ def user_prompt_for(python): return f""" Port this Python code to C++ with the fastest possible implementation that produces identical output in the least time. The system information is: {system_info} Your response will be written to a file called main.cpp and then compiled and executed; the compilation command is: {compile_command} Respond only with C++ code. Python code to port: ```python {python} ``` """
To view or add a comment, sign in
-
-
Is Python 🐍 safe?" That's a fascinating question! 🤔 In one critical way, YES, absolutely! 👍 Python 🐍 is wonderfully memory-safe. It has a garbage collector 🗑️, which means developers don't manually manage memory. This completely prevents a whole class of terrifying bugs 👻 like buffer overflows or use-after-frees that plague older languages. You're safe from those! 🥳 However, "safe" has other meanings! Python's 🐍 main "risk" is its dynamic typing. Type errors (like TypeError) are only caught when the code runs 🏃♂️, which can sometimes mean in production! 💥 This is a totally different philosophy from a language like Rust 🦀. Rust 🦀 is also memory-safe, but it provides this guarantee in a unique way: at compile-time ⏳ with its famous Borrow Checker 🛡️. No garbage collector needed! This means Rust 🦀 catches memory and type errors before the program can even run. 🚫 When it comes to concurrent programming, Rust 🦀 also provides "fearless concurrency" 🚀, a powerful compile-time safety guarantee against data races that Python doesn't offer. So, is Python 🐍 safe? Yes, it protects you from memory chaos. Is Rust 🦀 safer? It offers a stricter, more comprehensive set of safety guarantees (memory, type, and thread safety) that are enforced by the compiler 🦾. It's all about trade-offs! ⚖️ #PythonSafety #RustLang #Programming #TechTalk #MemorySafety #FearlessConcurrency #CodingDebate #SoftwareDevelopment #DeveloperLife
To view or add a comment, sign in
-
-
Just building a custom data type system in C for Python. Introducing Atom - a minimal, performant library implementing fundamental numeric types for Python, including: ➡️ Standard types: int8–64, float16–64, complex ➡️ Machine learning types: bfloat16 ➡️ Type introspection: finfo / iinfo similar to NumPy The system is written entirely from scratch using: ➡️ C11 for the core type definitions and operations ➡️ Python C API for bindings ➡️ Bazel for reproducible, incremental builds All unit tests are passing, and the next milestone is implementing type casting. This project has been a deep dive into low-level systems programming and an exploration of how libraries like NumPy handle data types internally. Repository: https://lnkd.in/e5UwKrfF
To view or add a comment, sign in
-
I think my initial hypothesis about the most valuable languages for GitHits was wrong. The data now points to C, C++, and Rust (among others) being in more demand than Python, TS, or JS. It makes sense in hindsight. LLMs are biased toward scripting languages because those dominate their training data. I’ll be rolling out support for some of the less-represented languages soon, after gathering feedback from users. Sometimes the data simply proves your assumptions wrong.
To view or add a comment, sign in
Explore content categories
- Career
- Productivity
- Finance
- Soft Skills & Emotional Intelligence
- Project Management
- Education
- Technology
- Leadership
- Ecommerce
- User Experience
- Recruitment & HR
- Customer Experience
- Real Estate
- Marketing
- Sales
- Retail & Merchandising
- Science
- Supply Chain Management
- Future Of Work
- Consulting
- Writing
- Economics
- Artificial Intelligence
- Employee Experience
- Workplace Trends
- Fundraising
- Networking
- Corporate Social Responsibility
- Negotiation
- Communication
- Engineering
- Hospitality & Tourism
- Business Strategy
- Change Management
- Organizational Culture
- Design
- Innovation
- Event Planning
- Training & Development