Python GIL explained: why it exists and how it affects multithreading Many developers hear about the Python GIL and assume one thing: “Python can't use multiple CPU cores.” But the reality is a bit more nuanced. What the GIL actually is The Global Interpreter Lock (GIL) is a mutex inside CPython that ensures only one thread executes Python bytecode at a time. Even if your program has multiple threads, only one can run Python code simultaneously. Example: Thread A → running Thread B → waiting Thread C → waiting This means Python supports threads, but they take turns executing code. Why Python has the GIL The main reason is memory management. CPython relies heavily on reference counting for garbage collection. Every Python object tracks how many references point to it. When the count reaches zero, the object can be safely removed from memory. If multiple threads updated these counters simultaneously, memory corruption could occur. The GIL prevents this by ensuring only one thread modifies Python objects at a time. Trade-off: simpler interpreter design safer memory management faster single-thread performance How the GIL affects multithreading The limitation mainly impacts CPU-bound workloads. Example: def compute(): for i in range(10_000_000): pass Running this across multiple threads does not give real parallelism because threads must wait for the GIL. Thread A → holds GIL Thread B → waiting Thread C → waiting The workload becomes effectively serialized. Why Python threads still work well for web servers Python threads still scale well for I/O-bound tasks. Examples: API requests database queries file operations While a thread waits for I/O, the interpreter releases the GIL, allowing another thread to run. That's why frameworks like FastAPI and Django can handle many concurrent requests. Common ways developers work around the GIL 1️⃣ Multiprocessing Each process has its own interpreter and its own GIL. 2️⃣ Native extensions Libraries like NumPy or PyTorch run heavy computations in C/C++ and release the GIL. 3️⃣ Async I/O Libraries like asyncio allow thousands of concurrent tasks in a single thread. ## Interesting development: Python without the GIL Simple visual model Threads → Queue → GIL → CPU Only one thread can pass through the GIL and execute Python bytecode. Question for the community: How do you handle CPU-bound workloads in Python? multiprocessing distributed workloads native extensions #Python #PythonProgramming #BackendDevelopment #SoftwareEngineering
Evgenii Klimenko’s Post
More Relevant Posts
-
Python isn't "just a script" it's optimized and compiled + interpreted engine. The classic Python Integer Cache concept that actually teaches a lot about memory and tells us that it's a highly optimized engine. Program: x = 256 y = 256 print(x is y) # True! a = 257 b = 257 print(a is b) # Expected was False but Actual Answer is True Range of Integer Cache in Python is -5 to 256. Wait... why does Python treat 256 and 257 as the exact same object in memory? Welcome to the Integer Cache and Python Optimization. Whenever you use a number in that range, Python doesn't create a new object. It just points your variable to the pre-existing object in the cache. Once you hit 257, you've stepped out of bounds, and Python starts dynamically allocating new memory on the heap for every variable. But it goes even deeper. Modern Python uses specialized bytecode instructions (like LOAD_SMALL_INT vs LOAD_CONST) to fetch these numbers based on physical 1 byte memory limits in the virtual machine itself. small_ints globally caches numbers from -5 to 256 at startup, whereas co_consts locally stores addresses for all other constants during compile time. LOAD_SMALL_INT uses its argument as a direct math shortcut to instantly fetch objects from the small_ints cache. LOAD_CONST uses its argument as an index to look up and retrieve objects stored in the co_consts locker. This is magic of Python implemented in CPython #CPython #PythonDeveloper #ComputerScience #BackendDevelopment #SoftwareArchitecture #TechTips #PythonProgramming #DeveloperCommunity #ProgrammingLife #TechCommunity
To view or add a comment, sign in
-
-
Waited 16 seconds for a Python loop to finish? I did. Then I added one decorator and it dropped to 0.18 seconds. That is a 128x speedup, and the only change was this: @njit In my latest post, I break down exactly why Python loops are slow at the bytecode level, how Numba bypasses the interpreter entirely, and where Numba actually makes things worse (hint: strings). Covered in the post: - How the Python interpreter processes opcodes on every loop iteration - Why @njit works so well for NumPy workloads - Why @njit(parallel=True) crashes on string operations - Benchmark results on Apple M4 Pro with 20 million element arrays If you work with numerical data in Python, this one is worth 5 minutes. https://lnkd.in/gV3W_YCg #Python #Numba #NumPy #Performance #MachineLearning
To view or add a comment, sign in
-
Python Variables Explained Simply (With Real Examples) In Python, everything you work with is data. When you create a variable, Python: Creates the data in memory Assigns a reference (variable name) to it Think of a variable as a label stuck on a box. Simple code = age = 25 Here , Age is a variable and 25 is the value assigned to it. Python does not require any type declaration. Because it's type is already determined. age = 25 # integer price = 19.99 # float name = "Alex" # string is_active = True # boolean A simple coding example which defines about user profile by using different datatypes : name = "Rahul" age = 24 height = 5.9 is_student = True print("Name:", name) print("Age:", age) print("Height:", height) print("Student Status:", is_student) Dynamic Typing : As informed earlier ,Python allows changing type dynamically. x = 10 print(type(x)) x = "hello" print(type(x)) Output : <class 'int'> <class 'str'> Because of this flexibility, python is fast for development. Important Concept: Checking Data Type Python provides type().Useful in debugging and validation. age = 22 print(type(age)) output : <class 'int'> #Python #Programming #Coding #LearnToCode #Beginners #TechCareer #SelfLearning
To view or add a comment, sign in
-
A fascinating design choice in Python: Everything is an object. Not just data. Also: • numbers • functions • classes • modules Even this: def hello(): pass print(type(hello)) Outputs: <class 'function'> That means functions are not just callable code. They are first-class objects. Internally, in CPython, functions are instances of PyFunctionObject, carrying: • a code object (__code__) • a global namespace reference (__globals__) • default arguments (__defaults__) • closure variables (__closure__) This design unlocks powerful behavior. You can: • pass functions as arguments • return them from other functions • store them in variables • attach attributes to them Which leads to patterns like: • decorators • higher-order functions • dynamic behavior at runtime And this is the key insight. Python doesn’t separate “data” from “behavior” the way many languages do. It treats both as objects with state and identity. That’s why the language feels so flexible. You’re not just calling functions. You’re manipulating objects that happen to be executable. Python is AMAZING!
To view or add a comment, sign in
-
🔒 Name Mangling in Python — Why Private Attributes Aren't Really "Private"! Just uncovered an interesting Python quirk—how "private" attributes can still be accessed (if you know the trick)! 🤔 🔍 What's Actually Happening? ✅ "Name Mangling" – Python renames '__balance' to '_BankAccount__balance' internally. ✅ "Private" is a convention—not true security, just a warning. ✅ "The Trap"—'b1.__balance = 9999999' creates a "new" public attribute. ✅ "Real "Balance"—Still protected inside '_BankAccount__balance' 💡 Key Insight: | "What You Wrote" | "What Python Stores" | |--------------------|------------------------| | 'self.__balance' | '_BankAccount__balance' | | 'b1.__balance' | Creates NEW attribute '__balance' | 📌 How to Actually Access Private Attribute (If You Must): ***python*** # Not recommended, but possible! print(b1._BankAccount__balance) # Output: 12000 💡 Best Practices: ✅ Use "single underscore" ('_balance') for "protected" (convention). ✅ Use "double underscore" ('__balance') for name mangling (prevents accidental access). ✅ "Respect privacy"—don't access mangled attributes directly. ✅ Use "getter/setter methods" for controlled access 📌 Real-World Lesson: Python trusts developers to be responsible. Private attributes are a "convention," not a security feature. The language says, "Here's how to protect it, but if you really want to break it, you can!" #Python #OOP #Encapsulation #NameMangling #Coding #Programming #LearnPython #Developer #Tech #PrivateAttributes #PythonTips #CodingLife #SoftwareDevelopment #PythonInternals #Day57
To view or add a comment, sign in
-
-
Python 3.15 might be shipping one of the most exciting performance tools Python has added in a while, that too right after the GIL-free release. It is called Tachyon. Python is adding a new statistical sampling profiler under `profiling.sampling` called Tachyon, and the positioning is what caught my attention: - attach to a running Python process - no code changes - no restart - designed for low-overhead profiling - supports outputs like flamegraphs, speedscope-compatible stacks, Firefox Profiler, heatmaps, live TUI, async-aware views, and even opcode-level profiling The docs go even further and describe it as capable of sampling at up to 1,000,000 Hz and suitable for production performance debugging. With ML pipeline domination and RESTful servers now written increasingly in Python, this profiler will be a great addition. No longer will the notion of “run a profiler locally and hope the issue reproduces” carry on, and profiling becomes a first-class citizen, as in languages like Java. This is from the Python 3.15 alpha docs, so details may change before final release. But this is absolutely worth watching. A big thanks to the Python Software Foundation and Hugo van Kemenade for the updates and articles. Read more in the link below: https://lnkd.in/eUJ3C4N7 #Python #Python315 #Performance #Profiling #SystemsEngineering #Observability #Backend #SoftwareEngineering
To view or add a comment, sign in
-
📢 Day 2 of my Python series is LIVE on MrCloudBook! 🐍 𝗣𝘆𝘁𝗵𝗼𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 & 𝗘𝘅𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻𝘀 — the building blocks that make your variables actually DO something! In Day 1, we covered variables and data types — the nouns of Python. Day 2 is all about the verbs. ✅ Here's what's inside: 🔢 Arithmetic operators — including the 3 that surprise every beginner: //, %, ** 🔍 Comparison operators — and the classic = vs == trap 🧠 Logical operators — and, or, not (with short-circuit evaluation!) ✅ Truthiness — what Python considers True or False 📝 Assignment operators — +=, -=, *= and more 🔤 String operators — +, *, and in 🎯 Operator precedence — so your expressions mean what you think they mean 💼 A complete Invoice Calculator project using every concept from the article If you're starting your Python journey or know someone who is — this one's for you. 🙌 👇 Read it here: https://lnkd.in/gSqznx_T #Python #LearnPython #PythonForBeginners #MrCloudBook #DevOps #100DaysOfCode #Programming #TechCommunity
To view or add a comment, sign in
-
Your Python logs are lying to you. 🚩 Most server logs are parsed line-by-line in Python. It’s the industry standard because it's easy. But it’s slow, and more importantly, it can be inaccurate. I just benchmarked a 10M row server log ingestion using standard Python vs. a custom C-Hybrid engine I built. Here are the results: 🚀 Execution Speed: 1.01s (Python) ➡️ 0.20s (Hybrid C) 🛡️ Data Integrity: Detected 180 "Ghost" errors that standard parsing missed. Why the difference? Standard line-by-line readers are "blind" to strings sliced exactly across I/O memory boundaries. If a status code like " 500 " is split between two chunks of data, standard iteration skips it. I solved this by building a Hybrid Engine that uses: 1️⃣ 8KB Binary Buffered I/O: Reading raw bytes directly into RAM. 2️⃣ Boundary Overlap Logic: Ensuring no string is ever "sliced" out of existence. 3️⃣ C-Python Bridge: Bringing C-level speed into a Python workflow using ctypes. The ROI: A 5x speedup and 100% data integrity. At enterprise scale (Netflix/Uber), this is the difference between catching a critical security signal and wasting thousands in unnecessary compute costs. 📂 Source Code: https://lnkd.in/g6Vv7DN2 I’m opening 3 slots for free performance audits on data pipelines this week. If your logs are slow or you suspect your numbers aren't 100% accurate, DM me 'OPTIMIZE'. #Python #CProgramming #DataEngineering #PerformanceOptimization #Backend #SoftwareArchitecture #ZeroLatency
To view or add a comment, sign in
-
-
🧠 Python Concept: zip() — Iterate Multiple Lists Together Stop using messy index loops 😵💫 ❌ Traditional Way names = ["Alice", "Bob", "Charlie"] scores = [85, 90, 95] for i in range(len(names)): print(names[i], scores[i]) ✅ Pythonic Way names = ["Alice", "Bob", "Charlie"] scores = [85, 90, 95] for name, score in zip(names, scores): print(name, score) 🧒 Simple Explanation Think of zip() like a zipper 🤐 ➡️ It joins multiple lists together ➡️ Pairs items one by one Example: ("Alice", 85), ("Bob", 90), ("Charlie", 95) 💡 Why This Matters ✔ Cleaner and readable code ✔ No index errors ✔ Works with multiple lists ✔ Very common in real-world data handling ⚡ Bonus Tip Unzip values: pairs = [("Alice", 85), ("Bob", 90)] names, scores = zip(*pairs) print(names) print(scores) 🐍 Python makes parallel iteration simple 🐍 Use zip() to write smarter loops #Python #PythonTips #CleanCode #LearnPython #Programming #DeveloperLife #100DaysOfCode
To view or add a comment, sign in
-
-
🔥 Python Set Methods If you're learning Python, mastering sets is a must! Here's a simple and clean reference 👇 📌 Python Set Methods Table Input → Method → Output {1, 2} → .add(3) → {1, 2, 3} {1, 2, 3} → .remove(2) → {1, 3} {1, 2, 3} → .discard(4) → {1, 2, 3} {1, 2, 3} → .pop() → Random element removed {1, 2} → .update({3, 4}) → {1, 2, 3, 4} {1, 2, 3} → .clear() → set() {1, 2} → .union({3, 4}) → {1, 2, 3, 4} {1, 2, 3} → .intersection({2, 3, 4}) → {2, 3} {1, 2, 3} → .difference({2}) → {1, 3} {1, 2} → .symmetric_difference({2, 3}) → {1, 3} {1, 2} → .issubset({1, 2, 3}) → True {1, 2, 3} → .issuperset({1, 2}) → True {1, 2} → .isdisjoint({3, 4}) → True {1, 2} → .copy() → {1, 2} 💡 Pro Tip: Use .discard() instead of .remove() when you're not sure if the element exists. 📚 Save this for quick revision & share with your friends! #Python #Coding #Programming #Developers #LearnPython #CodingTips #DataStructures
To view or add a comment, sign in
-
Explore content categories
- Career
- Productivity
- Finance
- Soft Skills & Emotional Intelligence
- Project Management
- Education
- Technology
- Leadership
- Ecommerce
- User Experience
- Recruitment & HR
- Customer Experience
- Real Estate
- Marketing
- Sales
- Retail & Merchandising
- Science
- Supply Chain Management
- Future Of Work
- Consulting
- Writing
- Economics
- Artificial Intelligence
- Employee Experience
- Workplace Trends
- Fundraising
- Networking
- Corporate Social Responsibility
- Negotiation
- Communication
- Engineering
- Hospitality & Tourism
- Business Strategy
- Change Management
- Organizational Culture
- Design
- Innovation
- Event Planning
- Training & Development