Python Memory Management: GC, References & Closures

Ever screamed at your screen because Python changed a variable you never touched?   Or a function suddenly "remembered" values from previous calls?   Or a SyntaxError pointed to a line that looked perfect? These aren't random bugs — they're Python's design decisions in action. And they trip up beginners and experienced devs alike. I wrote the guide I wish existed when I started:   "Getting Started with Python: Overview & Real-World Applications"   Not another "Python is readable" list — but a practitioner's breakdown of the **8 core surprises** that explain most "why does this behave that way?" moments in your first year. The 8 problems covered: - Terminal says Python doesn't exist (PATH hell) - Error on a line that looks fine (parser vs runtime) - Changing one variable changes another (name binding) - Function modifies input it should only read - Mutable defaults trap — function remembers across calls - "1992" isn't a number (input() strings) - Code runs but nobody understands it (naming/docstrings) - Windows paths break silently (escape sequences/raw strings) Plus: how these same concepts power real-world Python in data science (Pandas views/copies), web (Django/FastAPI), and automation. If you've ever wasted hours debugging a "perfectly logical" Python script — this post gives you the mental model to stop it. Read it here: https://lnkd.in/gcsHx66Q What's the #1 Python surprise that cost you the most time early on? Drop it below — let's commiserate and learn from each other. #Python #LearnPython #PythonBeginners #ProgrammingTips #DataScience #Coding (Full Python Fundamentals series linked inside — 13 articles building from install to production concepts)

🐍 Python Practice Progress ( 21 out of 50 question solved) Here are the problems I recently solved: ✅ Count even and odd numbers in a list ✅ Reverse a list without using reverse() ✅ Insert the third largest number at the second last position in a list. ✅ Find the sum of all elements in a list ✅ Merge two lists ✅ Find common elements between two lists ✅ Sort a list without using sort() Working through these exercises helped reinforce concepts like: • loops and indexing • list operations • conditional logic • problem-solving without relying on built-in shortcuts Sometimes going back to the basics is the best way to build stronger foundations. Day 3 ------------------- Count even and odd numbers in a list. try:     Num1=[1,2,3,4,5,6,7,8,9,9,88]     even=0     odd=0     for i in Num1:         if i%2==0:             even=even+1         elif i%2!=0:             odd=odd+1     print("even are ",even)     print("odd are ", odd) except ValueError:     print("Some error occured.") Reverse a list without using reverse(). list1=[1,2,3,4,5,6,7] rev_list=[] rev_list=list1[::-1] print(rev_list) # or try:     list1=[1,2,3,4,5,6,7]     rev_list=[]     for i in list1:         rev_list.insert(0,i)     print(rev_list) except ValueError:     print("error occured.") or try:     list1=[1,2,3,4,5,6,7]     rev_list=[]     for i in range(len(list1)-1,-1,-1):         rev_list.append(list1[i])     print(rev_list) except ValueError:     print("error occured.") Insert third largest number in a list and add it at second last position. try: list1=[1,2,76,84,989,23,4,5,666,7,788,8,9897] third_largest=sorted(list1)[-3] list1.insert(len(list1)-1,third_largest) print(list1) Find the sum of all elements in a list. try:     list1=[2,4,5,4,44,33,65,898,90]     s=0     for i in list1:         s=s+i     print(s) except ValueError:     print("error occured") Merge two lists. try:     list1=[2,3,4,4,5,6,7]     list2=[99,8,0,4,2,1]     c=list1+list2     print(c) except ValueError:     print("error occured") using extend method try:     list1=[2,3,4,4,5,6,7]     list2=[99,8,0,4,2,1]     list1.extend(list2)     print(list1) except ValueError:     print("error occured") # Find common elements between two lists. try:     list1=[2,3,4,4,5,6,7]     list2=[99,8,0,4,2,1]     list3=set(list1).intersection(set(list2))     print(list(list3)) except ValueError:     print("error occured") Sort a list without using sort(). try:         list2=[99,8,0,4,2,1]         for i in range(len(list2)):         for j in range(len(list2)-1):             if list2[j]>list2[j+1]:                 list2[j],list2[j+1]=list2[j+1],list2[j] ---i am using bubble sort logic     print(list2) except ValueError:     print("error occured") #Python #CodingPractice #PythonProgramming #ProblemSolving #DataScienceJourney

Python raises no error and produces no warning when an instance attribute shadows a @classmethod. The method is still on the class — it's just hidden from that specific instance. This happens because @classmethod is a non-data descriptor. It defines __get__ but not __set__, which puts it in tier 3 of Python's three-tier attribute lookup. An instance attribute with the same name sits in tier 2 (the instance __dict__) and wins every time. The result: c.create() raises TypeError with a message that never mentions shadowing. The bug can sit undetected for a long time. A new article on PythonCodeCrack covers how the descriptor protocol makes this possible, how to detect an active shadow using vars() and an MRO walk, and six prevention strategies — from naming conventions and __slots__ to ProtectedClassMethod data descriptors and a ProtectedMeta metaclass for hierarchy-wide coverage. There's also an interactive step-through visualizer, a Spot the Bug challenge, and a decision flowchart that routes to the right prevention strategy based on your codebase constraints. https://lnkd.in/ghRPQF9U #Python #PythonProgramming #SoftwareEngineering #DescriptorProtocol #PythonTips

Every line of Python creates objects. But who's tracking them? And what happens when they're no longer needed? Most developers trust Python to "just handle it." The ones who understand how — write faster, leaner, and more reliable code. Here's how it actually works: 1. Pymalloc — Python's own allocator Python doesn't ask the OS for memory every time. It grabs large chunks upfront and carves them into Arenas → Pools → Blocks. Small object allocation stays blazing fast. 2. Reference Counting — First line of defense Every object silently tracks how many things point to it. Count hits zero? Object destroyed instantly. No waiting. No pausing. Most objects never survive past this stage. 3. Garbage Collector — Second line of defense Reference counting has one weakness — circular references. Two objects pointing to each other never hit zero. Python's GC catches these using a generational strategy: Gen 0 — New objects. Collected most often. Gen 1 — Survived once. Collected less often. Gen 2 — Long-lived. Collected rarely. Most garbage dies young. Python bets on it — and wins. 💡 Things worth knowing: __slots__ — Replaces per-object dictionaries with compact fixed structures. Cuts memory by 40-50% for classes with millions of instances. weakref — References that don't keep objects alive. Essential for caches and observer patterns. tracemalloc — Tracks allocations to the exact line. Your best friend for hunting memory leaks in production. Generators over lists — Constant memory vs. allocating everything upfront. Always prefer generators when iterating once. The mindset shift: Python manages memory so you don't have to think about it. But the best developers choose to understand it anyway. Because when you know how objects live and die — every line of code becomes more intentional. What's the nastiest memory bug you've tracked down in Python? #Python #SoftwareEngineering #PythonInternals #PythonTips #CleanCode #BackendDevelopment

Slow python ? Meet pybind11 ! It started with a simple curiosity while exploring autograd in Deep Learning frameworks. To better understand how gradients work behind the scenes, we implemented #micrograd by Andrej Karpathy a tiny educational autograd engine written in Python. After building it, a natural question emerged: If Python is slow, how are performance-heavy libraries still so fast? Digging deeper, we discovered that many high-performance Python libraries rely on C++ under the hood, with Python acting as a clean interface. This led us to explore pybind11, a lightweight way to bridge Python and C++ ! We then, - Implemented micrograd in C++ - Exposed it to Python using pybind11 - Generated .so modules using setup.py - Recreated a simplified Python plus C++ architecture This small experiment helped us understand how Python can leverage C++ for performance while maintaining developer productivity.. Also, pybind11 is one of the several alternatives like SWIG, Boost.Python, and Cython that can also be used for language bindings. This exploration was done in collaboration with Kavin Kumar, where we jointly worked on both the medium article and the C++ micrograd implementation. To check our implementation, Github: https://lnkd.in/gz6GBuNV For a deep dive, Explore our article: https://lnkd.in/g2y8KRta PS: Not AI Generated :) #Python #CPP #CPlusPlus #Pybind11 #MachineLearning #DeepLearning #Autograd #Micrograd #PythonPerformance

🐍 Did you know? In Python, there’s a layer that sits above your classes, one that controls how classes themselves are created and behave. It’s called a metaclass. What exactly is a metaclass? - Objects are instances of classes - Classes are instances of metaclasses By default, Python uses type as the metaclass for all classes. But you can create your own metaclasses to customize how classes are defined and constructed. Example 1: Auto-injecting methods The metaclass automatically adds a greet method to the class. class Meta(type): def __new__(cls, name, bases, dct): def greet(self): return f"Hello from {name}!" dct["greet"] = greet return super().__new__(cls, name, bases, dct) class Person(metaclass=Meta): def __init__(self, name): self.name = name p = Person("Olivia") print(p.greet()) # Hello from Person! Example 2: Enforcing class rules Metaclasses can enforce constraints when a class is created. class MainClass(type): def __new__(cls, name, bases, attrs): if "foo" in attrs and "bar" in attrs: raise TypeError(f"Class {name} cannot define both foo and bar") return super().__new__(cls, name, bases, attrs) class SubClass(metaclass=MainClass): foo = 42 # bar = 34 # This would raise an error Example 3: Dynamic class generation class AnimalType: def __init__(self, ftype, items): self.ftype = ftype self.items = items def sub_animal(ftype): class_name = ftype.capitalize() def __init__(self, items): super(self.__class__, self).__init__(ftype, items) globals()[class_name] = type(class_name, (AnimalType,), {"__init__": __init__}) # create classes dynamically [sub_animal(a) for a in ["mammal", "bird"]] Metaclasses are a powerful (and sometimes mysterious) part of Python. Most developers rarely need them, but they are used in frameworks like Django. #Python #Metaclasses #SoftwareEngineering #BackendDevelopment #CleanCode #PythonTips

Whether you want to automate repetitive tasks, analyze data, build websites, or dive into machine learning — Python is the perfect starting point. And the best way to start is by getting solid on the fundamentals. In this guide you will learn every basic Python operation through concise explanations, real code examples, and the exact output you can expect to see when you run them. No fluff, no setup headaches — just Python. #Python #DataEngineering https://lnkd.in/g6bvfhHX

How Python Reads Your Code". It explains exactly what happens behind the scenes when Python encounters a simple line of code like r = 1 + 1: Step 1: Chopping It Up First, Python takes your line of code and breaks it down into individual, bite-sized pieces. For the equation r = 1 + 1, it specifically identifies r as the variable name, = as the assignment operator, the first 1 as the first operand, + as the addition operator, and the final 1 as the second operand. Step 2: Structuring & Trimming Once the pieces are separated, Python builds a blueprint by creating a structure that shows exactly how all of these parts connect together. To make sure it works as efficiently as possible, Python then "trims the fat" by removing any unnecessary complexity from this blueprint. Step 3: Analyzing the Ingredients Next, Python looks closely at each piece to identify its specific type—for example, recognizing that the number 1 is an "Integer". After figuring out exactly what kind of data it's holding, Python selects the correct operation (or "tool") needed to handle it. Step 4: The Final Cook Finally, Python executes the operation to process the code and produce your final result. The Process at a Glance The entire workflow boils down to four simple stages: 01 Chopping: Breaking the code into pieces. 02 Structuring: Building a logical blueprint. 03 Analyzing: Identifying data types and the right tools. 04 Executing: Running the code and producing the result.

🔹 Understanding Python Memory One important concept is how Python stores data in memory. When we write: a = 10 Most people think variable a stores the value 10. But in reality, Python variables store references to objects. Here 10 is the object, and a simply points to that object in memory. Multiple variables can reference the same object: a = 10 b = a Both a and b point to the same object in memory. 🔹 Mutable vs Immutable Objects Understanding this difference is very important in backend development. Immutable objects (cannot change after creation) ✴️ int ✴️ float ✴️ bool ✴️ str ✴️ tuple Example: a = 10 a = 20 Python creates a new object instead of modifying the old one. Mutable objects (can change after creation) ✴️ list ✴️ dictionary ✴️ set ✴️ custom classes Example: a = [1, 2] b = a b.append(3) Now a becomes: [1, 2, 3] Because both variables point to the same mutable object. This is a very common source of bugs in backend systems when shared state is not handled properly. 🔹 Generators in Python Generators are extremely useful for handling large data efficiently. A generator produces values one at a time instead of loading everything into memory. Example: def numbers(): for i in range(5): yield i for n in numbers(): print(n) Here, values are generated only when needed. 💡 Why generators are important in backend systems Generators are widely used for: ✴️ Streaming large API responses ✴️ Processing logs ✴️ Reading millions of database rows ✴️ Background workers ✴️ Data pipelines ✴️ Async streaming They help save memory and improve performance, especially when working with large datasets. #Python #BackendEngineering #SoftwareDevelopment

This blog covers everything from creating your first dictionary to advanced patterns like nested dicts, defaultdict, Counter, and dictionary comprehensions — with output tables after every example so you can see exactly what each operation produces. #Python #DataEngineering https://lnkd.in/g23zDgfm