NumPy Broadcasting Gotcha: Avoiding Shape Mismatches

Topic 4/100 🚀 🧠 Topic 4 — Generators Processing large data and running out of memory? 😵 This concept solves that problem. 👉 What is it? Generators are functions that return values one at a time using yield instead of returning everything at once. 👉 Use Case: Used in real-world applications for: Reading large files Data streaming Handling APIs with pagination 👉 Why it’s Helpful: Saves memory Improves performance Enables lazy evaluation (data generated only when needed) 💻 Example: def count_up_to(n): for i in range(n): yield i for num in count_up_to(5): print(num) 🧠 What’s happening here? Instead of storing all values in memory, the function generates them one by one when needed. ⚡ Pro Tip: If you're working with large datasets, always think “generator” before using lists. 💬 Follow this series for more Topics #Python #BackendDevelopment #100TopicOfCode #SoftwareEngineering #LearnInPublic

Why can’t we just search embeddings with a Python script instead of using a vector database? While working with RAG systems, this question came to mind. A simple approach would be to compare a query embedding with every vector using linear search. But that takes O(n) time — which becomes very slow when you have millions of vectors. Vector databases solve this using Approximate Nearest Neighbor (ANN) algorithms like HNSW and LSH. These structures allow fast similarity search by navigating the vector space efficiently instead of checking every vector. That’s why most modern RAG pipelines rely on vector databases. Interesting how much engineering goes into making retrieval fast. #MachineLearning #GenAI #RAG #VectorDB

DSA Tip: Graphs If you think data is always stored in a straight line… think again. Real-world systems are connected, not linear. Use Graphs. They represent data as nodes (points) and edges (connections). No strict order. No single path. Just relationships between data. Used in: - Social networks - Maps & navigation - Recommendation systems Insight: The real power of data isn’t in the elements, it’s in how they are connected. Quick Challenge: How would you represent your friend network as a graph? Drop your answer, I’ll review the best ones. FOLLOW FOR MORE DSA TIPS & INSIGHTS #DSA #Graphs #Python #CodingTips #LearnToCode

𝗜 𝗳𝗼𝘂𝗻𝗱 𝗮 𝟰𝟰× 𝘀𝗽𝗲𝗲𝗱 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗶𝗻 𝗮 𝘀𝗶𝗺𝗽𝗹𝗲 𝘀𝘂𝗺() 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻 🚀 I benchmarked three ways of summing 100,000 numbers: • Manual for loop → ~11.4 ms • Built-in 𝘀𝘂𝗺() → ~8.27 ms • 𝗻𝗽.𝘀𝘂𝗺() → ~0.259 ms 𝗡𝘂𝗺𝗣𝘆 𝘄𝗮𝘀 ~𝟰𝟰× 𝗳𝗮𝘀𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗮 𝗣𝘆𝘁𝗵𝗼𝗻 𝗹𝗼𝗼𝗽 ⚡ The real insight isn’t that “NumPy is faster.” It’s about execution layers. A Python loop runs inside the interpreter with dynamic checks every iteration. 𝘀𝘂𝗺() shifts the work into C. 𝗻𝗽.𝘀𝘂𝗺() operates on contiguous memory using optimized low-level code, avoiding Python-level iteration entirely. Same computation. Different execution layer. Massive performance gap. #Python #NumPy #DataScience #LearningInPublic

Honestly, a week ago I didn't fully understand why NumPy even existed. Now I just completed 10 hands-on NumPy problems from scratch. 💪 Here's what Level 1 covered: — Array creation (10+ different ways) — Array attributes (shape, ndim, size, dtype) — Data types and memory management — Random arrays and reproducibility with seed() — dtype conversions and overflow behavior One thing that surprised me: Converting float64 to float32 saves 50% memory. For large ML datasets — that's huge. I'm building this as an open source repo: 👉 numpy-100-challenges 100 problems from basics to ML-level NumPy. Level 2 starts tomorrow — Indexing & Slicing. If you're learning NumPy for ML, follow along! Repo link in comments 👇 #Python #NumPy #MachineLearning #LearningInPublic #DataScience #MLEngineer #100DaysOfCode #CSEStudent

Most people use scipy.optimize and move on. I wanted to understand what's actually happening underneath. So I built three optimization algorithms from scratch in Python, no external solvers, just NumPy: → Truncated Newton: finds local minima efficiently, approximating the Hessian without ever computing it fully → Sequential Penalty: handles constraints by turning them into a penalty, the more you violate them, the harder the algorithm pushes back → Filled Functions: the interesting one. When you're stuck in a local minimum, it builds an artificial "hill" on top of it to force the search elsewhere, that's how you find the global optimum Tested on 20+ benchmark problems. It works. In collaboration with Paolo Pascarelli and Silvia Alonzo Full code + writeup below 👇 https://lnkd.in/d8TwHtcS #python #optimization #numericalcomputing

𝗢𝗢𝗣𝗦 𝘄𝗮𝘀 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗼𝘀𝗲 𝘁𝗼𝗽𝗶𝗰𝘀 𝗜 𝗸𝗲𝗽𝘁 𝗿𝗲𝘃𝗶𝘀𝗶𝘁𝗶𝗻𝗴 — 𝗻𝗼𝘁 𝗯𝗲𝗰𝗮𝘂𝘀𝗲 𝗜 𝗳𝗼𝗿𝗴𝗼𝘁 𝗶𝘁, 𝗯𝘂𝘁 𝗯𝗲𝗰𝗮𝘂𝘀𝗲 𝗜 𝗻𝗲𝘃𝗲𝗿 𝗳𝘂𝗹𝗹𝘆 𝘁𝗿𝘂𝘀𝘁𝗲𝗱 𝗺𝘆 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗼𝗳 𝗶𝘁. Read about it from multiple sources. Understood the syntax, could follow the examples. But the mental model was always shaky. College lectures sorted that — not by covering something new, but by connecting things in the right order. Inheritance types, method resolution, how one class builds on another — it started making structural sense rather than just syntactic sense. Revisited it all. Coded through single, multi-level, hierarchical, and hybrid inheritance. And for the first time it felt like I actually owned the concept, not just recognized it. #Python #OOP #CS #DataScience

⚠️ Pandas trap: groupby() silently drops NaN keys by default, groupby() excludes rows where grouping columns contain NaN (dropna=True). This means: • Your training population may shrink • Group sizes may be biased • Downstream thresholds may fail Always define explicitly 💪 : Which rows you learn from. Whether NaN groups should be included (dropna=False). Your data quality assumptions before aggregation 🙅♀️ Silent defaults create silent bias. #Python #Pandas #DataScience #DataEngineering #DataQuality

Machine Learning Image Data using scikit image #machinelearning #datascience #dataimage #scikitimage scikit-image (formerly scikits.image) is an open-source image processing library for the Python programming language.[2] It includes algorithms for segmentation, geometric transformations, color space manipulation, analysis, filtering, morphology, feature detection, and more.[3] It is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy. The scikit-image project started as scikits.image, by Stéfan van der Walt. Its name stems from the notion that it is a "SciKit" (SciPy Toolkit), a separately-developed and distributed third-party extension to SciPy.[4] The original codebase was later extensively rewritten by other developers. Of the various scikits, scikit-image as well as scikit-learn were described as "well-maintained and popular" in November 2012.[5] Scikit-image has also been active in the Google Summer of Code. https://lnkd.in/gvmx22q2

Topic 5/100 🚀 🧠 Topic 5 — Iterators Ever wondered how a for loop actually works behind the scenes? 🤔 This is the concept powering it. 👉 What is it? Iterators are objects that allow you to traverse through data step-by-step using __iter__() and __next__() methods. 👉 Use Case: Used in real-world applications for: Custom data pipelines Streaming data Building your own iterable objects 👉 Why it’s Helpful: Gives full control over iteration Enables custom looping logic Foundation for generators 💻 Example: class Counter: def __init__(self, max): self.max = max self.current = 0 def __iter__(self): return self def __next__(self): if self.current < self.max: self.current += 1 return self.current raise StopIteration for num in Counter(3): print(num) 🧠 What’s happening here? We created a custom object that behaves like a loop by controlling how values are returned one by one. ⚡ Pro Tip: If you understand iterators, you’ll unlock how Python handles loops internally. 💬 Follow this series for more Topics #Python #BackendDevelopment #100TopicOfCode #SoftwareEngineering #LearnInPublic