Onkar Lapate’s Post

Django's only() and defer() methods are often overlooked, yet they are essential for optimizing memory usage when fetching data from the database. Every field retrieved from the database consumes memory, and this can become significant with large models. Consider the following examples: - Fetching all fields when only two are needed: ```python users = User.objects.all() ``` - Instead, fetch only the necessary fields: ```python users = User.objects.only('id', 'email') ``` - Alternatively, defer the heavy fields that are not immediately required: ```python users = User.objects.defer('bio', 'profile_picture') ``` For instance, with a User model that includes a TextField for bio and an ImageField for profile picture, fetching 10,000 users for an email report can lead to significant memory savings. Using only('id', 'email') reduced memory usage by 60% for that query alone. When to use which method: - Use only() when you know exactly which fields you need. - Use defer() when you want to retrieve everything except a few heavy fields. This small change can lead to a big impact at scale. 🚀 #Django #Python #DjangoORM #BackendPerformance #PythonDev #BackendDevelopment #HappyLearning

Show HN: WhiskeySour – A 10x faster drop-in replacement for BeautifulSoup https://ift.tt/kndh9Bg Show HN: WhiskeySour – A 10x faster drop-in replacement for BeautifulSoup The Problem I’ve been using BeautifulSoup for sometime. It’s the standard for ease-of-use in Python scraping, but it almost always becomes the performance bottleneck when processing large-scale datasets. Parsing complex or massive HTML trees in Python typically suffers from high memory allocation costs and the overhead of the Python object model during tree traversal. In my production scraping workloads, the parser was consuming more CPU cycles than the network I/O. Lxml is fast but again uses up a lot of memory when processing large documents and has can cause trouble with malformed HTML. The Solution I wanted to keep the API compatibility that makes BS4 great, but eliminates the overhead that slows down high-volume pipelines. It also uses html5ever which That’s why I built WhiskeySour. And yes… I *vibe coded the whole thing*. WhiskeySour is a drop-in replacement. You should be able to swap from "bs4 import BeautifulSoup" with "from whiskeysour import WhiskeySour" and see immediate speedups. Your workflows that used to take more than 30 mins might take less than 5 mins now. I have shared the detailed architecture of the library here: https://lnkd.in/eMjEWwmA Here is the benchmark report against bs4 with html.parser: https://lnkd.in/eeYk3pF3 Here is the link to the repo: https://ift.tt/HrUNktf Why I’m sharing this I’m looking for feedback from the community on two fronts: 1. Edge cases: If you have particularly messy or malformed HTML that BS4 handles well, I’d love to know if WhiskeySour encounters any regressions. 2. Benchmarks: If you are running high-volume parsers, I’d appreciate it if you could run a test on your own datasets and share the results. April 25, 2026 at 10:23AM

I've just published a new guide: "BeautifulSoup Web Scraper: A Beginner’s Guide to Scraping Web Data to CSV". Whether you're a student or a seasoned developer looking to automate data tasks, this guide shows you how to fetch, parse, and save web data efficiently using modern Python tools like uv.

🚀 Introducing numpy2: Production-Ready NumPy Just launched numpy2 on PyPI – the library that solves the #1 problem NumPy developers face when building APIs. The Problem: json.dumps(np.array([1, 2, 3])) # ❌ TypeError: Object of type int64 is not JSON serializable This happens constantly in production. NumPy types break JSON serialization, forcing developers to write messy custom encoders. The Solution: np2.to_json(arr) # ✅ '[1, 2, 3]' – Done! ⚡ Key Features: ✓ Zero-config JSON serialization for NumPy & pandas ✓ FastAPI, Flask, Django integration out-of-the-box ✓ Type-safe conversions (3.75x faster than manual code) ✓ Handles edge cases: NaN, Infinity, complex numbers ✓ MIT licensed, production-ready Why numpy2 stands out: • 1 import vs. 20 lines of boilerplate • Solves real-world pain points, not hypothetical ones • Used in high-traffic APIs handling millions of requests • Intuitive API – NumPy developers instantly understand it 📦 Install: pip install numpy2 📖 Docs: https://lnkd.in/djnmp7Jh Whether you're building data science APIs, ML pipelines, or real-time analytics platforms, numpy2 eliminates friction and lets you focus on your core logic. Drop a ⭐ on GitHub if this solves a problem you've faced! #NumPy #Python #WebDevelopment #FastAPI #DataScience #OpenSource #API #JSON #Serialization #Python3 #SoftwareDevelopment #GitHub #PyPI #DataEngineering #BackendDevelopment

🧠 Python Concept: dataclasses (Clean Data Models) Write less boilerplate code 😎 ❌ Traditional Class class User: def __init__(self, name, age): self.name = name self.age = age def __repr__(self): return f"User(name={self.name}, age={self.age})" 👉 More boilerplate 👉 Repetitive code ✅ Pythonic Way (dataclass) from dataclasses import dataclass @dataclass class User: name: str age: int 👉 Automatically generates: __init__ __repr__ __eq__ 🧒 Simple Explanation Think of it like a shortcut ➡️ You define data ➡️ Python builds the rest 💡 Why This Matters ✔ Cleaner code ✔ Less boilerplate ✔ Easier to maintain ✔ Used in real-world apps ⚡ Bonus Example @dataclass class User: name: str age: int = 18 👉 Default values supported 😎 🧠 Real-World Use ✨ API models ✨ Config objects ✨ Data handling 🐍 Write less code 🐍 Let Python do the work #Python #AdvancedPython #CleanCode #SoftwareEngineering #BackendDevelopment #Programming #DeveloperLife

How do you handle GET requests in your DRF projects? I have noticed a common point of confusion among Django developers whether to structure response data directly in the view or use a serializer. Here is my take. Always use serializers, even for read only operations. By default, read only fields in a serializer give you a clean declarative way to shape your API responses. Instead of manually reshaping dictionaries inside the view which quickly becomes unmaintainable, serializers act as a contract between your database models and the outside world. They allow you to rename fields conditionally, expose computed properties, nest related objects, and keep your views lean and focused on orchestration rather than transformation. But there is one critical performance caveat. If your serializer pulls data from multiple related objects, make sure you use prefetch related or select related in your queryset before passing it to the serializer. Otherwise you will run into the classic N plus one query problem, one query for the main object plus one query for each related object. That scales terribly. Good serialization is about control over your data shape. Good performance is about intention in your query planning. Do you structure your GET responses in serializers or directly in the view? What is your team's standard? #Django #DRF #APIDesign #Python #WebDevelopment #BackendBestPractices

🚀 𝐃𝐚𝐲 𝟐: 𝐌𝐚𝐬𝐭𝐞𝐫𝐞𝐝 𝐑𝐨𝐮𝐭𝐢𝐧𝐠 & 𝐏𝐚𝐭𝐡 𝐏𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 𝐢𝐧 𝐅𝐚𝐬𝐭𝐀𝐏𝐈! ⚡ The journey into FastAPI continues! Today was all about how we handle data directly within the URL. Coming from a Django background, I’m loving how clean and intuitive the routing feels here. 𝙃𝙚𝙧𝙚’𝙨 𝙬𝙝𝙖𝙩 𝙄 𝙩𝙖𝙘𝙠𝙡𝙚𝙙 𝙩𝙤𝙙𝙖𝙮 : 📍 𝙋𝙖𝙩𝙝 𝙋𝙖𝙧𝙖𝙢𝙚𝙩𝙚𝙧𝙨 & 𝙃𝙏𝙏𝙋 𝙈𝙚𝙩𝙝𝙤𝙙𝙨 : I explored how to capture dynamic values from the URL using {curly_brackets} and how they interact with standard HTTP methods like GET and POST. 🔢 𝙋𝙖𝙩𝙝 𝙋𝙖𝙧𝙖𝙢𝙚𝙩𝙚𝙧𝙨 𝙬𝙞𝙩𝙝 𝙏𝙮𝙥𝙚𝙨 : This is a game-changer! By using Python type hints (like : int or : str), FastAPI automatically handles: 𝗗𝗮𝘁𝗮 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻:It returns a clear error if the wrong type is sent. 𝐃𝐚𝐭𝐚 𝐂𝐨𝐧𝐯𝐞𝐫𝐬𝐢𝐨𝐧: It automatically converts the URL string into the correct Python type. 🔄 𝘿𝙤𝙚𝙨 𝙊𝙧𝙙𝙚𝙧 𝙈𝙖𝙩𝙩𝙚𝙧? (𝙋𝙖𝙩𝙝 𝙋𝙖𝙧𝙖𝙢𝙚𝙩𝙚𝙧 𝙊𝙧𝙙𝙚𝙧𝙨) : I learned that in FastAPI, the order of your route functions matters. If you have a static path like /users/me and a dynamic path like /users/{user_id}, the static one must come first to avoid being "caught" by the dynamic parameter! 📋 𝙋𝙧𝙚𝙙𝙚𝙛𝙞𝙣𝙚𝙙 𝙑𝙖𝙡𝙪𝙚𝙨 : Using Python’s Enum, I learned how to restrict a path parameter to a specific set of valid options. This makes APIs incredibly robust and self-documenting. 🛠️ 𝙋𝙖𝙩𝙝 𝘾𝙤𝙣𝙫𝙚𝙧𝙩𝙚𝙧𝙨 : I dived into using :𝗽𝗮𝘁𝗵 to capture entire file paths (like files/images/photo.jpg) within a single parameter. 𝐂𝐮𝐫𝐫𝐞𝐧𝐭 𝐒𝐭𝐚𝐭𝐮𝐬:Feeling more confident with every line of code. The way FastAPI handles documentation and validation simultaneously is a massive productivity boost! 🛠️💻 #FastAPI #Python #BackendDevelopment #WebAPI #LearningJourney #Coding #SoftwareEngineering #PythonDeveloper #Day2

Running Prophet forecasting inside a Node.js event loop? That's how you kill your server. When building SensorCore — AI-native analytics via MCP — we hit a wall: our ML tools (Isolation Forest, Decision Trees, Prophet) need heavy computation that doesn't belong in a JavaScript runtime. The solution: a dedicated Python microservice. Our analytics engine is a standalone FastAPI app running on Uvicorn. It owns ALL the heavy ML work: - 13 REST endpoints - 11 Python analyzers - Direct ClickHouse access for data - Python 3.11 + pandas + scikit-learn + Prophet + ruptures Why Python, not Node.js for ML? - scikit-learn, Prophet, ruptures — mature, battle-tested Python libraries - NumPy/pandas for vectorized data processing - No equivalent ecosystem exists in JavaScript - Python's data science stack is 10+ years ahead Why a separate service, not embedded? - Node.js stays fast — zero ML overhead on the API server - Independent scaling — scale ML horizontally without touching the API - Crash isolation — a Prophet OOM doesn't take down your ingestion pipeline - Independent deployments — update ML models without restarting the API The stack in one line: FastAPI + Uvicorn + pandas + scikit-learn + Prophet + ruptures + ClickHouse Packaged in a python:3.11-slim Docker container. One docker-compose up and it's running. The Node.js server handles 1000+ req/s for log ingestion. The Python engine crunches millions of rows for ML. Each does what it's best at. Do you separate your ML workloads from your API server? Or run everything in one process?

Here’s a simple Python roadmap to follow: 🔹 Step 1: Basics Build your foundation → Syntax, variables, data types → Conditionals, functions, exceptions → Lists, tuples, dictionaries 🔹 Step 2: Object-Oriented Programming Think like a developer → Classes & objects → Inheritance → Methods 🔹 Step 3: Data Structures & Algorithms Level up problem-solving → Arrays, stacks, queues → Trees, recursion, sorting 🔹 Step 4: Choose Your Path This is where things get interesting → Web Development Django, Flask, FastAPI → Data Science / AI NumPy, Pandas, Scikit-learn, TensorFlow → Automation Web scraping, scripting, task automation 🔹 Step 5: Advanced Concepts → Generators, decorators, regex → Iterators, lambda functions 🔹 Step 6: Tools & Ecosystem → pip, conda, PyPI

Day 15/365: Merging Two Dictionaries with Summed Values in Python 🧮🔗 Today I worked on a very common real-world task: merging two dictionaries where overlapping keys should have their values added together. 🧠 What this code does: I start with two dictionaries: d1 = {1: 10, 2: 20, 3: 30} d2 = {3: 40, 5: 50, 6: 60} Each key can represent something like: a product ID with its total sales, a student ID with total marks, a user ID with total points. The goal is to combine d2 into d1: If a key from d2 already exists in d1, I add the values. If the key doesn’t exist in d1, I insert it. Step by step: I loop over each key i in d2: for i in d2: For each key: If i is already a key in d1: I update d1[i] by adding d2[i] to it. Otherwise: I create a new entry in d1 with that key and its value from d2. After the loop finishes, d1 contains the merged result. For the given dictionaries: Key 3 exists in both, so its values are added: 30 + 40 = 70. Keys 5 and 6 only exist in d2, so they are added as new keys. Final output: {1: 10, 2: 20, 3: 70, 5: 50, 6: 60} 💡 What I learned: How to merge two dictionaries manually using a loop and conditions. How to update values in a dictionary when keys overlap. How this pattern appears in real data tasks like: combining monthly reports, merging user activity stats, aggregating counts from multiple sources. Next, I’d like to explore: Handling much larger dictionaries efficiently. Using dictionary methods like update() or Counter from collections to compare approaches. Trying the same logic with string keys (like product names) instead of numbers. Day 15 done ✅ 350 more to go. Got any other dictionary + loop problems (like counting frequencies from multiple sources or merging configs)? Drop them in the comments—I’d love to try them next. #100DaysOfCode #365DaysOfCode #Python #Dictionaries #DataStructures #LogicBuilding #CodingJourney #LearnInPublic #AspiringDeveloper