Onkar Lapate’s Post

I reduced my API response time from 2.3s to 140ms. No Redis. No CDN. No caching layer. Just 4 changes to my Django REST Framework setup that most tutorials never mention. N+1 queries everywhere. My serializer accessed post.author.name on every row. 100 posts = 101 database queries. One select_related('author') brought it down to 1. Response time: 2.3s to 800ms instantly. Using ModelSerializer for read endpoints. ModelSerializer builds fields dynamically on every request. It's up to 377x slower than raw Python dicts. Switched read-only endpoints to serializers.Serializer with explicit fields. Another 40% gone. No pagination on list endpoints. Returning the entire table. 10,000 rows. Every request. Added CursorPagination, constant-time queries regardless of dataset size. OFFSET-based pagination breaks at high page numbers. Cursor doesn't. Fetching fields I never used. Serializer returned 15 fields. Frontend used 6. Added .only() and trimmed the serializer. 2.3s to 140ms. Same server. Same database. Same $12/month VPS. The bottleneck was never my infrastructure. It was my code. Run queryset.explain(analyze=True) on your slowest endpoint. You'll probably find the same mistakes. Which of these have you tried? #Django #Python #API #WebPerformance #BuildInPublic

Django doesn't store what's passed to cache.set(). It stores a transformed version of it. The reality is Django serializes everything before storage and that serialization has consequences most engineers never consider. Here's what actually happens: 1. Before any value reaches the backend, Django serializes it using pickle. 2. Not JSON. Not a string representation. Python's pickle, a binary serialization of the entire object graph. 3. This is why caching a model instance, a queryset result or a complex nested object just works. But pickle is also why a network-exposed cache is a critical security vulnerability. Here is how - -> Pickle deserialization executes arbitrary Python code. That's not a bug, it's how pickle reconstructs complex objects. -> An attacker who can write to the cache can craft a malicious pickle payload. -> When Django deserializes it and that code runs. On the server. With full application privileges. Precautions for avoiding this attack - - Redis and Memcached should never be publicly accessible, bind to localhost or a private network only. - Use Redis AUTH or TLS for any cache that travels over a network. They're not enabled by default in Django. - django-redis supports pluggable serializers. Replace pickle with msgpack or a custom JSON encoder. Safer, often faster. The cache feels invisible until it isn't. Treat it like any other network service that touches application data. Has cache security ever been part of a security review in your stack or is it assumed safe by default? #Python #Django #BackendDevelopment #SoftwareEngineering

Swapping Django's cache backend is one settings change. The behavioral differences are not that simple. The catch - BaseCache defines the contract - get, set, incr, delete, get_or_set. Every backend implements this contract. Not every backend can fulfill it with the same guarantees! 1. incr() - where the divergence is most dangerous - cache.incr('counter') on Redis is a single INCR command sent to Redis. - Redis processes it atomically. One operation. Safe under any concurrency. - cache.incr('counter') on DatabaseCache - Django reads the current value, increments it in Python, writes it back. - Three steps. No lock between them. NOT safe under any concurrency. 2. get_or_set() - the race condition most of us miss - get_or_set('key', default, timeout) looks atomic. On no backend is it truly atomic. - Two concurrent requests both find the key missing. Both compute the default. Both set it. - This is the cache stampede problem. get_or_set() does not prevent it. 3. clear() - scope is more than just keys in an app - cache.clear() on a shared Redis instance clears the entire Redis database. Not just keys belonging to this application. - Multiple applications sharing one Redis instance, one clear() call wipes everything. - KEY_PREFIX only prevents key collisions. It does not scope clear(). The cache API is an abstraction over storage. Abstractions leak at the worst possible moment. Have you ever discovered a backend-specific behavior difference under load rather than in testing? #Python #Django #BackendDevelopment #SoftwareEngineering

cache.get('user_123') never touches Redis directly. Something else runs first. The assumption is Django's cache API is a thin wrapper. Call get, retrieve from storage and done. The reality is that every cache call passes through a pipeline before a single byte touches the backend. Here's what happens: 1. Every cache backend in Django inherits from BaseCache. BaseCache owns the entire caching contract - get, set, delete, incr, get_or_set. 2. The concrete backend like RedisCache, MemcachedCache implements only the storage-specific parts. The abstraction layer runs first. Always! 3. The first thing BaseCache does is transform the key. 4. Every key passes through make_key() — which prepends KEY_PREFIX and VERSION from settings. The key 'user_123' becomes ':1:user_123' in storage by default. Cache miss on a key that exists → check the actual key in storage first! Django's cache versioning lets the entire cache be invalidated by bumping VERSION in settings. Old keys still exist in storage, until eviction clears them. The cache API feels simple because BaseCache is doing the hard work invisibly. Have you ever debugged a cache miss only to find the key was there, just under a different name? #Python #Django #BackendDevelopment #SoftwareEngineering

🚀 Built my first CRUD API using FastAPI + MySQL and deployed it on Render! 🌐 Live URL: https://lnkd.in/gHKJaCXx Today I created a REST API with full CRUD operations using Python (FastAPI) and MySQL as the database, and deployed it using Render. What I built: ✔ GET → Read data from MySQL   ✔ POST → Insert data into MySQL   ✔ PUT → Update existing records   ✔ DELETE → Remove records from MySQL  Deployment: 🌐 Hosted on Render This project helped me understand how backend systems work in real-world applications—from API design to database integration and deployment. Key learnings: - REST API design principles   - CRUD operations with MySQL   - FastAPI backend development   - Deploying applications on Render  It’s a simple project, but it reflects real-world backend architecture. Next step: add authentication and improve security. #Python #FastAPI #MySQL #CRUD #BackendDevelopment #Render #Deployment #RESTAPI

🚀 Built My Own URL Shortener using Flask & MySQL! Excited to share my latest mini project — a URL Shortener Web Application 🔗 💡 What it does: This app converts long URLs into short, shareable links and redirects users seamlessly. ⚙️ Tech Stack Used: - Python (Flask) - MySQL Database - HTML & CSS - Hashing (SHA-256) + Base64 Encoding ✨ Key Features: ✔️ Generate unique short URLs ✔️ Store and retrieve links from database ✔️ Redirect to original URL instantly ✔️ Track click counts for each link ✔️ Simple and clean UI 🔍 How it works: - User enters a long URL - System generates a short hash - Data is stored in MySQL - Short URL redirects to original link when accessed 📌 This project helped me understand: - Backend development with Flask - Database integration - URL routing & redirection - Basic system design concepts #Python #Flask #WebDevelopment #Projects #BackendDevelopment #MySQL #Coding #DeveloperJo

I built a URL shortener. But not the tutorial kind. Everyone builds URL shorteners as a learning project. Mine does something different. It tells you what's inside the link before you click. Here's the problem I wanted to solve: You receive a shortened link in a message. You have no idea where it goes. You click blind. So I built one that doesn't let that happen. Before saving any link, an AI pipeline runs three things: 🔴 Safety check: is this URL malicious or spammy? 📝 Page summarisation: what is this page actually about? 🏷️ Category + tags + smart expiry: how long should this link live? When someone visits the short link, instead of a blind redirect they see a preview page -> title, summary, safety score, tags -> and choose whether to continue. The full stack: ⚙️ Java + Spring Boot + Spring AI 🤖 Llama 3.2 via Ollama (local LLM — no API key needed) 🗄️ MySQL + Redis (sub-millisecond redirects) 🐳 Docker Compose — one command to run everything ✅ 12 unit tests — all critical paths covered Every architectural decision was deliberate: → Base62 over UUID — shorter, URL-safe, human-readable → Redis cache populated on create — first redirect is already a cache hit → Soft delete over hard delete — analytics history preserved→ ConcurrentHashMap rate limiter in V1, Redis-based in V2 — deliberately designed to scale The part I'm most proud of: when you try to shorten a suspicious URL, the response is a clean 400 -> "Suspicious domain, potential phishing site." That's the whole point. GitHub: https://lnkd.in/gSTVakbG Feedback and Critics are welcome 🙌 #Java #SpringBoot #SpringAI #OpenSource #BackendEngineering #SystemDesign #BuildInPublic

designed a PostgreSQL schema for an e-commerce backend. 4 tables. clean relationships. production-ready structure. users place orders → orders contain items → items reference products every foreign key is intentional. every index exists for a reason. next step — connecting this to my FastAPI microservices and adding real data. this is how you build backends that scale. #PostgreSQL #Backend #DataEngineering #FastAPI #Python

This post makes me think about PostgreSQL schema adaptability. Even with only a single tenant, users and products can be a separate schema from the orders allowing users to be represented by multiple tables in a schema and products represented by multiple tables. For example, products can split into tables by store if the company owns multiple store brands with exclusive products or by region if the company has products that are only legal to sell in a region or only can be produced in that region. Users can have tables for additional info that can be more adaptibly changed in its own users schema. For example, a table tracking users linked universities can be used by an online course website, like Coursera, where universities want to track students linked to their university, like Standford, to give credit for online course completion with updates sent to users telling them the credit value of online courses before taking the online course. https://lnkd.in/gTHZPvt6

I just published spark-perf-lint to PyPI, the first dedicated Apache Spark performance linter for the Python ecosystem with built-in pre-commit hooks, CI/PR annotations, and deep audit capabilities. PyPI: https://lnkd.in/gu_qd5yB Live Webpage: https://lnkd.in/gje5sMac GitHub: https://lnkd.in/g6WF8-Yn pip install spark-perf-lint One command. That's all it takes for any PySpark team in the world to start catching performance anti-patterns before they reach production. There are 500,000+ PySpark projects on GitHub. Thousands of organizations run Spark ETL pipelines processing billions of rows daily. Yet until today, there was no dedicated Spark performance linter available on PyPI. We have linters for Python style, type safety, security, even framework-specific rules for Django and FastAPI. But the framework that processes more data than almost anything else in the enterprise? Nothing. Today that changes. What I've contributed to the ecosystem: → 93 Spark-specific performance rules — not generic Python lint. Every rule understands Spark internals: how the Catalyst optimizer works, when shuffles happen, what causes data skew, which join strategy Spark will choose and why. → 11 dimensions of coverage — cluster configuration, shuffle optimization, join strategy, partitioning, data skew, caching lifecycle, I/O and file formats, AQE tuning, UDF patterns, Catalyst optimizer, and monitoring gaps. This is the most comprehensive Spark performance rule set available anywhere — open source or commercial. → Every finding comes with a fix — not "consider optimizing." Actual before/after code. Specific config changes. Spark internals explanation of why the anti-pattern hurts. Estimated performance impact. Effort level to fix. → A complete knowledge base — 50+ Spark patterns with decision matrices. When to use broadcast join vs sort-merge vs bucket join. How to size partitions. When to cache vs checkpoint. This alone is worth reading even if you never run the linter. → Three tiers of analysis: Pre-commit hook, CI/PR analysis and Deep audit Why this matters at scale: A single missed .collect() without a filter can OOM your driver with 200M rows. A join on a low-cardinality column can create straggler tasks that run 50x longer than the median. A default spark.sql.shuffle.partitions=200 on a 500GB dataset creates 200 partitions of 2.5GB each guaranteed spills and GC pressure. These bugs don't show up in dev. They don't show up in code review. They show up at 2 AM when your production pipeline fails and the on-call engineer is staring at a Spark UI full of red. spark-perf-lint catches them at commit time. Before they ever run on a cluster. Before they cost compute. Before they wake anyone up. Try it today ! #ApacheSpark #PySpark #DataEngineering #OpenSource #Python #PerformanceEngineering #ETL #BigData #DevTools #PreCommit #PyPI #ClaudeCode #Fintech #DataPipelines