Improving Observability with MDC and Trace Context in Spring Boot

A Backend Lesson From a 30-Second Timeout Last week, a prod export API started failing. The Situation: End-to-end export time: ~32 seconds -Gateway timeout: 30 seconds (hard limit) -Works locally? Yes. -Works in lower envs? Yes. -Production? Nope. We evaluated a few alternatives first: Email the export? File size exceeded the 25MB attachment limit even after applying compression. Increase gateway timeout? Not configurable. Instead of fighting the constraints, we changed the architecture. The Solution: We moved the heavy lifting out of the request lifecycle. 1. Async processing: The API triggers a background job and returns immediately. 2. Stream to S3: The worker streams data directly to S3 using multipart upload (streamed and low memory footprint). 3. Pre-signed URL: Once complete, the client receives a secure, direct download link. The Result: ✅ Zero gateway timeouts ✅ Reliable exports regardless of file size ✅ Clean separation between HTTP handling and heavy I/O My biggest takeaway: Many backend problems are architectural rather than purely code-level. Sometimes the right fix isn’t faster code, it’s a better delivery strategy. How do you handle long-running jobs in your APIs? #BackendEngineering #SystemDesign #Java #SoftwareArchitecture #DevCommunity

Idempotency: The Bug That Only Shows Up in Production This was one of those bugs that never appeared in development or testing, but caused real damage in production. We had an API that created records based on client requests. Everything worked fine under normal conditions. Then one day, we noticed duplicate records showing up in the database. Not consistently. Not for every request. Just enough to cause confusion. The root cause wasn’t concurrency. It wasn’t a bad query. It was retries. A client would send a request, hit a network timeout, and retry the same request. From the client’s point of view, the first request might have failed. From the server’s point of view, both requests were processed successfully. The backend logic assumed requests would be processed exactly once. public Order createOrder(CreateOrderRequest request) { return orderRepository.save(new Order(request)); } Under retry scenarios, this code created duplicate orders. The fix wasn’t adding locks or increasing timeouts. The fix was idempotency. We introduced a request identifier and checked for existing processing before creating new records: if (orderRepository.existsByRequestId(requestId)) { return existingOrder; } In Python services, the same pattern applied when handling background jobs or API calls: if processed_request(request_id): return existing_result Once idempotency was in place, retries stopped being dangerous. The system could safely handle network failures, client retries, and message replays. The lesson was simple but important: in distributed systems, retries are normal. Designing APIs that assume “exactly once” execution is a production risk. Idempotency is not an optimization. It’s a requirement for reliable backend systems. #JavaDeveloper #PythonDeveloper #BackendEngineering #DistributedSystems #ProductionLessons #SystemDesign

🔥 𝐖𝐡𝐲 𝐌𝐨𝐝𝐞𝐫𝐧 𝐌𝐢𝐜𝐫𝐨𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 𝐒𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐏𝐫𝐨𝐭𝐨𝐛𝐮𝐟 𝐈𝐧𝐬𝐭𝐞𝐚𝐝 𝐨𝐟 𝐉𝐒𝐎𝐍 ‎ Let’s use one simple object to make everything clear: JSON {"id": 1, "name": "john"} Protobuf 08 01 12 04 6A 6F 68 6E What the bytes mean 08 -> field #1 (id), type = varint 01 -> integer value = 1 12 -> field #2 (name), type = length delimited 04 -> string length = 4 6A 6F 68 6E -> "john" in ASCII bytes This is the same data but massively more compact. ✔ No quotes ✔ No field names ✔ No commas ✔ No spaces ✔ No JSON overhead ‎ 📦 𝐁𝐲𝐭𝐞 𝐒𝐢𝐳𝐞 𝐃𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 JSON size: {"id": 1, "name": "john"} Total = 27 bytes (field names + quotes + spaces) Protobuf size: 08 01= 2 bytes 12 04 6A 6F 68 6E = 6 bytes Total = 8 bytes ‎ 📊 𝐅𝐢𝐧𝐚𝐥 𝐁𝐲𝐭𝐞 𝐂𝐨𝐦𝐩𝐚𝐫𝐢𝐬𝐨𝐧 JSON ~27–30 bytes Why? • Sends field names ("id", "name") • Sends quotes, commas, spaces, braces • UTF‑8 encoding adds invisible overhead Protobuf 8 bytes Why? • Sends only tiny numeric tags (08, 12) • Sends only raw binary values (01, 6A 6F 68 6E) • No quotes, no names, no formatting ‎ ➡️ ~70% smaller for the exact same object ➡️ At microservice scale → millions of requests → huge bandwidth savings ‎ 🧠 𝐒𝐡𝐨𝐫𝐭 𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲 JSON speaks to humans. Protobuf speaks to machines. smaller, faster and purpose built for high traffic microservices. ‎ ❓ What do you use for microservices communication, JSON or Protobuf? ‎ ♻️ Repost to help others ➕ Follow Syed Shahzaib Zafar for more system design insights ‎ #SystemDesign #Microservices #BackendEngineering #SoftwareArchitecture #Protobuf #gRPC #JSON #Performance #Developers

🚨 Memory Leaks in Backend Systems – A Silent Killer One thing I’ve learned while working on backend systems: Memory leaks don’t shout. They whisper. The application keeps running… GC runs more frequently… Response times slowly increase… And then one day, production crashes. Most memory leaks aren’t caused by “complex logic.” They’re usually small oversights. Here are some common patterns I’ve seen (or debugged): 🔹 A static Map/List that keeps growing and never gets cleaned 🔹 DB connections or streams not closed properly 🔹 Event listeners registered but never removed 🔹 Caches without size limits or eviction policies 🔹 ThreadLocal values not cleared in thread pools 🔹 ClassLoader leaks during redeployments The tricky part? Everything looks fine — until it isn’t. What helps: • Monitoring heap usage trends (not just CPU) • Observing GC behavior • Taking and comparing heap dumps • Using tools like VisualVM or MAT • Designing with object lifecycle in mind Biggest takeaway for me: Performance tuning isn’t just about faster queries or better algorithms. It’s about being mindful of how long objects live. If you’re building scalable backend systems, memory awareness is not optional — it’s essential. Have you ever debugged a memory leak that took hours (or days) to figure out? 👀 #Java #Backend #Performance #SystemDesign #SpringBoot #MemoryManagement

Stop using 50MB frameworks to manage one single instance of a Logger. Most developers think they know how to write a Singleton. They read a blog post a decade ago and haven't checked if that information is still valid for the modern JVM. The result? Production code filled with thread-safety issues, memory leaks, and "Senior" implementations that are actually maintenance liabilities. We’ve moved from "Singletons everywhere" to "DI frameworks for everything," but we’ve forgotten how to actually use the JVM. In my latest post, I break down: - The Reality of Failure: Why "Double-Checked Locking" is often just a sophisticated way to introduce bugs. - The "Resume-Driven Development" Trap: Why we over-engineer simple patterns just to look smart. - The 2026 Standard: Why the Enum Singleton remains the only truly bulletproof implementation. - Testing Myths: Why "Singletons are hard to test" is usually a symptom of poor interface design, not the pattern itself. Complexity is debt. Every line of extra code is something that can break at 3 AM. If you’re still writing ten lines of synchronized boilerplate for a task that requires one, you're building a liability, not a feature. #Java #Backend #SoftwareEngineering #JVM #CleanCode #Programming

Stop using 50MB frameworks to manage one single instance of a Logger. Most developers think they know how to write a Singleton. They read a blog post a decade ago and haven't checked if that information is still valid for the modern JVM. The result? Production code filled with thread-safety issues, memory leaks, and "Senior" implementations that are actually maintenance liabilities. We’ve moved from "Singletons everywhere" to "DI frameworks for everything," but we’ve forgotten how to actually use the JVM. In my latest post, I break down: - The Reality of Failure: Why "Double-Checked Locking" is often just a sophisticated way to introduce bugs. - The "Resume-Driven Development" Trap: Why we over-engineer simple patterns just to look smart. - The 2026 Standard: Why the Enum Singleton remains the only truly bulletproof implementation. - Testing Myths: Why "Singletons are hard to test" is usually a symptom of poor interface design, not the pattern itself. Complexity is debt. Every line of extra code is something that can break at 3 AM. If you’re still writing ten lines of synchronized boilerplate for a task that requires one, you're building a liability, not a feature. #Java #Backend #SoftwareEngineering #JVM #CleanCode #Programming

How I Think About Backend System Design Earlier, I focused mostly on how to build things. Now, I spend more time on why and what could go wrong. Before writing any code, I usually think about: -> What problem are we actually solving? -> What happens when traffic grows 10x? -> What will break first — performance, database, or auth? -> How easy will this be to debug in production? I’ve learned that good system design isn’t about fancy diagrams or big words. It’s about making simple, clear decisions and knowing the trade-offs. Things I value much more now: -> Simple designs over over-engineering -> Clear APIs over clever abstractions -> Observability and logs over assumptions -> Fixing root causes, not quick patches One mindset shift that helped me a lot: 👉 "Design systems for real usage and failures, not just for happy paths." I’ll keep sharing practical learnings from real backend work — not interview theory. Curious to hear from others: What’s one system design lesson you learned the hard way? #Java #SpringBoot #BackendEngineering #Microservices #APIs #PerformanceOptimization #SoftwareEngineering #SystemDesign #LearningInPublic #EngineeringGrowth

C# Deep Dive: struct vs class — When to use which? Many developers know that struct is a value type and class is a reference type. But the real question is: When should you actually use struct instead of class? 🔹 class (Reference Type) • Allocated on the heap • Passed by reference • Supports inheritance • Can be null • Better for complex domain models Example: public class User { public string Name { get; set; } } Use class when: ✔ The object is large ✔ You need inheritance ✔ The object represents a business entity 🔹 struct (Value Type) • Allocated on the stack (usually) • Passed by value (copied) • Cannot inherit from another class • Cannot be null (unless nullable struct) • Better for small, immutable data Example: public struct Point { public int X; public int Y; } Use struct when: ✔ The object is small (under ~16 bytes recommended) ✔ It represents a single value ✔ It is immutable ✔ High performance matters ⚠️ Important: Large structs can hurt performance because they are copied by value. Rule of thumb: If it behaves like a value → struct If it behaves like an entity → class Understanding this difference is critical in performance-sensitive .NET systems. Follow for more advanced C# and backend engineering content. #csharp #dotnet #backend #softwareengineering #systemdesign #performance

A backend skill that’s getting more attention lately: Observability As systems grow, logs alone stop being enough. When multiple services are involved, knowing what failed is less useful than knowing where and why. That’s where observability comes in. Instead of looking at logs, metrics, and traces separately, observability connects them. A slow API call can be traced across services, correlated with metrics, and backed by logs — all in one flow. This changes debugging completely. You don’t guess. You follow the request. Tools like OpenTelemetry also standardize how data is collected. That means less vendor lock-in and more consistent visibility across environments. Most production issues aren’t code bugs. They’re system behavior problems — and observability helps teams understand that behavior clearly. Good backend systems aren’t just built to run. They’re built to be understood in production. #Observability #BackendEngineering #OpenTelemetry #Microservices #DistributedSystems #Java