Kishan Kumar’s Post

Lessons from Real Backend Systems Short reflections from building and maintaining real backend systems — focusing on Java, distributed systems, and the tradeoffs we don’t talk about enough. ⸻ We had logs everywhere. Still couldn’t explain the outage. At first, it didn’t make sense. Every service was logging. Errors were captured. Dashboards were green just minutes before the failure. But when the system broke, the answers weren’t there. What we had: [Service A Logs] [Service B Logs] [Service C Logs] What we needed: End-to-end understanding of a single request The issue wasn’t lack of data. It was lack of context. Logs told us what happened inside each service. They didn’t tell us how a request moved across the system. That’s when we realized: Observability is not about collecting signals. It’s about connecting them. At scale, debugging requires three perspectives working together: Logs → What happened? Metrics → When and how often? Traces → Where did it happen across services? Without correlation, each signal is incomplete. The turning point was introducing trace context propagation. [Request ID / Trace ID] ↓ Flows across all services ↓ Reconstruct full execution path Now, instead of guessing: * We could trace a failing request across services * Identify latency bottlenecks precisely * Understand failure propagation Architectural insight: Observability should be designed alongside the system — not added after incidents. If you cannot explain how a request flows through your system, you cannot reliably debug it. Takeaway: Logs help you inspect components. Observability helps you understand systems. Which signal do you rely on most during incidents — logs, metrics, or traces? — Writing weekly about backend systems, architectural tradeoffs, and lessons learned through production systems. Keywords: #Observability #DistributedSystems #SystemDesign #BackendEngineering #SoftwareArchitecture #Microservices #Tracing #Monitoring #ScalableSystems

Our cron job was failing. Pods were restarting. The connection pool was full. Nobody knew why. That ticket landed on my desk. The Investigation: The cron job was processing entities in bulk. Looked simple on the surface. But when I went deeper into the code — I found three problems hiding inside each other. Problem 1 — Transactions inside transactions: Methods marked @Transactional were being called inside other @Transactional methods. In Spring Boot, by default, the inner transaction joins the outer one. So instead of small isolated transactions — everything was running inside one giant transaction that never released its database connection until the entire job finished. 300 entities. One transaction. Connection held the entire time. Problem 2 — 600 external API calls inside a single transaction: Each entity required 2 external API calls. 300 entities = 600 API calls. All of them inside the same open transaction. Every API call added latency. The transaction stayed open longer. The database connection stayed occupied longer. The connection pool slowly filled up. No connections left. Pod crashes. Job fails. Problem 3 — External API calls inside @Transactional: Calling external APIs inside a transaction is dangerous by design. If the API call hangs or times out — your transaction stays open. Your connection stays occupied. You have no control over how long that takes. The Fix: → One entity one transaction Moved processing to individual transactions per entity using @Transactional with REQUIRES_NEW propagation. After each entity — transaction commits, connection releases, memory frees. → Moved external API calls outside the transaction using Spring Events Published an event after each transaction committed. The event listener handled the external API calls asynchronously — completely outside the transaction boundary. Transaction commits fast. Connection releases immediately. API calls happen separately without blocking anything. Result — cron job runs cleanly. No pool exhaustion. No pod restarts. One ticket. Three hidden problems. One production system stabilized. @Transactional looks simple. The moment you put long running operations, external calls, or nested transactions inside it — it becomes one of the most dangerous annotations in Spring Boot. Have you ever debugged a transaction management issue in production? What gave it away? #SpringBoot #Java #TransactionManagement #BackendDevelopment #SoftwareEngineering #FAANG #SystemDesign #SpringFramework #ProductionEngineering #JavaDeveloper

𝗝𝗮𝘃𝗮 𝟮𝟭 𝗷𝘂𝘀𝘁 𝗯𝗲𝗰𝗮𝗺𝗲 𝗟𝗧𝗦. 𝗝𝗮𝘃𝗮 𝟮𝟱 𝗶𝘀 𝗮𝗹𝗿𝗲𝗮𝗱𝘆 𝗮𝗿𝗼𝘂𝗻𝗱 𝘁𝗵𝗲 𝗰𝗼𝗿𝗻𝗲𝗿.  𝗠𝗼𝘀𝘁 𝘁𝗲𝗮𝗺𝘀 𝗮𝗿𝗲 𝘀𝘁𝗶𝗹𝗹 𝗼𝗻 𝗝𝗮𝘃𝗮 𝟭𝟭. That gap is a ticking clock - not a badge of stability. I've seen systems processing $50M/month still running on JDK 11 because "it works." 𝑰𝒕 𝒅𝒐𝒆𝒔 𝒘𝒐𝒓𝒌. 𝑼𝒏𝒕𝒊𝒍 𝒊𝒕 𝒅𝒐𝒆𝒔𝒏'𝒕. Here's what the modernization wave actually means for backend engineers: 𝗩𝗶𝗿𝘁𝘂𝗮𝗹 𝗧𝗵𝗿𝗲𝗮𝗱𝘀 (𝗣𝗿𝗼𝗷𝗲𝗰𝘁 𝗟𝗼𝗼𝗺) 𝗮𝗿𝗲𝗻'𝘁 𝗷𝘂𝘀𝘁 𝗮 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 - 𝘁𝗵𝗲𝘆'𝗿𝗲 𝗮𝗻 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝘀𝗵𝗶𝗳𝘁. We replaced reactive WebFlux boilerplate in one service with virtual threads. Same throughput. Simpler code. Easier onboarding. 𝗣𝗮𝘁𝘁𝗲𝗿𝗻 𝗠𝗮𝘁𝗰𝗵𝗶𝗻𝗴 + 𝗥𝗲𝗰𝗼𝗿𝗱𝘀 𝗮𝗿𝗲𝗻'𝘁 𝗰𝗼𝘀𝗺𝗲𝘁𝗶𝗰 𝘀𝘂𝗴𝗮𝗿. In DDD, they let your domain model express intent clearly - no more 6-line null checks around a simple type branch. 𝗚𝗿𝗮𝗮𝗹𝗩𝗠 𝗻𝗮𝘁𝗶𝘃𝗲 𝗶𝗺𝗮𝗴𝗲𝘀 𝗮𝗿𝗲 𝗿𝗲𝗮𝗹 𝗻𝗼𝘄. We cut startup time by 60% on a Spring Boot microservice. Cold-start penalties in Kubernetes? Gone. 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: 🔄 Upgrade path matters more than upgrade speed - test your Kafka serializers and Hibernate dialect first ⚡ Virtual threads will obsolete most reactive code in I/O-heavy services 🏗️ Java 25 isn't disruption - it's the platform finally catching up to what we've been building workarounds for The teams that modernize incrementally win. The ones who wait for "the perfect migration window" fall two LTS cycles behind. Where is your team on the Java upgrade journey? Still on 11? Already on 21? What's the biggest blocker - legacy dependencies, risk appetite, or just bandwidth? #Java #SpringBoot #Microservices #BackendDevelopment #SoftwareEngineering #Kafka #SystemDesign #SoftwareArchitecture #Backend #Ascertia

API / BACKEND ENGINEERING (REAL-WORLD) Top 20 API & Backend Interview Questions → What is REST API? → Difference between REST and SOAP? → What is API authentication? → What is JWT (JSON Web Token)? → What are HTTP methods (GET, POST, PUT, DELETE)? → What is rate limiting? → What is API versioning? → What is idempotency? → What is middleware in backend? → How do you design scalable APIs? → What is database indexing? → What is caching in backend? → What is microservices architecture? → What is load balancing? → How do you handle concurrency? → What is event-driven architecture? → How do you secure APIs? → What is GraphQL? → How do you handle errors in APIs? → Best practices for backend development? Follow : Ashwina Mathur

Top 12 Microservices Design Patterns Every Developer Should Know Microservices architecture is powerful—but only when designed correctly. Understanding key patterns helps build scalable, resilient, and maintainable systems. Here are 12 essential patterns: API Gateway – Single entry point for all client requests Saga Pattern – Manages distributed transactions Event Sourcing – Stores changes as events CQRS – Separates read and write operations Strangler Fig – Gradual migration from monolith to microservices Service Discovery – Dynamic service registration & lookup Circuit Breaker – Prevents cascading failures Bulkhead – Isolates failures between services Database per Service – Independent data management Sidecar – Adds supporting features like logging/monitoring Retry Pattern – Handles temporary failures API Composition – Combines multiple service responses Mastering these patterns is key to building robust distributed systems. As a Java Full Stack Developer, I’m continuously exploring system design concepts to improve scalability and performance. 💡 Comment "Microservices" and I’ll share a complete solutions PDF with detailed explanations and coding examples. Follow Shail Tiwari for Java Full Stack interview questions, coding solutions, and career growth tips. #Microservices #SystemDesign #Java #BackendDevelopment #SpringBoot #SoftwareArchitecture #microservices

Hot take for backend engineers: Most teams do not have a scaling problem. They have a design problem. When a system slows down, the first reaction is usually: add retries add more pods add caching add a queue split another service That feels like engineering. But a lot of the time, the real issue is simpler: chatty service-to-service calls bad timeout values no backpressure weak DB access patterns too many synchronous dependencies in one request path I’ve seen systems with moderate traffic behave like they were under massive load. Not because traffic was insane. Because the architecture was burning resources on every request. That’s why “we need to scale” is often the wrong diagnosis. Sometimes the system does not need more infrastructure. It needs fewer moving parts. Debate: What causes more production pain in real systems? A) high traffic B) bad architecture C) poor database design D) weak observability My vote: B first, C second. What’s yours? #Java #SpringBoot #Microservices #DistributedSystems #BackendEngineering

𝗦𝗽𝗿𝗶𝗻𝗴 𝗕𝗼𝗼𝘁 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 – 𝗛𝗶𝗴𝗵 𝗹𝗲𝘃𝗲𝗹 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗕𝗿𝗲𝗮𝗸𝗱𝗼𝘄𝗻 When building scalable backend systems, having a clear architectural understanding of Spring Boot is a game changer. Here’s a simple yet powerful way to think about it 👇 𝗖𝗼𝗿𝗲 𝗟𝗮𝘆𝗲𝗿 (𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻) This is where everything starts. • Auto-Configuration – Reduces boilerplate, smart defaults • Dependency Injection – Loose coupling, easier testing • Application Context – Heart of Spring, manages beans lifecycle 👉 This layer makes Spring Boot “plug & play” 𝗪𝗲𝗯 𝗟𝗮𝘆𝗲𝗿 (𝗘𝗻𝘁𝗿𝘆 𝗣𝗼𝗶𝗻𝘁) Handles all incoming traffic. • REST Controllers – Expose APIs • Request Mapping – Route requests effectively • Validation – Ensure clean & safe inputs 👉 This is where your APIs meet the world 𝗗𝗮𝘁𝗮 𝗟𝗮𝘆𝗲𝗿 (𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝗲) Responsible for data handling. • Spring Data JPA – Abstracts DB interactions • Repositories – Clean data access layer • Transactions – Ensure consistency & reliability 👉 Focus: Integrity + performance 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗟𝗮𝘆𝗲𝗿 (𝗣𝗿𝗼𝘁𝗲𝗰𝘁𝗶𝗼𝗻) Because production ≠ demo apps. • JWT Authentication – Stateless & scalable • Role-Based Access Control (RBAC) – Fine-grained permissions 👉 Secure by design, not as an afterthought 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 (𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀) What you don’t measure, you can’t improve. • Actuator – Health & metrics endpoints • Prometheus – Metrics collection • Grafana – Visualization & alerts 👉 This is where real engineering begins 𝙁𝙞𝙣𝙖𝙡 𝙏𝙝𝙤𝙪𝙜𝙝𝙩: A good Spring Boot application isn’t just about writing controllers — it’s about designing layers that are scalable, secure, and observable. #SpringBoot #Java #BackendDevelopment #Microservices #SystemDesign #SoftwareArchitecture #DevOps #Observability #JWT #SpringFramework #CodeQuality #TechLeadership #codefarm

𝟵𝟬% 𝗼𝗳 𝗝𝗮𝘃𝗮 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿𝘀 𝗦𝘁𝗿𝘂𝗴𝗴𝗹𝗲 𝘄𝗶𝘁𝗵 𝗧𝗵𝗶𝘀 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻. → You understand Spring Boot architecture. → You’ve implemented scalable REST APIs. → Your projects demonstrate hands-on experience. 𝗕𝘂𝘁 𝘁𝗵𝗲𝗻 𝘁𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄𝗲𝗿 𝗮𝘀𝗸𝘀: How would you design a system that ensures high availability, consistency, and scalability under heavy traffic? → Most Java developers struggle because they focus on implementation, not architecture. → The real gap lies in system-level thinking, not coding ability. 𝗛𝗲𝗿𝗲’𝘀 𝘄𝗵𝗮𝘁 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝘁𝗲𝘀 𝗺𝗶𝗱-𝗹𝗲𝘃𝗲𝗹 𝗱𝗲𝘃𝘀 𝗳𝗿𝗼𝗺 𝘀𝗲𝗻𝗶𝗼𝗿 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀: → Instead of: “I know multithreading.” → They think: “How do I manage concurrency and avoid race conditions at scale?” → Instead of: “I build APIs,” → “How do I design fault-tolerant and idempotent services?” → Instead of: “I use ORM tools,” → “How do I optimize queries and manage database performance under load?” Senior Java engineers don’t just write code; they design resilient, distributed architectures. → Scalable concurrency models → Data consistency and distributed transactions → Fault tolerance with retries and circuit breakers → JVM tuning and performance optimization → Building highly available and resilient systems 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝘁𝗵𝗶𝘀 𝗶𝗻 𝗺𝗶𝗻𝗱, 𝗜 𝘄𝗲𝗻𝘁 𝗱𝗲𝗲𝗽 𝗮𝗻𝗱 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝗲𝗱 𝗲𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 𝗶𝗻𝘁𝗼 𝗮 𝗝𝗮𝘃𝗮 𝗕𝗮𝗰𝗸𝗲𝗻𝗱 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 𝗚𝘂𝗶𝗱𝗲. 𝗚𝗲𝘁 𝘁𝗵𝗲 𝗚𝘂𝗶𝗱𝗲 𝗵𝗲𝗿𝗲: https://lnkd.in/dRT_F8WS Use 𝗝𝗔𝗩𝗔𝟭𝟬 to get 𝟭𝟬% off. Stay Hungry, Stay Foolish!!

Stop making your users wait! 🛑 Master the 4 ways to handle "Off-Main-Thread" work. The hallmark of a Senior Dev isn't just writing fast code—it's knowing when not to run code on the main thread. 🚀 Background Process: For the "I'll do it now, but you don't have to watch" tasks. (Image resizing, cache warming). ⏰ Cron Job: The "Set it and forget it" worker. It lives by the clock. (Weekly backups, Monday morning reports). 🚜 Batch Job: The "Heavyweight Champion." It handles millions of records while the world is asleep. (Daily bank settlements). 📣 Pub/Sub Queue: The "Orchestrator." It allows different services to talk to each other without being "glued" together. (One event, five different actions). The Golden Rule: If it takes longer than 200ms and the user doesn't need the result to see the next screen, get it off the main thread! What’s your favorite tool for managing these? Hangfire? Quartz? RabbitMQ? AWS Lambda? Let’s talk architecture! 👇 #SystemDesign #BackendDevelopment #SoftwareArchitecture #DotNet #Java #CloudComputing #Microservices #ProgrammingTips

𝐋𝐚𝐭𝐞𝐧𝐜𝐲 𝐢𝐬 𝐮𝐩, 𝐛𝐮𝐭 𝐧𝐨𝐭𝐡𝐢𝐧𝐠 𝐢𝐬 𝐟𝐚𝐢𝐥𝐢𝐧𝐠. 𝐖𝐡𝐞𝐫𝐞 𝐝𝐨 𝐲𝐨𝐮 𝐥𝐨𝐨𝐤 𝐟𝐢𝐫𝐬𝐭? 🔍 One of the worst production situations: Latency is growing 📈 Users feel it 😐 Logs are clean 🧼 Nothing is obviously broken ❌ Most teams waste time here. They search for errors 🔎 Restart pods 🔄 Jump between dashboards 📊 But when nothing is failing, the problem is rarely an exception. It is usually one of these: 1. 𝗦𝗰𝗼𝗽𝗲 𝗳𝗶𝗿𝘀𝘁 🎯 One endpoint or all? One instance or all? Reads, writes, or async? If you skip this, you debug the whole system instead of a slice 2. 𝗧𝗵𝗿𝗲𝗮𝗱 𝗽𝗼𝗼𝗹𝘀 🧵 Active threads, queue size, blocked threads. If all workers are busy, requests are not failing - they are waiting to run. 3. 𝗧𝗵𝗿𝗲𝗮𝗱 𝗱𝘂𝗺𝗽 📸 Look for: * repeated stack traces * WAITING / BLOCKED threads * DB connection waits * socket reads * lock contention This shows where execution is actually stuck. 4. 𝗚𝗖 𝗯𝗲𝗵𝗮𝘃𝗶𝗼𝗿 ♻️ Pause time, frequency, heap pressure. If latency spikes in waves, GC is often involved. 5. 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻 𝗽𝗼𝗼𝗹𝘀 🧩 DB, HTTP clients, Redis, broker. Exhausted pool = requests wait instead of fail. Classic “slow but no errors”. 6. 𝗤𝘂𝗲𝘂𝗲𝘀 & 𝗹𝗮𝗴 📊 Queue depth, consumer lag, retries. The system may look fine while work silently accumulates. 7. 𝗗𝗼𝘄𝗻𝘀𝘁𝗿𝗲𝗮𝗺𝘀 🌐 DB, internal services, external APIs. Your service might be slow because it is efficiently waiting on something else. The key shift: No errors does not mean no problem. ❗ It usually means the bottleneck is in waiting, saturation, contention, or backlog. Stop hunting for exceptions first. Start finding where time is spent. How do you usually localize the bottleneck first in this situation? 🤔 #backend #java #springboot #observability #performance #distributedsystems #productionengineering