Stop Mocking Your Database in Tests

The service was up. The API was up. The database was up. Users were still complaining. That’s why I think a lot of engineers misunderstand reliability. They think reliability means keeping components alive. It doesn’t. It means keeping the full user path stable when systems start interacting badly. A service can be healthy. A dashboard can be green. And the product can still feel broken. Because real pain usually comes from: - slow dependencies - retry amplification - fragile request paths - components that are “up” but harmful together Debate: What do engineers misunderstand more often? A) scalability B) reliability My vote: B. What’s yours? #Java #SpringBoot #DistributedSystems #Microservices #BackendEngineering

One of the most fragile parts of any backend system is depending on external APIs. We learned this the hard way. We were integrating 3 third-party services into our platform. Payments. Notifications. Data providers. All called synchronously, one after another. The result? If any of those providers lagged even slightly, our entire API froze. Users waited. Requests piled up. The server choked. So we rethought the architecture completely. Here is what changed: Instead of calling third-party APIs directly from the request cycle, we offloaded those calls to background jobs using BullMQ The main server now just queues the job and immediately returns a response to the client A background worker handles the actual API call separately If the external service fails or times out, the job does not disappear. It gets pushed into a retry queue with exponential backoff and tries again automatically The result? A 70% drop in failure rates. The biggest mindset shift for me was this: Stop assuming your code will not fail. Start assuming the network will always fail at some point, and design your system to handle it gracefully. Synchronous = tightly coupled = one failure breaks everything Async + queues = decoupled = failures become recoverable events This is not premature optimization. This is just building systems that survive the real world. #Backend #SystemDesign #SoftwareEngineering #WebDevelopment #Architecture #NodeJS #BullMQ #API #DistributedSystems #Engineering #Tech #Programming #SoftwareDevelopment #BackendDevelopment #DevOps #Resilience #CloudComputing #Microservices #CodingLife #BuildInPublic

The .env Illusion & The Working Directory Trap The "Air" Tooling Trap: Why my Go backend suddenly forgot my database password Today, I took a massive step in securing my Go fintech backend: moving my hardcoded database URLs and secrets into a .env file using godotenv. I set it up, ran my server, and… crash. failed to connect to user=Fredrick Abodunrin database=: FATAL: password authentication failed Wait, who is Fredrick Abodunrin? Why is my Postgres database trying to log in with my Windows username instead of my DB user? Because my .env file was completely ignored. The Challenge: To make my development faster, I installed Air (Go's version of Nodemon for hot-reloading). In my main.go file inside cmd/server, I explicitly told Go to look two folders back for my env file: godotenv.Load("../../.env"). When I ran go run ./cmd/server/main.go, it worked perfectly. But when I ran air, it crashed. The Paradigm Shift: Working Directory ≠ File Location 🧠 Here is the massive lesson I learned about Go tooling today: When you run air, the tool executes from your project root, not from where your main.go file lives. Because air was already at the root, telling it to go ../../ meant it was searching for my .env file completely outside my project folder! The Fix: I changed it back to godotenv.Load(). Because air runs from the root, it automatically found the .env file sitting right there. Stop relying on relative path hacks (../../). Understand your execution context! The infra thinking is officially kicking in. We move! 💪🏾 To my Gophers: What was your most frustrating "file path" bug when you first started using Go or Docker? Let’s gist 👇🏾 #Golang #BackendEngineering #DevOps #SoftwareDevelopment #TechBro #TechInNigeria #WeMove

Day 10/30 — If you can’t trace a failed request across services in under 2 minutes, your logging is broken. Most teams realize this during an incident. At 2 AM. With leadership asking, “What happened?” A user reports: “My order failed.” You check: Order Service → request looks fine Payment Service → no record API Gateway → thousands of requests, impossible to isolate one 45 minutes later, you’re still grepping logs across 5 services. That’s not a debugging problem. That’s a logging architecture problem. 3 things every production log must have 1️⃣ Structure — log JSON, not sentences Human‑readable logs don’t scale. Machine‑queryable logs do. Structured logs let you filter by orderId, userId, traceId, amount, latency — instantly. When you have millions of log lines, you don’t read. You query. 2️⃣ Correlation — one traceId everywhere Without a correlation ID: Gateway logs are one story Order logs another Payment logs a third With a single traceId, they become one timeline. One query should tell you: When the request entered Which service failed Why At which millisecond If you need multiple terminal windows and manual grep… you’ve already lost. 3️⃣ Centralization — all logs, one place Logs on individual servers are effectively invisible. Ship everything to a central system: ELK, Datadog, Loki, CloudWatch — pick your poison. Key rule: ✅ Log to stdout ✅ Let your platform collect & forward ❌ Don’t SSH into servers to read files If logs aren’t searchable centrally, they don’t exist during incidents. What to log (and what not to) ✅ Request entry & exit (with duration) ✅ Every external call ✅ Every exception with full context ✅ Every state transition (order created → payment started → failed) ❌ Tight loops ❌ Sensitive data (passwords, cards, tokens) ❌ DEBUG by default in production INFO + structured fields + traceId beats verbose noise every time. The rule that covers everything: A developer who’s never seen your system should be able to: Take a traceId from a customer complaint Reconstruct exactly what happened Across all services Without touching a single server If that’s not true today, your logging isn’t done yet. #microservices #springboot #java #backend #softwareengineering

Source: https://lnkd.in/eTi_MWqa 🚀 28% Code Coverage Boost Without Writing a Single Test 🚀 Tom Noah’s approach to restructuring data models (Java records, @Value annotations) cut redundant code and boosted coverage naturally. Instead of bloating tests, he focused on system design—smart move! 💡 Key Takeaways: - Auto-generated code distorts metrics; removing it fixes the root issue. - Immutable patterns + Builder pattern = cleaner, thread-safe systems. - Less testing = faster CI pipelines and fewer false contracts. 🔧 Pro Tip: Audit your data models—redundant code is a hidden bottleneck! #CodeQuality #DevOps

Lessons from Real Backend Systems Short reflections from building and maintaining real backend systems — focusing on Java, distributed systems, and the tradeoffs we don’t talk about enough. ⸻ We had logs everywhere. Still couldn’t explain the outage. At first, it didn’t make sense. Every service was logging. Errors were captured. Dashboards were green just minutes before the failure. But when the system broke, the answers weren’t there. What we had: [Service A Logs] [Service B Logs] [Service C Logs] What we needed: End-to-end understanding of a single request The issue wasn’t lack of data. It was lack of context. Logs told us what happened inside each service. They didn’t tell us how a request moved across the system. That’s when we realized: Observability is not about collecting signals. It’s about connecting them. At scale, debugging requires three perspectives working together: Logs → What happened? Metrics → When and how often? Traces → Where did it happen across services? Without correlation, each signal is incomplete. The turning point was introducing trace context propagation. [Request ID / Trace ID] ↓ Flows across all services ↓ Reconstruct full execution path Now, instead of guessing: * We could trace a failing request across services * Identify latency bottlenecks precisely * Understand failure propagation Architectural insight: Observability should be designed alongside the system — not added after incidents. If you cannot explain how a request flows through your system, you cannot reliably debug it. Takeaway: Logs help you inspect components. Observability helps you understand systems. Which signal do you rely on most during incidents — logs, metrics, or traces? — Writing weekly about backend systems, architectural tradeoffs, and lessons learned through production systems. Keywords: #Observability #DistributedSystems #SystemDesign #BackendEngineering #SoftwareArchitecture #Microservices #Tracing #Monitoring #ScalableSystems

🛠️ How I Debug Slow APIs in Production (Step by Step) When an API is slow in production, guessing won't help. A structured investigation makes all the difference. Here's my step-by-step approach: 1. Verify if it's actually slow Start by measuring instead of assuming. Use Postman or curl to check response time. Slow only in browser → likely a frontend issue Slow in Postman/curl too → backend issue ✅ Lesson: Always validate with real data first. 2. Check network latency Latency = time taken for a request to reach the server. Using curl, you can analyze: DNS lookup time Connection time Time to first byte (TTFB) High latency → possible network or load balancer issue Low latency → issue is likely within the server ✅ Lesson: Separate network delay from processing delay. 3. Analyze server processing time Break down execution across layers: Controller Service Database Add structured logs with timestamps (tools like Winston or Logback work great here) to pinpoint exactly where time is being lost. ✅ Lesson: Isolate the bottleneck instead of debugging blindly. 4. Investigate database performance In most systems, the database is the #1 bottleneck. Check for: Slow queries Missing indexes High CPU usage Use query logs and EXPLAIN plans to optimize before you even think about scaling. ✅ Lesson: Fix inefficient queries before scaling infrastructure. 5. Use APM tools for full observability Tools like New Relic, Datadog, or Grafana give you: Method-level timing External API call breakdown Database query performance ✅ Lesson: You can't fix what you can't see. Observability is non-negotiable. 💡 Debugging in production is not about guesswork. It's about narrowing down the problem using data at every step. I've seen teams waste days on the wrong layer — just because they skipped step 1. What's your go-to tool or trick when debugging slow APIs? Drop it in the comments 👇 #BackendDevelopment #SoftwareEngineering #APIDevelopment #WebPerformance #DatabaseOptimization #Debugging #SystemDesign #DevOps #Observability #ProgrammingTips #CleanCode #TechLeadership #SoftwareDevelopment #100DaysOfCode #CodingLife

API Gateway: When It Helps, When It Hurts Another layer or actual value? API gateways are everywhere now. Auth, routing, rate limiting, logging - all in one place. Sounds perfect, right? Sometimes yes. Sometimes no. When It Helps ✅ You have many services. Each needs auth, SSL, rate limiting. Writing that logic ten times is painful. An API gateway does it once, in one place. You need to migrate or version APIs. The gateway can route old traffic to old services, new traffic to new ones. Smooth sailing. You have different teams owning different services. The gateway gives you a unified front door without coordination hell. When It Hurts ❌ You have one service and five users. The gateway is just another thing to deploy, monitor, and debug. Overkill. The gateway becomes a bottleneck. Every request goes through it. When it slows down, everything slows down. When it fails, everything fails. 🚨 You need low latency. Every hop adds milliseconds. For real-time systems, that hurts. Debugging becomes harder. "Is it the gateway or the service?" Another layer of mystery. What I've Learned Start without a gateway. Add one when the pain of not having it exceeds the pain of maintaining it. Don't add layers because they're trendy. Add them because you need them. #APIGateway #Microservices #SystemDesign #SoftwareArchitecture #BackendDevelopment #TechDecisions #Java

You hit “Enter” on a URL… and within milliseconds, you get a response. But here’s the truth most engineers miss 👇 👉 Your API doesn’t start in your controller… 👉 It starts in the OS kernel Before your Spring Boot app even sees the request: • DNS resolves the domain • OS creates a socket (file descriptor) • TCP handshake establishes a connection • TLS secures the channel • Data is split into TCP packets • Kernel buffers and reassembles everything And only then… your application gets a chance to run. --- 💡 The uncomfortable reality: Most developers spend 90% of their time optimizing: ✔ Controllers ✔ Queries ✔ Business logic But ignore the layers that actually control: ❌ Latency ❌ Throughput ❌ Scalability --- ⚙️ Real performance lives in: • Kernel queues (SYN queue, accept queue) • Socket buffers • Syscalls (accept, read, write) • Threading vs event-loop models • TCP/IP behavior --- 🚨 That’s why in production you see: • High latency with “fast” code • Thread exhaustion under load • Random connection drops • Systems that don’t scale --- 🧠 The shift that changed how I design systems: I stopped thinking in terms of “APIs” and started thinking in terms of: 👉 Data moving through layers Browser → OS → Kernel → Network → Server → App → Back --- If you understand this flow, you don’t just write code… 👉 You build systems that scale. --- 👇 I’ve broken this entire flow down (end-to-end) in the carousel Comment “DEEP DIVE” if you want the next post on: ⚡ epoll vs thread-per-request (what actually scales to millions of requests) #SystemDesign #BackendEngineering #DistributedSystems #Java #SpringBoot #Networking #Scalability #SoftwareEngineering #TechDeepDive

𝐒𝐨𝐦𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐜𝐫𝐢𝐭𝐢𝐜𝐚𝐥 𝐜𝐨𝐝𝐞 𝐢𝐧 𝐚 𝐬𝐲𝐬𝐭𝐞𝐦… 𝐫𝐮𝐧𝐬 𝐰𝐡𝐞𝐧 𝐧𝐨 𝐨𝐧𝐞 𝐢𝐬 𝐰𝐚𝐭𝐜𝐡𝐢𝐧𝐠. The most dangerous code in your backend… runs automatically. Not your APIs. Not your UI. Your background jobs. Recently, I implemented a simple cron job in NestJS to refresh exchange rates every hour: @𝗖𝗿𝗼𝗻('0 * * * *') async handleCron() { await this.service.fetchRates(); } At first glance, this looks straightforward. But in production, this “simple” job can introduce serious issues if not designed carefully. Here’s what actually matters: • Idempotency — The same job should run multiple times without corrupting data • Error Handling — External APIs fail, retries and fallbacks are essential • Concurrency — Multiple instances = duplicate executions unless controlled • Observability — Logs, metrics, and alerts are non-negotiable • Scalability — Heavy tasks should be moved to queues, not handled inline Cron jobs are not just schedulers. They are critical automation layers that directly impact system reliability. A well-designed cron job goes unnoticed. A poorly designed one becomes a production incident waiting to happen. How are you managing background jobs in your systems?