Part 1 was about the infra. This is Part 2 - what I learned I learned once the agents were actually running. Honestly, the hardest bugs weren't in the model. They were in the plumbing around it. Early on I had agents passing messages to each other. By the time a result reached the final node, nobody could tell where it came from or why. I removed that out and replaced it with a single shared state object, a TypedDict that every agent reads from and writes to. That one change made debugging go from impossible to just hard. Memory was harder than I expected. I assumed I could just stuff everything into the context window and call it done. I ended up with three layers: in-context for the current task, Redis for session state that needed to survive across turns, and a vector DB for long-term retrieval. Each agent has a router that decides which layer to hit. I also started treating prompts like code. Every agent has its own system prompt, versioned in Git, reviewed in PRs, tested before deploy. A prompt is just another file. I don't know why it took me this long to think about it that way. The last thing and maybe the most underrated is the Postgres checkpointer. When an agent workflow fails at step 14 of 20, it doesn't restart from zero. It picks up at step 14. That alone has saved me more times than I can count. If you want to talk through the architecture DMs are open. #AgenticAI #LangGraph #Python #AWS #AIEngineering #MLOps #AIEngineering #SystemDesign #Terraform #Pinecone #Redis #LangGraph #AgenticAI #LLMOps #RAG

🚀 Beyond the Cache: Building a Durable Redis Clone from Scratch I recently challenged myself to go "under the hood" of one of the most popular databases in the world. I’m excited to share that I’ve successfully built a Redis-compatible in-memory database from scratch using Python! While many use Redis as a simple black box, building it taught me the intricate balance between high-speed volatile storage and reliable disk persistence. 🛠️ The Architecture Breakdown I designed the system around a single-threaded event loop to handle concurrent client connections without the overhead of heavy threading. Here’s what’s happening inside: 🔹 Networking & Protocol Using Python’s socket and select modules, I implemented a Client Connection Handler that multiplexes requests from multiple telnet sessions over Port 6379. 🔹 Command Processing A central Command Handler validates and routes commands, ensuring they follow the logic expected by a standard Redis client. 🔹 Intelligent Storage I built a Data Store that manages a Key-Value engine alongside an Expiration Manager. It uses a hybrid strategy (Lazy + Active expiration) to ensure expired keys don't sit in memory forever. 🔹 Persistence Manager Implemented an AOF (Append-Only File) mechanism with configurable fsync policies (always, everysec, no) and background AOF rewriting to ensure data survives a crash. 💡 Key Takeaway: Building this from the ground up gave me a deep appreciation for non-blocking I/O and the complexity of ensuring data atomicity. 📂 Check out the project here: https://lnkd.in/gvbUjcji #SoftwareEngineering #Redis #Python #BackendDevelopment #SystemsDesign #DatabaseInternal #LearningByBuilding #Persistence

🚀 Backend Learning | Caching Patterns for High-Performance Systems While working on backend systems, I recently explored different caching strategies used to improve performance and scalability. 🔹 The Problem: • Frequent database hits increasing latency • High load under traffic • Need for faster response times 🔹 What I Learned: • Cache Aside (Lazy Loading): Load data into cache on demand • Write Through: Write to cache and DB simultaneously • Write Back (Write Behind): Write to cache first, DB updated later 🔹 Key Insights: • Cache Aside → Simple & widely used • Write Through → Strong consistency • Write Back → High performance but complex 🔹 Outcome: • Reduced database load • Faster API responses • Better system performance Caching is not just about storing data — it’s about choosing the right strategy. 🚀 #Java #SpringBoot #Redis #SystemDesign #BackendDevelopment #Caching #LearningInPublic

🚀 Built a full real-time CDC pipeline from scratch, here's what it looks like running. What you're seeing 👇 ① 📄 README & architecture PostgreSQL 16 (WAL logical replication) → Debezium 2.6 → Kafka→ Apache Flink 1.19 → DuckDB → dbt 1.11 → Grafana 10.4, orchestrated by Airflow 2.9 and provisioned with Terraform on GCP. ② 🐳 12 containers, all healthy The full stack runs on Docker Compose. One command brings up every service: source DB, CDC connector, message broker, stream processor, warehouse, transformation layer, orchestrator, and dashboard. ③ ⚡ CDC events in real time Two terminal panes side by side: the Python order simulator inserting events into PostgreSQL, and Kafka consuming the Debezium change stream live. Every INSERT and UPDATE flows through the WAL without touching application code. ④ 🔥 Apache Flink job running Flink Web UI showing the PyFlink job DAG: Kafka source → filter → filesystem sink. The job filters snapshot events and writes clean NDJSON to the warehouse volume continuously. ⑤ 🌀 Airflow orchestration The dbt_ecommerce_pipeline DAG running on a 5-minute schedule. A mix of completed, running, and retrying tasks visible across recent runs — a realistic view of a live pipeline in motion. ⑥ 📊 Grafana dashboard going live 4 panels updating in real time: orders by status, gross revenue by hour (trending up 📈), order volume by hour (trending up 📈), cancellation rate % (trending down 📉). All served via a custom Python SimpleJSON REST API bridging DuckDB to Grafana. ⑦ ☁️ Terraform The full infrastructure defined as code and ready for GCP deployment. Seven configuration files covering the network, database, storage, and access layers. The README documents every resource. #DataEngineering #CDC #Debezium #ApacheFlink #Kafka #dbt #DuckDB #Airflow #Terraform #GCP #Python #PortfolioProject

🚀 Built a local POC system that can process 100,000+ orders per second. Here's what I learned. In financial services and high-frequency trading, peak write rates aren't gradual — they're a vertical cliff. Thousands of transactions hit simultaneously, and every millisecond of latency has consequences. I just published a 5-part engineering deep-dive on building a horizontally-sharded, fault-tolerant order pipeline using: ⚡ Redis Streams (4-shard architecture) — 600,000 ops/sec ceiling 🐘 PostgreSQL — batched writes, fully decoupled from the HTTP layer 🐍 Python (FastAPI + asyncio) — sub-millisecond producer latency ☕ Spring Boot & Quarkus — polyglot consumer implementations 𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗯𝘂𝗶𝗹𝗱: ✅ Why Redis Streams beats Kafka for low-latency booking pipelines (no operational overhead, sub-ms write latency, built-in consumer groups + PEL for at-least-once delivery) ✅ Why Python's built-in hash() is dangerous for shard routing at scale — and how SHA-1 solves it ✅ How horizontal sharding makes scaling additive: bump NUM_SHARDS, get linear throughput — zero code changes ✅ Circuit breaker patterns and graceful degradation under shard failure ✅ Batch insert tuning that turns 1,000 individual DB writes into a single efficient operation This isn't just a side project — it's a distillation of 22 years in financial services (Lehman Brothers, Morgan Stanley, JPMorgan Chase) compressed into working, testable code. 📖 Full engineering write-up on Medium: https://lnkd.in/dc4SZu-Z 💻 Full source code on GitHub (Python + Spring Boot + Quarkus): https://lnkd.in/dwPTn9Qh 🙏 Special credit to Claude (Anthropic) — my AI pair programmer throughout this build. Claude helped architect the sharding logic, debug race conditions, and sharpen the engineering narrative. Human expertise + AI collaboration = faster, better systems. 💡 As Einstein said: "Everything should be made as simple as possible, but not simpler." That principle guided every design decision here — strip away what you don't need, keep exactly what you do. — #SystemDesign #HighThroughput #RedisStreams #PostgreSQL #SpringBoot #Python #Quarkus #FinTech #DistributedSystems #SoftwareEngineering #GenAI #ClaudeAI #BackendEngineering

Introducing ddbm (Dynamodb migrator): a high-performance migration tool built for speed and safety. With dedicated implementation for #RustLang dev and #Python dev. both perfectly sync for flexibility and choice I use dynamodb alot, moving and migrating data from stage to dev and changing access pattern has been everyday usage for me I built the mini version in python in 2025, but i have to go to the specific python directory to perform migration now I implemented cli version in rust and also improve the overall performance and transformation Why ddbm?  ✅ Type-Safe Architecture: The Rust core uses an Enum-based transformation registry, eliminating runtime typos and synchronization bugs. ✅ Dynamic Templates: Powerful {placeholder functions} allow you to transform data on the fly (case changes, math operations, substrings, and more). ✅ Built-in Safety: State management allows for interactive resumes and one-click rollbacks (Undo) if anything goes wrong. ✅ Developer-First: Fully hardened with a comprehensive test suite and CI/CD validation. Managing DynamoDB tables just got a whole lot easier. 🛠️ - Map old column to new - Merge column - Remove - Manipulate fields on the fly - Much more #AWS #DynamoDB #RustLang #Python #SoftwareEngineering #DataMigration #DevOps

I recently dedicated a couple of days to building a change-data-capture pipeline from scratch using the AWS free tier. Here's a breakdown of the process: Pipeline Overview: CoinMarketCap API → Python → RDS Postgres → Debezium → Kafka → S3 (JSON) 1. A Python script accesses CoinMarketCap's free-tier API and upserts the top 10 cryptocurrencies into Postgres. 2. RDS Postgres serves as the source of truth, with every INSERT/UPDATE recorded in the write-ahead log. 3. Debezium connects to the WAL via a logical replication slot, converting each row change into a CDC event and publishing it to Kafka. 4. A single-broker Kafka in KRaft mode (without Zookeeper) buffers the events. 5. The Confluent S3 Sink consumes the topic and outputs the events as JSON, creating one file per minute. This entire setup runs on a single t3.micro instance with 1 GB RAM and 1 GB swap, utilizing one IAM role and one bucket, without any managed Kafka or paid tier services. Key Learnings: - On RDS, the master user isn't a superuser and can't create a role WITH REPLICATION. Instead, grant the built-in rds_replication role. This term is crucial, as the documentation covers it, but the error message may lead you astray. - Debezium's default decimal.handling.mode is precise, which emits NUMERIC columns as base64-encoded bytes in your JSON. Change it to string to avoid prices appearing as "YmFzZTY0." - The S3 sink task reports RUNNING before attempting a PutObject. If your IAM policy lacks s3:PutObject on arn:aws:s3:::bucket/* (note the /*), the sink appears healthy until the first rotation, when it fails. Verify PutObject permissions before trusting the task state. - Home WiFi's public IP can rotate unexpectedly. If your EC2 security group is scoped to "my IP" and your ISP gives you a new one overnight, you're locked out until you update the SG. What's next: Phase 2 — add schema validation and move infrastructure to Terraform. Phase 3 — land the S3 data in an open table format so the bucket becomes directly queryable. Demo video is attached. Please watch and let me know your feedback. Github repo link is in the comments.

Redis isn't just "that caching thing" – it's a Swiss Army knife for backend performance. I just published a deep dive into Redis commands you'll actually use in production (with real Django examples): The essentials covered: • SET flags (NX, XX, EX, PX) – distributed locking made simple • SCAN vs KEYS – why KEYS will crash your production Redis • Rate limiting with INCR + EXPIRE – atomic, thread-safe, reliable • String commands (APPEND, STRLEN, INCR) – and the unbounded growth trap • Data structures: Hashes, Lists, Sets, Sorted Sets – choose the right tool Real patterns you can use today: ✓ Cache with jitter (stop the thundering herd) ✓ Distributed locks with auto-expiry ✓ Leaderboards with ZSET ✓ Task queues with LPUSH/RPOP Perfect for Django developers moving from "Redis works" to "I know exactly why this pattern matters." Link 👇 https://lnkd.in/ddJq94Hm #Redis #Django #BackendEngineering #Python #DatabaseOptimization

Building Resilient AI Agent Pipelines with Redis Streams 🤖 Building with LangGraph and FastAPI is exciting, but how do you handle high-volume event processing without the complexity of Kafka? I've been exploring Redis Streams as a lightweight, high-performance alternative for managing event-driven architectures. In my latest article, I break down: How to balance load across multiple workers. The "Janitor" pattern for self-healing systems. Production-ready Python implementation for Producers and Consumers. Check it out on Medium: https://lnkd.in/gp_KidYa #GenerativeAI #Redis #CloudArchitecture #PythonDev #TechCommunity

📣 Appify Prefab 0.5.0 is now live on Maven Central. This release is all about making your domain models more expressive while keeping the infrastructure invisible. Here is what is new: - Richer Types: Support for single-field value types. You can now use dedicated types like Email or PhoneNumber instead of generic Strings for your domain logic. - Automatic Docs: Async API documentation is now generated directly from your events. Your documentation stays in sync with your code automatically. - Persistence: MongoDB support is officially here. Whether you need relational or document-based storage, Prefab handles the boilerplate. - Schema-First: Generate your domain events directly from AVSC files if you prefer starting with the schema. Important Change: - Safety First: Fields are now @NotNull by default. To make a field optional, you must now explicitly use the @Nullable annotation. We are pushing for "Safe by Default" to catch potential issues before they ever hit a runtime environment. Read the full changelog here: https://lnkd.in/eweK7MCg #Java #SpringBoot #MongoDB #EDA #SoftwareArchitecture #DDD