Building Agentis Memory for Shared AI Agent Knowledge

Just shipped a distributed systems project And honestly i learned more debugging in this than in any project I've ever done 😅 I built a search recommendation engine from scratch, the kind of system that tracks what you click on and uses that data to re-rank future search results in real time The stack: → FastAPI for the APIs → Kafka as the message queue (click events flow through here) → PostgreSQL to store click scores → Redis to cache hot results → Docker Compose to wire everything together The whole flow looks like this: user clicks a result → event goes to Kafka → stream processor reads it → updates scores in postgres → next search returns re-ranked results with Redis serving the hot ones in <1ms sounds clean right? it was NOT clean getting there at all 😭 The issues i ran into (and how i fixed them): 1. Kafka advertising the wrong address to my Python service which kept getting a "connection refused" error, even though Kafka was running. Turned out Kafka was advertising itself as localhost:9092, which inside Docker means the container's own localhost, not the network address other containers could reach. fixed it by setting KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 so containers find each other by service name. 2. "depends_on" doesn't mean "ready" Docker's depends_on only waits for a container to start, not for the service inside to be ready. Kafka takes 20-30 seconds to fully boot. My stream processor was connecting before Kafka was ready and crashing. I fixed it with proper Docker healthchecks + retry logic in Python that actively probes Kafka every 5 seconds instead of blindly sleeping. 3. WSL2 dropping me into Docker's internal VM when i typed wsl it dropped me into Docker Desktop's internal Linux VM instead of a real Ubuntu distro. everything was broken — no sudo, no apt, wrong environment. Had to install Ubuntu properly via wsl --install -d Ubuntu and work from there. 4. Running Python outside Docker: I spent a while confused why my local Python couldn't reach Kafka. The fix was containerising the Python services too so everything runs on the same Docker network. Learned the hard way that localhost means something completely different inside vs outside a container. The most satisfying moment was watching the metrics endpoint show: cache hit rate: 57% kafka consumer lag: 0 cache HIT latency: 0.6ms vs cache MISS: 2.3ms That 12x latency difference between Redis and PostgreSQL is real and measurable not just theory anymore. Would apply this to an actual application next... Link to repo: https://lnkd.in/eDt65PvS #softwareengineering #distributedsystems #kafka #redis #python #docker #buildinpublic #devops Alright keep scrollingggg :)

Multi-cloud was never a tooling problem—it was a semantics problem, and solving that is what actually unlocks true portability. If MultiCloudJ delivers on its promise, it doesn’t just reduce lock-in—it fundamentally shifts control back to developers and architecture, where it belongs.

I recently dedicated a couple of days to building a change-data-capture pipeline from scratch using the AWS free tier. Here's a breakdown of the process: Pipeline Overview: CoinMarketCap API → Python → RDS Postgres → Debezium → Kafka → S3 (JSON) 1. A Python script accesses CoinMarketCap's free-tier API and upserts the top 10 cryptocurrencies into Postgres. 2. RDS Postgres serves as the source of truth, with every INSERT/UPDATE recorded in the write-ahead log. 3. Debezium connects to the WAL via a logical replication slot, converting each row change into a CDC event and publishing it to Kafka. 4. A single-broker Kafka in KRaft mode (without Zookeeper) buffers the events. 5. The Confluent S3 Sink consumes the topic and outputs the events as JSON, creating one file per minute. This entire setup runs on a single t3.micro instance with 1 GB RAM and 1 GB swap, utilizing one IAM role and one bucket, without any managed Kafka or paid tier services. Key Learnings: - On RDS, the master user isn't a superuser and can't create a role WITH REPLICATION. Instead, grant the built-in rds_replication role. This term is crucial, as the documentation covers it, but the error message may lead you astray. - Debezium's default decimal.handling.mode is precise, which emits NUMERIC columns as base64-encoded bytes in your JSON. Change it to string to avoid prices appearing as "YmFzZTY0." - The S3 sink task reports RUNNING before attempting a PutObject. If your IAM policy lacks s3:PutObject on arn:aws:s3:::bucket/* (note the /*), the sink appears healthy until the first rotation, when it fails. Verify PutObject permissions before trusting the task state. - Home WiFi's public IP can rotate unexpectedly. If your EC2 security group is scoped to "my IP" and your ISP gives you a new one overnight, you're locked out until you update the SG. What's next: Phase 2 — add schema validation and move infrastructure to Terraform. Phase 3 — land the S3 data in an open table format so the bucket becomes directly queryable. Demo video is attached. Please watch and let me know your feedback. Github repo link is in the comments.

Written by Luke Thompson - MongoDB Champion published on Friends of OpenJDK (Foojay.io), learn how to build a Java faceted full-text search API! In the tutorial, he'll walk through using a interesting dataset which showcases how you can effectively pair machine learning/AI-generated data with more traditional search to produce fast, cheap, repeatable, and intuitive search engines. Dive in here 👉 https://lnkd.in/gm6a2Y77 #mongodb #java #nosql #database #atlas

New blog post alert 🚨 "Serverless applications on AWS with Lambda using Java 25, API Gateway and DynamoDB – Part 6 Using GraalVM Native Image". In this article, we’ll introduce another approach to improve the performance of the Lambda function – GraalVM Native Image. If you like my content, please follow me on GitHub (github.com/Vadym79) and give my repositories like this https://lnkd.in/epud2eRf a star! Amazon Web Services (AWS) Oracle #Java #Serverless #AWS GraalVM https://lnkd.in/e-ZAAvaf

🚀 Built a local POC system that can process 100,000+ orders per second. Here's what I learned. In financial services and high-frequency trading, peak write rates aren't gradual — they're a vertical cliff. Thousands of transactions hit simultaneously, and every millisecond of latency has consequences. I just published a 5-part engineering deep-dive on building a horizontally-sharded, fault-tolerant order pipeline using: ⚡ Redis Streams (4-shard architecture) — 600,000 ops/sec ceiling 🐘 PostgreSQL — batched writes, fully decoupled from the HTTP layer 🐍 Python (FastAPI + asyncio) — sub-millisecond producer latency ☕ Spring Boot & Quarkus — polyglot consumer implementations 𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗯𝘂𝗶𝗹𝗱: ✅ Why Redis Streams beats Kafka for low-latency booking pipelines (no operational overhead, sub-ms write latency, built-in consumer groups + PEL for at-least-once delivery) ✅ Why Python's built-in hash() is dangerous for shard routing at scale — and how SHA-1 solves it ✅ How horizontal sharding makes scaling additive: bump NUM_SHARDS, get linear throughput — zero code changes ✅ Circuit breaker patterns and graceful degradation under shard failure ✅ Batch insert tuning that turns 1,000 individual DB writes into a single efficient operation This isn't just a side project — it's a distillation of 22 years in financial services (Lehman Brothers, Morgan Stanley, JPMorgan Chase) compressed into working, testable code. 📖 Full engineering write-up on Medium: https://lnkd.in/dc4SZu-Z 💻 Full source code on GitHub (Python + Spring Boot + Quarkus): https://lnkd.in/dwPTn9Qh 🙏 Special credit to Claude (Anthropic) — my AI pair programmer throughout this build. Claude helped architect the sharding logic, debug race conditions, and sharpen the engineering narrative. Human expertise + AI collaboration = faster, better systems. 💡 As Einstein said: "Everything should be made as simple as possible, but not simpler." That principle guided every design decision here — strip away what you don't need, keep exactly what you do. — #SystemDesign #HighThroughput #RedisStreams #PostgreSQL #SpringBoot #Python #Quarkus #FinTech #DistributedSystems #SoftwareEngineering #GenAI #ClaudeAI #BackendEngineering

Built a full stack async document processing system in a day. Here's what's under the hood 👇 DocFlow: an async document workflow engine where uploaded files go through a real multi-stage background processing pipeline, not a fake synchronous trick. The architecture: → Next.js 14 + TypeScript frontend with a full dashboard. Upload, live progress tracking, review/edit extracted fields, finalize & export → FastAPI (Python) backend with a clean 3 layer architecture: API routes → Service layer → Worker layer. Zero business logic in route handlers → Celery workers handle all processing in the background, completely outside the request-response cycle → Redis does double duty Celery message broker AND Pub/Sub channel for streaming 7 live progress events per job (job_queued → job_started → parsing_started → parsing_completed → extraction_started → extraction_completed → job_completed) → PostgreSQL + SQLAlchemy + Alembic for persistent job/document state with proper migrations → Server-Sent Events (SSE) to stream Redis Pub/Sub events to the frontend in real time → JWT auth, file storage abstraction layer (LocalStorage today, S3 ready interface), idempotent retry handling, cancellation support, and export as JSON/CSV → Docker Compose one command spins up all 5 services (API, Worker, Frontend, PostgreSQL, Redis) → Pytest test suite covering API, service layer, and workers For deployment I went through Docker on Render, Railway, and Vercel the peer dependency conflicts between TanStack/react-query v5 and Radix UI packages in the Node build made it a nightmare. The app runs perfectly locally via Docker Compose. Production cloud deploy was the only thing that didn't land in time. Both DocFlow and yesterday's project were built as 1 day take home assessments given by Predusk Technology Private Limited. Shipping production grade systems under real time pressure is a different kind of engineering test GitHub: https://lnkd.in/daAkhhKG #fullstack #python #fastapi #nextjs #celery #redis #docker #asyncprogramming #typescript #postgresql #buildinpublic #webdevelopment

🚀 Backend Learning | Caching Patterns for High-Performance Systems While working on backend systems, I recently explored different caching strategies used to improve performance and scalability. 🔹 The Problem: • Frequent database hits increasing latency • High load under traffic • Need for faster response times 🔹 What I Learned: • Cache Aside (Lazy Loading): Load data into cache on demand • Write Through: Write to cache and DB simultaneously • Write Back (Write Behind): Write to cache first, DB updated later 🔹 Key Insights: • Cache Aside → Simple & widely used • Write Through → Strong consistency • Write Back → High performance but complex 🔹 Outcome: • Reduced database load • Faster API responses • Better system performance Caching is not just about storing data — it’s about choosing the right strategy. 🚀 #Java #SpringBoot #Redis #SystemDesign #BackendDevelopment #Caching #LearningInPublic

MongoDB Atlas offers a powerful document model, enabling you to store data as JSON-like objects that closely resemble your application code. Read more 👉 https://lttr.ai/Ap3jo #Java #NoSQL #MongoDB

𝗦𝗽𝗿𝗶𝗻𝗴 𝗶𝘀 𝗲𝘃𝗼𝗹𝘃𝗶𝗻𝗴 𝗮𝗰𝗿𝗼𝘀𝘀 𝗮𝗹𝗹 𝗸𝗲𝘆 𝗮𝗿𝗲𝗮𝘀 𝗼𝗳 𝗯𝗮𝗰𝗸𝗲𝗻𝗱 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁. This week I read an update from InfoQ about the latest Spring ecosystem releases — and what stood out is how many areas are evolving at the same time. Here are some highlights: 🔹 Spring Boot → AMQP 1.0 support + MongoDB batch integration   🔹 Spring Data → improved Redis features and bulk operations in MongoDB   🔹 Spring Security → new authorization features + critical vulnerability fix   🔹 Spring Integration → better support for cloud events and messaging   🔹 Apache Kafka → improved acknowledgment handling and error strategies   🔹 Spring AMQP → stronger messaging support with AMQP 1.0   🔹 Spring AI → more flexible configuration for AI integrations   🔹 Spring Vault → simpler management of secrets and certificates  👉 Key takeaway: Java is not evolving in isolation. It’s advancing across   security, data, messaging, integration, and AI — all at once. From a backend perspective, this reinforces how important it is to understand not just frameworks, but the full landscape of modern systems: event-driven architectures, secure applications, and data flows. 💬 Curious — which of these areas is having the biggest impact in your projects? #Java #SpringBoot #Spring #BackendDevelopment #Microservices #Kafka #Security #Data #Cloud #DevOps #SoftwareArchitecture https://lnkd.in/ervTw5yN