Video search with your vector database is a hot topic now as multi-modal model capabilities is getting stronger. Check it out!

[Blog] Accelerate database migration to Amazon #Aurora #DSQL with #Kiro and Amazon #Bedrock #AgentCore In this post, we walk through the steps to set up the custom migration assistant agent and migrate a #PostgreSQL database to Aurora DSQL. We demonstrate how to use natural language prompts to analyze database schemas, generate compatibility reports, apply converted schemas, and manage data replication through #AWS #DMS. As of this writing, AWS DMS does not support Aurora DSQL as target endpoint. To address this, our solution uses Amazon Simple Storage Service (Amazon #S3) and AWS #Lambda functions. https://lnkd.in/dQZSdhzc

𝗧𝗼𝗱𝗮𝘆’𝘀 𝗔𝗪𝗦 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴: DynamoDB Hot Partition Problem 𝗪𝗵𝗮𝘁 𝗶𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁: DynamoDB table created → Reads/Writes failing or slowing down → Even though capacity seems sufficient 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼: Application stores data using a partition key (e.g., userId) Suddenly high latency / throttling errors CloudWatch shows uneven usage 𝗙𝗶𝗿𝘀𝘁 𝗔𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻: DynamoDB is slow Capacity too low AWS issue 𝗔𝗰𝘁𝘂𝗮𝗹 𝗜𝘀𝘀𝘂𝗲: Too many requests hitting the same partition key Uneven traffic distribution → one partition overloaded Other partitions idle 𝗪𝗵𝘆 𝗧𝗵𝗶𝘀 𝗛𝗮𝗽𝗽𝗲𝗻𝘀: Poor partition key design (e.g., same user, same region, same ID) Time-based keys (like “today”) causing concentrated traffic DynamoDB scales per partition, not globally 𝗪𝗵𝗮𝘁 𝗛𝗲𝗹𝗽𝘀: ✓ Design better partition keys (high cardinality) ✓ Add random suffix / sharding to keys ✓ Use composite keys (partition + sort key) ✓ Monitor ConsumedCapacity & partition usage 𝗞𝗲𝘆 𝗟𝗲𝘀𝘀𝗼𝗻: DynamoDB doesn’t fail because of size… it fails because of uneven access patterns. One Line Takeaway: If DynamoDB slows down unexpectedly, check your partition key design before scaling capacity. #AWS #DynamoDB #NoSQL #DatabaseDesign #CloudComputing #DevOps #Troubleshooting #LearningInPublic

Continuing my studies on databases, I present a summary focusing on NoSQL and AWS DynamoDB. It's important to remember that NoSQL doesn't replace SQL in all situations. But imagine a scenario where your application receives a large volume of requests during the account opening process. You need a consistent API with low latency. You need to think about data storage oriented towards access, not relationships. For this, a key-value database, perhaps with a document type like JSON as the value, is ideal for the response. Let's analyze DynamoDB in this scenario, focusing on 5 points:  🔹𝗧𝘆𝗽𝗲 (𝗗𝗮𝘁𝗮 𝗠𝗼𝗱𝗲𝗹): - In DynamoDB, you work with Key-Value and Document models; - No JOINs (Like UserTable join AddressTable). You design based on access patterns. 🔹𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁: - Fully serverless; - No server provisioning, no patching, no operational overhead; - TTL - define data lifetime. 🔹𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆: - Native integration with AWS IAM; - Transactional encryption. 🔹𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴: - Native integration with Amazon CloudWatch; - Real-time metrics (latency, throughput, throttling). 🔹𝗖𝗼𝘀𝘁: - Pay only for what you used. In a scenario with high request volume, latency requirements, and predictable access patterns, Amazon DynamoDB becomes a strong fit. Suggested article for further reading: https://lnkd.in/djm4dTN4 #AWS #SQLxNoSQL #Backend #DynamoDB #CloudComputing #SoftwareEngineering #Dev

This weekend, as part of working on a major RFP response, I had to design a multi-region architecture on Azure. OLTP system, PostgreSQL- For data layer my first instinct was to look for something like Aurora Global Database because in an ongoing digital transformation engagement, that is exactly what I am using to architect a multi-region data layer on AWS. Clean, elegant, and purpose-built for this problem. Turns out, Azure does not have an equivalent. After analyzing the RFP requirements specifically the availability, RTO, and RPO targets, Azure PostgreSQL Flexible Server turned out to be the right fit. But the curious engineer in me did not stop there. I wanted to understand why these two services are so different, not just in features, but in the engineering beneath. The diagram below is what I came up with. Aurora (top)- AWS built a custom distributed storage engine that is shared across all compute instances within a region. Cross-region replication happens at the storage layer using AWS custom redo log streaming. The PostgreSQL compute layer has zero involvement in replication. This is why Aurora’s intra-region failover is 10-30 seconds, it is purely a compute redirect, nothing more. Azure PostgreSQL Flexible Server (bottom)- three separate independent storage instances, each kept in sync using PostgreSQL native WAL streaming replication. Intra-region HA standby uses synchronous WAL primary waits for acknowledgement before confirming writes. Cross-region read replica uses asynchronous WAL where primary does not wait, RPO in seconds. Same category on paper. Fundamentally different engineering beneath. Shoutout to Ameya Pawar, had a great brainstorming session with him over the weekend on this. Always better when you think out loud with the right people. And to Sri Rajasekhar Jonnalagadda who challenged the architecture with sharp questions around replication, RTO, and RPO. That pushback led to a much deeper analysis than I had originally planned. The best kind of challenge. #happylearning PS- Azure PostgreSQL Flexible Server is architecturally closer to Amazon RDS PostgreSQL than to Aurora. Both use PostgreSQL native WAL streaming replication with separate independent storage per instance. The implementation details differ, particularly around how zone-redundant HA and cross-region replicas are managed, but the underlying replication principle is the same. Aurora is in a different league entirely, and that is the point.

🚨 MongoDB on Kubernetes: A Hidden Risk with hostPath Storage Migrating pods in Kubernetes sounds routine… until your database refuses to start. I recently faced a real-world issue while running a MongoDB StatefulSet using local storage (hostPath). After a pod rescheduling across nodes, everything looked fine at first—but: ❌ MongoDB wouldn’t start (DBPathInUse, Resource temporarily unavailable) ❌ Mongo-express returned ECONNREFUSED ❌ Data seemed “lost”… but it wasn’t 👉 The root cause? The data never moved. It stayed on the original nodes, leaving new pods with empty volumes. 💡 Key Takeaways from the Fix ✅ Reattach existing PersistentVolumes to the correct pods ✅ Force scheduling on the original nodes (nodeSelector / affinity) ✅ Reconfigure the ReplicaSet with proper Kubernetes DNS ✅ Update Mongo-express connection string 🛠️ Lessons Learned Avoid hostPath for production databases Prefer distributed storage or dynamic PVC provisioning Always plan for pod rescheduling scenarios Backups are not optional—they are critical Have you ever faced similar issues with stateful workloads in Kubernetes? Let’s discuss 👇 #Kubernetes #MongoDB #DevOps #CloudNative #StatefulSets #SRE

𝗔𝗺𝗮𝘇𝗼𝗻 𝗔𝘂𝗿𝗼𝗿𝗮 𝗣𝗼𝘀𝘁𝗴𝗿𝗲𝗦𝗤𝗟 𝗘𝘅𝗽𝗿𝗲𝘀𝘀 𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗮𝘁𝗶𝗼𝗻 VPCs. Subnets. Security groups. Database engines. Parameter groups. This was your Sunday evening just to get a dev database running. Not anymore. Amazon Aurora PostgreSQL Express Configuration is live. One click. Preconfigured intelligent defaults. Your database is ready in 60 seconds. 𝐇𝐞𝐫𝐞'𝐬 𝐰𝐡𝐚𝐭 𝐭𝐡𝐢𝐬 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐦𝐞𝐚𝐧𝐬: Aurora Serverless v2 under the hood — automatic scaling from 0.5 to 128 ACUs based on actual demand. Built-in monitoring, backup scheduling, and encryption. All the production-grade capability. Zero configuration burden. 𝐓𝐡𝐞 𝐫𝐞𝐚𝐥 𝐬𝐭𝐨𝐫𝐲: AWS is making a bet that developer experience is the next competitive battleground. Not just "can you run PostgreSQL at scale?" but "how fast can someone go from idea to executing a query?" Express Configuration removes the infrastructure decision fatigue. You still get full Aurora compatibility — pgvector for AI workloads, Babelfish for T-SQL migration, everything. You just don't need a week to provision it. 𝐖𝐡𝐞𝐧 𝐭𝐡𝐢𝐬 𝐦𝐚𝐭𝐭𝐞𝐫𝐬: - Hackathons. Students spinning up their first database. Startups testing product-market fit. Engineers who need a database NOW, not after 47 CloudFormation parameters. 𝐖𝐡𝐞𝐧 𝐲𝐨𝐮 𝐦𝐢𝐠𝐡𝐭 𝐬𝐤𝐢𝐩 𝐢𝐭: - Complex multi-AZ requirements, custom VPC peering scenarios, or when you need that level of control. The express option is opinionated by design. The broader pattern: AWS is bifurcating their services into two experiences. - Path A: Maximum control, maximum configuration (for architects who need it) - Path B: Sensible defaults, fast time-to-value (for teams who need to move) Both coexist. Both are valid. The skill is knowing which one your use case demands. --- When did you last spin up a database just to test an idea? Did the provisioning overhead stop you? #AmazonAurora #PostgreSQL #Serverless #DeveloperExperience #AWS #CloudArchitecture #ArchitectWithUs

From weeks of configuration to 60 seconds: Aurora PostgreSQL Express Configuration just changed the database provisioning game. Developer experience is the new competitive battleground. #AWS #AmazonAurora #AmazonRDS #ArchitectWithUs

Haven't posted about anything PostgreSQL in a long time. Until now! Why? Because I was blown away when I read that Ring stored 200 billion vector embeddings... In PostgreSQL! Ring just published how they built semantic video search at scale: - 200 billion embeddings stored - ~2 billion new embeddings per day - 9 AWS Regions across 4 continents - Billions of read requests daily - Under 2-second search latency!!! WHOA!!! And the database: Amazon RDS for PostgreSQL with pgvector. Not Pinecone. Not Weaviate. Not Milvus. PostgreSQL. This is one of the largest production pgvector deployments ever documented. For every team debating "dedicated vector database vs pgvector in our existing PostgreSQL", Ring just answered the question at a scale that dwarfs most use cases. The practical takeaway: --- If you're already running RDS PostgreSQL, you probably don't need a separate vector database. Enable pgvector. Add your embeddings. Scale it. If your scale exceeds what a single RDS instance can handle (and Ring's certainly does), the architecture patterns for sharding and multi-region are now documented. The vector database market just got disrupted by... PostgreSQL. Source: https://lnkd.in/gxxv5BXk #PostgreSQL #pgvector #VectorSearch #AWS #DataEngineering

AWS RDS vs DynamoDB (SQL vs NoSQL on AWS) If you're learning AWS, this is a concept you need to get right early: 👉 𝗥𝗗𝗦 𝗮𝗻𝗱 𝗗𝘆𝗻𝗮𝗺𝗼𝗗𝗕 𝗮𝗿𝗲 𝗯𝘂𝗶𝗹𝘁 𝗳𝗼𝗿 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗱𝗮𝘁𝗮 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀 Here’s a clear breakdown 👇 --- 🔵 𝗥𝗗𝗦 (𝗦𝗤𝗟 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲)  • Tables (rows & columns)  • Supports joins  • Flexible SQL queries  • Fixed schema 👉 Best for:  • Structured data  • Relationships (users, orders, payments)  • Complex querying --- 🟠 𝗗𝘆𝗻𝗮𝗺𝗼𝗗𝗕 (𝗡𝗼𝗦𝗤𝗟 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲)  • Key-value / document model  • No joins  • Query based on access patterns  • Flexible schema 👉 Best for:  • High-scale systems  • Low-latency access  • Predictable queries --- ⚡ 𝗖𝗼𝗿𝗲 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀  • Data model → RDS (tables) | DynamoDB (key-value)  • Queries → RDS (flexible) | DynamoDB (pattern-based)  • Scaling → RDS (vertical) | DynamoDB (horizontal)  • Schema → RDS (fixed) | DynamoDB (flexible) --- 💡 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗧𝗶𝗽: Design first, then choose the database. --- #AWS #AmazonWebServices #RDS #DynamoDB #CloudComputing #DevOps #CloudArchitecture #SystemDesign #TechLearning #Database