Shiv Iyer

We pushed PostgreSQL to its limit: 1 Petabyte of data. Michael Zhilin from Postgres Professional details the findings, bottlenecks, and architectural changes required to handle extreme volume in a distributed PostgreSQL environment. See what happens at the very edge of scale. #ExtremeScaling #DistributedDatabase #PostgresPro

23

Build a real-time lakehouse with Apache Doris + AWS Glue + S3 Tables ➡️ Join our meetup in SF to see the full demo: https://lnkd.in/gRmeZWUd 🔧 Stack: - S3 stores Iceberg table data - AWS Glue as a central metadata catalog for schema and table management - Apache Doris (via VeloDB) brings unified batch + real-time SQL analytics on top 💡 Pros of this setup: - Real-time analytics: Doris delivers sub-second, high-concurrency queries directly on S3 data. - Open architecture: Keep data in open formats (Parquet, Iceberg) for multi-engines access. - Simple AWS integration: Glue makes tables easily managable across AWS services (Athena, EMR, Redshift Spectrum...) - Cost-efficient: Cheap S3 storage, elastic Doris compute, and no complex ETL pipelines. Our quick demo below shows connecting Glue and S3 to Doris through VeloDB Studio, then create and query a new table, all in real time. 🎟️ RSVP link at the top, join our event in SF to see our demo and demos from Uber, Pinterest, and Datastrato. #AWS, #ApacheDoris, #Gravitino

13

𝐀𝐧𝐧𝐨𝐮𝐧𝐜𝐢𝐧𝐠 𝐂𝐨𝐬𝐦𝐨𝐬 𝐃𝐁 𝐢𝐧 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐚𝐛𝐫𝐢𝐜 𝐅𝐞𝐚𝐭𝐮𝐫𝐢𝐧𝐠 𝐍𝐞𝐰 𝐂𝐚𝐩𝐚𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬! (𝐏𝐫𝐞𝐯𝐢𝐞𝐰) Cosmos DB in Fabric brings the power of Azure Cosmos DB to the Fabric platform—combining high availability, dynamic scaling, and AI-optimized performance in a fully managed experience. Cosmos DB in Fabric is powered by the same Azure Cosmos DB service, now natively integrated into Fabric. 🧷𝑹𝒆𝒂𝒅 𝑴𝒐𝒓𝒆 𝒉𝒆𝒓𝒆: https://lnkd.in/gZZNRhyC #MicrosoftFabric #CosmosDB #DataEngineering #Lakehouse #Warehouse #OneLake #DataArchitecture #FabricWorkloads #Spark #TSQL #Databricks #Azure #LakehouseVsWarehouse Magudeswaran | Kaviya | Manikanta Reddy | Srinivasareddy | Sreethar M B | Suresh | Maureen | Krishnakanth | Satya Sekhar | Subhasis Das | RAMA | Santosh | Mahesh | India Microsoft Fabric Analytics and AI User Group

16

1 Comment

🚀 Big updates for #ApachePaimon! 1️⃣ Streaming Lakehouse • Snapshots now sync Iceberg V3 tables (incl. Deletion Vectors) into Paimon in near-real time—query latency in minutes. • Tight integration with Flink keeps stream & batch fully unified. 2️⃣ Multimodal for the AI era • Built-in Lance format lets Paimon store large blobs efficiently—perfect for audio, video, images, embeddings, etc. • Fast random reads and painless column adds. 3️⃣ Proven at scale • Alibaba runs hundreds of PB on Paimon; single tables hit 40 M rows/s and cut data freshness by an hour. • In production at vivo, Xiaomi, ByteDance, Shopee, and more. 4️⃣ Fully-managed in the cloud • Alibaba Cloud’s Data Lake Formation service now offers a managed Paimon catalog with smart tiering—30 % cheaper storage, 2× faster queries. • Public preview live in Singapore and Jakarta Paimon = one lake store for all your structured & unstructured AI data. 🌊🤖

22

🚀 Streaming to #AzureCosmosDB just got easier! The new Confluent-managed Kafka V2 connector is GA — with better scaling, observability, and native change feed support. Perfect for real-time data pipelines. 🔗 https://lnkd.in/e9tqSCBP #ApacheKafka #EventStreaming

1

▶️ In our "Getting to Grips with Confluent" video series, we have explored stream processing, Apache Kafka, and core Confluent mechanics. In our fourth video, Oscar Moores compares Confluent Platform and Confluent Cloud. While both services deliver similar core functionality, they differ in ease of use, management, and operational requirements. Confluent Cloud requires no physical infrastructure and comes with a managed schema registry and managed connectors, making it simpler for many organisations to deploy. 🔗 Find out more when you watch the full episode: https://lnkd.in/eH34nMRs Ozan Guzeldereli #Confluent #CloudSolutions #Kafka

3

Introducing pgraft: Raft Consensus Protocol for PostgreSQL Achieve true high availability with automatic leader election, zero split-brain scenarios, and crash-safe replication. Built on the same etcd-io/raft engine that powers Kubernetes. Production-ready for PostgreSQL 16, 17, and 18. #PostgreSQL #HighAvailability #RaftConsensus #DatabaseClustering #DistributedSystems #pgElephant #DevOps #SRE #DatabaseEngineering #OpenSource #CloudNative #Kubernetes https://lnkd.in/gcMDmRfi

1

🔥 Baserow 1.35 is live — focused on visualization, data sync, and flexibility. 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭’𝐬 𝐧𝐞𝐰: → Pie & doughnut charts for dashboards → Two-way sync with PostgreSQL → Formula fixes and improvements → Selective workspace exports + more 📖 Full release notes → https://bsrw.io/HOLsnh #baserow #nocode #appbuilder #productivity #charts

19