PureStream Java ETL Library for Fast Data Processing

🚀 Built an End-to-End Data Pipeline using API, Python & SQL Server! Excited to share a hands-on project where I implemented a complete data pipeline across two systems 💻 🔹 Project Overview: ✔ Extracted data from PostgreSQL (Laptop 1) ✔ Exposed data via Django API (JSON format) ✔ Accessed API from another machine (Laptop 2) ✔ Converted JSON → CSV using Python (pandas) ✔ Dynamically created table (no manual schema!) ✔ Loaded data into SQL Server using pyodbc 🔹 Architecture: PostgreSQL → Django API → JSON → Python → CSV → SQL Server 🔹 Key Learnings: 💡 API as a bridge between systems   💡 Handling JSON data in real-world scenarios   💡 Automating schema creation   💡 Cross-machine data transfer   💡 Building end-to-end ETL pipelines  This project gave me practical exposure to how modern data pipelines work in real-world data engineering 🚀 Looking forward to building more scalable and production-ready pipelines! #DataEngineering #Python #SQLServer #FastAPI #Django #ETL #DataPipeline #APIs #LearningInPublic #100DaysOfCode

💥 Apache Sedona 1.9.0 is here 🚀 . What changes for practitioners: • You can run Sedona on Spark 4.1. • ST_Transform and Rasters are now backed by proj4sedona, a pure Java library with EPSG and ESRI authority codes, URL-based custom CRS catalogs, and grid-based transformations. It allows round-trip across EPSG, WKT1, WKT2, PROJ, and PROJJSON. • Large GeoTiffs are no longer blocked by Spark's record size limit. The raster reader auto-tiles on read. • You can write Cloud Optimized GeoTiffs natively. RS_AsCOG handles compression, tiling, and overview generation in a single SQL call. • Bing Tiles are native. Nine new functions, including ST_BingTileAt and ST_BingTileCellIDs. • Data source readers get a lift: _metadata hidden columns for Shapefile and GeoPackage readers, additional metadata fields in the OSM PBF reader. • GeoPandas API on Sedona keeps closing the gap with upstream. Full release notes: https://lnkd.in/g3NSZcdu #ApacheSedona #Geospatial #ApacheSpark #bigdata #community

🚀 New Project Kickoff — Serverless ETL Pipeline Starting a new AWS-based project to build a Serverless ETL Pipeline. 📌 Architecture Designed: CSV → S3 → Lambda (Python) → Transform → RDS/S3 → SNS Alert Currently in the planning phase, and excited to implement this step-by-step. More updates coming soon as I build this pipeline! #AWS #Serverless #CloudProjects #DataEngineering #LearningJourney

I've understood CDC for a while — WAL, logical replication, how Debezium creates a replication slot and publication. The theory was solid. What I'd never done is actually work on it. So I built a small project. PostgreSQL with wal_level=logical, Debezium reading the WAL stream, Kafka storing the events, a Python simulator pushing orders through a full lifecycle. Kafdrop on top to watch the topics fill up in real time. There's a difference between knowing how it works and watching pg_replication_slots confirm the slot is there. Still a work in progress — analytics sink still to come. But the core flow is working end to end. If you're handling CDC in production — schema evolution, exactly-once delivery — I'd like to hear how you approach it. https://lnkd.in/dN5rSf5r #CDC #PostgreSQL #Debezium #Kafka #Docker #DataEngineering

Day 2/60: Production Infrastructure That Actually Scales What Most Developers Do: Start with SQLite. Hardcode credentials. Skip migrations. Write blocking database calls. Wonder why it breaks at 10K users. What I Built Today: ✅ Async SQLAlchemy 2.0 with connection pooling ✅ Docker Compose (PostgreSQL + Redis + Backend) ✅ Alembic migration system with rollback ✅ Database health checks and monitoring ✅ Multi-stage Docker builds (40% smaller images) ✅ Development scripts (init, validate, wait-for-db) ✅ 31 tests, 100% coverage on database layer Technical Decisions: Async Everything: Non-blocking I/O handles 100 concurrent users on single thread Connection Pooling: QueuePool (5+10) for PostgreSQL, NullPool for SQLite Health Checks: pg_isready with retry logic, services wait for dependencies Type Safety: mypy --strict passes, Mapped[T] catches bugs at compile time Architecture Highlight: DatabaseManager singleton manages lifecycle. Session context managers handle transactions. Automatic rollback on errors. Zero connection leaks. Why It Matters: Technical debt is a choice. Building for 10K users from day one means adding workers when growth comes, not rewriting the database layer. What's Working: ``` docker-compose up -d → All services healthy pytest → 31/31 tests passing Database connection → ✅ Validated ``` Metrics: - 11 new files - 1,800 lines of production code - 600 lines of documentation (DATABASE.md) - 100% test coverage on new code - 0 linting errors Day 3 Tomorrow: Database models (User, Organization, Channel, Post). First Alembic migration. Schema design for ML features. Buffer - Building a solid foundation for your API ecosystem. Would love to connect. Repository: https://lnkd.in/g8pdgJvM Medium Blog: https://lnkd.in/gRrs6WaR #BufferIQ #BuildingInPublic #DatabaseEngineering #Docker #Python #PostgreSQL #SQLAlchemy #SoftwareArchitecture #Buffer

SQL Views are often explained as “virtual tables” — but that doesn’t really capture how they work. A better way to think about a View is as a stored query, similar to a function. You write a query once, give it a name, and call it whenever you need specific data. Instead of rewriting complex joins, filters, or role-based logic every time, you simply call the view — and it executes the logic on current data. 👉 No data is stored 👉 Logic is reusable 👉 Output is always up-to-date Stop thinking of Views as tables. Start thinking of them as query functions for your database. #SQl #databases #webdev #python #backend #Mysql #SQLite

The last two mssql-python releases shipped big features: Bulk Copy in 1.4 for high-throughput data loading, and Apache Arrow in 1.5 for zero-copy analytics. Version 1.6 is about what happens next: you take those features into production, scale up your thread pool, and find out where the driver was quietly holding you back. https://lnkd.in/d-snaHPW

A Spark Config That Took Me Days to Find True story. I had a Spark job that worked perfectly in testing. Deployed it to production. The numbers were just... wrong. No crash. No error message. Just wrong output. I spent days checking the logic. Rewriting queries. Adding logging everywhere. Nothing. Turns out the problem had nothing to do with my code. Spark has a feature called "whole-stage code generation." Behind the scenes, it compiles your SQL into Java code to make it run faster. Cool, right? Except Java has a size limit on how big a single piece of compiled code can be. When your data has a lot of columns (we're talking 20-30+), the generated code gets too big. Spark doesn't tell you. It just quietly produces garbage. The fix? One line of config: spark.sql.codegen.wholeStage = false That's it. It tells Spark: "don't try to be clever with code generation, just process it the normal way." Slightly slower. But correct every time. I share this because I know someone out there is debugging the same thing right now, staring at perfectly fine-looking code, wondering why the numbers don't add up. Check your Spark codegen. It might save you a weekend. #ApacheSpark #DataEngineering #Debugging #TechTips #PySpark #RealTalk