Moving my Python projects from local scripts to the cloud. I recently finished a weather data project that helped me bridge the gap between writing code and deploying a functional system. Instead of just running a script on my pc, I wanted to see if I could build a pipeline that manages itself. The project tracks weather trends in Greece, and here is how I put it together: The Pipeline: I used Python and Pandas to fetch and clean data from the Visual Crossing API. The Database: To keep the data persistent, I used a cloud-hosted PostgreSQL database. I had to ensure the logic was idempotent so the script wouldn't create duplicates during daily runs. The Automation: I used GitHub Actions to schedule the ETL process. It now runs automatically every morning without me needing to touch it. The Environment: I wrapped the whole thing in Docker to make sure it works exactly the same in the cloud as it does on my machine. The UI: I built a simple Streamlit dashboard to visualize the results. The most challenging part wasn't the code itself, but managing the "plumbing" handling secrets securely, setting up CI/CD workflows, and troubleshooting environment mismatches. Live Dashboard: [https://lnkd.in/d6GK6DZb]  GitHub Repo: [https://lnkd.in/dvzBrqgM] #DataEngineering #Python #Docker #Automation #LearningInPublic

Hi! Mastering Apache Airflow DAGs: From Basics to Production‑Ready Pipelines Apache Airflow has become the de‑facto standard for orchestrating complex data workflows. Its declarative, Python‑based approach lets engineers model pipelines as Directed Acyclic Graphs (DAGs) that are version‑controlled, testable, and reusable. Yet, despite its popularity, many teams still struggle with writing maintainable DAGs, scaling the platform, and integrating Airflow into modern CI/CD pipelines. This article dives deep into Airflow DAGs—the core artifact that defines what, when, and how tasks run. We’ll explore everything from the low‑level API to production‑grade best practices, complete with runnable code snippets and a real‑world end‑to‑end example. By the end, you should be equipped to design, test, and operate robust Airflow pipelines that can survive the rigors of enterprise data engineering. > Note: The examples target Airflow 2.7+, which introduces the `@dag` and `@task` decorators, native task‑flow API, and improved type hints. Adjustments for older 1.x versions are highlighted where relevant. Read the full guide: https://lnkd.in/d3z-7SVU #apacheairflow #dataengineering #orchestration #python #devops

Hi! Mastering Apache Airflow DAGs: From Basics to Production‑Ready Pipelines Apache Airflow has become the de‑facto standard for orchestrating complex data workflows. Its declarative, Python‑based approach lets engineers model pipelines as Directed Acyclic Graphs (DAGs) that are version‑controlled, testable, and reusable. Yet, despite its popularity, many teams still struggle with writing maintainable DAGs, scaling the platform, and integrating Airflow into modern CI/CD pipelines. This article dives deep into Airflow DAGs—the core artifact that defines what, when, and how tasks run. We’ll explore everything from the low‑level API to production‑grade best practices, complete with runnable code snippets and a real‑world end‑to‑end example. By the end, you should be equipped to design, test, and operate robust Airflow pipelines that can survive the rigors of enterprise data engineering. > Note: The examples target Airflow 2.7+, which introduces the `@dag` and `@task` decorators, native task‑flow API, and improved type hints. Adjustments for older 1.x versions are highlighted where relevant. Read the full guide: https://lnkd.in/d3z-7SVU #apacheairflow #dataengineering #orchestration #python #devops

Hi! Mastering Apache Airflow DAGs: From Basics to Production‑Ready Pipelines Apache Airflow has become the de‑facto standard for orchestrating complex data workflows. Its declarative, Python‑based approach lets engineers model pipelines as Directed Acyclic Graphs (DAGs) that are version‑controlled, testable, and reusable. Yet, despite its popularity, many teams still struggle with writing maintainable DAGs, scaling the platform, and integrating Airflow into modern CI/CD pipelines. This article dives deep into Airflow DAGs—the core artifact that defines what, when, and how tasks run. We’ll explore everything from the low‑level API to production‑grade best practices, complete with runnable code snippets and a real‑world end‑to‑end example. By the end, you should be equipped to design, test, and operate robust Airflow pipelines that can survive the rigors of enterprise data engineering. > Note: The examples target Airflow 2.7+, which introduces the `@dag` and `@task` decorators, native task‑flow API, and improved type hints. Adjustments for older 1.x versions are highlighted where relevant. Read the full guide: https://lnkd.in/d3z-7SVU #apacheairflow #dataengineering #orchestration #python #devops

Excited to share that I’ve contributed to the latest release of AWS Lambda Powertools for Python (v3.26.0) https://lnkd.in/dG3C_R9P 🚀 This release introduces useful enhancements, including a new Lambda Metadata utility and improvements across logging, batch processing, and more. I’m particularly happy to have contributed to the logging space, helping improve how external loggers can be integrated with Powertools through buffering support - making observability in Lambda even more robust and flexible. Big thanks to the maintainers and contributors for the collaboration 🙌 If you’re building serverless applications on AWS, Powertools is definitely worth checking out. #AWS #Serverless #Python #OpenSource #CloudComputing #AWSLambda

I implemented a Diskless Kafka in Python in 1-2 days. Just finished optimizing its single-broker performance from 2 MB/s to 32 MB/s. 🔥 It blows my mind how far coding harnesses have come. A good chunk of the perf test -> fix -> test -> fix iteration I did solely from my phone. 🤯 If you've ever worked as any sort of technical lead, the workflow will be very familiar to you. It's conceptually similar to reading Slack, reviewing results then applying critical thinking on where things may have gone wrong: • are we stuck on a lock? • did we hit the max in-flight request limit? • are the commits parallelized? The difference is that the iteration literally happens 20-30x faster than with a teammate. It's a real force multiplier. Here's a quick view of how the perf iterations went: First test's starting point: 2.45 MB/s  → Reduced partitions 128→32: 4.24 MB/s (+61%)  → Parallelized metadata requests: 7.41 MB/s (+75%)  → Upped HTTP test clients to 512: 17.95 MB/s (+162%)  → Fixed a bad lock in partition init: 28.4 MB/s (+46%)  → A few more tweaks: 32 MB/s I'm 100% convinced the full codebase I wrote in a few days would have taken me a solid 30 focused days before, and I was considered a pretty productive member... That's what, ~$5,000-$10,000 in a regular salary? Now it's about ~$6 worth of AI subscriptions... and the ability to do it from your phone in a park. 🌲🌞

🚨 This DP problem almost looked impossible… until I broke it down step by step. Day 23 of my Backend Developer Journey — and today was all about thinking in transitions, not brute force 👇 🧠 LeetCode Deep Dive Solved Minimum Total Distance Traveled 💡 What clicked: → Sorting robots & factories simplifies decisions → DP helps track minimum cost efficiently → Key idea: assign k robots to one factory within capacity ⚡ Transition-based thinking = Game changer ⚙️ Approach Breakdown ✔️ Sort robots & factories ✔️ Use DP → dp[i][j] = min distance for first i robots using first j factories ✔️ Try assigning multiple robots to a factory (within limit) 🔗 My Submission: https://lnkd.in/gtj9xka3 ☕ Spring Boot Learning Today I strengthened my understanding of Spring Data JPA 🚀 🔹 What I Learned: 👉 Dynamic Sorting with Sort.by() 👉 Pagination using Pageable 👉 Writing cleaner and scalable repository queries 🧠 The Real Learning 👉 DP is not about memorizing patterns 👉 It’s about breaking problems into smaller decisions 👉 Backend is not just CRUD 👉 It’s about efficient data handling at scale 📘 Spring Boot Notes: https://lnkd.in/gpWQvkyK 📈 Day 23 Progress: ✅ Improved DP intuition ✅ Better understanding of scalable APIs ✅ Stronger problem-solving mindset 💬 Have you ever struggled with DP until one idea suddenly made everything click? 👇 #100DaysOfCode #BackendDevelopment #SpringBoot #Java #LeetCode #DataStructures #CodingJourney

𝗣𝘆𝘁𝗵𝗼𝗻 + 𝗞𝗮𝗳𝗸𝗮 𝗮𝘁 𝗦𝗰𝗮𝗹𝗲: 𝗗𝗲𝘀𝗶𝗴𝗻 𝗣𝗮𝘁𝘁𝗲𝗿𝗻𝘀 𝗳𝗼𝗿 𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 Building real-time systems that handle millions of events isn’t just about plugging Python into Kafka. It’s about designing systems that don’t break under pressure. I came across this solid breakdown of Kafka design patterns and it reinforces one thing 👇 👉 Architecture matters more than code when you scale. 🔗 Read the full article https://lnkd.in/gVUz6rT5

When code runs millions of times a day, even minor enhancements lead to significant compute savings. So I built xmltodict-fast. 🦀🐍 xmltodict is a Python library many of us use without a second thought. With ~5K GitHub stars, it’s a quiet workhorse powering ETL pipelines, SOAP clients, and invoice processors. It’s a drop-in replacement that maintains the same public API, but rewrites the performance-critical sections in Rust using PyO3 and quick-xml. Importantly: if the Rust extension isn't available on a platform, it seamlessly reverts to the original Python implementation. It's completely safe for incremental adoption. local benchmarks :  🚀 parse(): 2.1 × faster on typical XML  🚀 unparse():5.9 × faster (massive for serialization-heavy workflows)  On pathologically deep XML (500+ nesting levels), the Rust version is actually slower. :( (Side note: Thanks to my kind and patient AI coding assistant for helping me building this!) If you work with XML in Python, I welcome your feedback, testing, and pull requests! 🔗 Repo & Benchmarks: https://lnkd.in/exhfBuD7 #Python #RustLang #PyO3 #OpenSource #DataEngineering #PerformanceOptimization

I had 18,115 AWS API operation names in PascalCase that needed to become kebab-case. DescribeInstances to describe-instances. PutBucketAcl to put-bucket-acl. AWS's acronym casing is inconsistent across services, and I was not writing a custom Python converter for 18,000 edge cases. DuckDB has a community extension for this: INSTALL inflector FROM community; LOAD inflector; SELECT inflector_to_kebab_case('DescribeInstances'); -- describe-instances All 18,115 operations in one SQL pass. It also does snake_case, camelCase, train-case, pluralization, and bulk column renaming on structs. I used it to keep the raw PascalCase botocore contract in parquet and transform at query time — no slow Python string manipulation. https://lnkd.in/e8a_Aitd #duckdb #dataengineering #platformengineering #aws