Dinesh Kumar’s Post

Just wrapped up an end-to-end data engineering project as part of DataTalksClub’s Data Engineering Zoomcamp Built a pipeline to process GitHub Events data using Python, BigQuery, dbt, and Airflow. Some key things I worked on:   - Designed an ELT pipeline using GCS → BigQuery → dbt   - Implemented dimensional modelling (fact & dimension tables)   - Orchestrated workflows using Apache Airflow (Dockerised)   - Optimised performance using Parquet and partitioning   - Reduced query costs after initially scanning ~200GB per run One of the biggest learnings for me was how small design decisions (like partitioning and materialisation strategy) can have a huge impact on performance and cost. Also got to debug real-world issues like Airflow setup problems, inefficient dbt models, and I/O bottlenecks — which made the learning much more practical. Dashboard: https://lnkd.in/gxgQYVkH Github Repo: https://lnkd.in/gVtVvWR9 I will continue improving this project by adding new features and optimisations over time. Would love to hear any feedback! 🙂 #DataEngineering #BigQuery #ApacheAirflow

Most people use Claude Code like a smarter autocomplete. That's not what it is. If you structure your repo correctly, Claude Code operates more like a disciplined junior engineer — one that reads the docs before touching anything, follows your conventions, guards against dangerous operations, and leaves a clean audit trail after every session. The difference isn't the model. It's the project structure. Here's what actually matters: 1. CLAUDE.md — your AI onboarding doc. Client context, architecture diagram, coding conventions, known gaps. Auto-loaded every session. 2. A session brief (read.md) — what today's focus is, what was decided last time, what's locked. Prevents you repeating the same discovery work twice. 3. Slash commands — package your multi-step workflows as markdown files. /add-bronze-object, /add-gold-transform, /check-pipeline-status. One command, done correctly every time. 4. Hooks — Python scripts that intercept Claude before it runs a bash command or writes a file. Block destructive CLI calls. Catch bad SQL. Surface a git diff on exit. 5. Discovery docs — let Claude query your actual source DB and document what it finds. Real column names, real data patterns, real gotchas. No guesswork in the SQL. I ran this setup on a full Snowflake medallion pipeline — MSSQL source, Bronze → Silver → Gold, 25 objects. 25/25 built. 0 failures. One session. I also wrote a section on prompt pollution — what happens when vague or exploratory prompts silently contaminate your session context and why it's so hard to catch. Worth reading if you use any LLM in your data work. #DataEngineering #SnowflakeDB #ClaudeCode #ETL #ArtificialIntelligence #Python #DataPipeline #MLOps Full article 👇 https://lnkd.in/gc7tAXDA

🚀 Built an End-to-End Data Pipeline using API & SQL Server! Excited to share my recent hands-on project where I built a complete data pipeline from scratch 👇 🔹 What I did: 1. Source Database (SQL Server)   ↓ 2. Create API using FastAPI   ↓ 3. Expose endpoint (/data)   ↓ 4. Call API using Python (requests)   ↓ 5. Get data in JSON format   ↓ 6. Connect to Target SQL Server   ↓ 7. Auto-create table (if not exists)   ↓ 8. Insert data into target table   ↓ 9. Verify data in SSMS 🔹 Tech Stack: Python | FastAPI | SQL Server | pyodbc | requests 🔹 Key Learnings: 💡 How APIs act as a bridge between systems   💡 Converting JSON data into structured format   💡 Building real-world ETL pipelines   💡 Automating data movement without manual intervention  This project helped me understand how real-world data engineering pipelines work — from data extraction to loading 🚀 Looking forward to building more such projects and improving my skills! #DataEngineering #Python #FastAPI #SQLServer #ETL #DataPipeline #LearningInPublic #100DaysOfData #BuildingInPublic

I recently built a data pipeline that automatically tracks and visualizes real-time weather data. The project follows an ELT (Extract, Load, Transform) workflow to keep data moving quickly and accurately from the source to the final dashboard. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: • 𝗗𝗮𝘁𝗮 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻: A Python script pulls live weather data from an API every 5 minutes. • 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: The raw data is immediately loaded into a PostgreSQL database. • 𝗖𝗹𝗲𝗮𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗼𝗿𝘁𝗶𝗻𝗴: I use dbt to transform raw data into structured tables for analysis: • 𝘀𝘁𝗴_𝘄𝗲𝗮𝘁𝗵𝗲𝗿_𝗱𝗮𝘁𝗮: The staging table where raw API data is cleaned, validated, and prepared for further processing. • 𝘄𝗲𝗮𝘁𝗵𝗲𝗿_𝗿𝗲𝗽𝗼𝗿𝘁: A refined table designed for real-time monitoring with clear, analysis-ready weather insights. • 𝗱𝗮𝗶𝗹𝘆_𝗮𝘃𝗲𝗿𝗮𝗴𝗲: An aggregated table that summarizes daily weather metrics to track trends over time. • 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻: Apache Airflow orchestrates the entire process. • 𝗟𝗶𝘃𝗲 𝗗𝗮𝘀𝗵𝗯𝗼𝗮𝗿𝗱: Apache Superset displays results with a 5-minute auto-refresh. • 𝗦𝗲𝘁𝘂𝗽: Fully containerized using Docker for easy deployment. 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀: • 𝗡𝗲𝗮𝗿-𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲: Data updates every 5 minutes. • 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲: Prevents duplicates and ensures high-quality data. • 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: ELT enables scalable transformations inside the database. This project helped me build a complete, automated data system from scratch. #DataEngineering #ELT #Python #SQL #Airflow #Docker #DataPipeline #WeatherUpdate

New project unlocked🔓 I just finished building a 𝗖𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝗟𝗶𝗳𝗲𝘁𝗶𝗺𝗲 𝗩𝗮𝗹𝘂𝗲 (𝗖𝗟𝗩) 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝗶𝗼𝗻 𝗦𝘆𝘀𝘁𝗲𝗺. The starting question: 𝘩𝘰𝘸 𝘮𝘶𝘤𝘩 𝘳𝘦𝘷𝘦𝘯𝘶𝘦 𝘸𝘪𝘭𝘭 𝘦𝘢𝘤𝘩 𝘤𝘶𝘴𝘵𝘰𝘮𝘦𝘳 𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘦 𝘰𝘷𝘦𝘳 𝘵𝘩𝘦𝘪𝘳 𝘭𝘪𝘧𝘦𝘵𝘪𝘮𝘦 𝘪𝘯 𝘰𝘶𝘳 𝘣𝘶𝘴𝘪𝘯𝘦𝘴𝘴? Using the PostgreSQL DVD Rental dataset, I built an end-to-end pipeline: - Designed an ETL pipeline that processes ~14,000 transactions from 9 tables into a customer-level OLAP star schema - Engineered RFM-based features (Recency, Frequency, Monetary) for CLV modeling - Trained and compared multiple ML models (Linear Regression, Random Forest, Gradient Boosting) using chronological split and TimeSeriesSplit to avoid data leakage - Deployed everything into an interactive Django web app with a prediction form and business recommendations - The final model (Gradient Boosting) achieved strong performance, with R² close to 0.99 and low prediction error. One insight that came out of the analysis: customers who rent frequently, even at lower spend per transaction, often generate more lifetime value than occasional high spenders. Frequency matters more than monetary average! One limitation is that the dataset is static (historical DVD rental data), so the model reflects past behavior patterns rather than real-time customer activity. Additionally, some features like recency and tenure showed very low importance, likely due to the limited time range of the dataset, but they were still kept to ensures the model remains interpretable, aligned with business logic, and more generalizable to real-world scenarios beyond this dataset. This project helped me understand how data engineering, machine learning, and business thinking come together in a real system, not just a model. 🖇️GitHub → https://lnkd.in/g4k7iQuy Would love any feedback or thoughts!🖖🏻 #DataAnalytics #MachineLearning #Django #Python #PostgreSQL #PortfolioProject

Most data bugs I've seen in production didn't come from logic. They came from naming confusion 😬 . 🤔 Aliasing in SQL and Python is not syntax. It’s communication design. In production data work, clarity is not optional, it becomes technical debt very quickly. 💡 One small but consistent practice in well-structured codebases: using aliases wherever they add meaning. 👉 SQL example We use AS to align outputs with business understanding: SELECT salary AS monthly_salary FROM employees; Now the output speaks the business language, not just the database schema. 👉 Python example We standardize imports to reduce friction across teams: import pandas as pd import seaborn as sns Not just convenience, it improves shared understanding in collaborative codebases. Why it matters at scale: 🟢 Reduces ambiguity in pipelines and notebooks 🟢 Improves readability across teams 🟢 Enforces consistency in shared codebases 🟢 Improves long-term maintainability 💡 Senior takeaway: Clean data work is not about writing less code. It’s about writing code that disappears cognitively when someone reads it. #DataEngineering #SQL #Python #DataScience #CleanCode

𝗦𝗽𝗮𝗿𝗸 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗿𝘂𝗻 𝘆𝗼𝘂𝗿 𝗰𝗼𝗱𝗲. 𝗜𝘁 𝗯𝘂𝗶𝗹𝗱𝘀 𝗮 𝗽𝗹𝗮𝗻. A lot of Spark confusion comes from thinking it executes “line by line” like a normal program. In reality, Spark mostly does this: 𝗬𝗼𝘂𝗿 𝗰𝗼𝗱𝗲 -> 𝗹𝗼𝗴𝗶𝗰𝗮𝗹 𝗽𝗹𝗮𝗻 -> 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗹𝗼𝗴𝗶𝗰𝗮𝗹 𝗽𝗹𝗮𝗻 -> 𝗽𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗽𝗹𝗮𝗻 So when you write PySpark or Spark SQL, Spark isn’t “running Python” or “running SQL”. It’s building a plan for a distributed engine to execute. Here’s the simplified mental model I use: 𝟭) 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻𝘀 𝗯𝘂𝗶𝗹𝗱 𝘁𝗵𝗲 𝗽𝗹𝗮𝗻 (𝗹𝗮𝘇𝘆) select, filter, join, groupBy... These don’t immediately run a job. They describe what should happen. 𝟮) 𝗔𝗰𝘁𝗶𝗼𝗻𝘀 𝘁𝗿𝗶𝗴𝗴𝗲𝗿 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 count, show, collect, write… This is when Spark says: “ok, now I need to execute the plan”. 𝟯) 𝗧𝗵𝗲 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗿 𝗿𝗲𝘄𝗿𝗶𝘁𝗲𝘀 𝘆𝗼𝘂𝗿 𝘄𝗼𝗿𝗸 Before running, Spark tries to make it cheaper:   • push filters earlier   • prune unused columns   • reorder operations   • pick join strategies 𝟰) 𝗧𝗵𝗲 𝗽𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗽𝗹𝗮𝗻 𝗶𝘀 𝘄𝗵𝗮𝘁 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗿𝘂𝗻𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗰𝗹𝘂𝘀𝘁𝗲𝗿 This is where you’ll see the real cost drivers:   • join strategy (broadcast vs shuffle)   • number of stages/tasks   • shuffles, scans, exchanges   • partitioning decisions That’s why two bits of Spark code that look similar can behave completely differently. 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: If you can read the plan, you can explain most performance issues without guessing. Share your favourite Spark “aha” moment in the comments. #Spark #PySpark #SparkSQL #DataEngineering #BigData #Databricks #PerformanceTuning #SQL

Built Clarity because data teams were drowning in tools. One tool for SQL. Another for ETL. Another for dashboards. Another for reporting. None of them talk to each other. So we built one workspace that does it all — and made it AI-native from day one. → Full data lineage — trace every metric back to its source → Governed pipelines with audit trails and role-based access → A semantic layer your whole org trusts as the single source of truth → Query in SQL or plain English — every result is reproducible Full demo coming soon. Built with #Flutter #FastAPI #ClickHouse #Python #FlutterWeb #DataPlatform #Analytics #BuildInPublic #DataScience #SaaS #DataGovernance #RealTimeAnalytics #DataTransparency #DataQuality #TechStartup #DataOps #DataEngineering #DataDriven

🐍 Day 3/30 — Python for Data Engineers Dictionaries & Sets. The tools that make pipelines fast. Every Data Engineer works with dicts daily — whether parsing API responses, defining schemas, or managing configs. But here's the one that most beginners miss 👇 Sets are basically SQL operations: A & B → INNER JOIN (intersection) A | B → FULL OUTER JOIN (union) A - B → LEFT ANTI JOIN (difference) A ^ B → schema drift detector 🚨 That last one is genuinely useful in production: new_cols = incoming_cols - expected_cols # → {"total"} ← column you didn't expect. Alert! And remember: dict/set lookup is O(1) — hash table under the hood. List lookup is O(n) — it scans every element. On 10M rows, that difference is seconds vs milliseconds. 📌 Full cheat sheet in the image — methods, comprehensions, real DE patterns. Day 4 tomorrow: Functions & Lambda 🔧 What's your most-used dict method? .get() or .items()? Drop it below 👇 #Python #DataEngineering #30DaysOfPython #LearnPython #DataEngineer #SQL

🚀 Project Showcase : Personal Expense Tracker Web Application I’m excited to share one of my recent projects — a Personal Expense Tracker Web Application built using Python and Streamlit. This application helps users manage their finances by tracking expenses, monitoring budgets, and visualizing spending patterns through interactive dashboards. 💡 Key Features: ✅ Track daily expenses & income by category ✅ Interactive dashboard with Plotly charts & graphs ✅ Monthly spending heatmap ✅ Budget limits with progress tracking ✅ Recurring expense automation ✅ CSV import & PDF export ✅ Multi-user login system ✅ Deployed live on Streamlit Cloud 🛠️ Tech Stack: → Python | Streamlit → Supabase (PostgreSQL) → Plotly & Pandas for data visualization → ReportLab for PDF generation → GitHub for version control As data scientists, we often focus on models and insights — but building end-to-end data products is an equally important skill.I’m continuously learning and building projects in Data Science, AI, and Python development. 🔗 Live App: https://lnkd.in/dt3UDEDq 🔗 GitHub: https://lnkd.in/dgdw6_ME #DataScience #Python #Streamlit #Supabase #DataVisualization #Pandas #Plotly #BuildInPublic #PersonalFinance #DataScientist #OpenSource #ProjectShowcase