Mohamed Boughattas’ Post

A question I had when starting out: should I use Pandas or SQL for data transformation? Here's how I now think about it: Use SQL when: → Data lives in a database or warehouse → The dataset is large (millions of rows) → You need joins across multiple tables → You want the transformation to run server-side Use Pandas when: → Data is in files (CSV, Excel, JSON) → You need complex Python logic → You're doing exploratory analysis → The dataset fits comfortably in memory In data engineering, you'll use both. SQL for the heavy lifting, Pandas for the finishing touches. What's your go-to for data transformation? #Python #Pandas #SQL #DataEngineering

There are two ways to traverse hierarchies in SQL. Only one scales 👇 Recursive CTEs and self-joins solve the same problem: navigating hierarchical data. But they behave very differently as the data grows. Recursive CTEs let you define a single rule and let SQL iterate through the hierarchy until it reaches the end. No need to know the depth upfront. You also don’t need to keep adjusting the query every time the hierarchy changes, which makes it much more scalable in real-world systems. With recursive CTEs, the query adapts to the data. With self-joins, the query is fixed to the structure you assumed. For Python folks: think of recursive CTEs like a WHILE loop over a tree structure, with a termination condition to avoid infinite recursion. Got other SQL topics you want explained like this? Comment them 👇 📌Found it useful? Save it for later. #SQLTips #DataAnalytics #DataScience #SQL #Analytics #BusinessIntelligence #DataEngineer #LearnSQL

Behind every great business decision is a data engineer no one talks about. 🔧 They don't just move data — they build the infrastructure that makes insight possible. Here's what a modern data pipeline actually does: → Ingest: Pull raw data from APIs, databases, files → Transform: Clean, validate, enrich with SQL & Python → Warehouse: Store efficiently for fast querying → Visualize: Deliver truth to decision-makers via dashboards No reliable pipeline = no reliable decisions. #DataEngineering #DataEngineer #SQL #Python #PySpark #ETL #Databricks #PowerBI #DataPipeline #DataAnalytic #TechCareer #DataScience #BigData 

SQL vs PySpark vs Pandas cheat sheet If you’re working in Data Engineering or switching between tools on the fly during projects/interviews, this can save you a lot of time. 📌 What’s included: 13 structured sections 70+ commonly used concepts SELECT, JOINs, CTEs, Window Functions Aggregations, Date & String operations, Pivot Read/Write patterns + data quality checks Everything is shown side-by-side across SQL, PySpark, and Pandas, so you don’t have to keep searching for syntax differences every time. 💡 The idea is simple — faster recall, fewer mistakes, and more confidence in interviews and real projects. If you want the PDF, just drop a comment — I’ll share it for free. Feel free to repost if it helps someone in your network 👍 #DataEngineering #SQL #PySpark #Pandas #Python #BigData #DataEngineer #InterviewPrep #CheatSheet

🔄 From Pandas to PySpark — One Cheat Sheet to Rule Them All! Navigating between different data tools can be overwhelming, especially when switching between Pandas, Polars, SQL, and PySpark. This handy comparison simplifies everyday data operations like: ✔ Reading data ✔ Filtering & sorting ✔ Joins & aggregations ✔ Handling missing values ✔ Grouping & transformations 💡 Whether you're a beginner in data analytics or transitioning into big data tools, understanding these parallels helps you: Learn faster 🚀 Work smarter 💡 Adapt across technologies 🔁 In today’s data-driven world, flexibility across tools is a superpower! 📌 Save this for quick reference and level up your data skills. #DataAnalytics #DataScience #Python #Pandas #PySpark #SQL #Polars #BigData #DataEngineering #Learning #CareerGrowth #AnalyticsJourney #DataTools

Streamline Your Data Cleaning Workflow! 📊 Navigating data cleaning can be a challenge, but having the right tools at your fingertips makes all the difference. I came across this fantastic cheat sheet that compares SQL and Python methods for common data cleaning tasks, and I wanted to share it with my network! This side-by-side comparison covers: Missing Values: Efficiently finding and replacing them. Duplicates: Identifying and removing redundant data. Data Types & Formatting: Ensuring your data is in the correct format, including handling dates and text. Outliers (IQR): A clear method for detecting and managing outliers using the Interquartile Range. Whether you're a seasoned data professional or just starting out, this cheat sheet is a valuable resource for your next messy dataset. What are your go-to data cleaning techniques? Share your tips in the comments below! 👇 #DataCleaning #SQL #Python #DataScience #DataAnalysis #CheatSheet #BigData #DataManagement

🚀 Built an End-to-End Data Pipeline using API & SQL Server! Excited to share my recent hands-on project where I built a complete data pipeline from scratch 👇 🔹 What I did: 1. Source Database (SQL Server)   ↓ 2. Create API using FastAPI   ↓ 3. Expose endpoint (/data)   ↓ 4. Call API using Python (requests)   ↓ 5. Get data in JSON format   ↓ 6. Connect to Target SQL Server   ↓ 7. Auto-create table (if not exists)   ↓ 8. Insert data into target table   ↓ 9. Verify data in SSMS 🔹 Tech Stack: Python | FastAPI | SQL Server | pyodbc | requests 🔹 Key Learnings: 💡 How APIs act as a bridge between systems   💡 Converting JSON data into structured format   💡 Building real-world ETL pipelines   💡 Automating data movement without manual intervention  This project helped me understand how real-world data engineering pipelines work — from data extraction to loading 🚀 Looking forward to building more such projects and improving my skills! #DataEngineering #Python #FastAPI #SQLServer #ETL #DataPipeline #LearningInPublic #100DaysOfData #BuildingInPublic

🐍 Day 3/30 — Python for Data Engineers Dictionaries & Sets. The tools that make pipelines fast. Every Data Engineer works with dicts daily — whether parsing API responses, defining schemas, or managing configs. But here's the one that most beginners miss 👇 Sets are basically SQL operations: A & B → INNER JOIN (intersection) A | B → FULL OUTER JOIN (union) A - B → LEFT ANTI JOIN (difference) A ^ B → schema drift detector 🚨 That last one is genuinely useful in production: new_cols = incoming_cols - expected_cols # → {"total"} ← column you didn't expect. Alert! And remember: dict/set lookup is O(1) — hash table under the hood. List lookup is O(n) — it scans every element. On 10M rows, that difference is seconds vs milliseconds. 📌 Full cheat sheet in the image — methods, comprehensions, real DE patterns. Day 4 tomorrow: Functions & Lambda 🔧 What's your most-used dict method? .get() or .items()? Drop it below 👇 #Python #DataEngineering #30DaysOfPython #LearnPython #DataEngineer #SQL

𝗦𝘁𝗼𝗽 𝘂𝘀𝗶𝗻𝗴 .𝗰𝗼𝘂𝗻𝘁() to check whether a PySpark 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲 𝗶𝘀 𝗲𝗺𝗽𝘁𝘆. It looks harmless, but it can 𝗾𝘂𝗶𝗲𝘁𝗹𝘆 𝘀𝗹𝗼𝘄 𝗱𝗼𝘄𝗻 𝘆𝗼𝘂𝗿 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲. When you call `.count()`, Spark may scan the full DataFrame across all partitions just to answer a simple yes/no question. That means unnecessary compute, longer runtimes, and higher cost for something that should be lightweight. A small change can make a big difference on #spark jobs: Use a minimal read approach like `df.take(1)` when you only need to know whether data exists. Tiny optimization. Real impact. See the visual below for a quick comparison 👇 The best performance wins are often the simplest ones. Repost if this saved you a future debugging session. #PySpark #Databricks #DataEngineering #ApacheSpark #sql #Databricks #ETL #Python #DeltaLake #Spark #AzureDataEngineer #BigData #DataPipelines #SparkOptimization #DataArchitecture #InterviewQuestions #DataEngineer #Azure #CloudData #TableOptimization #DataPerformance #ProductionReady #DataCommunity #CareerTips

SQL or pandas, the tool is secondary. 💡 The logic is what matters. A classic use case: employees earning above their department average. 👉 SQL ,using a CTE: WITH avg_salary AS ( SELECT department, AVG(salary) AS dept_avg FROM employees GROUP BY department ) SELECT e.name, e.salary, a.dept_avg FROM employees e JOIN avg_salary a ON e.department = a.department WHERE e.salary > a.dept_avg; 👉 pandas, same logic: avg_salary = ( employees .groupby("department")["salary"] .mean() .reset_index(name="dept_avg") ) result = employees.merge(avg_salary, on="department") result = result[result["salary"] > result["dept_avg"]] ###Same pattern. Different syntax. 🟢 aggregate by group 🟢 join back to original dataset 🟢 filter using group-level context This is what defines data work across tools. Not memorizing syntax but recognizing reusable patterns. 😊 Master the logic. The syntax will follow. #SQL #Python #Pandas #DataEngineering #DataScience