Pivot Tables & Data Merge in Python with Pandas

🧠 Day 8 of 30 — Pandas: The Heart of  Data Analytics in Python If you want to work with data in Python, there is one library you cannot skip — Pandas. 🐼 Pandas lets you read, clean, analyse,  and manipulate data like Excel — but 100 times faster! Here are 5 must-know Pandas commands: 1️⃣ pd.read_csv()    Load any CSV file into a DataFrame 2️⃣ df.head()    Preview the first 5 rows of your data 3️⃣ df.describe()    Get instant stats — mean, max, min 4️⃣ df.dropna()    Remove rows with missing values 5️⃣ df.groupby()    Group and summarise data by category Quick real-world example: import pandas as pd df = pd.read_csv('sales_data.csv') df.groupby('city')['sales'].mean() Result? Average sales per city — in just 3 lines of code! 🚀 This is exactly what I use to  analyse data for my AI projects. Tomorrow → Day 9: Data Visualisation  with Matplotlib and Seaborn. Follow along — let us learn together! 🔥 Are you using Pandas in your projects? Drop a comment below! 👇 #Pandas #Python #DataAnalytics #LearnInPublic #Day8of30 #AI #MachineLearning #100DaysOfAI #ayyappanm #OpenToWork

Python Series – Day 22: Data Cleaning (Make Raw Data Useful!) Yesterday, we learned Pandas🐼 Today, let’s learn one of the most important real-world skills in Data Science: 👉 Data Cleaning 🧠 What is Data Cleaning Data Cleaning means fixing messy data before analysis. It includes: ✔️ Missing values ✔️ Duplicate rows ✔️ Wrong formats ✔️ Extra spaces ✔️ Incorrect values 📌 Clean data = Better results Why It Matters? Imagine this data: | Name | Age | | ---- | --- | | Ali | 22 | | Sara | NaN | | Ali | 22 | Problems: ❌ Missing value ❌ Duplicate row 💻 Example 1: Check Missing Values import pandas as pd df = pd.read_csv("data.csv") print(df.isnull().sum()) 👉 Shows missing values in each column. 💻 Example 2: Fill Missing Values df["Age"].fillna(df["Age"].mean(), inplace=True) 👉 Replaces missing Age with average value. 💻 Example 3: Remove Duplicates df.drop_duplicates(inplace=True) 💻 Example 4: Remove Extra Spaces df["Name"] = df["Name"].str.strip() 🎯 Why Data Cleaning is Important? ✔️ Better analysis ✔️ Better machine learning models ✔️ Accurate reports ✔️ Professional workflow ⚠️ Pro Tip 👉 Real projects spend more time cleaning data than modeling 🔥 One-Line Summary Data Cleaning = Convert messy data into useful data 📌 Tomorrow: Data Visualization (Matplotlib Basics) Follow me to master Python step-by-step 🚀 #Python #Pandas #DataCleaning #DataScience #DataAnalytics #Coding #MachineLearning #LearnPython #MustaqeemSiddiqui

📊 Today’s Learning: Mastering GroupBy in Python Pandas Continuing my journey in Data Analytics, today I explored one of the most powerful features in Pandas — GroupBy 🚀 🔹 What is GroupBy? GroupBy is used to split data into groups based on one or more columns, apply operations, and combine the results. It follows the Split → Apply → Combine concept. 🔹 Why is GroupBy important? ✔️ Helps summarize large datasets efficiently ✔️ Makes it easy to analyze patterns and trends ✔️ Essential for real-world data analysis tasks ✔️ Widely used in business reporting and dashboards 🔹 Common Operations with GroupBy: ✅ Sum, Mean, Count, Min, Max ✅ Multiple aggregations at once ✅ Grouping by multiple columns ✅ Filtering grouped data 🔹 Basic Syntax: df.groupby('column_name').agg({'column_name': 'function'}) 🔹 Examples: 👉 Total sales by category df.groupby('Category')['Sales'].sum() 👉 Average sales by region df.groupby('Region')['Sales'].mean() 👉 Multiple aggregations df.groupby('Category')['Sales'].agg(['sum', 'mean', 'count']) 👉 Grouping by multiple columns df.groupby(['Category', 'Region'])['Sales'].sum() 💡 Key Takeaway: GroupBy makes it simple to convert raw data into meaningful insights and is a core skill for any data analyst. 📈 Excited to apply this in real datasets and build more insights! #Python #Pandas #DataAnalytics #DataScience #LearningJourney #GroupBy #Analytics #DataSkills

When I started my data science journey, Python felt overwhelming. But honestly? You only need to master 3 core concepts to get started. 🐍 Here are the 3 Python concepts every data science beginner must know: ━━━━━━━━━━━━━━━━━━ 1. Pandas — Your data table tool ━━━━━━━━━━━━━━━━━━ Think of Pandas as Excel inside Python. It lets you load, clean, filter, and transform data in just a few lines. import pandas as pd df = pd.read_csv("data.csv") df.dropna(inplace=True)   # remove missing values df[df["age"] > 25]      # filter rows I used Pandas extensively in my Liver Failure Prediction project to clean 5000+ records from Kaggle. ━━━━━━━━━━━━━━━━━━ 2. NumPy — Your number crunching engine ━━━━━━━━━━━━━━━━━━ NumPy handles large arrays and mathematical operations at speed. It's the backbone behind Pandas, Scikit-learn, and almost every ML library. import numpy as np arr = np.array([10, 20, 30, 40]) print(arr.mean())  # 25.0 ━━━━━━━━━━━━━━━━━━ 3. Matplotlib — Your first visualisation tool ━━━━━━━━━━━━━━━━━━ Before Tableau or Power BI, Matplotlib helps you see your data right inside Python. import matplotlib.pyplot as plt plt.hist(df["age"], bins=10) plt.show() Why these 3 first? Because 80% of real data science work is cleaning, computing, and visualising data — before any ML model is even built. Master these and the rest becomes much easier. Are you learning Python for data science? Drop a comment — happy to share resources! 👇 #Python #DataScience #MachineLearning #Pandas #NumPy #Matplotlib #BeginnerTips #OpenToWork #DataAnalytics

📊 Beyond the Bell Curve: Handling "Messy" Data in Python As data scientists, we often dream of perfect, Gaussian (normal) distributions. But in the real world—especially with variables like car prices or housing data—the data is rarely "normal." I recently worked through a project involving Left-Skewed and Non-Parametric data. Here’s a breakdown of how I handled it using Python: 1️⃣ Identifying the Shape Before running any tests, I used Matplotlib to visualize the distribution. A high bin count (150) helped reveal a significant Left Skew, where the mean was being pulled down by a long tail of lower-priced entries. Python plt.hist(prices, bins=150) plt.show(); 2️⃣ The Transformation Strategy When data is left-skewed, standard parametric tests (like T-Tests) can become biased. To pull that "tail" back toward the center and achieve symmetry, I explored Square ($x^2$) and Cube ($x^3$) transformations. By stretching the right side of the distribution more than the left, these mathematical shifts can often "normalize" the data, allowing for more powerful statistical modeling. 3️⃣ When to Stay Non-Parametric If the data is truly "Non-Parametric" (multimodal or containing extreme gaps), forcing a transformation isn't the answer. In those cases, I pivot to Rank-Based tests like: ✅ Mann-Whitney U (instead of T-Test) ✅ Kruskal-Wallis (instead of ANOVA) ✅ Spearman’s Rank (instead of Pearson Correlation) The takeaway: Don't just import your library and hit "run." Understanding the geometry of your data is the difference between a biased model and an accurate insight. 💡 #DataScience #Python #Statistics #MachineLearning #Pandas #DataAnalytics #DataIntegrity

🧠 Day 9 of 30 — Data Visualisation: Matplotlib vs Seaborn Numbers alone do not tell a story. Charts do. 📊 Today I learned the two most powerful  Python libraries for Data Visualisation — Matplotlib and Seaborn. Here is the key difference: Matplotlib: → Full control over every detail → More code — more customisation → Best for precise, custom charts Seaborn: → Built on top of Matplotlib → Less code — beautiful by default → Best for statistical visualisations 5 charts every data analyst must know: 1️⃣ Bar Chart — Compare values across categories 2️⃣ Line Chart — Show trends over time 3️⃣ Scatter Plot — Find correlations in data 4️⃣ Heatmap — Spot patterns at a glance 5️⃣ Histogram — Understand data distribution The best part about Seaborn? A beautiful heatmap in just one line: sns.heatmap(df.corr(), annot=True,  cmap='coolwarm') That is it. One line. Production-ready chart. 🔥 Tomorrow → Day 10: SQL for Data Analytics — the skill every data professional needs. Follow along — let us learn together! 🚀 Which chart type do you use most? Drop a comment below! 👇 #DataVisualisation #Matplotlib #Seaborn #Python #LearnInPublic #Day9of30 #DataAnalytics #AI #100DaysOfAI #ayyappanm #OpenToWork

🚀 Data Cleaning & Exploratory Data Analysis (EDA) in Action Yesterday, I worked on cleaning and analyzing a real-world dataset using Python (Pandas, Matplotlib, Seaborn). Here’s a quick summary of what I explored: 🔹 Data Type Conversion Converted the Price column into numeric (float64) format, making it ready for analysis and calculations. 🔹 Descriptive Statistics Using df.describe(), I discovered: Most app ratings are between 4.0 – 4.5 App prices are mostly free, with a few outliers up to $400 Installs are highly skewed, with some apps reaching 1B+ downloads 🔹 Missing Values Analysis Found a total of 4,881 missing values Highest missing data in: Size (~15.6%) Rating (~13.6%) Other columns had minimal or no missing values 🔹 Data Quality Insights Detected outliers in Price and Rating Identified skewed distributions in Installs and Price Highlighted columns requiring data cleaning 🔹 Visualization Created a heatmap using Seaborn to visually identify missing values across the dataset 📊 💡 Key Learning: Before jumping into modeling, understanding your data through EDA and cleaning is critical. It helps uncover hidden patterns, errors, and insights that directly impact results. 🔥 More projects coming soon on my GitHub! Let’s connect and grow together in Data Analytics 🚀 #DataAnalytics #Python #Pandas #DataCleaning #EDA #Seaborn #Matplotlib #MachineLearning #DataScience

Day 4 — Python for Analytics When I started, I wasted weeks learning things I never used. Here are the 5 libraries that actually move the needle: 🐼 1. Pandas — The backbone of data analysis import pandas as pd df = pd.read_csv("sales_data.csv") top_product = (df.groupby("product")["revenue"]          .sum()          .sort_values(ascending=False)          .head(3)) print(top_product) If you learn nothing else — learn Pandas. 📊 2. Matplotlib / Seaborn — Turn numbers into stories Quick, beautiful charts with minimal code import seaborn as sns import matplotlib.pyplot as plt sns.lineplot(data=df, x="date", y="revenue") plt.title("Monthly Revenue Trend") plt.show() 🔢 3. NumPy — The engine under the hood Fast calculations on large datasets import numpy as np aov = np.mean(df["order_value"]) print(f"Average Order Value: ${aov:.2f}") 🤖 4. LangChain — Bridge between Python and LLMs Build GenAI workflows without starting from scratch from langchain_community.llms import OpenAI llm = OpenAI() response = llm("Summarize this sales report: ...") print(response) 📓 5. Jupyter Notebooks — Code + Story in one place Not just a coding tool — a communication format. Code → Output → Explanation → Chart All in one shareable document. Perfect for stakeholder walkthroughs. My honest learning path: Week 1 → Master Pandas Week 2 → Add Seaborn + Matplotlib Week 3 → Learn NumPy basics Week 4 → Explore LangChain Start with one. Build something real. Then add the next. #Python #Analytics #DataScience #Pandas #GenAI #30DayChallenge

Earlier, I used to think data analysis was all about dashboards, visualizations, and complex models. But while working with real datasets, I’ve realized something important — data preprocessing is where the real work happens. Most data is messy. It comes with missing values, inconsistent formats, duplicates, and sometimes even wrong entries. If we skip cleaning and preparing it properly, the final analysis can be completely misleading. Preprocessing may not look exciting, but it builds the foundation for everything that comes after — whether it’s analysis, visualization, or machine learning. I’m learning that even small steps like cleaning columns, handling missing data, or structuring information correctly can make a huge difference. In the end, it’s simple: Better data leads to better insights. #DataAnalytics #DataScience #LearningJourney #Python

🔶drop_duplicates() catches exact copies. But real data has a sneakier problem that it completely misses. Same person. Slightly different entry. Same Employee ID. Employee_ID: 10234 | Name: John Kamau | Dept: Sales Employee_ID: 10234 | Name: J. Kamau | Dept: sales Those look different enough that drop_duplicates() won’t touch them. But they’re the same person entered twice. Here’s how to catch it: # Find IDs appearing more than once duplicate_ids = df[ df.duplicated(subset=[“Employee_ID”], keep=False) ] print(f“Records with duplicate IDs: {len(duplicate_ids)}”) print(duplicate_ids.sort_values(“Employee_ID”).head(20)) 🔷This shows every row that shares an ID with another row. Now you can actually investigate instead of guessing. The fix depends on what you find: # Keep only the most recent entry per employee df = df.sort_values(“date_added”, ascending=False) df = df.drop_duplicates(subset=[“Employee_ID”], keep=“first”) Soft duplicates are dangerous for one reason: your analysis treats one person as two data points. Your model learns from the same person twice. Your headcount reports are wrong from the start. And none of it raises an error. Everything looks fine. 📍Check for duplicates by key columns, not just identical rows. That extra step catches what the default function misses. ❓Have you ever found soft duplicates in a dataset? What gave it away? #DataCleaning #Python #DataScience