Master Pandas with 30 Essential Functions for 10X Faster Analysis

📊 What I Learned Today — Percentiles & Quantiles (Pandas) Today I fixed a confusion I had for a long time: 👉 Percentiles are NOT based on total sum — they’re based on position in sorted data. Key takeaways: 🔹 Quantile → value below which a % of data lies 🔹 Position formula: (n − 1) × q 🔹 Decimal position → interpolation 🔹 Result may not exist in dataset (and that’s okay) 💡 Example: Data → [10, 20, 30, 40] 75th percentile → position = (4−1)×0.75 = 2.25 So pandas doesn’t pick a value directly — it interpolates between 30 and 40 → 32.5 💡 Big insight: Even if the 75th percentile isn’t directly present, pandas computes it using values in between — not by summing anything. This cleared a major confusion: ❌ Percentage = sum-based ✅ Percentile = position-based Small concept, but a big clarity boost. Consistency > Perfection 🚀 #DataAnalytics #Pandas #Python #LearningJourney #InterviewPrep

📊 𝗖𝗵𝗲𝗰𝗸 𝗠𝗶𝘀𝘀𝗶𝗻𝗴 𝗩𝗮𝗹𝘂𝗲𝘀 𝗶𝗻 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 Before building any ML model, always check for missing values ❗ Ignoring them can lead to poor results 😬 🔍➤ 1) Check total missing values (count) df.isna().sum() ➡️ Shows missing count per column 📊 📉 ➤ 2) Missing values percentage (in %) (df.isna().sum() / len(df)) * 100 ➡️ Helps decide whether to drop 🗑️ or fill(Imputation) 🧩 📊 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗲 𝗠𝗶𝘀𝘀𝗶𝗻𝗴 𝗩𝗮𝗹𝘂𝗲𝘀 📌 ➤ 1) Bar Chart df.isna().sum().plot(kind='bar', figsize=(15,4)) 🔥 ➤ 2) Heatmap import seaborn as sns import matplotlib.pyplot as plt plt.figure(figsize=(12,6)) sns.heatmap(df.isna(), cbar=False) plt.title("Missing Value Heatmap") plt.show() 🎨 Dark color (almost black / blue) → Value is NOT missing ✅ (data is present) ⚪ Light / white color → Value IS missing ❌ (NaN) 📑 𝗦𝘂𝗺𝗺𝗮𝗿𝘆 𝗧𝗮𝗯𝗹𝗲 (Clean Report) missing_report = pd.DataFrame({ "missing_count": df.isna().sum(), "missing_pct": df.isna().mean() * 100 }).sort_values(by="missing_pct", ascending=False) missing_report 🚀 Clean Data = Better Models 💯 Always handle missing values before training! #DataScience #MachineLearning #Python #DataAnalysis #GitHub #LearningJourney

Day 19 — Merging & Joining Data in Pandas As I continue deepening my understanding of pandas, today’s focus was on something very practical: combining datasets. In real-world scenarios, data rarely comes in a single clean table. You often have multiple datasets that need to be brought together before any meaningful analysis can happen. That’s where pandas functions like merge(), join(), and concat() come in. Here’s a quick breakdown of what I learned: 🔹 merge() This is similar to SQL joins. It allows you to combine datasets based on a common column. You can perform: Inner joins Left joins Right joins Outer joins Example: pd.merge(df1, df2, on="id", how="inner") 🔹 join() Used mainly for combining DataFrames based on their index. It’s a bit more concise when working with indexed data. 🔹 concat() Used to stack DataFrames either: Vertically (adding more rows) Horizontally (adding more columns) Example: pd.concat([df1, df2], axis=0) 💡 Key Insight: Understanding when to use each method is crucial. Use merge() when working with relational data Use concat() when stacking data Use join() for index-based alignment This concept is especially important in data cleaning and preprocessing, where datasets often come from different sources. Each day, pandas feels less like a tool and more like a language for working with data. #M4aceLearningChallenge #Day19 #DataScience #MachineLearning #Python #Pandas #DataAnalysis

Numerous Methods to Accomplish a Task: Task: Order the cities columns on the basis of number of cells populated in descending order. In case of same number of cells populated, higher number suffix in Cities will take precedence like in case of Cities1 and Cities2, Cities2 will be listed first. Workbook Link: https://lnkd.in/ghGXkYrV Solution: Excel: =LET(a,A1:E19,SORTBY(a&"",-BYCOL(a,COUNTA),,TAKE(a,1),-1)) Power Query: let   Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],   Result = Table.ReorderColumns(Source, List.Sort(Table.ColumnNames(Source), {each -List.Count( List.RemoveNulls(Table.Column(Source,_))), {each _, 1}})) in   Result Python in Excel: df = xl("A1:E19", headers=True) Cols = sorted(df.columns, key = lambda x: (-len(df[x].dropna()), -df.columns.get_loc(x)) ) Result = pd.concat([df[i] for i in Cols], axis = 1).fillna('')

🐼 Pandas Cheat Sheet for Data Analysis Working with data? These Pandas functions save hours 👇 📌 Must Know: • Load → read_csv(), read_excel() • Explore → head(), info(), describe() • Clean → dropna(), fillna(), rename() • Filter → loc[], iloc[], query() • Aggregate → groupby(), pivot_table() • Combine → merge(), concat() 💡 Learn these once. Use them in every project. 💬 Which Pandas function do you use the most? 👇 #Python #Pandas #DataAnalytics #DataScience #Learning #Programming

🐼 Pandas Cheat Sheet for Data Analysis Working with data? These Pandas functions save hours 👇 📌 Must Know: • Load → read_csv(), read_excel() • Explore → head(), info(), describe() • Clean → dropna(), fillna(), rename() • Filter → loc[], iloc[], query() • Aggregate → groupby(), pivot_table() • Combine → merge(), concat() 💡 Learn these once. Use them in every project. 💬 Which Pandas function do you use the most? 👇 #Python #Pandas #DataAnalytics #DataScience #Learning #Programming

🐼 Pandas Cheat Sheet for Data Analysis Working with data? These Pandas functions save hours 👇 📌 Must Know: • Load → read_csv(), read_excel() • Explore → head(), info(), describe() • Clean → dropna(), fillna(), rename() • Filter → loc[], iloc[], query() • Aggregate → groupby(), pivot_table() • Combine → merge(), concat() 💡 Learn these once. Use them in every project. 💬 Which Pandas function do you use the most? 👇 #Python #Pandas #DataAnalytics #DataScience #Learning #Programming

Ever opened a dataset and thought… “why is this so messy?” 😅 Same here. While working with Pandas, I realized data cleaning isn’t complicated — it’s just a few powerful steps repeated smartly 👇 🧹 Missing values? → isna() to find them, fillna() or dropna() to handle them 🔁 Duplicate rows? → drop_duplicates() and move on 🔧 Wrong data types breaking your logic? → astype() fixes it in seconds 🧼 Messy text (extra spaces, weird formats)? → str.strip() and str.lower() clean it instantly 📊 Before trusting data? → info() and value_counts() give a quick reality check Good analysis starts with clean data first. That simple shift has already changed how I look at datasets. Still learning, but this is one of the most useful lessons so far. #DataAnalytics #Python #Pandas #DataCleaning #LearningJourney

Data management is all about understanding how to work with data and store it efficiently. In this piece, I explored some essential techniques in Pandas that make data handling more effective and reliable: ♦ Using sample() to extract random, reproducible subsets of data for analysis ♦ Understanding the difference between direct assignment and .copy() to avoid unintended changes to datasets ♦ Building Pivot Tables with .pivot_table() to transform raw data into meaningful insights One key takeaway: small decisions in data handling like whether or not to use .copy() when using pandas, can significantly impact the integrity of your analysis. #DataAnalysis #Python #Pandas #DataManagement #DataAnalytics #LearningInPublic

groupby() in Pandas Most beginners use Pandas but completely ignore groupby(). That’s a mistake. Because groupby() is where real data analysis starts. Think of it like this: You don’t just want data, you want insight by category. Example: - Average salary by department - Total sales by city - Count of customers by country That’s exactly what groupby() does. df.groupby("department")["salary"].mean() If you’re not using groupby properly, you’re not doing analysis you’re just looking at data. Learn how to use groupby effectively : Link in Comment #Python #Pandas #DataAnalysis #MachineLearning