Cleaning Pandas Data with isna(), dropna(), and More

Real-world data is messy. And that’s where I started understanding Pandas better 👇 While practicing, I noticed something: Data is rarely clean. You’ll find: - missing values - inconsistent formats - unwanted columns So I tried a simple example: 👉 Dataset with student marks Some values were missing Using Pandas, I: - identified missing values - filled them with default values - removed unnecessary data What I realized: Data cleaning is not just a step… 👉 it’s the foundation of any data workflow Even the best analysis fails if the data is not clean. Now I’m focusing more on: - handling missing data - making datasets usable Because clean data = better results If you're learning Pandas, don’t just read… try cleaning a messy dataset That’s where real learning happens. What’s the most common issue you’ve seen in datasets? #Pandas #DataCleaning #Python #DataEngineering #DataScience #CodingJourney #TechLearning

🚀 Day 70 – String Methods in Pandas Today’s learning was all about String Manipulation in Pandas — a powerful skill when working with messy real-world data! 🧹📊 🔹 String Methods in Pandas Explored how to clean and transform text data using functions like: .str.lower() / .str.upper() .str.strip() .str.replace() .str.contains() These methods make it easy to standardize and analyze textual data efficiently. 🔹 Detecting Mixed Data Types Real-world datasets often contain inconsistent data types in the same column. Learned how to: Identify mixed types Use astype() and to_numeric() to fix them Ensure data consistency for better analysis 💡 Key Takeaway: Clean and well-structured data is the foundation of accurate insights. String manipulation plays a crucial role in making data analysis reliable and effective. 📈 Step by step, getting closer to becoming a better Data Analyst! #Day70 #DataScience #Pandas #Python #DataCleaning #DataAnalytics

Today, I stepped deeper into data analysis by working with Pandas which is a powerful library for handling structured data. I learned how to: 🔹 Create and explore DataFrames 🔹 Select and filter data 🔹 Perform basic data inspection 🔹 Understand how datasets are structured for analysis My key insight is that before building any machine learning model, you must first understand your data and Pandas makes that process much easier and more efficient. This session made me realize that data analysis is not just about numbers, but about extracting meaningful insights from structured information. I'm excited to keep building! #Python #Pandas #DataAnalysis #MachineLearning #M4ACE

Starting to understand why Pandas is the first tool every data scientist learns. ● I built a simple Student Marks Analyzer — nothing fancy, but it clicked something for me. With just a few lines I could: → Build a table from scratch → Explore rows, columns, specific values → Get average, highest and lowest marks instantly ● Average: 84.0 | Highest: 95 | Lowest: 70 The interesting part? I didn't write a single formula. No Excel. No manual counting. Just Python doing the heavy lifting in milliseconds. This is exactly what data analysis feels like at the start — small project, but you can already see the power behind it. Still a lot to learn. But this one felt good. 🐼 ●  Code is on my GitHub — link in the first comment. #Python #Pandas #DataScience #MachineLearning #AI #100DaysOfCode #PakistanTech

🔷A simple train test split is not always enough. I learned this the hard way when my model looked great on paper and struggled on real data. 📌Here is what nobody tells you about splitting data properly. The basic split gives you two sets. Training and testing. That works for simple projects. But what if you need to tune your model? You test different settings, pick the best one, and evaluate on the test set. The problem is that you have now indirectly used the test set to make decisions. It is no longer a fair judge. This is where a three way split becomes important. 🔹X_train, X_temp, y_train, y_temp = train_test_split( X, y, test_size=0.3, random_state=42 ) 🔹X_val, X_test, y_val, y_test = train_test_split( X_temp, y_temp, test_size=0.5, random_state=42 ) Now you have three sets. Training set. The model learns here. 70 percent of your data. Validation set. You tune and compare models here. 15 percent. Test set. You evaluate the final model here. Once. Never again. 15 percent. The test set is sacred. You look at it exactly one time at the very end. One more thing that most people miss. Always stratify your split when your target column is imbalanced. 🔹train_test_split(X, y, stratify=y, test_size=0.2) stratify=y makes sure both sets have the same proportion of each class. Without it you might end up with a training set that barely sees the minority class and a model that has no idea it exists. The split is not a formality. It is a decision that shapes every result that follows. Get it right before you touch anything else. ❓What split ratio do you use for your projects and why? #DataScience #MachineLearning #Python

Everyone talks about learning more tools. But the real shift happens when you start building with what you already know. Lately, I’ve been focusing on: • Writing better SQL to extract meaningful data • Using Python to automate repetitive tasks • Improving data quality through validation checks Not chasing everything — just getting better at the fundamentals. Because in the end: 👉 It’s not about doing more. It’s about creating more value. Still learning. Still building. #Python #SQL #Automation #DataEngineering #Analytics #Learning

Unpopular opinion: You don’t need 10 tools to work in data. You need 3 — and you need to use them well. • SQL → to actually understand your data • Python → to process and automate it • Thinking → to solve the right problem Everything else is optional. Most of the time, the issue isn’t lack of tools — it’s lack of clarity. Lately, I’ve been focusing more on mastering the basics, improving data quality, and automating repetitive workflows instead of chasing every new tool. Still learning — but this shift has made a real difference. #DataEngineering #SQL #Python #Automation #Learning

Hot take: Most people aren’t struggling with data science because it’s “too hard” they’re learning it in the wrong order. They start with: Python → Libraries → Models When they should start with: Problem → Data → Decisions → THEN tools. Here’s the reality: • A simple model with a clear problem beats a complex model with no direction • Understanding your data is more important than memorizing algorithms • Metrics matter more than model complexity • Business/context thinking beats tool proficiency Data science is less about using models and more about solving problems with data. If you can clearly define the problem, understand the data, and choose the right approach the tools become easy. #DataScience #MachineLearning #DataAnalytics #ProblemSolving

Ever noticed how much time goes into just handling files and data every day? I was stuck in a loop — opening multiple Excel files, cleaning data, fixing formats, updating sheets, and repeating the same steps daily. Easily 1.5–2 hours gone. Then one simple thought hit me — what if this entire flow could run on its own? So I built a automation using: 1. Python 2. Pandas (for data handling) 3. Openpyxl (for working with Excel files) Built-in tools like datetime, pathlib, and logging for structure and tracking Now, what used to take hours runs in just a few minutes. More than saving time, it made me realize — a lot of “routine work” is just an automation waiting to happen. Still learning, but definitely seeing work differently now. #Python #Automation #DataAnalytics #Learning

Data management is all about understanding how to work with data and store it efficiently. In this piece, I explored some essential techniques in Pandas that make data handling more effective and reliable: ♦ Using sample() to extract random, reproducible subsets of data for analysis ♦ Understanding the difference between direct assignment and .copy() to avoid unintended changes to datasets ♦ Building Pivot Tables with .pivot_table() to transform raw data into meaningful insights One key takeaway: small decisions in data handling like whether or not to use .copy() when using pandas, can significantly impact the integrity of your analysis. #DataAnalysis #Python #Pandas #DataManagement #DataAnalytics #LearningInPublic