Exploratory Data Analysis with Dr. Satyajit Pattnaik

Garbage In, Garbage Out: Cleaning Data with NumPy Day 54/100 A machine learning model is only as good as the data it's fed. For Day 54, I tackled the problem of Outlier Detection. In any real-world sensor system, glitches happen. If you don't filter them, they skew your averages and ruin your predictions. I implemented a Z-Score Filter using strictly NumPy to automatically identify and remove statistical anomalies. Technical Highlights: 📐 Standardization: Calculating Mean ($\mu$) and Standard Deviation ($\sigma$) to measure data spread. 🔢 Z-Score Implementation: Determining how many standard deviations a data point is from the mean to assess its 'normality.' 🛡️ Boolean Masking : Using absolute value thresholds to programmatically 'clean' a dataset in a single vectorized step. The Professional Insight: Data Cleaning accounts for nearly 80% of a Data Scientist's work. Building these filters manually in NumPy gives me a fundamental understanding of data distribution that a library can't teach. Do check my GitHub repository here : https://lnkd.in/d9Yi9ZsC #NumPy #DataScience #100DaysOfCode #BTech #AIML #Statistics #DataCleaning #Python #SoftwareEngineering #LearningInPublic

Hidden truth about Missing values 👇 Your model isn't underperforming. It’s being lied to. 🛑 One of the most dangerous mistakes in Data Science? Treating 0 as a value when it's actually a void. In many datasets: 0 ≠ actual value Sometimes, 0 actually means: ➡ data not recorded ➡ patient skipped test ➡ value unavailable In a recent Diabetes Dataset project: ✔ Identified hidden zeros ✔ Converted them into NaN ✔ Handled missing values properly ✔ Model performance improved significantly That’s when I realised: Data cleaning isn’t boring. It’s where real Data Science begins. Before building any model, always ask: “Does this value make logical sense?” Because better data → better insights → better decisions. #DataCleaning #Python #EDA #MachineLearning #DataAnalytics #DataScienceJourney #StudentToDataScientist

Day 110 – Data Science Learning Journey Today I continued yesterday’s article and learned about Interquartile Range (IQR), Percentiles, and Quartiles — important concepts in statistics for understanding data distribution and detecting outliers. Key Learnings: • IQR = Q3 − Q1 • Helps measure data spread • Used in box plots to detect outliers • Percentiles divide data into 100 parts • Quartiles divide data into 4 parts Understanding these concepts is very useful for data analysis, data cleaning, and visualization. Statistics is truly the backbone of Data Science, and I’m continuing to strengthen my fundamentals step by step. #DataScience #Statistics #LearningJourney #DataAnalytics #Python #MachineLearning #Day110

Day 40 of my Data Engineering journey 🚀 Today I went deeper into data filtering, sorting, and aggregation using Pandas. 📘 What I learned today (Pandas Filtering & Aggregation): • Filtering rows using conditions • Combining multiple conditions • Sorting values with sort_values() • Selecting specific columns • Grouping data using groupby() • Applying aggregate functions (sum, mean, count) • Understanding how Pandas handles missing values • Writing cleaner transformation logic Pandas feels like SQL inside Python but more flexible. Instead of just querying data, I’m now transforming it programmatically. This is real data manipulation. Why I’m learning in public: • To stay consistent • To build accountability • To improve daily Day 40 done ✅ Next up: data cleaning & handling missing values in Pandas 💪 #DataEngineering #Python #Pandas #LearningInPublic #BigData #CareerGrowth #Consistency

Pandas in Practice: How to Survive a Data Analysis Test 🐼📊 I consider today's Pandas test closed! ✅ Although starting to work with data frames can be challenging, today I managed to go through the full data preparation process: Cleaning and Selection: Selecting key variables and removing duplicates (the number of rows has been reduced to 30 for greater clarity!). Missing Management: Efficiently filling NA values with the arithmetic mean using fillna(). Statistical Analysis: Calculating the correlation matrix for independent variables. Visualization: Creating boxplots and histograms with groups (e.g., the relationship between fuel consumption and the number of cylinders). Nothing teaches you as much as working with a living organism (in this case, the mtcars dataset). Python and Pandas are powerful tools, and today marks another step towards effortless data analysis in the Jupyter Notebook environment. Thank you for your support and keeping your fingers crossed! 🚀 #Python #Pandas #DataAnalysis #Jupyter #MachineLearning #DataScience #StudentLife

Started the analytical workflow by focusing on Data immersion and wrangling, building the foundation for all later analysis. The first step was understanding the dataset from both technical and business perspectives before moving into deeper exploration. 1. Created a detailed data dictionary covering variable definitions, data types, and business relevance. 2. Performed initial profiling to identify missing values, duplicates, inconsistent formats, and outliers. 3. Standardized important fields such as dates, time values, and categorical variables. 4. Prepared a clean dataset ready for downstream analysis. GitHub Link : https://lnkd.in/guaN2xNT #DataAnalytics #DataScience #Python #Pandas #DataCleaning #DataWrangling

Strengthening Data Science Foundations – Day 41 (Applied) Today’s work focused on data cleansing and insight generation using the dummy dataset. -> The dataset was processed using Pandas to handle real-world data quality challenges. -> Missing values were examined and addressed through different approaches, including filling missing values, ignoring incomplete records where appropriate, and applying logical rules to maintain dataset integrity. -> After cleaning the dataset, exploratory analysis techniques were applied to derive initial insights regarding user behavior patterns, engagement indicators, and lifestyle attributes reflected in the data. This exercise reinforced the importance of data preparation as a foundational step in data science, since meaningful insights depend heavily on clean, reliable datasets. #DataScience #Pandas #DataCleaning #ExploratoryDataAnalysis #Kaggle #RealWorldData #AppliedLearning #Python #ContinuousLearning

Most people use Random Forest only for prediction. But one of its most powerful use cases? 👉 Feature Importance for Exploratory Data Analysis (EDA). Instead of relying only on correlation heatmaps, Random Forest helps: • Capture non-linear relationships • Handle multicollinearity • Rank features based on real predictive power • Reveal hidden drivers in your dataset In my recent credit risk analysis project, this approach helped identify the most influential variable contributing nearly 47% of the model’s decision power. Sometimes, modeling isn’t just about prediction , it’s about understanding what truly drives outcomes. #Python #DataAnalytics #MachineLearning #RandomForest #EDA #FeatureEngineering #DataScience #PowerBI #SQL #AnalyticsProjects #AspiringDataAnalyst #BusinessIntelligence #Sklearn #Excel

Beyond the "Equals" Sign: Mastering the Broadcast Nested Loop Join! ♾️🔍 On Day 38 of my 365-day Spark journey, I’m exploring how Spark handles complex logic like Range Joins and Inequality Joins. Standard joins are great for ID-to-ID matching, but real-world data is often messier. When you need to join based on a "between" condition or a "greater than" threshold, the Broadcast Nested Loop Join (BNLJ) is Spark's secret weapon. Key Takeaways: ✅ Non-Equi Joins: It’s the go-to strategy when you can’t use a simple = sign in your join condition. ✅ Small Table Requirement: Because this uses a nested loop (O(N \times M) complexity), broadcasting the small table is essential to prevent the cluster from grinding to a halt. ✅ Execution Plan: Always check your .explain()—if you see BNLJ on two large tables, beware! It’s time to rethink your logic. Spark optimization isn't just about speed; it's about choosing the right tool for the right logic! 🛠️ #DataEngineering #ApacheSpark #PySpark #BigData #Algorithm #Performance #Python #100DaysOfCode #Day38 #DataScience #CloudComputing

📊 Applying NumPy on Real Data — Learning Beyond Basics 🚀 After understanding NumPy fundamentals, I practiced working with a small dataset to explore how numerical data can be analyzed efficiently. Instead of just creating arrays, I worked on analyzing structured data using NumPy operations. 🔹 What I practiced: • Creating 2D arrays (dataset structure) • Calculating total and average values • Finding maximum and minimum values • Accessing specific rows and columns • Performing vectorized operations This practice helped me understand how data is structured and analyzed before moving to advanced tools like Pandas. Building strong foundations step by step in Data Analytics. 📈 Next goal: Start learning Pandas and work with larger datasets. Open to feedback and connections in Data & Tech. #DataAnalytics #NumPy #Python #DataScienceJourney #AnalyticsSkills #ContinuousLearning #AspiringDataAnalyst