Data Engineering Journey: Pandas Filtering & Aggregation

Day 110 – Data Science Learning Journey Today I continued yesterday’s article and learned about Interquartile Range (IQR), Percentiles, and Quartiles — important concepts in statistics for understanding data distribution and detecting outliers. Key Learnings: • IQR = Q3 − Q1 • Helps measure data spread • Used in box plots to detect outliers • Percentiles divide data into 100 parts • Quartiles divide data into 4 parts Understanding these concepts is very useful for data analysis, data cleaning, and visualization. Statistics is truly the backbone of Data Science, and I’m continuing to strengthen my fundamentals step by step. #DataScience #Statistics #LearningJourney #DataAnalytics #Python #MachineLearning #Day110

🚀 Day 11/70 – NumPy Array Operations & Indexing Today I went deeper into NumPy 📊 After learning the basics yesterday, today I explored array operations and indexing — which are very important in real data analysis. 📌 Array Indexing import numpy as np arr = np.array([10, 20, 30, 40, 50]) print(arr[0]) # First element print(arr[-1]) # Last element 📌 Slicing Arrays print(arr[1:4]) # Elements from index 1 to 3 Slicing helps in selecting specific parts of data. 📌 Mathematical Operations print(np.sum(arr)) # Sum of elements print(np.mean(arr)) # Average print(np.max(arr)) # Maximum value print(np.min(arr)) # Minimum value These operations are used frequently in data analysis. 📌 2D Array (Matrix) matrix = np.array([[1, 2, 3], [4, 5, 6]]) print(matrix) print(matrix.shape) # Rows & Columns Understanding 2D arrays is important because real datasets are structured in rows and columns. Today’s Key Learning: NumPy makes data manipulation faster, cleaner, and more efficient compared to traditional Python lists. 11 Days of Consistency 💪 Step by step toward becoming a Data Analyst. #Day11 #NumPy #Python #DataAnalytics #LearningInPublic #FutureDataAnalyst #70DaysChallenge

Today I learned about three important statistical concepts in Data Analytics 📊🐍 🔹 Mean (Average) The sum of all values divided by the number of values 🔹 Median (Middle Value) The middle value when data is sorted 🔹 Mode (Most Frequent Value) The value that appears most often Example in Pandas: df["Sales"].mean() df["Sales"].median() df["Sales"].mode() 💡 Important Insight: • Mean is affected by outliers • Median is more stable for skewed data • Mode is useful for categorical data Understanding these basics helps in better data interpretation and decision making. Learning step by step and strengthening my foundation in Data Analytics 🚀 #Python #Pandas #DataAnalytics #Statistics #LearningJourney

Started the analytical workflow by focusing on Data immersion and wrangling, building the foundation for all later analysis. The first step was understanding the dataset from both technical and business perspectives before moving into deeper exploration. 1. Created a detailed data dictionary covering variable definitions, data types, and business relevance. 2. Performed initial profiling to identify missing values, duplicates, inconsistent formats, and outliers. 3. Standardized important fields such as dates, time values, and categorical variables. 4. Prepared a clean dataset ready for downstream analysis. GitHub Link : https://lnkd.in/guaN2xNT #DataAnalytics #DataScience #Python #Pandas #DataCleaning #DataWrangling

Day 42 of my Data Engineering journey 🚀 Today I learned how to merge and join datasets using Pandas a core skill when working with multiple data sources. 📘 What I learned today (Merging & Joining in Pandas): • Combining datasets using merge() • Understanding inner, left, right, and outer joins • Joining datasets based on keys • Using concat() to stack datasets • Handling duplicate columns after merges • Aligning data from different sources • Thinking about relational data in Python • Understanding how this mirrors SQL joins Most real-world data lives in multiple tables or files. Learning how to merge them correctly is essential for building reliable pipelines. SQL joins tables. Pandas merges datasets. Same concept different tool. Why I’m learning in public: • To stay consistent • To build accountability • To improve daily Day 42 done ✅ Next up: data transformation & feature engineering with Pandas 💪 #DataEngineering #Python #Pandas #LearningInPublic #BigData #CareerGrowth #Consistency

Headline: 🛠️ 80% of data science is data cleaning. Here is how I tackle it. I just published a new project on GitHub: The Customer Data Cleaning Pipeline. Raw data is rarely "model ready." To bridge that gap, I built a comprehensive pre-processing workflow in Python that transforms noisy, inconsistent records into high-quality data for business intelligence. The Pipeline Highlights: Data Integrity: Evaluated and fixed missing values using advanced imputation. Standardization: Uniformed categories and corrected inconsistent data formats. Feature Engineering: Implemented Data Normalization, Binning, and created Indicator (Dummy) Variables. Visualization: Developed bin distribution charts to validate data segments. You can run the entire cleaning process directly in your browser via the "Open in Colab" link in my repo! Check out the project below in the comments: #DataCleaning #Python #Pandas #DataScience #DataQuality #OpenSource #GitHub #Numpy #Matplotlib

📘 Data Science Journey | Day 24 🔥 Day 49 of my #100daysofcodechallenge Today I started learning Data Collection Techniques in my Data Science journey. Here’s what I covered today: 📌 Introduction to Data Collection ▫ Understanding how data is gathered from different sources ▫ Importance of data collection in the data science pipeline 📌 Introduction to Web Scraping ▫ Extracting data from websites automatically ▫ Real-world use cases like price tracking, news data, and research 📌 Basics of HTML for Web Scraping ▫ Understanding structure of a webpage using HTML tags ▫ Learning key elements ▫ Importance of class and id attributes for targeting data 👉 See you tomorrow for Day 50. #DataScience #Python #WebScraping #HTML #DataCollection #LearningJourney #Consistency #CodeWithHarry #100daysofcode

Just finished exploring Pandas—and it’s amazing how powerful it is for data work 🚀 From understanding core structures like Series (1D) and DataFrames (2D) to handling missing values, indexing, and performing fast, vectorized operations—Pandas truly feels like a blend of SQL + Excel + Python in one place. What stood out the most? 👉 Clean data manipulation 👉 Efficient analysis workflows 👉 Ability to turn raw data into insights quickly If you're stepping into data analytics or data science, mastering Pandas is a game changer. #Python #Pandas #DataAnalytics #DataScience #LearningJourney

🚀 Day 2 of My Data Analytics / ML Journey Today I explored the fundamentals of Pandas, one of the most powerful Python libraries for data analysis. Here’s what I built 👇 ✅ Created a structured DataFrame (like an Excel table) ✅ Added a new subject column dynamically ✅ Calculated Total and Average marks ✅ Implemented Grade logic (A, B, C, D) ✅ Built Pass/Fail system using functions 💡 Key Learning: Writing code that works is not enough — writing code that is scalable and dynamic is what makes you industry-ready. Instead of hardcoding values, I used a subjects list and applied operations across columns — just like real-world datasets. 📊 Tools Used: Python 🐍 | Pandas | Logical Thinking 🎯 This is just the beginning — next I’ll be working on: ➡️ Data filtering (like SQL) ➡️ Sorting & ranking systems ➡️ Real-world datasets #DataAnalytics #Python #Pandas #MachineLearning #LearningInPublic #100DaysOfCode #DataScienceJourney

"Code Every Day": Python journey with Data Science (Day 103) Today was another productive day in my Machine Learning journey, where I explored the concept of Ridge Regression. I learned that Ridge Regression is a regularization technique (L2 regularization) used to reduce overfitting in linear models. • It works by adding a penalty term to the cost function, which discourages large coefficient values. • I studied the Ridge Regression formula • I understood the role of lambda (2): • Small ^ → model behaves like normal linear regression Large 1 → coeffi-ients shrink more, reducing model complexity • Large 1 → more shrinkage of coefficients I also analyzed the graphical representation, understanding how Ridge Regression smooths the model and reduces variance compared to normal regression • Overall, today helped me understand how to balance bias and variance using regularization techniques in machine learning. #100DaysOfPython #PythonJourney #LearnInPublic #CodeEveryday #PythonForDataScience #sheryianscodingschool