Detecting and Treating Outliers in Python Data Analysis

 Day 2 of Learning Python – And I Just Built My First Real Data Audit System 📊🐍 Today I didn’t just “learn Python”… I used it to analyze structured company-style audit data and built a Mistake Scoring System that automatically evaluates performance. And honestly, It felt like stepping into real business intelligence work. 💡 What I built today: Using Pandas, I processed an audit dataset and generated insights like: 📌 Total deals per responsible person 📌 Pipeline distribution per team member 📌 Mistake scoring based on missing actions (follow-ups, updates, documents) 📌 Final performance summary ranking everyone by errors ⚙️ The idea behind the system: Instead of manually checking performance, I created a logic-based scoring system where: Missing documents = +1 error No follow-up = +1 error No comment update = +1 error Unresolved status = +3 heavy penalty This turns raw data into actionable performance insights. 💻 Code I used: import pandas as pd file_path = r " Instered your excel data file here" Note: The r before the file path means it is a raw string, which helps Python correctly read the path without treating backslashes as escape characters. Also, make sure your Excel file is saved in the same folder where your Python script is located, or ensure the correct full file path is provided. df = pd.read_excel(file_path) # CLEAN DATA df.columns = df.columns.str.strip() df = df.fillna("No") # MISTAKE SCORE SYSTEM df["Mistake Score"] = 0 df.loc[df["Document/RF Request"] == "No", "Mistake Score"] += 1 df.loc[df["Comment Updates"] == "No", "Mistake Score"] += 1 df.loc[df["Follow up"] == "No", "Mistake Score"] += 1 df.loc[df["Status"].str.lower() == "unresolved", "Mistake Score"] += 3 # ANALYSIS print(df["Responsible"].value_counts()) print(df.groupby(["Responsible", "Pipeline"]).size()) mistakes = df.groupby("Responsible")["Mistake Score"].sum().sort_values(ascending=False) print(mistakes) summary = df.groupby("Responsible").agg( Total_Deals=("Responsible", "count"), Total_Mistakes=("Mistake Score", "sum") ) print(summary.sort_values("Total_Mistakes", ascending=False)) 🚀 Key takeaway: Even simple Python + Excel data can be transformed into a decision-making system that highlights performance gaps instantly. Day 2 of learning — and I’m already seeing how powerful data can be in real business environments. Can’t wait to build dashboards and automate even more next 🔥 #Python #DataAnalysis #Pandas #LearningInPublic #DataScience #Automation #BusinessIntelligence #CareerGrowth

🐍 Data Types & Type Casting in Python (Small Concept, Big Impact) When working with data in Python, one mistake beginners often make is ignoring data types. And trust me, this small thing can break your entire analysis. When you load a dataset in Python, it doesn't always read your data the way you expect. A column full of numbers might be stored as text. A date column might be treated as a random string. A true/false column might come in as an object. And if you don't fix this early, your entire analysis will give you wrong results. 🔹 So What Are Data Types? Every value in Python has a type - it tells Python what kind of data this is and what you can do with it. The most common ones in data analysis: int → Whole numbers → 25, 100, -5 float → Decimal numbers → 3.14, 99.9, -0.5 str → Text → "John", "Mumbai", "Yes" bool → True or False → True, False datetime → Dates & times → 2024-01-15 👉 Think of data types as the language your data speaks, If you misunderstand it, your analysis goes wrong. 🔹 Why Data Types Matter in Data Analysis Because Python behaves differently based on data types. Example: 👉 "100" + "20" → "10020" (string concatenation) 👉 100 + 20 → 120 (numeric addition) Same values. Different result. 🔹 A Simple Real-Life Example Imagine a salary column in your dataset. You try to calculate the average: df['salary'].mean() But Python throws an error. You check the data type and you see - salary is stored as object (string), not a number. Python literally can't do math on it. That's where Type Casting comes in. 🔹 What is Type Casting? Type casting means converting one data type into another. Your salary column is stored as "50000" (a string). Every calculation you run will give wrong results or fail completely. After type casting: # Convert salary column to number df['salary'] = df['salary'].astype(float) # Now calculate average salary df['salary'].mean() # works perfectly # Convert joining date to datetime df['join_date'] = pd.to_datetime(df['join_date']) # Convert employment status to boolean df['is_active'] = df['is_active'].astype(bool) Now Python understands your data — and you can calculate average salaries, find top earners, compare departments, and build models correctly. 🔹 Why This Matters in Real Projects Wrong data types silently break your analysis. - Calculations fail on string columns - Sorting dates goes wrong if stored as text - Visualizations won't plot numeric data stored as objects - Machine learning models reject incorrect types completely Checking and fixing data types is not optional — it is one of the first things a professional analyst does. 🔹 When Should You Always Check Data Types? ✔ Right after loading your dataset ✔ Before doing any salary calculations ✔ During data cleaning df.dtypes # check all column types at once One wrong data type = one wrong insight. And in salary analysis, one wrong insight can mislead an entire business decision. #DataAnalytics #Python #DataTypes #TypeCasting #pandas

🚀 Strings & String Methods in Python #Day31 If variables are containers, strings are how Python stores and handles text data. Names, emails, passwords, customer data, file paths, web scraping, data cleaning — strings are everywhere. 🔹 What is a String? A string is a sequence of characters enclosed in quotes. name = "Harry" city = 'Delhi' Both single and double quotes work the same. Strings can contain: ✅ Letters ✅ Numbers (as text) ✅ Symbols ✅ Spaces "Python" "12345" "Hello @2026" 🔹 Multiline Strings Use triple quotes for text spanning multiple lines: message = """This is a multi line string""" Useful for documentation, SQL queries, or long messages. 🔹 String Indexing Each character has a position (index). text = "Python" P y t h o n 0 1 2 3 4 5 print(text[0]) # P print(text[3]) # h ⚡ Indexing starts from 0. Python also supports negative indexing: text[-1] # n text[-2] # o Very useful when working from the end of a string. ✂️ String Slicing Slicing extracts a portion of a string. text[0:3] # Pyt text[2:] # thon text[:4] # Pyth Negative slicing: text[-3:] # hon Powerful and widely used in data manipulation. 🔹 len() Function Find the length of a string: len("Python") Output: 6 Even spaces are counted. 🛠 Common String Methods 1. lower() and upper() "PYTHON".lower() "python".upper() Useful for standardizing text. 2. strip() Removes extra spaces: " hello ".strip() Great for cleaning raw data. 3. replace() "Hello World".replace("World","Python") Output: Hello Python 4. split() Turns a string into a list: "apple,banana,orange".split(",") Used heavily in data parsing. 5. join() Opposite of split: ",".join(["apple","banana","orange"]) 6. find() Find position of text: "Hello World".find("World") Returns index or -1 if not found. 7. startswith() and endswith() email.endswith(".com") email.startswith("test") Very useful in validation. 🔍 Checking String Content isalpha() isdigit() isalnum() Examples: "Python".isalpha() "123".isdigit() "Python123".isalnum() Useful for validation logic. 🔄 Strings Are Immutable Important concept: text="Python" text[0]="J" ❌ Error Strings cannot be modified directly. Any change creates a new string. 💡 Why Strings Matter in Data Analytics Strings are everywhere in analytics: 📌 Cleaning messy datasets 📌 Working with CSV files 📌 Parsing emails & text 📌 Filtering data 📌 Web scraping 📌 Text analysis Mastering strings makes data cleaning much easier. Python strings may look simple, but they’re one of the most powerful tools in programming. #Python #PythonProgramming #DataAnalytics #PowerBI #Excel #MicrosoftPowerBI #MicrosoftExcel #DataAnalysis #DataAnalysts #CodeWithHarry #DataVisualization #DataCollection #DataCleaning

Day 4 of My Data Analyst Journey – Data Cleaning in Python Today, I practiced data cleaning techniques using Python, focusing on handling real-world messy text data. Problem Statement: I had a dataset of customer feedback containing: • Extra spaces • Mixed casing (UPPER/lower) • Punctuation (., !, ?) Objective: Clean and standardize the feedback text for better analysis. What I implemented: Removed punctuation using .replace() Converted text to lowercase Removed leading & trailing spaces using .strip() Handled lists inside a dictionary Python Code: import string feedback_data = { 'S_No': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'Name': ['Ravi', 'Meera', 'Sam', 'Anu', 'Raj', 'Divya', 'Arjun', 'Kiran', 'Leela', 'Nisha'], 'Feedback': [ ' Very GOOD Service!!!', 'poor support, not happy ', 'GREAT experience! will come again.', 'okay okay...', ' not BAD', 'Excellent care, excellent staff!', 'good food and good ambience!', 'Poor response and poor handling of issue', 'Satisfied. But could be better.', 'Good support... quick service.' ], 'Rating': [5, 2, 5, 3, 2, 5, 4, 1, 3, 4] } punctuation = ".,!?" cleaned_feedbackdata = {} for key, value in feedback_data.items(): if isinstance(value, list): new_list = [] for item in value: if isinstance(item, str): item = item.strip().lower() for p in punctuation: item = item.replace(p, "") new_list.append(item) cleaned_feedbackdata[key] = new_list else: cleaned_feedbackdata[key] = value print(cleaned_feedbackdata) Outcome: Cleaned and structured feedback data ready for analysis like sentiment detection, keyword extraction, and insights generation. Key Learning: Data cleaning is one of the most important steps in data analysis—clean data = better insights! #Python #DataCleaning #DataAnalytics #LearningJourney #BeginnerToPro #CodingPractice #100DaysOfCode

𝗖𝗮𝗻 𝗦𝗤𝗟 𝗱𝗼 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗮𝗻𝗮𝗹𝘆𝘀𝗶𝘀? We usually do feature analysis in Python, but what if we cannot load millions of rows in Python? Can we do that with SQL? To figure this out, I took the problem of customer churn and tried to understand why customers are leaving and what we can do about it. For this, I tried to understand the behavior of churned customers across the different groups of each feature. For example, does a high number of support calls lead to churning? To study customer behavior, I calculated the churn rates across the groups of each feature using AVG() in SQL. I used churn rate because it allows comparison irrespective of group size. For calculating the churn rate for numerical features like payment delay, I first divided this feature into groups using GROUP BY in SQL. I did this by identifying the sudden difference in churn rates between two values. Consequently, I identified the thresholds of behavioral change and labeled the groups using a CASE conditional statement. For categorical features, it can be easily calculated. To decide which feature is important, I used this criteria: 1. The churn rate difference must be significant for at least one group compared to others. This suggests that after this threshold is the breaking point of customer behavior. 2. The pattern should be stable, to avoid random noise. 3. Group sizes should be comparable. Example: Issue Level (Support Calls) +------------------+------------------+ | Issue Level | Churn Rate | +------------------+------------------+ | Low      | 0.10     | | Medium    | 0.25     | | High     | 0.80     | +-------------------+-----------------+ Churn rate stays stable across low and medium but increases sharply at high issue level. Customers waited patiently until the support calls were in the medium issue level. Once the threshold is crossed, 80% of the customers leave. That means one should respond to support calls before reaching the high issue level; otherwise, the customer will leave. In customer churn, the features are: Age, Gender, Tenure, Usage Frequency, Support Calls, Payment Delay, Subscription Type, Contract Length, Total Spend, Last Interaction, and Churn. For more detailed analysis, check out github repo (Notebooks/SQL_Analysis folder): https://lnkd.in/gUx9vgyE #SQL #FeatureAnalysis #CustomerChurn #DataAnalytics #DataScience #SQLAnalytics #ChurnAnalysis #DataEngineering #BehavioralAnalysis #AnalyticsEngineering #BigData #DataCommunity

🚀 Day 7 of My Python Learning Journey | String Methods | Business Analyst Aspirant Continuing my Python journey to strengthen my skills for a Business Analyst role 📊 Today, I worked on String Methods in Python, which are extremely useful for data cleaning, transformation, and preprocessing — key tasks in real-world analytics. 💻 Topic: String Methods in Python # Remove spaces text1 = " hello python learners " print("Clean text:", text1.strip()) # Upper & Lower case print("Upper:", text1.upper().strip()) print("Lower:", text1.lower().strip()) # Replace text print("Replace:", text1.replace("python", "SQL").strip()) # Count occurrences print("Count of 'o':", text1.count("o")) # Check start print("Starts with hello:", text1.strip().startswith("hello")) # Check numeric mobile = "9876543210" print("Is numeric:", mobile.isnumeric()) # Split & Join msg = "Welcome to python Course" words = msg.split() print("Words list:", words) joined_text = "_".join(words) print("Joined text:", joined_text) # Find position print("Index of 'p':", msg.find("p")) # Extract domain email = "student@example.com" domain = email[email.find("@") + 1:] print("Domain:", domain) # Data Cleaning Example (Price) price_text = "Price : ₹3500/-" clean_price = price_text.replace("Price :", "")\ .replace("₹", "")\ .replace("/-", "")\ .strip() print("Clean price:", clean_price) 💡 Key Learnings: Cleaned raw text data using strip() and replace() Transformed text using upper(), lower(), split(), and join() Extracted useful information (like email domain) Practiced real-world data cleaning (price formatting) 📌 These skills are directly applicable in: ✔ Data Cleaning ✔ Excel / SQL transformations ✔ Power BI datasets I’m learning Python through Satish Dhawale sir course (SkillCourse) and practicing daily 💻 🔥 Next step: Applying these concepts on real datasets and analytics projects Let’s connect if you're also learning Python or Data Analytics 🤝 #Python #StringMethods #DataCleaning #BusinessAnalyst #DataAnalytics #LearningJourney #SkillDevelopment #SatishDhawale #SkillCourse #UpGrad

I have started a python series to learn the knowledge of it. 𝐏𝐲𝐭𝐡𝐨𝐧 𝐃𝐒𝐀 𝐃𝐚𝐲 𝟏 - 𝐏𝐲𝐭𝐡𝐨𝐧 𝐁𝐚𝐬𝐢𝐜𝐬 𝐜𝐨𝐯𝐞𝐫𝐢𝐧𝐠. ✅ 𝐃𝐚𝐲 𝟏: 𝐏𝐲𝐭𝐡𝐨𝐧 𝐁𝐚𝐬𝐢𝐜𝐬 𝗧𝗼𝗽𝗶𝗰𝘀 𝗖𝗼𝘃𝗲𝗿𝗲𝗱: 1. Variables 2. Data Types 3. Input/Output (I/O) 4. Operators 5. Basic Programs:   * Calculator   * Swapping Numbers 1️⃣ 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞𝐬 * Definition: A variable stores data that can change during program execution. * Syntax: x = 10 name = "Alice" * Python is dynamically typed, meaning you don’t need to declare the data type. 2️⃣ 𝗗𝗮𝘁𝗮 𝗧𝘆𝗽𝗲𝘀 Common data types in Python: int     10 float   10.5 str   "Hello" bool   True/False list   [1, 2, 3] tuple   (1, 2) dict   {"a": 1} 3️⃣ 𝗜𝗻𝗽𝘂𝘁 𝗮𝗻𝗱 𝗢𝘂𝘁𝗽𝘂𝘁 (𝗜/𝗢) * Input from user: name = input("Enter your name: ") * Convert input to int/float: age = int(input("Enter your age: ")) * Print Output: print("Hello,", name) 4️⃣ 𝗢𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 Arithmetic   + - * / % // ** Comparison   == != > < >= <= Logical     and or not Assignment   = += -= *= /= 5️⃣ 𝗕𝗮𝘀𝗶𝗰 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝘀 🔹 Calculator a = float(input("Enter first number: ")) b = float(input("Enter second number: ")) print("Sum =", a + b) print("Difference =", a - b) print("Product =", a * b) print("Quotient =", a / b) 🔹 Swap Numbers (with temp) x = 5 y = 10 temp = x x = y y = temp print("x =", x) print("y =", y) 🔹 Swap Numbers (without temp) x = 5 y = 10 x, y = y, x print("x =", x) print("y =", y) ✅ 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲 𝗧𝗶𝗽𝘀 * Use `input()` to take values and `print()` to display them. * Play around with different data types. * Try basic math programs on your own. ➣ 𝗦𝘂𝗴𝗴𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗮𝗻𝗱 𝗖𝗼𝗿𝗿𝗲𝗰𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝘄𝗲𝗹𝗰𝗼𝗺𝗲𝘀 ♻️ Share 👍🏻 React 💭 Comment ☑️ Repost ✴️ Follow Rishabh Bhat for more. Guided by Ratan Kumar jha #Python #Coding #Programming #DataScience #MachineLearning #AI #WebDevelopment #PythonProgramming #Automation #Tech #LearnPython #PythonForBeginners #SoftwareDevelopment #Developers #CodeNewbie #PythonProjects #TechEducation #Scripting #PythonCommunity

My Data Science Journey — Python Tuple, Set, Dictionary & the Collections Library Today’s focus was on Python’s core data structures — Tuples, Sets, and Dictionaries — along with the powerful collections module that enhances their functionality for real-world use cases. 𝐖𝐡𝐚𝐭 𝐈 𝐋𝐞𝐚𝐫𝐧𝐞𝐝: Tuple – Ordered, immutable, allows duplicates – Single element tuples require a trailing comma → ("cat",) – Supports packing and unpacking → x, y = 10, 30 – Cannot be modified after creation (TypeError by design) – Faster than lists in certain operations – Used in scenarios like geographic coordinates and fixed records – Can be used as dictionary keys (unlike lists) Set – Unordered, mutable, stores unique elements only – No indexing or slicing support – Empty set must be created using set() ({} creates a dict) – .remove() raises KeyError if element not found – .discard() removes safely without error – Supports operations like union, intersection, difference, symmetric_difference – Methods like issubset(), issuperset(), isdisjoint() help in set comparisons – frozenset provides an immutable version of a set – Offers O(1) average time complexity for membership checks Dictionary – Key-value pair structure, ordered, mutable, and keys must be unique – Built on hash tables for fast lookups – user["key"] → raises KeyError if missing – user.get("key", default) → safe access with fallback – Methods: keys(), values(), items() for iteration – pop(), popitem(), update(), clear(), del for modifications – Widely used in real-world data like APIs and JSON responses – Common pattern: list of dictionaries for structured datasets Collections Library – namedtuple → tuple with named fields for better readability – deque → efficient queue with O(1) operations on both ends – ChainMap → combines multiple dictionaries without merging copies – OrderedDict → maintains order with additional utilities like move_to_end() – UserDict, UserList, UserString → useful for customizing built-in behaviors with validation and extensions Performance Insight – List → O(n) – Tuple → O(n) – Set → O(1) (average lookup) – Dictionary → O(1) (average lookup) 𝐊𝐞𝐲 𝐈𝐧𝐬𝐢𝐠𝐡𝐭: Understanding when to use each data structure — and how collections enhances them — is crucial for writing efficient, scalable, and clean Python code. Read the full breakdown with examples on Medium 👇 https://lnkd.in/gvv5ZBDM #DataScienceJourney #Python #Tuple #Set #Dictionary #Collections #Programming #DataStructures

📊 Handling Missing Values in Python - The First Real Data Problem You’ll Face You’ve loaded your dataset. Everything looks fine… until you notice this: 👉 Some values are missing. Blank cells. NaN values. Incomplete records. And here’s the truth: 👉 Almost every real-world dataset has missing data. 🔹 What Are Missing Values? Missing values are simply gaps in your dataset — places where data should exist but doesn't. In Python, they usually appear as: NaN # Not a Number — most common in pandas None # Python's version of empty 🔹 Why Do Missing Values Matter? Because they can silently break your analysis. ❌ Wrong averages ❌ Incorrect insights ❌ Errors in calculations ❌ Poor model performance 👉 Ignoring missing data = trusting wrong results 🔹 Simple Real-Life Example Imagine you’re analyzing employee salaries. Some entries are missing. Now if you calculate average salary: 👉 Your result will be misleading But once you handle missing values properly: 👉 Your analysis becomes accurate and reliable 🔹 How to Detect Missing Values In Python, it’s very simple: df.isnull().sum() 👉 This shows how many values are missing in each column 🔹 How to Handle Missing Values There is no “one right way”—it depends on the situation. But commonly, analysts use: ✔ Remove missing data df.dropna() ✔ Fill with mean (for numerical data) df['salary'] = df['salary'].fillna(df['salary'].mean()) ✔ Fill with mode (for categorical data) df['city'] = df['city'].fillna(df['city'].mode()[0]) ✔ Forward fill (for time-based data) df.fillna(method='ffill') 🔹 One Rule to Always Remember Missing % What to DoLess than 5% Safely drop the rows 5% to 30% Fill with mean, median or mode More than 30% Investigate — the column may be unreliable 🔹 When Should You Handle Missing Values? Always: ✔ Right after loading your dataset ✔ Before doing calculations ✔ Before building any model 👉 Cleaning comes before analysis. 🚀 Final Thought Dirty data is not the problem. Not knowing how to clean it — that is. Every professional dataset has missing values. What separates a good analyst from a great one is knowing exactly how to handle them. 💡 #DataAnalytics #Python #MissingValues #DataCleaning #pandas #DataAnalyst #LearningInPublic #PythonForData #AnalyticsJourney #DataScience

🚀 Variables & Data Types in Python #Day27 If you're starting your Python journey, understanding Variables and Data Types is your first big step toward becoming a pro 💡 Let’s break it down in a simple and practical way 👇 🔹What is a Variable? A variable is like a container 🧺 that stores data values which you can use and update in your program. 👉 Example: name = "Ishu" age = 20 Here, name stores text and age stores a number. 🔹Rules for Naming Variables 📝 ✔ Start with a letter or underscore _ ✔ Cannot start with a number ❌ ✔ Use only letters, numbers, and underscores ✔ Case-sensitive (age ≠ Age) 👉 Best Practice: student_name = "Rahul" total_marks = 95 🔹Dynamic Typing in Python 🔄 Python automatically understands the type of data — no need to declare it! x = 10 # Integer x = "Hello" # Now String 🔥 This flexibility makes Python beginner-friendly. 🔹What are Data Types? 📊 Data types define the type of value a variable can store. 🔸1. Numeric Types 🔢 Used for numbers: a = 10 # int b = 3.14 # float c = 2 + 3j # complex 🔸2. String (str) 🔤 Used for text: name = "Python" msg = 'Hello World' 🔸3. Boolean (bool) ⚖️ Represents True or False: is_logged_in = True 🔸4. List 📋 (Ordered & Mutable) Can store multiple values: fruits = ["apple", "banana", "mango"] ✔ Changeable (mutable) 🔸5. Tuple 📦 (Ordered & Immutable) Similar to list but cannot be changed: point = (10, 20) ✔ Faster than lists ❌ Cannot modify 🔸6. Set 🔗 (Unordered & Unique) Stores unique values only: numbers = {1, 2, 3, 3} 👉 Output: {1, 2, 3} 🔸7. Dictionary 📚 (Key-Value Pair) Stores data in pairs: student = {"name": "Ishu", "age": 20} 🔹Type Checking 🔍 Check the type of any variable: print(type(name)) 🔹Type Conversion 🔄 Convert one data type to another: x = int("10") y = str(20) z = float(5) 🔹Multiple Variable Assignment ⚡ a, b, c = 1, 2, 3 x = y = z = 0 🔹Constants in Python 🔒 Python doesn’t have fixed constants, but we follow naming convention: PI = 3.14 MAX_LIMIT = 100 💡Pro Tips: ✔ Use meaningful variable names ✔ Follow snake_case naming style ✔ Keep your code readable and clean ✔ Understand data types deeply to avoid bugs 🔥Conclusion Variables store your data, and data types define what kind of data you're working with. Master these fundamentals, and you’ll unlock the true power of Python 💪 💬 What’s your favorite Python data type? #DataAnalytics #DataAnalysts #DataAnalysis #Excel #PowerBI #PythonProgramming #Python #MicrosoftExcel #MicrosoftPowerBI #DataCleaning #DataCollection #DataVisualization #SQL #CodeWithHarry #Variables #DataTypes #learningJourney #Learning #Consistency