Cleaning Data with Python: The Key to Meaningful Insights

Anti-hot take: Python and SQL aren’t going anywhere. Even with AI. In fact, if you’re a data professional, they’re more valuable now than they were two years ago. 📈 The current narrative is that "natural language is the new programming language" and we’ll all just prompt our way to a dashboard. That sounds great in a pitch deck, but anyone who actually works with messy, real-world data knows the reality. AI is an incredible co-pilot, but it’s a dangerous captain. When an LLM spits out 50 lines of code, you aren't just a "user"—you are the Editor-in-Chief. If you don't actually know the syntax, you're just copy-pasting your way toward a logic error. Here is why the fundamentals matter more now than ever: 🔹 The "Looks Right" Trap AI is a master of the "hallucination"—writing code that is syntactically perfect but logically catastrophic. Without a deep understanding of SQL or Python, it’s nearly impossible to spot the subtle error that doubles a revenue metric or incorrectly handles a null value. 🔹 Debugging is 80% of the Job AI excels at the "happy path." But business data is never happy. It’s siloed, inconsistent, and poorly labeled. When a script breaks because of a schema change, "prompting harder" won't fix it. You have to be able to go under the hood yourself. 🔹 The Cost of Inefficiency An AI can write a query that "works." It can also write a query that scans 10TB of data and spikes your compute costs because it used a nested loop instead of a proper join. You need to know the fundamentals to optimize for scale. 🔹 AI doesn't know your business An LLM doesn’t know why "Active User" means something different in your warehouse than it does in a textbook. Python and SQL are the tools you use to bake your specific company logic into the data. AI can't guess your internal definitions. The bottom line? We’re moving from a world of writing from scratch to a world of auditing and verifying. Python and SQL remain the foundation. AI is the accelerator, NOT the foundation. If you can’t audit the code the AI gives you, you can’t trust the results. And in data science, if you can’t trust the data, the work is worthless. Stop asking if AI will replace these skills. Start using AI to master them faster. 💡

Precisely! 👌🏻💯 From my experience, many people of my generation fail to be convinced that AI is not flawless and perfect, especially when it comes to programming languages. I sometimes hear colleagues say things like "you only need to tell it what to do and it'll cook" or "learning programming is not useful anymore", but I always argue that they are making a horrible mistake that would eventually leave them lagging far behind the curve.

✅ *Python Interview Questions with Answers* *1. How do you handle missing data in Pandas?* Use `df.isnull().sum()` to detect, then `df.fillna(value)` or `df.dropna()` to handle. For forward/backward fill: `df.fillna(method='ffill')` or `df.interpolate()`. *2. What is the difference between loc[] and iloc[]?* - `loc[]`: label‑based indexing (e.g., `df.loc['row_label', 'col_name']`). - `iloc[]`: position‑based (integer) indexing (e.g., `df.iloc[0, 1]` for first row, second column). *3. What are lambda functions in data analysis?* Anonymous one‑line functions: `lambda x: x*2`. Used in `apply()`, `map()`, `filter()` for quick transformations, like `df['col'].apply(lambda x: x.upper())`. *4. How do you remove duplicates from DataFrame?* `df.drop_duplicates(subset=['col1', 'col2'], keep='first')`. Reset index after if needed: `df.drop_duplicates().reset_index(drop=True)`. *5. Explain groupby() and agg().* `groupby()` splits data into groups: `df.groupby('category')`. `agg()` applies multiple functions: `df.groupby('category').agg({'sales': ['sum', 'mean'], 'profit': 'max'})`. *6. How do you merge/join DataFrames?* `pd.merge(df1, df2, on='key', how='inner/left/right/outer')` or `df1.join(df2, on='key')`. For multiple keys: `on=['key1', 'key2']`. *7. What is vectorization?* Performing operations on entire arrays/DataFrames without loops (e.g., `df['col'] * 2` vs loops). Uses NumPy under the hood for speed; avoid `apply()` for simple math. *8. How do you handle outliers using IQR method?* ```python Q1 = df['col'].quantile(0.25) Q3 = df['col'].quantile(0.75) IQR = Q3 - Q1 df = df[(df['col'] >= Q1 - 1.5*IQR) & (df['col'] <= Q3 + 1.5*IQR)] ``` *9. What is the difference between list, tuple, dict?* - List `[]`: mutable, ordered. - Tuple `()`: immutable, ordered. - Dict `{}`: mutable, key‑value pairs, preserves insertion order (Python 3.7+). *10. How do you pivot data with pivot_table()?* `pd.pivot_table(df, values='sales', index='category', columns='region', aggfunc='sum', fill_value=0)`. *11. What libraries do you use for viz (Matplotlib/Seaborn)?* - Matplotlib: base plotting (`plt.plot()`, `plt.bar()`). - Seaborn: high‑level stats viz on top of Matplotlib (`sns.scatterplot()`, `sns.heatmap()`). *12. Explain apply() vs map() vs applymap().* - `df.apply(func)`: row/column‑wise (Series‑level functions). - `Series.map(func)`: element‑wise on a Series. - `df.applymap(func)`: element‑wise on entire DataFrame (older style; today you’d often use `map()` on elements). *13. How do you read CSV with chunks?* ```python for chunk in pd.read_csv('file.csv', chunksize=10000): process(chunk) ``` This lets you process large files without loading everything into memory. *14. What is NumPy broadcasting?* NumPy automatically expands arrays of different shapes for element‑wise operations (e.g., `arr + 5` adds 5 to every element, or adding a 1D array to each row of a 2D array).

✅ *Python Interview Questions with Answers* *1. How do you handle missing data in Pandas?* Use `df.isnull().sum()` to detect, then `df.fillna(value)` or `df.dropna()` to handle. For forward/backward fill: `df.fillna(method='ffill')` or `df.interpolate()`. *2. What is the difference between loc[] and iloc[]?* - `loc[]`: label‑based indexing (e.g., `df.loc['row_label', 'col_name']`). - `iloc[]`: position‑based (integer) indexing (e.g., `df.iloc[0, 1]` for first row, second column). *3. What are lambda functions in data analysis?* Anonymous one‑line functions: `lambda x: x*2`. Used in `apply()`, `map()`, `filter()` for quick transformations, like `df['col'].apply(lambda x: x.upper())`. *4. How do you remove duplicates from DataFrame?* `df.drop_duplicates(subset=['col1', 'col2'], keep='first')`. Reset index after if needed: `df.drop_duplicates().reset_index(drop=True)`. *5. Explain groupby() and agg().* `groupby()` splits data into groups: `df.groupby('category')`. `agg()` applies multiple functions: `df.groupby('category').agg({'sales': ['sum', 'mean'], 'profit': 'max'})`. *6. How do you merge/join DataFrames?* `pd.merge(df1, df2, on='key', how='inner/left/right/outer')` or `df1.join(df2, on='key')`. For multiple keys: `on=['key1', 'key2']`. *7. What is vectorization?* Performing operations on entire arrays/DataFrames without loops (e.g., `df['col'] * 2` vs loops). Uses NumPy under the hood for speed; avoid `apply()` for simple math. *8. How do you handle outliers using IQR method?* ```python Q1 = df['col'].quantile(0.25) Q3 = df['col'].quantile(0.75) IQR = Q3 - Q1 df = df[(df['col'] >= Q1 - 1.5*IQR) & (df['col'] <= Q3 + 1.5*IQR)] ``` *9. What is the difference between list, tuple, dict?* - List `[]`: mutable, ordered. - Tuple `()`: immutable, ordered. - Dict `{}`: mutable, key‑value pairs, preserves insertion order (Python 3.7+). *10. How do you pivot data with pivot_table()?* `pd.pivot_table(df, values='sales', index='category', columns='region', aggfunc='sum', fill_value=0)`. *11. What libraries do you use for viz (Matplotlib/Seaborn)?* - Matplotlib: base plotting (`plt.plot()`, `plt.bar()`). - Seaborn: high‑level stats viz on top of Matplotlib (`sns.scatterplot()`, `sns.heatmap()`). *12. Explain apply() vs map() vs applymap().* - `df.apply(func)`: row/column‑wise (Series‑level functions). - `Series.map(func)`: element‑wise on a Series. - `df.applymap(func)`: element‑wise on entire DataFrame (older style; today you’d often use `map()` on elements). *13. How do you read CSV with chunks?* ```python for chunk in pd.read_csv('file.csv', chunksize=10000): process(chunk) ``` This lets you process large files without loading everything into memory. *14. What is NumPy broadcasting?* NumPy automatically expands arrays of different shapes for element‑wise operations (e.g., `arr + 5` adds 5 to every element, or adding a 1D array to each row of a 2D array).

How can video data be transformed into structured data suitable for analysis? Transforming video into structured data for analysis with Snowflake #Python. There are several approaches depending on what you want to extract: 1️⃣. 🇲🇪🇹🇦🇩🇦🇹🇦 🇪🇽🇹🇷🇦🇨🇹🇮🇴🇳 Duration, resolution, FPS, codec, file size Libraries: ffmpeg-python, moviepy, opencv-python 2️⃣. 🇫🇷🇦🇲🇪 🇪🇽🇹🇷🇦🇨🇹🇮🇴🇳 (🇮🇲🇦🇬🇪 🇩🇦🇹🇦) Extract frames as images at intervals Convert to pixel arrays (NumPy) for analysis Libraries: OpenCV (cv2), ffmpeg python import cv2 cap = cv2.VideoCapture('video.mp4') while cap.isOpened(): ret, frame = cap.read() # frame is a NumPy array # Process frame... 3️⃣. 🇴🇧🇯🇪🇨🇹/🇸🇨🇪🇳🇪 🇩🇪🇹🇪🇨🇹🇮🇴🇳 Detect and count objects per frame (people, vehicles, products) Libraries: YOLO, TensorFlow, PyTorch, AWS Rekognition, Google Vision API 4️⃣. 🇦🇺🇩🇮🇴/🇸🇵🇪🇪🇨🇭 🇹🇴 🇹🇪🇽🇹 Extract audio track → transcribe to text → analyze Libraries: whisper (OpenAI), speech_recognition, Google Speech-to-Text 5️⃣. 🇴🇵🇹🇮🇨🇦🇱 🇨🇭🇦🇷🇦🇨🇹🇪🇷 🇷🇪🇨🇴🇬🇳🇮🇹🇮🇴🇳 (🇴🇨🇷) Extract on-screen text (dashboards, slides, signage) Libraries: pytesseract, EasyOCR, PaddleOCR 6️⃣. 🇲🇴🇹🇮🇴🇳/🇦🇨🇹🇮🇻🇮🇹🇾 🇦🇳🇦🇱🇾🇸🇮🇸 Optical flow, motion heatmaps, activity recognition Libraries: OpenCV, MediaPipe, MMAction2 7️⃣. 🇫🇦🇨🇮🇦🇱/🇪🇲🇴🇹🇮🇴🇳 🇦🇳🇦🇱🇾🇸🇮🇸 Detect faces, recognize emotions, track gaze Libraries: DeepFace, dlib, MediaPipe 8️⃣. 🇸🇹🇷🇺🇨🇹🇺🇷🇪🇩 🇩🇦🇹🇦 🇴🇺🇹🇵🇺🇹 All the above techniques produce structured data (CSV, JSON, tables) that can be loaded into Snowflake for analysis: ---------------------------------------------------------------------------------------- Frame/Timestamp | Objects Detected | Text Found | Speech Transcript | Emotion 00:01:05 | 3 people, 1 car | "EXIT" | "Turn left here" | Happy ---------------------------------------------------------------------------------------- In Snowflake Context You can combine this with Snowflake by: Pre-processing video externally (Python) → extract structured data Load extracted data into Snowflake tables. Use Cortex AI functions like AI_CLASSIFY, AI_EXTRACT, AI_SUMMARIZE on the extracted text/transcript data. Use AI_PARSE_DOCUMENT if you convert frames to images/PDFs for document-style extraction. The key insight: video itself isn't directly queryable — you must first transform it into structured/semi-structured data (text, numbers, labels) using the techniques above, then analyze that data. #DataEngineer #ETL #DataAnalysis

Hello Techies, Did you know you can train a Machine Learning model using just SQL — no Python, no setup, no deployment headaches? I recently explored BigQuery ML and honestly, it changed how I think about ML workflows for data teams. Let me show you what I mean. The Traditional Way (Python) Imagine you work at an e-commerce company. Your manager asks: "Can we predict which website visitors are likely to make a purchase?" As a data scientist, here's your to-do list: # 1. Extract data from warehouse to your machine df = bigquery_client.query("SELECT ...").to_dataframe() # 2. Clean it manually df['country'] = df['country'].fillna("") df['pageviews'] = df['pageviews'].fillna(0) # 3. Encode text columns (ML doesn't understand strings) df['country'] = LabelEncoder().fit_transform(df['country']) # 4. Split train/test X_train, X_test, y_train, y_test = train_test_split(X, y) # 5. Train model = LogisticRegression() model.fit(X_train, y_train) # 6. Save model pickle.dump(model, open('model.pkl', 'wb')) # 7. Deploy an API so others can use it # ... another few days of engineering work Total time from question to answer: days to weeks Skills needed: Python, sklearn, MLOps, deployment knowledge The BigQuery ML Way (SQL) Same problem. Same data. Here's your to-do using Biq Query: 1. Train CREATE MODEL `ecommerce.purchase_predictor` OPTIONS (model_type = 'logistic_reg', input_label_cols = ['will_purchase']) AS SELECT will_purchase, device_type, country, pageviews, session_duration FROM `ecommerce.visitor_data`; 2. Evaluate SELECT * FROM ML.EVALUATE(MODEL `ecommerce.purchase_predictor`); 3. Predict & get business insight SELECT country, SUM(predicted_will_purchase) AS expected_buyers FROM ML.PREDICT(MODEL `ecommerce.purchase_predictor`, (SELECT * FROM `ecommerce.next_month_visitors`)) GROUP BY country ORDER BY expected_buyers DESC; Total time from question to answer: minutes Skills needed: SQL What BigQuery ML handles automatically that you'd do manually in Python: > Null/missing value handling > Encoding text columns (country, OS, etc.) > Train/test splitting > Model storage — saved directly in BigQuery > Deployment — ML.PREDICT IS your API > Scaling — handles petabytes natively Supported model types in BigQuery ML today: > Logistic & Linear Regression > K-Means Clustering > XGBoost & Random Forest > Deep Neural Networks > Time Series Forecasting (ARIMA+) > Import TensorFlow/PyTorch models directly BigQuery ML won't replace data scientists — but it puts ML in the hands of every analyst who knows SQL. And that's a massive unlock for any data-driven organization. Have you tried BigQuery ML? What was your experience? Drop it in the comments #BigQuery #GoogleCloud #MachineLearning #DataScience #SQL #BigQueryML #GCP #DataEngineering #MLOps #Analytics #CloudComputing #AI #DataAnalytics #Python #TechLearning

You have been learning Python for months.  But can you load a messy CSV and tell me what the business should do next? If not - you are learning the wrong things. I have seen candidates spend months learning algorithms and data structures - then freeze when I ask them to load a CSV and answer a basic business question from it. That is not a Python problem. That is a direction problem. Here is the exact Python roadmap for data analysts, from someone who interviews them: 𝗦𝘁𝗮𝗴𝗲 𝟭 - 𝗧𝗵𝗲 𝗕𝗮𝘀𝗶𝗰𝘀 Variables, data types, loops, conditionals, and functions. Do not spend more than 2 weeks here. Resource: CS50P by Harvard - free at cs50.harvard.edu/python 𝗦𝘁𝗮𝗴𝗲 𝟮 - 𝗣𝗮𝗻𝗱𝗮𝘀 & 𝗡𝘂𝗺𝗣𝘆 This is where data analyst Python actually starts. -- Load data with pd.read_csv() -- Explore with head(), info(), describe() -- Clean with fillna(), dropna(), drop() -- Summarize with groupby(), pivot_table(), value_counts() -- Combine with merge() and join() If you cannot do this on a messy dataset without Googling - you are not ready for an interview. Resource: Kaggle Learn - free at kaggle.com/learn 𝗦𝘁𝗮𝗴𝗲 𝟯 - 𝗗𝗮𝘁𝗮 𝗖𝗹𝗲𝗮𝗻𝗶𝗻𝗴 & 𝗘𝗗𝗔 This is what most of a real analyst's job looks like. Handle missing values with context. Remove duplicates. Detect outliers. Convert data types. Explore distributions and trends. Clean data is the foundation of every insight. Resource: Keith Galli - youtube.com/@KeithGalli 𝗦𝘁𝗮𝗴𝗲 𝟰 - 𝗗𝗮𝘁𝗮 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 -- Matplotlib for basic charts -- Seaborn for statistical visuals -- Plotly for dashboards Can you take messy data and create a visualization that answers a business question - without being told which chart to use? That judgment is the skill. Resource: freeCodeCamp - https://lnkd.in/gvKw8x2W 𝗦𝘁𝗮𝗴𝗲 𝟱 - 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 -- rolling() and cumsum() for time series -- apply() and lambda for logic SQL + Python together. Automate reports. This is what gets you promoted. 𝗦𝘁𝗮𝗴𝗲 𝟲 - 𝗔𝗜 + 𝗣𝘆𝘁𝗵𝗼𝗻 -- Use Claude to pressure test your analysis -- Use it to draft summaries -- Use GitHub Copilot to speed up code Python without AI in 2026 is like knowing SQL but refusing to use indexes. You do not need to know all of Python. You need to know the 20% that does 80% of the work - deeply. The candidates I hire are not the ones who learned the most. They are the ones who can clean, analyze, visualize, and explain what the business should do. That is the roadmap. Everything else is noise. Where are you on this right now? ♻️ Repost to help someone learning Python for data analytics 💭 Tag someone learning Python without direction 📩 Get my full data analytics career guide: https://lnkd.in/gjUqmQ5H

Learning Python by putting this roadmap and resources attached into practice can build practical skills needed, especially augmenting its impact by combining with AI-based capabilities 👇

Day 24 - Automate KPI Reports with Python I turned 3 hours of weekly KPI reporting into 90 seconds using Python + SQL + AI. import pandas as pd import pyodbc from openai import OpenAI from datetime import datetime conn = pyodbc.connect("DSN=your_db;UID=user;PWD=pass") query = """ SELECT metric_name, current_value, target_value, ROUND((current_value/target_value)*100, 1) AS pct_of_target FROM kpi_dashboard WHERE report_week = DATEPART(week, GETDATE()) """ df = pd.read_sql(query, conn) df['status'] = df['pct_of_target'].apply(   lambda x: '🔴 Below' if x < 80   else ('🟡 At Risk' if x < 95 else '🟢 On Track') ) kpi_table = df[['metric_name','current_value','target_value','status']].to_string(index=False) client = OpenAI() response = client.chat.completions.create(   model="gpt-4o",   messages=[{     "role": "system",     "content": "You are a senior business analyst. Write concise, professional executive summaries."   }, {     "role": "user",     "content": f"""Write a 4-sentence executive KPI summary. KPI Data: {kpi_table} Report Date: {datetime.today().strftime('%B %d, %Y')}"""   }] ) print(response.choices[0].message.content) print(kpi_table) Example output: This week the team achieved strong results in customer acquisition 103% of target and delivery time 98%. Revenue per user is at risk at 82% of target pricing adjustments recommended before month-end. Churn remains the top concern at 71% of target, immediate customer success outreach is advised. No more staring at spreadsheets trying to write summaries. Your Monday mornings just got easier. Which part would you use first: A) SQL pull B) Status flagging C) AI narrative D) All of it #Python #KPIReporting #DataAutomation #SQL #OpenAI #AIEngineer #BusinessIntelligence

April 4, 2026. Day 2 of the new month. Still moving. Introduction to Data Visualization with Matplotlib — 4 hours — DataCamp. First course in the Data Visualization in Python track. And I want to talk about visualization honestly. Because there's a conversation here that goes deeper than charts and graphs. I've been visualizing data for a while now. Matplotlib has been in my toolkit. I've used it in projects — plotted distributions, drawn correlation matrices, built figures for EDA reports. So technically, I've been here before. But here's what I've come to understand about revisiting tools you think you already know: familiarity is not the same as fluency. I could produce a chart. I couldn't always produce the right chart, built the right way, communicating the right thing with intention and precision. There's a difference. Matplotlib is one of those libraries that rewards depth. On the surface it looks straightforward — you call a function, a plot appears. But underneath, it has a full object-oriented architecture. Figures. Axes. Artists. A structured way of thinking about every visual element as something you can control deliberately. Most people — myself included at earlier stages — use Matplotlib like a blunt instrument when it's actually a precision tool. This course made me slow down and learn the precision. And as someone who has spent over 10 years in a classroom drawing diagrams on a board — sketching graphs of quadratic functions, plotting velocity-time relationships in Physics, drawing titration curves in Chemistry — I know what it means to make a visual land. I know the difference between a graph that confuses and a graph that clarifies. I know that the choice of scale, label, color, and emphasis changes what a student — or a stakeholder — takes away completely. That teaching instinct is now being formalized into code. And it feels right. I'm also stepping into this new track — **Data Visualization in Python** — with a clear sense of where it fits in the bigger picture. Visualization is not decoration. It's not the thing you do after the "real" analysis. It IS part of the analysis. It's how you find patterns before you can name them. It's how you communicate what the data revealed after you've named them. Yesterday I completed the Data Manipulation in Python track — NumPy and pandas, the engine and the structure. Today, Matplotlib — the voice. The way data speaks to people who weren't in the room when it was collected. These things connect. Deliberately. That's the whole point. April is already demanding. But so am I. 📊 #Matplotlib #DataVisualization #Python #DataCamp #DataVisualizationInPython #DataScience #DataAnalysis #ContinuousLearning #3MTT #DeepTechReady #Nigeria #RealTalk #BuildingInPublic #April #TheGrind