Data Analytics News Roundup: CDC, LLM, and Pandas

I finally understand why data scientists say they spend 80% of their time on data. 📊 This week, instead of just reading about the ML lifecycle, I actually did the second step: Data Collection. 🎯 I built my own dataset called "TMDB Top Rated Movies" using their public API. 🎬 It was interesting to see how data can come from different sources some datasets are already available in formats like CSV and JSON, while others can be retrieved using SQL databases. I also learned that data can be collected through APIs or even web scraping depending on the use case. Nothing fancy. Just: 🐍 Python 📡 A bunch of API calls 🔄 Figuring out how to loop through pages without breaking everything In the end, I pulled together 10,000+ movie records clean, structured, and ready for actual analysis or ML. 📁✅ This part felt more like real engineering than anything I have done in a notebook. 🛠️ Small step. But it's real. 🚀 dataset link: https://lnkd.in/dG7EcE5q #MachineLearning #DataScience #Python #LearningByDoing

Debugging Data Flow: Resolving Silent Mismatches in FastAPI Architecture isn't just about the code you see; it's about the data that flows between layers. I just closed a persistent bug in my Todo App that served as a masterclass in dictionary-key synchronization and JWT payload extraction. The Challenge: Despite having the correct logic in my create_todo endpoint, my owner_id column was returning null. The Breakthrough: I discovered a silent mismatch in my data bridge. My authentication dependency was returning a user dictionary with the key user_id, but my CRUD logic was searching for the key id. Because Python’s .get() method returns None instead of crashing when a key is missing, the issue remained hidden until I inspected the dictionary structure. The Fix: By aligning my get_current_user dependency and my SQLAlchemy mapping to use a consistent key structure, I've successfully implemented Row-Level Security. Every task is now perfectly mapped to its creator. This taught me the value of explicit data contracts—a critical skill as I continue building toward complex Agentic AI systems where data integrity is the primary safety guard. Portfolio: 🔗 [https://lnkd.in/ehPH7fwh] @tiangolo | @FastAPI | @PythonNigeria | @LagosDev #FastAPI #Python #BackendEngineering #Debugging #JWT #BuildInPublic #AgenticAI #DataIntegrity #AdedaraBenson

I just built a very basic Natural Language to SQL Generator using LLM with LangChain, Groq, and Streamlit A natural language to SQL generator - you type a question in plain English, and it writes the SQL, runs it against a real database, and explains the results back to you. "Which customer has spent the most money?" → Generates a 3-table JOIN query automatically → Runs it against SQLite → Returns the answer with a plain English explanation No SQL knowledge needed. Code on GitHub : https://lnkd.in/g9bKNb_Y Stack: Llama 3.1 via Groq · LangChain · SQLite · Streamlit It's experimental. It's not perfect. But it taught me more about prompt engineering in one afternoon than a week of reading about it. #MachineLearning #Python #AI #BuildInPublic #LLM

📊 𝗖𝗵𝗲𝗰𝗸 𝗠𝗶𝘀𝘀𝗶𝗻𝗴 𝗩𝗮𝗹𝘂𝗲𝘀 𝗶𝗻 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 Before building any ML model, always check for missing values ❗ Ignoring them can lead to poor results 😬 🔍➤ 1) Check total missing values (count) df.isna().sum() ➡️ Shows missing count per column 📊 📉 ➤ 2) Missing values percentage (in %) (df.isna().sum() / len(df)) * 100 ➡️ Helps decide whether to drop 🗑️ or fill(Imputation) 🧩 📊 𝗩𝗶𝘀𝘂𝗮𝗹𝗶𝘇𝗲 𝗠𝗶𝘀𝘀𝗶𝗻𝗴 𝗩𝗮𝗹𝘂𝗲𝘀 📌 ➤ 1) Bar Chart df.isna().sum().plot(kind='bar', figsize=(15,4)) 🔥 ➤ 2) Heatmap import seaborn as sns import matplotlib.pyplot as plt plt.figure(figsize=(12,6)) sns.heatmap(df.isna(), cbar=False) plt.title("Missing Value Heatmap") plt.show() 🎨 Dark color (almost black / blue) → Value is NOT missing ✅ (data is present) ⚪ Light / white color → Value IS missing ❌ (NaN) 📑 𝗦𝘂𝗺𝗺𝗮𝗿𝘆 𝗧𝗮𝗯𝗹𝗲 (Clean Report) missing_report = pd.DataFrame({ "missing_count": df.isna().sum(), "missing_pct": df.isna().mean() * 100 }).sort_values(by="missing_pct", ascending=False) missing_report 🚀 Clean Data = Better Models 💯 Always handle missing values before training! #DataScience #MachineLearning #Python #DataAnalysis #GitHub #LearningJourney

📊 Just wrapped up my Mastering Pandas series — a 4-part deep dive into the library every data professional relies on. If you're learning pandas or want a solid reference to come back to, this series covers the full workflow from raw data to insights: 🔹 Part 1 — Reading, Sorting & Displaying Data https://lnkd.in/dg2ujnKC 🔹 Part 2 — GroupBy & Indexing https://lnkd.in/d3SaX-vu 🔹 Part 3 — Data Cleaning & Merging/Joining https://lnkd.in/dZaabdui 🔹 Part 4 — Data Visualization with Matplotlib & Seaborn https://lnkd.in/dxyhPhPv Each article walks through the core properties and methods with clean examples, comparison tables, and the "why" behind each tool — not just the syntax. Whether you're just starting out or brushing up, I hope this helps 🙌 Feedback and thoughts are always welcome. #Pandas #Python #DataScience #DataAnalysis #MachineLearning

🌐 Most people work with datasets… But where does the data actually come from? One of the most interesting things I explored recently was web scraping collecting data directly from websites instead of relying on pre-built datasets. 💡 What I realized: Real-world data is rarely clean or readily available. Before any analysis or AI model, the first step is often: → Extracting the data → Structuring it properly → Handling inconsistencies 🔧 In this project, I worked on: • Extracting data from web pages • Parsing and cleaning raw HTML content • Converting unstructured data into usable format • Preparing data for analysis 💡 Key takeaway: Data collection itself is a major part of the pipeline and sometimes more challenging than the analysis. This gave me a better understanding of how data pipelines actually begin. I’ve shared the project here: 👉 https://lnkd.in/eRzXNgsZ Curious to hear: 💬 Have you ever worked on collecting your own dataset instead of using ready-made data? #WebScraping #Python #DataEngineering #DataCollection #DataScience #BuildInPublic

🚀 NumPy for Data Science – The Backbone of Fast Computing! If you're stepping into the world of Data Science, one library you cannot ignore is NumPy (Numerical Python). 🔍 What is NumPy? NumPy is a powerful Python library used for numerical computing. It provides support for large, multi-dimensional arrays and matrices, along with a collection of mathematical functions to operate on them efficiently. 💡 Why NumPy? ✔ Faster than Python lists ✔ Memory efficient ✔ Supports vectorized operations ✔ Foundation for libraries like Pandas, Matplotlib, and Scikit-learn 📌 Key Concepts & Functions with Examples 1️⃣ Creating Arrays Definition: Used to create structured data (arrays) instead of traditional lists. import numpy as np arr = np.array([1, 2, 3, 4]) print(arr) 2️⃣ Zeros & Ones Functions Definition: Create arrays filled with zeros or ones. np.zeros((2,3)) # 2 rows, 3 columns of zeros np.ones((3,2)) # 3 rows, 2 columns of ones 3️⃣ Arange Function Definition: Generates values within a given range. np.arange(0, 10, 2) # Output: [0 2 4 6 8] 4️⃣ Reshape Function Definition: Changes the shape of an array without changing data. arr = np.array([1,2,3,4,5,6]) arr.reshape(2,3) 5️⃣ Statistical Functions Definition: Perform quick calculations on datasets. arr = np.array([1,2,3,4]) np.mean(arr) # Average np.sum(arr) # Total np.max(arr) # Maximum value 6️⃣ Mathematical Operations Definition: Apply operations element-wise. arr = np.array([1,2,3]) arr + 5 # [6 7 8] arr * 2 # [2 4 6] 📊 Real-Time Example Imagine analyzing student marks: marks = np.array([85, 90, 78, 92, 88]) print("Average:", np.mean(marks)) print("Highest:", np.max(marks)) 🎯 Conclusion NumPy is the foundation of Data Science. Mastering it will make your data processing faster, cleaner, and more efficient.

What if cleaning messy datasets took seconds instead of hours? 👀 🚀 I built an industrial-grade data cleaning tool that turns messy datasets into ML-ready data in seconds. While working with real-world datasets, I kept facing the same problem: ❌ messy columns ❌ missing values ❌ inconsistent formats ❌ hours wasted before even starting ML So I built DataForge Pro 👇 ⚙️ What it does: • Auto-cleans datasets (missing values, duplicates, types) • Detects & handles outliers (IQR / Z-score) • Converts messy strings like "$1,200" → numeric • Generates a full visual report (6 charts) • Gives an ML Readiness Score (0–100) 💡 Why this matters: Data scientists spend ~70–80% of time on cleaning. This tool reduces that to seconds. 🌐 Live Demo: https://lnkd.in/ggr8TjQK 📂 GitHub: https://lnkd.in/g6eSXaz2 📊 Built with: Python • Streamlit • pandas • scikit-learn This is just v1 — planning to add: → AI-powered cleaning suggestions → Polars for big data → REST API version Would love your feedback 🙌 Open to collaborations & improvements! #DataScience #Python #Streamlit #MachineLearning #OpenSource #BuildInPublic

t-SNE: Visualizing What We Can't See Imagine 784 dimensions compressed to 2 — and the clusters you see tell you everything about the structure of the data. t-SNE makes the invisible visible. Day 27 of 60 → t-SNE — the most beautiful data visualization tool in ML. PCA finds linear components. t-SNE finds NON-LINEAR structure — preserving local neighborhoods. The idea: 1. Measure which points are close in high-dimensional space 2. Lay them out in 2D preserving those closeness relationships 3. Similar points cluster together, dissimilar ones spread apart What good t-SNE output looks like: → Tight clusters = data has natural groupings → Fuzzy boundaries = gradual transitions between groups → Outlier points far from clusters = anomalies CRITICAL caveats: 1. Distances between clusters are NOT meaningful (only within-cluster distances) 2. Results depend on "perplexity" parameter (try 5, 30, 50) 3. Never interpret the x/y axis — they're arbitrary t-SNE is for EXPLORATION, not prediction. But for making the invisible visible? Nothing compares. #tSNE #DataVisualization #MachineLearning #Python #60DaysOfML

Pandas is the workhorse of EDA, but it’s dangerously easy to write bad code. If your data exploration is slow, crashing your Jupyter notebook, or throwing endless warnings, you might be falling into one of these 5 common traps. Here are the biggest Pandas anti-patterns and how to fix them: 1. The "For-Loop" Trap (df.iterrows) ❌ The Mistake: Looping through rows to apply logic. It is painfully slow because it bypasses Pandas' C-backend. ✅ The Fix: Vectorization. Use np.where() or native Pandas math operations. They are optimized and run exponentially faster. 2. The .apply() Bottleneck ❌ The Mistake: Thinking .apply() is fast. It's often just a glorified, hidden for-loop under the hood. ✅ The Fix: Use built-in vectorized string (.str) or datetime (.dt) methods whenever possible. 3. Ignoring Memory Optimization ❌ The Mistake: Using pd.read_csv() on massive datasets without defining data types. Everything loads as float64 or object, eating up your RAM. ✅ The Fix: Downcast your types. Convert strings with low cardinality to category, and float64 to float32. 4. Chained Indexing (SettingWithCopyWarning) ❌ The Mistake: Subsetting data like this: df[df['A'] > 5]['B'] = 10. You don't know if you are modifying a view or a copy. ✅ The Fix: Always use .loc[] for assignments: df.loc[df['A'] > 5, 'B'] = 10. 5. Blindly Dropping Nulls ❌ The Mistake: Slapping .dropna() on your dataframe just to make the code run, destroying valuable data context. ✅ The Fix: Investigate why data is missing. Use .fillna(), interpolation, or treat "missing" as its own valuable category. Efficiency in EDA isn't just about saving time; it’s about writing scalable code that doesn't break in production. What is your biggest Pandas pet peeve? Let me know below! 👇 #DataScience #Python #Pandas #DataEngineering #MachineLearning #TechTips