Mastering JSON in Machine Learning and Data Science with Pandas

After cleaning and preparing the dataset, today I made the chatbot talk to a real database. No CSV reading anymore. No in-memory DataFrame queries. The data is now stored in SQLite and accessed using real SQL. What I did today: • Established a connection between Python and SQLite • Converted the cleaned Pandas DataFrame into a SQL table using to_sql() • Designed the table structure directly from the dataset • Ensured data is permanently stored and queryable • Closed the connection properly to avoid database locks Now the system architecture looks like this: User Question → Rule Logic → SQL Query → SQLite Database → Answer This is where the project stops being a script… and starts becoming a real data system. Why this step matters: Because AI systems don’t answer from files. They answer from structured, queryable data sources. The chatbot is now able to answer questions directly from the database, not from Python memory. Next step: Use if / elif logic to map user questions directly to SQL queries and make the chatbot answer real questions from the database. Screenshots from Jupyter Notebook will be shared in the final project. #Python #SQL #SQLite #DataEngineering #AI #MachineLearning

You all know I released the first GA version of QQL v1.0.0 yesterday 🚀 If you’re curious what this project is really about and why it matters, I’ve put together a detailed blog that walks through everything. 👉 https://lnkd.in/gSrcuJhB In this blog, I break it down simply: • The architecture behind QQL — how a SQL-like query gets translated into native Qdrant operations • The motivation — why QQL exists and the gap it is trying to solve • A hands-on walkthrough of QQL CLI so you can see it in action The goal is straightforward: Make vector search feel natural for developers who think in queries, not SDK calls. If you’ve ever felt friction while working with vector databases, this might resonate with you. Would love to hear your thoughts after reading 👇 #AI #genAI #LLM #AIAgents #RAG #VectorSearch #QQL

Most people don’t struggle with PySpark because it’s hard. They struggle because they write it like Python… instead of Spark. This cheat sheet is a reminder that PySpark is built for: ➡️ Transformations, not step-by-step logic ➡️ Distributed execution, not local thinking ➡️ Optimization by design, not manual tuning everywhere A few patterns that change everything: 1. Read smart, write smarter Using Parquet instead of CSV isn’t just a format choice. It’s a performance decision. 2. Select early, reduce data The fastest data is the data you never process. Projection matters more than most people realize. 3. Joins & aggregations = shuffle zones If your job is slow, start here. This is where most pipelines break at scale. 4. Window functions > complex logic Cleaner, more expressive, and built for analytics use cases. 5. Lazy evaluation is your superpower Nothing runs until an action is triggered. Spark optimizes the entire DAG before execution. The difference I’ve seen in real projects: Same pipeline Same data ➡️ 200+ lines (script mindset) ➡️ 50 lines (Spark mindset) Cleaner code. Better performance. Easier debugging. If you’re learning PySpark, don’t just focus on syntax. Focus on: How Spark executes Where shuffles happen How to minimize data movement That’s where real engineering starts. 📌 𝗥𝗲𝗴𝗶𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝗼𝗽𝗲𝗻 𝗳𝗼𝗿 𝗼𝘂𝗿 𝟮𝗻𝗱 𝗯𝗮𝘁𝗰𝗵 𝗼𝗳 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗖𝗼𝗵𝗼𝗿𝘁 , 𝗘𝗻𝗿𝗼𝗹𝗹 𝗵𝗲𝗿𝗲- https://rzp.io/rzp/May2026

📊𝗗𝗮𝘆 𝟲𝟳 𝗼𝗳 𝗠𝘆 𝗗𝗮𝘁𝗮 𝗦𝗰𝗶𝗲𝗻𝗰𝗲 & 𝗗𝗮𝘁𝗮 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗝𝗼𝘂𝗿𝗻𝗲𝘆 Today I explored an important Python concept that strengthens how we safely handle data structures in real-world analytics projects — Dictionary Comparison, Shallow Copy, and Deep Copy. At first, copying a dictionary may look simple. But when working with nested data structures like JSON files, API responses, configuration objects, or feature-engineered datasets, understanding how Python handles memory references becomes extremely important. Here’s what I learned today: 🔹 Dictionary Comparison in Python Dictionary comparison helps verify whether two datasets or configurations are identical by checking both keys and values. This is especially useful during data validation, debugging transformations, and ensuring correctness in preprocessing pipelines. Example use cases: • Checking whether cleaned data matches expected output • Validating configuration dictionaries in ML workflows • Comparing original vs transformed datasets during feature engineering This improves reliability and reduces silent errors in analytics workflows. 🔹 Shallow Copy – Understanding Reference Behavior A shallow copy creates a new dictionary object, but nested objects inside the dictionary still reference the same memory locations as the original dictionary. That means: If we modify nested elements, the changes appear in both copies. This concept is important when working with: • Nested dictionaries • Lists inside dictionaries • Structured dataset representations Shallow copy is faster and memory-efficient, but must be used carefully in data preprocessing tasks. Example: Useful when copying only top-level structures without modifying nested elements. 🔹 Deep Copy – Creating Fully Independent Data Structures A deep copy creates a completely independent duplicate of the dictionary, including all nested objects. That means: Changes made in one dictionary will NOT affect the other dictionary. This is extremely useful in Data Science when: • Performing multiple transformation experiments on the same dataset • Creating safe backup versions of datasets before cleaning • Handling nested JSON responses from APIs • Building reliable machine learning preprocessing pipelines Deep copy ensures data integrity and prevents accidental overwriting of original datasets. 💡 Key Learning Insight from Today Understanding how Python handles memory references is not just a programming concept — it directly impacts how safely and efficiently we manipulate datasets in analytics and machine learning workflows. The more I learn about Python internals like these, the more confident I feel working with real-world data structures used in Data Science projects. #Day67 #PythonLearning #DataScienceJourney #DataAnalytics #LearningInPublic #PythonForDataScience #FutureDataScientist #WomenInTech #ConsistencyMatters

We need to start caring about data packaging again. I migrated Rahu’s Python AST from a pointer-heavy recursive structure to an arena-backed one, and it improved both analysis and lookup much more than I expected. Rahu is a Python language server I’m building from scratch in Go. The old AST used separate structs, pointers, and slices to model recursive trees. That made it easy to work with, but it also meant many small allocations, pointer chasing, and poor cache locality in hot paths. The new AST is stored as a flat arena: compact nodes in a contiguous slice, stable NodeIDs, sibling-linked children, and side tables for names, strings, and numbers. A good example is attribute access. In the old AST, obj.field was an Attribute node pointing to both the base expression and a separate Name node. In the new one, it’s just a NodeAttribute plus child IDs into the same array. Traversal involves indexed access instead of following heap pointers. The result: AnalysisSmall: ~84 µs → ~55 µs AnalysisMedium: ~183 µs → ~117 µs AnalysisLarge: ~2.15 ms → ~1.85 ms DefinitionLookup: ~205 ns → ~30 ns HoverLookup: ~207 ns → ~34 ns DefinitionLookupAll: ~12.2 µs → ~1.36 µs The geomean across the benchmark set dropped by about 45%. Some construction-heavy paths worsened slightly, which is expected: the arena model added bookkeeping and shifted work into indexing and side tables. The edit-time analysis path improved, and lookup improved significantly, which matters more for the actual LSP experience. The main takeaway for me was simple: data layout matters. I didn’t change the language features. I changed AST storage and traversal, and that had a large effect on end-to-end performance.

Python Series – Day 22: Data Cleaning (Make Raw Data Useful!) Yesterday, we learned Pandas🐼 Today, let’s learn one of the most important real-world skills in Data Science: 👉 Data Cleaning 🧠 What is Data Cleaning Data Cleaning means fixing messy data before analysis. It includes: ✔️ Missing values ✔️ Duplicate rows ✔️ Wrong formats ✔️ Extra spaces ✔️ Incorrect values 📌 Clean data = Better results Why It Matters? Imagine this data: | Name | Age | | ---- | --- | | Ali | 22 | | Sara | NaN | | Ali | 22 | Problems: ❌ Missing value ❌ Duplicate row 💻 Example 1: Check Missing Values import pandas as pd df = pd.read_csv("data.csv") print(df.isnull().sum()) 👉 Shows missing values in each column. 💻 Example 2: Fill Missing Values df["Age"].fillna(df["Age"].mean(), inplace=True) 👉 Replaces missing Age with average value. 💻 Example 3: Remove Duplicates df.drop_duplicates(inplace=True) 💻 Example 4: Remove Extra Spaces df["Name"] = df["Name"].str.strip() 🎯 Why Data Cleaning is Important? ✔️ Better analysis ✔️ Better machine learning models ✔️ Accurate reports ✔️ Professional workflow ⚠️ Pro Tip 👉 Real projects spend more time cleaning data than modeling 🔥 One-Line Summary Data Cleaning = Convert messy data into useful data 📌 Tomorrow: Data Visualization (Matplotlib Basics) Follow me to master Python step-by-step 🚀 #Python #Pandas #DataCleaning #DataScience #DataAnalytics #Coding #MachineLearning #LearnPython #MustaqeemSiddiqui

Ever wondered how machine learning can predict house prices with real-world data? I built an end-to-end House Price Prediction system using Machine Learning and deployed it using Django. This project covers the complete pipeline—from raw data to real-time predictions: - Data Cleaning & Preprocessing (handling missing values) - Exploratory Data Analysis (Univariate & Bivariate) - Statistical Testing (VIF, T-Test, ANOVA) - Data Visualization (Histogram & Scatter Plot) - Feature Selection (Forward & Backward Selection) - Model Training (Linear Regression) - Model Evaluation using R² Score - Model Deployment using Django Web App Through this project, I gained hands-on experience in: - Building a complete ML pipeline from scratch - Understanding statistical techniques in real-world datasets - Feature engineering & selection strategies - Scaling data correctly using StandardScaler - Saving & loading models using Pickle - Integrating ML models into a Django web application - Debugging real-world issues like data shape, scaling & deployment 📌 Follow me for more AI & Data Science projects 📌 Stay connected 🚀 #MachineLearning #DataScience #Python #AI #Django #Projects #.Net

Ever wished you could just talk to your data instead of writing pandas scripts? 📊🤖 I’m thrilled to share my latest project: Data Copilot-LLM POWERED DATA ANALYSIS ASSISTANT You simply upload a CSV, ask a question in plain English, and the system generates instant charts, statistics, and insights. Here is a look under the hood: 🧠 Brains: LangChain + LLaMA 3 (via Groq) handles code generation and dynamic chart selection. ⚙️ Engine: A modular FastAPI backend integrated with a Streamlit frontend. 🛡️ Sandbox: All LLM-generated Python code is executed in a heavily restricted, safe environment using RestrictedPython. 🔄 Auto-Fix: If the LLM writes bad code, the backend catches the error and sends it back to the AI for a self-correction loop! ⚡ Infrastructure: Fully containerized with Docker, utilizing Redis for DataFrame caching, SQLite for persistent history, and MLflow for query telemetry. It even supports multi-CSV joins! ⚙️Tech Stack : FastAPI, LangChain + Groq, pandas + matplotlib + seaborn, Redis, SQLite, MLflow Building this project was the perfect way to bridge the gap between experimental AI and production-ready software. It required a true end-to-end mindset-blending LLM engineering, prompt engineering, and RAG concepts for the intelligence layer, while relying on robust backend architecture, rigorous security sandboxing, MLOps for tracking, and DevOps for seamless containerization I’d love to hear feedback from the community! While this is not deployed yet feel free to clone the repo and test it👇 🔗 https://lnkd.in/gVdu2Ke2 #DataScience #MachineLearning #GenAI #FastAPI #LangChain #Docker #Python #DataEngineering #PromptEngineering #MLOps #SriLankaTech #CareerGrowth

Detailed & Professional (Best for showing deep learning) Headline: Leveling Up: Integrating Flask with SQLAlchemy for Dynamic Data Persistence! 🖥️🗄️ A great user interface means nothing if the data doesn’t stick around. Today, I took a massive leap forward in my backend development journey by connecting my Flask applications to a real database using SQLAlchemy. Moving beyond volatile, in-memory data, I mastered the art of making information persistent. Here’s a breakdown of my latest CRUD (Create, Read, Update, Delete) integration milestones: 🏗️ Database Connection & Setup: Successfully configured my Flask app to communicate with a database using SQLAlchemy, mapping Python classes to database tables (the magic of ORM!). 📩 Create: Inserting Data: I bridged the gap between my front-end Flask forms (WTForms) and the backend. User input is now captured and securely stored directly into the database. 📊 Read: Displaying Data: Mastered querying the database to retrieve stored information and dynamically rendering it in my HTML templates. Seeing real-time data flow from the DB to the user interface was highly rewarding! This isn't just about code; it’s the foundation for the student portal I’m building. Now, I can ensure that user registrations and test submissions are saved permanently. Up next: implementing the 'Update' and 'Delete' functionalities to complete the full CRUD cycle! #Python #Flask #SQLAlchemy #Database #WebDevelopment #BackendDeveloper #CRUD #CodingJourney #LearningToCode #DataPersistence #TechCommunity #FullStack #ORM

#dataengineering A data ingestion problem. 20–30K files. Download. Chunk. Vectorize. Every few hours. Forever. Imagine this. It's 2am. Your pipeline is running — barely. A single Python script, looping through files one by one. Download. Chunk. Vectorize. Wait. Repeat. 25,000 files taking hours. And by the time it finishes, it's almost time to start again. Sound familiar? Here's how you'd think through it — and why the "obvious" answers are often wrong. Lets walk through available options. Option 1 — Serial (single-threaded) Does one thing, finishes, moves on. Simple to write and debug. If it fails, you know exactly where. But for 25K files? You're waiting all night. Fine for a weekend prototype. A disaster in production. Option 2 — Async / Concurrent Send 100 requests before the first one comes back. A right step to take. Python's asyncio let us fire off dozens of downloads simultaneously. I/O-bound work — waiting for HTTP responses — is where async shines. Runtime dropped dramatically. But we're still on one machine, one CPU core. Vectorization is CPU-heavy. Async won't help there. Option 3 — Multi-threaded Put every core to work. ThreadPoolExecutor or multiprocessing let us use all CPU cores for the chunking and embedding work. Combined with async for downloads, this was a real upgrade. But Python's GIL limits true CPU parallelism in threads — you need multiprocessing to escape it. Still a single machine. Still a single point of failure. Option 4 — Apache Spark Distribute the job across a cluster. Spark is extraordinary — when you need it. Petabytes? Millions of files? Yes. 25K files every few hours? You're spending more time on cluster management than the actual work. Spark has high overhead. Don't bring a rocket ship to a road trip. Option 5 — Highly Available Distributed Service A queue. Workers. Retries. Observability. Always on. This is where we landed for production. A task queue (Celery, RQ, or a cloud-native option like Cloud Tasks) pulls jobs off a queue. Workers process independently. Failed jobs retry automatically. New files? Push to queue. Workers scale up. It's more complex to set up than the first 3 options — but it's the only option that handles real-world messiness: flaky APIs, partial failures, midnight spikes. The lesson? Each step wasn't an upgrade in prestige. It was an upgrade in the problem being solved. Serial → Async: you're I/O-bound. Async → Multi-process: you're CPU-bound. Single node → Distributed: you need fault tolerance. Spark → HA service: you need continuous operation, not just scale. Know which problem you actually have before you reach for the hammer.