PyMuPDF outperforms pypdf for speed and AI integration

Polars or pandas for dataframes? I recently asked one of the developers, and this is what I found: 🖥️From a technical perspective, there is little reason to remain with pandas: 👉Polars is significantly ahead. It has addressed many of the long-standing issues pandas has struggled with, while offering a clearer API and much faster performance. 👉Pandas is unlikely to change dramatically, while polars is evolving quickly. That means the tech gap between the performance of the 2 libraries will continue to widen. In practice: 👉Few people move from polars to pandas, while many users are transitioning from pandas to Polars. 👉Still, pandas is huge compared to Polars. In fact, if you check the summary made by MLcontests about the data science competitions in 2025, you’ll notice that Pandas is still the go-to library for dataframe manipulation, used in 61 competitions vs 5 using polars. 💡Pandas popularity will not change overnight, which means that pandas will likely remain widely used and, for a long time, more popular overall. So, which library should you use? In short: 👉Are you new to Python and dataframes ⇒ then learn polars 👉Working with legacy code? You are not alone and pandas is here to stay for many years, so your learnings will not be wasted Which library do you use? Let me know in the comments 👇 #machinelearning #ml #dataframes #polars #pandas #mlonline #mlcourse #trainindata #datascience #datascientist #dataengineer #dataengineering #mleducation #mlcareer #ai #python

🚀 Banknote Authentication System using Machine Learning & FastAPI I recently built a machine learning-powered API that can detect whether a banknote is real or fake based on key statistical features. 🔍 Project Highlights: - Built a classification model using Scikit-learn - Used features like variance, skewness, curtosis, and entropy - Saved and deployed the model using Pickle - Developed a high-performance API with FastAPI - Tested endpoints using Postman & Swagger UI ⚙️ Tech Stack: Python | FastAPI | Scikit-learn | NumPy | Pandas | Uvicorn 📌 How it works: The API accepts input data and returns a prediction indicating whether the banknote is genuine or counterfeit. 💡 This project helped me understand: - Model deployment in real-world applications - API development and testing - Handling model serialization and version issues 🔗GitHub Repository:https://lnkd.in/gYi6eSnU Looking forward to enhancing this with a frontend and deploying it on the cloud! #MachineLearning #FastAPI #Python #AI #DataScience #BackendDevelopment

I just built a very basic Natural Language to SQL Generator using LLM with LangChain, Groq, and Streamlit A natural language to SQL generator - you type a question in plain English, and it writes the SQL, runs it against a real database, and explains the results back to you. "Which customer has spent the most money?" → Generates a 3-table JOIN query automatically → Runs it against SQLite → Returns the answer with a plain English explanation No SQL knowledge needed. Code on GitHub : https://lnkd.in/g9bKNb_Y Stack: Llama 3.1 via Groq · LangChain · SQLite · Streamlit It's experimental. It's not perfect. But it taught me more about prompt engineering in one afternoon than a week of reading about it. #MachineLearning #Python #AI #BuildInPublic #LLM

New LangChain project: Document QA with RAG Load any text or PDF, embed it into FAISS, and answer natural-language questions — grounded strictly in the document. No hallucinations, with full source attribution. What's inside: → TextLoader + RecursiveCharacterTextSplitter for chunking with overlap → OpenAI text-embedding-3-small + FAISS for semantic vector search → RetrievalQA chain (chain_type="stuff") with a custom grounding prompt → return_source_documents=True — every answer shows which chunks backed it → Interactive Q&A mode + PDF-ready (swap TextLoader for PyPDFLoader) Difficulty: Beginner LangChain Part of the ai-projects series (now 13 projects). https://lnkd.in/g7f_iyTN #LangChain #RAG #Python #AI #GenerativeAI #AWS #MachineLearning #OpenAI

I recently had an interview where I was asked how I would build an AI system that can answer questions from 10,000 files. I didn’t have a strong answer. My AI experience was mostly chat history and summarization — not retrieval across a large document set. At the end the interviewer gave me a hint: RAG. So I built it from scratch — a document Q&A API where you upload files and ask questions about them. The workflow: 1. Split documents into chunks 2. Embed each chunk locally using sentence-transformers (free, runs on your machine) 3. Store vectors in PostgreSQL with pgvector 4. Embed the user query 5. Retrieve top 20 candidates via approximate nearest neighbor search 6. Rerank using a cross-encoder model to select the true top 5 7. Generate a grounded answer via Groq API (free tier, Llama 3.1) Built with Python, FastAPI, and containerized with Docker Compose. Used Azure Blob Storage (free tier) for file storage and Groq for inference — the entire stack costs $0 to run. I didn’t get the job. But I turned one weak answer into a project and a much better understanding of retrieval systems. Next time I get that question, I’ll have a real answer. GitHub: https://lnkd.in/e7cDAjdx #RAG #Python #FastAPI #PostgreSQL #LLM #SoftwareEngineering

🚀 NumPy – The Foundation of Machine Learning If you're starting Machine Learning, NumPy is the first concept you must master. Here’s what I’ve covered in this beginner-friendly guide: ✔️ What NumPy is and why it's powerful ✔️ Arrays vs Python Lists (performance + structure) ✔️ Creating arrays (1D & 2D) ✔️ Array attributes (shape, dimensions, data types) ✔️ Indexing & slicing ✔️ Mathematical operations ✔️ Important functions (zeros, ones, arange, linspace) ✔️ Reshaping arrays ✔️ Real-world use in Machine Learning NumPy is not just a library — it’s the core engine behind ML models. Everything from data processing to model computation depends on it. I’ve created a clear and practical material so you can actually understand and apply, not just memorize. 📚 Additional Resource to go deeper: https://lnkd.in/gQ-8CH4m w3schools.com Don’t just read — try every line of code. Let’s build a strong foundation together 💡 💬 Comment your add-ons 🤝 Let’s learn together 🧠 Let’s explain each other #MachineLearning #AIBasics

Nobody talks about the quiet revolution that already happened in Python data tooling. Pandas was the default for years. Comfortable. Familiar. Everywhere. But in 2024–2025, something shifted. Here's what the modern Python data stack actually looks like now: → DuckDB for analytical queries on local files No server. No setup. Just SQL that runs faster than you expect directly on CSVs and Parquets. → Polars for dataframe operations Written in Rust. Built from scratch for multi-core CPUs. Lazy evaluation by default. On large datasets, it's not 2× faster than Pandas. It's often 10–50×. → Pandas is still useful. But mostly as a last step for compatibility, not for computation. The real insight here isn't the tools. It's the mental model. The old stack was: load → transform → analyze (all in Pandas). The new stack is: query first (DuckDB) → transform fast (Polars) → output clean (Pandas if needed). If you're still running df.groupby() on a 5M-row CSV in Pandas and wondering why your laptop fan is screaming this is for you. I wrote a deep dive on exactly this shift covering benchmarks, real code comparisons, and when to use which tool. Follow for more practical AI & data engineering content. What's your current go-to for data wrangling? Still Pandas, or have you made the switch? 👇 #Pandas #Python #DataScience #AI #DataCleaning

🔬 I built a Production-Grade RAG System from Scratch — Here's How. A research assistant that answers questions with EXACT page citations. How I Built It: 📄 Step 1: Document Ingestion • Load PDFs using PyPDFLoader • Split into semantic chunks (1000 chars, 200 overlap) • Each chunk = 1 searchable unit 🔢 Step 2: Vector Embeddings • Convert text chunks to numerical vectors • Used sentence-transformers (all-MiniLM-L6-v2) • Similar meaning = closer vectors in space 🔍 Step 3: Vector Search • User question → converted to vector • Cosine similarity search across 70+ chunks • Retrieved top-k most relevant chunks 🤖 Step 4: LLM Generation • Retrieved chunks = context • Google Gemini API generates answer • Answer based ONLY on retrieved context • Every answer includes exact page numbers 💾 Step 5: Database & Export • SQLite stores all Q&A pairs • Bookmark important answers • Export to CSV for research documentation Technical Challenges Overcome: Challenge 1: Rate Limits → Implemented retry logic with exponential backoff → Optimized model selection for performance Challenge 2: Slow Startup (25 seconds) → Implemented caching for embeddings → Reduced to 2-3 seconds startup time Challenge 3: Section Detection → Built regex patterns for Roman numerals & numbering → Generated hierarchical tree diagram of document structure Tech Stack: Python | Streamlit | LangChain | Google Gemini API | SQLite | Sentence-Transformers | PyPDF Results: ✅ Processes 12-page papers → 70+ searchable chunks ✅ Sub-3 second response time ✅ 80% faster research analysis ✅ Production-ready web interface #RAG #RetrievalAugmentedGeneration #VectorDatabase #Embeddings #LangChain #GoogleGemini #Streamlit #Python #LLM #GenerativeAI #PortfolioProject

Workflow Experiment Tracking using steppy #machinelearning #datascience #workflowexperimenttracking #steppy Steppy is a lightweight, open-source, Python 3 library for fast and reproducible experimentation. It lets data scientist focus on data science, not on software development issues. Steppy’s minimal interface does not impose constraints, however, enables clean machine learning pipeline design. What problem steppy solves? In the course of the project, data scientist faces multiple problems. Difficulties with reproducibility and lack of the ability to prepare experiments quickly are two particular examples. Steppy address both problems by introducing two simple abstractions : Step and Tranformer. We consider it minimal interface for building machine learning pipelines. Step is a wrapper over the transformer and handles multiple aspects of the execution of the pipeline, such as saving intermediate results (if needed), checkpoiting the model during training and much more. Tranformer in turn, is purely computational, data scientist-defined piece that takes an input data and produces some output data. Typical Transformers are neural netowrk, machine learning algorithms and pre- or post-processing routines. https://lnkd.in/gUJZpVPD

Built something interesting this week. I created a simple tool that turns raw datasets into step-by-step analysis without the usual mess. You upload your file, describe what you need, add your own API key, and it handles the rest. Clean code, proper flow, and most importantly complete outputs (not half-baked results). Kept it very intentional: – No internal API usage – No guessing or skipping steps – No unnecessary visuals unless asked Just a controlled system that does exactly what you tell it to do. Also added export options (Python, Jupyter, Colab, Streamlit) so you can actually use the work outside the tool. UI is minimal, fast, and built with a futuristic feel (green + black theme). Still early, but it works and that’s what matters. Curious to hear: What would you improve in something like this? Hisham Sarwar Saad Hamid Mehroze Munawar Muhammad Umar Nazir #AISeekho2026 #VibeKaregaPakistan #DataScience #AI #Python #BuildInPublic #SaaS