Python Script for Efficient URL Crawling with Custom Headers

Day 7/10 🚀 This is where your code grows up. Modules & Packages — the foundation of every scalable Python project. Without them? Spaghetti code, repetition, no structure. With them? Clean, reusable, production-ready code. 📋 What I covered today: 01 → Modules & package structure 02 → Creating & importing .py files 03 → init.py & sub-packages 04 → Import styles — import, from, alias 05 → Relative vs absolute imports 06 → Standard library — os, json, datetime, re 07 → Third-party packages — pip, numpy, pandas 08 → Virtual environments & requirements.txt 09 → Mini Project — Config Loader Package Built a small config package with loader & validator modules — a real-world pattern used in production apps. Day 1 ✅ Day 2 ✅ Day 3 ✅ Day 4 ✅ Day 5 ✅ Day 6 ✅ Day 7 ✅ 3 more to go. Drop a 📦 if you’ve ever put everything in one giant .py file 😄 #Python #Modules #Packages #DataEngineering #LearningInPublic #CleanCode #10DaysOfPython #SoftwareEngineering

Built a small Python utility to refresh core fundamentals and structure. Built a CLI-based Contact Management system with: • Full CRUD operations • JSON-based persistent storage • Modular function design The idea wasn’t complexity, but to reinforce clean logic, data handling, and building systems from scratch without relying on frameworks. Planning to extend this further towards API-based and AI-driven workflows. GitHub: https://lnkd.in/gW8BXbKP #Python #SoftwareDevelopment #Backend #ContinuousLearning

One of the most common FastAPI mistakes is reusing SQLAlchemy models as Pydantic response schemas. This doesn't just couple your layers—it makes it easy to accidentally expose fields like hashed passwords. In my latest post, I share 8 Sections of battle-tested patterns for structuring your FastAPI applications: 1️⃣ Centralized Config: Using pydantic-settings instead of scattered os.environ calls. 2️⃣ Dependency Injection: Leveraging Depends() for cleaner auth and DB sessions. 3️⃣ Business Logic: Moving core logic into pure Python service functions for easier unit testing. 4️⃣ Error Handling: Using app-level exception handlers to remove "try/except" noise from routes. Swipe through the PDF for the code examples and folder structures! ➡️ #FastAPI #Python #BackendDevelopment #WebDevelopment #CleanCode

🐍 Exploring NumPy Indexing & Slicing in Python Today I practiced how to access and manipulate data inside NumPy arrays. This is a very important step for Data Science and Machine Learning 🚀 ✔ Accessed elements using indexing (arr[9]) ✔ Used slicing to get parts of arrays (arr[:9], arr[8:]) ✔ Learned that slicing creates a view (changes affect original array!) ✔ Used .copy() to avoid modifying original data ✔ Worked with 2D arrays (matrix indexing) ✔ Applied conditional filtering (arr[arr > 3]) ✔ Explored slicing in 2D arrays (rows & columns selection) 💡 Key takeaway: NumPy slicing returns a view, not a copy. So changing a slice can change the original array — this is very important to remember! 📌 Full code available here: 👉 https://lnkd.in/dCMhYQey #Python #NumPy #DataScience #MachineLearning #100DaysOfCode #LearningJourney

Day 30 of #100DaysOfPython 𝐔𝐩𝐠𝐫𝐚𝐝𝐞𝐝 𝐭𝐡𝐞 𝐏𝐚𝐬𝐬𝐰𝐨𝐫𝐝 𝐌𝐚𝐧𝐚𝐠𝐞𝐫 𝐀𝐩𝐩 𝐭𝐨𝐝𝐚𝐲. I moved from saving data in a text file to using JSON, which makes the data more structured and easier to manage. I also added a search feature to retrieve saved credentials. 𝐖𝐡𝐚𝐭 𝐈 𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐝: 𝑺𝒘𝒊𝒕𝒄𝒉𝒆𝒅 𝒕𝒐 𝑱𝑺𝑶𝑵 𝒇𝒐𝒓 𝒔𝒕𝒓𝒖𝒄𝒕𝒖𝒓𝒆𝒅 𝒅𝒂𝒕𝒂 𝒔𝒕𝒐𝒓𝒂𝒈𝒆 𝑰𝒎𝒑𝒍𝒆𝒎𝒆𝒏𝒕𝒆𝒅 𝒔𝒆𝒂𝒓𝒄𝒉 𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏𝒂𝒍𝒊𝒕𝒚 𝒇𝒐𝒓 𝒔𝒂𝒗𝒆𝒅 𝒑𝒂𝒔𝒔𝒘𝒐𝒓𝒅𝒔 𝑯𝒂𝒏𝒅𝒍𝒆𝒅 𝒆𝒓𝒓𝒐𝒓𝒔 𝒖𝒔𝒊𝒏𝒈 𝒕𝒓𝒚/𝒆𝒙𝒄𝒆𝒑𝒕 (𝒎𝒊𝒔𝒔𝒊𝒏𝒈 𝒇𝒊𝒍𝒆, 𝒎𝒊𝒔𝒔𝒊𝒏𝒈 𝒅𝒂𝒕𝒂, 𝒆𝒎𝒑𝒕𝒚 𝑱𝑺𝑶𝑵) 𝑰𝒎𝒑𝒓𝒐𝒗𝒆𝒅 𝒐𝒗𝒆𝒓𝒂𝒍𝒍 𝒖𝒔𝒆𝒓 𝒆𝒙𝒑𝒆𝒓𝒊𝒆𝒏𝒄𝒆 𝒊𝒏 𝒕𝒉𝒆 𝑮𝑼𝑰 This version feels much closer to a real application. Managing data properly and handling edge cases made a big difference. Also a good reminder that writing code is one thing, but making it robust and user-friendly is another level. #100DaysOfCode #100DaysOfPython #Python #Tkinter #PasswordManager #JSON #ErrorHandling #PythonProjects #LearningToCode #CodingJourney #BuildInPublic

Day 8 of #30DaysOfPython ✅ Until today, every single thing I built vanished the moment I closed the terminal. Run the script, see the output, close it — gone. No trace. No memory. Nothing. File handling changed that completely. I can now write a script that creates a file, puts data inside it, and saves it to my computer — permanently. I can open that same file an hour later and read every line back. My code has storage now. That shift in what Python can do hit harder than I expected. The concept itself isn't complicated. But the details will get you. I spent 20 minutes wondering why my file kept getting wiped every time I ran my script. Turns out I was opening it in "w" mode — which doesn't add to a file. It replaces it. Completely. Every. Single. Time. Switched to "a" for append. Problem solved. Data survived. Lesson learned permanently. 😅 The other habit I picked up today: always use with open() instead of plain open(). It automatically closes the file when you're done — even if your code crashes midway. I didn't think it mattered until I read about what happens to files that never get closed properly. Now I will never skip it. What I covered today: "r" to read, "w" to write, "a" to append with open() as the safe, correct way to handle files .read(), .readlines(), and looping line by line Checking if a file exists before trying to open it Day 8 done. My code has a memory now. 🧠 👇 What was the first script you wrote that actually saved data to a file? I'd love to know what made file handling click for you — drop it below! #Python #30DaysOfPython #FileHandling #BuildInPublic #PythonProjects

Built a Python-based Directory Sync Tool to compare and synchronize files between two directories with reliability and control. Instead of relying only on file names or timestamps, the tool uses a combination of metadata and SHA-256 hashing to accurately detect new, modified, and missing files. Key highlights: • Recursive directory scanning with structured metadata (name, extensions, size, hash) • Efficient change detection using size-first filtering followed by hash comparison • Memory-efficient hashing using chunk-based file reading (handles large files) • Synchronization support with metadata preservation using shutil.copy2 • Safe cleanup by optionally removing extra files from the destination While building this, I focused on moving beyond a basic script and treating it like a real tool, structuring the code into clear components, improving output readability, and adding validation and error handling to make it more reliable in real use. GitHub:https://lnkd.in/gt-Ec3rF #Python #CLI #GitHubProjects #SoftwareDevelopment #LearningByBuilding #SystemsThinking

I built a mini Product API with FastAPI. If you’re learning backend, build this API first. Simple project, but it covers important backend fundamentals: - GET routes - POST routes - dynamic route parameters - request validation with Pydantic - basic error handling with HTTPException - in-memory data handling The project is small, but the concepts are used everywhere in real backend systems. Next step: replacing the in-memory product list with SQL. #FastAPI #Python #BackendDevelopment #SoftwareEngineering #APIDevelopment #RESTAPI #BuildInPublic #PythonDeveloper

A 40ms API became a 4ms API. Here's the only thing that changed. We were making 3 separate DB queries to assemble a response. Each was fast in isolation. Together, they were sequential — each waited for the previous. The fix: run them concurrently. In Python (asyncio), this went from: result_a = await get_a() result_b = await get_b() result_c = await get_c() To: result_a, result_b, result_c = await asyncio.gather(get_a(), get_b(), get_c()) That's it. No caching, no infra change, no complex refactor. The mental model that helps: always ask "are these operations actually dependent on each other?" before assuming they need to run in sequence. Most API latency problems aren't hard — they're just unexamined. #BackendDevelopment #PythonAsyncio #APIOptimization #SoftwareEngineering

🚀 Just shipped my latest Python project — a CLI-based Log Analyzer! Log debugging is one of those tasks that can eat up hours. I built a tool to make it faster and smarter. 🔍 What it does: Takes raw log files in multiple formats — plaintext, CSV, XML, and YAML — and transforms them into structured, actionable reports right in your terminal. 📊 The output includes: → KPI Summary (Total Events, Error Rate, Uptime Score) → Exception Analysis (SQL Timeouts, NullPointerExceptions, and more) → Intelligent Insights (e.g., detecting cascading failures across services) So instead of manually grepping through hundreds of lines like: [2026-04-03 10:16:12.003] [Thread-09] ERROR [com.store.Database] SQL State: 08001 - Connection Timeout ...you get a clean, parsed report that tells you exactly what went wrong and where. Building this taught me a lot about: ⚙️ Multi-format file parsing in Python ⚙️ Pattern recognition across log structures ⚙️ Designing clean CLI interfaces ⚙️ Turning raw noise into meaningful diagnostics Check it out on GitHub 👇 https://lnkd.in/gnBpFnPi Feedback and contributions are always welcome! 🙌 #Python #CLI #OpenSource #SoftwareEngineering #BuildInPublic #DevTools #GitHub