DAY 5 - THE TECH STACK (Tools & Why I Chose Them) Every tool in my stack was a deliberate choice. Here's why - and what I'd tell my past self about each one. No random picks. No "I've always used this." Every decision had a reason. > Python :Primary language for the crawler and extraction logic. Chose it for its rich ecosystem - BeautifulSoup, Playwright, asyncio - nothing else comes close for this use case. Wish I knew: Async Python has footguns. Mixing sync and async code will ruin your day silently. > Playwright (Headless Browser) : Handles JS-rendered pages that raw HTTP requests miss completely. Chose it over Puppeteer for Python-native support and better async handling. Wish I knew: Resource usage adds up fast at scale. Spin up only what you need. > Message Queue (e.g., Redis / RabbitMQ) : Acts as the backbone between crawlers and processors - total decoupling. Chose it because I needed workers to fail independently without cascading crashes. Wish I knew: Queue monitoring is not optional. A silent backlog will sneak up on you. > Database (Structured Storage) : Stores normalized technographic profiles per company, queryable and enriched. Chose a relational model to keep tech profiles structured and joinable. Wish I knew: Schema design early saves migrations later. Do not skip this. > Docker: Containerized each service so workers can be spun up and torn down without friction. Chose it for portability and reproducibility across environments. Wish I knew: Docker networking between containers is its own learning curve. Start simple. Which tool here are you most curious about? Happy to go deeper on any of them 👇 #TechStack #Python #DistributedSystems #WebScraping #BackendDevelopment
My Tech Stack: Python, Playwright, Redis, Database, Docker
More Relevant Posts
-
Most codebases are still navigated with grep… and a lot of guesswork. When you jump into a new repo—at work or in open source—you’re often stuck: → Guess a keyword → Open a file → Repeat… across 50 tabs It’s slow, frustrating, and breaks your flow. I got tired of that. So I built something to change it. Meet Ziv — a local semantic search tool for Python codebases. Instead of guessing keywords, you just ask: → ziv search "where is request context handled?" → ziv search "session management" Ziv understands intent and returns the most relevant files—ranked by meaning, not exact matches. No noise. No digging. Just answers. Under the hood: • FAISS for fast similarity search • all-MiniLM-L6-v2 (ONNX) for embeddings • Optimized to run efficiently on CPU And everything stays local: • No cloud • No API keys • No code leaving your machine It’s open source. It’s free. And it’s now in public beta. If you work with Python or contribute to open source, I’d love for you to try it. If it breaks, open an issue. If it helps, a ⭐ on GitHub would mean a lot right now. Links are in the comments 👇 #Python #OpenSource #DevTools #BuildInPublic #MachineLearning
To view or add a comment, sign in
-
-
Production-ready multi-agent orchestration is finally here. With Agent Framework v1.0 introducing some exciting new features. My personal favourites: 1. Agent workflows 2. Middleware hooks 3. Integration with managed agents in Foundry Agent Service 4. Multi-agent orchestration with HITL 5. Skills 6. Foundry Tools, Memory, Observability and Evaluations and many more. Check out the amazing blog post from Shawn Henry detailing the various features with concrete examples in Python and .NET https://lnkd.in/dKBRDujY
To view or add a comment, sign in
-
🚀 Small Learning, Big Impact! While working on a Web application with Flask, I came across a subtle but important best practice that can save a lot of debugging time 👇 ❌ What I used to do: <𝚕𝚒𝚗𝚔 𝚛𝚎𝚕="𝚜𝚝𝚢𝚕𝚎𝚜𝚑𝚎𝚎𝚝" 𝚑𝚛𝚎𝚏="𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜"> At first glance, this seems fine. But here’s where things go wrong 👇 In Flask, static files (like CSS, JS) are not served directly from the root. They are served through a configured path: - "𝚜𝚝𝚊𝚝𝚒𝚌_𝚏𝚘𝚕𝚍𝚎𝚛" → the actual directory in your project (default: /𝚜𝚝𝚊𝚝𝚒𝚌) - "𝚜𝚝𝚊𝚝𝚒𝚌_𝚞𝚛𝚕_𝚙𝚊𝚝𝚑" → the URL route used to access those files (default: /𝚜𝚝𝚊𝚝𝚒𝚌) So when you write: <𝚕𝚒𝚗𝚔 𝚛𝚎𝚕="𝚜𝚝𝚢𝚕𝚎𝚜𝚑𝚎𝚎𝚝" 𝚑𝚛𝚎𝚏="𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜"> 👉 The browser tries to fetch the file from: /𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜 But Flask actually serves it from: /𝚜𝚝𝚊𝚝𝚒𝚌/𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜 ⚠️ Now imagine this: - You rename "𝚜𝚝𝚊𝚝𝚒𝚌_𝚏𝚘𝚕𝚍𝚎𝚛" to "/𝚊𝚜𝚜𝚎𝚝𝚜/" - Or change "𝚜𝚝𝚊𝚝𝚒𝚌_𝚞𝚛𝚕_𝚙𝚊𝚝𝚑" to "/𝚛𝚎𝚜𝚘𝚞𝚛𝚌𝚎𝚜" Your file is now served from: /𝚛𝚎𝚜𝚘𝚞𝚛𝚌𝚎𝚜/𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜 But your hardcoded path still points to: /𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜 💥 Result? Broken styling. ✅ Correct Approach: <𝚕𝚒𝚗𝚔 𝚛𝚎𝚕="𝚜𝚝𝚢𝚕𝚎𝚜𝚑𝚎𝚎𝚝" 𝚑𝚛𝚎𝚏="{{ 𝚞𝚛𝚕_𝚏𝚘𝚛('𝚜𝚝𝚊𝚝𝚒𝚌', 𝚏𝚒𝚕𝚎𝚗𝚊𝚖𝚎='𝚜𝚝𝚢𝚕𝚎.𝚌𝚜𝚜') }}"> 🔷️ In Flask’s Jinja templates, 𝚞𝚛𝚕_𝚏𝚘𝚛() dynamically generates the correct URL for a given endpoint (like routes or static files) based on your app’s configuration. It prevents hardcoding paths, so links remain valid even if routes or folder structures change. 💡 Why this works: - Flask dynamically generates the correct URL based on "𝚜𝚝𝚊𝚝𝚒𝚌_𝚏𝚘𝚕𝚍𝚎𝚛" and "𝚜𝚝𝚊𝚝𝚒𝚌_𝚞𝚛𝚕_𝚙𝚊𝚝𝚑" - No hardcoding → no broken paths - Your app becomes flexible, scalable, and production-ready. 🔍 Key takeaway: Never hardcode static file paths in Flask. Let the framework handle it. Small practices like this make a big difference in writing clean, reliable backend systems. #Flask #Python #BackendDevelopment #LearningInPublic #DataScience #MachineLearning #AIML #SoftwareEngineering #CodingJourney
To view or add a comment, sign in
-
-
One instruction file doesn't scale. The moment your codebase has a Python service and a TypeScript frontend and a Go worker, a single CLAUDE.md becomes either too generic to be useful or too bloated to trust. Scoped context solves this the way filesystems already do — by nesting. Org-level rules wrap user-level rules wrap project-level rules wrap directory-level rules. The agent reads whichever scope it's working inside, the same way a developer picks up conventions walking into a new folder. Example: the org says "never commit .env files." The project says "use Zod for validation." The ./src/api/ directory says "return JSON, validate schema." The agent sees all three, cleanly composed. The trade-off is discoverability. When rules live in four places, it's harder to answer "what does the agent actually see right now?" Good tooling here isn't optional — it's the whole pattern. Treat context as a tree, not a file. How are you organizing rules across a multi-language codebase? #AI #AgenticAI #SoftwareArchitecture #DeveloperTools #Clausey
To view or add a comment, sign in
-
-
🚀✨ Sharing something I’ve been building — Code Crawler 🕷️ Ever joined a large Python codebase and spent days 😵💫 figuring out what calls what? Yeah… that frustration led me to build this 👇 💡 Code Crawler helps you understand any Python codebase visually and instantly. Just point it to a GitHub repo (or even a local folder 📂), and it parses everything using AST to generate a function & class-level lineage graph 🧬 🔍 What makes it powerful: ⚙️ Resilient crawl pipeline (Temporal) — survives restarts & scales smoothly 🔗 Cross-repo lineage — track relationships across repos + inheritance chains 🌿 Branch comparison — see what changed & what it impacts downstream ▶️ Run functions from UI — no test setup needed 🧪 Smart mocking — auto-detect dependencies & stub them easily 📊 Batch testing — run multiple test cases together with instant results 📂 Local support — drag & drop projects, no GitHub required 🏢 Multi-tenant architecture — full isolation + invites + admin panel 💭 Why I built this: Understanding a new codebase shouldn’t feel like solving a puzzle 🧩 Instead of jumping across hundreds of files: 👉 Visualize relationships 📈 👉 Trace flows instantly 🔄 👉 Run functions with real inputs ⚡ All in one place. 🚧 Still building, but already super useful for exploring codebases I didn’t write Would love feedback from folks who’ve struggled with large Python projects 🙌🔥 🔗 https://lnkd.in/gtD-5juu #Python 🐍 #SoftwareEngineering 💻 #DevTools 🛠️ #OpenSource 🌍 #FastAPI ⚡ #React ⚛️
To view or add a comment, sign in
-
#GitHub trending looks different this week. Not another JavaScript framework. Not another Python wrapper. Wall-to-wall Rust. claw-code hit 100K stars faster than any repository in GitHub history. Behind it: zeroclaw, ironclaw, pi_agent_rust, rtk, agent-browser. All Rust. All AI agent infrastructure. All shipped in April 2026. I've been writing C++ and Rust for embedded systems for years. Watching the AI tooling community discover what systems programmers already knew feels like deja vu. Why Rust is winning this specific race: Single binary deployment. No virtualenv. No Docker. No "works on my machine." Your agent ships as one file, runs everywhere. Memory safety without a garbage collector. AI agents run for hours, sometimes days. They ingest untrusted tool output, parse arbitrary JSON, manage dozens of concurrent connections. A memory leak at hour 47 is not acceptable. Neither is a use-after-free from a malformed API response. Startup in milliseconds. When your agent orchestrates 20 tool calls per minute, the runtime overhead matters. Python's import time alone is longer than Rust's entire cold start. The numbers back this up. rtk, a Rust CLI proxy, cuts LLM token consumption by 60 to 90 percent through intelligent caching. That's not a micro-optimization. That's the difference between viable and unaffordable at scale. Python built the AI prototype. Rust is building the AI production stack. #Rust #AI #OpenSource #DeveloperTools #LLM #SoftwareEngineering #GitHub
To view or add a comment, sign in
-
𝗦𝘁𝗼𝗽 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗲𝘃𝗲𝗿𝘆 𝗯𝗮𝗰𝗸𝗲𝗻𝗱 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝘄𝗮𝘆. FastAPI isn't just "another Python framework." It's a deliberate choice — and knowing when to reach for it matters more than knowing how to use it. 𝗣𝗶𝗰𝗸 𝗙𝗮𝘀𝘁𝗔𝗣𝗜 𝘄𝗵𝗲𝗻: • You're building ML/AI-powered APIs and your team already lives in Python • You need async performance without the boilerplate of Go or Java • Auto-generated docs (Swagger/OpenAPI) aren't a nice-to-have — they're a requirement • You want type safety that actually catches bugs before production 𝗦𝘁𝗶𝗰𝗸 𝘄𝗶𝘁𝗵 𝘁𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗯𝗮𝗰𝗸𝗲𝗻𝗱𝘀 (𝗦𝗽𝗿𝗶𝗻𝗴, 𝗗𝗷𝗮𝗻𝗴𝗼, 𝗘𝘅𝗽𝗿𝗲𝘀𝘀, .𝗡𝗘𝗧) 𝘄𝗵𝗲𝗻: • Your org already has deep expertise and infra around them • You need battle-tested ORM support and a massive plugin ecosystem • You're building monoliths where convention-over-configuration saves months 𝗧𝗵𝗲 𝗿𝗲𝗮𝗹 𝗮𝗻𝘀𝘄𝗲𝗿? 𝗜𝘁'𝘀 𝗻𝗲𝘃𝗲𝗿 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸. 𝗜𝘁'𝘀 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺. FastAPI shines where speed-to-deploy, async I/O, and Python-native ML pipelines intersect. Forcing it into a legacy enterprise CRUD app is like using a scalpel to chop wood. Choose your tools like an engineer, not a fan. Thoughts? When did FastAPI click (or not click) for you? #FastAPI #Python #BackendDevelopment #SoftwareEngineering #WebDevelopment #APIDevelopment #TechCommunity #Programming #MLOps #SystemDesign
To view or add a comment, sign in
-
-
📣 12,000 downloads. We shipped the first release 6 weeks ago. No launch event. No press. No paid promotion. Just engineers who found it, tried it, and kept using it. And 17 contributors who decided to build instead of wait. Here's what 12,000 downloads actually built: 33 LLM providers behind one interface. 45 agent tools. 46 document loaders. 20 retrieval strategies. 9 vector store backends. MCP client and server. Built-in RAG evaluation. OpenTelemetry observability. Production guardrails. EvalCI for quality gates in CI. Fully local RAG with GPT4All and vLLM. 2 hard dependencies. Async-native from day one. Apache 2.0. To every engineer who pip installed, opened an issue, sent a message, or pushed a PR: this is yours too. The door is still open. Many issues waiting. Good first issues stocked. PR reviews fast. ⭐ https://lnkd.in/d2fGSPkX #OpenSource #Python #LLM #AI #RAG #BuildInPublic #MachineLearning #SynapseKit
To view or add a comment, sign in
-
📣 We've run 22 benchmarks comparing LangChain, LlamaIndex, and SynapseKit across cold start, dependencies, lines of code, RAM, chunking, BM25, hybrid search, memory, async throughput, and more. Every number is reproducible. Every notebook is public on Kaggle. Fork and run yourself. Week 3 starts next week. Agents, tool calling, multi-agent orchestration. The scorecard is about to get interesting. engineersofai.com #Python #LLM #RAG #AI #EngineersOfAI #SynapseKit
📣 12,000 downloads. We shipped the first release 6 weeks ago. No launch event. No press. No paid promotion. Just engineers who found it, tried it, and kept using it. And 17 contributors who decided to build instead of wait. Here's what 12,000 downloads actually built: 33 LLM providers behind one interface. 45 agent tools. 46 document loaders. 20 retrieval strategies. 9 vector store backends. MCP client and server. Built-in RAG evaluation. OpenTelemetry observability. Production guardrails. EvalCI for quality gates in CI. Fully local RAG with GPT4All and vLLM. 2 hard dependencies. Async-native from day one. Apache 2.0. To every engineer who pip installed, opened an issue, sent a message, or pushed a PR: this is yours too. The door is still open. Many issues waiting. Good first issues stocked. PR reviews fast. ⭐ https://lnkd.in/d2fGSPkX #OpenSource #Python #LLM #AI #RAG #BuildInPublic #MachineLearning #SynapseKit
To view or add a comment, sign in
-
For the past few weeks, I've been building Termagent — a natural language terminal assistant for Windows, powered by Groq and LangGraph. What started as a small experiment turned into something I'm genuinely excited about. It's available as a Python package right now: pip install termagent-cli==1.1.1 Then just run: termagent Run 'termagent-reset' to reset your credentials Requires Node.js to be installed on system. Instead of remembering PowerShell syntax, you just tell it what you want in plain English. Here's what it can do: 🖥️ Terminal Control — Run PowerShell commands using plain English. "Create a folder called src and move all .py files into it" just works. 📧 Email — Compose and send emails with attachments directly from the terminal. Connects to Gmail via SMTP. Uses Gmail appPasswords. Create one for Termagent here 'https://lnkd.in/eBsRemp9' 📄 Documentation — Generate well-formatted .docx reports and documents with web-searched, up-to-date content. 💻 Coding Agent — Understands your entire project directory and writes or edits multiple files while maintaining context across all of them. Works like a lightweight "Claude Code"(not exactly), right in your terminal. 🐙 GitHub Integration — Commit, push, create releases, open issues, list PRs, and manage branches — all in plain English via the GitHub MCP server. 🎤 Voice Input — Press Ctrl+M to speak your command. Transcribed instantly via Groq's Whisper API. 📋 Clipboard — Ctrl+V to paste into the input, Ctrl+Shift+C to copy the last output. It'll walk you through setting up your Groq API key, email, and GitHub token on first run. Everything is saved locally in ~/.termagent/.env — your credentials never leave your machine. ⚠️ Disclaimer: This is an early release — it may have bugs. That said, your credentials are stored locally and are completely safe. Also worth noting: the coding agent, GitHub features, and document generation can result in higher token usage on your Groq account, so keep that in mind. I'd love to hear your feedback — what works, what breaks, and what features you'd want next. #Python #AI #LLM #Groq #LangChain #LangGraph #OpenSource #WindowsTerminal #DevTools #BuildInPublic
To view or add a comment, sign in
-
Explore related topics
Explore content categories
- Career
- Productivity
- Finance
- Soft Skills & Emotional Intelligence
- Project Management
- Education
- Technology
- Leadership
- Ecommerce
- User Experience
- Recruitment & HR
- Customer Experience
- Real Estate
- Marketing
- Sales
- Retail & Merchandising
- Science
- Supply Chain Management
- Future Of Work
- Consulting
- Writing
- Economics
- Artificial Intelligence
- Employee Experience
- Workplace Trends
- Fundraising
- Networking
- Corporate Social Responsibility
- Negotiation
- Communication
- Engineering
- Hospitality & Tourism
- Business Strategy
- Change Management
- Organizational Culture
- Design
- Innovation
- Event Planning
- Training & Development