We just published a step-by-step guide to building your first MCP server in Python. But the most important thing we learned building production MCP servers for clients isn't in any official doc: **Your tool description is worth more than your code.** Here's why: When an LLM decides whether to call your tool, it reads the docstring. Not the function name. Not the parameter types. The description. We had a client whose MCP tool for querying their CRM was called 300 times in a session — causing API rate limits. The tool description was: "Get customer data." We changed it to: "Retrieve full customer profile by customer ID. Use only when the user asks about a specific customer. Do not call multiple times for the same customer in one conversation." Calls dropped by 80%. Your tool descriptions are prompt engineering. Treat them that way. → Full tutorial with 6 steps + production deployment: https://lnkd.in/dK7skARH

Waiting on outdated scrapers feels like watching a loading bar that never ends. 😱 While traditional tools struggle with JavaScript and CAPTCHAs, your data pipeline doesn’t have to. Crawlbase is built for how the web actually works today so you can focus on using data, not chasing it. What you get: 🔹 Reliable JavaScript rendering 🔹 Automatic IP rotation at scale 🔹 Intelligent handling of blocks and CAPTCHAs 🔹 Clean, structured data delivered faster Less waiting. More doing. 👉 crawlbase.com #Crawlbase #WebScraping #DataEngineering #Automation #Developers #BigData #DataExtraction #AIAutomation #MachineLearning #APIs #SaaS #TechTools #DevTools #Programming #Python #NoCode #DataPipeline #GrowthHacking

🚀 𝗖𝗵𝗮𝗶𝗻𝗪𝗲𝗮𝘃𝗲𝗿 𝘃𝟬.𝟭.𝟬 𝗶𝘀 𝗼𝘂𝘁 A lot of “agent orchestration” is still doing something wasteful: Tool A → LLM decides → Tool B → LLM decides → Tool C That makes sense when the next step is genuinely uncertain. But in many cases, the flow is already known. And then those extra LLM calls add latency, cost, and unpredictability without adding much value. That’s the problem 𝗖𝗵𝗮𝗶𝗻𝗪𝗲𝗮𝘃𝗲𝗿 is built for. It’s an open-source Python library for compiling 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗺𝘂𝗹𝘁𝗶-𝘁𝗼𝗼𝗹 𝗳𝗹𝗼𝘄𝘀 into 𝗟𝗟𝗠-𝗳𝗿𝗲𝗲 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 𝗽𝗮𝘁𝗵𝘀 𝗯𝗲𝘁𝘄𝗲𝗲𝗻 𝘀𝘁𝗲𝗽𝘀. So instead of re-reasoning at runtime, you define the flow once and execute it in a predictable, testable way. 𝗪𝗵𝗮𝘁’𝘀 𝗶𝗻 𝘃𝟬.𝟭.𝟬 🔧 @𝘁𝗼𝗼𝗹 𝗱𝗲𝗰𝗼𝗿𝗮𝘁𝗼𝗿 Wrap a plain Python function with much less boilerplate. ChainWeaver uses type hints to build the input schema automatically. 🏗️ 𝗙𝗹𝗼𝘄𝗕𝘂𝗶𝗹𝗱𝗲𝗿 𝗳𝗹𝘂𝗲𝗻𝘁 𝗔𝗣𝗜 Build flows with a cleaner, more Pythonic interface instead of assembling the objects manually. 🔀 𝗗𝗔𝗚-𝗯𝗮𝘀𝗲𝗱 𝗳𝗹𝗼𝘄 𝘀𝘂𝗽𝗽𝗼𝗿𝘁 Model flows with explicit dependencies, branching structure, and topological execution order — not just simple linear chains. The goal is simple:  • fewer unnecessary LLM calls  • lower latency and cost  • more reproducible tool orchestration  • easier testing and debugging 📦 pip install chainweaver 🔗 https://lnkd.in/eBqCCnke Curious to see where this is useful — especially for MCP-heavy or multi-tool agent workflows. #Python #AI #Agents #OpenSource #MCP #LLMOps

🚀 Day 9/10 — Optimization Series Config-Driven Pipelines (Avoid Hardcoding) 👉 Basics are done. 👉 Now we move from working code → optimized code. You build a pipeline… It works perfectly… But you hardcode everything 😐 file_path = "data/sales_2024.csv" api_url = "https://lnkd.in/gsfHEDWP" 👉 Looks simple… but becomes a problem later. 🔹 The Problem Hard to update values ❌ Not reusable ❌ Breaks across environments ❌ 🔹 What is Config-Driven Approach? 👉 Move all dynamic values to a config file 🔹 Example (config.json) { "file_path": "data/sales_2024.csv", "api_url": "https://lnkd.in/gsfHEDWP" } 🔹 Use in Python import json with open("config.json") as f: config = json.load(f) file_path = config["file_path"] api_url = config["api_url"] 🔹 Why This Matters Easy to update 🔄 Reusable pipelines ♻️ Environment-friendly 🌍 🔹 Real-World Use 👉 Dev / Test / Prod configs 👉 Data pipelines 👉 API integrations 💡 Quick Summary Config-driven = flexible + scalable pipelines 💡 Something to remember If your values change often… they don’t belong in your code. #Python #DataEngineering #LearningInPublic #TechLearning

🚀 rst-queue v0.1.6: Scaling Terabytes with Megabytes In a world of bloated data systems, we often find ourselves throwing more hardware at software problems. But what if our tools were engineered to be small, grounded, and incredibly powerful? Introducing rst-queue v0.1.6, a high-performance async queue system built for the modern developer who values efficiency above all else. Inspired by the psychology of the Leafcutter Ant, this project is the first major release from the Datarn initiative. Why rst-queue? Most Python-based queues are limited by the Global Interpreter Lock (GIL) and high memory overhead. rst-queue is different. By using Rust and the Crossbeam framework, we’ve built a system that: ⚡ Bypasses the GIL: Achieve true parallelism with native Rust worker pools. 🐜 Microscopic Footprint: 30-50x less memory usage than traditional message brokers. 🛡️ Dual Modes: Choose between AsyncQueue (In-memory for 1M+ items/sec) or the new AsyncPersistenceQueue (Durable storage with Sled KV). Grounded in the Kernel The secret to our speed is "Simple OS Layering." We’ve designed rst-queue to sit as close to the OS kernel as possible, utilizing direct system calls and memory-mapped I/O. This isn't just a library; it's a high-velocity data crossing (Taran) for your most critical applications. Get Started in Seconds We believe in zero-setup excellence. You can add high-performance queuing to your Python project with a single command: Bash pip install rst-queue==0.1.6 Join the Datarn Movement At Datarn, we are building a suite of "Small but Mighty" tools for data-intensive domains like B2B e-commerce and real-time analytics. rst-queue is just the beginning. Explore the project on PyPI: https://lnkd.in/d54yqdea Contribute on GitHub: https://lnkd.in/d_x3E-zj #Python #RustLang #DataEngineering #OpenSource #Efficiency #Datarn #PerformanceOptimization #SoftwareArchitecture

DAY 5 - THE TECH STACK (Tools & Why I Chose Them) Every tool in my stack was a deliberate choice. Here's why - and what I'd tell my past self about each one. No random picks. No "I've always used this." Every decision had a reason. > Python :Primary language for the crawler and extraction logic. Chose it for its rich ecosystem - BeautifulSoup, Playwright, asyncio - nothing else comes close for this use case. Wish I knew: Async Python has footguns. Mixing sync and async code will ruin your day silently. > Playwright (Headless Browser) : Handles JS-rendered pages that raw HTTP requests miss completely. Chose it over Puppeteer for Python-native support and better async handling. Wish I knew: Resource usage adds up fast at scale. Spin up only what you need. > Message Queue (e.g., Redis / RabbitMQ) : Acts as the backbone between crawlers and processors - total decoupling. Chose it because I needed workers to fail independently without cascading crashes. Wish I knew: Queue monitoring is not optional. A silent backlog will sneak up on you. > Database (Structured Storage) : Stores normalized technographic profiles per company, queryable and enriched. Chose a relational model to keep tech profiles structured and joinable. Wish I knew: Schema design early saves migrations later. Do not skip this. > Docker: Containerized each service so workers can be spun up and torn down without friction. Chose it for portability and reproducibility across environments. Wish I knew: Docker networking between containers is its own learning curve. Start simple. Which tool here are you most curious about? Happy to go deeper on any of them 👇 #TechStack #Python #DistributedSystems #WebScraping #BackendDevelopment

 My FastAPI Journey — and things are getting exciting! Module 2: Core Basics Coming from a Django background, I've been exploring FastAPI and today I completed Module 2: Core Basics — and I'm genuinely blown away by how clean and powerful it is! 🔥 What I built today: A complete Notes REST API with full CRUD operations using just Python! ✅ Here's what I implemented: 📌 GET /notes — Fetch all notes from the database 📌 GET /notes/{note_id} — Fetch a specific note using Path Parameters 📌 POST /notes — Create a new note with auto-generated ID 📌 PUT /notes/{note_id} — Update an existing note by ID 📌 DELETE /notes/{note_id} — Delete a note and return confirmation 💡 Key concepts I learned: 🔹 Path Parameters — Passing values directly in the URL like /notes/1 to fetch specific records. FastAPI automatically converts and validates the data type! 🔹 Query Parameters — Filtering data using URL params like /search?keyword=Python — no extra configuration needed! 🔹 Request Body — Sending structured JSON data to the server using Pydantic models for automatic validation 🔹 Response Model — Controlling exactly what data is returned to the client for security and clean APIs 🔹 Auto Swagger Docs — FastAPI automatically generates interactive API documentation at /docs — no extra tools needed! ⚡ What makes FastAPI special compared to Django: → No need for separate serializers — Pydantic handles validation automatically → No need for urls.py — routes are defined directly on functions → Auto generated API docs out of the box → Built on ASGI (Uvicorn) making it significantly faster This is just the beginning — next up: 📦 Pydantic Data Validation 🗄️ Database Integration with SQLAlchemy 🔐 JWT Authentication 🚀 Deploying a Production API Learning in public — follow along for daily updates! 💪 #FastAPI #Python #REST #API #BackendDevelopment #Django #LearningInPublic #100DaysOfCode #WebDevelopment #SoftwareEngineering #CRUD

If you're a Claude Code user, check out these terminal tools! Glad to see Starship and CShip getting the love they deserve!

I wrote a new article on dev.to about building a multi-agent pipeline with Microsoft's new Agent Framework. The use case: reading a product backlog from Markdown and creating Epics and Stories on Jira automatically. I had already built the same pipeline with CrewAI (amazing framework), so I already knew the dependency setup and could jump straight into the code. The framework is radically explicit. Agent orchestration is just async Python. MCP integration is native and works well. Full walkthrough from hello world to modular pipeline with Pydantic validation. All code on GitHub (https://lnkd.in/da-YA6Km). #AI #AgentFramework #Python #MCP #OpenAI #Microsoft

Half my context window was gone before I typed a single prompt. Claude Code indexed my entire monorepo at session start — Python files, Airflow DAGs, three months of task logs. Then it generated a migration that referenced a table that doesn't exist. I spent weeks rebuilding my project setup from scratch. Token usage dropped over 60%. But the real win was rework time going down significantly. Here's what actually moved the needle: - permissions.deny in settings.json — the official way to block files Claude shouldn't read. Read(./.env), Read(./airflow/logs/), Read(./.venv/). The airflow/logs line alone cut 15%. - .claudeignore — an unofficial shortcut that works like .gitignore. Not in the docs yet, but a lot of people use it. Same result, cleaner syntax. - CLAUDE.md hierarchy — root file under 200 lines. Subdirectory files load only when needed. Past 200 lines, Claude starts treating your instructions as optional. - MCP servers (BigQuery + Airflow) — live database access without pre-loading schemas into context. Deferred by default, costs almost nothing until Claude actually queries one. - Skills & agents — on-demand workflows at ~100 tokens each instead of 3,000-5,000 tokens baked into CLAUDE.md every session. - /compact and /context — the two commands I run multiple times a day to manage what's eating my context window. 30 minutes of setup. Every session after that starts lean. Full walkthrough with real configs from a data pipeline project: https://lnkd.in/gaNuSUta -- What does your Claude Code project setup look like? Are you using permissions.deny or .claudeignore — or just letting it index everything? #AICoding #SoftwareEngineering #DataEngineering #ClaudeCode #DeveloperTools #AIEngineering #SystemDesign