Ibis SQL for 22+ Databases with Python

NEW VIDEO Text-to-SQL: Create Logical Plans & Prune Your Schema w/ piglets & Apex-SQL https://lnkd.in/eqeFnfB7 In this video I introduce two techniques from the Apex-SQL paper which can be used to pre-process context in your text-to-SQL workflow. This is also the first time I'm revealing a python library I've been working on: piglets 🐷 (link in the comments) piglets is an open source modular text-to-SQeaL toolkit. This is in pre-release so expect the API to change regularly. piglets is something I believe needs to exist, there are many monolithic text-to-SQL solutions all with different trade-offs, piglets empowers users to bolt on the parts they believe are missing from their current solution for free. For example: piglets currently enables users to drastically reduce the schema (and therefore the context and tokens) that would be needed to give a text-to-SQL solution enough context to answer a given natural language question. piglets is bring your own LLM (from any provider) and compatible with BigQuery, Snowflake and many other traditional SQL dbs (other major cloud data warehouse connectors are coming soon).

PydanTable 1.17.0 has been released, and MongoDB is now officially part of the story. This release introduces an optional MongoDB execution engine, allowing work to remain on the MongoDB database side when supported. This means you can materialize data only when it is actually needed in the application, rather than pulling full result sets into Python first. Additionally, this version adds integration with Beanie, a popular Python ODM (object-document mapper) for MongoDB built on Pydantic. If your application already models MongoDB documents with Beanie, PydanTable can seamlessly integrate with that layer, ensuring your document models and typed, table-shaped workflow remain aligned without the need for a parallel schema. For more details, check out the documentation and release notes: - PyPI: https://lnkd.in/ez4NZMjT - Documentation: https://lnkd.in/eV4RTqZQ - Repository: https://lnkd.in/eVpjrcRX #Python #Pydantic #MongoDB #DataEngineering #OpenSource

You won’t master SQL in a month ❌ You won’t master Python in a month ❌ You won’t master PySpark in a month ❌ But here’s what actually works 👇 --- 🔷 SQL 💻 → Solve 1 problem every day Resources → LeetCode, HackerRank, StrataScratch --- 🔷 Python 🐍 → Write scripts on weekends Resources → Codebasics, CodeWithHarry --- 🔷 PySpark ⚡ → Spend 30 mins daily understanding concepts Resources → Darshil Parmar, Deepak Goyal, Shubham Wadekar --- ▪️You don’t need to study like crazy ❌ ▪️You just need to improve a little every day 💡 --- Here’s the truth most people ignore 👇 ▪️1.00^365 = 1 ▪️1.01^365 = 37.7 🚀 --- Do nothing → you stay the same ❌ Improve 1% daily → massive growth 📈 --- 🔹Small steps 🔹Every day That’s your real advantage 🧠🔥 --- 🔸Save this 🔸Stay consistent 🔸Trust the process 🚀 --- #dataengineering #sql #python #pyspark #learningjourney #consistency #careergrowth

SQLAlchemy 2.0 made querying simpler — but also more explicit. When working with Async SQLAlchemy in FastAPI, one important thing to understand is: 👉 Query building is synchronous 👉 Execution is asynchronous Most confusion comes from mixing these two. In this post, I’ve focused on the core query building patterns you’ll use daily: ✔️ select() — building the base query ✔️ where() — filtering data ✔️ join() — working with relationships And then executing them using: 👉 await session.execute() No unnecessary theory — just practical patterns that map closely to SQL. 📌 This is Part 1 of a series: Part 2 → Execution Layer Part 3 → Insert, update, delete + transactions If you're using FastAPI with PostgreSQL, this will make your ORM usage much clearer. 💬 Do you prefer explicit queries style like this, or more ORM abstraction like in the previous version? #sqlalchemy #fastapi #postgresql #python #backenddevelopment

I just published a piece on something I keep seeing in Python APIs: using SQLAlchemy by default — even when it’s not needed After working more directly with PostgreSQL, I started questioning this habit. Because the database is not just storage — it’s a core part of performance and system behavior. In many APIs, especially simple or performance-critical ones, I’ve found that: - ORM adds unnecessary abstraction - raw SQL gives better control over query shape - PostgreSQL features are easier to leverage directly and in some cases, it actually improves performance due to lower overhead So I wrote about: ->> when ORM makes sense ->> when it becomes overengineering ->> and why I prefer asyncpg + raw SQL in many cases Do you stick with ORM everywhere, or go raw SQL when performance matters? https://lnkd.in/dzZ7xvCS #python #postgresql #fastapi #backend #softwareengineering

Day 10/30 — Social Network Analyzer (Python + MySQL) 🔹 Project Overview: Developed a Social Network Analyzer system using Python and MySQL to model user relationships, analyze connections, and recommend new links using graph-based algorithms. 🔹 Tools Used: Python | MySQL | Data Structures | Graph Algorithms | NetworkX | Matplotlib 🔹 Key Features: • Designed relational database to manage users and connections • Built graph structure to represent real-world relationships • Implemented BFS to find shortest connection paths • Identified mutual connections between users • Developed recommendation engine based on shared connections • Added network visualization for interactive analysis • Created CLI-based interface with clean and colored output 🔹 What I Learned: • Applying graph algorithms in real-world scenarios • Working with MySQL for structured data management • Building scalable backend logic using Python • Visualizing relationships using network graphs • Designing modular and maintainable code 🔗 GitHub Repository: https://lnkd.in/dpSCzhQG Would appreciate your feedback and suggestions 🙌 #30DaysOfCoding #PythonProjects #SQL #DataStructures #BackendDevelopment #LearningByDoing

UDFs vs. Native Functions: Don’t Reinvent the Wheel (It Just Gets Slower)! Are you a Python master who loves creating custom functions (UDFs) to apply to your PySpark DataFrames? 🛑 Stop right now! Although UDFs may seem like the perfect solution, they are Spark performance’s “kryptonite” because they force data serialization between the JVM (Java) and Python, killing speed.   The evolution of PySpark has brought hundreds of native functions in the pyspark.sql.functions module. Using native functions instead of Python UDFs can make your code 10 to 100 times faster, since they run directly on the JVM with the Catalyst Optimizer (Source: Spark Performance Benchmark, 2022).   Want a practical example? Instead of creating a UDF to convert strings to uppercase and add a value, look for native functions such as upper(), concat(), when(), col(). Almost everything you need already exists in an optimized form! Leave UDFs only for extremely complex business logic that cannot be solved with built-in functions. It’s the difference between riding a bicycle and taking a rocket!   What was the most complex UDF you managed to replace with native functions? Tell us in the comments!   #DataEngineering #PySpark #Productivity

Ever stuck with unstructured data in Excel sheets or spreadsheets and needed to push that messy data into a structured database? 🤯 Recently, I faced a similar challenge, a large spreadsheet filled with inconsistent, unstructured data that needed to be transformed into multiple clean tables. Doing it manually would’ve been time consuming and error prone. Here comes Python 🐍 Instead of struggling with manual cleanup, I built a small data pipeline using Python to automate the entire process from parsing and structuring the data to inserting it directly into a PostgreSQL Supabase database. What could’ve taken hours was reduced to minutes with better accuracy and scalability. As software engineers, knowing the right tool can turn a messy problem into an elegant solution. #Python #DataEngineering #Automation #PostgreSQL #Supabase #SoftwareEngineering

I was loading CSV files into SQL Server. It was slow. Then I switched to BULK INSERT. 💥 Everything changed. BULK INSERT is a native SQL method. It is built for speed. But the real power comes when you combine it with Python. 𝗪𝗵𝗮𝘁 𝗱𝗶𝗱 𝗜 𝗱𝗼? ✔️ Python to handle multiple CSV files ✔️ Python to clean and normalize data ✔️ BULK INSERT for fast loading into SQL Server This combination is simple. And very powerful. Python manages flexibility. SQL manages performance. 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁: a faster ingestion process a cleaner pipeline a more reliable system 𝗗𝗮𝘁𝗮 𝗶𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻 𝗶𝘀 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝗹𝗼𝗮𝗱𝗶𝗻𝗴. 𝗜𝘁’𝘀 𝗮𝗯𝗼𝘂𝘁 𝘀𝗽𝗲𝗲𝗱 𝗮𝗻𝗱 𝗰𝗼𝗻𝘁𝗿𝗼𝗹. Curious how it works? 🔗 GitHub repository: https://lnkd.in/dwjwP-bh P.S.  I know BULK sounds a lot like “HULK”… not very original, but I like it 😄

1.2s → 85ms. The LATERAL join 90% of devs have never written. Four nested Python loops. One slow endpoint. I replaced all of it with one Postgres LATERAL join. Most backend devs don't know it exists. Full breakdown (7 min read) → https://lnkd.in/gyPiyjsQ #PostgreSQL #Database #SQL #BackendEngineering #DataEngineering