How to connect Python to SQL Server using pyodbc and SQLAlchemy

Databricks just made it easier to connect any data source to Spark. With the Python Data Source API (now generally available), developers can build custom connectors entirely in Python — for both batch and streaming workloads — without touching JVM code. It’s a simple, powerful way to bring external data into your lakehouse and keep it governed with Unity Catalog.

Databricks just made it easier to connect any data source to Spark. With the Python Data Source API (now generally available), developers can build custom connectors entirely in Python — for both batch and streaming workloads — without touching JVM code. It’s a simple, powerful way to bring external data into your lakehouse and keep it governed with Unity Catalog.

Databricks just made it easier to connect any data source to Spark. With the Python Data Source API (now generally available), developers can build custom connectors entirely in Python — for both batch and streaming workloads — without touching JVM code. It’s a simple, powerful way to bring external data into your lakehouse and keep it governed with Unity Catalog.

Databricks just made it easier to connect any data source to Spark. With the Python Data Source API (now generally available), developers can build custom connectors entirely in Python — for both batch and streaming workloads — without touching JVM code. It’s a simple, powerful way to bring external data into your lakehouse and keep it governed with Unity Catalog.

We just published our blog on the Python Data Source API for you to extend and incorporate your custom data sources as part of your data pipeline. 🐍 Python Spark connectors - Build custom connectors for diverse data sources using the Python ecosystem 🔄 Batch & streaming support - Real-time data pipelines for structured/unstructured sources and destinations 🔒 Unity Catalog integration - Govern external data with lineage, access control, and auditability 📤 Declarative Pipeline sinks - Stream records to external services via Python data sources # Python #PythonDataSource #datapipelines #apacheSpark https://lnkd.in/gYuVajnp.

🚀 Exciting news for data engineers and Python developers! The latest release of Apache Spark (version 4.0 on Databricks Runtime 15.4 LTS +) now fully supports the Python Data Source API - making custom connector development easier than ever. ✅ Key highlights: - Develop readers and writers fully in Python—no JVM coding required. - Supports batch and streaming use cases out of the box. - Seamless Declarative Pipeline Integration: connectors defined once can plug directly into Spark SQL, DataFrames, and workflows. - Works with Unity Catalog for end-to-end governance and lineage. - Ideal for integrating APIs, ML datasets, and external or proprietary sources. Upgrade to Spark 4.0 or Databricks Runtime 15.4 LTS to start building Python-native connectors today. https://lnkd.in/gHYU3adK #ApacheSpark #Databricks #SparkDeclarativePipeline #PySpark #SparkSQL #StructuredStreaming #SparkStreaming #UnityCatalog #Connectors #Python #Spark4

🎉 Day 99: Final Python Push! I capped off my Python series today with an intensive focus on practical application and data persistence! This was a full-stack experience, demonstrating how Python moves beyond simple scripts into building real-world tools. 🛠️ Data Persistence: From File to NoSQL I built three versions of a "YouTube Manager" CLI application, mastering different data storage methods: File-Based (JSON): The basic version used the json module to save video data to a local file (youtube.txt). This taught me crucial concepts like error handling (try...except FileNotFoundError) and using json.dump and json.load for persistence. Relational (SQLite3): I migrated the project to use sqlite3, showcasing SQL operations (CREATE TABLE, INSERT, SELECT, UPDATE, DELETE) within Python. This is essential for understanding relational databases. NoSQL (MongoDB): I deployed the final version using the pymongo library, connecting to a MongoDB cluster. This introduced me to NoSQL concepts and using unique IDs (ObjectId) for document management. 🌐 Handling APIs in Python I also practiced interacting with external services using the requests library. The example demonstrated how to: Make a GET request to a public API endpoint. Parse the JSON response data. Extract specific nested data points (e.g., username and country). Implement basic response validation. 📈 Wrapping Up the Series My Python journey concludes here (for now!), having covered everything from fundamentals (data types, control flow) to advanced topics (OOP, Decorators, Scopes) and practical tools (APIs, Databases, Virtual Environments). I also touched on the broader ecosystem, including Conda and Jupyter Notebooks, before the farewell to the series. What a ride! Now, where should I apply this Python knowledge next? #Python #PythonProject #Databases #MongoDB #SQLite #APIs #SoftwareDevelopment #LearningInPublic

Python Data Source API is now Generally Available. This is a big deal for anyone shipping pipelines on Databricks. You can build read and write connectors in pure Python. No JVM. It runs on Spark 4.0 in DBR 15.4 LTS and above, including Serverless. What I care most about is speed and costs tbh... and at least that's the promise. The API is built on Apache Arrow. Data moves between your connector and Spark with minimal overhead. Fewer copies, fewer conversions. If you currently fetch from an API dump to temp, then load to Spark, you can skip steps and cut costs. It works for batch and streaming. You can pull from event sources in real time or push out to external sinks, and you can expose the source to SQL once you register it, which keeps analytics workflows simple. Governance stays in place. You can read from custom sources and write to Unity Catalog tables with lineage, ACLs, and audits.. so “non-native” data is controlled. What I expect in practice is: - Lower ingest latency for Python connectors because Arrow reduces serialization overhead. - Less intermediate I/O. - Simpler pipelines because the connector lives in your Python ecosystem. - Clearer debugging since the data path is shorter. I’ll start with APIs where I still do fetch -> stage -> load. I’ll measure time, batch size and the share of runtime spent serializing. _____________________ What will you connect first with Python? #databricks #python #dataengineer

🚀 New in Snowpark: Read External Data Sources Directly with the Python DB API is now GA! You can now connect Snowpark directly to external databases and query them right from your Snowflake environment — no data movement required. This new Python DB API integration makes it easier to bring data from systems like PostgreSQL, MySQL, and others into your Snowflake workflows for seamless analysis and data engineering. Another big step toward simplifying multi-system data access — all through Snowpark. 🔗 Learn more: https://lnkd.in/dFd4VvK4 #Snowflake #Snowpark #Python #DataEngineering #DataCloud #Innovation

How modernising your Python–PostgreSQL stack can pay off As an architect focused on database performance (especially with psycopg3 + PostgreSQL), I found key take-aways from this article that speak directly to high-throughput, low-latency systems. ✅ Key Takeaways • Native async I/O support: With AsyncConnection and AsyncCursor, your Python code can avoid blocked threads and better scale in event-loop frameworks. • Better connection pooling and resource use: Psycopg3 offers more advanced pool semantics compared to older drivers. • Improved row factories (custom result mappings) and more fluent API: This helps cleaner code and less plumbing in your DB layer. • Enhanced COPY / streaming capabilities: Ideal for bulk loads or real‐time ingestion pipelines. • Support for SQL composition and safe type binding: Reduces risk of injection, improves maintainability. • Better type adaptation (Python–PostgreSQL) and binary protocol usage: Means less overhead and more native leverage of PostgreSQL capabilities. • Overall: The art of “fast” isn’t just hardware—it’s the driver, the integration, the code. As the article states: “fast isn’t a feature; it’s a habit.” 🎯 Why this matters for database and application architects • If you manage high-throughput OLTP/real-time ingestion (as in many of my audits), you’ll benefit from reducing idle time, blocking threads, inefficient pools, legacy drivers. • For migrations or modernisation efforts (e.g., older Python + PostgreSQL stacks), choosing a driver that supports async, binaries, advanced pooling gives you a layer of competitive advantage. • Cleaner code + modern driver = less fragility, easier readability, fewer surprises when scaling. • With architectures involving hybrid systems (e.g., OLTP + OLAP decoupling, CDC pipelines, etc.) every micro-improvement counts, and driver-level upgrades are low hanging fruit. 🔍 My recommendation for teams at Mydbops or similar • Evaluate your Python–Postgres stacks: Identify if you’re still using older drivers (e.g., psycopg2) or synchronous patterns that block threads. • Prototype a migration to psycopg3: Measure metrics like latency, connection reuse, threads blocked, memory usage under load. • Pair this with best practices in schema/indexing/partitioning (your domain) to maximise end-to-end performance. • Document the transition path and rollback options—driver changes still carry risk. • Communicate the business value: better responsiveness, fewer connection overheads, improved maintainability. Would you like me to include the blog link in the post text so readers can click directly through? Here’s the link again: https://lnkd.in/gXbYqSi5 ⸻ What do you think — ready to post? Or would you like me to craft a version in Tamil or with a different tone? #PostgreSQL #Python #DatabasePerformance #Psycopg3 #DevOps #ScalableSystems