Transform Sybase ASE with Python for Smarter Operations

UDFs vs. Native Functions: Don’t Reinvent the Wheel (It Just Gets Slower)! Are you a Python master who loves creating custom functions (UDFs) to apply to your PySpark DataFrames? 🛑 Stop right now! Although UDFs may seem like the perfect solution, they are Spark performance’s “kryptonite” because they force data serialization between the JVM (Java) and Python, killing speed.   The evolution of PySpark has brought hundreds of native functions in the pyspark.sql.functions module. Using native functions instead of Python UDFs can make your code 10 to 100 times faster, since they run directly on the JVM with the Catalyst Optimizer (Source: Spark Performance Benchmark, 2022).   Want a practical example? Instead of creating a UDF to convert strings to uppercase and add a value, look for native functions such as upper(), concat(), when(), col(). Almost everything you need already exists in an optimized form! Leave UDFs only for extremely complex business logic that cannot be solved with built-in functions. It’s the difference between riding a bicycle and taking a rocket!   What was the most complex UDF you managed to replace with native functions? Tell us in the comments!   #DataEngineering #PySpark #Productivity

I’m pleased to share my recent project: a Distributed Caching System in Python, designed to improve application performance by reducing database dependency and network latency. The system leverages core data structures (LRU using HashMap and Doubly Linked List) along with object-oriented design principles to deliver efficient in-memory data retrieval. It also includes multi-node distribution, TTL-based expiration, REST API endpoints, and a Python SDK for seamless integration. Additional features such as rate limiting, logging, and a SQL-backed demo were implemented to simulate real-world backend scenarios. Through benchmarking, the system demonstrates a clear improvement in response time, with repeated requests being served from cache instead of querying the database. https://lnkd.in/d2-EGs7v This project was a valuable exercise in backend system design, performance optimization, and building production-oriented solutions. #Python #BackendEngineering #SystemDesign #Caching #APIs #SoftwareDevelopment #DistributedSystems

I published my first open-source library — uow-lib A minimalist Unit of Work implementation for Python 3.12+ that works with any async driver (asyncpg, aiosqlite) and doesn't force an ORM on you. What it does: – Tracks entity mutations automatically via instrumentation – Guarantees one in-memory object per identity (Identity Map) – Orders operations by dependency: INSERTs parents before children, DELETEs children before parents – Auto-registers child entities when added to a collection What you get: – Models are plain classes — domain stays clean from infrastructure – Full control over SQL in your DataMappers – Works with query builders, raw SQL, or any Connection that implements commit/rollback – Applicable to NoSQL if your store supports transactions – No need to manually track changes in your models First release, so rough edges are expected. If you try it and something breaks or feels off — open an issue. That's the fastest way to make it better. 📦 https://lnkd.in/eJbmXitV 🔗 https://lnkd.in/eV2n-JAx #python #opensource #unitofwork #architecture #asyncio

🚀 Day 1 of SQL Series! Starting from the absolute basics — because strong fundamentals beat shortcuts every time. Today I covered: ✅ What is SQL & DBMS ✅ Types of SQL Commands (DDL, DML, DQL, DCL, TCL) ✅ SQL Datatypes (CHAR, VARCHAR, NUMBER, DATE, CLOB, BLOB) One image. Everything you need to know on Day 1. Drop a 🔥 if you're learning SQL too! #SQL #DataScience #100DaysOfCode #LearningInPublic #Python #Database

1.2s → 85ms. The LATERAL join 90% of devs have never written. Four nested Python loops. One slow endpoint. I replaced all of it with one Postgres LATERAL join. Most backend devs don't know it exists. Full breakdown (7 min read) → https://lnkd.in/gyPiyjsQ #PostgreSQL #Database #SQL #BackendEngineering #DataEngineering

I found a data type I had never seen before Today I was reading about data types in Rust. Many people hate Rust for all kinds of reasons — that is actually what made me curious enough to look for myself. But that is beside the point. During my study I found something very interesting. The classification of types is very different from traditional languages like C++, Python, and Java. Data types are classified into scalar types and compound types — which are further broken down into many more. And at the very bottom of this classification I found a type of integer I had never seen anywhere else. Architecture dependent integers. usize and isize. Here is everything that makes them interesting: → Their size is not fixed. On a 32-bit system they are 4 bytes. On a 64-bit system they are 8 bytes. The processor decides, not the developer. → They are used exclusively for indexing and memory addressing — because they are guaranteed to address any byte in memory on the system they run on. → Code written with usize works on both 32-bit and 64-bit architectures without a single modification. I had never encountered a type whose size is defined by the machine it runs on rather than the language specification. Most languages abstract this away entirely. Rust exposes it — and makes it useful. #Rust #SystemsProgramming #Programming #LearningInPublic #SoftwareEngineering

Spark Connect — I kept seeing this term but never really understood what problem it was solving. So I dug deeper. Before Spark Connect, the client and Spark driver were tightly coupled. Your PySpark script ran directly inside the driver process. This meant: → Heavy dependency overhead (matching Java, Scala, Python versions) → Client crashes could take down the driver → Building non-JVM clients was a difficult process → PySpark relied on Py4J to bridge into the driver's JVM Spark Connect changes all of this by clearly separating the client and the server. Here's the simplified flow: 1. The client converts your DataFrame or SQL query into an Unresolved Logical Plan 2. That plan is serialized using Protocol Buffers 3. Sent to the Spark server via gRPC 4. The server deserializes, optimizes, and executes it 5. Results come back as Apache Arrow record batches — streamed, not dumped all at once The result? The client no longer needs a full Spark installation. The server can be updated independently. And since the entire communication stack (gRPC + Protobuf + Arrow) is language-agnostic, building Spark clients in Python, Go, Rust — much simpler. Check out my detailed write-up on Spark Connect on Medium 👇https://lnkd.in/gmZegTXn #ApacheSpark #PySpark #SparkConnect #DataEngineering 

Zero-copy data sharing between Python, Java, C++ without serialization overhead. That's Apache Arrow. Arrow is not a file format. It's an in-memory columnar format. The Arrow ecosystem: Arrow: language-agnostic in-memory columnar format. Arrow Flight: high-performance RPC over gRPC. No more JSON/CSV serialization tax. Flight SQL: send and execute SQL queries over Flight. ADBC: standardized API for Arrow-native database interactions. DuckDB, Polars, DataFusion, Spark - they all use Arrow under the hood. If you're building anything that moves data between processes, Arrow is the standard to know. https://lnkd.in/dfXtSSwN

Databricks Notebook A Databricks Notebook is a document that allows you to execute code in multiple languages (like Python, SQL, Scala, and R), visualize results, and combine code with text, images, and charts in a single place. Multi-language support – Python, SQL, Scala, R Interactive execution – Run code step by step Visualization – Create charts and graphs instantly Collaboration – Multiple users can work together Integration with big data – Works with Apache Spark

PART 3 of designing my python final! In the previous post, I outlined the overall system and its scope. With that defined, the next step was structuring the application. A three-layer architecture was used: • GUI Layer (Tkinter frames) • Service Layer (business logic and validation) • DAO Layer (database operations) Each layer interacts only with the one directly below it, ensuring proper separation of concerns. The GUI never directly executes SQL queries, and all data access is routed through DAO functions. With the structure in place, the focus shifted to how different users would interact with the system. DAY 12 #SystemDesign #CleanArchitecture