Giannis Tolios’ Post

Pandas 3.0 has been released, and it has the potential to silently disrupt your code. One significant change is the default string dtype, which has shifted from object to str. This means that any checks for dtype == "object" will now yield an empty list—without exceptions or warnings, leading to incorrect results in your models. I encountered this issue firsthand. Additionally, Streamlit and mlflow still rely on pandas versions less than 3, indicating that it's not only your code that could be affected. For a detailed explanation of the changes and their implications, I recommend reading this article by Marc Garcia, a core developer of pandas: https://lnkd.in/gbBm6qgR Make sure to search your repository for dtype == "object" before upgrading. #Python #pandas #DataScience #MachineLearning #DataEngineering

The Default Sort that Misleads 🐍 Nobody tells you this early enough. 👉 Pandas doesn't sort your data for you. It loads it exactly as it arrives. The dates look right in the first few rows. You move on to analysis. But unsorted data breaks everything that depends on sequence: 🔹Rolling averages calculated in the wrong order 🔹Lag values referencing the wrong rows 🔹Cumulative metrics that compound incorrectly 🔹Trends that look smooth but aren't real 🔹Any analysis that assumes row order matters One habit that prevents all of it — First line. Every time.  python df = df.sort_values('date').reset_index(drop=True) 👉 Always sort explicitly before any time-based calculation. 👉 Never trust the default order. #DataAnalytics #Python #AnalyticsThinking #LearningInPublic

Today’s Python breakthrough: Rethinking the Fibonacci sequence. I started with a Recursive (Top-Down) approach—it looks clean but recalculates the same values repeatedly. It's fine for small numbers, but a nightmare for scaling. I moved to Tabulation (Bottom-Up). By using a list to store results as I go: ✅ Time complexity dropped from O(2 n) to O(n). ✅ No more redundant calculations. ✅ The code actually scales. As an economist moving into Data Science, these efficiency wins are what I love most. It’s not just about getting the right answer; it’s about the most effective way to get there. Check out the clean code here: https://lnkd.in/dgw_sRVM #Python #DataScience #BuildInPublic #DynamicProgramming #WomenInTech

Machine Learning Image Data using scikit image #machinelearning #datascience #dataimage #scikitimage scikit-image (formerly scikits.image) is an open-source image processing library for the Python programming language.[2] It includes algorithms for segmentation, geometric transformations, color space manipulation, analysis, filtering, morphology, feature detection, and more.[3] It is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy. The scikit-image project started as scikits.image, by Stéfan van der Walt. Its name stems from the notion that it is a "SciKit" (SciPy Toolkit), a separately-developed and distributed third-party extension to SciPy.[4] The original codebase was later extensively rewritten by other developers. Of the various scikits, scikit-image as well as scikit-learn were described as "well-maintained and popular" in November 2012.[5] Scikit-image has also been active in the Google Summer of Code. https://lnkd.in/gvmx22q2

Most people default to Pandas. Works fine… until your data scales. That’s where Polars wins: > Similar syntax for most operations > Faster execution > Lazy evaluation (big performance boost) Don’t ditch Pandas. But ignoring Polars now? That’s a mistake. Learn both. Use what fits. Found Insightful? ♻️ Repost in your network and follow Sahil Alam for more. #DataEngineering #Python #Pandas #Polars #BigData #DataAnalyticsSahil Alam for more.

Data is messy, but Python is the glue that brings it all together. 🛠️📊 I love visuals that turn complex technical concepts into a clear roadmap. This "Pythonic Universe" chart highlights why Python remains the top choice for everything from simple automation scripts to cutting-edge Machine Learning. My favorite takeaway: The "Pancake Stack" for Memory Management. It’s a great reminder that while the syntax is simple, there’s a lot of powerful logic happening under the hood. 🥞 What’s your favorite Python library to work with? (Mine is definitely Pandas! 🐼) #PythonProgramming #DataAnalytics #Infographic #TechVisuals #SoftwareEngineering #AI

Day 24: Iterators — The Engine Behind the Loop ⚙️ In Python, we often talk about "looping over data." But how does Python actually keep track of where it is in a list of 1 million items? It uses the Iterator Protocol. 1. Iterable vs. Iterator (The "Book" vs. The "Bookmark") Iterable: Any object you can loop over (List, String, Dictionary). Think of it as a Book. It contains all the data, but it doesn't "know" which page you are on. Iterator: A special object that represents a stream of data. Think of it as a Bookmark. It knows exactly where you are and what comes next. 2. The Tools: iter() and next() You can turn any Iterable into an Iterator using these two built-in functions: iter(my_list): Creates the "Bookmark" for that list. next(my_iterator): Moves the bookmark to the next "page" and returns the value. numbers = [10, 20] my_iter = iter(numbers) print(next(my_iter)) # Output: 10 print(next(my_iter)) # Output: 20 # print(next(my_iter)) # Error: StopIteration (The book is finished!) 3. Does it store data in memory? 🧠 This is the "Senior Engineer" secret. Lists/Tuples store all their data in memory at once. If you have 1 billion numbers, your RAM will crash. Iterators (and specifically Generators) are "Lazy." They don't store the whole list; they only calculate or fetch the next item when you ask for it. 💡 The Engineering Lens: This is why we use Iterators for massive datasets (like reading a 10GB log file). We process one line at a time, keeping our memory usage near zero! 4. The StopIteration Exception When an iterator runs out of items, it doesn't return None or 0. It raises a StopIteration error. 💡 The Engineering Lens: A for loop is actually just a "Fancy While Loop" that calls next() repeatedly and automatically handles the StopIteration error so your program doesn't crash. #Python #SoftwareEngineering #ComputerScience #MemoryManagement #Iterators #LearnToCode #CodingTips #TechCommunity #BackendDevelopment

After three years of relying on pandas as my daily driver, I finally dipped my toes into Polars today. At first glance, the semantics feel comfortably familiar. But once you look under the hood, it’s clear that the underlying philosophy is a total departure from the "eager" execution we’re used to in Python. In fact, it feels more like returning to the tidyverse in R. It’s refreshing to see data manipulation evolving toward this "query engine" mindset. I believe if you’re coming from a background in R or SQL, Polars might just feel like coming home. #DataScience #Python #Polars #Rust #Pandas #DataEngineering #MachineLearning

Machine Learning Geospatial Data using folium #machinelearning #datascience #geospatialdata #folium Folium is a powerful Python library that helps you create several types of Leaflet maps. By default, Folium creates a map in a separate HTML file. Since Folium results are interactive, this library is very useful for dashboard building. You can also create inline Jupyter maps in Folium. Python data, leaflet.js maps Folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js library. Manipulate your data in Python, then visualize it in a Leaflet map via Folium. Concepts : Folium makes it easy to visualize data that’s been manipulated in Python on an interactive leaflet map. It enables both the binding of data to a map for choropleth visualizations as well as passing rich vector/raster/HTML visualizations as markers on the map. The library has a number of built-in tilesets from OpenStreetMap, Mapbox, etc, and supports custom tilesets. Folium supports both Image, Video, GeoJSON and TopoJSON overlays and has a number of vector layers built-in. https://lnkd.in/gCrsfJXA

Today I practiced Sorting data using Pandas in Python 📊🐍 Sorting is very useful when analyzing datasets to identify trends, top values, or patterns. Two important functions: 🔹 sort_values() – Sort data based on column values 🔹 sort_index() – Sort data based on index Example: df.sort_values(by="Sales", ascending=False) df.sort_index() This helps quickly identify top-performing products, highest sales, or important insights in a dataset. Small concepts like these make data analysis much more efficient. Continuing to build strong foundations in Python for Data Analytics. #Python #Pandas #DataAnalytics #LearningJourney