"Python Data Scientists' Dream Cloud Platform by Hiren Dave"

Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled, uses VMs instead. By David Cassel

Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled, offers an alternative. By David Cassel

Microsoft is actively improving #fsspec, powering seamless Azure Blobs integration for Python AI/ML workflows! The newest release of adlfs (2025.8.0) brings 2-5x faster uploads and plug-and-play support for #Ray, #PyTorch, #Pandas and more. Just switch to az:// for scalable cloud data pipelines. Upgrade now and help shape the future by providing feedback in the GitHub page. Continuous open-source collaboration is our priority. Blog post: https://lnkd.in/gWiEYrXT Adlfs GitHub page: https://lnkd.in/g3B5JamB #Azure #adlfs #opensource #DataScience #MachineLearning #Python #Storage #rayio Kyle Knapp Sean McCullough Scott Hoag

Great set of enhancements for the Python ecosystem. adlfs is also the foundational access layer for Microsoft OneLake. OneLake provides ADLS-compatible APIs for both Blob and DFS endpoints, enabling seamless integration with Azure Blob Storage and Azure Data Lake Storage.

🚀 Shaving off 70% of your Cloud Build time - A Tiny Dockerfile Tweak That Makes a Huge Difference. The following takes forever on Google Cloud Build: “Building wheel for pandas…” “Building wheel for scikit-learn…” The time adds up - every single build. After some digging, I realized that I was using #python 3.12, which didn’t have prebuilt wheels for #pandas, #sklearn or #numpy. Since these were all in my requirements.txt, Cloud Build was actually compiling them from source every time 🤦♂️ The fix is actually pretty simple - shown in the code snippet! ✅ Use a Python version with existing binary wheels ✅ Install heavy libraries (numpy, pandas, scikit-learn) early and separately Doing this lets Cloud Build cache the layer, skipping slow recompiles. Build time dropped by nearly 70%. Wow! 🚀 One of those small "aha!" moments that makes you appreciate how much time is lost to tiny details in CI/CD pipelines. #GCP #cicd #docker #ML

Testing your app locally and then deploying to the cloud via Infrastructure as Code (IaC) tools is a good approach in most cases. Using FastAPI for your Python backend is a good idea. An example of using these with the Cloud Development Kit (CDK) is below. Using managed and serverless components on AWS is an easy way to get a fully functional app up and running quickly. These services are typically highly scalable and for the most part you will pay nothing or very little unless you're seeing large amounts of traffic. The example below from Adrian Mudzwiti shows how to take a FastAPI based API developed locally and set it up in AWS with a Lambda function and a serverless DynamoDB database. https://lnkd.in/emThKDsF

🚀 AI Coding in Colab with Gemini Google Colab now integrates directly with Gemini, turning AI into a true collaborator in coding and data science. In this hands-on tutorial, we connect Colab to GitHub and MongoDB, securely manage credentials, and let Gemini generate, explain, and run Python code — building a full Twitter network analysis in minutes, all in the cloud. See how AI transforms coding from automation to collaboration: 👉 https://lnkd.in/eRYKRjMh #AICoding #GoogleColab #Gemini #DataScience #Python #LBSocial

It’s now incredibly easy to analyze Twitter data with AI. Using Google Colab and Gemini, we can ask AI to write, explain, and run Python code that connects to a dataset, maps relationships, and builds a complete social network analysis — all in the cloud, no setup required. AI enables faster, smarter, and more intuitive data exploration than ever. #AICoding #Python #GoogleColab #Gemini #DataScience #SocialNetworkAnalysis #TwitterData

🚀 From Code to Cloud: Automating the Full Stack Lifecycle. I just developed and deployed a full-stack Python/Flask application, and it's a perfect example of modern DevOps in action. 🔹 App: A dynamic To-Do List (because even DevOps engineers need to stay organized! ✅) 🔹 Tech Stack: Python, Flask, MySQL 🔹 Infrastructure: AWS EC2 🔹 Security: Managed via AWS Security Groups for controlled access. 🔹 Monitoring: Implemented Prometheus & Grafana dashboards for real-time insights into server & database performance. This wasn't just about writing code; it was about building a resilient, monitored, and secure system. It reinforced the critical cycle of: Develop → Deploy → Monitor → Iterate. #DevOps #CloudComputing #AWS #Python #Flask #InfrastructureAsCode #Monitoring #Prometheus #Grafana #SysAdmin #Cloud

This week, I deep dived into something that quietly saves huge amounts of cost and time data compression in GCP. I wanted to see how GZIP and Snappy behave across different layers like BigQuery, Cloud Storage, and Dataflow pipelines. The goal? - Find the right balance between storage efficiency and processing speed. Here’s what I found: 🔹 BigQuery automatically handles compression using its columnar storage format you don’t have to lift a finger. 🔹 Parquet + Snappy turned out to be ideal for analytical pipelines. 🔹 GZIP gave better compression ratios, making it great for archiving and long-term storage. 🔹 Smart compression choices directly reduced GCS storage size and improved pipeline execution times. I also created a small .ipynb notebook on GitHub showing how to test and visualize compression results feel free to check it out! https://lnkd.in/dcKDCPP8 Here’s a quick snapshot from the experiment #DataEngineering #GoogleCloud #BigQuery #DataCompression #Python #ETL #GCP #DataPipeline #CloudComputing