Data scientist argues against Kubernetes for Python workloads

Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled, offers an alternative. By David Cassel

A Cloud Built for Python Data Scientists, Not Infrastructure Engineers. Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled.io, uses VMs instead.

🌟 New Blog Just Published! 🌟 📌 Python Data Scientists' Dream Cloud Platform 🚀 ✍️ Author: Hiren Dave 📖 The modern cloud is built to serve any coder, yet it feels like a sandbox for engineers who wrestle with Docker, Kubernetes, and networking. Python -loving data scientists just want a clean API for...... 🕒 Published: 2025-10-05 📂 Category: Cloud 🔗 Read more: https://lnkd.in/d6sezfxT 🚀✨ #pythoncloud #datasciencecloud #cloudml

“Data scientists didn’t sign up to be Kubernetes engineers.” Yet too often, that’s exactly what happens. Brilliant Python minds spend more time wrangling infrastructure than building models. This article makes the case for a different approach: cloud platforms designed for data scientists, by data scientists, where scaling experiments and integrating with the Python ecosystem is seamless, and infrastructure fades into the background. I’ve seen firsthand how much time gets lost fighting with tools that weren’t really built for us. A purpose-built cloud could change the game. Curious... what’s been your biggest pain point with running data science workloads in the cloud? A Cloud Built for Python Data Scientists, Not Infrastructure Engineers... The New Stack https://lnkd.in/giU7su7z #DataScience #Python #CloudComputing #MachineLearning

Excited to share my latest project — AI Voice & Text Chatbot using Groq API, FastAPI & Redis! This intelligent chatbot can understand both voice and text queries, generate real-time responses using Groq’s LLMs, and handle contextual memory via Redis Cloud. ⚙️ Tech Stack: FastAPI Groq API (LLaMA 3.1) Redis Python SpeechRecognition gTTS 💡 Key Features: • Voice-to-text and text-to-voice communication • Real-time AI conversation using Groq API • FastAPI backend with Redis caching • Scalable and deployable as a web-based assistant 🔗 GitHub Repository: https://lnkd.in/dW9fNr2v 📊 Architecture Overview is attached below. I learned a lot about integrating APIs, handling real-time speech, and optimizing latency through caching. Would love to hear your thoughts and suggestions! 🚀 #AI #FastAPI #Groq #Redis #VoiceChatbot #Python #LLM #MachineLearning #SoftwareDevelopment

Implementing Machine Learning Models with Python on MCP Servers The rapid advancements in machine learning (ML) and #ArtificialIntelligence (AI) have catalyzed numerous businesses to integrate these #technologies into their infrastructure. #MCP (Managed Cloud Platform) servers offer a robust environment for deploying machine learning models with the power and scalability needed for complex computations. This article not only delves into the fundamentals of implementing machine learning models using #Python on MCP servers but also explores best practices, tools, and considerations for optimizing your setup. https://lnkd.in/dmEyNitV

Recently, I ran into an issue where log files stored in a Google Cloud Storage (GCS) bucket were growing too large, risking the disk quotas. I implemented a log rotation script in Python using the google-cloud-storage library. Below was the script I implemented: from google.cloud import storage from datetime import datetime MAX_SIZE = 5 * 1024 * 1024 # 5 MB BUCKET_NAME = "Animesh_Data" LOG_PATH = "logs/my_log.txt" def get_blob():   client = storage.Client()   bucket = client.get_bucket(BUCKET_NAME)   blob = bucket.blob(LOG_PATH)   return blob def get_blob_size():   blob = get_blob()   try:     blob.reload() # to refresh metadata     size = blob.size     if size is None:       return 0     return size   except Exception as e:     print(f"Error accessing blob size: {e}")     return 0 def rotate_log_if_needed():   size = get_blob_size()   if size <= MAX_SIZE:     return   blob = get_blob()   bucket = blob.bucket   # Build rotated filename base with date   date_str = datetime.utcnow().strftime("%Y%m%d")   base_name = f"logs/my_log_{date_str}"   # Find next available index for rotation   i = 1   while True:     rotated_name = f"{base_name}_{i}.txt"     if not bucket.blob(rotated_name).exists():       break     i += 1   # Copy original log to rotated file   bucket.copy_blob(blob, bucket, rotated_name)   # Delete the original log file   blob.delete() def write_log_entry(entry):   blob = get_blob()   try:     content = blob.download_as_text() #get the existing content as text   except Exception:     content = ""   content += entry + "\n"   blob.upload_from_string(content) #upload the incoming content into the file def log(entry):   rotate_log_if_needed()   write_log_entry(entry) The time complexity of the entire code was O(N + S) where N = number of rotated logs for the current day and S = size of current log file (in bytes). This setup ensures that no log file exceeds 5MB in size, and old logs are archived with a timestamp and index. This is especially useful when managing logging on cloud infrastructure with limited storage budgets.

A Cloud Built for Python Data Scientists, Not Infrastructure Engineers - The New Stack: Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled.io, ... #bigdata #cdo #cto

Testing your app locally and then deploying to the cloud via Infrastructure as Code (IaC) tools is a good approach in most cases. Using FastAPI for your Python backend is a good idea. An example of using these with the Cloud Development Kit (CDK) is below. Using managed and serverless components on AWS is an easy way to get a fully functional app up and running quickly. These services are typically highly scalable and for the most part you will pay nothing or very little unless you're seeing large amounts of traffic. The example below from Adrian Mudzwiti shows how to take a FastAPI based API developed locally and set it up in AWS with a Lambda function and a serverless DynamoDB database. https://lnkd.in/emThKDsF

🚀 Architecting the Future: Launching into Python for Serverless & Automation on AWS! Starting a strategic course focused on mastering the intersection of Python and advanced AWS services. This isn't just about scripting; it’s about building highly scalable, secure, and automated cloud solutions. The Core Pillars of Cloud Mastery The curriculum is meticulously designed to move from robust Python fundamentals (covering variables, conditionals, functions, lists, and dictionaries) to sophisticated enterprise architecture: Serverless Architectures: I'll be deploying complete serverless solutions using Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. This foundation is critical for developing modern, cost-efficient microservices. Intelligent Automation: The focus then shifts to practical, real-world application, writing and running powerful automation scripts with Python and AWS Systems Manager. This empowers essential automation for Support and Operations. Data & Security: The course also integrates crucial topics like managing data with Amazon RDS and Amazon DynamoDB , all while enforcing robust access control using AWS IAM roles and safeguarding credentials with AWS Secrets Manager. This course is designed to give a deep conceptual understanding of Python and how to use it strategically for enterprise-level automation. I am committed to leveraging these skills to drive significant efficiency and innovation in cloud operations. Let's Connect on Cloud Strategy! For my network focused on cloud and DevOps: What Python library or AWS service do you believe is currently the single most underestimated tool for automation? Share your insights below—I'm ready to learn from your experience! 👇 #AWS #Python #Serverless #Automation #CloudArchitecture #LinkedInLearning #CareerGrowth