Why Kubernetes isn't best for Python workloads: Coiled.io founder

A Cloud Built for Python Data Scientists, Not Infrastructure Engineers. Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled.io, uses VMs instead.

A Cloud Built for Python Data Scientists, Not Infrastructure Engineers Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled.io, offers an alternative. The cloud is incredibly useful — but what if you’re a Python-loving data scientist? The prevailing advice has been that if you want to run industrial-grade Python, then run it on Kubernetes. “We just think that’s dead wrong,” said Matthew Rocklin. In 2020, Rocklin co-founded Coiled.io to offer an even easier way to unlock the cloud’s potential. “The answer is just ‘Go use raw VMs [virtual machines]’,” Rocklin said on the “Talk Python” podcast. “They’re actually pretty good, if you do a few things around them.” (Like configuring the right software environments and appropriate logs.) In 2015, Rocklin created Dask, a Python library to spin up lots of VMs for analyzing and manipulating data. And after years contributing to Python projects for data science (like Tools, Multiple Dispatch, and SimPy), Rocklin co-founded Coiled.io to make it even easier to deploy that VM-creating software. https://lnkd.in/eQYGEeVx Please follow Divye Dwivedi for such content.  #DevSecOps, #SecureDevOps, #CyberSecurity, #SecurityAutomation, #CloudSecurity, #InfrastructureSecurity, #DevOpsSecurity, #ContinuousSecurity, #SecurityByDesign, #SecurityAsCode, #ApplicationSecurity, #ComplianceAutomation, #CloudSecurityPosture, #SecuringTheCloud, #AI4Security #DevOpsSecurity #IntelligentSecurity #AppSecurityTesting #CloudSecuritySolutions #ResilientAI #AdaptiveSecurity #SecurityFirst #AIDrivenSecurity #FullStackSecurity #ModernAppSecurity #SecurityInTheCloud #EmbeddedSecurity #SmartCyberDefense #ProactiveSecurity

A Cloud Built for Python Data Scientists, Not Infrastructure Engineers Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled.io, offers an alternative. The cloud is incredibly useful — but what if you’re a Python-loving data scientist? The prevailing advice has been that if you want to run industrial-grade Python, then run it on Kubernetes. “We just think that’s dead wrong,” said Matthew Rocklin. In 2020, Rocklin co-founded Coiled.io to offer an even easier way to unlock the cloud’s potential. “The answer is just ‘Go use raw VMs [virtual machines]’,” Rocklin said on the “Talk Python” podcast. “They’re actually pretty good, if you do a few things around them.” (Like configuring the right software environments and appropriate logs.) In 2015, Rocklin created Dask, a Python library to spin up lots of VMs for analyzing and manipulating data. And after years contributing to Python projects for data science (like Tools, Multiple Dispatch, and SimPy), Rocklin co-founded Coiled.io to make it even easier to deploy that VM-creating software. https://lnkd.in/eR7NdmMB Stay Connected to Sidharth Sharma, CPA, CISA, CISM, CFE, CDPSE for content related to Cyber Security. #CyberSecurity #JPMC #Technology #InfoSec #DataProtection #DataPrivacy #ThreatIntelligence #CyberThreats #NetworkSecurity #CyberDefense #SecurityAwareness #ITSecurity #SecuritySolutions #CyberResilience #DigitalSecurity #SecurityBestPractices #CyberRisk #SecurityOperations 

“Data scientists didn’t sign up to be Kubernetes engineers.” Yet too often, that’s exactly what happens. Brilliant Python minds spend more time wrangling infrastructure than building models. This article makes the case for a different approach: cloud platforms designed for data scientists, by data scientists, where scaling experiments and integrating with the Python ecosystem is seamless, and infrastructure fades into the background. I’ve seen firsthand how much time gets lost fighting with tools that weren’t really built for us. A purpose-built cloud could change the game. Curious... what’s been your biggest pain point with running data science workloads in the cloud? A Cloud Built for Python Data Scientists, Not Infrastructure Engineers... The New Stack https://lnkd.in/giU7su7z #DataScience #Python #CloudComputing #MachineLearning

🚀 Day 13 of #30DaysDataScience: Google Image Scraper Project Complete! Today, I built a Python-based Image Scraper that: ✅ Scrapes images from Google based on a search query ✅ Downloads and saves them locally ✅ Logs activities and errors ✅ Optionally stores image data in MongoDB This project helped me gain hands-on experience with: Python scripting Web scraping with Requests & BeautifulSoup File handling and logging MongoDB for structured/unstructured data Even without deploying to the cloud, I made the project GitHub-ready and documented it professionally. 💻 Check out the project here: https://lnkd.in/dDiX-JT8 #DataScience #Python #WebScraping #MongoDB #LearningByDoing #Projects #Day13

Python has long been the favorite language for data scientists, but scaling from a local notebook to enterprise-grade #ML on terabytes of data has always been a challenge. BigQuery DataFrames (#BigFrames) changes that, bringing the familiar pandas and scikit-learn experience to BigQuery's massive scale — no infrastructure headaches, just pure data science power. It's more than "pandas at scale", translating Python operations into efficient BigQuery SQL, integrating seamlessly with #VertexAI and #Gemini models, and removing the friction that slows down innovation. Check out our blog, "BigQuery DataFrames: Python at Scale for Data Science" 👉 https://lnkd.in/dWqieqFQ #kartaca #cloud

Amazon DynamoDB Integration Lab – Data Engineering Mastery Lab Overview:  → Started the lab by accessing a fully managed AWS environment, geared towards practical cloud database integration. Python Scripting & Table Review:  → Opened the preloaded Python script in VS Code IDE and explored the schema & records in the LanguagesTable using both AWS Console and CLI. Challenge 1 – Insert Operation:  → Updated the Python code (using Boto3) to insert a new item—Danish language and ISO code—into DynamoDB. Applied best practices for data modeling and dictionary structures. Challenge 2 – Query Operation:  → Modified the script to query and retrieve the Danish entry from the LanguagesTable, demonstrating efficient use of get_item for NoSQL database operations. Testing & Troubleshooting:  → Ran and validated the script in the IDE terminal, confirming both insert and query actions worked seamlessly. Fine-tuned code with troubleshooting and solution validation. Knowledge Check & Reflection:  → Completed the lab with reinforced skills in cloud database management, Python automation, and practical understanding of CRUD operations on AWS DynamoDB. Conclusion:  → Successfully integrated and operated on DynamoDB—enhancing my proficiency in cloud-native data engineering and real-world application development. Grateful for the hands-on exposure! Looking forward to leveraging these skills for impactful cloud data solutions and career growth. #AWS #DynamoDB #Python #CloudEngineering #HandsOnLearning #DataScience #ContinuousImprovement

Amazon DynamoDB Integration Lab – Data Engineering Mastery Lab Overview:  → Started the lab by accessing a fully managed AWS environment, geared towards practical cloud database integration. Python Scripting & Table Review:  → Opened the preloaded Python script in VS Code IDE and explored the schema & records in the LanguagesTable using both AWS Console and CLI. Challenge 1 – Insert Operation:  → Updated the Python code (using Boto3) to insert a new item—Danish language and ISO code—into DynamoDB. Applied best practices for data modeling and dictionary structures. Challenge 2 – Query Operation:  → Modified the script to query and retrieve the Danish entry from the LanguagesTable, demonstrating efficient use of get_item for NoSQL database operations. Testing & Troubleshooting:  → Ran and validated the script in the IDE terminal, confirming both insert and query actions worked seamlessly. Fine-tuned code with troubleshooting and solution validation. Knowledge Check & Reflection:  → Completed the lab with reinforced skills in cloud database management, Python automation, and practical understanding of CRUD operations on AWS DynamoDB. Conclusion:  → Successfully integrated and operated on DynamoDB—enhancing my proficiency in cloud-native data engineering and real-world application development. Grateful for the hands-on exposure! Looking forward to leveraging these skills for impactful cloud data solutions and career growth. Special thanks to my mentors Morgan Willis and Russell Sayers for their constant guidance and inspiration. Your mentorship makes every achievement possible—grateful for your support! #AWS #DynamoDB #Python #CloudEngineering #HandsOnLearning #DataScience #ContinuousImprovement

Coiled python library that makes cloud infrastructure management easier Coiled is a Python library and cloud platform designed to simplify scaling Python workloads in the cloud, particularly with Dask. It allows data engineers and scientists to leverage cloud resources without needing to manage complex infrastructure like Docker or Kubernetes. Key features and functionalities of Coiled include: Scaling Python with Dask: Coiled simplifies the deployment and management of Dask clusters in the cloud, enabling parallel and distributed computing for large datasets and complex computations. Environment Synchronization: It automatically replicates your local Python environment, including packages and dependencies, onto cloud machines, ensuring consistent execution Cloud Resource Management: Coiled handles the provisioning, configuration, and management of cloud resources (CPUs, GPUs, memory) in your cloud account (AWS, Azure, GCP). Cost Management and Visibility: It provides tools for tracking cloud costs, setting user quotas, and monitoring resource usage across teams. Security: Coiled operates within your cloud network and implements security best practices to protect your data and resources. Serverless Functions: Coiled Functions offer a serverless-like API for executing Python functions in the cloud, ideal for bulk processing. Local Development with Remote Execution: Coiled Notebooks enable local development while executing computations on powerful remote cloud machines, with real-time file synchronization. Integration: Coiled integrates easily with existing Python workflows and can be used in various environments where Python runs. In essence, In nutshell Coiled aims to make cloud-based Python scaling accessible and efficient, allowing users to focus on their data science and engineering tasks rather than infrastructure management. #python #cloud #inframanagement

🚀 Architecting the Future: Launching into Python for Serverless & Automation on AWS! Starting a strategic course focused on mastering the intersection of Python and advanced AWS services. This isn't just about scripting; it’s about building highly scalable, secure, and automated cloud solutions. The Core Pillars of Cloud Mastery The curriculum is meticulously designed to move from robust Python fundamentals (covering variables, conditionals, functions, lists, and dictionaries) to sophisticated enterprise architecture: Serverless Architectures: I'll be deploying complete serverless solutions using Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. This foundation is critical for developing modern, cost-efficient microservices. Intelligent Automation: The focus then shifts to practical, real-world application, writing and running powerful automation scripts with Python and AWS Systems Manager. This empowers essential automation for Support and Operations. Data & Security: The course also integrates crucial topics like managing data with Amazon RDS and Amazon DynamoDB , all while enforcing robust access control using AWS IAM roles and safeguarding credentials with AWS Secrets Manager. This course is designed to give a deep conceptual understanding of Python and how to use it strategically for enterprise-level automation. I am committed to leveraging these skills to drive significant efficiency and innovation in cloud operations. Let's Connect on Cloud Strategy! For my network focused on cloud and DevOps: What Python library or AWS service do you believe is currently the single most underestimated tool for automation? Share your insights below—I'm ready to learn from your experience! 👇 #AWS #Python #Serverless #Automation #CloudArchitecture #LinkedInLearning #CareerGrowth