Parallelising API calls in Python does not have to mean rewriting everything. A common performance issue in Python code looks like this: • a loop that makes many API calls • each call waits for the previous one • runtime grows linearly as requests increase In this post, we break down how to improve that pattern step by step: • measure where time is actually being spent • remove small but costly Pandas inefficiencies • run independent, I/O-bound API calls in parallel using concurrent.futures.ThreadPoolExecutor The key idea is this: when requests do not depend on each other, they do not need to run one after another. The walkthrough uses a small public dataset purely as a demonstration, but the same approach applies to production APIs, data pipelines, and application backends. Full post is linked in the comments. #Python #SoftwareEngineering #PerformanceOptimization #DataEngineering #APIs #Pandas
Optimize Python API Calls with Parallel Processing
More Relevant Posts
-
Python Dataclasses — Write Less, Do More 🚀 Ever felt your code getting bloated with __init__, __repr__, and repetitive setup for simple data models? Python Dataclasses solve exactly that clean, readable, and safer code with minimal effort. 📌 What’s inside this carousel: ✅ The problem with boilerplate-heavy data containers ✅ How @dataclass simplifies class definitions ✅ A quick, practical code demo ✅ Key benefits: readability, safety, maintainability ✅ Real-world use cases like configs & API models If you work with Python models regularly, this is a must-know feature. 🐍✨ #Python #Dataclasses #PythonTips #CleanCode #SoftwareDevelopment #BackendDevelopment #PythonProgramming #CodeBetter #DeveloperTools #CodeXLancers
To view or add a comment, sign in
-
Day 293: Python asyncio — Modern Python Concurrency ⏳ Why asyncio feels different Unlike threading or multiprocessing, asyncio is about cooperation, not parallelism. Tasks don’t run at the same time — they pause and resume intelligently while waiting. 👉 Simple async example import asyncio async def hello(): print("Hello") await asyncio.sleep(1) print("World") asyncio.run(hello()) The magic is await. It tells Python: “I’m waiting, let something else run.” 💡 Perfect for •Web scraping •API calls •Async servers •Non-blocking applications 🧠 Challenge Write an async program that downloads data from multiple URLs at the same time. #Python #AsyncIO #AsyncProgramming #ModernPython
To view or add a comment, sign in
-
In C, an integer takes 4 bytes. In Python, the number 1 takes 28 bytes. Why? As python is a "Dynamically Typed" language, it needs to store more than just the value 1. It needs to store "metadata" about that value so the interpreter knows what to do with it. How? In the CPython source code, every single thing is a PyObject. So when we create x = 1, following structure is created with it: 1. ob_refcnt (8 bytes): The Reference Counter. It tracks how many variables point to this object. 2. ob_type (8 bytes): A pointer to the "Type Object" (telling Python the datatype). 3. ob_size (8 bytes): For variable-sized objects (like lists) 4. The Actual Value (4-8 bytes): The actual number 1 Understanding this overhead explains why Python is memory-intensive being a dynamically typed language. I am trying to learn Python Internals in detail and will share my learnings. Do follow along and tell your experiences in comments. #Python #PythonInternals #SoftwareEngineering #BackendDevelopment
To view or add a comment, sign in
-
-
In Python, dictionary lookups become faster than list scans after just a few repeated queries. I often filter lists of objects by name or ID using a simple loop. But I wanted to understand when it makes sense to build an index, such as a {name: person} dictionary. As I was getting restless during the holidays I ran benchmarks on datasets ranging from 1k to 100k items. The results were consistent: • Building the dictionary is inexpensive; a few milliseconds, even at 100k items • Once built, dictionary lookups are about 10 to 70 times faster than looping • The performance benefit appears quickly • For 1k to 10k items, about 2 to 4 lookups are enough • For 100k items, around 4 to 7 lookups Based on these results, if I only need one lookup or I am accessing most of the data anyway, I stay with a loop. But if I need even a few targeted lookups, building the dictionary quickly becomes worthwhile. Have you done similar comparisons? #Python #SoftwareEngineering #DataStructures #CodePerformance #CleanCode #ProgrammingTips
To view or add a comment, sign in
-
-
🟦🟨🟩 🐍 PYTHON – DAY 2 🟦🟨🟩 📌 Variables & Data Types Variable: A variable is used to store data values in Python. Python does not require declaring the data type explicitly. ✨ Example: name = "Python" age = 22 marks = 85.5 is_student = True 🔹 Common Data Types in Python 🔸 int → Integer (10, 20) 🔸 float → Decimal (10.5, 3.14) 🔸 str → Text ("Hello") 🔸 bool → True / False
To view or add a comment, sign in
-
It felt more natural to go from SQL to Python than I had anticipated. Once you get the reasoning behind SQL, it is primarily syntax, that needs to be translated into Python. The concepts of filtering, grouping, and joining remain unchanged. The terminology just shifts. For this reason, foundations are important. It is far less scary to switch tools when the concepts are obvious. I'm sharing this because switching from SQL to Python doesn't have to feel like a fresh start. #DataProfessional #SQL #Python #DataAnalytics #AnalyticsLearning
To view or add a comment, sign in
-
-
Day 38 / 100 – Implement Stack Using Python List => Building a basic stack data structure with core operations: push, pop, top, and empty. Time Complexity: O(1) for all operations => Python lists make stack operations efficient: append() for push pop() for removing the top element Index access for top Length check for empty state Learning Insight This problem show how choosing the right data structure makes implementation simple and efficient. Python lists are dynamic arrays, making them a perfect fit for stack behavior when operations are restricted to one end. Consistency and clarity matter more than complexity — mastering the basics builds a strong DSA foundation. rewrite in simple word so not look loke ai Code pushed to Git https://lnkd.in/g3NUT5HM #100DaysOfCode #Python #DSA #Stack #DataStructures #LeetCode #CodingJourney #ProblemSolving
To view or add a comment, sign in
-
An overlooked Python behavior that can silently affect ML pipelines While working on an in-place array problem recently, I was reminded how important Python’s object model is in real-world data workflows. In Python: 1. Reassigning a variable (x = new_list) → does not modify the original object 2. Mutating an object (x[i] = ...) → modifies the object in-place Why it matters: 1. Pandas: df2 = df vs df.copy() 2. NumPy: in-place operations (X /= max_val) vs out-of-place (X = X / max_val) 3. ML pipelines: unintended in-place changes can affect downstream processing Takeaway: Be deliberate about in-place vs out-of-place operations. Using .copy() or new assignments when necessary to maintain data integrity. #Python #NumPy #Pandas #DataScience #MachineLearning #MLOps
To view or add a comment, sign in
-
-
💻 100 Days of Python Challenge – Day 12: Dictionary 🧾 Dictionary in Python A dictionary is a built-in data structure in Python used to store data in key–value pairs. ✔️ Key Features Unordered (but maintains insertion order in modern Python) Mutable (you can add, update, or remove items) Keys must be unique Values can be anything (numbers, strings, lists, other dictionaries)
To view or add a comment, sign in
Explore content categories
- Career
- Productivity
- Finance
- Soft Skills & Emotional Intelligence
- Project Management
- Education
- Technology
- Leadership
- Ecommerce
- User Experience
- Recruitment & HR
- Customer Experience
- Real Estate
- Marketing
- Sales
- Retail & Merchandising
- Science
- Supply Chain Management
- Future Of Work
- Consulting
- Writing
- Economics
- Artificial Intelligence
- Employee Experience
- Workplace Trends
- Fundraising
- Networking
- Corporate Social Responsibility
- Negotiation
- Communication
- Engineering
- Hospitality & Tourism
- Business Strategy
- Change Management
- Organizational Culture
- Design
- Innovation
- Event Planning
- Training & Development
Link to blog post: https://www.jumpingrivers.com/blog/python-package-part-three/