Mastering Database Sharding in Python (Django, Flask, FastAPI)

What is Sharding

Sharding is the technique of splitting a large database into multiple smaller, faster, independent pieces called shards.

Each shard contains a subset of the data.

Example: Instead of a single user table with 50 million rows, you divide users based on:

• User ID ranges
• Region
• Hash values
• Tenant
• Customer type

Each shard lives on its own server, meaning:

• Queries hit less data
• Lower latency
• Higher reliability (if one shard dies, the system still works)
• Horizontal scaling becomes easy

Core Sharding Pattern Used by Professionals

Every Python backend framework follows the same general logic:

1. Define shards (DB connections).
2. Create a routing function (decides which shard to use).
3. Use the router in your ORM / queries.
4. Auto-route all read/write operations at runtime.

Below, we implement it for Django, Flask, and FastAPI.

1. Database Sharding in Django

Django already supports sharding via Database Routers.

Step 1 Define Multiple Shards in settings.py

DATABASES = {
    "default": {},
    "shard_1": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "users_europe",
    },
    "shard_2": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "users_asia",
    },
}
        

Step 2 Create a Router With Type Hints

from typing import Optional


class UserShardRouter:
    def db_for_read(self, model, **hints) -> Optional[str]:
        user_id: int = hints.get("user_id", 0)
        return "shard_1" if user_id < 50000 else "shard_2"

    def db_for_write(self, model, **hints) -> Optional[str]:
        return self.db_for_read(model, **hints)
        

Step 3 Use the Router in Views/Services

def get_user(user_id: int):
    return User.objects.using(
        "shard_1" if user_id < 50000 else "shard_2"
    ).get(id=user_id)
        

• Professional
• Clean
• Scalable

2. Sharding in Flask

Flask is lightweight, so sharding is done manually using SQLAlchemy.

Step 1 Initialize Shards

from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from typing import Dict


shards: Dict[str, Session] = {
    "shard_1": Session(create_engine("postgresql://.../eu")),
    "shard_2": Session(create_engine("postgresql://.../asia")),
}
        

Step 2 Shard Selector Function

def get_shard(user_id: int) -> Session:
    return shards["shard_1"] if user_id < 50000 else shards["shard_2"]
        

Step 3 Querying Data

def fetch_user(user_id: int):
    session = get_shard(user_id)
    return session.execute(
        "SELECT * FROM users WHERE id = :id",
        {"id": user_id}
    ).fetchone()
        

Simple and effective.

3. Implementing Sharding in FastAPI

FastAPI = cleanest structure + async support.

Step 1 Setup Async Shards (Postgres Example)

from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
from sqlalchemy.orm import sessionmaker
from typing import Dict


shards: Dict[str, sessionmaker] = {
    "shard_1": sessionmaker(
        create_async_engine("postgresql+asyncpg://.../eu"),
        class_=AsyncSession, expire_on_commit=False
    ),
    "shard_2": sessionmaker(
        create_async_engine("postgresql+asyncpg://.../asia"),
        class_=AsyncSession, expire_on_commit=False
    ),
}
        

Step 2 Routing Logic

def select_shard(user_id: int) -> sessionmaker:
    return shards["shard_1"] if user_id < 50000 else shards["shard_2"]
        

Step 3 FastAPI Endpoint

from fastapi import FastAPI

app = FastAPI()


@app.get("/users/{user_id}")
async def get_user(user_id: int):
    session_factory = select_shard(user_id)
    
    async with session_factory() as session:
        result = await session.execute(
            "SELECT * FROM users WHERE id = :id",
            {"id": user_id}
        )
        return result.fetchone()
        

• Modern
• Async
• Production-ready

Final Thoughts

Sharding feels difficult only until you understand:

“It’s just splitting your data and routing cleverly.”

Python frameworks don’t provide sharding out of the box (except Django), but with small routing logic + separate DB connections, you get:

• Unlimited scalability
• Reduced latency
• Faster reads/writes
• More stable production systems

GitHub Demo (Starter Pack):

https://github.com/UmanSheikh/python-sharding-boilerplate