Hybrid Database Model in Django for Speed

Introduction

Handling a high volume of requests per second while managing an extensive product database is a key challenge in modern e-commerce platforms. In this post, I'll show the solutions we implemented in our "catalog" microservice to address these challenges.

We initially built the service in Django — driven by the need for rapid development and Django's convenient admin interface. But as the platform scaled, storing everything in a relational database with multiple joins became a bottleneck.

Our idea: combine the strengths of both relational and non-relational databases. Keep the relational integrity needed for certain data, while using MongoDB for fast reads on product fields. And keep everything compatible with Django models, Django Admin, and Django REST Framework.

Solution

The goal is to link each Django model object with a MongoDB document. When reading, the model is automatically populated from MongoDB. When writing, changes are synced back to the MongoDB document.

We achieve this by dynamically creating descriptors within the Django model. These descriptors act as proxies to MongoDB fields and are generated using a metaclass.

Step 1: The MongoDB Field Descriptor

class MongoFieldDescriptor:
    def __init__(self, field_name):
        self.field_name = field_name
        self.private_name = f"_m_{field_name}"

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        # Read directly from the MongoDB document
        if obj._mongo_document:
            return obj._mongo_document.get(self.field_name)
        return getattr(obj, self.private_name, None)

    def __set__(self, obj, value):
        # Store in private field for batch saving later
        setattr(obj, self.private_name, value)

The descriptor reads fields directly from the MongoDB document. For writes, it stores data in a private _m_<field> attribute so all fields can be saved at once.

Step 2: The Model Metaclass

class HybridModelMeta(type(models.Model)):
    def __new__(mcs, name, bases, namespace):
        cls = super().__new__(mcs, name, bases, namespace)
        mongo_fields = getattr(cls, "mongo_fields", [])
        for field_name in mongo_fields:
            setattr(cls, field_name, MongoFieldDescriptor(field_name))
        return cls

The metaclass iterates through each MongoDB field name and creates a descriptor with that name, attaching it to the model class.

Step 3: The Base Hybrid Model

class HybridModel(models.Model, metaclass=HybridModelMeta):
    class Meta:
        abstract = True

    _mongo_document = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # Connect to the corresponding MongoDB document
        self._mongo_document = self.get_mongo_collection().find_one(
            {"_id": str(self.pk)}
        )

    def save(self, *args, **kwargs):
        super().save(*args, **kwargs)
        # Sync _m_ prefixed fields back to MongoDB
        update_fields = {}
        for field_name in self.mongo_fields:
            private = f"_m_{field_name}"
            if hasattr(self, private):
                update_fields[field_name] = getattr(self, private)
        if update_fields:
            self.get_mongo_collection().update_one(
                {"_id": str(self.pk)},
                {"$set": update_fields},
                upsert=True,
            )

This abstract base class connects a MongoDB document to the model on initialization and automatically syncs _m_-prefixed fields back to MongoDB on save.

Step 4: The Product Model

class Product(HybridModel):
    # Regular Django fields (relational)
    category = models.ForeignKey(Category, on_delete=models.CASCADE)
    created_at = models.DateTimeField(auto_now_add=True)

    # Fields stored in MongoDB for fast reads
    mongo_fields = ["name", "description", "price", "attributes", "images"]

    @classmethod
    def get_mongo_collection(cls):
        return mongo_db["products"]

The implementation is straightforward: define a MongoDB document containing product fields and connect it to the Product model. The model can still use Django relationships normally.

Performance Comparison

We created an endpoint to retrieve products within a specific category and benchmarked it using the OHA tool — 10,000 requests across 50 threads.

Conclusion

The hybrid model approach significantly enhanced performance by offloading read-heavy product data to MongoDB while keeping relational integrity in PostgreSQL. There's room for further improvement — loading all MongoDB fields at once, batch-loading documents for entire querysets, and adding caching.

This approach offers a practical balance between relational and non-relational databases while staying fully compatible with Django Admin and Django REST Framework.