Welcome to the World of MongoDB!

Hey there! Before we dive into the nitty-gritty of MongoDB’s aggregation powers, let’s kick things off with a quick intro to what MongoDB is and why it’s such a big deal in the database world.

What is MongoDB?

MongoDB is a document-oriented NoSQL database that’s designed to handle large amounts of data at high speed. It’s all about giving developers flexibility and power to store and retrieve data in a way that’s intuitive and aligned with how developers actually think and code. Instead of tables and rows as in traditional relational databases, MongoDB works with collections and documents. This structure allows it to handle all sorts of data formats and to scale up easily, making it a favourite for everything from small startups to massive enterprises.

Why MongoDB?

1. Schema-less Nature: Each document in a collection can have a completely different structure. This flexibility means you can rapidly iterate and deploy new features without having to worry about restructuring your entire database.
2. Scalability: MongoDB handles horizontal scaling with ease. You can spread your data across more servers easily as your app grows and attracts more users.
3. Speed: Its indexing capabilities allow for fast queries, and its document model means that related data can be stored together and accessed quickly in a single query.

So, whether you’re developing a new app, working on a complex data analysis task, or just trying to optimize your database, MongoDB offers a robust set of tools to make your life easier.

Now, let’s dive into some of the advanced features that truly show off what MongoDB can do when it comes to managing and querying your data. Buckle up; it’s going to be an enlightening ride!

Introduction

When working with MongoDB, understanding how to efficiently fetch and manipulate related data from multiple collections is crucial. To illustrate, let’s consider a simple eCommerce application that uses three main collections: customers, orders, and items.

To try out all the queries discussed below use this MongoDB Playground. (Except for the populate query)

Schemas

Customers Collection

• _id: Unique identifier for each customer, an ObjectId.
• name: The customer’s name.
• email: The customer’s email address

[
  {
    "_id": ObjectId("cust100"),
    "name": "Jane Doe",
    "email": "jane@example.com"
  },
  {
    "_id": ObjectId("cust101"),
    "name": "John Smith",
    "email": "john@example.com"
  }
]

Orders Collection

• _id: Unique identifier for each order, an ObjectId.
• customerId: Reference to the customer’s ObjectId.
• itemIds: Array of item IDs purchased in this order.

[
  {
    "_id": ObjectId("ord200"),
    "customerId": ObjectId("cust100"),
    "itemIds": [ObjectId("item300"), ObjectId("item301")]
  },
  {
    "_id": ObjectId("ord201"),
    "customerId": ObjectId("cust101"),
    "itemIds": [ObjectId("item302")]
  },
  {
    "_id": ObjectId("ord202"),
    "customerId": ObjectId("cust100"),
    "itemIds": [ObjectId("item301"), ObjectId("item303")]
  }
]

Items Collection

• _id: Unique identifier for each item, an ObjectId.
• name: Name of the item.
• price: Price of the item.

[
  {
    "_id": ObjectId("item300"),
    "name": "Bluetooth Speaker",
    "price": 99.99
  },
  {
    "_id": ObjectId("item301"),
    "name": "Wireless Mouse",
    "price": 20.50
  },
  {
    "_id": ObjectId("item302"),
    "name": "Keyboard",
    "price": 45.00
  },
  {
    "_id": ObjectId("item303"),
    "name": "USB-C Charging Cable",
    "price": 12.99
  }
]

Use Case 1: Viewing Orders with Customer Information

Initially, you might want to view orders along with the information of the customer who placed them. Using Mongoose’s .populate method, you can easily replace the customerId with the actual customer document, enriching the orders with customer details. This is straightforward because the orders collection directly references the customer collection through the customerId.

db.orders.find().populate('customerId');

[
  {
    "_id": "ord200",
    "customerId": {
      "_id": "cust100",
      "name": "Jane Doe",
      "email": "jane@example.com"
    },
    "itemIds": [
      "item300",
      "item301"
    ]
  },
 ...
]

Use Case 2: Viewing Customers with Their Orders

However, consider the reverse scenario: you want to see all customers along with a list of all orders they have placed. Here, the challenge arises because the customers schema does not inherently contain references to orders. Using .populate would not be possible directly, as it requires a reference within the document being queried.

This limitation illustrates a perfect scenario for using the $lookup operator from MongoDB’s aggregation framework. $lookup can perform a left outer join to the orders collection on the customerId field, effectively appending the matching orders to each customer document.

Introducing the $lookup Operator

MongoDB Lookup Docs

The $lookup stage in MongoDB's aggregation pipeline is a powerful tool that performs a left outer join to another collection within the same database. This capability is crucial for enriching documents with data from related collections and provides a versatile alternative to simpler methods like Mongoose’s .populate.

Knowing the existence of this powerful tool, let’s try to now solve the above Use Case 2 using $lookup .

db.customers.aggregate([
  {
    $lookup: {
      from: "orders",               // The collection to join.
      localField: "_id",            // The field from the customers collection.
      foreignField: "customerId",   // The field from the orders collection.
      as: "orders"                  // The output array containing joined documents.
    }
  }
]);

[
  {
    "_id": "cust100",
    "email": "jane@example.com",
    "name": "Jane Doe",
    "orders": [
      {
        "_id": "ord200",
        "customerId": "cust100",
        "itemIds": [
          "item300",
          "item301"
        ]
      },
      {
        "_id": "ord202",
        "customerId": "cust100",
        "itemIds": [
          "item301",
          "item303"
        ]
      }
    ]
  },
  ...
]

$lookup with Pipeline for Deep Population

To enhance the depth of our aggregation queries further, let’s address the remaining limitation where the itemIds in the orders are still showing as ObjectIds instead of being populated with item details. We can achieve this by using a nested $lookup within an existing $lookup—essentially a lookup inside another lookup.

Using the pipeline option within $lookup, we can execute a more complex query that not only joins documents but also allows for operations like matching and projecting within the nested $lookup. This method is perfect for fetching detailed item data within each order.

db.customers.aggregate([
  {
    $lookup: {
      from: "orders",
      let: { customer_id: "$_id" }, // Define the local variable to use in the pipeline
      pipeline: [
        { $match: { $expr: { $eq: ["$customerId", "$$customer_id"] } } },
        { $lookup: {
          from: "items",
          let: { order_item_ids: "$itemIds" },
          pipeline: [
            { $match: { $expr: { $in: ["$_id", "$$order_item_ids"] } } } // Match items from the order's itemIds
          ],
          as: "items"
        }}
      ],
      as: "orders"
    }
  }
]);

[
  {
    "_id": "cust100",
    "email": "jane@example.com",
    "name": "Jane Doe",
    "orders": [
      {
        "_id": "ord200",
        "customerId": "cust100",
        "itemIds": [
          "item300",
          "item301"
        ],
        "items": [
          {
            "_id": "item300",
            "name": "Bluetooth Speaker",
            "price": 99.99
          },
          {
            "_id": "item301",
            "name": "Wireless Mouse",
            "price": 20.5
          }
        ]
      },
      {
        "_id": "ord202",
        "customerId": "cust100",
        "itemIds": [
          "item301",
          "item303"
        ],
        "items": [
          {
            "_id": "item301",
            "name": "Wireless Mouse",
            "price": 20.5
          },
          {
            "_id": "item303",
            "name": "USB-C Charging Cable",
            "price": 12.99
          }
        ]
      }
    ]
  },
  ...
]

Using nested $lookup operations with the pipeline option allows for much more sophisticated queries, including conditions and transformations within each join. This method is especially useful for deeply nested data structures or when multiple collections must be joined with specific conditions. It also makes the aggregation process more efficient by reducing the amount of data processed in each stage by applying filters early in the pipeline.

Introducing the $unwind Operator

MongoDB Unwind Docs

The $unwind stage in MongoDB's aggregation pipeline is used to deconstruct an array field from the input documents to output a document for each element. Each output document is the input document with the value of the array field replaced by the element. This is particularly useful when operations such as grouping or calculating aggregates on array elements are needed.

Consider a scenario where a business wants to analyse the total sales for each item across all orders, potentially to assess which items are best sellers. To accomplish this, we must first flatten the items array that we enriched in the previous section using $lookup. After unwinding, we can then group by item and sum up quantities or calculate total sales.

db.orders.aggregate([
  {
    $lookup: {
      from: "items",
      localField: "itemIds",
      foreignField: "_id",
      as: "itemDetails"
    }
  },
  { $unwind: "$itemDetails" },
  {
    $group: {
      _id: "$itemDetails._id",
      name: { $first: "$itemDetails.name" },
      totalSales: { $sum: "$itemDetails.price" }
    }
  }
]);

[
  {
    "_id": "item300",
    "name": "Bluetooth Speaker",
    "totalSales": 99.99
  },
  ...
]

To explore a more complex scenario involving all three collections (customers, orders, and items), let's consider a business use case where we want to generate detailed sales reports for each customer. This report will include the total amount spent by each customer, detailed by each item they've purchased, reflecting the quantities and total prices.

db.customers.aggregate([
  {
    $lookup: {
      from: "orders",
      localField: "_id",
      foreignField: "customerId",
      as: "orders"
    }
  },
  {
    $unwind: "$orders"
  },
  {
    $lookup: {
      from: "items",
      localField: "orders.itemIds",
      foreignField: "_id",
      as: "orders.items"
    }
  },
  {
    $unwind: "$orders.items"
  },
  {
    $group: {
      _id: {
        customerId: "$_id",
        orderId: "$orders._id"
      },
      customerName: {
        $first: "$name"
      },
      customerEmail: {
        $first: "$email"
      },
      items: {
        $push: {
          itemId: "$orders.items._id",
          itemName: "$orders.items.name",
          itemPrice: "$orders.items.price"
        }
      },
      orderTotal: {
        $sum: "$orders.items.price"
      }
    }
  },
  {
    $group: {
      _id: "$_id.customerId",
      name: {
        $first: "$customerName"
      },
      email: {
        $first: "$customerEmail"
      },
      orders: {
        $push: {
          orderId: "$_id.orderId",
          items: "$items",
          total: "$orderTotal"
        }
      },
      totalSpent: {
        $sum: "$orderTotal"
      }
    }
  }
]);

[
  {
    "_id": "cust100",
    "email": "jane@example.com",
    "name": "Jane Doe",
    "orders": [
      {
        "items": [
          {
            "itemId": "item300",
            "itemName": "Bluetooth Speaker",
            "itemPrice": 99.99
          },
          {
            "itemId": "item301",
            "itemName": "Wireless Mouse",
            "itemPrice": 20.5
          }
        ],
        "orderId": "ord200",
        "total": 120.49
      },
      {
        "items": [
          {
            "itemId": "item301",
            "itemName": "Wireless Mouse",
            "itemPrice": 20.5
          },
          {
            "itemId": "item303",
            "itemName": "USB-C Charging Cable",
            "itemPrice": 12.99
          }
        ],
        "orderId": "ord202",
        "total": 33.49
      }
    ],
    "totalSpent": 153.98
  },
  ...
]

Bonus Section: Optimising MongoDB Queries

Thank you for following along with our in-depth exploration of MongoDB’s powerful aggregation framework! Your willingness to dive deep into these complex queries will surely enhance your skills in managing and querying MongoDB databases. As a token of appreciation, here are two essential tips that can help optimise your MongoDB queries, especially when working with complex data models involving multiple lookups and unwinds.

1. Applying Proper Indexing to Improve Query Speed

MongoDB Indexing Docs

Indexing is a critical factor in enhancing the performance of database operations. In MongoDB, indexes support the efficient execution of queries. Without indexes, MongoDB must perform a collection scan, which scans every document in a collection, to select those documents that match the query statement. Here’s how proper indexing can help:

• Targeted Indexing: Ensure that fields used in $match, $sort, and $group stages of your queries are indexed. MongoDB uses these indexes to quickly locate and retrieve the data without scanning every document.
• Compound Indexes: When multiple fields are used in query conditions, consider creating a compound index that includes these fields. MongoDB can then use this single index to fulfill complex queries efficiently.
• Index Usage in $lookup: For operations that use the $lookup stage, indexing the foreign field can drastically improve the performance, as MongoDB can use the index to quickly perform the join operation.

2. Optimising Queries with Strategic Ordering of $lookup and $unwind

When your queries involve multiple $lookup and $unwind operations, the order in which you place these operations can significantly impact performance:

• Minimise Early Unwinds: Avoid using $unwind at an early stage in your aggregation pipeline. The $unwind operation can significantly increase the number of documents processed by subsequent stages. If possible, filter your data using $match or reduce the dataset with $group before you apply $unwind.
• Ordering Lookups: When applying multiple $lookup, make sure you perform lookups on array field at the last and if possible, based on your use case only apply unwinds after all the lookups have been performed. This will prevent redundant lookups performed for the unwinded array objects.
• Filter Before Join: Apply $match operations to filter data as much as possible before performing a $lookup. This reduces the amount of data that needs to be joined and can prevent unnecessary processing.
• Nested $lookup Optimisation: When using nested $lookup, ensure that the inner lookup only processes the necessary documents. Use the let and pipeline options in $lookup to apply conditions that reduce the number of documents that need to be joined.

Suboptimal Query Structure

Here’s a version of the query where we first join and unwind items, followed by a join on customers, which could lead to redundant operations:

db.orders.aggregate([
  {
    $lookup: {
      from: "items",
      localField: "itemIds",
      foreignField: "_id",
      as: "itemDetails"
    }
  },
  { $unwind: "$itemDetails" },  // Unwinding early increases the document count
  {
    $lookup: {
      from: "customers",
      localField: "customerId",
      foreignField: "_id",
      as: "customerDetails"
    }
  }
]);

In this structure:

• Each order is first joined with the items collection, potentially multiplying the number of documents if orders contain multiple items.
• After unwinding the items, the number of documents may increase significantly — each document now represents an order with exactly one item.
• A subsequent $lookup on customers is applied to each of these unwound documents. If the initial order contained 5 items, and there were 100 such orders, after unwinding, we're performing the $lookup on 500 documents.

Optimised Query Structure

A more efficient approach would be to first join customers and then join and unwind items, as shown below:

db.orders.aggregate([
  {
    $lookup: {
      from: "customers",
      localField: "customerId",
      foreignField: "_id",
      as: "customerDetails"
    }
  },
  {
    $lookup: {
      from: "items",
      localField: "itemIds",
      foreignField: "_id",
      as: "itemDetails"
    }
  },
  { $unwind: "$itemDetails" }  // Unwind after all lookups
]);

In this optimised structure:

• Each order is first joined with the customers collection, which does not increase the number of documents—each order still corresponds to one document.
• Only after all orders have customer details attached do we perform the $lookup on items.
• Unwinding the items now does not affect the performance of joining customers, as it happens afterward.

That’s a Wrap — Thanks for Sticking With Me!

We’ve finally hit the end of our deep dive into some of the cooler parts of MongoDB, and I just want to say thanks for nerding out with me. It’s been a real treat to unpack all these tricky bits like nested $lookups and getting the most out of $unwind.

Catch you later, and keep on querying!

In this tutorial we will learn how to use JWT (JSON Web Tokens) to create register, login and logout views in Django Rest framework (DRF).

We will use two tokens in this tutorial Refresh Token and Access Token. The client side server will store both the tokens and use them to send a request to an API endpoint which requires authentication of the user. When the user sends the login request with his credentials they get in return the two above mentioned two tokens which they need to store in their session. Now to access any end point which requires user authentication they need to send their access token in the request header, but then whats the use of Refresh token?

As the name suggests Refresh Token is like a refresh button which is used to refresh the Access Token, we will discuss it in detail when we will create User Logout view but for the time being just know that Access Token is very short lived and gets expired after certain time and to get the new token user needs to request it using refresh token.

Let’s first install the required packages —

pip install djangorestframework
pip install djangorestframework-simplejwt

Now let’s set up the settings.py file now and make the following changes in it

Now let’s start by first extending the User model to create tokens.

In this user model we created a function tokens to assign the user with Refresh and Access Tokens.

Now let’s create the serializers for Register, Login and Logout.

Let’s try to understand these serializers in detail starting from RegisterSerializer.

In RegisterSerializer we set the password field to write only so that the user can’t see that in the request sent, then we accept fields - email, username and password to register the user. We then create a validate method to validate the username and then the final create method to create the user with the provided details.

In LoginSerializer we again set the password field to write only so that the user cant see that in the request sent. Then we need a method get_tokens which in returns call the method tokens from User model to get the tokens for the user that’s trying to send the login request to our API. After that we create a method to check whether the user provided correct username and password and if yes then return him the details with the Access and Refresh Tokens.

After the user gets tokens he can send request to any API endpoint that requires user authentication by adding access token in headers and in case the token expires we request a new token by using refresh token which we will see when we will create urls file.

Coming to the logout view which is a bit tricky we need to understand that we need to keep the JWT stateless, in simple words the authentication details are stored on client side and not on server side so we need to somehow blacklist those details/tokens so that the user can’t use them again after logging out.

To do that we use a Simple JWT blacklist option that blacklists the refresh token so that it can not be used again. Now you may be wondering that why only refresh tokens when the actual token we use for authentication is access token? The answer to that is as I already said the beginning that access token is very short lived in fact every time you need to send request to an endpoint you will need to get the new access token. So whats the better way to blacklist the refresh token itself so that the user cant request for new access token in other words user successfully logged out booyaaahhh!!!!.

Now all the heavy work is almost done we just need to create the views and urls and we are done. So let’s create views first —

Just link the serializers to the respective views, call the methods and we are done with views as all the heavy lifting was already done in serializers file.

Now lets assign these views to the urls.

We link the views to their respective urls but notice the token_refresh url, as I mentioned earlier about the refresh token, Simple JWT provide a predefined TokenRefreshView which we can directly use and all the heavy lifting is already done for us!! isn’t DRF just simply awesome 😜.

Now its time to finally test our API so just run our own home sweet home localserver 127.0.0.1, oh wait there’s command for that too in django 😜 just run

python manage.py runserver

Open your postman and lets check our endpoints.

So our task is complete here, you can now import this app in any of your projects which requires Login, Logout and Registration of the user.

Register, Login and Logout in Django Rest Framework using JWT was originally published in Mozilla Firefox Club VIT Vellore on Medium, where people are continuing the conversation by highlighting and responding to this story.