MongoDB Aggregation Framework

I have explained below topics in brief, so that you get an idea on how to get started with MongoDB Aggregation Framework.

What You Will Need

1. Server — MongoDB Community Server
2. Client — MongoDB Compass
3. MongoDB Database Tool
4. For performing operations I have added sample databases with collections and sample operations, check out this repository (take a look at README for better understanding on how you can use this repository for your learning) — MongoDB Aggregation

Table of Contents :

1. Cursor

2. Aggregation Framework

a. Aggregation Pipeline Stages

b. Single Purpose Aggregation methods

c. Update with Aggregation Pipeline

d. Aggregation Expressions

3. SQL Procedure like StoredJS Function on the Server

4. Aggregation Pipeline as Alternative to Map-Reduce

5. Custom aggregation $function or $expr in JavaScript

6. Quiz on Aggregation

1. Cursor

The Cursor is a MongoDB Collection of the document which is returned upon the find method execution.

db.collection.find() function returns a cursor

Note: If a cursor inactive for 10 min then MongoDB server will automatically close that cursor.

Cursor Methods

Using Query tool, execute these methods and observe the behavior.

2. Aggregation Framework

Aggregation is a way of processing a large number of documents in a collection by means of passing them through different stages.

a. Aggregation pipelines, which are the preferred method for performing aggregations.

• An aggregation pipeline consists of one or more stages that process documents.
• Each stage performs an operation on the input documents. For example, a stage can filter documents, group documents, and calculate values.
• The documents that are output from a stage are passed to the next stage.
• All except the $out, $merge, and $geoNear stages can appear multiple times in a pipeline.
• Note that $out, $merge must be the last stage under pipeline.
• allowDiskUse()option allows MongoDB to use temporary files on disk to store data exceeding the 100 megabyte system memory limit while processing a blocking sort operation

Pipeline Stages

Stages

- $project

This stage is used to select certain fields from a collection. We can also add, remove or reshape a key.

run playground

- $match

It is used in filtering operation and it can reduce the number of documents that are given as input to the next stage.

run playground

- $group

It groups all documents based on some keys.

run playground

- $sort

It is used to sort all the documents. Sorting stage cannot use more than 100 MB of RAM

- $skip

With this, it is possible to skip forward in the list of documents for a given amount of documents

- $limit

This limits the number of documents to look at, by the given number starting from the current positions

run playground

- $unwind

We can use unwind for all documents, that are using arrays.

run playground

- $lookup

Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. To each input document, the $lookup stage adds a new array field whose elements are the matching documents from the "joined" collection. The $lookup stage passes these reshaped documents to the next stage.

Get Singh Jyoti’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

- $out & $merge

Takes the documents returned by the aggregation pipeline and writes them to a specified collection. The $out / $merge stage must be the last stage in the pipeline.

b. Single Purpose Aggregation Methods, which are simple but lack the capabilities of an aggregation pipeline.

- distinct(), count(), estimatedDocumentCount()

c. Updates with Aggregation Pipeline

- With the update operations, the aggregation pipeline can consist of the following stages:

$addFields, $set $unset, $replaceRoot, $replaceWith

d. Aggregation Expressions

- There are many types of stages in the Aggregation Framework that don’t allow expressions to be embedded. Examples of some of the most commonly used of these stages are:

$match, $limit, $skip, $sort, $count, $lookup, $out

- There are Expressions like Arithmetic, Data, Array, Conditional, Boolean, Set, Text Search, String, Variable, Literal, Accumulator. (Will cover only accumulator)

- Accumulator Operators

• $sum, $avg, $max, $min — can be used with both $group and $project stages.
• $first, $last, $push, $addToSet — Available in $group stage only.

check repository files for more detailed examples on aggregation pipeline

3. Store a JavaScript Function on the Server

Why we use SQL Stored procedures and How we achieve this in MongoDB?

In RDBMS (SQL Server),

- A stored procedure is SQL code stored on a database for repeated usage and to perform complex query operations. You can think of a stored procedure as equivalent to functions for databases.

- However, modern databases such as MongoDB have solved this problem by using a document model.

In NoSQL (MongoDB)

- Early versions of MongoDB tried to implement a concept similar to stored procedures as stored functions.

- There is a special system collection named system.js ( as a collection under system folder) that can store JavaScript functions for reuse.

- These JavaScript functions could be stored on the server and later reused as part of a Map-Reduce API.

- However, these stored functions are now slowly being phased out.

Note — Do not store application logic in the database. There are performance limitations to running JavaScript inside of MongoDB. Application code also is typically most effective when it shares version control with the application itself.

run this file using mongoDB query tool— StoredJSExample

4. Aggregation Pipeline as Alternative to Map-Reduce

- An aggregation pipeline provides better performance and usability than a map-reduce operation.

- Various map-reduce expressions can be rewritten using aggregation pipeline operators, such as $group, $merge, and for custom functions Use $accumulator , $function operators to define custom aggregation expressions in JavaScript.

Example

The following map-reduce operation on the orders collection groups by the cust_id, and calculates the sum of the price for each cust_id:

mapReduce example

Ø Using mapReduce (deprecated)

run these file using mongoDB query tool — mapReduce-example/ loadScript-example

Ø Using Aggregation Pipeline as Alternative (Recommended)

• You can rewrite the operation into an aggregation pipeline without translating the map-reduce function to equivalent pipeline stages using $group, $merge

run this file using mongoDB query tool — withoutFunction-example

5. Custom aggregation $function or $expr in JavaScript

- Executing JavaScript inside an aggregation expression may decrease performance.

- Only use the $function operator if the provided pipeline operators cannot fulfill your application's needs.

Ø Using $function Aggregation

run this file using mongoDB query tool — function-example

Ø $expr in JavaScript

run this file using mongoDB query tool — expr-example

QUIZ on Aggregation

check for Questions and Solutions— Quiz folder

References -

https://www.mongodb.com/docs/manual/tutorial/store-javascript-function-on-server/

https://www.mongodb.com/features/stored-procedures

https://www.mongodb.com/docs/manual/reference/map-reduce-to-aggregation-pipeline/

https://www.mongodb.com/docs/manual/reference/aggregation-commands-comparison/

https://www.mongodb.com/docs/manual/reference/operator/aggregation/

https://www.mongodb.com/docs/manual/tutorial/update-documents-with-aggregation-pipeline/

https://www.mongodb.com/docs/manual/core/aggregation-pipeline-optimization/

https://www.marcelbelmont.com/nosql-workshop/docs/mongodb-aggregation.html

https://www.mongodb.com/docs/manual/reference/sql-comparison/