MongoDB Aggregate

This is my original tutorial, republished on this website. The examples are preserved. Tool versions and interfaces may have changed.

In this tutorial, we will learn about MongoDB Aggregate. To start with, what is aggregate? The simple definition is “ to form or group into a class or cluster”. Whenever we talk about aggregate function, may it be MongoDB or otherwise, we are trying to get a result that is the combined value extracted from various sources. Therefore, the Aggregation framework is an operation process that formats several documents, retrieves data according to the query, and then gives us the results. Data from a number of documents are extracted and grouped together to return the desired data. Learning this will be helpful as MongoDB currently provides official driver support for all popular programming languages like C, C++, C#, Java, node.js, Perl, PHP, Python, Ruby, Scala, Go, and Erlang. It also is NoSQL instead of SQL, which makes manipulation of new documents easier.

Overview

This article will cover:

What is aggregation in MongoDB?
How does aggregation work in MongoDB?
The aggregation pipeline method
Map-reduce function method
Single-purpose method
What is an example of aggregation in MongoDB?
How fast is MongoDB aggregation?

What is aggregation in MongoDB?

Dealing with a lot of documents, the information we require is almost a combination of small data from a large chunk. MongoDB Aggregation operations process the large chunk of data and give us the computed value. MongoDB Aggregation group values from multiple documents together. You can use various operations on the grouped data to get a desired single result. Here is the documentation for aggregation by MongoDB.

How does aggregation work in MongoDB?

Namely, there are three ways to perform aggregation in MongoDB:

the aggregation pipeline
the map-reduce function
single-purpose aggregation

The aggregation pipeline method

MongoDB aggregation pipeline is a method of data processing pipelines where the output of the one processed data is used as the input in the next process. The process has various aggregation pipeline stages. This way the process is conducted in different pipeline stages which are the subset of the original data. Documents enter a multi-stage pipeline that transforms the documents into an aggregated result. Collection => (Aggregate functions) => Result.
Collection => $project => $match => $group => $sort => Result.

MongoDB Aggregation Example

Here, we have an example of the Aggregation pipeline method. Here, we have a document collection called students. In SQL we would have tables. In Mongo, we call them collections. Bear in mind, MongoDb is NoSQL. The collection students is a record of people signing up for online courses. This will be our input document. Students are given an id (This id field will always have a unique value), their classes, section, and the course fee. The below file is in JSON. MongoDB uses BSON to store files. BSON is the binary encoding of JSON-like documents that MongoDB uses when storing documents in collections. students

{
student_id: "P0001"
class: 101,
section: "A",
course_fee: 12
}, 
{
student_id: "P0002"
class: 102,
section: "A",
course_fee: 8
}, 
{
student_id: "P0002"
class: 101,
section: "A",
course_fee: 8
},
{
student_id: "P0004"
class: 103,
section: "B"
course_fee: 19
}

Now, If we run the function on a mongo shell or MongoDBCompass, then the pipeline uses the result of the aggregate function as input for the next function (first match stage then group stage). This causes data transformation. The query:

db.students.aggregate([
                { $match: { section: "A" } },
                { $group: { student_id: "student_id", total: {$sum: "amount" }}} 
                        ])

The result we get:

{
student_id : "P001",
total: 12
},
{
student_id : "P002"
total: 16
}

Here, the computed results show the total number of students. The computed fields are student_id and the total they have to pay for their courses. These are all students from section A. This is the output document that has all the information that you need. The pipeline goes, students => $match => $group => desired result. Based on your need you can add other aggregate functions at any stage in the pipeline. Commonly, match stage is found at the beginning and the sort stage at the end.

Map-reduce function method

MongoDB also provides map-reduce operations to perform aggregation. Map-reduce uses custom JavaScript functions to perform the map and reduce operations, as well as the optional finalize the operation.
.
Taking the same collection as above

{
student_id: "P0001"
class: 101,
section: "A",
course_fee: 12
}, 
{
student_id: "P0002"
class: 102,
section: "A",
course_fee: 8
}, 
{
student_id: "P0002"
class: 101,
section: "A",
course_fee: 8
},
{
student_id: "P0004"
class: 103,
section: "B"
course_fee: 19
}

Using MongoDB Aggregate:

db.students.mapReduce(
                        function() {emit this.student_id, this.course_fee );},
                        function(key,values) {return Array.sum(values) },
                        {
                        query: { section: "A" },
                        out: "Students"
                }
)

Here, we get first query:

{
student_id: "P0001"
class: 101,
section: "A",
course_fee: 12
}, 
{
student_id: "P0002"
class: 102,
section: "A",
course_fee: 8
}, 
{
student_id: "P0002"
class: 101,
section: "A",
course_fee: 8
}

Then map:

{ "P0001": [12] }

{ "P002": [8,8] }

Lastly, Reduce. The result we get:

{
student_id : "P001",
total: 12
},
{
student_id : "P002"
total: 16
}

The first method of the pipeline is preferred rather than this one. Here, you can see the comparison between different commands. If you want better performance and a coherent interface then the Aggregation pipeline is better than map-reduce. You can find more details about this topic with Map-Reduce Examples and Map-Reduce to Aggregation Pipeline.

Single purpose method

Single-purpose aggregation uses the concept of sorting to present the result in a certain manner. It also limits the number of results that appear at the end of a query. The syntax for Single purpose aggregation operators are:

db.collection.estimatedDocumentCount({<options>})

Use: Returns the count of all documents in a collection or view. The method wraps the count command.

db.collection.count({<query>,<options>})

Use: Returns the count of documents that would match a find() query for the collection or view.

db.collection.distinct({<field>,<query>,<options>})

Use: Finds the distinct values for a specified field across a single collection or view and returns the results in an array. All of these operations aggregate documents from a single collection. They have the advantage of simple access to common aggregation processes, they lack the flexibility and capabilities of the aggregation pipeline and map-reduce. For embedded objects or arrays, the Single purpose aggregate method can’t be used. Some of the various aggregate functions are:

$project - To select specific fields from a collection.
$match- To filter thereby reducing the input data for the next stage.
$group - To group together data based on given criteria.
$sort - To sort the documents. The sort stage is usually at the end.
$skip - To skip forward in the list of documents provided.
$limit - To limit the number of input documents to process.
$unwind - To unwind documents that are using arrays.

The documentation for MongoDB Aggregation commands and methods.

How fast is MongoDB aggregation?

Now, how fast is the aggregation query: The aggregation query's speed depends upon the amount of data stored in the embedded doc. If you store a million entries into the embedded doc qualities, it will create a performance overhead while unwinding those millions of entries. So, it all comes to how you create your database schema. Where there is an array field within the input documents, you will sometimes need to output the document several times. This is because you need to output once for every element of that array from the array field. Also, for faster querying you can look into the multi-key indexing and sharding approaches for MongoDB. There are advanced concepts you should look into when you get the hang of MongoDB. Especially understand how to optimize pipeline. If you want datasets to experiment with, MongoDB has samples you can download and use.