We will spend nearly all our time in the mongo shell. This post assumes that the reader has MongoDB installed locally, and has basic familiarity with MongoDB and mongo shell commands. The mongo shell is a JavaScript environment; we’ll use basic JavaScript in the examples below.

For a nice introduction to MongoDB, refer to An Overview of MongoDB & Mongoose by Jim Medlock.

Indexes

MongoDB indexes are defined at the collection level, and can be created on any field (or combination of fields) present in that collection’s documents. By default, MongoDB creates an index on the _id field. We can create additional indexes to support our application’s access patterns. This simply means that we should create indexes for fields that we commonly include in our queries. See here for more information on indexing strategies.

Setup

We need a database. In the mongo shell, the use command creates (or switches to) a specified database. We’ll call ours indexingdemo:

use indexingdemo

Now we need some documents. We’ll use JavaScript in the mongo shell to generate an array of 3 million documents. Then we can bulk-insert the array into the database. Our collection will consist of incrementing student IDs and random scores for quizzes, exams and homework.

I use a nested for loop to populate a docs array; 1 million outer iterations * 3 inner iterations = 3 million total documents:

var docs = [];
for (var i = 0; i < 1000000; i++) {
  var types = ["quiz", "exam", "homework"];
  for (var j = 0; j < 3; j++) {
    docs.push({
      student : i,
      type    : types[j],
      score   : Math.round(Math.random()*100)
    });
  }
}

Then insert the array of documents into the scores collection (which will be created on the fly):

db.scores.insert(docs)

That could take a few minutes depending on the speed of your hardware. When finished, MongoDB will return a BulkWriteResult. The nInserted property should equal 3000000:

We can query our database now. For example, we may want to see all of one student’s scores:

db.scores.find({ student: 585534 }).pretty()

Our query returns 3 documents — one for each of the student’s quiz, exam, and homework scores.

Before creating our index, let’s get a performance baseline. We can chain the .explain() method to the end of our query, passing in "executionStats" to return detailed performance information:

db.scores.find({ student: 585534 }).explain("executionStats")

That produces a lot of output. Scroll up a bit to find the property executionTimeMillis. We’ll use this execution time when comparing test runs.

Now let’s create our index. Assume that queries typically include both student and type fields. We will create a compound index on student and type. This will support efficient queries for just student or student and type.

db.scores.createIndex({ student: 1, type: 1 })

Note the "numIndexesBefore" : 1 — that prior index is the one MongoDB automatically created for ObjectId.

Now let’s re-run the test query:

db.scores.find({ student: 585534 }).explain("executionStats")

You should notice that the query feels faster — it should complete almost instantaneously. Because — haha! — it does:

Now, executionTimeMillis of 0ms doesn’t really mean it took zero time. If I re-run the query multiple times I occasionally see some non-zero results. Either way, the indexed query is fast.

The above results were obtained in a VirtualBox VM. Windows 7 Pro x64 host OS + Ubuntu 16.04 guest OS using 2 CPU cores (Intel Core i5–3470) and 4GB of RAM. Not elite hardware by today’s standards.

Scaling

Since this is a virtual machine and I have control over the system resources, I can examine how performance scales across various (virtual) hardware configurations by adjusting the VM’s settings. In VirtualBox this is as easy as moving some sliders around. Let’s play with CPU core count and RAM.

The indexed results (in blue) for each configuration are to the left of each of the non-indexed results (in orange) — they’re not easy to see! Each bar in the chart represents the average of 5 runs for each of the following CPU/RAM combinations:

                       Indexed avg     Non-Indexed avg
--------------------------------------------------------
2 cores, 4 GB RAM         0 ms             1100 ms
2 cores, 3 GB RAM         0 ms             1113 ms
2 cores, 2 GB RAM         0 ms             1062 ms
2 cores, 1 GB RAM         2 ms             1196 ms

1 cores, 4 GB RAM         1 ms             1102 ms
1 cores, 3 GB RAM         2 ms             1088 ms
1 cores, 2 GB RAM         1 ms             1061 ms
1 cores, 1 GB RAM         2 ms             1140 ms

What’s going on here? First, we can generalize that our sample query’s performance is not bound by the number of CPU cores or system RAM ≥ 1GB. In all configurations, indexed queries execute within 1–2 ms of zero and non-indexed queries complete in a little over 1 second.

Why?

Without indexing MongoDB has to perform a full collection scan, which means hitting the disk(s). My host system has a single Crucial MX100 SATA3 SSD. I suspect the non-indexed results are limited by my disk I/O. To test this, I created a collection of 1.5 million documents (half the original size) and re-ran the queries. I did not test VM configurations other than the min/max configurations used above. The results below are averages of 5 runs:

                      Indexed avg      Non-Indexed avg
--------------------------------------------------------
2 cores, 4 GB RAM         0 ms              497 ms
1 cores, 1 GB RAM         1 ms              552 ms

Non-indexed queries against a collection of 1.5 million documents execute in about half a second. Compared to the 3 million document collection: half the documents = half the time. This supports my hypothesis re: full collection scans in a disk I/O limited environment.

But what about the lack of scaling for our indexed queries? Recall that our results are essentially 0ms regardless of hardware config. To answer this, let’s begin by looking at the size of our index:

db.scores.totalIndexSize()

Our index is 85,639,168 bytes (81.67 MB). For best performance, indexes should fit in available RAM. And even with just 1 GB of system RAM there is plenty of available RAM to hold our index.

Conclusions

The main takeaway is obvious: create indexes to support your queries — the benefits are incredible! Otherwise queries require full collection scans, which are limited by your server’s disk subsystem I/O. Server CPU and RAM resources were less of a factor than expected. This is encouraging for those who use inexpensive products from VPS hosting providers like Digital Ocean and OVH.

Indexing and MongoDB Query Performance was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

We ended part 2 with a working modular web application. We wrote our modules and stylesheets as separate files, linked together through our index.html and bootstrapped by our app.js. This is an efficient and scalable development process for all the reasons discussed in part 1 of this series.

But it’s a process that produced eight separate files. Our browser has to open and maintain eight separate http requests to retrieve each of those files, and that’s before we include the hosted font or jQuery. Although our files are not large and modern browsers can make these requests in parallel, in a large application the cumulative latency of those requests would result in slower page loads. We can reduce the number of http requests through concatenatation.

We can also increase performance through minification, a kind of compression where our function and variable names are replaced with smaller “tokens”, and all comments, white space and line breaks are removed. This results in smaller file sizes, which means less waiting for our users.

Our demo application includes a feature of es6/es2015: template literals. Template literals are amazingly useful, but their backtick delimiters cause problems for some code minifiers (including the one we will use). Additionally, browser support for es6/es2015 syntax is still evolving. We want to transpile our source code into traditional every-browser-understands-it es5 JavaScript.

Finally, imagine that we composed our stylesheets with SASS instead of CSS. Browsers do not natively understand SASS, so we want to convert that into regular CSS. We won’t actually rewrite our stylesheets — instead we’ll cheat a little for demonstration purposes. And while we’re at it, we should add all necessary vendor-prefixed selectors so different browsers render our application as expected.

That’s a lengthy wish list. In this article we will use Gulp to provide solutions to each of those “wishes”. Gulp will process our source files and create new production-ready files that we can use when deploying our application.

Although our demo is a front-end application, because Gulp depends on Node.js — typically used in back-end applications — the reader should have basic familiarity with Node and NPM and have them installed locally. If you do not, please refer to the official docs for installation instructions.

Aside from a few Node-specific functions (require and pipe), the rest of our code will be written in regular es5 JavaScript. We will spend a bit of time in the command line, but you needn't be a CLI ninja.

Application Structure

Part 2 left us with the following file/folder structure:

|-- /src
|    |
|    |-- /css
|    |    |
|    |    |-- app.css
|    |    |-- background.css
|    |    |-- greeting.css
|    |    |-- quote.css
|    |
|    |-- /js
|         |
|         |-- app.js
|         |-- background.js
|         |-- greeting.js
|         |-- quote.js
|
| index.html

We will work mainly in the root of our project folder; Gulp will be working on all those /src files.

Initialization

Begin by initializing a new project with NPM. While in our project’s root directory, issue the following terminal command:

npm init -y

If you omit the -y flag, NPM will ask you a series of questions about your project. The -y just tells NPM to accept all the default answers. npm init creates a new package.json file in your project's root directory, populated with the answers to npm init's questions.

Gulp

Gulp is a Node.js task runner — it runs the tasks that we will later write. But by itself, Gulp cannot fulfill our introduction’s wish list. To do that, we first need to install some plugins to equip Gulp with new abilities.

But before that, we need to install Gulp. Gulp installs using NPM. There are three parts to its installation: install Gulp globally, then install the local project package, and finally install any needed plugins.

You may need elevated privileges to install Gulp globally. You’ll know if your installation attempt errors out! In a Debian/Ubuntu environment you need to preface the installation command with sudo. Elevated privileges are not required to install the local Gulp package or any of the plugins we need.

Install the global Gulp package using the following command (sudo as needed):

npm install gulp-cli -g

Once that has finished, install Gulp locally. Make sure you are still in your demo project’s root folder and issue the following command:

npm install gulp --save-dev

The --save-dev command-line switch registers the package as a developer dependency in package.json. If you open package.json now, you'll see that it has a new "devDependencies" : {...} property listing "gulp" as its sole dependency. All our various plugins will be registered here after we install them. Let's do that now.

We need the following plugins:

1. gulp-concat — to concatenate (merge together) our separate JavaScript and SASS files
2. gulp-sass — to convert SASS files into traditional .css files
3. gulp-autoprefixer — to automatically add vendor-prefixed selectors to our stylesheets based on CanIUse rules
4. gulp-babel — to transpile any es6 syntax into traditional es5 JavaScript using Babel
5. babel-preset-es2015 — a preset Babel configuration
6. gulp-sourcemaps — to generate a JavaScript sourcemap so that when we’re debugging we can make sense of our browser’s console output
7. gulp-uglify — to minify our JavaScript

We install these plugins using NPM, again using the --save-dev switch to add each module to package.json. You can install each individually or all at once:

npm install gulp-concat gulp-sass gulp-autoprefixer gulp-babel babel-preset-es2015 gulp-sourcemaps gulp-uglify --save-dev

You may have noticed that a node_modules folder was created in your project’s root directory. NPM stores everything required by our dependencies in there. We will not need to touch anything in node_modules.

If you use Git for version control, now would be a good time to add node_modules/ to your .gitignore file. There's no reason for us to track or push those files to a remote repository. Should they be deleted, we can easily recreate them with a single npm install command.

Our "devDependencies" object should now look like this (version numbers current as of publication date):

"devDependencies": {
  "babel-preset-es2015": "^6.22.0",
  "gulp": "^3.9.1",
  "gulp-autoprefixer": "^3.1.1",
  "gulp-babel": "^6.1.2",
  "gulp-concat": "^2.6.1",
  "gulp-sass": "^3.1.0",
  "gulp-sourcemaps": "^2.4.1",
  "gulp-uglify": "^2.1.0"
}

Gulp has everything it needs to process our JavaScript and SASS files. But before we can start writing tasks, we have to create a gulpfile to contain them. Create a new gulpfile.js in your project’s root directory and begin by importing our plugin modules:

/* gulpfile.js */

var gulp         = require('gulp'),
    concat       = require('gulp-concat'),
    sass         = require('gulp-sass'),
    autoprefixer = require('gulp-autoprefixer'),
    sourcemaps   = require('gulp-sourcemaps'),
    uglify       = require('gulp-uglify'),
    babel        = require('gulp-babel');

We’ll build our tasks below those module assignments.

Processing CSS

We want Gulp to concatenate our separate stylesheets, convert SASS to CSS, add any necessary vendor-prefixed selectors and output the result as a single file. But wait — we don’t actually have any SASS. Lucky for us, traditional CSS is also valid SASS, so we can fake it by just renaming our file extensions. In our project’s /src/css/ folder, rename all the .css files to .scss:

|-- /src
|    |
|    |-- /css
|    |    |
|    |    |-- app.scss
|    |    |-- background.scss
|    |    |-- greeting.scss
|    |    |-- quote.scss

Et voilà: SASS!

We’re finally ready to write our first Gulp task. Gulp tasks are methods that take two arguments: the task’s name (as a string) and an anonymous callback function:

// example
gulp.task('taskName', function () {

    // do stuff

});

Create a styles task in our gulpfile.js below the module assignments:

/* gulpfile.js */

var gulp         = require('gulp'),
    concat       = require('gulp-concat'),
    sass         = require('gulp-sass'),
    autoprefixer = require('gulp-autoprefixer'),
    sourcemaps   = require('gulp-sourcemaps'),
    uglify       = require('gulp-uglify'),
    babel        = require('gulp-babel');

// process stylesheets
gulp.task('styles', function () {

  gulp.src('src/css/**/*.scss')
    .pipe(concat('quote-app.scss'))
    .pipe(sass().on('error', sass.logError))
    .pipe(autoprefixer({
      browsers: ['last 2 versions']  // config object
    }))
    .pipe(gulp.dest('dist/css'));

});

Inside the task’s callback, we first give gulp.src() the location of our .scss files. The strange-looking /**/ wildcard instructs Gulp to look for matching files in /src/css and its subfolders. This is especially handy if you organized your project folders by feature. Then we .pipe() those source files through each of our plugins, processing them along the way:

1. The separate .scss source files are first concatenated into a single quote-app.scss file
2. that file is then converted to a standard .css file
3. then vendor-prefixed selectors are added based on the config object (ours targets the two most-recent browser versions)
4. finally, the processed CSS file is written to /dist/css/ (Gulp will create folders if they don't already exist)

Task #1: done. Run it from your terminal: gulp styles

Gulp created new file /dist/css/quote-app.css - take a look at it in your editor. All of our separate .scss files were merged, converted to .css and vendor-prefixed (specifically the flexbox properties):

In index.html, we can now replace the four separate links to our original /src/css/ files with a single <link> to our new /dist/css/quote-app.css:

<!-- ===================== css ====================== -->
<link rel="stylesheet" href="dist/css/quote-app.css">

If you make changes to your source /src/css files, those changes will not propagate to the production /dist/css/quote-app.css file until you re-run gulp styles. That's annoying; we'll learn how to automate this process after dealing with our JavaScript.

Processing JavaScript

Before we write the task to process our JavaScript files, we need to address a potential issue. In the preceding section we passed a string ('src/css/**/*.scss') to gulp.src() which grabbed any SASS files found in src/css and its subfolders. Gulp grabs them irrespective of any order. That's not necessarily a problem with SASS/CSS, especially since we namespaced our selectors. But order is critical with our JavaScript - our modules need to be initialized first so that their public methods exist to be called by app.js.

We enforce a specific file order by passing an array to gulp.src() consisting of our source file names. Create the new array below our imported modules:

/* gulpfile.js */

var gulp         = require('gulp'),
    concat       = require('gulp-concat'),
    sass         = require('gulp-sass'),
    autoprefixer = require('gulp-autoprefixer'),
    sourcemaps   = require('gulp-sourcemaps'),
    uglify       = require('gulp-uglify'),
    babel        = require('gulp-babel');

// ordered array of javascript source files
var sourceJS = [
    'src/js/background.js',
    'src/js/greeting.js',
    'src/js/quote.js',
    'src/js/app.js'        // must come last!
];

Now add a scripts task below the existing styles task:

/* gulpfile.js */

/* ... snip ... */

// process scripts
gulp.task('scripts', function () {

  gulp.src(sourceJS)       // <-- note new array
    .pipe(sourcemaps.init())
    .pipe(concat('quote-app.min.js'))
    .pipe(babel({
      presets: ['es2015']  // babel config object
    }))
    .pipe(uglify())
    .pipe(sourcemaps.write())
    .pipe(gulp.dest('dist/js'));

});

After passing our source file array to gulp.src we .pipe() them from one plugin to the next, just like we did in our styles task:

1. We first initialize the sourcemaps plugin
2. then we concatenate our separate files into a single quote-app.min.js file
3. Babel then transpiles any es6/es2015 to es5 JavaScript using our Babel preset
4. then we minify the transpiled code
5. the generated sourcemap is appended to the the end of our code
6. and finally the processed file is written to /dist/js/

Task #2: done. Now run it in your terminal: gulp scripts

Open the newly-created /dist/js/quote-app.min.js in your editor to see the effects of uglify:

That illegible mess is functionally equivalent to all of our original source code. And if you think that’s funky, check out line 2 — that nonsense is the sourcemap.

We can now replace the separate links to each of our application and module files with a single <script> tag:

<!-- ================ our javascript ================== -->
<script src="dist/js/quote-app.min.js"></script>

You may have noticed that the total size of our four original files (5.3kB) is actually less than our minified quote-app.min.js (12.3kB). That doesn’t seem right — what's going on? Our minified file is larger because we appended the sourcemap to it. If we omit sourcemaps, our minified file weighs in at a lithe 1.6kB.

Watching Files for Changes

Manually re-running Gulp tasks after each minor code change adds an annoying interruption to our work flow. Gulp is supposed to make life easier. And it does, with one of Gulp’s greatest features: watchers. A watcher automatically runs tasks in response to events. We will write one that watches our source .scss files for changes, and runs our styles task in response.

But first we’ll create a default task to contain our watcher. Add it after our scripts task:

/* gulpfile.js */

/* ... snip ... */

// default task contains our watcher
gulp.task('default', ['styles'], function() {

  // watch sass source files and convert on changes
  gulp.watch('src/css/**/*.scss', ['styles']);

});

This task’s construction is slightly different than our previous tasks. Notice the additional array passed to .task() as its second parameter — think of that as a list of the tasks we'll be using inside our default task. Inside the task's callback, gulp.watch() takes two parameters: the path to the files we want to watch, and an array of tasks to run when those files change (we have only one, our styles task).

Gulp default tasks can be executed with just gulp - we do not need to provide the task's name. Running this task initiates a watcher — it will execute but not terminate. It continues to run, watching for additional changes to our files:

So let’s change something. Our quote text currently turns red (color: #F33) on hover - let's try a blueish lavender instead. Make the following change to /src/css/quote.scss:

#quote > a:hover {
    color: #AAF;
}

Keep an eye on your terminal window and save the file — Gulp will automatically re-run our styles task in response to the save event. Since that task generates a new /dist/css/quote-app.css, you can reload your browser window to see the changes:

Good looking, indeed. We can now generate production-ready CSS automatically and without workflow interruption.

Summarizing

At the end of part 2, we had a perfectly functional modular application. We could have left it at that; we had achieved our design goals after all. But in the process, we introduced the potential for degraded performance.

Foreseeing that and the other issues in our introduction, we used Gulp to generate production-ready files in preparation for deployment. We added vendor-prefixed selectors to our CSS, transpiled our SASS and es6 to CSS and es5 JavaScript, and minified our scripts for additional performance. Finally, we added a watcher to automate CSS processing in response to changes to our source stylesheets.

There are other build tools — Webpack and Grunt are very popular and do similar things. We picked Gulp because of how quickly we can get results, allowing us to spend more time developing applications.

Just like we can reuse our modules between projects, we can reuse our gulpfile in any other project that shares the same modular file/folder structure — another reason to adopt modular application design patterns.

Conclusion

This wraps up this series on writing modular JavaScript. It describes a method for planning and building modular applications that is easy to understand and implement, and does not require any exotic syntax, libraries or frameworks. This makes it well-suited for beginning JavaScript programmers.

You might ask, “why not use a framework?” It’s a fair question — Angular for example, is modular by design. I have nothing against frameworks (and I love Angular), but time is precious. The modular design structure described in this series can be used by anyone immediately — it does not require that you spend time first gaining proficiency in a framework’s syntax and methods.

And ultimately, modular design is more about the process. Framework or no framework, the applications you build will benefit from a sound foundation in modular thinking. I hope you can translate these ideas to your own projects.

Update — our greeting module now makes its displayMessage() method public. This change allows our random quote module to call it, rendering both features simultaneously.

Introduction

In this installment we will write a simple application applying the modular design concepts discussed in part 1 of the series. Our application will have multiple features, each controlled by its own module and written as a separate file.

Remember that modular code helps satisfy the design goals we set in part 1: encourage code reuse (modules “plug-in” to projects), enhance code legibility (smaller, more-focused code is easier to make sense of), and facilitate collaboration (modules reveal a standard API for easy interconnection).

The demo application will be simple by design — our goal is to understand the modular design concepts by putting them into practice, not to explore in-depth any features of the JavaScript language. You should have some familiarity with client-side / front-end JavaScript and jQuery. We use jQuery because it simplifies AJAX and DOM manipulation. We will use a few promises and es6 template literals, but nothing too abstract or complex.

Application Features

We are going to build a random quote application. If you’ve worked through Free Code Camp’s front-end curriculum, you have probably already built one. We’ll add a few extra features to ours for additional practice writing and wiring up modules.

Our application will consist of:

1. a greeting feature that displays a different message based on the time of day
2. a random background image feature that asynchronously gets an image from a remote service and displays it as our background
3. a random quote feature that makes an AJAX request to a remote API, processes the response and displays the quote in our view.

Application Structure

We start with a simplified version of the by type structure described in part 1 of this series. We don’t need a /dist folder or anything related to Node just yet — we’ll add those in part 3. Begin by creating a project directory that contains the following files and sub-folders:

|— /src
|    |
|    |— /css
|    |
|    |— /js
|
| index.html

Initial Boilerplate

Our index.html will load our separate application and module files and define some "containers" for our feature modules to hook into. For now, we need a basic html5 document with links to a font, our main stylesheet and jQuery, and <div>s for our features.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">

  <title>Modular Random Quote Machine</title>

  <!-- ==================== fonts ===================== -->
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Condensed">

  <!-- ===================== css ====================== -->
  <link rel="stylesheet" href="/src/css/app.css">

</head>
<body>

  <div id="background"></div>

  <div class="container">
    <div id="greeting"></div>
    <div id="quote"></div>
  </div>

  <!-- =============== vendor javascript ================ -->
  <script src="https://code.jquery.com/jquery-3.1.1.js"></script>

  <!-- ================ our javascript ================== -->

</body>
</html>

Let’s create our application’s main stylesheet now too. Create a new file app.css in your /src/css/ folder:

/* /src/css/app.css */

html,
body {
  margin: 0;
  padding: 0;
}

body {
  background-color: #222;
  color: #fff;
  font-family: 'Roboto Condensed', sans-serif;
}

.container {
  display: flex;
  height: 100vh;
  flex-direction: column;
  align-items: center;
  justify-content: center;
}

Now we can start writing our feature modules.

Random Background Feature

Our application consists of three features, but what about those features — what do they need? You can easily structure your modules by answering that question.

For example, our backgrounds module needs to:

1. cache DOM elements
2. asynchronously get an image
3. assemble an element
4. render that to the DOM
5. provide us a way to initialize it

You can convert and use that list of requirement as pseudo code in your module — instant function documentation! Let’s write the module — create a new file background.js in your /src/js/ folder:

/* /src/js/background.js */

var Background = (function() {

  'use strict';

  // placeholder for cached DOM elements
  var DOM = {};

  /* =================== private methods ================= */

  // cache DOM elements
  function cacheDom() {
    DOM.$background = $('#background');
  }

  // coordinate async assembly of image element and rendering
  function loadImage() {

    var baseUrl = 'https://source.unsplash.com/category',
        cat     = 'nature',
        size    = '1920x1080';

    buildElement(`${baseUrl}/${cat}/${size}`)
      .then(render);
  }

  // assemble the image element
  function buildElement(source) {

    var deferred = $.Deferred(function (task) {

      var image = new Image();

      image.onload = function () {
        task.resolve(image);
      };

      image.onerror = function () {
        task.reject();
      };

      image.src = source;

    });

    return deferred.promise();
  }

  
  // render DOM
  function render(image) { 

    DOM.$background
      .append(image)
      .css('opacity', 1);
  }

  /* =================== public methods ================== */

  // main init method
  function init() {
    cacheDom();
    loadImage();
  }

  /* =============== export public methods =============== */
  return {
    init: init
  };

}());

This module appends an <img> to our target <div> once it receives an image from unsplash. There's a bit of promise voodoo going on in there, because getting the image is asynchronous — we can't set an image’s src attribute until we actually have a image! So we have to wait; promises happen to excel at waiting. We'll see them again in the random quote feature.

Now that we have a module, we should load it with a <script> tag in index.html. But our app still won't do anything because we haven't written our app.js. You can manually call Background.init() in your browser’s console, but that’s … um, not the best user experience. Remember: app.js bootstraps our modules. Let's instruct it to do so — create a new file app.js in your /src/js/ folder:

/* /src/js/app.js */

$(document).ready(function () {

  Background.init();

});

That’s it for now — we’ll add more as we complete additional modules. Recap: our app.js has only one job — to call our modules’ public methods once the document has finished loading. At this point, you can launch index.html in your browser and it will fetch and display a random image. Yay — it works! Also boo — it looks like crap! Let's fix that with a bit of CSS.

Just like we are writing separate JavaScript files for each module, we will write separate stylesheets for each feature. This makes it even easier to reuse a whole module —JjavaScript and CSS — in another application. Just copy both files. Don’t forget to <link> to the separate stylesheet(s) in the <head> of your index.html.

Create a new file background.css in your /src/css/ folder:

/* /src/css/background.css */

#background {
  position: fixed;
  left: 0;
  top: 0;
  height: 100vh;
  width: 100%;

  background-attachment: fixed;
  background-position: bottom center;
  background-repeat: no-repeat;
  background-size: cover;

  opacity: 0;
  transition: opacity 0.75s linear;

  z-index: -1;
}

#background > img {
  height: 100vh;
  width: 100vw;
  object-fit: cover;
}

The above CSS positions, sizes and anchors the image on the page, and sets the opacity of the containing <div> to 0 — meaning it will be invisible. The <div> begins life invisible so that our module's render() function can fade-in the background feature using a CSS opacity transition.

It’s a good idea to namespace each selector in your module stylesheets. Otherwise you may accidentally override some other stylesheet’s rules. For example, above we always include our feature’s selector when referring to any child elements: #backgrounds > img. We might have other modules with images, and we wouldn't want our background feature’s rules to affect them.

That wraps up the random background feature — not too bad, right? The other modules follow a similar pattern.

Greeting Feature

Define the module’s requirements — this module needs to:

1. cache DOM elements
2. determine the time of day
3. craft a greeting based on the time of day
4. render the greeting to our view
5. provide us some way to initialize it

We’ll add a random name picker for fun. Create new file greeting.js in your /src/js/ folder:

/* /src/js/greeting.js */

var Greeting = (function() {

  var DOM = {},
      names = [
        'handsome',
        'smarty pants',
        'good looking',
        'classy',
        'ace',
        'Mr Roboto'
      ],
      dummy = selectName();

  /* =================== private methods ================= */

  // cache DOM elements
  function cacheDom() {
    DOM.$greeting = $('#greeting');
  }

  // pick a name from names array
  function selectName() {
    var ind = Math.floor(Math.random() * names.length);

    return names[ind];
  }

  // assemble time-based greeting message
  function makeMessage() {
    var timeOfDay,
        theDate = new Date(),
        theHour = theDate.getHours();

    if (theHour < 12) {
      timeOfDay = "morning";
    } else if (theHour >= 12 && theHour < 17) {
      timeOfDay = "afternoon";
    } else {
      timeOfDay = "evening";
    }

    return `Good ${timeOfDay}, ${dummy}.`;  // :D
  }

  // render DOM
  function displayMessage() {
    DOM.$greeting
      .text(makeMessage());
  }

  /* =================== public methods ================== */

  // main init method
  function init() {
    cacheDom();
  }

/* =============== export public methods =============== */
  return {
    init: init,
    displayMessage: displayMessage
  };

}());

This module crafts a custom greeting using a name from the names array and a specific message that depends on the current time. Our init() method just caches our DOM element. And we export two public methods: init and displayMessage. In a previous version of this article, we called displayMessage within init(). We now publish displayMessage separately because our quotes module needs to call it. This is explained in the Random Quote Feature section below.

Add a <script> tag to index.html for our new module, and include a call to its public init() method in app.js:

/* /src/js/app.js */

$(document).ready(function () {

  Background.init();
  Greeting.init();

});

Let’s quickly style our greeting. Add a new file greeting.css to our /src/css/ folder:

/* /src/css/greeting.css */

#greeting {
  font-size: 4em;
  font-weight: bold;
  text-align: center;
  text-shadow: 0 0 30px #000;
}

Reload thy browser and be greeted! On to our final feature…

Random Quote Feature

Like we did with our previous modules, begin by defining what this module needs to do:

1. cache DOM elements
2. fetch a random quote from a remote API
3. process the JSON response
4. call our greeting module’s public Greeting.displayMessage() method
5. render the quote to the DOM
6. provide us some way to initialize it

Note: you may have difficulty getting the API to respond if you are not serving your application from a proper server. Both Chrome and Firefox threw up CORS errors when I loaded the static files directly in those browsers. Brackets’ Live Preview works fine. Hosting anywhere (GitHub pages, for example) would also work.

Create a new file quote.js in your /src/js/ folder:

/* /src/js/quote.js */

var Quote = (function () {

  'use strict';

  var DOM = {};

  /* =================== private methods ================= */

  // cache DOM elements
  function cacheDom() {
    DOM.$quoteFeature = $('#quote');
    DOM.$quoteLink    = $(document.createElement('a'));
    DOM.$author       = $(document.createElement('p'));
  }

  // get random quote
  function getQuote() {

    var api = {
      endpoint: 'https://quotesondesign.com/wp-json/posts',
      params: {
        'filter[orderby]'       : 'rand',
        'filter[posts_per_page]': 1,
        'cachingHack'           : (new Date()).getTime()
      }
    };

    $.getJSON(api.endpoint, api.params)
      .then(renderQuote)
      .catch(handleError);
  }

  // handle errors
  function handleError(err) {
    console.log(err);
  }

  // render
  function renderQuote(response) {

    // call Greeting module's public displayMessage() method
    Greeting.displayMessage();

    DOM.$quoteLink
      .attr('target', '_blank')
      .attr('href', response[0].link)
      .html(response[0].content);

    DOM.$author
      .html(response[0].title);

    DOM.$quoteFeature
      .css('background-color', 'rgba(0, 0, 0, .2)')
      .attr('href', response[0].link)
      .attr('target', '_blank')
      .html(DOM.$quoteLink)
      .append(DOM.$author);
  }

  /* =================== public methods ================== */
  function init() {
    cacheDom();
    getQuote();
  }

  /* =============== export public methods =============== */
  return {
    init: init
  };

}());

We use promise syntax again to coordinate the timing of the .getJSON() AJAX response with subsequent function calls. In this case, once the request has successfully resolved, we .render() the quote (wrapped in an anchor tag) and author's name.

In a previous version of this article, our greeting module’s init() method called its own renderer (displayMessage()). Although that works, it results in a diminished user experience: the greeting renders immediately, and then the quote renders after a short delay. This is because the quote module can't render until it receives a quote from the API. We can do better.

Our modular structure makes it easy to improve upon the previous version. We refactored the greeting module above to export its displayMessage() method instead of calling it internally. That allows our quote module (or any other module) to call it. We added that call to renderQuote() so that both features render at the same time.

Special thanks to Chris Coyier for granting permission to use his Quotes on Design API. In addition to curating all those quotes, Chris founded CSS-Tricks and co-founded Codepen.

Add a <script> link for our random quote module to index.html and call the module’s public method in app.js:

$(document).ready(function () {

  Background.init();
  Greeting.init();
  Quote.init();

});

Almost done — we have one last file to write: the quote module’s stylesheet. Create a new file quote.css in your /src/css/ folder:

/* /src/css/quote.css */

#quote {
  font-size: 1.5em;
  padding: .5em 1em;
  width: 80%;
  max-width: 720px;
  border-radius: 8px;
  text-shadow: 0 0 40px #000;
}

#quote > a {
  color: #fff;
  text-align: justify;
  text-decoration: none;
}

#quote > a:hover {
  color: #f33;
}

#quote p {
  padding: .25em 0;
  margin: 0;
}

#quote > p {
  font-style: italic;
  text-align: right
}

Add a link to the stylesheet in index.html and we are done. Our final index.html is a well-organized, logical foundation for our application:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">

  <title>Modular Random Quote Machine</title>
  
  <!-- ==================== fonts ===================== -->
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Condensed">

  <!-- ===================== css ====================== -->
  <link rel="stylesheet" href="src/css/background.css">
  <link rel="stylesheet" href="src/css/greeting.css">
  <link rel="stylesheet" href="src/css/quote.css">
  <link rel="stylesheet" href="src/css/app.css">

</head>
<body>

  <div id="background"></div>
  
    <div class="container">
      <div id="greeting"></div>
      <div id="quote"></div>
    </div>

  <!-- =============== vendor javascript ================ -->
  <script src="https://code.jquery.com/jquery-3.1.1.js"></script>

  <!-- ================ our javascript ================== -->
  <script src="src/js/background.js"></script>
  <script src="src/js/greeting.js"></script>
  <script src="src/js/quote.js"></script>
  <script src="src/js/app.js"></script>
  
</body>
</html>

Summarizing

If all went well, your browser should show you something like this:

We wrote a simple web application based on the modular design principles outlined previously. Each feature’s application logic and CSS is contained in separate files, loaded by index.html and bootstrapped by our app.js. We used a systematic requirements-based approach to build each module. Our modules’ public methods are namespaced to avoid collisions in the global scope. Their CSS selectors are similarly namespaced to avoid colliding with any generic selectors.

A live demo of this application is hosted here. The code and markdown versions of the article can be found at the GitHub repo.

At this point our application structure looks like this:

|-- /src
|    |
|    |-- /css
|    |    |
|    |    |-- app.css
|    |    |-- background.css
|    |    |-- greeting.css
|    |    |-- quote.css
|    |
|    |-- /js
|         |
|         |-- app.js
|         |-- background.js
|         |-- greeting.js
|         |-- quote.js
|
| index.html

Looking good! Time to collect on all those promised mint juleps.

Up Next: Preparing for Deployment

Our application, modules and stylesheets are working together. But the browser has to make a separate request for each of those eight separate files. While the files themselves are not large, the cumulative latency of all those requests delays application load time. In Part 3 we will prepare our application for deployment using Gulp to concatenate, transpile, and minify our files.

Introduction

If you have so much as glanced at a computer programming language, you have surely heard the mantra: Don’t Repeat Yourself. Basically, if you find yourself writing the same snippet of code more than once, you are better off sticking that code inside a function instead. In this post, I will show you how we can extend that mantra beyond its use within a single file. We’ll apply the concept at the application level — within a single project and even across projects — by breaking monolithic applications into smaller, self-contained, reusable modules.

Maybe you want to use some previously-written utility in your new application — instead of rewriting the whole thing, you can just copy your file into your new project and link to it with a <script> tag. The time savings alone makes this a worthwhile endeavor.

But there’s another, possibly greater benefit to modular design: it makes collaborating with other people a whole lot easier. When you are working with a team of developers, where each of you is creating a part of a larger application, you will eventually need to merge all of those separate parts together. If you’ve adopted a modular design structure in advance, joining everything up at the end is trivially easy. Develop modular habits today and you’ll be the hero of your team tomorrow.

Finally, modular design patterns make it much easier to read your code. You should personally care about this because you (or others — see previous paragraph) will eventually have to make sense of code you previously wrote. The fewer lines of code, the quicker you and others can make sense of it. A module is by definition a smaller, focused part of some larger whole.

This article is aimed at beginning JavaScript developers who already have some experience with the language, but who may not yet know how to structure code into modules, or why the modular design pattern is valuable.

I assume that you have basic familiarity with client-side / front-end JavaScript. My examples use a small amount of jQuery, but no other libraries or frameworks are required to get started. I will mention IIFEs and closures, but not go into too much detail about them here.

Application Structure

A modular application begins with a solid file/folder structure. There are many ways to organize a project’s files. I tend to use one of two ways: either by type of file (css, js, assets, etc.) or by application feature (header, landing, login, about, footer, etc). On large projects, organizing by feature is almost a requirement. But we’re going to use the simpler by type system for this post.

|— /dist
|    |
|    |— /js
|    |— /css
|    |— /assets
|
|— /src
|    |
|    |— /js
|    |   |
|    |   | app.js
|    |   | module1.js
|    |   | module2.js
|    |   | module3.js
|    |
|    |— /css
|    |   |
|    |   | style.css
|    |
|    |— /assets
|         |
|         | image.jpg
|         | etc.jpg
|
|— /node_modules
|
| .gitignore
| gulpfile.js
| package.json
| index.html

Briefly:

• /dist - ultimately our production-ready files will live here. Gulp will create them later - we don't need to do anything here.
• /src - we'll write all our in-development code here (css, JavaScript, etc). Later, Gulp will process these files to produce the files in /dist. The module file names used above are just examples - you should definitely use more meaningful file names!
• /node_modules - Node for front-end apps? Yes! We will use Node and NPM to install and run Gulp tasks.
• .gitignore - files and folders specified in this file will be ignored (not tracked) by Git. They also won't be pushed to GitHub. We'll be ignoring the node_modules/ folder - no reason to track or push all those 3rd-party packages.
• gulpfile.js - we'll write our Gulp concatenation, transpilation and minification tasks in here.
• index.html - during development, we'll itemize each script separately. Later, we will just link to a single production-ready JavaScript file.
• package.json - Created by NPM. Among other things, this file catalogs all the packages our application depends on. Ours will include dependencies for Gulp and various Gulp plugins.

The Nitty and the Gritty

Let’s begin by looking at the structure of our index.html. During development, we'll simply link to each of the JavaScript files in our /src/js folder. So our index.html might look like this:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Nice Boilerplate</title>
</head>
<body>

  <!-- =============== vendor javascript ================ -->
  <script src="https://example.com/jquery.min.js"></script>

  <!-- ================ our javascript ================== -->
  <script src="src/js/module1.js"></script>
  <script src="src/js/module2.js"></script>
  <script src="src/js/module3.js"></script>
  <script src="src/js/app.js"    ></script>

</body>
</html>

Order is important! Our modules must load first so that their public methods are available to be called. That statement will make more sense as we progress. For now, know that app.js must load last because we're going to use it to call all our other modules' public methods. Other than that, it's just a normal index.html file. Scaffold your application as you would normally.

Things start to look a little different when we get to our app.js file:

$(document).ready(function () {

  Module1.init();
  Module2.init();
  Module3.init();

);

That’s it! If you’re used to seeing/writing volumes of code inside of $(document).ready(), the above may look a little strange. Our app.js has only one responsibility: call our separate modules' .init() methods. All of the actual application logic resides in those other modules. Each time you add a new module to your application, you add a call to its .init() method here. Easy and organized.

Because we use app.js to bootstrap our other modules, it now makes sense that index.html has to load those modules first. If the modules were loaded after app.js, calling their .init() methods would fail.

We will write each module in a separate file to promote reuse. And we’ll adopt the style known as the revealing module pattern. The “revealing” qualifier hints at the module’s internal structure: some features are private, others are revealed publicly.

Here’s an example:

var Module1 = (function() {

  'use strict';

  // placeholder for cached DOM elements
  var DOM = {};

  /* =================== private methods ================= */

  // cache DOM elements
  function cacheDom() {
    DOM.$someElement = $('#some-element');
  }

  // bind events
  function bindEvents() {
    DOM.$someElement.click(handleClick);
  }

  // handle click events
  function handleClick(e) {
    render(); // etc
  }

  // render DOM
  function render() {
    DOM.$someElement
      .html('<p>Yeah!</p>');
  }

  /* =================== public methods ================== */

  // main init method
  function init() {
    cacheDom();
    bindEvents();
  }

  /* =============== export public methods =============== */

  return {
    init: init
  };

}());

We begin by declaring a variable (Module1) and assigning a function to it. All of the module’s code goes inside this function.

Note that the function is both (wrapped in parentheses and ends with another set of parentheses)()) :

var Module1 = (function () {

  // all module logic goes here

}());

The parentheses () at the end causes the function to be interpreted and evaluated as an expression. This is known as an IIFE, or Immediately Invoked Function Expression. The IIFE means we don't call the function separately — the function is called when the JavaScript engine assigns it to our variable. And that happens as soon as the module loads. The wrapping () are not technically required, but are used by convention to let other programmers know that this function is an IIFE.

In addition to “auto-loading” our module, the function also creates its own local scope for all the variables and functions within it. Anything declared inside a function is scoped to the function — it can’t be called or referenced outside of the function (except when it can! We’ll get to this shortly). Isolating scope this way also prevents adding all kinds of variables and functions to the global scope.

Within the IIFE, our example module uses /* comments */ to organize its contents into four sections.

At the very beginning we declare all required module-scope variables. Standard stuff you already do — no big deal. This example only declares a single variable, an empty object that will cache a DOM element.

Next we define a few private methods. The example methods are commonly found in many front-end applications: cacheDom(), bindEvents(), and render(). They do exactly what you expect them to do. Additional private methods might handle form submission, AJAX requests, manipulating JSON responses, etc. What’s important to note is that these methods are private — we don't need or want them to be available outside of the module they're defined in. And they're not, because they're encapsulated within our IIFE. For example, there's no reason for any module's bindEvents() function to be available globally — it's irrelevant to any other module, so we keep it private.

Also note that multiple modules can have the same private functions. They will never collide with each other because they are restricted to each module’s local function scope. Beyond that, each module can have the same public methods as well, since each of those is namespaced to its own module. For example, you might have the three modules, each with the same public method: modals.init(), greeting.init(), login.init(). Those identically-named methods will never collide with each other because they're all called on unique modules. Very courteous.

In the next section we define our public methods. In this case we only have one: init(). But you could have any number of them. Notice that init() does only one thing: make calls to our private methods. When called, init() closes over the private module-scoped variables and functions, making them available outside the scope they were declared in (yay, closure).

You may ask, “what makes the init() function public? — it looks just like the other private functions above it." Good question — now go to the next paragraph!

The final section returns an object literal. This is what makes the revealing module pattern so cool. What we’re returning is essentially a set of “links” to any private methods that we want to make public outside our module. This allows us to separate the private and public parts of our modules, and — to answer the preceding paragraph’s question — is what makes init() a public method.

Remember that an object’s properties are { key : value } pairs:

return {
  init : init
};

In the above, the left-hand ‘key’ is the name of the method we’ll call outside our module. The right-hand ‘value’ refers to the name of the function defined inside the module. Although it looks strange to see init : init, it's common practice to use the same name externally that we use internally. It's also one less name you have to remember.

Bringing Everything Together

By returning public methods via the object literal, we “attach” them to the variable that we declared at the beginning: Module1 gets the init() method. And because Module1 was declared globally, it and its public methods are available in the global namespace. Once a module exposes its public methods globally, any other module can call those methods. So we can call Module1.init() anywhere we like — which is exactly what we did earlier in app.js. Module1 might have additional public methods — those can be called anywhere in the application as well.

See how it’s all starting to fit together? And how, in a team setting, each team member can work independently and then publish their module’s public methods, adding any necessary calls to their module’s public method(s) in app.js?

I typically limit each module to a single unique application feature or a single unique application utility. An application feature might have many private methods and not need anything other than a single init() method exposed publicly. In contrast, a utility module might have few (or zero) private methods and instead expose all its methods publicly. And utility modules may not need init() methods at all.

Summarizing

We break our application logic into chunks based on single feature or utility. Each team member composes their module(s) as an IIFE, consisting of all the business logic needed by the module. Each module reveals specific public methods, which can be called elsewhere in the application. Our main app.js bootstraps those modules by calling each of their public init() methods. Other project team members recognize your brilliance and buy you mint juleps.

Up Next: Let’s Build a Modular App!

Now that we understand the principles of basic modular application design, we’ll put them to use building a simple web application in part 2.