The Problem: Data Downtime

Like most tech companies, Storyblocks has application databases and ETLs importing them into a data warehouse. Bulk-copying production MySQL databases hosted on Amazon RDS to a data warehouse in Amazon Redshift overnight is easy enough, but three issues with that naive approach are Data Latency, Data Reliability, and Interrupted Workloads.

Those three problems can arise from bulk load operations which drop the target data in the data lake or warehouse. Some ETL tools such as Amazon DMS make it very easy to setup a routine data migration, but by default they will drop or clear the target tables.

Data Reliability can suffer if an ETL process drops target tables before bulk-loading the new data. Any unhandled error in the load process can leave the data lake/warehouse in an incomplete state when people come to work the next day. When more and more data processes are added to the ETL or different workloads are added to the data lake/warehouse, some bulk load operations can lead to Interrupted Workloads. For example, a long-running SELECT operation for an analyst’s derived table or a data scientist’s recommendation engine job could be clobbered by the ETL. As a result of those limitations, the ETL is often only scheduled once per day overnight and Data Latency can forever be a day behind.

A Solution: “Blue-Green” Data Deploys

To avoid the issues above, one solution is to maintain two sets of target data for each data source — a “blue” dataset and “green” dataset — and then a simple view layer on top which all consumers actually use. For example, if we have an accounts database in production, we would keep an accounts_blue and accounts_green database in the data warehouse with populated tables, and an accounts database which just holds views of each table pointing to the “active” database.

The “blue-green” name comes from the software engineering concepts of Continuous Integration/Continuous Delivery and refers to deploying new code to a set of servers and slowly cutting traffic over to the new servers. The analogy doesn’t exactly fit, but I figured it is close enough (also there’s likely a more appropriate data term I don’t know about).

A lot of other ETLs will load data into a temporary table, then change the table name at the end of the process. This is a perfectly reasonable solution, but in some systems it can still lead to interrupted workflows, and at least for Amazon DMS it is not possible to do this out of the box.

Airflow and Task Groups

At Storyblocks, we use Apache Airflow to schedule most data processes. Airflow 2.0 introduced Task Groups, a much-easier-to-use feature compared to SubDags for reusable/composable groupings of tasks. In this case, we had a group of three distinct tasks which could logically be grouped into a repeatable pattern for a given database: 1. Query the warehouse to see which color is inactive, 2. Run the ETL process for the inactive color, 3. Switch the active color in the warehouse (point the view layer to the newest color).

The Storyblocks Nightly ETL

Our main ETL process for loading application databases into our data warehouse consists of Amazon DMS jobs running in parallel, followed by a series of Apache Spark jobs to generate derived tables, and finally a massive collection of SQL scripts maintained by our Analytics department (called “etl_runner” below).

Originally, the DMS jobs would drop and bulk load the target tables. That often led to the issues outlined above. Our machine learning team would attempt to read data in their own overnight data jobs and would often get incomplete results from the still-in-progress DMS jobs or the ETL would clobber their process and delay their team by a day.

The white tasks on the left indicate standard DMS tasks, and the blue ones are the Task Groups. If you click on a Task Group in the Airflow UI it will expand:

A simplified version of the DAG code with Task Groups is here:

The code for selecting the active color in the warehouse is here:

And the code (Redshift-specific) for switching all of the views in a database to point to the new active color is here:

Blue-Green ETLs with Airflow Task Groups was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Storyblocks platform produces millions of events each day. These events allow our growing number of microservices to communicate with each other, let us incorporate user behavior into our search algorithms, and allow product managers and analysts to evaluate new features. These structured event payloads are produced and consumed by frontend code, backend services, and data-intensive scripts and jobs.

Apache Kafka is a wonderful tool that enables asynchronous communication and data transportation at scale, and Apache Avro is a schema management and encoding technology which ensures structured and valid data. Using the new GitHub Actions platform, we created an extremely developer-friendly experience for managing schemas across our entire engineering organization.

Problem

Until recently, our method for gathering frontend and backend event data has been to produce JSON-encoded events onto a Kinesis stream, which then had a custom application validating the events with json-schema and an EMR job storing them in S3. Event schemas were stored in a forked version of Iglu deployed to an EC2 server.

While simple in theory, we faced many challenges with this system: Schemas were not stored in a central location, registering a schema was an error-prone process that required manually running a custom python script, versioning was unenforced and unclear, custom infrastructure and code led to bugs and outages, schema validation was incomplete, backwards-incompatible schemas changes broke downstream scripts, schema files were large and human-unfriendly, common fields were copied and repeated in every file, and invalid events were not discoverable until code was deployed to staging.

Developers found the system difficult to use and fragile, which created friction when creating new events or updating existing ones. Also, the system did not allow easy asynchronous communication between services. As a result, we were slow to track new data, slow to enhance existing data, and stuck in synchronous REST calls between our services. Not a good situation for a startup that “hearts data”.

Solution

Enter Avro. Avro is a widely-used technology for structuring and encoding data, and is a first-class citizen in the awesome Kafka-based Confluent ecosystem. It has schema evolution built-in, and its Avro IDL language makes it easy to write human-readable schemas that can import and reuse common fields.

Given our developers’ lack of enthusiasm for creating events, we wanted to take things a step further and create a pain-free developer experience for managing schemas. Something that Just Works™.

Here are the components:

Central Events Repository

Every developer and product manager on our team knows how to use GitHub, so having a simple “events” repository to hold our schemas made the most sense. It’s easy to browse the schemas using the GitHub interface, and to update or create schemas you follow the same workflow as any other codebase: Branch, code, pull request, and merge.

Avro IDL

Avro IDL is extremely easy to read and understand (much more than JSON, in my opinion). More importantly, it allows importing schemas. This important feature lets us share common schemas between common events, vastly simplifying and DRY-ing up our schema code. For example, all of our backend events have a “meta” field that contains uuid, timestamp, and origin (the service or site that produced it).

Event Libraries

Typed code generally creates less bugs and speeds up the development process. We wanted our developers to be able to import typed event classes and populate them with all of the benefits of types and typeahead. That is much preferable to sending arbitrary JSON or arrays-of-arrays onto the wire hoping they conform to a schema. It also makes it easy to “click into” event classes to see the schemas and available fields.

Lucky for us, there are open-source libraries which compile Avro schemas into Java, Typescript, and Python files. There wasn’t a PHP version, so we built that ourselves. We just needed to figure out how to link all these things together in a user-friendly fashion…

Continuous Integration with GitHub Actions

Continuous Integration offloads the work of testing and deploying to machines so that people can just do their jobs. GitHub recently released their own system for executing code automatically on their platform in the form of GitHub Actions. We configured custom actions to run when engineers pushed code or merged to master in the “events” repository.

On every push to a branch, a validate schemas action checks all schemas against Confluent Schema Registry, hosted on Aiven.io. If a schema change is not forwards- or backwards-compatible, the action fails and GitHub shows a red “x” on that branch. This is also a required check before a pull request can be merged to master. As a result, only valid, non-breaking changes can be made to the master branch of our event repository.

On every merge to master, a “sync schemas and tag” action puts the changes into the schema registry, followed by a “publish event libraries” action. The publish action auto-generates files and a versioned artifact for each language, and publishes the artifact. For example, the “compile typescript” action generates Typescript files for each event and publishes a private npm package to npmjs.org. A developer can install the library by running npm install @videoblocks/events-ts.

Simple Workflows == Happy Developers

Now, when a product manager wants to track a new feature (beyond simple telemetry data) or a developer needs to communicate a message to another system, the workflow looks like this:

1. Design the schema.
2. Create a pull request in the events repository.
3. Merge to master.
4. Update the client library.
5. Code.

Developers can write tests for their events, produce and consume them locally, and can rest assured knowing that they won’t anger coworkers by breaking their downstream scripts. Our latest high-impact microservice for billing customers successfully used the new system to communicate with the other services in our platform, and the engineers found the system easy to use. Events are a tool that enables them rather than a requirement they must implement.

Next Steps

We hope to open-source a version of our events repository in the near future, including our GitHub Actions and example client libraries. Combined with the powers of Kafka and Kafka Connect, these tools can be a huge asset to any development team dealing with structured event data.

Foolproof Schema Management with GitHub Actions and Avro was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Fighting Meetings with Technology and Psychology… and Light Bulbs

Repository: https://github.com/Footage-Firm/meeting-room-traffic-lights

With two-dozen Oscars to present during the Academy Awards, there are a great many acceptance speeches. To keep speakers within a reasonable time limit, play-off music is famously used to tell folks to wrap it up — and it nearly always does the trick!

All too often, I have been stuck in a dragging meeting at work wishing that someone would start the play-off music. For our third annual “Innovation Day” at Storyblocks, my coworker and sometimes-boss Aaron Silverman had the idea to create traffic lights for our meeting rooms to tell people the meeting is ending. Having previously written a blog post on gathering meeting metrics using the Google Calendar API, this idea fit very well with my general displeasure at meetings and my desire to stomp them out at Storyblocks — or at least make them more tolerable.

The idea is simple: Put smart RGB light bulbs in each of our meeting rooms, then connect each bulb up to Google Calendar and change the color based on meeting time. If there is no meeting, the bulb is off. If there is currently a meeting in the room, the bulb is green. If the meeting is almost over, the bulb turns orange. When the meeting is over, the bulb turns red.

Materials

• LIFX Mini Color bulbs
• INJUICY bulb stands
• Meeting Rooms defined as bookable resources in Google Calendar
• 2.4 GHz ONLY network (mixed networks don’t work well).
• AWS

Process

We wanted a system that would be simple, inexpensive, flexible, and reliable. To achieve that, we knew that we wanted little-to-no infrastructure and bulbs that could be controlled remotely over the internet. Cursory internet research showed that others had success controlling LIFX bulbs via their API and their bulbs were relatively cheap at about $30 per bulb (even cheaper on Ebay), so we ordered some along with simple stands that fit our office aesthetic. After that, our game-plan was to write up a simple Typescript application to be deployed on AWS Lambda using the dead-simple serverless module.

Code

The application was coded to allow automatic configuration of rooms and bulbs. If our company changed any room names or added any rooms, we wanted it to be as simple as changing the name of a light bulb or purchasing a new one. Also, if any other groups wanted to use our system, they shouldn’t have to touch the code other than deploying it.

// Handler.ts (syncBulbs function)

const lights = new MeetingTrafficLights()

const lifxNetwork = new LifxBulbNetwork()
lights.addNetwork(lifxNetwork)

const calendarService = new CalendarService()
lights.setCalendar(calendarService)

return await lights.syncBulbs()

The code works by polling Google Calendar to get the list of all meeting rooms in an organization, querying the LIFX API to get the list of bulbs, then mapping rooms to bulbs before setting bulb colors. It will attempt to match bulb labels to room name automatically (so you don’t have to touch the code), but you can optionally override the pairings. For each mapped room/bulb pair, it then checks if a meeting is currently in-progress, almost finished, or finished, and sets the corresponding color.

/// MeetingTrafficLights.ts

if (currentMeeting) {
    logger.debug('Meeting in progress. Setting color to Green.', {room, bulb})
    await bulb.setColor(Color.GREEN)
} else {
    logger.debug('No meeting. Turning bulb off.', {room, bulb})
    await bulb.powerOn(false)
}

The app runs every minute during normal business hours Monday through Friday (to save energy and cut down on Lambda executions).

// serverless.yml

functions:
  syncBulbs:
    handler: bin/Handler.syncBulbs
    name: sync-bulbs
    description: Sets light bulb colors for meeting rooms during weekday business hours (12-24 GMT).
    events:
      - schedule:
          rate: cron(* 12-23 ? * 2-6 *)

Technical Difficulties

Our biggest issue by far was WiFi. The bulbs that we used can only communicate on a 2.4 GHz network, and our office WiFi had the same SSID for each band. It was quite common for a bulb to initially connect successfully and then be completely unable to reconnect after turning off. For our demo, it was easy enough to coast on the initial connection, but we knew that we’d need a dedicated 2.4 GHz network before staking our reputations on a real implementation.

Additionally, connecting to the Google Calendar API is not the most straightforward process. We used a service account that was setup for the Meetrics project (see instructions in that blog post or in this repo), but it still took a good couple hours of experimenting with the googleapis module to get it right.

Live Demo

Our demo went well! The Innovation Day judges were able to move meetings around on Google Calendar and see the bulbs change color. We also showed how easy it was to add a bulb without making any code changes. People were excited to see the bulbs “rolled out to production” in the coming weeks.

After Innovation Day, we were able to do live tests of a bulb in one of our meeting rooms called “The Studio” and see what effects, if any, it had. The first runs were definitely biased by the novelty of the addition to the meeting room, but there did seem to be a natural increased awareness of meeting schedule. Once the bulb turned red, people picked up their hustle to get out.

Results

At the very least, we won the “Cultural Award” at Innovation Day and got our working prototype out the door for live experimentation in our company’s meeting rooms. We will see if the bulbs continue to work in spite of WiFi issues and live users who don’t like being told when to end their meetings.

Feel free to implement your own traffic lights using our open-sourced code at Footage-Firm/meeting-room-traffic-lights! The links to necessary materials and instructions are included in the README.

Happy hacking, and DOWN WITH ALL THE MEETINGS (some are ok).

Interested in monitoring your own organization’s meeting health? Check out Meetrics: https://meetrics.io

Fighting Meetings with Technology and Psychology was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the release of Amazon Aurora Serverless, which recently went into public beta, you can now build a standard web app which is 100% serverless. This post walks through a basic serverless implementation using Lambda, Aurora, and NodeJs.

Lambda Aurora Node… LAN Stack.

Overview

The high-level view of our LAN application is an REST API written in NodeJs and deployed to Amazon API Gateway + Lambda using the wonderful serverless framework with a database hosted using Aurora Serverless. Although not necessary for all web apps, you may easily add a frontend to this setup with a single page app hosted on S3 and served with CloudFront, putting a nice decoupled cherry on top of this delicious serverless cake.

Serverless Framework

The serverless.js library is a great tool which makes it dead simple to deploy Node.js code to Amazon Lambda. The only steps you need to get up and running are to npm install serverless and then configure serverless.yml in the project root.

A bare-bones serverless.yml configuration may look like this:

# serverless.yml
# @see docs.serverless.com

service: my-serverless-api

provider:
  name: aws
  runtime: nodejs8.10

functions:
  helloWorld:
    handler: src/helloWorld
    events:
      - http: ANY /hello

This deploys a function exported in src/helloWorld.js to Lambda and configures API Gateway to launch that function when someone makes an HTTP request (using any method) to https://<generated endpoint>/hello. The endpoint is returned after deploying serverless via serverless deploy.

That setup works well enough if you want to just launch a simple javascript function when you hit an endpoint, but what if you want to create a fully-fledged web app? When I’m developing my API or web app, I don’t want to think about functions, gateways, or infrastructure at all; I just want to install the web framework of my choice and get down to writing routes.

The Proxy Function

Most NodeJs web frameworks support “inject” functionality for their server classes, which is the key to wrapping a normal web app in a Lambda function. Lambda functions launched by API Gateway are invoked with an event parameter which contains the relevant HTTP request information. We just map that event to a fake request object and inject into the server, then translate the response.

// proxy.js
const url = require('url')

const server = require('../server')

module.exports = async (event, context, callback) => {
  const request = {
    method: event.httpMethod,
    url: url.format({
      protocol: event.protocol,
      pathname: event.path,
      query: event.queryStringParameters
    }),
    payload: event.body,
    headers: event.headers
  }

  const response = await server.inject(request)

  return {
    headers: response.headers,
    statusCode: response.statusCode,
    body: response.payload
  }
}

We now just need to map every incoming API Gateway request to this function, which is easy using serverless.yml:

functions:
  proxy:
    handler: src/functions/proxy
    description: Proxies an API Gateway request to the web server.
    events:
      - http: ANY /
      - http: ANY {proxy+}

There are some other important tweaks to serverless.yml, such as installing serverless-offline for local development, serverless-api-cloudfront to setup custom domain names, adding VPC configurations for the function, and adding environment variables.

The full configuration I used is here (on a side project that is in-the-works):

openrecord/turntable

The Database

Aurora Serverless isn’t really “serverless”, it’s just Amazon’s way of abstracting server management away from us developers. To date, this has been a popular trend in cloud servers and storage, but not many (if any?) cloud providers have released a stateful database solution. Their documentation sums up the service well:

With Aurora Serverless, you simply create a database endpoint, optionally specify the desired database capacity range, and connect your applications. You pay on a per-second basis for the database capacity you use when the database is active.

For something like side projects, Aurora Serverless can be a convenient way to deploy a production-ready environment while not worrying about shutting down servers when you go on vacation. As of August 2018, one catch is that you can’t connect directly to the endpoint from outside the VPC. To connect from my computer, I just configured my ~/.ssh/config file to do port-forwarding to an EC2 jump-box in the same VPC:

Host aurora-serverless-tunnel
    IdentityFile ~/.ssh/my-pem.pem
    User ubuntu
    Hostname <EC2 IP Address>
    StrictHostKeyChecking no
    LocalForward 3306 <aurora cluster endpoint>:3306

When connecting from an application deployed in the same VPC, it is functionally the same as a normal RDS/Aurora setup.

Conclusion

The best parts of this setup are that it can be written and configured almost exactly as you would write a normal web app or API and that it costs pennies on the dollar for a side project. I haven’t used Aurora Serverless at scale, so I can’t recommend it for a production application since I’m not sure how well it performs at scale, but the proxy function approach can easily be used in production. In fact, at Storyblocks Engineering we have started using this approach for our production API. I look forward to seeing the proliferation of technologies like this in the coming years, as it makes web development extremely simple to deploy and scale.

The LAN Stack: Going 100% Serverless was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re running a Nodejs process on dedicated servers and not using all of the available CPUs, adding less than 20 lines of code could get you more bang for your buck.

A normal server function might look like this:

// startServer.js

const server = require('../src/app/server');
const logger = require('../src/app/util/logger');

server.start();

Using the builtin NodeJs node-cluster library, you can launch multiple instances of your server process. The best part? The instances will automatically share the original server port, meaning you can implement this with zero infrastructure or deployment changes:

// startClusteredServer.js

const cluster = require('cluster');
const os = require('os');

const server = require('../src/app/server');
const logger = require('../src/app/util/logger');

if (cluster.isMaster) {

  logger.info('Cluster master is running. Forking workers.');

  const NUM_CPUS = os.cpus().length;
  for (let i = 0; i < NUM_CPUS; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    logger.error('Cluster worker died.', {code, signal, pid: worker.process.pid});
  });

} else {
  logger.info('Worker starting server.');
  server.start();
}

At Storyblocks, we have been using this approach for over a year on our NodeJs services which are still on servers, and it has appreciably cut down the hardware required. While we are moving more towards Elastic Container Service to more efficiently use our hardware, this is definitely a quick win which requires no infrastructure changes.

NodeJs Clustering: Don’t Spare the CPU Horses was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

This past month at Storyblocks engineering we held a team-wide interactive AWS Fargate workshop. Since we are moving our core products from docker containers on EC2 over to ECS/Fargate, we figured it would be fun to get everyone up to speed using a competitive team-based workshop. The tutorial is open-source and hosted here: https://github.com/Footage-Firm/learn-fargate.

Why Fargate?

We are migrating our core site and services to Fargate in order to simplify our DevOps. Fargate allows users of Amazon’s Elastic Container Service to deploy a collection of containerized services without managing the servers on which they run. This is a very powerful distinction: before Fargate, one had to manage both the deployment and scaling of containers via ECS in addition to the auto-scaling of the EC2 boxes running the containers. This added simplicity (subtracted complexity?) had just the activation energy we needed to make the switch for our DevOps processes.

ECS Fargate Basics

Terminology

• Task Definition: The blueprint for a task, which is used by ECS to launch containers.
• Task: A group of one or more containers launched and maintained by ECS based on a preset Task Definition. This is the smallest unit.
• Service: A service encapsulates a specific group of like tasks and is most commonly used to scale tasks. A service is not required to run a task.
• Cluster: A physical grouping of underlying servers that holds all of your services, tasks, etc. This is the largest unit.
• ECR: Elastic Container Registry hosts images similar to Docker Hub.

Overview

Any process running on Fargate starts with a Task Definition, which you can set up via the AWS Console or via Cloudformation. It both defines the limits and capabilities of the task, and defines the container parameters to tell the task which docker images to run and how to map ports, etc. The CPU and Memory properties of the Task Definition will ultimately determine how much horsepower your Task gets from Fargate and how much the Task will cost to run.

With a Task Definition in place, you can either launch individual Tasks or encapsulate multiple Tasks within a Service, allowing you to connect them to a load balancer and scaled just like EC2 instances. Tasks can be configured to run multiple containers, such as a side-car logging service for a web container. Multiple logically-related Services and/or Tasks can be launched on a Cluster.

The Tutorial

To teach our team how to use ECS Fargate, we created a simple open-source Laravel app which lets teams compete in a race to watermark 10,000 images located in AWS S3.

https://github.com/Footage-Firm/learn-fargate

The application works by letting teams of two or three people register a team name via a simple form. When the team name is registered, a worker on the default queue immediately creates a folder for their team name in the teams folder of the S3 bucket and then creates 10,000 image-watermark jobs on a queue with the given team name. The non-watermarked source images were located in the source-photos directory of the S3 bucket.

In order to start watermarking images, a team would have to check out the codebase, build a worker image with the correct queue name, and then create Fargate Tasks to start executing jobs on their queue. They could make progress easily by running the worker process locally, but the idea was to get everyone using hyper-parallelization on Fargate with the stroke of a few keys.

Results

After a couple hours and a few pizzas, all of the teams had made good progress on their watermarking queues, with the victorious team having figured out how to launch beefy Task Definitions with dozens of worker containers for each Task (extra-hyper parallelization). Everyone walked away with an appreciation of how easy it is to get production-ready applications deployed and parallelized with minimal infrastructure configuration.

If you want an interactive way to bring Fargate to your team or just try it out yourself, check out the tutorial!

For an introduction to this series, see Phasing into a Microservice with Zero Downtime: Intro

Most microservice projects should begin with the API definition, which means figuring out every way that the consumers of the service interact with it. For a legacy monolithic codebase, this can be very complicated due to either missing or leaking service boundaries in the code. In our case, we had a PHP ORM for our user entities which had escaped and run rampant through every conceivable part of the code (See our VP of Engineering Jonathan Fulton’s post on how a leaky ORM can get you into trouble).

Creates, Reads, and Updates were happening everywhere, which made it difficult to conceive what the service boundary API should look like. Unfortunately, this means the first step on the microservice journey (and really most of the work) was to refactor that legacy code.

Our high-level approach for refactoring the legacy code:

1. Create a UserService class to encapsulate all user CRUD operations.
2. Create a MySQL and a REST driver to be used by the service.
3. Identify all existing ORM Create operations in the codebase.
4. Implement all Create operations inside of the MySQL driver and connect the driver to the UserService API.
5. Write unit tests for the new driver methods and service API.
6. Convert all original Create operations to use the UserService API.
7. (Repeat for Updates and Reads).

That took a long time. A very long time. Before the refactor, a cursory search in the PHPStorm IDE for the ORM user objects would prompt you to cancel the search process after 1000 references were found.

Technical debt of this scale leads to challenges in project planning, project management, and even team morale. How can you know how long it will take to finish? How many engineers should work on it? Is it even worth refactoring or should a new system be created from the ground up? How can engineers work for months without losing interest and productivity? WHY DO UNIT TESTS KEEP FAILING ON MY BRANCH?! (For this project, it took 3 engineers about 6 months and by the end everyone needed a break).

Some general guidelines:

• If your codebase has unit tests which actually call the old source of record, try to mock the responses from the service (your other option is actually spinning up an instance of the service when running unit tests… not ideal). In our case, we had over two thousand tests with the majority relying on user queries (due to the unclean boundaries), and mocking would add too much time. It worked, but mocking the service would have been much more ideal.
• “Services” in your codebase (or whatever you call the code that encapsulates interactions with the data) should only interact with their own drivers. The UserService may not use the PaymentService MySQL driver, for example.
• When you are identifying the common CRUD operations used in the code, look for opportunities to minimize the final number of operations required by the service API. Try to converge and simplify the actions and API, reusing functionality as much as possible. For example, if your admin section of the code searches for users created in the last week while a part of your billing code fetches users older than a month, create a generic service method which takes optional beginDate and endDate parameters.
• TDD is the name of the game with major refactors. If you do not have decent test coverage already in place, you will need to add time to the project to create test coverage. It may be worth investing in your CI/CD pipeline to get fast-running and non-flakey tests to run on every commit.

Phasing into a Microservice with Zero Downtime: Part I (Refactor) was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

*Update: Meetrics is now a real product! Check out meetrics.io for an easy way to monitor your own organization’s meeting health.

A group of us at Storyblocks recently decided to take a look at how much time the company was spending in meetings. We created meeting “rules of etiquette”, started polling employees to generate an NPS score for our meetings, and experimented with No Meeting Wednesday as a way to give people some dedicated heads-down time. As an engineer, I wanted a way to get some hard data from Google Calendar to track the impact of these efforts.

The “Meetrics” Dashboard

The ultimate outcome of my project was a dashboard (created using Periscope) showing compiled metrics for meetings by employees in our company’s Google “G Suite” domain. The metrics currently include average time spent in meetings and one-on-ones over time, a breakdown by department, and highest-impact meetings.

How It Works

The workflow is straightforward: A dockerized Go app (made extra-small using some docker multistage build tips) runs once per night via Jenkins and hits the Google Calendar API to calculate the previous day’s meeting statistics for each employee in our company.

The list of employee emails is manually-curated to sort each employee into a department and to prevent email aliases and miscellaneous email addresses in the organization from polluting the data. Alternatively, one could use the Google Admin API to list users in a given organization.

For each email address, all events are retrieved from the previous day and classified as having 0 guests, 1 guest, or 2+ guests. The number of minutes for each of these categories is then saved in a database entry. Additionally, large recurring meetings have some metadata stored so the dashboard can flag the most “impactful” (or expensive) events in the organization.

With the number of minutes each person spends in each of those three buckets of meetings during a day, we can try to make a reasonable guess at how much time on average they spend in large meetings, in one-on-ones or quick catch-ups, and on personal “crank” time (as many of our employees book their own time to get work done in a normal day).

One of the more challenging parts of the application was getting Google’s complicated authentication system configured correctly.

Authentication

Accessing all of a “G Suite Domain’s” user information from the calendar API requires some finagling of their OAuth 2.0 authentication system. I ended up creating a

Google’s service account documentation outlines how to do this. The basic steps I took were the following:

1. Create a project in the Google APIs dashboard to hold the service account.
2. Create the service account in the IAM & admin section of your project.
3. Delegate domain-wide authority to the service account.
4. Log into the G Suite Admin section (as an admin user) to authorize the service account for API scopes. For listing calendar information, you need the calendar.readonly scope: https://www.googleapis.com/auth/calendar.readonly. For listing users in the domain, you need the admin.directory.user.readonly scope: https://www.googleapis.com/auth/admin.directory.user.readonly.
5. Go to the credentials page in the Google APIs dashboard for your project, click Create credentials > Service account key, select your service account and JSON as the key type, and click Create. This will download your credentials JSON file containing your private key which will be used in Google API SDKs.

Note: This service account will allow your application to query various Google APIs as users in your organization. This means you will want to restrict the scopes as much as possible (stick to readonly):

Your application now has the authority to make API calls as users in your domain (to “impersonate” users). When you prepare to make authorized API calls, you specify the user to impersonate.

Conclusion

We have not revealed the dashboard to the company at-large yet, so I’m not sure if this will have a positive, negative, or null impact. It has proved an interesting conversation-starter: we’ve had fun speculating about peaks/troughs in the data during particular weeks and wondering if some of the larger meetings really need to be that large.

In the future I’d like to add meeting-related NPS data from a Slack poll to the dashboard to try and track employee satisfaction with meetings over time. The event data also needs to be filtered and analyzed more-accurately, as it currently suffers from duplicate events, miscounted all-day (vacation) events, misinterpreted invitation information (when a list or group is invited to an event in addition to individual members), etc. Assumptions also need to be validated, such as the one that meetings with 0 people are actually crank time as opposed to personal chores.

And of course I’ll be adding unit tests… at some point… probably.

Analyzing Meeting Metrics Using the Google Calendar API was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When a pilot hands the controls over to a copilot (or vice-versa), they both need to execute what is called the Positive Exchange of Controls: The person with their hands on the yoke tells new pilot “you have the flight controls,” to which the new pilot replies “I have the controls,” which is followed by a final “you have the flight controls” from the pilot. This method ensures that the important thing — control of the airplane — is maintained without any confusion or lapses. Here at Storyblocks, we used a similar concept to hand-off control of our user accounts from their original MySQL databases to a centralized microservice.

Until recently, all user account data for our three core websites (audioblocks.com, videoblocks.com, and storyblocks.com) existed in separate databases. This means that if you signed up with your@email.com on VideoBlocks and then wanted to get access to AudioBlocks, you would have to sign up again on the new site. If you had accounts on both sites and updated your password on one of them, you would now have two different passwords for two of our sites. Not a great user experience. The solution? Put our user accounts in a single place.

In late 2016, we decided to carve user accounts into a single microservice. This would help us do the following: Pay down technical debt caused by old code and hacked-together features, unlock in-product capabilities like more targeted upgrade offers, simplify the overall user experience, scale our infrastructure, and shift our company mindset regarding users to be more holistic.

There are a few approaches you can take to a microservice migration like this, the simplest being to just build the service, temporarily shut-down the production sites, import the data from the old databases to the microservice, then turn the sites back on (what we called the “Big Bang” approach). In our case, because A) this refactor touched extremely high-risk business logic and B) this was our first stateful microservice, we opted to ease into it with a phased approach that also enabled zero downtime. The phases were as follows:

1. Phase I / Refactor: Rewrite the code in the monolith so that all reads and writes to the data “domain” of the service have clean boundaries. This encapsulates all read/write logic in a single place and makes it easier to manage the hand-off between original database and microservice.
2. Phase II / Write-Only: Write user data to the original database and also send to the new service, but the original database remains the source of record for reading user data. (“You have control of the airplane”)
3. Phase III / Backfill: Import the data from the old service to the new microservice. While users created since Phase II will have their information in the new service, pre-existing users will not. We’ll need their data before switching the source of record to the new service, and thus need to retroactively backfill older users.
4. Phase IV / Attempt-Read: In the monolith, start attempting to read from the service but allow minor data inconsistencies or even a total service failure. Log any data inconsistencies to help troubleshoot any bugs that pop up. Writes are still going to the monolith and it is STILL the source of record at this point. (“I have control of the airplane”)
5. Phase V / Read & Write: Turn off writes and reads to the old database, completing the transition to the new service. The microservice is finally the source of record. (“You have control of the airplane”)

Conclusion

By activating writes and backfilling old records in the background, we achieve zero downtime for our production applications. Also, by creating an attempt-read phase, we get a live preview for how the production sites are reading from the service. Response times, failed writes or reads, data integrity issues, etc., can all be troubleshot on-the-fly and in a low-risk environment. This Positive Exchange of Controls approach to a microservice is useful when refactoring a risky core domain of the monolith, and was invaluable to our migration of user accounts at Storyblocks.

This upcoming series of posts outlines those phases and some of our learnings from this process.

Phasing into a Microservice with Zero Downtime: Intro was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In late 2015, the VideoBlocks engineers decided to start a semi-annual VideoBlocks hackathon (which has since turned into our first company-wide “Innovation Day” this month). We sent out a spreadsheet to collect project ideas from the company, and among the great ideas (#-of-downloads LED ticker, a 24/7 CS 1.5 office server, a Slack doorbell for the office, a peapod API for Slack), there was this gem added by our now-CEO TJ Leonard:

In the style of the Cliffhanger Game on the Price is Right, we should have a Swiss Hiker marching towards our revenue goals each month, showing daily (or even real time) progress. When we reach our goal, the Swiss Hiker should break out in a Yodeling Song that the entire office can hear.

Since it was our first hackathon, we took it as the joke that it was and moved on to more approachable projects. When the time came for hackathon #2, though, our very own @MathiasHansen dared to ask “why not?”

Design & Execution

Home Depot, Kinkos, Arduino, 3D Printing, and APIs.

Step 1: How will he climb?

This was the biggest unknown for us, so we took to the internet. We found an open source project that looked pretty similar to our requirements, and had some schematics and how-to’s already in place. We would just need to 3D print some parts and buy a servo, belt, and rails. Once we were confident in our motor/rail design and knew its limitations, we could get the dimensions for the final product.

The motor is a Sparkfun Stepper Motor, which lets you give the driver a number of “steps” you want for it to rotate. We picked a rotation speed that we liked, and then had the motor run a full circuit in order to get a measurement for our total number of steps.

Step 2: How will he yodel?

I happened to have an old assembled Adafruit Wave Shield from a previous project, so it was a simple matter of finding an mp3 of the Cliffhangers song and loading it onto an SD card. The shield has a built-in 3.5mm headphones jack, so any speakers would do.

We knew the wave shield would not be driving the motor or running the yodeler logic (its only job is to make sound), so we created a very simple program that would loop the desired MP3 when an input pin went to 5 volts. The main microcontroller would activate sound this way.

Step 3: Making the Mountain

We knew how big it could be from Step 1, so the next step was heading to Home Depot with some spitballed dimensions and little idea of what we actually wanted to buy. We ended up using 2x4 boards for the frame (and the mount for the motor/rail unit), and sheets of MDF for the mountain and background. After a little woodworking and hammering, we were good to go!

Step 4: Make it Look Good

A vector image of the mountain and background was clearly the way to go. Fortunately, we had access to VideoBlocks world-class graphic designer (Nick), who took some time off of beautifying our websites to draw up the vector. Using the FedEx online portal, we ordered a giant print on white vinyl. A couple layers of adhesive, and the vinyl was easy to attach.

Step 5: Give it Data

The last step was giving the yodeler some company data so it knew when to yodel. We aimed for simplicity, so a very simple endpoint running on our hubot server (something like /yodeler-info) that responded with a minimal JSON object would work just fine:

{min: 0, max: 100, value: 63, percent: 0.63}

The server computes that information using our analytics database living in Amazon Redshift, and is (currently) based on a monthly cash goal.

For the client, we went super simple and used a SparkFun ESP8266 Thing board (Mathias had previous experience with it), which has a wifi chip on board and costs less than $15. The client simply establishes a wifi connection and then queries our endpoint in a loop. The code is not public, but is very similar to the public code I wrote for our “build status light bulbs”, here.

The Thing board computes the correct number of steps using the “percent” field from the endpoint, and then drives the yodeler to that point. When it reaches 100%, the unit runs a full circuit while activating the yodeling wave shield by setting an output pin to 5 volts. Then, it goes back to the bottom.

Step 6: ???

Step 7: Profit

And that’s that. We hope to continue the VideoBlocks tradition of quirky hackathon (now Innovation Day) products, and continue to grow this blog!

The Goal Yodeler was originally published in The Storyblocks Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.