On Saturday, May 21st, Twitch hosted the Global Ember Meetup Conference (GEMConf) at our headquarters in San Francisco. Global Ember Meetup is an organization that hosts Ember meetups online for developers who do not have a regional meetup group near them. They have held many remote meetups, as well as a few that brought together local attendees with remote viewers and presenters. This was Global Ember Meetup’s first full-day event and their first time streaming via Twitch, and we were very excited to host them! We love being part of the Ember community, and we’ve been lucky to have the opportunity to give back by providing a venue for events like this.

There were over 100 people in attendance at GEMConf, with viewers tuning in to the stream on Twitch from over a dozen countries! It was great to see such a strong turnout from remote attendees, since the Twitch platform is all about bringing live content creation to the global audience. I was especially excited to see the mix of on-site and remote presenters for the talks; the event switched back and forth between the two throughout the day.

The whole day was filled with great talks that really resonated with what we value in Twitch Engineering. All of the talks were excellent, and two stood out to me as particularly relevant to us at Twitch:

Juan Pablo Buritica’s presentation on “Building Excellent Engineering Culture” highlighted something that we’re really passionate about. We work very hard to engender engineering practices that create a web product of the highest quality and availability, through best practices, infrastructure, tooling, and by surrounding ourselves with engineers who share our vision of that culture.

“Building Real-Time Applications w/ EmberJS & WebSockets” by Ben Limmer was also a great example of what we strive for. As a live content platform, it’s really important for us to leverage the best technology for creating a real-time experience for our users. In particular, our team has worked very hard to build a seamless chat experience enabled by our real-time distributed Go platform. You can read more about our platform architecture here.

I was lucky enough to have the opportunity to present my own talk at GEMConf, and I spoke about an addon I built called Ember User Activity (EUA). EUA provides an event service bus for tracking global user actions in your application, such as mousedown, keydown, scroll, and more. One of its key use-cases at Twitch is for our new Friends feature, where we use EUA to determine whether a user is currently active on the page. The user-idle service that implements this functionality is also provided by the addon and can be configured to your app’s needs! I’ve done a lot of work lately to improve the performance of EUA, utilizing requestAnimationFrame and adding event throttling. Check out the demo and play around with it, then let me know what you think!

I was also incredibly interested in the CSS layout performance talk by Chris Thoburn and Nathan Hammond’s lightning talk on accessibility. These are all topics that we strive to improve upon in our work at Twitch, and it was great to get community input and thought leadership from the speakers at GEMConf!

You can check out all the talks and their slides through the GEMConf Recap.

Overall, it was great to bring the Global Ember Meetup here to San Francisco and host the very first GEMConf! I can’t wait to see what more we can do with streaming meetups and conference talks on Twitch to enable remote participation in these great events. It’s really exciting to be able to build a product that can not only benefit the Ember community through our team’s upstream open-source contributions, but also to provide a platform that can enable the global spread of information and communication for our events as well!

If you’re in the Bay Area, be sure to join us for the next SF Ember meetup at our downtown San Francisco HQ on Tuesday, July 26th!

To find out about all of our upcoming Tech events and more, follow us on Twitter at @TwitchTech.

Twitch Hosts Global Ember Meetup Conference was originally published in Twitch Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

So you’re interviewing at Twitch? That’s great! Hopefully we can give you a few pointers that will make interviewing at Twitch a more fulfilling and successful experience.

What we’re looking for:

When we interview engineers, we’re looking for a few things that are important for everyone that works at Twitch:

• Enthusiasm for engineering and craftsmanship — you should enjoy the challenge of solving engineering problems, striving to improve your skills, and care deeply about the quality and polish of the products you build.
• Practical coding skills and problem-solving ability — we want you to be able to work quickly and safely, as well as solve difficult problems.
• Communication skills — yes, you’re going to be spending a lot of time writing code, but all of the important decisions that you make will usually require you to communicate clearly and effectively with a variety of people, both technical and non-technical.

Overall Interview Tips

Here are a few tips as to how you can make the interview experience better for yourself (and us):

Ask us questions! Interviews are as much about you learning about us as us learning about you. If there are things that you would like to know about Twitch that will help you decide if we’re a good fit, ask them. Write down a list ahead of time if you’re not good at being spontaneous.

Don’t worry if you don’t know everything. We’re looking for folks who are able to succeed when given difficult challenges. Part of that is often seeing how you deal with things that may be out of your comfort zone or area of expertise.

Explain your thinking. If neither you nor the interviewer are speaking, that probably means that you could be conveying more information about how you’re approaching the problem that would allow us to evaluate you.

Be yourself! We aren’t robots, and we don’t want to hire robots. We are looking for the wealth of your experiences, and want to know what you would be like to work with on a day to day basis — let your personality and passion show!

Phone Interview Tips

Phone interviews in particular have lots of things that you can do in order to make your experience better. To wit: A great deal of what can go wrong in a phone screen is technical in nature.

• Find a quiet environment that doesn’t have distracting noises for both yourself and your interviewer.
• If you’re using a cell phone, make sure that you are in a location with a reliable connection. Instead of a cell phone, consider using a landline or VOIP solutions like Skype or Google Hangouts with a headset to get especially good voice quality.
• We will usually provide you with an URL for a shared editor environment like CoderPad. Take a few minutes to familiarize yourself with the environment, and make sure that you can connect to the provided URL before the interview starts.

On-site Interview Tips

Dress comfortably. Twitch is a casual environment, but if dressing sharp helps boost your confidence, do what works for you!

Technical Interview Tips

As mentioned above, we’re looking to challenge you and explore the boundaries of what you do and don’t know. So it’s more important that you feel comfortable thinking on your feet and describing your approach than knowing every little detail. If you haven’t interviewed for a while, get someone to throw a practice question at you so you can practice verbalizing your thoughts.

Interview questions that we give you may often seem like they’re vague or unclear. Hint: This is often by design! Real world engineering is messy — we’re looking to find people who are able to bring clarity and order to it. Ask questions to determine the scope of the problem so you can clean and tidy it up.

Iterate while answering questions! If you find an optimal solution first, that’s great! But it’s often better to solve the problem quickly first with a suboptimal solution and then follow up with a better solution later.

Coding

As an engineering candidate, it is expected that you will be able to demonstrate proficiency in day-to-day coding skills. We expect that you will be able to write correct, clean, and maintainable code on the fly. We suggest that you brush up on a language that lends itself well to rapid prototyping, like Python, Ruby, or JavaScript, so you can spend more time demonstrating your coding skills and less time writing uninteresting boilerplate. You should have an understanding of fundamental algorithms and data structures (Breadth-first search, depth-first search, hash tables, lists, binary trees, etc.), as well as their runtime characteristics and common applications. For frontend positions, we emphasize expertise programming for the Web (including an understanding of modern JS tooling and cross-browser compatibility issues) and sensibilities for design and UX over a strong algorithmic or CS background.

Programming Language and Tools Expertise

One of the things that you develop as a good engineer is expertise in the programming languages and tools that you use to accomplish your job. While we do not require that you know any particular programming language, you should be able to demonstrate that you have gained proficiency with at least one commonly used language such as JavaScript, Ruby, Go, Python, Java, C++, or C#. Expect to be asked about unique language features, commonly used libraries, and warts of the languages that you are proficient in.

Design and Architecture

Twitch engineering runs the gamut from crafting delightful user experiences on our platforms (Web, Xbox, PS4, iOS, etc.) to building large-scale distributed systems, so the scope of what you may be asked could be large depending on the position you’re applying for. We may ask you questions outside your past experience or comfort zone to see how you think on your feet and adapt to new problems. When designing solutions, we value simplicity, ease of maintenance, and durability more than algorithmic and design cleverness.

Distributed systems that communicate over HTTP are widely used at Twitch, so if you’re interviewing for a backend position, it’s helpful to have a basic understanding of common web architecture patterns and the various components that are used in building distributed systems, including databases and other data stores, load balancers, caches, and message queues.

Final Thoughts

We look forward to talking to every single one of our candidates — we’re always excited to be able to add a new member to the Twitch family! While interviews can be stressful, we hope that we’re able to do our best to make your experience as fun and informative as possible.

Good luck!

A Guide to Interviewing at Twitch was originally published in Twitch Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Among other things, we have:

• One of the largest live video distribution systems in the world
• A real-time chat system
• Web services that provide access to functionality and data
• Data storage systems
• Client applications on the web, and on a multitude of platforms — mobile and console in particular
• Data science infrastructure
• Internal tooling and systems — configuration management, deployment systems, hardware and software provisioning, testing and QA
• Network infrastructure that keeps the bits between all of these systems and the end users flowing

Each of these different components of Twitch has different challenges that need to be tackled — among them low-level optimization of video encoding, large scale system scaling, implementing products across multiple platforms and form factors, and improving our engineering efficiency via improved tooling.

Read More.

Engineering at Scale

Before we dive into what technology we currently use, it’s important to note that many of the decisions that we make surrounding technology are driven by the size of Twitch and how rapidly it is growing:

• We have over 30,000 people streaming video on Twitch simultaneously.
• We have peaked at over 2 million concurrent video streams on our site.
• Our chat service delivers over 10 billion messages a day.
• Our web APIs handle over 50 thousand requests a second on average.
• Over the last 3 years, the number of engineers at Twitch has grown by over 800%.

This means that we have to scale at both an architectural and organizational level. As each individual systems grows, we expose different kinds of failure modes that aren’t problems in smaller systems, but become problematic as systems grow larger. This includes problems like hardware and network failures that are rare in small systems but commonplace in larger ones, and doing things like capacity planning to make sure that you can build and architect new infrastructure and systems before they are needed.

Beyond individual systems growing larger, we are also adding more systems to Twitch to make our product better. Each of these new systems adds more complexity to our overall system, which makes engineering more complex for everyone. And adding the extra engineers to build those systems also adds organizational complexity — communication and coordination is an O(n2) scaling problem if left alone. There have been many great articles written (and ideas) about how to try to solve these problems, but there isn’t enough room here to do them justice.

Our Technology Stack (A 50 thousand foot view)

There is way too much to talk about in depth here, but I’ll attempt a high-level overview of the the key pieces of technology that keep Twitch up and running.

Video System

The video system is responsible for getting video from the broadcaster to our viewers. This includes the following core components:

• Video ingest — We take RTMP video in, and then transport it to the transcode system.
• Transcode system — We take the incoming RTMP stream from the broadcaster, and transcode it into multiple HLS streams. This is implemented via a combination of C/C++, and Go.
• Distribution and Edge — Distribute the HLS streams to our geographically disparate POPs, such that you have the highest quality video streaming experience. Again, mostly written in Go.
• VOD (Video on Demand) — we take all of our incoming video systems and archive them for our VOD system.

Chat

Chat is a highly scalable real-time distributed system written in Go. It delivers hundreds of billions of messages per day to users who are watching video via our own protocols, as well as supporting IRC as a communication protocol, making it easy for developers to create IRC bots to add their own custom chat features. Core chat components include:

• Edge — Receives and distribute messages between clients and backend services. Edge speaks the IRC protocol over both raw TCP and WebSockets.
• Pubsub — Distributes messages internally across Edge nodes. Pubsub and Edge combine to form a hierarchical message distribution system which executes massive fanout.
• Clue — Perform analysis on user actions and send them through our business logic pipeline. Iis the user banned in the channel they’re speaking in? Are they a subscriber? Are they banned or exhibiting abusive behavior? To inform these decisions, Clue aggregates data across many sources, including databases, internal APIs and caches.
• Room — Responsible for the viewer list. Room aggregates, stores, and queries membership data across all our edges to retrieve viewer lists for each channel’s chat room.

Web APIs and Data

Besides our real-time services (video and chat) we also have a substantial number of services including, but not limited to:

• Web APIs that allow users to manage and customize their profiles and subscriptions
• Search and Discovery services that help you to find the streams that you want.
• Revenue systems — these are systems that allow us to manage advertising and subscriptions, and make sure that our partners get their income.
• Administrative tools that help our support staff resolve your problems.

This is all built on a mix of Rails, Go, and various different open-source applications that are used for routing, caching, and data storage. We have also been doing a lot of vertical and horizontal partitioning of our data and data APIs in order to make our systems scale better — both at a technical and organizational level.

Web and Client Applications

All of these services are lovely, but useless without easy to use client applications that create a good user experience. We have a multitude of different client applications that make it easy to access Twitch.

• Our desktop web application, which started off as a vanilla Rails application, but has evolved into an Ember app.
• Native apps for the major mobile and tablet platforms (iOS and Android).
• Various console applications, including XBox One, XBox 360 and Playstation 4.

Data Science Infrastructure

Science Engineering builds data collection systems and analysis tools to study how our users interact with our products. These systems include:

• The data pipeline, which collects, cleans, and loads over a billion events a day into our data warehouse
• Streaming aggregators to summarize metrics in near real-time for broadcasters.

Tools and Operational Infrastructure

The above services directly support the application and product. However, the operational infrastructure that we need to manage these services is also extremely important. We have and maintain tools for:

• Testing all of these services by manual QA and automated testing — we currently use Jenkins for automated build and test.
• Tools for rapid deployment and rollback of changes — we have a couple of home-grown tools here.
• Monitoring and alerting systems that let us know when things are going wrong, and help us to diagnose and fix them. Currently a mix of tried and tested tools like ganglia, graphite, and Nagios.
• Distributed trace. This is something new that we’ve been working on, to help us understand and debug our systems as we move further and further into the realm of distributed systems and microservices.

At the physical level, we currently run an actually fairly large number of “bare metal” POP (points of presence) all over the world — this allows us to deliver higher quality video due to the unusual requirements of video delivery (lots of bandwidth!).

We also have been moving an increasing amount of our services to Amazon Web Services — this helps to reduce the amount of operational overhead, as well as to take advantage of the convenience and scalability of many of their services.

In Closing

We hope this has been an interesting overview of what makes Twitch tick under the hood. If you are interesting in learning more about these (or other) systems, please look at the Twitch Engineering site!

Twitch Engineering: An Introduction and Overview was originally published in Twitch Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Establishing the Path Forward

The initial schema was not designed with a billion relationships in mind. It lacked correct indexing and was a part of a large, centralized PostgreSQL cluster — which was succumbing under load during peak times. One of the primary use cases for our following system is to notify users when their favorite broadcasters have come online. This means iterating over all of a channel’s followers as quickly as possible. To illustrate the problem, this is a graph of time required to retrieve all followers sequentially, fetching 100 results at a time. The old model is unsustainable, while the new model is a perfect fit.

At peak, the following-related queries represented 40% of our PostgreSQL cluster’s available CPU time. Having pushed that cluster to its limits already, we sought to move to a more optimal data model. Inspired by Facebook’s TAO, we realized we could simply model our follower data as associations in the form of [Entity A, Association Type, Entity B], as such:

With the target data model established, we created a high level migration plan:

1. Take a database dump of the PostgreSQL table
2. Simultaneously start recording all new write events (follows and unfollows) on to a time ordered event queue.
3. Import the database dump into a new database and convert the data into the new schema format.
4. Replay all queued up writes onto the new data store until it’s caught up
5. Divert traffic to the new database

We had a separate program consume the write events and write them to the new database using the new schema. Although on paper that makes us able to divert the traffic and switch over, we wanted to be sure that our newly formatted data matched the semantics of our old data. Clients of our API should not be aware that this change was happening and all previously stored data should be retrievable. Since each of these migration steps has a possibility to introduce non-obvious faults, we devised a strategy to ensure consistent behavior between the two data representations.

Ensuring Success

To ensure data consistency, we took inspiration from Github’s Scientist library for refactoring critical paths. We wrote our service to support more than a single data store: a primary one and an arbitrary number of secondaries. We can mirror, in parallel, each incoming request to all the data stores and return the result from the primary one to the client. We can then, in a different thread, compare the result of a secondary data store with the primary to see whether it’s correct. With this information, we can say with confidence when the migration is ready to proceed. We can also measure request times for each secondary data store to ensure response times are within expectations. The idea is illustrated here, with the steps being numbered in order:

Below is a code snippet that illustrates the flow mentioned. The client supplies ‘from’ and ‘to’ IDs, as well as the type of the association. Our service responds with a single result if the record existed, otherwise nothing. The interfaces look like this (note that I’ve excluded Contexts everywhere, trimmed package names and simplified some things for brevity’s sake):

type Reader interface {
    GetAssoc(Association) (*AssocResponse, error)
}

type Backender interface {
    GetAssoc(Association, *Params) (AssocResponse, error)
}

// Backend provides storage and stats implementations
type Backend struct {
    primaryReader Reader
    secondaryReaders []Reader
}

This makes it possible to create multiple readers. Every incoming request comes into the backend, which then routes them to the primary and the secondaries.

func (b *Backend) GetAssoc(assoc Association) ([]AssocResponse, error) {
    res, err := b.PrimaryReader.GetAssoc(assoc)
    if err == nil {
        go func() { b.secondaryGet(assoc, res) }()
    }
    return res, err
}

Secondary reads are handled after the primary has already returned. We spin up a goroutine for each of the registered secondary readers, and time how long each one takes. That allows us to compare the result it got back with the primary result and report stats around to our statsd cluster. Finally, these all execute within a timeout function for safety.

func (b *Backend) secondaryGet(assoc Association, primaryResult AssocReponse) {
    for _, r := range b.secondaryReaders {
        go func(reader Reader) {
            res, err := reader.GetAssoc(assoc)
            if err != nil || res != primaryResult {
                // Log failed comparison
            } else {
                // Log successful comparison
            }
        }(r)
    }
}

Logging the successful and failed comparisons in our statsd clusters allow us to measure mismatch rates:

In the case of a mismatch we logged the result alongside the expectation and compared the two. The dips in the graph above represent our progress from steadily investigating inconsistencies and fixing them. A few example issues that we found were:

• Result sets not being ordered correctly
• Records being included when they should not have been
• Time stamps being returned in the wrong timezone

Conclusion

By the end of it, we knew that the results we were serving to our users were the ones they were expecting and felt confident in switching over to the new database cluster. As an extra safeguard for rolling back, we added the original database as a secondary data store when we switched over — meaning that it wouldn’t get out of date, and it’d be easy to switch back if any problems occurred. It’d just be a matter of changing which database we call primary and which we call secondary. We first did this with reads and subsequently with writes by implementing a secondary writer.

Migrating 500 million rows without downtime was a lot of fun, especially given our confidence about the outcome. We knew our new TAO-like schema would be a significant upgrade over the old schema, and since we moved this information into its own separate data store we took about 30–40% of load away from our main database cluster. And no one noticed!

We're hiring!

This was a fun problem for us to work on. We have a ton of other fun problems left. If you are interested in working on systems and scale, please look at the Twitch Engineering site!

How we migrated over half a billion records without downtime was originally published in Twitch Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.