Summary

From November 20th, 2020 to January 4th, 2021, some Medium authors saw intermittent spikes in post view stats.

Impact

Approximately 700 authors were experiencing large single-day spikes of story views (in most cases only in one post) of up to 1000 additional views in their stats details page. This did not impact the Member Reading time. As such, no Partner Program earnings revenue was lost or impacted. This also did not affect average read time.

Timeline

• January 3, 5:04 pm: We received a tweet at Medium Support describing the issue
• January 4, 7:09 am: One of our employees sees the tweet then follows up by creating a ticket and exploring the issue
• January 4, 8:56 am: We discover that a bad actor started emitting thousands of post viewed events to our server
• January 6, 4:55 pm: More reports of the issue come in and this issue is marked as urgent
• January 7, 2:42 pm: A new service was deployed to prevent this issue from happening again

Root cause

A malicious external script was sending thousands of fake view events inflating views on stats pages.

Action taken

The IP of the script was blocked and we added additional security to detect and block these spam events. We will also be doing a data backfill to eliminate the fraudulent events.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Monday 4 January 2021 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On January 26th, 2018, Medium had a rolling outage from 4:51PM until 5:32PM.

Impact

A majority of requests to medium.com failed (including visits from a web browser, interactions from our native apps, and API requests).

Timeline

• 4:51PM: Server errors began
• 5:31PM: A bad cache instance was identified and restarted
• 5:32PM: The last sustained server errors ended

Root cause

At Medium, we run experiments that affect the behavior of the site, the apps, emails, and many other things — all with the goal of refining the Medium experience to make it better for our users. When a server processes a request or handles an event, it checks the status of all running experiments to ensure the correct response is delivered.

This outage was caused by the failure of a cache that stored metadata for currently running experiments. The resulting surge of requests that failed over to our database were beyond provisioned capacity and requests began to fail.

Action taken

We have taken steps to ensure that the cache is more resilient against the failures of instances serving hot cache keys.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Wednesday 26 January 2018 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On September 19, 2017, version 3.3 of the Medium iOS app was released to the App Store. Apple released the official version of iOS 11 earlier that same day. Through a combination of factors, the app started crashing on launch for users running iOS 11.

Timeline

• September 19, 3:22pm: Version 3.3 was released to the App Store.
• 4:16pm: iOS engineers received an alert from Crashlytics about a crash affecting a large percentage of users on iOS 11 only.
• 6:03pm: A fix for the crash was submitted in version 3.4 of the app with an expedited review request.
• September 20, 11:00pm: Version 3.4 of the app was approved and released.

Root cause

The crash was caused by a runtime type mismatch between what type an object was declared to be and what type it was at runtime. This type mismatch was introduced a while ago, but didn’t start causing crashes until iOS 11.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Tuesday 19 September 2017 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On July 11th, 2017, Medium.com was not serving requests from 4:10 pm to 4:14 pm due to a deployment issue caused by a miscommunication.

Impact

No one could access https://medium.com for 4mn.

Root cause

At Medium, all services are deployed following a continuous integration and continuous deployment model. What this means is that deployments of our services happen automatically at a high frequency (this service, in particular, gets deployed dozens of times a day). To make this process safe, our deployment pipeline runs a health check on a canary to make sure that there is no obvious regression.

If the health check step passes, all production traffic is redirected to a fleet of servers running the new build.

If anything bad is discovered, the deployment pipeline stops and notifies the on-call team. The team then investigates the possible regression and decides between retrying the health check step or abandoning a build.

In this case, the health check step did not pass, and two different protocols were implemented simultaneously, causing an outage.

Action Taken

We took the necessary actions to prevent such issues from happening again by adding a lock to prevent an abandoned build from being retested and adding extra steps prior to the promotion of a build to make sure enough hosts are running the service before switching traffic to the new fleet.

Timeline

• 4:10 pm PDT
  Health check of our deployment pipeline mistakenly allows a terminated build to be promoted taking down https://medium.com
• 4:14 pm PDT
  Oncall engineer rolls back the bad fleet and returns Medium to service

Tuesday July 11th, 2017 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thursday 27 April, 2017 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Wednesday morning at 6:45am PST, custom domains on Medium were subject to a 350x increase in network traffic due to a Distributed Denial of Service (DDoS) attack. Direct traffic to custom domains was affected, while direct traffic to medium.com remained unaffected.

Our on-call team was alerted within minutes and began working to resolve the issue. By 8:20am load times began improving and at 12:45pm custom domain performance was restored to 95%. Traffic reverted to normal at 5:30am Thursday morning and custom domain performance was back at 100%.

Timeline

• 6:45am PST
  On-call team registers 350x increase in network traffic. Within minutes the team increases the size and number of reverse proxies to manage the flood of traffic.
• 8:20am PST
  Load times begin improving to many custom domains, but access remains intermittent.
• 12:45pm PST
  Custom domains restored to 95% normal performance.
• 5:30am PST (Thursday)
  DDoS attack ends. Custom domains restored to 100% normal performance.

Explanation of root cause

The source addresses of the attack appear to have been spoofed, making it difficult to identify its origin.

Explanation of resolution

We mitigated the attack by increasing the number and size of reverse proxies to enable us to handle the flood of traffic to our servers.

Preventative measures

• We’re looking into a host of measures to prevent this type of attack from reaching custom domains’ reverse proxies in the future.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Wednesday 18 January 2017 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Monday lunchtime a change was pushed to production that caused post pages to become unavailable. Due to an increase in load, a small percentage of traffic to other parts of the site were also intermittently unavailable, though at a much smaller scale. The CMS was unaffected.

The error triggered alerts and our on-call team reverted the bad build. The incident lasted 18 minutes, from 1:08pm to 1:26pm PST.

Timeline

• 1:03pm PST
  Health checks on the new build pass.
• 1:06pm PST
  The deployment completes.
• 1:08pm PST
  Alerts indicate abnormally high pressure on an internal service.
• 1:16pm PST
  Alerts indicate server responses are too slow.
• 1:19pm PST
  On-call triggers a rollback to the previous build deploy.
• 1:26pm PST
  The rollback succeeds, all pages are again fully accessible.

Explanation of root cause

On Monday a new feature, making a change to the rendered output of a post page, was rolled out to 100% of users, due in part to a desire to have a consistent experience for all users.

The data used to render Medium’s post pages comes from a number of backend services, supported by a number of different kinds of caches. During development of this new feature, a misconception about how cache abstractions work meant that the new feature queried Hopper (a backend data service) for every rendering of every post page.

As a result, once the deploy completed, Hopper saw a substantial spike in traffic, and was unable to handle the additional load. The machines backing Hopper rapidly approached 100% CPU, which caused requests to back up. Post pages in turn began to timeout, and the reverse proxies shed traffic.

The additional load on Hopper caused cascading slowdowns to other parts of Medium which relied upon it, causing intermittent — though less noticeable — failures throughout the rest of the site. Successful responses returned by Medium frontends were at 25% of what they were prior to the incident.

This class of issue is usually prevented through use of variants to roll new features out to progressively larger audiences. We start with Medium staff, then typically go to 10%, 25%, then 100% of Medium users. This is done both to assess the effectiveness of the feature, and to understand the impact of the feature on infrastructure. The Monday afternoon change bypassed this progressive rollout.

Explanation of resolution

We resolved the short term issue by turning off the new feature. A permanent caching fix reduced the load on Hopper. The feature was then rolled out to 5% of users, then in small increments, allowing the team to understand the required capacity and add machines to the Hopper fleet as necessary.

Preventative measures

• Restatement of policy to avoid going from 0–100% of users in one step. We know this to be best practice, and we made a mistake by not doing so. In particular, launch plans should be designed to account for the fact that we can’t jump to 100% for anything that touches the post page.
• Implement graceful degradation of non-critical features.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Monday 31 October 2016 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Wednesday morning a change was pushed to production that caused some publication home pages to throw errors, including landing pages for hosted domains. Direct traffic to medium.com, CMS, and post pages was unaffected.

The error triggered alerts and our on-call team reverted the bad build. The incident lasted 21 minutes, from 10:04am to 10:25am PST.

Timeline

• 9:42am PST
  Health checks pass, despite errors on publication home pages.
• 10:04am PST
  The deployment completes.
• 10:09am PST
  Alerts indicate several major publication pages are offline.
• 10:10am PST
  On-call triggers a roll-back to the previous build deploy.
• 10:25am PST
  The rollback succeeds, publication home pages accessible.

Explanation of root cause

Medium’s publication home pages are divided into “sections” which display stories from a variety of sources (e.g. the latest stories in the publication, featured stories, or stories from a tag). On Medium, stories can be tagged with topics that help readers discover related stories. Publications can use these tags to divide up their homepage on a per-topic basis.

Medium assumes that every tag uses title case by default. For certain popular tags that should not use title case, we manually override the name of the tag with the correct casing (for example, we use “MLB” instead of “Mlb”). Unfortunately, we weren’t using the overridden cases when rendering sections on publication home pages. In the process of fixing this issue, a bug was introduced where it was assumed that tag object would always exist for the corresponding slug:

section.postListMetadata.tagName = tagsBySlug[tagSlug].name

When the corresponding tag object does not exist, tagsBySlug[tagSlug] evaluates to undefined. We then attempt to access the name property of an undefined object, which throws an error.

Errors like this are ordinarily caught by unit tests, or during the “health check” phase of Medium’s deployment process. During this final phase, we test a single machine with production traffic before deploying the build to the rest of the fleet. We track the errors on this machine and halt the build if we encounter any unexpected errors.

The Wednesday morning build succeeded due to a false negative with the health checks. This caused our deployment process to interpret the build as safe, and escalate it to production.

Explanation of resolution

We resolved the initial bug by checking whether the tag object exists before attempting to access its properties. We also verified that the case where the tag object didn’t exist was safe, and corrected that case in a later fix.

Preventative measures

• Upgraded health check process with more probes for publication home pages and hosted domains.
• Hardened health check process to prevent false negatives during health checks for this class of bug.
• Will add additional unit tests for affected areas.
• Will add additional end-to-end tests for publication features.
• Identified steps to speed up rollback following event such as this.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Wednesday 22 June 2016 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Thursday morning a change was made to how EC2 instances are provisioned. This change, combined with lack of termination protection on key long-lived instances and loose IAM policies, resulted in instances being removed in a key availability zone. This impacted core infrastructure required for Medium’s operation.

The outage lasted 31 minutes, from 11:25am to 11:56am PST. Error rates remained elevated until 12:35 PST.

Timeline

• 11:20am PST
  A change to how we provision EC2 instances was made and executed.
• 11:26am PST
  Increase in average response times coupled with a drop in traffic to application servers.
• 11:27am PST
  Elevated server errors. On-call engineers are paged and respond immediately.
• 11:32am PST
  On-call engineers identify that several reverse proxies are down, begin re-provisioning.
• 11:44am PST
  Instances for several other services are offline. Process begins to rebuild fleet.
• 11:55am PST
  Core services operational. medium.com and hosted sites are accessible.
• 12:09pm PST
  Source of terminations identified as within our network. Production access tightened for remainder of incident.
• 12:35pm PST
  All services operational and stable. Offline tasks are backed up, but not affecting user experience.
• 12:56pm PST
  Root cause identified. Restrictions on production access lifted.

Explanation of root cause

Medium uses Ansible for configuration management and orchestration. The EC2 module has a feature which can enforce a specific number of instances based on the uniqueness of EC2 tags. This can be done with either tag/value pairs or based on the presence of a tag.

For example, the following guarantees a single instance with a given “Name” tag:

count_tag:
  Name: "{{ fullname }}"
exact_count: 1

For a particular internal service, we don’t want more than one instance, regardless of zone constraints. At the time, it wasn’t clear to us that the Ansible EC2 module has specific checks that apply to a given zone. And the zone was being set automatically elsewhere.

This leads us to the root cause of Thursday’s issue.

While debugging this particular problem the configuration was set to:

count_tag: Name
exact_count: 1

This means only allow one instance that has a Name tag, regardless of value.

Unfortunately this wasn’t executed in dry-run mode (-C) and ended up removing all but one instance in an availability zone.

We distribute most services across all availability zones, which meant we lost about 25% of our fleet.

Explanation of resolution

The immediate resolution was to re-provision the instances we lost.

Preventative measures

• Turning on termination protection for all our key long-lived instances, and making it the default behavior for new instances, will reduce impact of a similar incident in the future.
• Making the dry-run (-C) flag on by default will give more insight into what will change and help avoid accidental changes while debugging configurations.
• Creating an isolated testing environment for config changes will protect production services from bad config changes.
• Tightening up IAM role policies will help ensure that destructive actions (e.g. ec2:StopInstances) are restricted to automation systems that require them.
• Re-running disaster planning for loss of an availability zone will identify actions that will reduce the impact of such an event on core services.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Thursday 3 March 2016 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.

A bug was introduced Thursday that subtly broke caching for a subset of our database requests. This, combined with a lack of caching on some requests from logged out users, aggressive retry logic, and an increase in traffic from Talia Jane’s open letter increased pressure on our web tier Sunday morning.

For 30 minutes, between 8:55am to 9:25am PST, Medium’s primary web servers returned 504s to 20% of requests. On-call engineers were paged and were online within five minutes. It took ten minutes to identify the root cause and another fifteen to fully resolve it.

Timeline

• 02/17 11:00am PST
  A change in how tags are cached is deployed to production.
• 02/21 8:55am PST
  The tags table starts throttling and on-call engineers are paged.
• 02/21 9:00am PST
  On-call engineers respond online.
• 02/21 9:10am PST
  The problem is determined and our on-call engineers begin increasing the read capacity on the tags table.
• 02/21 9:15am PST
  The first increase in read capacity is fully pushed to production. At this point, the error rate is reduced but there are still errors happening in production.
• 02/21 9:20am PST
  On-call engineers bump read capacity on the table for the second time.
• 02/21 9:25 PST
  The second increase in read capacity is fully pushed to production and the site is recovered.

Explanation of root cause

Medium uses Amazon’s DynamoDB database for serving much of the site’s content. We also have a Redis cache in front of Dynamo to help smooth out our traffic load and prevent spikes that would otherwise result in DynamoDB throttling our request load.

On Thursday, a change was made that was intended to make the fetching of cached post tags more efficient. An unintended side effect of this change was that we stopped caching an empty list for posts with no tags. With this change out in production, requesting the list of tags on a post without any tags started hitting DynamoDB 100% of the time.

This issue wasn’t surfaced until the traffic spike on Sunday morning when we were paged for user facing errors and throttled DynamoDB requests.

Under the hood, the Node AWS SDK automatically retries requests to DynamoDB — including throttled requests — up to 10 times. This led to an increase in requests being processed concurrently on the Medium web servers, and subsequently an increase in latency. Our proxy servers compensated by shedding load and returning 504s. From 8:55am to 9:25am, approximately 20% of the requests to medium.com and other domains hosted on our platform returned 504s.

Explanation of resolution

Because the Dynamo table did not have the full protection of our cache, it became more vulnerable to increases in traffic. Temporarily bumping the read capacity Sunday morning helped us weather the spike, but was not a long term resolution.

Preventative measures taken

• Tuning the DynamoDB retry logic will decrease the perceived latency of throttled Dynamo queries. With fewer concurrent requests running on the system at once, the load caused by throttling shouldn’t spill over to other requests on the system like it did Sunday morning.
• Increasing test coverage will help detect the class of bug where caching was bypassed.
• Adding alerts that detect when access patterns to tables change will allow us to identify issues before users are affected.
• Reevaluate Dynamic Dynamo as a mechanism for adjusting provisioning levels more quickly than humans can respond.

The Medium Engineering team have committed to publishing a technical postmortem for serious outages to Medium core services, in order to build trust and hold us accountable to our users. More background on this program.

Sunday 21 February 2016 was originally published in Postmortems on Medium, where people are continuing the conversation by highlighting and responding to this story.