July 29 incident

At 11:49 AM JST, we received an alert stating that our API pods were becoming unavailable and the service went down for roughly 5 minutes.

Pods were getting restarted very frequently until eventually all of them started going down, causing a full API outage. We were initially a little confused because the time of the incident is when the number of requests typically wind down rather than peak.

When looking at the API latency, we also observed a severely slow response for certain endpoints.

The endpoint shown in the screenshot above is one we added for our calendar integration feature that we released just a few weeks prior. The API response time was slowly increasing after the release so there’s definitely a correlation between the feature’s release and the incident itself.

Investigation

The endpoint in question was actually doing two things.

1. Accept the code from the OAuth provider and exchange it for an access_token
2. Attempt to create resources required for the calendar integration to work (typically some sort of subscription so that we can be notified on changes in user’s status (Teams) or calendar events (Google Calendar)

While the former is generally a pretty quick operation, the latter can occasionally be an extremely slow one. Microsoft Teams Graph API can take up to ten seconds to respond, causing a time-out when attempting to create the necessary subscription. Google Calendar is typically not that bad but could still take a couple of seconds to complete the requested operation. On top of that, the subscription process could randomly fail and require the user to re-attempt the request, meaning the response from our API to our front-end could take an extremely long time.

Mitigation

We rolled back one of our integration releases to reduce the load on our API, bumped up the API resources and moved the subscription logic into background jobs. This way, it doesn’t block the API, and the endpoint is now only responsible for the token exchange process and firing up the background job.

We thought this one was a fairly easy incident to resolve, but it turns out the issue we addressed was not the root cause. The cause described above actually just made the real issue worse to a point that it finally caused an outage.

After further research, the actual cause was a default setting in our email client Swoosh that is causing troubles when ran in production. You can see the adventures we went through to reproduce it in this blog post.

This issue is now resolved at the time of writing and we are not expecting to see similar issues again. Furthermore, when developing new endpoints or adjusting old ones (which we’re still doing both as part of our ongoing effort to migrate away from PHP to Elixir, as well as whenever we identify endpoints that could or should be improved), we have now set rather strict limits on ourselves in terms of how fast an endpoint has to typically respond (Apdex 100ms), and if a long-lasting operation is required for the endpoint’s operation, it has to be developed as a background job with our job scheduler.

August 3 incident

At 8:51 AM JST, Datadog alerted us that our API was down. Five minutes later we started getting customer reports that people couldn’t access any workspaces. So at this point, it was clear that we were dealing with a full site outage.

Attempted fixes

Learning from previous incidents, our initial attempt was to rollback the most recent release of the API. There was a recent change that fixed a performance issue, and even though we didn’t think that it could be related, we proceeded with the rollback regardless because our first goal is always to return to a known good state.

Unfortunately, rolling back the API didn’t resolve the issue and also made it worse because the performance fix was now gone. During this time, we continued to observe high CPU load, memory usage (up to 3 GiB) and latency issues with our API.

The latency issue was actually what caused the API pods to keep getting restarted. Due to the health checks we set in Kubernetes, if Kubernetes doesn’t get a response within the predetermined time frame (in our case, 10s), it will consider the pod unhealthy and will force a restart of the pod in an attempt to fix the situation.

This restart caused a vicious cycle for the APIs to start up because,

1. Pod latency increases
2. Kubernetes thinks that the pod is dead because it doesn’t respond in time even though it’s still alive
3. Pod gets restarted but gets hammered with requests again and repeats from #1

Sentry

During all the scrambling, we noticed one interesting notification from Sentry.

Errors depleted for the month? After only two weeks into the Sentry subscription month?

Something looked extremely weird, and after digging through logs of crashed pods, we did see the following. And we saw it often, very very often.

00:04:31.923 [info] JOINED user_call:3694632 in 10ms

Parameters: %{}

00:04:31.923 request_id=a48af8b8029476faa05538f691f299d4 trace_id=708587351750908735 span_id=874386591673519646 [info] Sent 200 in 3086ms

00:04:31.924 [error] an exception was raised:

** (FunctionClauseError) no function clause matching in OviceWeb.UserChannel.join/3

(ovice 0.1.0) lib/ovice_web/channels/user_channel.ex:10: OviceWeb.UserChannel.join(“user:xxxxxx”, %{}, %Phoenix.Socket{assigns: %{user_id: xxxxxx}, channel: OviceWeb.UserChannel, channel_pid: #PID<0.1117.1>, endpoint: OviceWeb.Endpoint, handler: OviceWeb.UserSocket, id: “user_socket:xxxxxxx”, join_ref: “xxxx”, joined: false, private: %{log_handle_in: :debug, log_join: :info}, pubsub_server: Ovice.PubSub, ref: nil, serializer: Phoenix.Socket.V2.JSONSerializer, topic: “user:xxxxxx”, transport: :websocket, transport_pid: #PID<0.4437.0>})

(phoenix 1.6.6) lib/phoenix/channel/server.ex:376: Phoenix.Channel.Server.channel_join/4

(phoenix 1.6.6) lib/phoenix/channel/server.ex:298: Phoenix.Channel.Server.handle_info/2

(stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4

(stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6

(stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

00:04:31.926 [info] JOINED user_call:3583827 in 8ms

Parameters: %{}

00:04:31.930 [info] JOINED user_call:3563518 in 8ms

Parameters: %{}

Sure enough, we were generating tons of similar errors in a very short amount of time.

This helped us pinpoint the issue to a couple of things.

1. A recent front-end code refactoring accidentally removed a check that was supposed to act as a guardrail to not hit our API if the conditions were not met (or to be more exact, if a parameter is missing due to how the space is configured, this API should not be called as it doesn’t make sense without the parameter in place)
2. API accepting the request relied on Elixir’s function pattern matching and didn’t have a fallback implementation to handle requests without the required parameters

Once the former was identified, we quickly released a front-end fix to add back the missing check. At this point, the site was able to restore since the UI started limiting the requests it should be making — to give you a more concrete idea, well over 99% of spaces we maintain are currently configured in such a way that they should not be calling the API, yet without the check in place, they all were calling the API unexpectedly.

The latter was quickly fixed as a follow-up in the backend as well to make sure we don’t just error out when an invalid request gets sent.

Learnings

We reconfirmed how well Elixir can scale due to this incident. At the end of the day, it still went down but it was able to hold itself together even when generating 5M+ errors in the span of 30 minutes. Even though it still crashed, this just gives us even more confidence in the language choice we made.

August 8 incident

1:07 PM JST, an alert came in telling us the API was down again. But this time with a slightly different twist. The database (MySQL) CPU is at 100% usage.

This can only mean a couple of things.

1. Someone (or we ourselves) is DDoSing us
2. Some very inefficient query is running and is locking up the whole database (e.g. large DB migration)

Cause

Resolving a DB issue is usually pretty straightforward. You just need to login to the database and check what existing queries are causing the problem and then kill them, right?

Not so easy this time, especially when the database is using 100% of the instance’s CPU time. Getting into the database itself took a couple of attempts because the CPU usage was so high, it didn’t have any more resources available to grant external access.

Once the database let us in, it’s a matter of checking its running processes.

| 98293642 | ovice_admin | 10.xx.x.xxx:60756 | ovice | Execute | 439 | removing tmp table | <QUERY> |

| 98293835 | ovice_admin | 10.xx.xx.xxx:51358 | ovice | Execute | 438 | Sending data | <QUERY> |

Removing a temporary table…? That means that there was a scan going on somewhere that was so large it couldn’t fit into memory and had to be written to the disk first.

Another query (omitted from the listing above) seems to be called thousands of times — each of them are just queued up on the database side, waiting to be processed.

Upon further inspection, the query was related to fetching user’s personal history of most recent away memos. I’ll explain how that’s possible in just a few paragraphs.

Remediation

As an urgent and temporary measure, we immediately disabled the API endpoint that was issuing the inefficient query by returning an empty response instead, and we started seeing the CPU usage of the database instance drop significantly.

The table that caused this incident was already somewhat of a known issue internally. Since it was pretty quiet for most of the time and wasn’t causing any trouble, we decided to leave it alone until we had some bandwidth to address it in the future.

This is a typical example of technical debt getting out of hand if left around for too long. We’ve identified the following steps that we either have taken already or are in the process of implementing:

1. Identify that the query that caused the problem itself was necessary, but the underlying database schema could be altered in such a way that the query can be written in a much more efficient matter. As we’re slowly migrating not only from PHP to Elixir, but also from MySQL to Postgres, and because the access to the service was restored, it was decided that this endpoint will get a much better database schema to operate on, with one efficient table instead of two inefficient, and the new table will reside in our shiny new PostgreSQL instance. In the coming days, we have tested the new implementation thoroughly and finally last week, re-enabled the feature altogether, this time backed by schema that will hopefully work for a long time.
2. Unfortunately, the table that caused this incident is not the only known problem in our legacy database — there are two more tables with whose only purpose is to accumulate historical data, so we have started the process of identifying which data are truly important and must be kept, as well as the process of moving this data from MySQL to the appropriate data store.

August 23 incident

9 AM JST alert for workspace not being accessible was once again being triggered. Immediately we observed our API pods running out of memory.

There were a couple of endpoints that we were working on recently so we started rolling some changes back. The issue seemed to disappear momentarily but resurfaced after just a little while.

Cause

While the rollbacks we did seem to have resolved the issue at the time, there was still some doubt as to whether it was actually the root cause due to the amount of resources the endpoint generally uses.

After digging around, we realized an API endpoint that was supposed to be only used internally was being called pretty frequently during the incident.

What the problematic API endpoint does is generally querying all the users associated with a certain workspace. This is initially created for our own needs so while performance of the API is not good, it’s considered acceptable since the intention was never to be used by UI clients or by customers.

Unfortunately, it turns out that wasn’t the case. With some recent UI improvements, we were looking to see if there were existing endpoints that we could reuse in order to expedite the updates.

This internal API was found and was deemed as something that will meet our needs for the time being. And while we were aware of the performance issues, we didn’t expect it to be bad enough to cause a site outage.

So what happened? Turns out we had a workspace that has over 400,000 users associated with it. And the problematic API’s job was to retrieve all associated users in a space so it ended up querying all 400,000 users, attempting to store them in memory and of course, utterly failing to do so, leading to the Erlang VM crashing due to not being able to allocate the necessary memory, causing the incident.

Mitigation

We immediately limited the maximum number of users retrievable for any space being accessed. The fix itself was simple enough but I have to admit we didn’t expect to see 400,000 users associated with a single space. In the next round of fixes, we’ll implement proper pagination so that all records are still accessible as before, but without affecting the performance.

Learnings

Not bugs

While a couple of the causes were careless mistakes, these were all not bugs. When tested individually in Staging environment and our Release Candidate environments, things worked as expected.

Only happens at scale

Our current environment is set up to be able to test and verify features and bugs but it’s not built in a way to help verify scaling issues.

In order to address this, we’re looking into setting up a Canary environment where we’ll be doing load testing before making the releases to production to our paid customers via a rolling release to a subset of users.

Furthermore, we’re looking into ways to lower the impact area should similar issues happen in the future. One misbehaving endpoint, or one endpoint that suddenly becomes painfully slow due to the environment or data it relies on, should not bring down the entire system.

Technical debt

A couple of the causes were somewhat known issues that had been deprioritized due to higher priorities. They all happened to come and bite us this month, which was unfortunate timing.

Moving forward, any known issues will be addressed either before release or as an immediate follow-up. If it can’t be immediately resolved, we’ll at least mitigate the impact to make sure it doesn’t blow things up.

Summary

Thanks to you, our customers, for using our service. These problems only start occurring with high usage so this is a good problem to have in our eyes.

However, we understand how critical our service is for our customers and we will continue to improve on the process and automation of delivering changes. Our goal is to minimize disruptions and outages for our customers regardless of the origin of the issue.

A Post Mortem on August 2022 incidents was originally published in oVice on Medium, where people are continuing the conversation by highlighting and responding to this story.

まずは8月に起きた障害において、皆様に多大な迷惑をおかけしたこと、この場をお借りしてお詫びいたします。本当に申し訳ありませんでした。このブログでは、7月の終わりと8月に起きた各障害において、1. 何が起きたのか、とその 2.対策 についてユーザーの皆様に共有したいと思います。

より正確に情報を伝えるため、内容はテクニカル寄りになりますがご了承ください。

2022年7月29日の障害について

11:49 AM JST、APIがダウンしたというアラートが届き、5分ほどサービスが落ちたという通知が届きました。

内容を見てみると、k8sのPod（サーバーの一部だと思っていただければ大丈夫です。以降は「Pod」とのみ記載します。）が再起動する頻度が急に増え、そしてそれはいずれすべてのAPIにアクセスできなくなるほどでした。

また、APIの遅延を確認すると、いくつか非常に遅いものが目につきました。

上記のスクリーンショットは数週間前にリリースされたカレンダー連携機能と関連しているのですが、リリース後、徐々に遅延の数値が上がっているので、障害との関連性は高いと推測しました。

調査

問題のAPIは二つの処理を行っており、

1. OAuth認証成功後に渡されたcodeをaccess tokenに交換する処理
2. 認証したユーザーのカレンダーへ連携を試みる処理（会議が始まる、などのイベントをoVice側が探知できるようにするためのもの）

基本的に、#1の処理は早く終わるものなのですが、#2の処理はかなり遅くなることがあります。例えばMicrosoft TeamsのGraph APIはレスポンスが帰ってくるまで10秒もかかることがあり、それによってtimeoutされ、処理がキャンセルされることもしばしば起こります。

Google calendarはそこまでひどくはないものの、それでも数秒ほどの遅延は発生します。そのうえ、両者（特にMSの方）の連携処理が不明瞭な理由で失敗することも頻繁に起きるため、我々からretryをしなければならず、さらに処理時間が伸びることもあります。

対応

まず、緊急対応として

1. APIを一つ前のバージョンに戻す
2. APIの利用できるリソースを増やす
3. 連携処理をAPIではなくて、Background processorに移す

の最初の2つの対応を行い、連携処理がAPIのレスポンス遅延に影響を与えないようにし、あくまでOAuthのcodeとaccess tokenのみの交換だけを行うように変更しました。

対応としては簡単だったため、比較的軽い問題だと思っていたのですが、APIの状態をその後も続けて観察していると、頻度自体は減ったものの、問題そのものは解消されていませんでした。どうやら連携処理自体は本来の問題をただ悪化させていただけで、根本原因ではなかったようです。

引き続き調査を行ったところ、原因はAPIが使っているSwooshというメールクライアントであることが特定できました。詳細は別のブログ記事に掲載していますので、興味がある方はぜひそちらをご覧ください。

根本原因の特定とその対策をリリースしたため、現在問題は収束しています。今後の対策としても、APIの開発や修正に当たってresponse timeには厳重な速度規定を設定しています（Apdex 100ms）。もし、レスポンスタイムがこの規定時間を超えるようなことがあれば、background processorで実行されるように書き換えることになります。

2022年8月3日の障害について

8:51 AM JST、APIが落ちたと監視システムのDatadogから通知が来ました。5分後にはユーザーからもサービスが落ちたという報告が入り始め、スペースに入場できない事態になりました。

試み

過去の障害の学びから、まずは直近でリリースされた変更をロールバックしました。その変更は主にパフォーマンス改善関連のものだったので、あまり関連があるようには思えなかったものの、まず動作が確認できている状態へ戻すことが先決だったため、ロールバックを決行しました。

しかし、残念なことに、APIのロールバックでは問題は解消されず、さらにパフォーマンスの改善も見られなかったため、全体的に状態が悪化してしまいました。そのときどんな状態だったかというと、CPUの利用率が常に高く、メモリーの利用量も3G（普段は1Gほどの利用量なので、2 ~ 3倍の利用量）まで跳ね上がってしまっており、さらにAPIの遅延も非常に多くなっていました。

APIの遅延の問題は、Podの再起動に起因するものでした。具体的にご説明すると、Kubernetesで設定しているhealth checkがあるのですが、予め設定された時間内（我々の場合は10秒）に反応がないと、Podが落ちたと判断し自動で再起動を行い状態を改善しようとします。

本来、問題の改善を自動化するための設定が、今回に限っては裏目に出て状態を悪化させていました。現象としては以下になります。

1. Podのhealth checkの反応の遅延が長くなる
2. Kubernetesは時間内で反応がないので、Podが落ちたと判断されていたが、実際はまだ稼働している
3. Podが再起動され、立ち上がる際に大量のtrafficが流れ込み、処理しきれなくなって #1 に戻る

Sentry

問題をより正確に把握するために、複数人でさまざまなものをチェックしていた際、Sentry（システムのエラーレポートに使うツール）から一つの興味深い通知が届きました。

8月はもうエラーレポートの予算がなくなった。。？まだ2週間しか経っていないのに？

どう見てもおかしいので、落ちているPodのログを見てみると、以下のエラーが発生していました。しかもかなり頻繁に。

00:04:31.923 [info] JOINED user_call:3694632 in 10ms

Parameters: %{}

00:04:31.923 request_id=a48af8b8029476faa05538f691f299d4 trace_id=708587351750908735 span_id=874386591673519646 [info] Sent 200 in 3086ms

00:04:31.924 [error] an exception was raised:

** (FunctionClauseError) no function clause matching in OviceWeb.UserChannel.join/3

(ovice 0.1.0) lib/ovice_web/channels/user_channel.ex:10: OviceWeb.UserChannel.join(“user:xxxxxx”, %{}, %Phoenix.Socket{assigns: %{user_id: xxxxxx}, channel: OviceWeb.UserChannel, channel_pid: #PID<0.1117.1>, endpoint: OviceWeb.Endpoint, handler: OviceWeb.UserSocket, id: “user_socket:xxxxxxx”, join_ref: “xxxx”, joined: false, private: %{log_handle_in: :debug, log_join: :info}, pubsub_server: Ovice.PubSub, ref: nil, serializer: Phoenix.Socket.V2.JSONSerializer, topic: “user:xxxxxx”, transport: :websocket, transport_pid: #PID<0.4437.0>})

(phoenix 1.6.6) lib/phoenix/channel/server.ex:376: Phoenix.Channel.Server.channel_join/4

(phoenix 1.6.6) lib/phoenix/channel/server.ex:298: Phoenix.Channel.Server.handle_info/2

(stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4

(stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6

(stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

00:04:31.926 [info] JOINED user_call:3583827 in 8ms

Parameters: %{}

00:04:31.930 [info] JOINED user_call:3563518 in 8ms

Parameters: %{}

他の落ちていたPodも見てみると、案の定同様のエラーが短時間で大量に発生していたことが確認できました。

これによって、問題の原因が何なのかを以下のように絞り込むことができました。

1. 直近リリースされたUIの変更で、特定の条件を満たさない限りAPIへアクセスしなくていいようにするチェックを誤って外してしまっていたこと
2. 受け取るAPI側で、パターンに合わないリクエストが来たときの処理が実装されていなかったこと

問題を把握した段階で、まず #1 の対応に取り掛かり、急いで外されてしまったチェックを戻しました。この時点でサイトはすぐにアクセス可能な状態に戻りましたが、少しだけ事象をまとめると、

• 99%以上のスペースはまだこの機能を使っていません
• チェックを外してしまったことでUIが全スペースへのアクセスを試みるようになり
• エラーを大量に発生させる状態を引き起こし
• そのエラーが少しずつAPIへの負荷としてのしかかり
• 最終的にはKubernetesがトドメをさしたことになります

#2の問題もすぐさま修正し、期待されたリクエストが来てもエラーを起こさずに対応できるように変更しました。

学び

Elixirがいかにスケールできるかということは、今回の障害で再確認できました。何を言ってるんだ？と思われる方もいるかと思いますが、たしかに最終的に落ちはしたものの、500万以上のエラーを30分以内に起こしながらも、かなり長い時間耐えられていたということが正直驚きです。少なくとも、技術選定における自信が今回の事象で確かなものになりました。

2022年8月8日の障害について

1:07 PM JST、またしてもAPIが落ちたというアラートが飛びます。しかし、今回は今までのものとは少し違い、どうやらデータベースにアクセスできなくなっているようでした。

CPU稼働率100%、これが意味することはそんなに多くありません。

1. 誰かがDDoSをしている
2. 非常に効率の悪いSQLクエリが実行されていて、DBのリソースを使い切ってしまっている

の2つくらいです（もちろん他にもあるけど、簡略化するために今回は割愛）。

原因

DBの問題を解消するプロセスはそんなに難しくありません。

1. DBへログインする
2. 犯人と思われる実行中のSQLクエリを見つけて、キャンセルする

これで、おおよその問題は解決します。

ただ、そんな簡単なプロセスも、今回のようにCPUの稼働率が100%だと簡単ではなくなっていきます。なぜなら、処理するキャパがほとんどないのでアクセスするだけでも一苦労です。

何度も試した末、やっとの思いでDBへログインし、実行中のクエリ一覧を吐き出させました。

| 98293642 | ovice_admin | 10.xx.x.xxx:60756 | ovice | Execute | 439 | removing tmp table | <QUERY> |

| 98293835 | ovice_admin | 10.xx.xx.xxx:51358 | ovice | Execute | 438 | Sending data | <QUERY> |

Removing tmp table…? これが意味することは、スキャンを行う際に、取り出そうとしてるデータ量が多すぎるために、メモリー内に収まらず、ディスク上に一時的に書き込まなければならない状態です。

もう一つあるクエリはどうやら何千回も呼ばれているようで、ほとんどが実行待ち状態になっていました。

もう少し掘り下げると、どうやら関連のクエリは個人のawayメモの履歴を引き出そうとしているものでした。

解決

緊急対応として、まず該当するクエリを実行するAPIを無効にしました。無効といっても、実行自体をスキップしてデータを返さないだけで、API自体はまだ残っている状態です。

その変更を加えたところ、すぐさまCPUの稼働率が落ちていくのが目に見えて分かりました。

実は、今回の障害を引き起こしたDBのテーブルは、すでにエンジニアリングチームで問題児だと認識していました。そこで当然最初に疑問に思われるのは、なぜ事前に対応しなかったのか？だと思います。

問題児と言っても、定期的に内部でのデータの読み込みの際にデータ量が多すぎて処理しきれなくなるくらいで、実際のところ、本番ではほとんど問題はありませんでした。また、スタートアップである以上、リソースの配分はどうしてもシビアにならざるをえません。そのため、問題になりそうなタイミングで対処しようと考え、問題を先送りにしていました。

これは、技術負債の放置によって引き起こされる典型的な問題です。とは言え、問題を引き起こしてしまったため、これ以上放置するわけにはいかないので、下記のような対策をすでに実行済み、あるいは現在進行形で対処しています。

1. 問題のクエリ自体は必要なものですが、DBの設計自体はもっと効率よく書き直せるものです。また、MySQL => PostgreSQLへの移行も行っているところなので、より効率のいいDBで実行させるように書き直しました。この書き直しはすでにリリースされていて、しばらくの間、面倒見なくてもいいようになると思います。
2. 実は、問題のテーブルと比較的近い構造のテーブルは、他にも2つあります。ただ用途的にはRelational DBに保存するものではないので、近い将来、的確なDBへ移行することになります。

2022年8月23日の障害について

9:00 AM JST、スペースへアクセスできない非常通知が鳴り響きます。すぐさまPodのメモリーを使い切ってしまっている状態が確認できました。

直近でいくつかのAPIの変更を行っていたので、すぐに変更をロールバックしました。一瞬問題が解決されたように見えましたが、まもなく同じ問題が浮上しました。この時点で、直近の変更によるものではないということが、はっきりとしましたが、不可解なことに問題は少し経つと自然と収束していきました。

原因

この現象がどうしても不可解だったため、チーム一丸となり調査していると、一つのことに気付きました。

それは、本来内部利用のために作られたAPIが、障害中かなり頻繁に呼ばれていたため、これが社内の人間が実行していたものではないということがはっきりしました。なぜなら実行回数的に人間のできる速度ではないからです。

問題となったこの内部用のAPIは何をしているかというと、スペース内の特定のステータスのユーザーを抽出するというものです。元々内部のために作られていたもののため、多少のパフォーマンスの悪さには目をつむってきました。実行頻度自体が少なく、問題になりえないからです。

しかし、どうやらそうした状態ではなくなっていました。一般利用でも使われるようになったからです。最近のUI変更で、オフラインのユーザー一覧というものが増えたことをご存じでしょうか？

これは、その内部用のAPIを再利用することで実装されているものです。というのも、リソースが圧迫されながらも、変更を出し続けなければならないという状態において、エンジニアとしては再利用できるものがあるならしようと考えます。

この考え方自体は全く問題ではなく、私個人としてもそうしていたと思います。API自体も多少パフォーマンスに問題があるのは承知していたうえで、それでもサービスが落ちるほどの問題になりえないと思い再利用する意思決定をしました。

では一体何が起きたのでしょうか？

調査してわかったことは、どうやら、ある一つのスペースで、400,000人以上のユーザーが紐付いていたことです。つまり、そのスペースにアクセスする時に、自動で400,000人分ものデータを引き出そうとします。

しかも、これは誰かがスペースにアクセスする度に起こります。その結果、APIは何度も400,000人のデータを読み込もうとし、Podのメモリーが足りなくなったことで、最終的に許可しているリソース以上を使おうとしていたため、KubernetesがPodを自動で落としていました。これが何回か連続、かつ高速で起こると全Podはアクセスできなくなり、結果としてこの障害が起きてしまったのです。

自然と収束したこともこれで簡単に説明できます。そのスペースへのアクセスが一時的に増え、そのアクセスが減ると共に、膨大なデータ量にアクセスしようとする試みも一緒に収束していきます。

対策

最初の対応策として、引き出せるユーザーの量を強制的に絞るようにしました。その引き出せるリミットはoViceで2番目に大きいスペースに合わせていますが、それでも問題となったスペースの1/100ほどで、APIが十分対応できるレベルです。

また、フォローアップとして、リミット以上のデータを取得しようとする場合は、Paginationを行うようAPIを少し変更します。

学び

バグではないこと

最初の2つの原因は不注意により引き起こしてしまったことですが、これらすべての障害で共通していることは、どれもバグではないということです。エンジニアたちは実装後に自分たちで検証環境でのテストを行っており、QAも共に通っているため、個別に見ると挙動はあっています。

スケールした時のみに起きる問題

oViceの現在の環境は、機能やバグなどのチェック（自動化も含めて）を行うことは可能ですが、大規模なスケールで起きる問題は検知することができません。

この問題は、今回の一連の障害ではっきりとわかったため、今後の対策としてはカナリア環境というものを追加し、一部のユーザーで、ある程度の規模以上での挙動を確認してから全体にリリースするように変更します。

さらには、将来似たような問題が起きたとしても、インパクトの規模を制限できるような対策も講じています。進歩とは、つまり変化することで、変化すれば必ず問題は一緒に付いてきます。

故に「すべての問題を防ぐ」ことは考えていませんし、現実的に不可能です。

しかし、本来の予想と違った挙動をするただ一つのAPIが全体を落とすような事態になってはならないので、問題が発生した際の規模を絞れるようにします。

技術負債

ご覧の通り、いくつかの問題の原因は対応を後回しにしていた技術負債から一気にしっぺ返しを食らったことで起こったものでした。運が悪いかと言えば、そう言えるかもしれません。

しかし、運が悪かったで終わらせるつもりは当然ありません。今後も、今回の問題のようにショートカットを選択することはありえます。これに関しては避けようのないことです。ただし、問題を把握していれば規模を制約することも可能なので、これからはまず、完全ではなくとも、把握している問題の対策を行ってからリリースするようチーム全体でしていきます。

最後に

日頃からoViceをご利用いただいてる皆様、本当にありがとうございます。反省することは多くあるのですが、少し見方を変えてみると、これらは本当に多くの方々がoViceをたくさん使っていただいたことでしか起きない問題でもあるため、個人的には非常に良い悩みの機会をいただいたと思っています。

当然、oViceというサービスがいつでも正常に稼働していないと多くの方にご迷惑をおかけすることも認識しています。エンジニアチーム一同として、社内のプロセスや自動化の発展に精進し、また、たとえどんな原因のものであろうと、問題や障害を可能な限り最小限に抑えられるよう引き続き努力してまいります。

A Post Mortem on August 2022 incidents — JP was originally published in oVice on Medium, where people are continuing the conversation by highlighting and responding to this story.

September 3rd, 2021 was my last day at Cloudflare. It was an exciting 2 years, being in the front seat of a fast-growing company.

I was part of an SWE/SRE team within the Infrastructure organization that builds out automation for data center and hardware operations, helping the company to handle its ever-growing numbers of servers. The work was fun and fulfilling, and the team is great - people you can truly trust and know that they will have your back is not an environment you often find.

Also, it was an eye-opening experience being able to see how certain things get done from the inside. For example, Cloudflare’s blog is a well-known and highly rated content in the industry. One of the main reasons is its transparency and the sophistication of the content itself. A lot of folks will agree that they’ve learned a lot just by reading through Cloudflare’s technical blogs. If there’s an incident or something went wrong, you’ll also be able to know what exactly went wrong and how does Cloudflare makes sure it doesn’t happen again. This creates trust for customers and eventually led to better business.

During my final executive interview with Matthew (CEO), I asked him, “what does trust mean to you?”. To this day, his answer still sticks with me.

There are 3 types of trust:

Trust from customers

Letting them know you have their backs and will always be there for them. Making sure issues are transparently communicated and how they’re handled to make sure it doesn’t happen again.

Trust from your team (employees)

A corporation is ultimately just a group of people. Without them, you can’t operate. As a company, we should all be heading to the same direction, so making sure that there’s trust within the company will result in more happy people, which will ultimately lead to having more happy customers.

Trust from society (and communities)

A company with happy employees can’t operate on it’s own. It’s built on top of layers of infrastructure and communities. While making money is a priority for a profit organization, we need to make sure we fulfill our share of responsibilities and give back to the communities we build our business on. Without them, there’s no us, so having that trust between Cloudflare and our communities is also very important.

Pretty insightful, right? His answer was the ultimate push for me to decide to join Cloudflare 2 years ago and I can tell you he has been true to his words even by observing from afar.

While the company is doing extremely well with a 600%+ value growth for the past 2 years, I’m pretty sure it won’t stop there and it will continue to do well in the coming years.

You can tell how much I like being at Cloudflare, so leaving was a very hard decision to make, including the financial incentives. But having said that, I’m quite excited about my next opportunity.

Why oVice?

The title pretty much speaks for itself. I’ll be joining oVice (headquartered in Japan) as the VP of Engineering starting September 27th, 2021.

So what is oVice and why I made the jump from a successful public company to a startup?

Obviously, it’s the Metaverse!!
That’s going to be next big thing!!!

Jokes aside, the actual reasons for me to join oVice are,

1. Being to be able to provide options for a more flexible work-life balance
2. Solving the technical challenges to provide for #1

The offline/online hybrid work style

After we all went into lockdown with COVID-19, I’ve got nothing much to do after work so I started helping out other companies as a technical advisor. oVice was one of them.

The product was extremely interesting. The company positions itself as a service provider for virtual real estate. Meaning you can sign up for spaces with oVice, customize it to your liking and then network and connect with people there, or do whatever you want to do like how you would with a piece of land in the real world.

While I love the idea of being able to work remotely and not having to commute to the office — the traffic around San Francisco is just pure horror and on top of that, you might get your car’s window smashed. Why in the world would someone want to go pay for parking in the city for 8 hours and head home with a $500 bill? Just doesn’t make sense —, one of the major pain points has been the loss of human interactions when people are present in a physical space.

Replicating human interactions online

There are numerous times we’ve overheard some other interesting conversations happening nearby, jumping in, and then lead to ideas and cross-team collaboration efforts you wouldn’t have come up with if you’ve just been sitting by your own in a little box.

Being all by yourself in an isolated space 24/7 can be pretty lonely as well. Humans are social animals so being able to chit-chat and talk with people can be good for one’s mental health. Zoom and Google Meet are not helping in that sense, and they are not designed to be used that way so you really can’t blame them for it.

oVice tries to solve that problem digitally. The avatar that represents you can move around freely within each space, and there’s a concept of direction and distance built into the app. Meaning, there’s only so far your voice can travel. So if you’re too far away from one another, you’ll no longer be able to hear what other people are saying, which is pretty close to what you’ll expect in the real world.

This means that you can virtually sneak up to a group of people and start eavesdropping on them, like how you’ll do it in the real world!!
I mean, that’s how you network with other folks, right? :P

As time passes, it can get harder to keep in touch with folks that were once part of our lives, due to physical reasons or maybe other responsibilities. While most existing customers use oVice for work or school, it can be also used for all kinds of other gatherings, like one-time events and class reunions, etc.

How will work look in the future?

Reducing the need or requirement to be at a physical location could help people lead a lifestyle they want as well. Perhaps you don’t like to live in crowded places like New York City or San Francisco (or Tokyo) but still want to be able to work for a company located there. oVice can help achieve that, and can still make you feel like you’re close to the people you work with regardless of the physical distance apart.

On the other hand, if you’re a company looking to access talent globally, oVice can help you make sure employees in remote locations feels like they’re part of the team regardless of where they live. Having that sense of belonging can be a crucial part of both the company’s and the hire’s success.

When I see technology having the potential to help bridge the gaps, and provide alternatives to the traditional ways of doing things, that’s when I get most excited.

NOTE: We can’t solve time zones just in case if you’re wondering so you’ll need to figure that part out yourself

Technical challenges

I’m an engineer at heart, and solving problems are what I enjoy doing most.

For oVice to fulfill its full potential, there are a tremendous amount of technical challenges to solve. While WebRTC technology is not something new, the major difference between oVice and other existing video collaboration tools like Zoom or Google Meets is that you have the extra layer of real-time visual interactions on top. That adds a new set of complexity for human-machine interactions and making this work will be something quite interesting to solve.

• What would mobile interactions look like?
• How do we allow other people to develop on top of oVice and create their own experience?
• How do we make sure things scale and handle connections well even with 10s of thousands of connections in certain spaces?
• How do we make sure the people working in an office and people working remotely can still collaborate well together?

These are all interesting problems to solve, both complexity-wise and scalability-wise. And being one of the early companies in this space, our solutions could have the potential to become the next norm of how people interact with each other moving forward.

Just to be clear, we’re not trying to make you ditch the real world and make you throw yourself completely into the digital world like some companies are. We’re merely trying to extend what’s possible, and allow people to choose the best options for themselves and how they want to live their lives.

Technologies have evolved a lot in the last 2 decades, and I believe we’re at a turning point that we can achieve what our predecessors couldn’t.

Sounds interesting?

We’ve just closed on our Series A and we’re expanding globally (Will share English press once released).

バーチャル空間「oVice」運営、シリーズAで18億円を調達【日経報道】

Come join us!!

• Product Designer
• Software Engineer (Backend)
• Software Engineer (WebRTC)
• Site Reliability Engineer

The location is fully remote. The job description is set to the US because LinkedIn won’t let me just set it as remote.

Joining oVice as VP of Engineering was originally published in oVice on Medium, where people are continuing the conversation by highlighting and responding to this story.

At first, I was just trying to deal with the pain handling Cron tasks and offline workers, and looking into existing solutions wasn’t very satisfying. Building a prototype was relatively easy since the concept itself was simple.

The foundation of the project was done in a couple days, a couple evenings after work and BOOM! 💥 MVP completed. After that, it was just adding a small feature here and there every week or two.

Mistakes 🈲

A few weeks later, after talking to a colleague which I don’t recall what the exact conversations were, I got excited and thought it was a good idea to launch it as a service and make it publicly available.

I know how to launch a service. What could possibly go wrong, right?

Allow me to share some of the mistakes I’ve made.

Pre-Optimization

After deciding to publicly launch the product, a ton of engineering work suddenly showed up.

• Sign up / Login
• Authentication
• Authorization
• Access Controls
• API designs

etc.

So far in my career, I’m lucky to have been able to work with and learn from engineers that would try to do things “the right way”. Which is extremely important when working with multiple engineering teams on an established product. However, that’s not the right approach for a product in an early stage.

My mind has been so biased towards the “right engineering approaches” that I’ve been ignoring what actually matters.

Just get it out of the door, talk to people and iterate

As a result, it cost me months instead of weeks before putting the product before people’s eyes, just because I wanted to do a fancy Typescript React Front-End with a Rails API and using Procotol Buffers for communication, because I don’t want to maintain the API schema. I over engineered the infrastructure as well and while it can scale well, it has become a financial burden because doing things the right way is not cheap.

To be honest, it did help catch a lot of issues when doing some major schema changes. Oh, and did I mentioned that my colleague actually tried to DDoS the service and it withhold? Pretty impressive right? But it doesn’t matter, it’s still solving the wrong problems at the stage it was in.

Underestimating the amount of work outside of engineering

Being an engineer and knowing engineers, we tend to think that building a good product will automatically get us customers and the rest will be history. While there are some portion of truth to it, having a good product will likely give you some advantages over your competitors, it’s definitely not going to take over the world just like that.

After making LivePing public, I started looking into landing pages, marketing, SEOs, in-corporation, accounting, etc..

There’s a lot of work besides just coding something out and throwing it on the internet, and I’ve seriously underestimated that. And critically enough, I’ve done it in the wrong order. Building a product, then market it was a complete waste of effort and time, especially when there’s not even a close to guarantee that the product will be what you think it would become.

Positive side, while I’ve never really thought that other jobs would be easier than mine (maybe a little bit :P), I’ve learned to really appreciate the work of the people in marketing, sales and all the other departments that operates and runs a company. Experiences like this makes you more humble and I think I understand why a lot of successful CEOs have a humble mindset now.

Not verifying the market size

Business is about people and solving their problems. Verifying the needs and pains your product solve is the first step which most people knows, clearly. However, another important portion that tends to get overlooked is the size of the market.

Sure, the product is solving a pain, I’ve conducted user interviews before deciding to make it an actual product, and there already are competitors in the field so that helps me verify that there are need for such a tool. But if the market is not big enough, it might not be worth the effort pursuing.

Trying to make $1000 in a $10,000 market vs a $1,000,000 market is a huge difference. Neither will be easy, but the effort required for the former could be way higher than the latter, resulting in the cost and effort of trying to grow the product becoming higher than the rewards.

In the past couple months, I’ve learned that the market LivePing is in are one of those markets that efforts can be greater than rewards, hence making it harder justifying the additional effort to be put into the service.

You might be thinking, “Clearly you’re not doing a good enough job of researching the market beforehand, otherwise you’ve known”.

While I do admit that I did ran with optimism, it’s also hard to know when and what is “good enough”. I’ve come to learn that, there’s never going to be a “good enough” because at some point, you’ll just have to run with it and learn while doing so. Knowing the border between research vs action is kind of an art, and it’s different for everyone. On top of that, knowing when to withdraw is even harder because there will always be some level of emotional attachment to your product. Hence, knowing those borders requires experience, and LivePing had taught me that in the hard way.

In hindsight, even though I knew all these principles from reading famous startup books, I still committed the same mistakes that people tell you not to. And because of the mistakes, I’ve wasted a lot of time and money which otherwise, I didn’t have to.

Another interesting aspect was that mistakes can have compounding effects on your motivation, and will put your passion to the test. Turns out, I don’t have as much passion as I thought I had for developer tooling after all and it was a precious insight that I would not have known unless having been put on the spot.

End of Life 🌆

LivePing will be shutting down at the end of February 2020.

Any active subscriptions will be cancelled and payments that extends beyond the end date will be refunded on a pro rata basis.

If you’re looking for alternatives for your workloads, here are some options available.

• Cron monitoring and uptime monitoring for websites and APIs - Cronitor
• Free Cron Job Monitoring - Healthchecks.io

I apologies for the inconvenience, and like to thank you for trying out LivePing. One of the positives that came out of this experience was being able to experience the joy of seeing someone else using something I’ve built from scratch, and receiving feedback for it. It’s fulfilling in a different kind of way compared to having a full time job working on an established product.

While there’s nothing specific on the pipeline for the next project, I’m likely not going to be working on developer tooling for business anymore, and will be looking at other industries where I’ve confirmed my passion.

Shutting Down LivePing was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

ソフトウェアの会社なら、必ず使ってると言っても過言ではない普遍的な素晴らしい技術です。しかし、実行中にエラーが発生した時はいったい何が起こるんでしょうか。。？

Silent Failure

ここ数ヶ月は、自分のためのサイドプロジェクトに取り掛かっていて、そしてできたものをしばらく使っていました。ある程度公開しても大丈夫な仕様になりましたので、ぜひ見てみてください。

LivePing

LivePingは一言で言うと死活監視のためのシステムです。ただ、死活監視と言っても大きくは、

• 外部ネットワークからチェックできるUptime Monitoring
• ネットワークからそもそもアクセスできない代わりに、自身がシステムを通知するHeartbeat Monitoring

の２種類に分かれます。LivePingは後者のHeartbeat Monitoring（直訳だと鼓動監視？）のシステムです。

さて、Cronってそこまで重要なものでしょうか？実は、TwilioのCEOが半冗談でツィートしたように、少なくともアメリカのソフトウェア業界を裏で支えてる重要な技術の一つです。しかし、どんな技術にも長所と短所があります。そしてCronも例外ではありません。利用したことのある人なら一度は考えたことがあるのではないでしょうか？

もし実行時に何か問題が発生した場合はどうやってそれを認知する？

Cronを筆頭に、オフラインで実行するソフトウェアは基本的に、全く音を立てずに、静かに障害を起こします。今頭に浮かぶだけでも、

• データベースのバックアップ失敗
• 外部向けのSSL/TLSの証明書の自動更新の失敗により、後日サービス全体へのアクセス障害が発生
• 再訓練された機械学習のモデルがデプロイされず、パーソナライズのサービスが日に日に不備を蓄積
• 社内やサービス間のTLS証明書の更新失敗により、サービス間の通信に障害が発生
• 毎月に顧客のために自動で生成されるレポートが生成されずに、クレームに繋がる

以上の例以外にも他に多くの問題を経験しました。

そして、畳み掛けるかのように、時には問題が発覚するのは数カ月先ということも珍しくありません。

Attempts in the past

当然問題を認知しているなら、それを解決しようとします。過去には何度か社内ツールとしてこの問題に対応してきました。そして、毎度同じように、メンテナンスの優先順位という課題にぶつかるのです。

監視システムである以上、稼働時間が短かったり、簡単に落ちていては意味がありませんが、片手間で簡単にできるものでもありません。少なくとも社内で唯一のSRE、あるいは数少ないSREの場合、障害対応以外にも対応しなければならないものや自動化させるものが他に大量にある中で、どうしても社内ツールの対応する優先順位は下がってしまうのです。

ならば社内ツールを担当するチームが存在するほどの規模の会社なら解決する問題なのか？それが面白いことに、意外とそうでもないことが多いのです。ツールのチームが担当する範囲のものは多岐に渡り、同じ社内ツールでも、エンジニアたちが直接影響されやすい、言語系フレームワークやCI/CDのような継続開発をサポートするツールに意識が行きやすく、Heartbeat Monitoringのシステムへの注意がどうしても後回しになってしまうことが多いです。

また、システムの担当や所有権も一つの問題になりえます。簡単に言えば、もし現在担当してるエンジニアが会社を去った場合、誰がシステムを引き継ぐのかという問題です。シリコンバレーは特に人材流動の激しい地域です。それによって多くの会社ではシステム構築者がその会社を去った後に、所有権が有耶無耶のシステムが数多くあり、誰も触ろうとせず、その上エンジニアの多くはドキュメンテーションを書きたがらないため、現実と認識のズレが発生し、知らない合間に会社のソフトウェアサービスが静かに危機的な状況に陥ることも珍しくありません。

そして最後に、「監視者を誰が監視する？」という問題も付いてきます。香ばしくなってきましたね？

じゃあ外部サービスを使えばいいじゃないか？と思うかもしれませんが、当然すでに試しました。最低限に必要な機能は確かに備えていますが、最低限ということは、つまり追加機能は結局自分が実装することになるので、別に大して仕事量が減ることもありませんでした。

Problems LivePing solves (for now)

現時点のLivePingはまず、僕個人の問題を解決するために作っています。一つは上記のような優先順位や会社を去った後の劣化の問題と、もう一つは自分が何度も何度も同じようなシステムを作る羽目になる問題です。

そして、これは問題ではないですが、僕が得意とするスケールとシステムの信頼性の一部を雇用関係になることなく、多くの人に提供できるかもしれません。

さて、ここで一旦僕個人の問題はさておき、LivePingはどのような問題が解決できるのか、少しお見せしましょう。

• サイレント障害の防止
• Cronが仕事をしていなかった場合のためのアラート
• データベースバックアップやTLS証明書の更新がちゃんと走ったかどうかのチェック

他に使える定時実行の例としては、

• データ同期のパイプライン
• データの集約やデータの廃棄
• システムの脆弱性のチェック

などがあります。当然、何かしら定時実行する必要なものであれば、基本的に対応できると思っていただいて大丈夫です。

現在のLivePingのアラートはEmailとSlackのみ対応していますが、これからどんどん機能と他のアラート先を実装していくつもりです。

短期のロードマップとしては、

• PagerDutyのアラート
• Webhookの対応
• アラートを飛ばす前の猶予期間
• 一時的なアラートのミュート
• APIアクセスとドキュメント
• OAuth2による認証

これ以外にもまだまだいっぱい機能を追加する予定です。

上記の機能の実装以外にもシステムの信頼性をあげるための自動化も全力で取り組んでおります。利用された際にはぜひフィードバックをいただけたら幸いです。また、希望する機能やアラート先がある場合はFeature requestを提出してください。各種のフォームや連絡情報はContactページに掲載しております。

簡単にお試しができるように、LivePingは無料で5つのPingを提供しております。有料プランもすべて14日の試用期間を設けております。詳細はPricingページを参照してください。

What’s next?

短期的には、まず僕が過去にぶつかった多くの問題を解決するための機能を追加していく予定です。

しかし、長期的には会社やチームとともに成長できるシステムにしていきたいです。最初はもしかしたら、ただひと握りのCronを監視するために。次第にはそれぞれのチームで使い分け、アクセス制御やダッシュボードの提供。そして究極的にはサービスとともに監視を自動化していく。

Monitoring（監視）は、あくまでObservability（可観測性）の一部に過ぎません。監視の中でもサービスの死活監視と、分散されたスケジュールシステムだと対応も違ってきます。故にPrometheusにもHeartbeat Monitoringのために、Push Gatewayという副次的な機能が備えているのです。

もしPrometheusをすでに利用されているのであれば、Heartbeat MonitoringもそのままPrometheusを利用し続けることをお勧めします。しかし、もし運用してない、あるいは社内でそのようなチームを作る余裕がない場合は、ぜひLivePingを一度試してみてください！

質問、または単に話してみたいだけでも、遠慮なく連絡をください！

最後まで読んでくれてありがとうございます！

Launching LivePing (日本語) was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cron is a simple but excellent piece of software that most software companies in the world rely on. But what happens when the job or script cron fails during execution…?

Silence failure

I have been working on a side project of mine and was using it for a little while. I have now made it available as a service, and it is accessible here.

LivePing

LivePing is a heartbeat monitoring tool that helps check workloads that don’t accept HTTP requests, like Cron and background workloads.

So, what’s the big deal about cron anyway? There’s a tweet that jokes about the importance of cron in the economy, but the impact it has is no joke. While cron is something a lot of companies rely on, there is also a problem that needs consideration when operating them.

How do you know when they are not doing their job?

Background workloads like cron tended to fail silently and had led to multiple issues in the past for me:

• Database backups were not running
• External facing SSL/TLS certificates did not get updated and caused a site outage
• Retrained machine learning models did not get deployed
• Internal services went down silently (related to internal TLS), creating a cascading effect on other services
• Monthly email reports for customers did not get generated and sent out

these were merely some of the numerous examples.

Certain times, they might not even cause any side effects and only found out that the workloads have not been working for some extended amount of time.

Attempts in the past

I’ve solved this problem a couple of times in the past as an internal tool, but it always comes back to the same maintenance priority.

Since it’s a monitoring system, it needs to have a decent amount of uptime, but guaranteeing that is typically not something you do single-handedly. At least, not if you’re the solo, or maybe one of the few members of a small SRE team when you have a thousand other things to handle and automate. Don’t mention unexpected incidents too.

Even with an internal tooling team at a larger company, that team is likely to be dealing with other projects, maybe language frameworks, CI/CD pipeline, and have other responsibilities. When trying to support more visible and direct tooling for other engineers, handling of a heartbeat monitoring system tends to get de-prioritized.

Another problem is ownership of the system. This is not a problem while you’re still with the company, but it emerges likely after you left.

Developers are not necessarily good writers, and there are always gaps between documentation and the running systems. I’m guilty of that too. So things get missed, and systems can reach critical states without being noticed after a certain amount of time.

And last but not least, you get into the “who’s going to watch the watchdog” problem. :)

So you might be thinking now, “I can see operating as an internal tool can be burdensome for people, then why don’t just use Cronitor or Dead Man’s Snitch?”

Yup, I’ve tried those, and yes, they do fulfill the bare minimum requirements I need. But they have their own set of problems, and I might touch-base on that in the future.

Problems LivePing solves (for now)

First and foremost, LivePing solves my problem of not needing to create another monitoring system that doesn’t age well. And the issue of doing roughly the same thing over and over again at different places. ;)

Also, maybe providing something that I’m good at, which is reliability and scalability to a broader audience.

Enough of my issues, let’s see what LivePing can do for you.

• Eliminating silent failures that can lead to disastrous situations
• Provides alerts before problems start to snowball
• Preventing the issues I’ve listed earlier like DB backups and TLS certificate updates, etc.

LivePing can also help you make sure these scheduled tasks are running and working as expected:

• Data synchronization pipelines
• Data aggregation, garbage collection
• System/vulnerability scans

While LivePing is functional at this point with Email and Slack as alerting targets, there are still a lot of lacking features and integrations.

Some of the stuff on the near term roadmap would be:

• PagerDuty alert integration
• Custom webhooks
• Grace period before alerting
• Temporarily silencing alerts
• API access and documentation
• OAuth2 authentication

and there are many more to come.

I’m working hard to improve and automate the system overall to provide a high SLA uptime and a reliable system and would appreciate your feedback using it. You can find other various forms like feedback and feature request on the contact page.

In effects to make it accessible, LivePing comes with 5 pings for free and doesn’t require credit card information to sign up. Any plans going forward will have a 14 day trial period. You can see the pricing details on the pricing page.

What’s next?

In the short term, I’m planning to be focusing on adding features to solve problems I’ve encountered in the past, including scalability issues.

But in the long term, I want to create a system that can grow with the team’s needs. Initially, maybe just simple cron monitoring for the small number of jobs the company has. Then, moving up to dividing up jobs between teams, access controls, and dashboarding. Eventually, to a system that is easy to integrate into the automation of services.

When it comes to the observability of systems, monitoring is just a subset of it. And monitoring uptime vs. a distributed scheduling system requires a different approach. That’s why even Prometheus has a push gateway.

If you’re familiar with Prometheus, which is an excellent piece of software, by all means, go with it. If not, or you don’t have the resources to set up such a team, give LivePing a try!

If you have any questions or simply just want to chat, please reach out!

Thanks for reading! :)

Launching LivePing was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

Web development technologies, especially front-end related ones are changing rapidly in the past few years.

One of the unexpected areas they have entered (personally) are the text editor space. The well known ones these days built from web technologies are Atom and Visual Studio Code.

I consistently get questions regarding Emacs from people considering switching to it but are not sure what kind of horror stories are awaiting them so hopefully this would shed some light.

What is Emacs?

In case for people who are not familiar, Emacs is an editor that was released in 1976, which is around 43 years ago at the time of writing this article.

It’s built from Lisp and C, and is cross platform till this day. Meaning you won’t have issues running it in Apple, Linux or Windows (actually I don’t use Windows so not sure :P) devices.

So like everything in life, there are always some kind of opportunity cost to decisions and everything is a trade-off. Here are some hopefully unbiased Pros & Cons if you’re considering something like Emacs.

Pros

Fast

Yes, that’s right. It’s actually fast and doesn’t take like 16ms to render text on your bloody screen. So you can actually have a pleasant experience if you are used to doing blind touch typing and not get irritated every time when latency happens from your keystroke until when it shows up on the screen.

It can also be extremely fast to setup, since configurations are just dot files which makes synchronizing configurations between computers extremely easy. Download the dot file, boot up Emacs and you’re ready to go!

Wide list of plugins and support

Since it has been around for over 40 years at this point, you can basically find a plugin for almost anything you want to do.

Programming language support, git operations, google search, you name it. I bet it’s actually harder to find something it can’t do.

You can also go pretty crazy with it, like view, compose, send emails, even browse the internet with Emacs. If you’re using the GUI version of Emacs, HTML can actually render so it’s a pretty interesting experience, literally having code and browser side by side.

To be honest, I figured that I just want to see text on my editor so I don’t do the those crazy things anymore. You know, Mistake of Youth.

Keyboard only navigation

To be accurate, you can still click around in Emacs, but you don’t need to. Everything can be handled with your keyboard and this actually contributes a lot to the speed of editing too. It doesn’t mean that writing fast is always good. What good is there if you produce s**t 10x faster, you know like (my) definition of a 10x engineer? But if you can express your thoughts in a shorter amount of time, why not? That small amount of time will add up and you can get hours back per day. Like literally.

It’s always kind of funny to hear a fellow Emacs user who have to use a different editor temporarily and complains that using a mouse just slows them down tremendously.

No offense to anyone who uses a mouse, it just feels very weird to have to use one when you got use to handle everything with your keyboard. It’s like, “wait, where is my hand supposed to go now?” and have to pause for a few seconds deciding on your hands’ next move.

Oh and it also makes file editing in servers easier too.

Multiple files in one screen

This is actually unique to Emacs (and Vim) where each file is a buffer and the concept of a frame and buffer is different so you can have a loooooooot of buffers in one frame.

This work very well if you have a very short memory span. Like literally have your memory wiped while switching screens and you have no idea what you were supposed to be looking at so you switch back to the previous screen, repeating this back and forth like I don’t know how many times wasting hours just trying to figure out what you were thinking about 2 seconds ago.

At a bare minimum, I will always have 3 files opened and sometimes more than 8. I get the same comment from all my past and present colleagues when they walk pass my desk.

“I have no f***ing idea how you even operate with that.”

Cons

High learning curve

I’m pretty sure this is one of the common reasons that bars people from using the tool. Emacs Lisp, while it’s extremely flexible and fast (compiles to byte code), it has a syntax a lot of people are not used to.

Default key strokes in Emacs can be awkward at first as well. For example, saving a file is Ctrl-x Ctrl-s . That’s 2 key strokes and seems more effort for a lot of people when you’re used to just doing Ctrl-s or Cmd-s for saving a file.

If you’re willing to stick with it, you’ll be amazed how fast it’ll become going through all those weird key stroke combinations thanks to muscle memory, but your productivity will definitely slow down for some amount of time until that memory sticks.

Extreme customization

Related to the key stroke combination above, some crazy people (like me) go extra steps to customize Emacs to fit their needs. This is one of the double edged sword characters of Emacs, because it makes pairing soooooooooo much more painful. You can hardly pair with even a fellow Emacs user, since it will feel like you’ve suddenly entered a different universe.

But hey, I don’t work at Pivotal, therefore I don’t pair 100% of the time so it should be fine right?

Documentation

If it has been around for such a long time, wouldn’t there be plenty of documentations?

Yes, there are tons of them. But also a lot of them are outdated so there could be struggles when you try to do something following the documentation you found online and it completely blows up on your face.

Package repositories are also kind of scattered around both up to date and outdated so you might need to connect to all of them in order to get the package you want. Looks like people are finally deciding on a single source of truth these days so hopefully life will be easier when you are just trying to install a plugin to test it out.

Some distributions like Spacemacs can help by providing a layer of abstraction so you don’t need to handle all the details yourself. It also provides a nice set of defaults, but if you’re the kind of person that wants to know what each of you configurations does… oh well, you have been warned.

Why Emacs

So, should you switch to Emacs?

To be honest, you don’t need to. Like I said earlier, everything is a trade-off. So if you’re the kind of person who don’t care about the tool and want it to just work, then VSCode, Atom or Sublime is a better fit for you.

If you’re the kind of person who cares more about performance and don’t mind investing a little time to get that, then I think Emacs is a great editor, even after 40+ years since its creation.

How about Vim?

Pros and cons wise, Vim is pretty much the same. I was initially a Vim user, but had to switch to Emacs for performance reasons when editing large files x amount of years ago.

To provide a little more context before I get backslash from enthusiastic vim users.

Back at the time when I decided to switch to Emacs, I was doing

• Linting
• Code completion
• Auto code format after save

on all opened files. Vim wasn’t able to do async operations back then so whenever I opened a giant file, it takes like 5 seconds for my typing to show up on the screen, and I think everyone will agree that 5 seconds is unbearable. I’m complaining with 16ms so you can tell that I was tearing my hair out with that amount of latency. But I didn’t want to disable all those settings so I decided to give Emacs a try since I heard it compiles to byte code and I thought I would be able to keep the best of both worlds.

Being lazy with code and have it render fast on the screen.

And as expected, it worked great ever since! Vim and the newer rewrite NeoVim both have async APIs now so you shouldn’t encounter the issue I had anymore.

If you’re considering Vim, definitely give it a go. I still uses Vim as my sub editor for quick edits, because to be honest, it actually loads way faster than Emacs but I just don’t have the motivation to maintain 2 configurations.

The next cool kid? editor? around the block

I’ve been tracking the Xi editor for a while now. The design philosophy is something I like a lot, and it looks like it a Emacs/Vim rebuilt with modern technologies, also learning from past lessons.

Unfortunately, it is not ready for a day to day work level yet, so if you are interested in a performant text editor, give Emacs a try for now.

I personally think it’s worth the investment but I would like to know what you think about it too!