At first, I was just trying to deal with the pain handling Cron tasks and offline workers, and looking into existing solutions wasn’t very satisfying. Building a prototype was relatively easy since the concept itself was simple.

The foundation of the project was done in a couple days, a couple evenings after work and BOOM! 💥 MVP completed. After that, it was just adding a small feature here and there every week or two.

Mistakes 🈲

A few weeks later, after talking to a colleague which I don’t recall what the exact conversations were, I got excited and thought it was a good idea to launch it as a service and make it publicly available.

I know how to launch a service. What could possibly go wrong, right?

Allow me to share some of the mistakes I’ve made.

Pre-Optimization

After deciding to publicly launch the product, a ton of engineering work suddenly showed up.

• Sign up / Login
• Authentication
• Authorization
• Access Controls
• API designs

etc.

So far in my career, I’m lucky to have been able to work with and learn from engineers that would try to do things “the right way”. Which is extremely important when working with multiple engineering teams on an established product. However, that’s not the right approach for a product in an early stage.

My mind has been so biased towards the “right engineering approaches” that I’ve been ignoring what actually matters.

Just get it out of the door, talk to people and iterate

As a result, it cost me months instead of weeks before putting the product before people’s eyes, just because I wanted to do a fancy Typescript React Front-End with a Rails API and using Procotol Buffers for communication, because I don’t want to maintain the API schema. I over engineered the infrastructure as well and while it can scale well, it has become a financial burden because doing things the right way is not cheap.

To be honest, it did help catch a lot of issues when doing some major schema changes. Oh, and did I mentioned that my colleague actually tried to DDoS the service and it withhold? Pretty impressive right? But it doesn’t matter, it’s still solving the wrong problems at the stage it was in.

Underestimating the amount of work outside of engineering

Being an engineer and knowing engineers, we tend to think that building a good product will automatically get us customers and the rest will be history. While there are some portion of truth to it, having a good product will likely give you some advantages over your competitors, it’s definitely not going to take over the world just like that.

After making LivePing public, I started looking into landing pages, marketing, SEOs, in-corporation, accounting, etc..

There’s a lot of work besides just coding something out and throwing it on the internet, and I’ve seriously underestimated that. And critically enough, I’ve done it in the wrong order. Building a product, then market it was a complete waste of effort and time, especially when there’s not even a close to guarantee that the product will be what you think it would become.

Positive side, while I’ve never really thought that other jobs would be easier than mine (maybe a little bit :P), I’ve learned to really appreciate the work of the people in marketing, sales and all the other departments that operates and runs a company. Experiences like this makes you more humble and I think I understand why a lot of successful CEOs have a humble mindset now.

Not verifying the market size

Business is about people and solving their problems. Verifying the needs and pains your product solve is the first step which most people knows, clearly. However, another important portion that tends to get overlooked is the size of the market.

Sure, the product is solving a pain, I’ve conducted user interviews before deciding to make it an actual product, and there already are competitors in the field so that helps me verify that there are need for such a tool. But if the market is not big enough, it might not be worth the effort pursuing.

Trying to make $1000 in a $10,000 market vs a $1,000,000 market is a huge difference. Neither will be easy, but the effort required for the former could be way higher than the latter, resulting in the cost and effort of trying to grow the product becoming higher than the rewards.

In the past couple months, I’ve learned that the market LivePing is in are one of those markets that efforts can be greater than rewards, hence making it harder justifying the additional effort to be put into the service.

You might be thinking, “Clearly you’re not doing a good enough job of researching the market beforehand, otherwise you’ve known”.

While I do admit that I did ran with optimism, it’s also hard to know when and what is “good enough”. I’ve come to learn that, there’s never going to be a “good enough” because at some point, you’ll just have to run with it and learn while doing so. Knowing the border between research vs action is kind of an art, and it’s different for everyone. On top of that, knowing when to withdraw is even harder because there will always be some level of emotional attachment to your product. Hence, knowing those borders requires experience, and LivePing had taught me that in the hard way.

In hindsight, even though I knew all these principles from reading famous startup books, I still committed the same mistakes that people tell you not to. And because of the mistakes, I’ve wasted a lot of time and money which otherwise, I didn’t have to.

Another interesting aspect was that mistakes can have compounding effects on your motivation, and will put your passion to the test. Turns out, I don’t have as much passion as I thought I had for developer tooling after all and it was a precious insight that I would not have known unless having been put on the spot.

End of Life 🌆

LivePing will be shutting down at the end of February 2020.

Any active subscriptions will be cancelled and payments that extends beyond the end date will be refunded on a pro rata basis.

If you’re looking for alternatives for your workloads, here are some options available.

• Cron monitoring and uptime monitoring for websites and APIs - Cronitor
• Free Cron Job Monitoring - Healthchecks.io

I apologies for the inconvenience, and like to thank you for trying out LivePing. One of the positives that came out of this experience was being able to experience the joy of seeing someone else using something I’ve built from scratch, and receiving feedback for it. It’s fulfilling in a different kind of way compared to having a full time job working on an established product.

While there’s nothing specific on the pipeline for the next project, I’m likely not going to be working on developer tooling for business anymore, and will be looking at other industries where I’ve confirmed my passion.

Shutting Down LivePing was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

New

Documentation for LivePIng

We’ve added documentation regarding usage of LivePing and how to interact with alert integrations. These are frequently asked questions so we hope it helps clarify things better.

PagerDuty integration

We have implemented PagerDuty integration so you can get paged for Ping violations. The integration app has been reviewed and officially approved by PagerDuty. Here are the docs for integrating PagerDuty.

https://www.liveping.io/docs/integrations/pagerduty

PagerDuty integration is only available in paid plans.

Custom Webhook integration

LivePing can now send custom webhooks as alerts. If you have an endpoint that accepts webhooks, you can use the Custom Webhook integration for sending alerts to your system. This is useful if you have a tool that you’re using for checking alerts but it’s not officially supported by us yet.

We have populated the default payload when you try to create a webhook integration, but you can modify it to whatever you want as long as it’s valid JSON.

https://www.liveping.io/docs/integrations/webhook

Custom webhook are available for all plans including Free.

Grace Period for Alerting

Pings can now set a grace period before sending out alerts. This is particularly useful for heartbeat monitoring where intervals could vary and might not always check-in within the pre-defined interval. Alerting on a slightly late heartbeat could result in floppy alerts. Grace periods will help resolve that problem.

The way that Grace Period works is it will ignore up to the number of times for missed check-ins. In the example above, that number would be 5 and based on the interval being set to every 2 minutes, the grace period for this example would be 10 minutes. If the the workload continues to not check-in for 10 minutes straight, then alerts will start firing.

When a Ping is stepping into the grace period, you would be able to see it on the UI as a yellow bell icon.

That’s it for November updates. Happy holidays and hope you enjoy the last month of 2019!

See you in 2020!

Feel free to contact us for questions and inquiries at info@liveping.io.

We’re accepting feature requests!

November 2019 Updates was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

ソフトウェアの会社なら、必ず使ってると言っても過言ではない普遍的な素晴らしい技術です。しかし、実行中にエラーが発生した時はいったい何が起こるんでしょうか。。？

Silent Failure

ここ数ヶ月は、自分のためのサイドプロジェクトに取り掛かっていて、そしてできたものをしばらく使っていました。ある程度公開しても大丈夫な仕様になりましたので、ぜひ見てみてください。

LivePing

LivePingは一言で言うと死活監視のためのシステムです。ただ、死活監視と言っても大きくは、

• 外部ネットワークからチェックできるUptime Monitoring
• ネットワークからそもそもアクセスできない代わりに、自身がシステムを通知するHeartbeat Monitoring

の２種類に分かれます。LivePingは後者のHeartbeat Monitoring（直訳だと鼓動監視？）のシステムです。

さて、Cronってそこまで重要なものでしょうか？実は、TwilioのCEOが半冗談でツィートしたように、少なくともアメリカのソフトウェア業界を裏で支えてる重要な技術の一つです。しかし、どんな技術にも長所と短所があります。そしてCronも例外ではありません。利用したことのある人なら一度は考えたことがあるのではないでしょうか？

もし実行時に何か問題が発生した場合はどうやってそれを認知する？

Cronを筆頭に、オフラインで実行するソフトウェアは基本的に、全く音を立てずに、静かに障害を起こします。今頭に浮かぶだけでも、

• データベースのバックアップ失敗
• 外部向けのSSL/TLSの証明書の自動更新の失敗により、後日サービス全体へのアクセス障害が発生
• 再訓練された機械学習のモデルがデプロイされず、パーソナライズのサービスが日に日に不備を蓄積
• 社内やサービス間のTLS証明書の更新失敗により、サービス間の通信に障害が発生
• 毎月に顧客のために自動で生成されるレポートが生成されずに、クレームに繋がる

以上の例以外にも他に多くの問題を経験しました。

そして、畳み掛けるかのように、時には問題が発覚するのは数カ月先ということも珍しくありません。

Attempts in the past

当然問題を認知しているなら、それを解決しようとします。過去には何度か社内ツールとしてこの問題に対応してきました。そして、毎度同じように、メンテナンスの優先順位という課題にぶつかるのです。

監視システムである以上、稼働時間が短かったり、簡単に落ちていては意味がありませんが、片手間で簡単にできるものでもありません。少なくとも社内で唯一のSRE、あるいは数少ないSREの場合、障害対応以外にも対応しなければならないものや自動化させるものが他に大量にある中で、どうしても社内ツールの対応する優先順位は下がってしまうのです。

ならば社内ツールを担当するチームが存在するほどの規模の会社なら解決する問題なのか？それが面白いことに、意外とそうでもないことが多いのです。ツールのチームが担当する範囲のものは多岐に渡り、同じ社内ツールでも、エンジニアたちが直接影響されやすい、言語系フレームワークやCI/CDのような継続開発をサポートするツールに意識が行きやすく、Heartbeat Monitoringのシステムへの注意がどうしても後回しになってしまうことが多いです。

また、システムの担当や所有権も一つの問題になりえます。簡単に言えば、もし現在担当してるエンジニアが会社を去った場合、誰がシステムを引き継ぐのかという問題です。シリコンバレーは特に人材流動の激しい地域です。それによって多くの会社ではシステム構築者がその会社を去った後に、所有権が有耶無耶のシステムが数多くあり、誰も触ろうとせず、その上エンジニアの多くはドキュメンテーションを書きたがらないため、現実と認識のズレが発生し、知らない合間に会社のソフトウェアサービスが静かに危機的な状況に陥ることも珍しくありません。

そして最後に、「監視者を誰が監視する？」という問題も付いてきます。香ばしくなってきましたね？

じゃあ外部サービスを使えばいいじゃないか？と思うかもしれませんが、当然すでに試しました。最低限に必要な機能は確かに備えていますが、最低限ということは、つまり追加機能は結局自分が実装することになるので、別に大して仕事量が減ることもありませんでした。

Problems LivePing solves (for now)

現時点のLivePingはまず、僕個人の問題を解決するために作っています。一つは上記のような優先順位や会社を去った後の劣化の問題と、もう一つは自分が何度も何度も同じようなシステムを作る羽目になる問題です。

そして、これは問題ではないですが、僕が得意とするスケールとシステムの信頼性の一部を雇用関係になることなく、多くの人に提供できるかもしれません。

さて、ここで一旦僕個人の問題はさておき、LivePingはどのような問題が解決できるのか、少しお見せしましょう。

• サイレント障害の防止
• Cronが仕事をしていなかった場合のためのアラート
• データベースバックアップやTLS証明書の更新がちゃんと走ったかどうかのチェック

他に使える定時実行の例としては、

• データ同期のパイプライン
• データの集約やデータの廃棄
• システムの脆弱性のチェック

などがあります。当然、何かしら定時実行する必要なものであれば、基本的に対応できると思っていただいて大丈夫です。

現在のLivePingのアラートはEmailとSlackのみ対応していますが、これからどんどん機能と他のアラート先を実装していくつもりです。

短期のロードマップとしては、

• PagerDutyのアラート
• Webhookの対応
• アラートを飛ばす前の猶予期間
• 一時的なアラートのミュート
• APIアクセスとドキュメント
• OAuth2による認証

これ以外にもまだまだいっぱい機能を追加する予定です。

上記の機能の実装以外にもシステムの信頼性をあげるための自動化も全力で取り組んでおります。利用された際にはぜひフィードバックをいただけたら幸いです。また、希望する機能やアラート先がある場合はFeature requestを提出してください。各種のフォームや連絡情報はContactページに掲載しております。

簡単にお試しができるように、LivePingは無料で5つのPingを提供しております。有料プランもすべて14日の試用期間を設けております。詳細はPricingページを参照してください。

What’s next?

短期的には、まず僕が過去にぶつかった多くの問題を解決するための機能を追加していく予定です。

しかし、長期的には会社やチームとともに成長できるシステムにしていきたいです。最初はもしかしたら、ただひと握りのCronを監視するために。次第にはそれぞれのチームで使い分け、アクセス制御やダッシュボードの提供。そして究極的にはサービスとともに監視を自動化していく。

Monitoring（監視）は、あくまでObservability（可観測性）の一部に過ぎません。監視の中でもサービスの死活監視と、分散されたスケジュールシステムだと対応も違ってきます。故にPrometheusにもHeartbeat Monitoringのために、Push Gatewayという副次的な機能が備えているのです。

もしPrometheusをすでに利用されているのであれば、Heartbeat MonitoringもそのままPrometheusを利用し続けることをお勧めします。しかし、もし運用してない、あるいは社内でそのようなチームを作る余裕がない場合は、ぜひLivePingを一度試してみてください！

質問、または単に話してみたいだけでも、遠慮なく連絡をください！

最後まで読んでくれてありがとうございます！

Launching LivePing (日本語) was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cron is a simple but excellent piece of software that most software companies in the world rely on. But what happens when the job or script cron fails during execution…?

Silence failure

I have been working on a side project of mine and was using it for a little while. I have now made it available as a service, and it is accessible here.

LivePing

LivePing is a heartbeat monitoring tool that helps check workloads that don’t accept HTTP requests, like Cron and background workloads.

So, what’s the big deal about cron anyway? There’s a tweet that jokes about the importance of cron in the economy, but the impact it has is no joke. While cron is something a lot of companies rely on, there is also a problem that needs consideration when operating them.

How do you know when they are not doing their job?

Background workloads like cron tended to fail silently and had led to multiple issues in the past for me:

• Database backups were not running
• External facing SSL/TLS certificates did not get updated and caused a site outage
• Retrained machine learning models did not get deployed
• Internal services went down silently (related to internal TLS), creating a cascading effect on other services
• Monthly email reports for customers did not get generated and sent out

these were merely some of the numerous examples.

Certain times, they might not even cause any side effects and only found out that the workloads have not been working for some extended amount of time.

Attempts in the past

I’ve solved this problem a couple of times in the past as an internal tool, but it always comes back to the same maintenance priority.

Since it’s a monitoring system, it needs to have a decent amount of uptime, but guaranteeing that is typically not something you do single-handedly. At least, not if you’re the solo, or maybe one of the few members of a small SRE team when you have a thousand other things to handle and automate. Don’t mention unexpected incidents too.

Even with an internal tooling team at a larger company, that team is likely to be dealing with other projects, maybe language frameworks, CI/CD pipeline, and have other responsibilities. When trying to support more visible and direct tooling for other engineers, handling of a heartbeat monitoring system tends to get de-prioritized.

Another problem is ownership of the system. This is not a problem while you’re still with the company, but it emerges likely after you left.

Developers are not necessarily good writers, and there are always gaps between documentation and the running systems. I’m guilty of that too. So things get missed, and systems can reach critical states without being noticed after a certain amount of time.

And last but not least, you get into the “who’s going to watch the watchdog” problem. :)

So you might be thinking now, “I can see operating as an internal tool can be burdensome for people, then why don’t just use Cronitor or Dead Man’s Snitch?”

Yup, I’ve tried those, and yes, they do fulfill the bare minimum requirements I need. But they have their own set of problems, and I might touch-base on that in the future.

Problems LivePing solves (for now)

First and foremost, LivePing solves my problem of not needing to create another monitoring system that doesn’t age well. And the issue of doing roughly the same thing over and over again at different places. ;)

Also, maybe providing something that I’m good at, which is reliability and scalability to a broader audience.

Enough of my issues, let’s see what LivePing can do for you.

• Eliminating silent failures that can lead to disastrous situations
• Provides alerts before problems start to snowball
• Preventing the issues I’ve listed earlier like DB backups and TLS certificate updates, etc.

LivePing can also help you make sure these scheduled tasks are running and working as expected:

• Data synchronization pipelines
• Data aggregation, garbage collection
• System/vulnerability scans

While LivePing is functional at this point with Email and Slack as alerting targets, there are still a lot of lacking features and integrations.

Some of the stuff on the near term roadmap would be:

• PagerDuty alert integration
• Custom webhooks
• Grace period before alerting
• Temporarily silencing alerts
• API access and documentation
• OAuth2 authentication

and there are many more to come.

I’m working hard to improve and automate the system overall to provide a high SLA uptime and a reliable system and would appreciate your feedback using it. You can find other various forms like feedback and feature request on the contact page.

In effects to make it accessible, LivePing comes with 5 pings for free and doesn’t require credit card information to sign up. Any plans going forward will have a 14 day trial period. You can see the pricing details on the pricing page.

What’s next?

In the short term, I’m planning to be focusing on adding features to solve problems I’ve encountered in the past, including scalability issues.

But in the long term, I want to create a system that can grow with the team’s needs. Initially, maybe just simple cron monitoring for the small number of jobs the company has. Then, moving up to dividing up jobs between teams, access controls, and dashboarding. Eventually, to a system that is easy to integrate into the automation of services.

When it comes to the observability of systems, monitoring is just a subset of it. And monitoring uptime vs. a distributed scheduling system requires a different approach. That’s why even Prometheus has a push gateway.

If you’re familiar with Prometheus, which is an excellent piece of software, by all means, go with it. If not, or you don’t have the resources to set up such a team, give LivePing a try!

If you have any questions or simply just want to chat, please reach out!

Thanks for reading! :)

Launching LivePing was originally published in LivePing on Medium, where people are continuing the conversation by highlighting and responding to this story.