<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Healthchecks.io on Medium]]></title>
        <description><![CDATA[Stories by Healthchecks.io on Medium]]></description>
        <link>https://medium.com/@healthchecks?source=rss-d2288c51f7d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*m_ad1TBlJEKTn9n2sKZqgg.png</url>
            <title>Stories by Healthchecks.io on Medium</title>
            <link>https://medium.com/@healthchecks?source=rss-d2288c51f7d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:23:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@healthchecks/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running One-man SaaS, 9 Years In]]></title>
            <link>https://medium.com/@healthchecks/running-one-man-saas-9-years-in-8f22c8b46414?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f22c8b46414</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[indiedev]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[solopreneur]]></category>
            <category><![CDATA[mrr]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Mon, 29 Jul 2024 11:54:16 GMT</pubDate>
            <atom:updated>2024-08-08T08:37:23.118Z</atom:updated>
            <content:encoded><![CDATA[<p>Healthchecks.io launched in July 2015, which means this year we turn 9. Time flies!</p><p>Previous status updates:</p><ul><li>In 2018, <a href="https://blog.healthchecks.io/2018/08/my-one-person-saas-side-project-celebrates-its-third-birthday/">My One-person SaaS Side Project Celebrates its Third Birthday</a></li><li>In 2021, <a href="https://blog.healthchecks.io/2021/07/healthchecks-turns-6-status-update/">Healthchecks Turns 6, Status Update</a></li></ul><h3>Money</h3><p>Healthchecks.io currently has 652 paying customers, and the monthly recurring revenue is 14043 USD. MRR graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2YkXGycli5tYhFj4.png" /></figure><p>Side note: to minimize the number of data sub-processors, I am not using revenue analytics services. I used a script and a spreadsheet to make the MRR graph!</p><p>I’m happy to see MRR gradually go up, but I’m not optimizing for it. Healthchecks.io is sustainable as-is, and so I’m optimizing for enjoyment and life/work balance.</p><p>More stats (user count, check count, pings/day) are available on the <a href="https://healthchecks.io/about/">Healthchecks.io About page</a>.</p><h3>Still a one-man business?</h3><p>Yes, Healthchecks.io is still a one-man business. Until 2022, I was part-time contracting. Since January 2022 Healthchecks.io has been my only source of income, but I work on it part-time.</p><p>At least for the time being I’m not looking to expand the team. A large part of why I’m a “solopreneur” is because I do not want to manage or be managed. A cofounder or employee would mean regular meetings to discuss what’s done, and what’s to be done. It would be awesome to find someone who just magically does great work without needing any attention. Just brief monthly summaries of high-quality contributions, better than I could have done. But I don’t think I can find someone like that, and I also don’t think I could afford them.</p><h3>Growth Goals</h3><p>I’m not planning to tighten the limits of the free plans. I started Healthchecks in 2015 because I thought the existing services (Dead Man’s Snitch and Cronitor) were overpriced. I started with “I think this can be done better and cheaper”, and I’m sticking with it.</p><p>For the same reason, I’m also not planning to raise pricing for paid plans.</p><p>I’m choosing not to pursue enterprise customers who ask about PO billing, payments by wire transfer, custom agreements, and signing up to vendor portals. “But you are leaving money on the table!” — yes, it is a conscious decision. In my situation, the extra money will not make a meaningful difference, but the additional burden will make me more busy and grumpy.</p><p>Feature-wise, I am happy with the current scope and feature set of Healthchecks. I am <em>not</em> planning to expand the scope and add e.g. active uptime monitoring, hosted status pages, or APM features.</p><p>Healthchecks the product is <a href="https://hachyderm.io/@danderson/112766460393943288">hobbit software</a> and Healthchecks.io the business is a lifestyle business.</p><h3>Hosting Setup</h3><p>The hosting setup is mostly the same as in <a href="https://blog.healthchecks.io/2022/02/healthchecks-io-hosting-setup-2022-edition/">2022</a>. Just a few updates:</p><ul><li>Web servers upgraded to Hetzner’s AX42 (AMD 8700GE, 8 cores). On the old machines, saw a few nonsensical Python exceptions. A kernel update and a reboot didn’t fix it. Rather than messing with hardware troubleshooting, I upgraded to newer, faster, and more efficient machines.</li><li>Database servers upgraded to Hetzner’s EX101 (Intel 13900, 8+16 cores). I was setting up new database replicas after <a href="https://status.healthchecks.io/en/incidents/m7Qv7s8KCsVdMVjGvMbpJb/">an outage and failover event</a> and took the opportunity to upgrade hardware.</li><li>Healthchecks.io <a href="https://blog.healthchecks.io/2023/08/notes-on-self-hosted-transactional-email/">now sends its own email using maddy</a>.</li><li>Healthchecks.io <a href="https://blog.healthchecks.io/2022/04/we-moved-some-data-to-s3/">now stores ping body data in S3-compatible object storage</a>. This keeps the PostgreSQL database size down but adds reliance on an external service.</li></ul><p>That’s it for now, thank you for reading! Here’s to another 9 years, and in the closing here’s a complimentary picture of me trying to fit through pull-up bars, and my kids, Nora and Alberts, cheering:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sXeV8hbszOg28R8X.jpg" /></figure><p>Happy monitoring,<br>Pēteris,<br>Healthchecks.io</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2024/07/running-one-man-saas-9-years-in/"><em>https://blog.healthchecks.io</em></a><em> on July 29, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f22c8b46414" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Healthchecks Sends Signal Notifications]]></title>
            <link>https://medium.com/@healthchecks/how-healthchecks-sends-signal-notifications-d5720b7f02d4?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5720b7f02d4</guid>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[signal]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 09:33:21 GMT</pubDate>
            <atom:updated>2023-01-13T13:31:54.816Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Voc7LEk7qmz4iY9-.jpg" /></figure><p>When a cron job does not run on time, Healthchecks can notify you using various methods. One of the supported methods is Signal messages. <a href="https://signal.org/">Signal</a> is an end-to-end encrypted messenger app run by a non-profit Signal Foundation. Signal’s mobile client, desktop client, and server are free and open-source software (with some exceptions-read on!).</p><h4>No Incoming Webhooks</h4><p>Unlike most other instant messaging services, Signal does not provide incoming webhooks for posting messages in chats. If you want to send messages on the Signal network, you must run a full client, and follow all the same cryptographic protocols that normal end-user clients follow. This is inconvenient for the integration developer but makes sense: the main feature of Signal is strong cryptography and as little as possible sharing of information with the Signal servers. The servers pass around messages, and help with peer discovery, but (as far as I know) cannot send their own messages on the user’s behalf. Official incoming webhooks would conflict with the overall architecture of the system.</p><h4>signal-cli</h4><p><a href="https://github.com/AsamK/signal-cli">signal-cli</a> is a third-party open-source Signal client. It uses <a href="https://github.com/signalapp/libsignal">the same signal client libary</a> that the official clients use but offers a programmatic interface for sending and receiving messages. signal-cli supports command-line, <a href="https://github.com/AsamK/signal-cli/wiki/DBus-service">DBUS</a>, and <a href="https://github.com/AsamK/signal-cli/wiki/JSON-RPC-service">JSON-RPC</a> interfaces.</p><p>Signal’s official position on the signal-cli client seems to be-they do not support it, but they also have not explicitly banned it. When I asked Signal Support about their stance regarding signal-cli (and also about advice regarding rate-limit issues discussed below), I got just this short response back:</p><blockquote><em>Due to our limitations as a non-profit organization, we can only provide support for the product we provide. Signal-cli is not provided or maintained by us, therefore we cannot provide any support for it.</em></blockquote><h4>Using signal-cli in Healthchecks</h4><p>I coded the initial Signal integration in January 2021. To send messages, it was running a signal-cli send -m &#39;text goes here&#39; command for every message. Each send took a minimum of one second, as every signal-cli invocation was initializing JVM, and initializing network connections, just to do one small send operation. A more efficient approach was to run signal-cli in daemon mode and talk to it via DBUS or JSON-RPC.</p><p>Also in January 2021, I upgraded the integration to talk to signal-cli over DBUS. This took some tinkering to figure out the DBUS interface configuration and to get python code to talk to it. But it worked, and message delivery was now much quicker.</p><p>In December 2021, signal-cli added the JSON-RPC interface, and I switched the Healthchecks integration to it. Again, it took a fair bit of tinkering and <a href="https://github.com/AsamK/signal-cli/discussions/799">support from the signal-cli author</a> until I figured out how it all hangs together, how to read and write messages over a UNIX socket, and how to interpret them. There were two important improvements over the previous DBUS code:</p><ul><li>Simpler operations: I did not need the DBUS service with its associated configuration files anymore.</li><li>The Healthchecks project did not need the “dbus-python” dependency anymore.</li></ul><h4>Rate limiting and CAPTCHAs</h4><p>Around April 2022 I started to notice that some send operations were failing with an error message asking to solve a CAPTCHA challenge. These errors were infrequent at first and seemed to only affect the very first messages to new recipients. I added code to email me the CAPTCHA challenges, and I added a crude command-line utility to submit the CAPTCHA solutions. As the CAPTCHA challenges came in, I manually solved and submitted them. Signal was using Google reCAPTCHA, and I got plenty of opportunities to demonstrate my intelligence by expertly clicking on fire hydrants, crosswalks, and traffic lights. Sometimes at odd hours, sometimes roadside over a mobile hotspot.</p><p>As the frequency of CAPTCHAs gradually increased, I tried to make solving them less annoying:</p><ul><li>I figured out that being logged in gmail.com helps the CAPTCHA solving a lot. Usually just a single click, <em>no fire hydrants</em>.</li><li>I set up my computer to <a href="https://github.com/AsamK/signal-cli/issues/701#issuecomment-1293254365">automatically put the CAPTCHA solution in the clipboard</a>.</li><li>I made a web form for submitting CAPTCHA solutions. No need to fire up the terminal, just click a link in the email, and paste the solution.</li></ul><p>Now solving a CAPTCHA challenge took just a few clicks, but the end-user experience was still not great. For some users, Signal notifications would not work until I showed up and solved yet another CAPTCHA. I did some spelunking in the <a href="https://github.com/signalapp/Signal-Server">signal-server</a> code base. There is <a href="https://github.com/signalapp/Signal-Server/blob/main/service/src/main/java/org/whispersystems/textsecuregcm/configuration/RateLimitsConfiguration.java">a class</a> listing various rate-limiters and their parameters. For any rate limiter, I could trace back where and how it was used. But I still could not pinpoint the piece of code that triggers the specific rate-limit errors I was seeing. Signal-Server has an <a href="https://github.com/signalapp/Signal-Server/blob/main/.gitmodules">“abusive-message-filter” module</a>, which is private code, perhaps the logic lives there.</p><p>It seemed only the initial messages to new recipients were triggering rate-limit errors. After a single message got through, the following messages would work with no issues. So my next idea was to change the Signal integration onboarding flow:</p><ul><li>After the user has entered the phone number of the Signal recipient, ask them to send a test message</li><li>If the test message generates a rate-limit error, ask the user to initiate a conversation <em>with us from their side</em>, then try again:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6gKVEkpPuNMUmYyK.png" /></figure><p>My working theory is that users initiating the conversation with Healthchecks will look less abusive to Signal’s abusive message filter, and will help avoid hitting rate limits. But, if the theory fails and we still get rate-limit errors, at least the users will not create dysfunctional integrations (the “Save Integration” button becomes available only after successfully sending a test message).</p><h4>In Summary</h4><p>In summary, Healthchecks uses signal-cli to send Signal messages. It talks to signal-cli over JSON-RPC. To avoid rate limits, it asks the user to send the first message from their end. Building and maintaining the Signal integration has taken more effort than any other integration. But that is fine and, aside from the manual CAPTCHA solving, time well spent. I’m glad Healthchecks supports it, and I’m happy to see that the Signal integration is popular among Healthchecks.io users.</p><p>Happy monitoring and messaging,<br>Pēteris</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2023/01/how-healthchecks-sends-signal-notifications/"><em>https://blog.healthchecks.io</em></a><em> on January 13, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5720b7f02d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Schedule Cronjob for the First Monday of Every Month, the Funky Way]]></title>
            <link>https://medium.com/@healthchecks/schedule-cronjob-for-the-first-monday-of-every-month-the-funky-way-b86b52fb001b?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b86b52fb001b</guid>
            <category><![CDATA[unix]]></category>
            <category><![CDATA[tricks]]></category>
            <category><![CDATA[cron]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 15:19:24 GMT</pubDate>
            <atom:updated>2022-09-23T10:28:10.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dD5OOKFiIXtyd5ck.png" /></figure><p>The crontab man page (“man 5 crontab” or <a href="https://man7.org/linux/man-pages/man5/crontab.5.html">read online</a>) contains this bit:</p><blockquote><em>Note: The day of a command’s execution can be specified by two fields — day of month, and day of week. If both fields are restricted (i.e., don’t start with *), the command will be run when either field matches the current time. For example, </em><em>30 4 1,15 * 5 would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.</em></blockquote><p>What does it mean precisely? If you specify both the day of month and the day of week field, then cron will run the command when <em>either</em> of the fields match. In other words, there’s a logical OR relationship between the two fields. Let’s look at an example:</p><pre>0 0 1 * MON</pre><p>This expression translates to “Run at the midnight if it’s the first day of the month OR Monday”. An expression like this could be handy for a job that sends weekly and monthly reports as it probably needs to run at the start of every week, and the start of every month. However, if either field is unrestricted, the logical relationship between the fields changes to “AND”. For example:</p><pre>0 0 * * MON</pre><p>Here, the day of month field is unrestricted. Cron will run this command when both the day of month field AND the day of week fields match. Since * matches any day of month, this expression effectively translates to &quot;Run at midnight every Monday&quot;.</p><p>So far so good! The sometimes-OR relationship between the two fields is a relatively well-known cron gotcha. But let’s look closer at what values cron considers “unrestricted”. Star (*) is of course unrestricted, but, according to the man page, any value that starts with the star character is <em>also</em> unrestricted. For example, */2 is unrestricted too. Can we think of any useful schedules that exploit this fact? Yes, we can:</p><pre>0 0 1-7 * */7</pre><p>Here, the day of month is restricted to dates 1 to 7. Cron will interpret */7 in the day of week field as &quot;every 7 days starting from 0 (Sunday)&quot;, so, effectively, &quot;Every Sunday&quot;. Since the day of week field starts with *, cron will run the command on dates 1 to 7 which are <em>also</em> Sunday. In other words, this will match <strong>midnight of the first Sunday of every month</strong>.</p><p>In the above example, */7 is a trick to say &quot;every Sunday&quot; in a way that starts with the star. Unfortunately, this trick only works for Sunday. Can we make an expression that runs on, say, the first Monday of every month? Yes, we can!</p><pre>0 0 */100,1-7 * MON</pre><p>The day of month field here is */100,1-7, meaning &quot;every 100 days starting from date 1, and also on dates 1-7&quot;. Since there are no months with 100+ days, this again is a trick to say &quot;on dates 1 to 7&quot; but with a leading star. Because of the star, cron will run the command on dates 1 to 7 that are <em>also</em> Monday.</p><p>OK, but does any of this work? Is the man page accurate? Yes: you can check <a href="https://salsa.debian.org/debian/cron/-/blob/8836264d360843f3294a986fede165385d847827/entry.c#L183">the cron source here</a> and see how it initializes the DOW_STAR and DOM_STAR flags by testing just the first character of the fields. I&#39;ve also tested both expressions empirically by setting up dummy cron jobs and monitoring when they run. I ran them in a VM with an accelerated clock, <a href="https://blog.healthchecks.io/2021/10/how-debian-cron-handles-dst-transitions/#qemu-magic">which I&#39;ve used for experiments before</a>.</p><p>An important caveat before you use these tricks for scheduling your tasks: there are many systems that support cron-like syntax for scheduling tasks. It’s a safe bet not all of them implement all the quirks of the classic cron. Always check if your scheduler supports the syntax and logic you are planning to use. And always monitor if your scheduled tasks do run at the expected times (<a href="https://healthchecks.io/docs/monitoring_cron_jobs/">wink wink</a>)!</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2022/09/schedule-cron-job-the-funky-way/"><em>https://blog.healthchecks.io</em></a><em> on September 22, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b86b52fb001b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Send Email From Cron Jobs]]></title>
            <link>https://medium.com/@healthchecks/how-to-send-email-from-cron-jobs-326ba4365de6?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/326ba4365de6</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[unix]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[sysadmin]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Sun, 21 Nov 2021 09:47:02 GMT</pubDate>
            <atom:updated>2021-11-21T13:01:57.276Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s say you are writing a shell script for a systems housekeeping task. Perhaps the script is uploading backups to a remote server, or it is cleaning up old data, or it is making a measurement and submitting it to a somebody else’s HTTP API. What is the least cumbersome way for the script to contact you in case of problems? Let’s say you want to be contacted via email — how to make it work?</p><h3>Cron and MAILTO</h3><p>If you run the script from cron, you can look into using cron’s MAILTO= option. You put a MAILTO=you@example.org line in your crontab, and, when a job fails, cron will send a notification to the specified address using system&#39;s MTA. OK, then, what MTA to use?</p><h3>sSMTP</h3><p><a href="https://wiki.debian.org/sSMTP">sSMTP</a> is a send-only MTA which hands off messages to an external SMTP server that you configure in /etc/ssmtp/ssmtp.conf. For example, you can create a dedicated Gmail address for sending notifications, and use it in ssmtp.conf like so:</p><pre>Root=username@gmail.com <br>Mailhub=smtp.gmail.com:465 <br>AuthUser=username@gmail.com <br>AuthPass=gmail-password-or-app-password-here <br>UseTLS=YES</pre><p>Gmail-specific note: for the Gmail SMTP service to accept your credentials, you will need to enable “Less secure app access” in your new Google account’s settings, Security section. This is not ideal, and putting your password in ssmtp.conf is also not ideal. This is why I would strongly recommend to use a separate Gmail account, not your main account, for this.</p><p>With sSMTP installed and configured, sSMTP will pass messages on to Gmail, and Gmail will deliver them to your inbox. This will make cron’s MAILTO option <em>just work</em>, and you can then also send messages from shell scripts using the mailx program:</p><pre>echo &quot;Hello world&quot; | mailx -s &quot;Subject goes here&quot; you@example.org</pre><p>Similar sSMTP alternatives are <a href="https://wiki.debian.org/nullmailer">nullmailer</a> and <a href="https://wiki.debian.org/msmtp">msmtp</a>.</p><p>I used Gmail as example, but if you use a different email provider, it likely provides an SMTP interface as well. Transactional email services also typically provide an SMTP interface. For example, I’ve used sSMTP with Mailgun, and it works great. In short, sSMTP needs working SMTP credentials — it does not matter if they are from Fastmail, Zoho, AWS SES, Sendgrid, Mailjet or something else.</p><h3>HTTP API</h3><p>The transacional email services usually provide a HTTP API. Depending on the provider, the API can be so simple you can use it with a simple curl call. Here’s an example with the already mentioned Mailgun:</p><pre>curl --user &quot;api:your-api-key-here&quot; \<br> https://api.mailgun.net/v3/yourdomain.com/messages \<br> -F from=&#39;sender@yourdomain.com&#39; \<br> -F to=&#39;you@example.org&#39; \<br> -F subject=&#39;Subject goes here&#39; \<br> -F text=&#39;Message body goes here&#39;</pre><p>The upside of this approach is you don’t need to install and configure anything on your server beforehand (assuming curl is preinstalled). One downside is you cannot use this method with cron’s MAILTO, as curl is not a MTA. But you can use this method just fine from scripts.</p><p>HTTP API is also easy to use from a Python script:</p><pre>import os<br>import requests<br><br>def email(subject, text):<br>    url = &quot;https://api.mailgun.net/v3/yourdomain.com/messages&quot;<br>    auth = (&quot;api&quot;, os.getenv(&quot;MAILGUN_KEY&quot;))<br>    data = {<br>        &quot;from&quot;: &quot;sender@yourdomain.com&quot;,<br>        &quot;to&quot;: &quot;you@example.org&quot;,<br>        &quot;subject&quot;: subject,<br>        &quot;text&quot;: text,<br>    }<br><br>    requests.post(url, auth=auth, data=data)</pre><h3>Hosted Log Management Systems</h3><p>Here, I’m thinking of systems like <a href="https://www.logdna.com/">LogDNA</a> and <a href="https://www.papertrail.com/">Papertrail</a>. You configure your system’s syslog to ship system logs to your chosen log management system. In that system, you set up alerting rules like “alert me if this specific keyword appears in logs this many times in this long time window”. And that is all!</p><p>Logging to syslog from scripts is easy using the logger command:</p><pre>logger Hello World!</pre><p>As an example, here’s a notification I received from Papertrail a few days ago. It runs a saved search once per hour and sends a notification if the search produces any results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*IL6wFpKxHlXPvuf-.png" /></figure><h3>Cron Monitoring Services</h3><p>And here I am of course thinking of <a href="https://healthchecks.io/">Healthchecks.io</a>, but <a href="https://blog.healthchecks.io/2020/01/comparison-of-cron-monitoring-services-january-2020/">there are good alternatives too</a>. The monitoring service provides an unique URL that the housekeeping script must request regularly. When an HTTP request from the script does not arrive on time, Healthchecks.io detects that and alerts you. This takes care of scenarios, where, for example, the server has been shut down, and so is unable to contact you by itself. Healthchecks.io offers a lot more features, but this is the basic idea.</p><p>Here’s an example notification I received from Healthchecks.io. My home router is pinging Healthchecks.io. When home connection goes down, I get an alert:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/0*dN8b808nm2TjI5JF.png" /></figure><p>In summary:</p><ul><li>sSMTP is a good choice for use in combination with cron’s MAILTO option</li><li>HTTP APIs are handy for sending emails from scripts, without needing to install or configure any additional software</li><li>Log management systems and cron job monitoring systems can send email notifications, but they specialize in specific tasks. Log management systems can notify you about patterns in log files, cron job monitoring systems can detect irregularities in the heartbeat signals sent by your background tasks.</li></ul><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2021/11/how-to-send-email-from-cron-jobs/"><em>https://blog.healthchecks.io</em></a><em> on November 21, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=326ba4365de6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Feature: Slug URLs]]></title>
            <link>https://medium.com/@healthchecks/new-feature-slug-urls-6fc5855f86e5?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fc5855f86e5</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[announcements]]></category>
            <category><![CDATA[changelog]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 13:13:04 GMT</pubDate>
            <atom:updated>2021-09-29T13:38:27.461Z</atom:updated>
            <content:encoded><![CDATA[<p>Healthchecks.io pinging API has always been based on UUIDs. Each Check in the system has its own unique and immutable UUID. To send a success signal (“ping”), clients make a request to <a href="https://hc-ping.com/">https://hc-ping.com/</a> with check’s UUID added at the end:</p><pre><a href="https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9">https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9</a></pre><p>To signal a start, a failure, or a particular exit status, clients can add more bits after the UUID:</p><pre>https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9/start https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9/fail <a href="https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9/123">https://hc-ping.com/d1665499-e827-441f-bf13-8f15e8f4c0b9/123</a></pre><p>This is conceptually simple and has worked quite well. It requires no additional authentication. The UUID value <em>is</em> the authentication, and the UUID “address space” is so vast nobody is going to find valid ping URLs by random guessing any time soon.</p><p>Still, UUID-based ping URLs have downsides too.</p><p><strong>UUIDs are not particularly human-friendly.</strong> Unless you are good at memorizing UUIDs, it is not easy to associate a ping URL with a check just by looking at it. But it <em>is</em> easy to make mistakes when copy/pasting UUIDs around.</p><p><strong>Each UUID is a secret.</strong> Therefore, if you have many things to monitor, you must keep many secrets. Let’s consider a specific example: a web application that does various housekeeping tasks on a schedule. Each housekeeping task has a corresponding Check in Healthchecks.io and a ping URL. The web app stores its configuration, including ping URLs, in environment variables: FOO_TASK_URL, BAR_TASK_URL, and so on. This is all well and good. But, as the web app grows and adds new types of housekeeping tasks, the number of environment variables can get out of hand. In one specific project I&#39;m working on, there are already 15 environment variables for storing ping URLs, and there will likely be more. Wouldn&#39;t it be nice if there was a way to store just a single secret, and derive all ping URLs from it?</p><h3>Introducing: Slug URLs</h3><p>In slug URLs, we replace the UUID with two components, a <strong>ping key</strong> and a <strong>slug</strong>:</p><pre>https://hc-ping.com/&lt;ping-key&gt;/&lt;slug&gt;</pre><p>Here’s a concrete example:</p><pre><a href="https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups">https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups</a></pre><p>Slug URLs support start and failure signals the same way as UUID URLs do:</p><pre>https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups/start https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups/fail <a href="https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups/123">https://hc-ping.com/fqOOd6-F4MMNuCEnzTU01w/db-backups/123</a></pre><p>All checks in a single project share the same <strong>ping key</strong>. You can look up or generate the ping key in your project’s Settings screen, right next to your project’s API keys:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*FZvnL59nMnExfD_d.png" /></figure><p>Healthchecks.io derives <strong>slug</strong> from Check’s name using Django’s <a href="https://docs.djangoproject.com/en/3.2/ref/utils/#django.utils.text.slugify">slugify</a> function. The slugify function applies the following transformations:</p><ul><li>Converts to ASCII.</li><li>Converts to lowercase.</li><li>Removes characters that aren’t alphanumerics, underscores, hyphens, or whitespace.</li><li>Replaces any whitespace or repeated hyphens with single hyphens.</li><li>Removes leading and trailing whitespace, hyphens, and underscores.</li></ul><p>Here are a few specific examples of check names and the resulting slugs:</p><p>Name: “DB Backup”, slug: “db-backup”.<br>Name: “Backup /opt/some/path”, slug: “backup-optsomepath”.<br>Name: “server1 -&gt; server2”, slug: “server1-server2”.</p><p>Going back to the web app with housekeeping tasks example, with slug URLs the web app would need to store just one secret — the pinging key, and would be able to construct all ping URLs off that. Here is a rough example in Python:</p><pre>// foo_task.py <br>from os import getenv<br>import requests<br> <br>// ... do some work here ...<br><br>requests.get(&quot;https://hc-ping.com/%s/foo_task&quot; % getenv(&quot;PING_KEY&quot;))</pre><p>The ping URLs are also more human-friendly. The slug part helps you tell them apart.</p><h3>Q&amp;A</h3><p>Q: How can I use slug URLs in my Healthchecks.io project?<br>A: First, generate the ping key in your project’s Settings page. Next, click on “slug” on the Checks page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*TgExK05QmzuO45Cm.png" /></figure><p>Q: What if a check has no name?<br>A: The check will have no corresponding slug URL then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*edSbHbWsUwtSzWT-.png" /></figure><p>Q: What if multiple checks in the project have the same name?<br>A: They will also have the same slug. When you try to ping them, you will get an HTTP 409 response with the text “ambiguous slug” in the response body.</p><p>Q: Can I use UUID and slug URLs at the same time?<br>A: Yes, you can use UUID URLs and slug URLs interchangeably.</p><p>Q: Then what does the uuid / slug selector do, exactly?<br>A: It selects which URL format is used <em>for display</em> in the list and details views on Healthchecks.io.</p><h3>Current Status</h3><p>Slug URLs are implemented and ready for use on Healthchecks.io. This is a brand new feature and it will likely receive refinements over time. If you notice problems when using slug URLs, or want to suggest improvements, please send your feedback to contact@healthchecks.io. I will appreciate it!</p><p>-Pēteris</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2021/09/new-feature-slug-urls/"><em>https://blog.healthchecks.io</em></a><em> on September 29, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fc5855f86e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Healthchecks Turns 6, Status Update]]></title>
            <link>https://medium.com/@healthchecks/healthchecks-turns-6-status-update-4d6da412aa15?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d6da412aa15</guid>
            <category><![CDATA[bootstrapping]]></category>
            <category><![CDATA[status-report]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[building-in-public]]></category>
            <category><![CDATA[indiedev]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 07:16:13 GMT</pubDate>
            <atom:updated>2021-08-17T12:37:17.030Z</atom:updated>
            <content:encoded><![CDATA[<p>Time flies and Healthchecks.io is already 6 years old. Here’s a quick review of notable recent events and the project’s current state.</p><h3>Database Migration</h3><p>Healthchecks.io database used to run PostgreSQL 10. In March 2021 I migrated it to PostgreSQL 13. For the upgrade method, I used logical replication, <a href="https://www.reddit.com/r/PostgreSQL/comments/ll1hsh/upgrading_postgresql_from_v10_to_v13_using_pg/">as suggested on Reddit</a>.</p><p>The idea is to set up a Postgres 13 replica, replicate the data to it, and then failover to it. But there are of course several gotchas and everything has to be thoroughly tested before. I found <a href="https://www.cybertec-postgresql.com/en/upgrading-postgres-major-versions-using-logical-replication/">this guide</a> and worked through it. I made a step-by-step migration plan and tested it on Vagrant VMs. I then iteratively improved the plan and did more test migrations until everything was working smoothly, and I knew the order of commands to run almost by heart.</p><p>Then it was time to announce maintenance, provision new hardware (two Ryzen 5950X machines: 16 cores, 64GB RAM, and 2x4TB NVMe drives for each, <em>aw yiss</em>), set them up, and do the migration for real. And it all worked as planned!</p><h3>Wireguard</h3><p>Hetzner has a feature called <a href="https://docs.hetzner.com/robot/dedicated-server/network/vswitch/">vSwitch</a> for setting up private networks between hosts. I had it set up, and the infrastructure servers (load balancers, app servers, databases) were communicating between themselves over internal IPs.</p><p>In my experience, vSwitch turned out to be less reliable than the regular network. There was an incident where the vSwitch network interface on one machine was not working while the public interface was still fine. The issue got resolved after contacting Hetzner support, but I decided to go back to using public interfaces. I used firewall rules to control which IPs can connect to which ports.</p><p>Although Hetzner support says their internal network is secure, and customers cannot snoop on other customer traffic, I wanted to reduce the trust placed on Hetzner, and set up Wireguard tunnels between the servers. I did not use <a href="https://tailscale.com/">Tailscale</a> or anything fancy like that, just a few Fabric recipes for initial setup, and for updating peers (when a server is added or removed from the network).</p><p>A small gotcha here was services not always automatically starting after system reboot. I had to tweak systemd service definitions to make sure network-dependent services (nginx, postgres) start only after Wireguard has initialized.</p><p>Self-hosted Postgres, bespoke Wireguard tunnels, can you hear <a href="https://mcfunley.com/choose-boring-technology">the innovation tokens</a> burning up yet? 🙂</p><h3>Signal</h3><p>Healthchecks.io has had a <a href="https://signal.org/">Signal</a> integration for a couple months now. I think Signal has been the most tricky to implement and set up so far. Unlike most other services, Signal does not have a public HTTP API you can call to send messages. Instead, you have to run a local Signal client locally and communicate with it to send the messages. Luckily there is <a href="https://github.com/AsamK/signal-cli">signal-cli</a>, a wrapper around the official Signal Java client library. I run signal-cli under a separate OS user account, and Healthchecks communicates with it over DBus ( <a href="https://github.com/AsamK/signal-cli/wiki/DBus-service">details</a>). Multiple app servers are sending out notifications, each one runs signal-cli, and all signal-cli instances are linked to a single Signal account (phone number).</p><p>After deploying and announcing the Signal integration, I was glad to see a quick uptake:</p><ul><li>SMS was introduced in July 2017, and has approx. 500 configured integrations</li><li>WhatsApp was introduced in July 2019, and has approx. 450 configured integrations</li><li>Signal was introduced in January 2021, and has approx. 350 configured integrations</li></ul><p>When looking at these numbers, one factor to keep in mind is that SMS and WhatsApp have a minimal sending quota in free accounts (because sending these notifications costs money), while Signal is unrestricted.</p><h3>Dark Mode</h3><p>Healthchecks now has an optional dark theme. You can activate it in Account Settings — Appearance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ImdY0TVm7FjmsZU.png" /></figure><p>Implementing dark mode was, as expected, lots of work, and there is more work left. Aside from the obvious — page background, body text, panels, buttons — various other bits needed theming, each in their specific way:</p><ul><li>Bootstrap components like menus</li><li>Selectize dropdowns</li><li>Period and Grace sliders</li><li>The icon font with integration logos</li><li>Syntax highlighting for code samples</li></ul><p>It was interesting work. I use Sublime Text, and found the <a href="https://packagecontrol.io/packages/Color%20Highlighter">Color Highlighter</a> plugin very handy when working with colors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LzaeP0I0g4XbORES.png" /></figure><p>After publishing the initial dark mode implementation, I was happy to see people starting to use it. It was not work-for-nothing, a significant number of users prefer the dark mode over the default!</p><h3>Fuzz Testing croniter, Introducing cronsim</h3><p>Healthchecks.io had <a href="https://blog.healthchecks.io/2020/02/incident-report-7-february-2020/">an incident</a> where a single bad cron expression caused system-wide issues. The bad cron expression was making <a href="https://github.com/taichino/croniter">the croniter library</a> throw an unexpected exception. This lead to a crash-restart loop in the notification sending process. The initial fix was to add “try .. except” around croniter calls, but I later also spent time fuzz testing croniter. I found and filed several crashing issues. The worst one was to do with expressions like: 0-1000000000 * * * *. By varying the number of zeroes I could get the python process to use up all system memory and eventually crash. I reported this issue privately in January 2021, and the maintainer fixed it the same day.</p><p>After diving around the croniter code, I wanted to try my hand at writing a slimmed-down version. And so I did, welcome <a href="https://github.com/cuu508/cronsim">cronsim</a>. It is 250 lines of code, and it does just one thing: it takes a cron expression and returns a datetime iterator.</p><p>I’ve tested cronsim with a large corpus of cron expressions, and, for every expression I tested, it produced the same results as the croniter library. Except for one case, where both libraries produce incorrect results: the handling of daylight saving time (DST) transitions. Getting this right has been surprisingly hard, and I have not cracked this problem yet. But I did come up with a cool toy: I installed a Debian system inside qemu ( <a href="https://pqsec.org/2021/01/05/intall-debian-ubuntu-in-qemu.html">instructions</a>) and used qemu emulator flags to speed up the system clock inside the VM. With this contraption, I can test cron expressions with the actual running Debian cron daemon, and see results in minutes instead of hours or days. Anyway, more work is needed here.</p><h3>Development Roadmap</h3><p>The default plan is to continue making small iterative improvements.</p><p>In the background, I am also bouncing around ideas around product architecture and reliability. One area is the reliability of the Ping API. Whenever a client makes an HTTP request to a ping endpoint, there is a small, but non-zero probability the request will fail due to TCP packet loss. The probability increases as the distance from the client to the server increases. It would be ideal to put the server close to the client. There are different ways to go about this and lots to explore. One potential building block is <a href="https://github.com/cockroachdb/cockroach">CockroachDB</a>. Very impressively, in my testing Healthchecks test suite passed with CockroachDB backend out of the box. It <em>Just Worked</em>, but to make it perform well, I would need to make several changes. For example, the big and write-heavy “api_ping” table has an auto-incrementing integer primary key. It would not work well in a distributed database.</p><h3>Healthchecks.io the Business</h3><p>I’ve reduced my other work commitments, and Healthchecks.io is now my main occupation and my main source of income. Not quite “full-time” yet, but getting there!</p><p>I regularly update the <a href="https://healthchecks.io/about/">About page</a> with running stats (ping volume, the number of users, revenue, …), you can check out the numbers there!</p><p>As the project’s revenue slowly creeps up, I start to get more regular “Acquisition?” emails. I don’t have plans to sell the project in the foreseeable future. Too much work and soul put into it, and I also simply enjoy working on it and running it (aside from dealing with infra outages I have no control over, these are not fun at all!).</p><p>That’s it for now, thank you for reading! Here’s to another 6 years, and in the closing here’s a complimentary picture of me attacking a hornet nest with a pressure washer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*jVNoYR2OqG8J3HBy.jpg" /></figure><p>Happy monitoring,<br>Pēteris,<br>Healthchecks.io</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2021/07/healthchecks-turns-6-status-update/"><em>https://blog.healthchecks.io</em></a><em> on July 15, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d6da412aa15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fighting Packet Loss with Curl]]></title>
            <link>https://medium.com/@healthchecks/fighting-packet-loss-with-curl-51030bb015e6?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/51030bb015e6</guid>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[curl]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Fri, 03 Jan 2020 14:22:52 GMT</pubDate>
            <atom:updated>2020-01-13T09:01:32.241Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u4iK5D9UAEnPPS1_SWfEmg.jpeg" /></figure><p>One class of support requests I get at <a href="https://healthchecks.io/">Healthchecks.io</a> is about occasional failed HTTP requests to ping endpoints (hc-ping.com and hchk.io). Following an investigation, the conclusion often is that the failed requests are caused by a packet loss somewhere along the path from the client to the server. The problem starts and ends seemingly at random, presumably as network operators fix failing equipment or change the routing rules. This is mostly opaque to the end users on both ends: you send packets into a “black hole” and they come out at the other end — and sometimes they don’t.</p><p>One way to measure packet loss is using the mtr utility:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*Yxnk0b0AAeOab0Kc-c3ZYw.png" /></figure><p>The command line parameters used:</p><ul><li>-w Puts mtr into wide report mode. When in this mode, mtr will not cut hostnames in the report.</li><li>-c 1000 The number of pings sent to determine both the machines on the network and the reliability of those machines. Each cycle lasts one second.</li><li>-s 1000 The packet size used for probing. It is in bytes, inclusive IP and ICMP headers.</li><li>-r Report mode. mtr will run for the number of cycles specified by the -c option, and then print statistics and exit.</li></ul><p>The last parameter is the IP address to probe. You can also put a hostname (e.g. hc-ping.com) there. The above run shows a 5.2% packet loss from the host to one of the IPv6 addresses used by Healthchecks.io ping endpoints. That’s above what I would consider “normal”, and will sometimes cause latency spikes when making HTTP requests, but the requests will still usually succeed.</p><p>Packet loss cannot be completely eliminated: there are always going to be equipment failures and human errors. Some packet loss is also allowed by IP protocol’s design: when a router or network segment is congested, it is <em>expected</em> to drop packets.</p><p>I’ve been experimenting with curl parameters to make it more resilient to packet loss. I learned that with enough brute force, curl can get a request through fairly reliably even at 80% packet loss levels. The extra parameters I’m testing below <em>should not be needed</em>, and in an ideal world the HTTP requests would <em>just work</em>. But sometimes they don’t.</p><p>For my testing I used iptables to simulate packet loss. For example, this incantation sets up 50% packet loss:</p><pre>iptables -A INPUT -m statistic --mode random --probability 0.5 -j DROP</pre><p>Be careful when adding rules like this one over SSH: you may lose access to the remote machine. If you do add the rule, you will probably want to remove it later:</p><pre>iptables -D INPUT -m statistic --mode random --probability 0.5 -j DROP</pre><p>I made a quick bash script to run curl in a loop and count failures:</p><pre>errors=0<br>start=`date +%s`<br><br>for i in {1..20}<br>do<br>    echo -e &quot;\nAttempt $i\n&quot;<br>    # This is the command we are testing:<br>    curl --retry 3 --max-time 30 https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19<br>    if [ $? -ne 0 ]; then<br>        errors=$((errors+1))<br>    fi<br>done<br><br>end=`date +%s`<br>echo -e &quot;\nDone! Attempts: $i, errors: $errors, ok: $(($i - $errors))&quot;<br>echo -e &quot;Total Time: $((end-start))&quot;</pre><p>For the baseline, I used the “-retry 3” and “-max-time 30” parameters: curl will retry transient errors up to 3 times, and each attempt is capped to 30 seconds. Without the 30 second limit, curl could sit for hours waiting for missing packets.</p><p>Baseline results with <strong>no packet loss</strong>:</p><ul><li>👍 Successful requests: 20</li><li>💩 Failed requests: 0</li><li>⏱️ Total time: 4 seconds</li></ul><p>Baseline results with <strong>50% packet loss</strong>:</p><ul><li>👍 Successful request: 20</li><li>💩 Failed requests: 0</li><li>⏱️ Total time: 2 min 4 s</li></ul><p>Baseline results with <strong>80% packet loss</strong>:</p><ul><li>👍 Successful requests: 13</li><li>💩 Failed requests: 7</li><li>⏱️ Total time: 17 min 43 s</li></ul><p>Next, I increased the number of retries to 20, and reduced the time-per-request to 5 seconds. The idea is to fail quickly and try again, rinse and repeat:</p><pre>curl --retry 20 -m 5 <a href="https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19">https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19</a></pre><p>When using the -retry parameter, curl delays the retries using an exponential backoff algorithm: 1 second, 2 seconds, 4 seconds, 8 seconds, … This test was going to take hours so I added an explicit fixed delay:</p><pre>curl --retry 20 -m 5 --retry-delay 1 <a href="https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19">https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19</a></pre><p><strong>Fast retries with 1 second retry delay</strong> and 80% packet loss:</p><ul><li>👍 Successful requests: 15</li><li>💩 Failed requests: 5</li><li>⏱️ Total time: 18 min 18 s</li></ul><p>Of the 5 errors, in 3 cases curl simply ran out of retries, and in 2 cases it aborted with the “Error in the HTTP2 framing layer” error. So I tried HTTP/1.0 instead. To make the results more statistically significant, I also increased the number of runs to 100:</p><pre>curl -0 --retry 20 -m 5 --retry-delay 1 <a href="https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19">https://hc-ping.com/6e1fbf8f-c17e-4749-af44-0c81461bdd19</a></pre><p>Fast retries <strong>over HTTP/1.0</strong> with 80% packet loss:</p><ul><li>👍 Successful requests: 98</li><li>💩 Failed requests: 2</li><li>⏱️ Total time; 51 min 3 s</li></ul><p>For a good measure, I ran the baseline version again, now with 100 iterations. Baseline results:</p><ul><li>👍 Successful requests: 75</li><li>💩 Failed requests: 25</li><li>⏱️ Total time: 60 min 22 s</li></ul><p>Summary: in a simulated 80% packet loss environment, the “retry early, retry often” strategy clearly beats the default strategy. It would likely reach 100% success rate if I increased the number of retries some more.</p><p>Forcing HTTP/1.0 prevents curl from aborting prematurely when it hits the “Error in the HTTP2 framing layer” error.</p><p>Going from HTTPS to plain HTTP would likely also help a lot because of the reduced number of required round-trips per request. But trading privacy for potentially more reliability is a questionable trade-off.</p><p>From my experience, IPv6 communications over today’s internet are more prone to intermittent packet loss than IPv4. If you have the option to use either, you can pass the “-4” flag to curl and it will use IPv4. This might be a pragmatic choice in short term, but we should also keep pestering ISPs to improve their IPv6 reliability.</p><p>If you experience failed HTTP requests to Healthchecks.io, and fixing the root cause is outside of your control, adding the above retry parameters to your curl calls can help as a mitigation. Also, curl is awesome.</p><p>Happy curl’ing,<br>-Pēteris, <a href="https://healthchecks.io/">Healthchecks.io</a></p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2020/01/fighting-packet-loss-with-curl/"><em>https://blog.healthchecks.io</em></a><em> on January 3, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51030bb015e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Look at Healthchecks.io Hosting Setup, Summer 2019]]></title>
            <link>https://medium.com/@healthchecks/a-look-at-healthchecks-io-hosting-setup-summer-2019-3299e787dbdf?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/3299e787dbdf</guid>
            <category><![CDATA[indiedev]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[hetzner]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Sun, 25 Aug 2019 13:16:57 GMT</pubDate>
            <atom:updated>2019-08-25T13:16:57.384Z</atom:updated>
            <content:encoded><![CDATA[<p>For a monitoring service, uptime and reliability is of course a critical feature: customers are placing trust in the service to detect problems and deliver timely and accurate alerts. While I cannot guarantee that <a href="https://healthchecks.io/">Healthchecks.io</a> will absolutely never let you down, I can offer transparency on how it is currently being hosted and operated.</p><p>I will use bullet point lists liberally in this blog post (otherwise, it could turn into “The Opinionated and Frugal SRE Book”). Let’s go:</p><ul><li>Main values: Simple is good. Efficient is good. Less is more.</li><li>The core infrastructure runs on <a href="https://www.hetzner.de/">Hetzner</a> bare metal machines. Hetzner offers amazing value for money and is a big part of the reason why Healthchecks.io can offer its current pricing.</li><li>No containers, no auto-scaling, no “serverless”. Plain old servers, each dedicated to a single role: “load balancer”, “application server” and “database server”.</li><li>The machines are closer to “pets” than “cattle”: I have provisioning scripts to set up new ones relatively quickly, but in practice the working set of machines changes rarely. For example, the primary database server currently has an uptime of 375 days.</li></ul><h3>Load balancers</h3><ul><li><strong>Hardware:</strong> Two Intel 9900K machines, costing €69/month each.</li><li><strong>Software:</strong> HAProxy.</li><li><strong>Fault tolerance:</strong> A watchdog script on an external VM, detects unhealthy load balancers and removes their DNS records.</li><li>Supports HTTP/2. Supports IPv6. Uses ECDSA certificates with modern clients and falls back to RSA certificates with old clients.</li><li>Used <a href="https://loader.io/">Loader.io</a> to test performance-related configuration changes.</li></ul><p>Before going HAProxy, I used Cloudflare Load Balancer. Compared to Cloudflare,</p><ul><li>The monthly running cost ends up being slightly lower but similar.</li><li>It took more work to research, set up, and maintain the dedicated load balancer servers.</li><li>HTTP request latencies seen from clients are much more consistent. Cloudflare was great on some days, and not so great on others.</li><li>More control over the load balancer configuration, and more options to diagnose problems.</li></ul><h3>Application Servers</h3><ul><li><strong>Hardware:</strong> Three Intel 6700/7700 machines, costing €39/month each.</li><li><strong>Software:</strong> Nginx -&gt; uWSGI -&gt; Django.</li><li><strong>Fault tolerance:</strong> load balancers detect unhealthy or “in maintenance mode” machines and remove them from rotation.</li><li>For daemon processes that need to be constantly running (“sendalerts”, “sendreports”) I use systemd services.</li><li>Communication with load balancers is over private network (<a href="https://wiki.hetzner.de/index.php/Vswitch/en">Hetzner’s vSwitch</a>).</li></ul><p>Side note: when I started out, I took the private network idea to the extreme: I asked Hetzner support to install two NICs in each machine, and to connect the secondary NICs using a dedicated switch. A custom order like that takes more time (days instead of minutes) and costs more, but is possible. I later moved away from that setup, for a couple of reasons:</p><ul><li>All servers had to be in the same rack. Cannot add more machines if the rack is full.</li><li>Also, with this setup, one critical switch failure could take out all of the servers.</li></ul><h3>Database</h3><ul><li><strong>Hardware:</strong> Two Intel Xeon E3–1275 machines, ECC RAM, NVMe storage, costing €59/month each.</li><li><strong>Software:</strong> PostgreSQL 10.</li><li>Primary/hot standby setup.</li><li><strong>Fault tolerance:</strong> I am not brave enough to completely automate database failover — I have a tested procedure to perform the failover, but the decision has to be made manually.</li><li>Application servers specify <a href="https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/">multiple hosts in the database connection strings</a>. When primary changes, the applications can fail over to the new primary without any additional configuration changes.</li><li><strong>Backups:</strong> Full daily backups encrypted with GPG and uploaded to S3. I keep 2 months worth of backups and delete older ones.</li></ul><h3>Monitoring</h3><ul><li><a href="https://www.opsdash.com/">OpsDash</a> agent and hosted dashboard for a “big picture” view of the servers, and alerting.</li><li><a href="https://github.com/netdata/netdata">Netdata</a> agent on each machine for investigating specific issues.</li><li>Four VMs in different locations sending regular pings, and logging everything, including TCP packet captures.</li><li>Logs from the VMs aggregated to <a href="https://papertrailapp.com/">Papertrail</a>. Papertrail sends alerts on specific log events.</li><li>An always-on laptop dedicated to showing live Papertrail logs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yY2kw0vQnck6v5I-.jpg" /><figcaption>My workplace, Papertrail logs in the top right screen</figcaption></figure><h3>Ops</h3><ul><li>Yubikey for signing commits, logging into servers and decrypting database backups. For initial setup, I used <a href="https://github.com/drduh/YubiKey-Guide">DrDuh’s guide</a>.</li><li>A small laptop with development and deployment environments, and another Yubikey. Travels with me when I leave home.</li><li><a href="http://docs.fabfile.org/en/1.13/">Fabric</a> scripts for server provisioning, updates and maintenance.</li><li>Operated by one person. I’m not in front of my PC 100% of the time, so incidents <em>can</em> take time to fix.</li></ul><h3>Custom Bits</h3><ul><li>I maintain a private branch of the Healthchecks project with non-public customizations: branding, customized footer template, the “About”, “FAQ”, “Privacy Policy” pages and similar.</li><li>For serving the ping endpoints (hc-ping.com, hchk.io) efficiently, I wrote a small Go application, which I have not open sourced.</li></ul><p>Last but not least, I am eating my own dog food and am monitoring many of the periodic and maintenance tasks using Healthchecks.io itself. For example, if the load balancer watchdog silently fails, I might not notice until much later when it fails to do its duty. With Healthchecks.io monitoring, if the watchdog runs into issues (the script crashes, VM loses network connectivity, anything) I receive an email and an SMS alert within minutes.</p><p>Thanks for reading,<br>- Pēteris, Healthchecks.io</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2019/08/a-look-at-healthchecks-io-hosting-setup-summer-2019/"><em>https://blog.healthchecks.io</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3299e787dbdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I monitor the Ingress Sojourner Medal using Healthchecks.io]]></title>
            <link>https://medium.com/@healthchecks/how-i-monitor-the-ingress-sojourner-medal-using-healthchecks-io-79a07dd7f5e3?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/79a07dd7f5e3</guid>
            <category><![CDATA[life-hacking]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[ingress]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Thu, 04 Apr 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-04-04T19:22:43.179Z</atom:updated>
            <content:encoded><![CDATA[<p>The primary intended use, in the case of <a href="https://healthchecks.io/">Healthchecks.io</a>, is to monitor the regularly running tasks on servers, such as cron jobs. However, the “alert me if X doesn’t happen on time” functionality can be useful in many other contexts too. One of the quirkier ways I’ve been personally using Healthchecks.io is to help my progress towards the Sojourner medal in Ingress.</p><p><a href="https://www.ingress.com/">Ingress</a> is a location-based, augmented-reality game. It is a predecessor of Pokemon GO, and is made by the same company. In Ingress, players visit physical points of interest (“portals”) and perform various in-game actions on them. One of the gamification features in Ingress is Medals. Medals are awarded to players as they reach set milestones. For example, the “Trekker” medal is awarded for walking a long distance while playing Ingress:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*WnP_xGea-48H5XWN.png" /><figcaption>Bronze: 10 km, Silver: 100 km, Gold: 300 km, Platinum: 1000 km, Black: 2500 km</figcaption></figure><p>Another medal in Ingress is Sojourner. For the Sojourner medal, the player must visit and “hack” portals in consecutive 24 hour periods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*u1UBCOHFvrecuXMO.png" /><figcaption>Bronze: 15-day hacking streak, Silver: 30-day hacking streak, Gold: 60-day hacking streak, Platinum: 180-day hacking streak, Black: 360-day hacking streak</figcaption></figure><p>The black Sojourner medal is notoriously hard to obtain. If the player misses their 24 hour window, the current hacking streak resets to zero, and they have to start from scratch. The player must keep track of “When was my last hack? How much time do I have left?” day in day out. My own highest hacking streak is 233 days. Everything was going well until one day I forgot to go outside and do my “daily hack” on time. And just like that, I was back to square one!</p><p>After a period of “I’m never doing this again!”, I started another attempt. This time I was also using Healthchecks.io to keep track of how much time I have left each day and to remind me when the time is about to run out. Here’s how this works: I’ve set up a check on Healthchecks.io with a period of 20 hours:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/0*0nHzRhUy7j0W501i.png" /></figure><p>For notifications, I’m using <a href="https://pushover.net/">Pushover</a> with the “emergency” priority: if the “Daily Hack” check doesn’t get pinged at least every 20 hours, my phone will get a notification. The “emergency” priority causes the notification to be repeated every 5 minutes until it is acknowledged. The emergency notifications also ignore the “Do Not Disturb” mode on the phone.</p><p>I will not be reverse engineering the Ingress mobile app, so I also need a convenient way to manually send HTTP requests to the “Daily Hack” check when I’m returning home from playing. I’ve found the <a href="https://play.google.com/store/apps/details?id=ch.rmy.android.http_shortcuts&amp;hl=en_US">HTTP Shortcuts</a> Android app and its home screen widget useful for that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SFK-TLKbnUDoaRVk.jpg" /><figcaption>Tap on “Daily Hack” and the HTTP Shortcuts app fires off an HTTP request to Healthchecks.io</figcaption></figure><p>I also bought an <a href="https://aws.amazon.com/iotbutton/">AWS IoT button</a> and stuck it on the inside of my apartment’s front door. When passing the door, I can press the button, which triggers a Lambda function, which then pings Healthchecks.io. <a href="https://twitter.com/healthchecks_io/status/1073545947622330369">Here is a quick demonstration.</a> In practice, I am using the phone’s home screen widget most of the time, but this was a good excuse to play around with the AWS IoT button.</p><p>And that is all! Whenever 20 hours pass without a ping to Healthchecks.io, my phone then sounds notifications every 5 minutes. At this point, I still have 4 hours left to go out and hack a portal. I don’t silence the notifications until the hack is done–just to be safe and stay “motivated” by the annoying notifications. This setup has been working well for me so far. My current hacking streak is 201 days and counting. Fingers crossed!</p><p>Do you also have found an unconventional or creative use for Healthchecks.io? Please let me know via <a href="https://twitter.com/healthchecks_io">Twitter</a> or <a href="/service/mailto:contact@healthchecks.io">email</a>!</p><p>Pēteris,<br>Healthchecks.io</p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2019/04/how-i-monitor-the-ingress-sojourner-medal-using-healthchecks-io/"><em>https://blog.healthchecks.io/2019/04/how-i-monitor-the-ingress-sojourner-medal-using-healthchecks-io/</em></a><em> on April 4, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79a07dd7f5e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Projects on Healthchecks.io]]></title>
            <link>https://medium.com/@healthchecks/introducing-projects-on-healthchecks-io-9243c663a0b4?source=rss-d2288c51f7d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9243c663a0b4</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Healthchecks.io]]></dc:creator>
            <pubDate>Mon, 04 Feb 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-02-04T11:53:01.392Z</atom:updated>
            <content:encoded><![CDATA[<p>If your Healthchecks account is growing and is getting a little hard to manage, <a href="https://healthchecks.io/">Healthchecks.io</a> has a new feature for you: Projects. Use Projects to organize your monitoring dashboards, to have finer-grained team and API access controls, and to simplify your check-integration mapping.</p><p>You start will all your existing checks in your default project. You can create more projects and move your existing checks between them. There is no limit to the number of projects you can have. However, your account-level check limit is shared by the projects you own. For example, if you are on the Hobbyist plan (free, 20 check limit) and have three projects, the total number of checks across all three projects must not exceed 20.</p><h3>Overview of Your Projects</h3><p>Click on the Healthchecks.io logo in top navigation to see an overview of your projects. You can also use the drop-down menu in the top navigation to switch between projects and to access the “Project Settings” pages. The red badges show the number of checks currently down:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GDkWoPNlxos3Czv6.png" /></figure><h3>Project Settings</h3><p>API keys and team memberships are now project-scoped. In the “Project Settings” page you can generate the API keys and invite people to the project’s team. The “Project Settings” page is currently only available to the project owner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/0*HNrGusxc4zZVMvpr.png" /></figure><h3>Transferring Checks Between Projects</h3><p>You can move checks between the projects you have access to. For example, you can create a check in your personal account, but later decide to move it to your company account. The check’s URL and ping history is preserved, but the enabled integrations change: the check loses its current integrations, and gets assigned all integrations of the new project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/0*exYfxBT-ENjJl3WQ.png" /></figure><p>This is the initial release of Projects. I will appreciate <a href="/service/mailto:contact@healthchecks.io">any and all feedback</a> as I keep iterating on the interface and the features.</p><p>Happy monitoring, <br>Pēteris,<br><a href="https://healthchecks.io/">Healthchecks.io</a></p><p><em>Originally published at </em><a href="https://blog.healthchecks.io/2019/02/introducing-projects-on-healthchecks-io/"><em>https://blog.healthchecks.io/2019/02/introducing-projects-on-healthchecks-io/</em></a><em> on February 4, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9243c663a0b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>