<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Crafting Cronitor - Medium]]></title>
        <description><![CDATA[Adventures in Engineering - Medium]]></description>
        <link>https://medium.com/crafting-cronitor?source=rss----dda536166ffb---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Crafting Cronitor - Medium</title>
            <link>https://medium.com/crafting-cronitor?source=rss----dda536166ffb---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 07:49:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/crafting-cronitor" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[5 places cron jobs live in Linux]]></title>
            <link>https://medium.com/crafting-cronitor/5-places-cron-jobs-live-in-linux-4913a57b7c1b?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/4913a57b7c1b</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cron]]></category>
            <dc:creator><![CDATA[Shane Harter]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 07:42:35 GMT</pubDate>
            <atom:updated>2018-10-17T07:42:34.579Z</atom:updated>
            <content:encoded><![CDATA[<p>It might surprise you that the version of cron that runs on your server today is largely compatible with the crontab spec written in the 1970s. One downside of this careful backwards compatibility is that jobs, even on the same server, can be created and scheduled differently.</p><p><strong>1. The user crontab<br></strong>In Linux, each user has their own crontab that can be viewed and updated using the crontab command. The user crontab is the easiest place to add a job so it’s commonly used and is the first place to look when you’re unsure how a job has been scheduled.</p><p>One challenge is to know which user crontab it is in. If you can see the job in syslog, the username may be included. For example, here i can see the ubuntu user running the database backup:</p><blockquote>Aug 5 4:05:01 dev01 CRON[2128]: (ubuntu) CMD (/var/cronitor/bin/database-backup.sh)</blockquote><p><strong>2. The root user crontab<br></strong>Like any other user, root has a user crontab. Essentially the same as any other user crontab, you are editing the root crontab when you run sudo crontab -e.</p><p><strong>3. The system crontab file<br></strong>The original and still commonly used way to schedule system-wide cron jobs is by adding entries to /etc/crontab. Writing to /etc/crontab requires sudo privileges and jobs scheduled here can be run as any user. If the first word in your job command matches a user account the command is executed as that user. If no user is specified it’s run by root. In this example, the job will be run by user dataproc:</p><blockquote>* 0 * * * dataproc /var/cronitor/bin/database-backup.sh</blockquote><p><strong>4. A crontab file in the system drop-in directory /etc/cron.d<br></strong>At some point we realized it was bad practice to have a single system-wide crontab mutated by installers and automated tooling and the /etc/cron.d/ directory was born. Jobs are scheduled using /etc/cron.d by copying or creating a symlink to a crontab file. Like jobs scheduled in /etc/crontab, the first word of the command can optionally specify the user to run the job.</p><p><strong>5. A script or command in /etc/cron.hourly, /etc/cron.daily, etc<br></strong>When precise scheduling is less important than simple frequency, cron jobs can be created by copying or symlinking a script to /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly or /etc/cron.monthly. In distributions like Ubuntu, you can see when these scripts will be invoked by looking for lines in /etc/crontab:</p><blockquote>7 * * * * root cd / &amp;&amp; run-parts — report /etc/cron.hourly<br>5 23 * * * root cd / &amp;&amp; run-parts — report /etc/cron.daily<br>5 23 * * 7 root cd / &amp;&amp; run-parts — report /etc/cron.weekly<br>5 23 1 * * root cd / &amp;&amp; run-parts — report /etc/cron.monthly</blockquote><p><strong>Bonus: Understanding /var/spool/cron<br></strong>When individual user crontabs are edited using crontab -e, the crontab files themselves are stored in /var/spool/cron. Avoid the temptation to edit files here directly and instead run crontab -e as the user that will run the scheduled job. If you edit files in /var/spool/cron directly you will lose the benefit of syntax checking that crontab -e provides and you may find that your distribution doesn’t detect changes without an explicit cron daemon reload.</p><p><strong>Cron jobs can fail due to bugs, system failure, accidental deletion, and thousands of other reasons. </strong><a href="https://cronitor.io/"><strong>Cronitor</strong></a><strong> will ensure you’ll never lose sight of an important job and that you’ll be alerted first when things go wrong.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4913a57b7c1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/5-places-cron-jobs-live-in-linux-4913a57b7c1b">5 places cron jobs live in Linux</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 common reasons cron jobs fail]]></title>
            <link>https://medium.com/crafting-cronitor/4-common-reasons-cron-jobs-fail-ce067948432b?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/ce067948432b</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Shane Harter]]></dc:creator>
            <pubDate>Tue, 02 Oct 2018 03:13:49 GMT</pubDate>
            <atom:updated>2018-10-02T03:14:31.710Z</atom:updated>
            <content:encoded><![CDATA[<p>Cron job failures create chaos for your users and your team. After analyzing over a million failure reports these 4 problems emerged as the most likely causes of preventable job failures:</p><h4>1. System resources have been depleted</h4><p>Without careful cleanup and log rotation, system resources like disk space can be consumed slowly over months or years before leading to job failures. By carefully cleaning up any files your cron job creates and using a tool like logrotate to automatically prune log files you can prevent disk space related failures.<br>Other resource problems, especially when running cron jobs in a virtual machine, are limits to the number of open file descriptors, threads, and memory usage. You can view limits on your system by running ulimit -a as the cron user.</p><h4>2. Recent infrastructure updates were made</h4><p>When developers are making changes to database connections or upstream APIs it’s easy to overlook cron jobs, especially when they’re running on a separate host. It can help if cron jobs are deployed like the rest of your app and are included in post-release verification steps.</p><h4>3. Your dataset has grown too large</h4><p>Cron jobs are often used for batch processing, event sourcing and other data intensive tasks that can reveal the constraints of your stack. Jobs may work fine until your data size grows to a point where a bottleneck is reached.<br>Optimizing database queries and API calls is the right place to start, and depending on the job, running it more frequently may be effective by reducing the data processed each time it runs.</p><h4>4. Cron job invocations begin to overlap</h4><p>Cron will run your command at the scheduled time even if the last invocation is still running, or even if the last 10 are. If your job invocations are close to overlapping, consider spacing them out further or adding a tool like flock to ensure only 1 instance is running.</p><p><strong>When failure prevention is not enough, </strong><a href="https://cronitor.io/"><strong>Cronitor</strong></a><strong> will keep you ahead of the chaos by monitoring your cron jobs and alerting you instantly when things go wrong.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce067948432b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/4-common-reasons-cron-jobs-fail-ce067948432b">4 common reasons cron jobs fail</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWS spend of a SaaS small business]]></title>
            <link>https://medium.com/crafting-cronitor/the-aws-spend-of-a-saas-side-business-30bd5dbd91b?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/30bd5dbd91b</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[saas]]></category>
            <dc:creator><![CDATA[Shane Harter]]></dc:creator>
            <pubDate>Fri, 09 Jun 2017 18:04:34 GMT</pubDate>
            <atom:updated>2021-03-03T18:08:05.313Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12wb6XsUPXfAUfoG_FBC5Q.png" /><figcaption>Cronitor AWS spend for the last 12 months</figcaption></figure><p>In the first 30 days after moving <a href="https://cronitor.io/">Cronitor</a> to AWS in January, 2015 we collected $535 in MRR and paid $64.47 for hosting, data transfer and a domain name. In the time since we’ve continued to increase our footprint, level-up instances and add more managed services. Despite the AWS reputation as an expensive foot-gun we’ve improved availability while keeping our bill consistently close to 12.5% of revenue. Here’s a look.</p><h4>The bumps and scrapes of AWS on the cheap</h4><p>When it became clear to us that our idea had a little traction we knew we needed to raise the bar from side-project<em> </em>to small business. Our first goal was not high availability, only <em>higher availability </em>than our previous setup on a single 2GB Linode. We really just wanted to restart our database without losing incoming telemetry data. Our initial build-out was simple:</p><ul><li>ELB</li><li>SQS</li><li>A pair of t2.small instances running our web app and data collection, both in us-west-2</li><li>A single m3.medium running MySQL and our daemon that evaluates failures and sends alerts</li></ul><p>We completed the migration in two hours with essentially no downtime and we were really pleased with ourselves. Beers were raised. Celebratory tweets were tweeted.</p><p>The joy was brief.</p><h4><strong>Issue 1: Breaking the ELB</strong></h4><p>Our users send telemetry pings when their jobs, tasks and daemons run. With NTP, the average server today has a very accurate clock and we see emergent traffic spikes at the top of every second, minute, hour, and day — as much as 100x our baseline traffic.</p><p>Immediately after the migration, users reported intermittent timeouts and we started double-checking server configs and tailing ELB logs. With traffic spikes still under 100 requests per second we resisted blaming the ELB and looked for mistakes in our own configuration. Finally, we set up a test to ping continuously from a few moments before 00:00 UTC to a few afterward and we saw requests failing that were never recorded in the ELB logs. Individual instances could be reached and the ELB request queue never backed up. It was clear that connections were being dropped at the load balancer, probably because our traffic spikes were too large relative to our baseline and too short-lived for it to be warmed up to more capacity. With an expensive AWS support plan we could’ve asked them to manually increase the size of our ELB but instead we chose to round-robin requests using DNS and remove the need for the load balancer altogether.</p><p>Lesson learned:</p><ul><li>Cloud solutions like elastic load balancing are generally engineered for the average use case. Think about the ways you are not like the average.</li></ul><h4><strong>Issue 2: Getting familiar with the CPU credit balance</strong></h4><p>The T2 family of burstable instances provides cost-effective performance for occassional workloads, or so the website says. What I wish it said was: Once you start running this instance at a consistent 25% of CPU you will drain your CPU credit balance and once it’s empty you essentially have the computing power of a RasperryPi. No alarms will sound when this happens and your CPU% will not reflect the diminished capacity. The first time we exhausted the credit balance we mistakenly thought connections were being dropped due to an upstream issue.</p><p>Lessons learned:</p><ul><li>If something is a lot cheaper there’s a good reason, so understand it.</li><li>You should only use T2 instances within an auto-scaling group.</li><li>Just in case, create a CloudWatch alarm to warn you when the credit balance drops below 100.</li></ul><h4><strong>Issue 3: Reading the fine print</strong></h4><p>Last year at re:Invent Amazon updated their Reserved Instance offering, probably in response to more generous terms from Google Cloud. The press release said Reserved Instances would be cheaper and could now be moved between availability zones. I’ll drink to that!</p><p>When it came time to retire our final T2 instances in October we rolled-out new M3’s with these cheaper, more flexible twelve-month reservations. After on-boarding several large users in Q1 we decided in April to level-up instance types again to M4.large. We had 6 months left on our October resevations so I planned to sell them like I always had before. Then I learned the expensive truth that the trade-off to these cheaper more flexible reservations is… you can’t resell them.</p><p>Lessons Learned:</p><ul><li>If something is a lot cheaper there’s a good reason, so understand it.</li><li>Always read the fine print twice. AWS billing is incredibly complicated.</li></ul><h4>A peek at real-world AWS costs</h4><p>Our infrastructure today remains fairly straightforward:</p><ul><li>M4.large cluster handling incoming telemetry collection</li><li>M3.medium cluster serving our web app and API</li><li>M4.large worker running our monitoring daemon and alerting system</li><li>M4.xlarge for MySQL and Redis</li></ul><p>We continue to use a number of managed services including SQS, S3, Route53, Lambda and SNS.</p><h4><strong>Elastic Compute</strong></h4><p>We use partial upfront reservations for all of our instances.</p><p>You can see that on our monthly bill we pay 2/3 as much for provisioned iops as we do for instances. In contrast to most cloud metrics guaranteed iops (input/output operations per second) are a concrete SLA, something that has a real cost to provide. They’re also an essential part of your Ec2 budget for any host where disk performance matters. If you don’t pay for iops your workloads will wait in line for remnant capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OTencrYv1CufKowHOL8pA.png" /><figcaption>Please don’t ask me what an “alarm-month” is</figcaption></figure><h4><strong>SQS</strong></h4><p>We use SQS extensively to queue incoming telemetry pings and results from our healthchecks service. One optimization we added a few months after migration is max-batching of reads. You pay by number of requests, not number of messages, so it reduces costs and makes it significantly faster to gulp down messages.</p><p>During the migration we were concerned about SQS as a single point of failure for our data collection pipeline. To mitigate the risk we deployed a small daemon on each host to buffer and retry SQS writes on failure. It’s had to buffer messages only once in 2.5 years so 1) it was 100% worth building and 2) SQS has proven incredibly reliable in us-west-2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1T0sQYCzZQUvr7l4tj3fA.png" /></figure><h4><strong>Lambda</strong></h4><p>Our <a href="https://cronitor.io/micro-service-monitoring">Healthchecks service</a> is built in part on Lambda workers deployed in the regions below. It’s worth calling out that Lambda has a generous free tier that applies to each region independantly. Currently the free tier is <a href="https://aws.amazon.com/lambda/pricing/">advertised as “indefinite”</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZEaHxC3sXy-Cs15yZsjVtA.png" /></figure><h4><strong>S3</strong></h4><p>We backup database snapshots and log files to S3 with replication to us-east-1 for disaster recovery.</p><p>An AWS pro-tip is that backups and EBS snapshots are vital for disaster recovery but serious failures within a region are not usually contained to a single service. If you can’t launch an instance there’s a good chance you won’t be able to copy your snapshots to another region. Do that ahead of time!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45iAPvNL21BcZzxR71jn-Q.png" /></figure><h4><strong>Wrapping it up</strong></h4><p>Having worked on large corporate and investor-funded AWS deployments, I can personally vouch that you can run up a big tab. They’ve created a wonderland of tools and shiny objects that all snap together — and you’re charged for <em>everything you touch</em>. Not only that, you’re also charged for the data flowing in and out each of those systems. This can be a real gotcha if you’re not careful. Our friends at <a href="https://cloudforecast.io/">Cloudforecast.io</a> helped us get a better understanding of this, and also wrote a handy guide to <a href="https://www.cloudforecast.io/blog/understand-and-optimize-your-aws-data-transfer-costs">reducing AWS data transfer costs</a>.</p><p>AWS can be dangerous. You have to be frugal and restrain your appetites, but you’re rewarded with the ability to grow a small business into a bigger one with on-demand access to any resources you need. It’s good to pause for a moment and appreciate how awesome that is — then get back to work.</p><h3>Cronitor is your missing monitoring tool. <a href="https://cronitor.io/pricing">Try it free</a>.</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30bd5dbd91b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/the-aws-spend-of-a-saas-side-business-30bd5dbd91b">The AWS spend of a SaaS small business</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring for developers]]></title>
            <link>https://medium.com/crafting-cronitor/monitoring-for-developers-ab5cb4490d68?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/ab5cb4490d68</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Shane Harter]]></dc:creator>
            <pubDate>Sun, 19 Feb 2017 00:26:30 GMT</pubDate>
            <atom:updated>2017-02-19T01:51:16.090Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Monitoring and QA are the same thing. You’d never think so until you try doing a big SOA. But when your service says “oh yes, I’m fine”, it may well be the case that the only thing still functioning in the server is the little component that knows how to say “I’m fine, roger roger, over and out” in a cheery droid voice. — Steve Yegge</blockquote><p>Ten years ago, still young in my career building software things, I got my first taste of DevOps. We didn’t call it that back then but for the first time I had to worry about how my software would be delivered to the world.</p><p>It was an awakening. A new professional class of tooling was being built in companies like Google and Amazon but things for most of us were still primitive: Usually you paid somebody to host servers that you drove to the data center in the back of your Subaru. Then you setup everything else, which is quite a bit when you really need to be on top of things. The process where I worked, and sadly I think many can relate, was just to clone everything in the most recent working system, manually updating where necessary. Automation where I found it was often a bundle of Perl scripts or .bat files that came with their own folklore. I learned that ops was just another legacy system that needed to be maintained by people who hadn’t built it.</p><p>As a developer, I was changed. I started thinking about and building ops configuration into my (Subversion) repos. Have we built a daemon? The init conf should be auto-generated. A website? Dev and prod Apache conf should be committed with the code that relies on it. I still remember where I was when I first read about Puppet 😍.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hwdxh-MILJ-8bHYHgfNqXg.png" /></figure><p>Developers have kept ourselves fed for the last 5 years in part by breaking big apps into littler apps and calling them microservices. I enjoy slaying a monolith as much as anybody. In the second decade of my career it helps soothe the guilt of having built so many to begin with. But often microservices are bespoke, one-off mini-monoliths. This is not the way to engineer good outcomes. To avoid developer malpractice in 2017 the table stakes for microservices include:</p><ul><li><strong>Conformity of build process</strong>. If your services cannot be developed, tested, staged, and deployed in a uniform way you will pay with decreased velocity and more mistakes. This is the assembly line part of your software product. It should be boring, consistent and reliable.</li><li><strong>Conformity of craftsmanship</strong>. In a service-oriented architecture you will often need to make changes in multiple code bases to ship a new feature. It’s an anti-pattern for teams or developers to seal-up services to themselves. The only way that people can move from one service to the next is consistency of craftsmanship and expectations. If you don’t have a runbook, linter, or tests that run with a single command you’re doing it wrong.</li><li><strong>Aggregation of errors and logs</strong>. Use an error aggregation tool like Sentry. Combine logs into Elastic Search or Splunk. Give yourself operational visibility into your distributed system.</li><li><strong>Uniform visibility into the health of your services</strong>. The smaller a micro service is, the easier it is to overlook when the system is down and you’ve temporarily lost a dozen IQ points. In the chaos you might just forget about something like the <em>single sign on redirect service</em>. It happens. Distributed systems are just more complicated and you need better monitoring.</li></ul><p>In the era of the monolith, these tasks didn’t really need to scale. You configured Nagios once and tweaked it every now and then when something important changed. Today, the only sensible way to do this work is in code. Tools like Chef, Puppet, Terraform, Ansible and database migrations are the result. <em>Things aren’t as primitive anymore.</em></p><p>In 2014 I started working <a href="https://www.indiehackers.com/businesses/cronitor">nights and weekends</a> with a friend to launch <a href="https://cronitor.io/">Cronitor</a>, a service for monitoring cron jobs and scheduled tasks. A lot of great companies use Cronitor and we’ve had fun building it. Last year on the hop from San Francisco to Portland we found ourselves talking about what to do next. Cronitor is built with microservices and discussing our own pain points we were startled by the question <strong><em>why can’t we just keep monitoring configuration directly within our code?</em></strong></p><p>On the return flight a few days later we drafted a plan to build web service health checks and SDKs for popular frameworks that will enable developers to configure monitoring directly from application code, with updates that are published automatically when you deploy your app.</p><h4>Introducing: Auto Health Checks</h4><p>In the fall we launched the first part, <a href="https://cronitor.io/micro-service-monitoring">web application health checks</a> with assertions like response body contains &quot;SUCCESS&quot;.</p><p>This week, we are launching our first SDK for Django. With <a href="https://github.com/cronitorio/django_auto_healthchecks"><strong>django_auto_healthchecks</strong></a>, you attach monitoring directly to routes in urls.py and when your app is restarted, health checks are collected in a batch and published. Cronitor runs them immediately for post-release validation and as frequently as every 30 seconds thereafter. SDKs for Express, Laravel and Rails are coming soon.</p><p>Our goal with Auto Health Checks is simple: a version controlled single source of truth that defines the health of your app, with instant alerts when you need them. Check out <a href="https://github.com/cronitorio/django_auto_healthchecks">django_auto_healthchecks on Github</a> and sign up free at <a href="https://cronitor.io/">https://cronitor.io</a> to get started.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab5cb4490d68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/monitoring-for-developers-ab5cb4490d68">Monitoring for developers</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons learned with Stripe subscriptions]]></title>
            <link>https://medium.com/crafting-cronitor/lessons-learned-with-stripe-subscriptions-d6c8d408eb1?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/d6c8d408eb1</guid>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[apps]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[ecommerce]]></category>
            <dc:creator><![CDATA[Shane Harter]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 16:20:06 GMT</pubDate>
            <atom:updated>2017-01-08T20:45:51.186Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeSXeH2lrnM1aHKrW1bPHA.png" /><figcaption>The MRR Grind</figcaption></figure><p>In June, 2014 we shipped an MVP for <a href="https://cronitor.io/">Cronitor</a> that was so basic I cringe a little when I think about it. It didn’t do very much, most features were cut, but it shipped with paid subscriptions on day one. We sold one to a friend.</p><p>My earliest work on subscription integration was primitive: The first month’s charge was captured during the upgrade but we created each subscription by hand in the Stripe dashboard. It sounds sketchy but recurring billing is just <em>not</em> a problem that needs to be solved by launch day. Now, 2.5 years and 2 iterations later, I’ve had time to meditate on a few lessons learned.</p><h4>Embrace the platform</h4><p>Don’t worry about lock-in: a billing migration is not a high leverage project for a small team anyway. As a Stripe user this means:</p><ul><li>Use coupons to manage price segmentation, reducing overall number of plans.</li><li>For usage based or variable billing, push the quantity of users or widgets consumed to Stripe. Don’t try to manage these line items directly from your code.</li><li>A plan in your app should map 1:1 to a plan in Stripe. If you have monthly and yearly options for a plan, this is actually two plans. Define feature sets in your code that can be shared among multiple plans to simplify implementation.</li></ul><h4>Stripe as the source of truth</h4><p>Your code will need to know which plan a user is subscribed to, but the source of truth for these details should remain in Stripe.</p><ul><li>Stripe webhooks are incredibly reliable — they are versioned and retried on failure. Use webhooks to push necessary details to your application.</li><li>If you offer a free plan, create subscriptions for these users in Stripe. You will be able to find and manage all users from your Stripe dashboard and there will be fewer conditionals in your code if every user has a subscription of some kind.</li><li>With webhook integration the Stripe dashboard can become your billing UI. Save yourself the effort of rebuilding it with API calls.</li></ul><h4>There are a few missing pieces</h4><p>Stripe has a well deserved reputation for quality but they have plenty of work yet to do.</p><ul><li>Invoicing users must be done manually today using a webhook. Stripe sends payment receipts but they lack the details many companies need on an invoice. At Cronitor we created invoices in Excel for several months until we found time to hack on this. We hoped Stripe would beat us to it. Two years later they still haven’t.</li><li>Failed payment retry logic is limited. Failed charges can only be retried three times and there is no notification to the user when their payment fails. We’ve found it’s best to give users 4–6 weeks to resolve failed payments with 1-2 reminders per week. None of that is possible today without dev work.</li></ul><h4>Monitor important Stripe events</h4><p>Cronitor is built to monitor scheduled jobs, workers, and services but it can also be used to consume Stripe webhooks and send notifications to your team:</p><ul><li>Be notified instantly of failed or disputed charges.</li><li>Trigger an alert if no subscriptions have been created recently.</li></ul><p><a href="https://cronitor.io/">Use Cronitor for free</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6c8d408eb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/lessons-learned-with-stripe-subscriptions-d6c8d408eb1">Lessons learned with Stripe subscriptions</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The JIT Startup]]></title>
            <link>https://medium.com/crafting-cronitor/the-jit-startup-bb1a13381b0?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb1a13381b0</guid>
            <category><![CDATA[lean-startup]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Cronitor.io]]></dc:creator>
            <pubDate>Thu, 18 Feb 2016 19:20:00 GMT</pubDate>
            <atom:updated>2016-08-12T23:07:16.544Z</atom:updated>
            <content:encoded><![CDATA[<p>A few nights ago, nearly two years after my co-founder and I wrote the first lines of code for <a href="https://cronitor.io/">Cronitor</a>, I finally added the ability for a user to change their email address. Why did it take two years to add a near universal feature for any SaaS service? Truth be told, even though we’ve grown our customer base and the size of our app significantly during that time, it hadn’t been worth it.</p><p>A few days ago, after receiving a support request asking us how one might change their email address, I decided to quickly look at how many times we had been contacted about changing an email address. From what I can tell it’s been 9 times. 8 of them are still customers by the way.</p><p>Some quick math (~5 min/email * 9 email) told me we had probably spent more time updating the db manually than it would take me to automate it, and thus, unfortunately, the time had come to finally build it. It ended up being 47 loc in 5 files, and about 30 min total time including deploy.</p><p>My cofounder and I have been jokingly calling this “The JIT Startup” after the way some programming languages delay compiling code into lower level machine instructions until the code is actually called.</p><p>That edit email form is just one of probably a hundred features competing for our attention. And even though we use a <a href="https://stripe.com/">bunch</a> <a href="https://www.twilio.com/">of</a> <a href="http://mandrill.com/">great</a> <a href="https://slack.com/">SaaS</a> <a href="https://www.pobox.com/">services</a> to mitigate the need for writing our own solutions to common problems, it still requires a lot of time and code to write the glue layer to integrate with these services.</p><p>Here are just a few of the many things we’ve written software for in the past 18 months, and <strong><em>none</em></strong> of these are our actual product.</p><ul><li>Collecting Payments</li><li>Changing Plans</li><li>Sending Invoices</li><li>Editing Payment Information</li><li>Editing Account Information</li><li>Sending Automated Emails</li><li>Managing Customer Support Requests</li><li>Billing Failures &amp; Account Termination</li></ul><p>Many of those have been multi-day projects that have required significant thought and investment on our part. And, it’s worth repeating, none of them are related to our core business — <a href="https://cronitor.io/">monitoring scheduled tasks and services</a>. By the way, the only one of those we launched with was the first one — Collecting Payments.</p><p>We’ve been fortunate to find a niche for a product that many people actually have a need for (we were our own first customer, but that’s another story). But, it’s not dumb luck that we have been able to grow from weekend hack project to over 100 paying customers. Several similar projects have failed to gain the traction we have.</p><p>The JIT mindset helps us focus our priorities, and has helped keep our software flexible. Said in a single sentence it’s this:</p><blockquote>Unless the project under consideration is immediately applicable to our core product, distribution, or will lead to a closed deal with a new customer, then delay doing it for as long as possible.</blockquote><p>If you’re a software developer it’s especially tempting to justify spending more time on your software. You’ve worked with tangled messes of code in the past and suffered through others’ poor product choices. This is your chance to “do it right” from the beginning, but that’s a trap! Never write a line of code today that can be put off until tomorrow. Focus exclusively on the essentials, and handle everything else over a support channel.</p><p>Writing software is hard. It’s easy to convince yourself that if you’re working on something hard you’re working on something worthy of your time and attention. It’s a mistake we’ve personally made many times. Embracing a JIT mentality keeps our attention focused, our churn low, and our product ahead of our competition.</p><h4>Cronitor is a simple monitoring tool for scheduled jobs, periodic tasks, external SaaS tools, and almost anything else.</h4><h4><a href="https://cronitor.io/">Try Cronitor free for 2 weeks</a>.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb1a13381b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/the-jit-startup-bb1a13381b0">The JIT Startup</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Django: Breaking out not breaking up]]></title>
            <link>https://medium.com/crafting-cronitor/scaling-django-breaking-out-not-breaking-up-c48b4a9e4a96?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/c48b4a9e4a96</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Cronitor.io]]></dc:creator>
            <pubDate>Wed, 04 Nov 2015 15:09:36 GMT</pubDate>
            <atom:updated>2016-09-03T20:55:35.849Z</atom:updated>
            <content:encoded><![CDATA[<p>We launched <a href="https://cronitor.io/">Cronitor</a> in June 2014 after a few weeks of hacking on an MVP. With an eye on shipping quickly we built everything within Django, including our crucial ping tracking endpoints, and it was a valuable force multiplier. The first version of our tracker was a normal view function that persisted directly to the database. Keep it simple, right?</p><p>When your traffic is under a few requests per second your only real scalability task is to not screw it up. The specific point where that changes will depend on your circumstances, but my best advice if you’re concerned about how Django will scale is wait and see. At Cronitor, we reached our inflection point 8 months after launch. With traffic above 10 requests per second, we began seeing intermittent timeouts during particularly busy moments like 00:00:00 UTC. We needed a plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-Yj_oKBiOzrQG2wLciYtg.png" /></figure><h4>First, do fewer things</h4><p>It can be easy to convince yourself that the truly lean thing to do is build a bulletproof tracking endpoint that will never need to be touched again. Having too much to do can be a real antidote to this type of thinking and it kept us focused on the real problems, like smoothing our load curve. When our customers ping our tracking endpoints, they predominantly do so in the first 10 seconds of each minute. Our database would redline that entire time and it was obvious we needed to queue requests and spread out the write load.</p><p>We updated our ping tracking endpoints — still within Django — to do only basic validation before injecting into SQS. A simple worker daemon reads from the queue into our database at a constant rate and can easily be paused, decoupling the DB from our endless stream of ping traffic.</p><p>It bought us 6 months.</p><h4>The high availability factor</h4><p>Deploying code should be painless and reliable but the only truly safe code is what’s been running successfully in production.</p><p>By combining our ping tracking application with our website in a single Django monolith we were pairing an API that is highly critical and changes infrequently with a web application that gets updated almost every day. More than performance, we knew that decoupling would be a real win for service availability.</p><h4>Breaking out</h4><p>As clever as Django is with generators and lazy evaluation it’s a feature-rich framework that emphasizes developer productivity over request throughput. While we had spare CPU capacity it wasn’t an interesting problem to solve, but a few months ago we were facing the need to level-up our instance types or fan-out more ping collectors. We chose, instead, to break out ping tracking from Django and build a simple tracker using the <a href="https://falcon.readthedocs.org/en/stable/">Falcon micro framework</a>. Developing with a framework like Falcon means fewer creature comforts but you’re well compensated by lower request overhead and improved throughput.</p><pre><em>Logging requests to a flat file on a t2.small instance:</em></pre><pre>Django:  519.93 trans/sec<br>Falcon: 2083.19 trans/sec</pre><p><em>(</em><a href="https://www.joedog.org/siege-home/"><em>Siege</em></a><em> was used to run these tests. You can install Siege with homebrew and compile stats like this using `siege -c 30 -t 30S -b http://example.com/path`)</em></p><h4>The Nginx Config</h4><p>Before the break out, our Nginx config had a single <a href="http://nginx.org/en/docs/http/ngx_http_core_module.html#location">location block</a>. To send ping requests to their own application we first need to migrate the affected route from <a href="https://docs.djangoproject.com/en/1.8/topics/http/urls/">urls.py</a> to a second location block:</p><pre>server {<br>  listen 80;<br>  server_name cronitor.io;<br><br>  access_log /var/log/cronitor/access.log;<br>  error_log  /var/log/cronitor/error.log;</pre><pre>  location ~* &quot;^/[A-Za-z0-9]{4,12}/(r|c|f|run|complete|fail)$&quot; {<br>    include uwsgi_params;<br>    uwsgi_pass unix:////var/cronitor/tracker-uwsgi.sock;<br>  }<br><br>  location / {<br>    include uwsgi_params;<br>    uwsgi_pass unix:////var/cronitor/cronitor-uwsgi.sock;<br>  }<br>}</pre><h4>Rolling it out</h4><p>Before deploying the updated Nginx config we needed to stand up the new uWSGI application:</p><ul><li>We created a new <a href="http://upstart.ubuntu.com/">Upstart</a> service for the tracking server. (Supervisor, RunIt, etc, are fine alternatives to Upstart.)</li><li>A new command in our <a href="http://www.fabfile.org/">fabfile</a> will deploy code and reload the service.</li></ul><p>With the pieces in place we deployed to a hot spare, then to a pair of production servers. This snapshot was taken 18 hours later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVLLcMVD7r6JAumj60UPzg.png" /></figure><p>Isn’t that just the prettiest graph you’ve seen all day?</p><h4>Cronitor is a simple monitoring tool for scheduled jobs, periodic tasks, external SaaS tools, and almost anything else.</h4><h4><a href="https://cronitor.io/">Try Cronitor free for 2 weeks</a>.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c48b4a9e4a96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/scaling-django-breaking-out-not-breaking-up-c48b4a9e4a96">Scaling Django: Breaking out not breaking up</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to ignore large requests with Nginx and uWSGI (and several ways not to)]]></title>
            <link>https://medium.com/crafting-cronitor/how-to-handle-large-requests-with-nginx-and-uwsgi-and-several-ways-not-to-d231dafb9b90?source=rss----dda536166ffb---4</link>
            <guid isPermaLink="false">https://medium.com/p/d231dafb9b90</guid>
            <dc:creator><![CDATA[Cronitor.io]]></dc:creator>
            <pubDate>Wed, 04 Nov 2015 15:06:21 GMT</pubDate>
            <atom:updated>2019-08-12T19:15:40.079Z</atom:updated>
            <content:encoded><![CDATA[<p>At Cronitor, our primary engineering objective is to never miss a tracking ping. We process ping requests into telemetry data and a missed ping can lead to errant reporting and, even worse, false-positive alerts. With this focus we’ve grown from a Django app on a single Linode box to a stack that today handles our traffic bursts of 100 requests/sec.</p><p>Last week our LogWatcher daemon alerted us to failed pings in our Nginx error log. The daemon’s job is to rescue failures by re-processing them but it was puzzled by these requests and flagged them for manual review. I was on the train when my phone buzzed in my pocket.</p><pre><em>Client intended to send too large body: 14601004 bytes, client: xx.xx.xx.xx, server: cronitor.link.</em></pre><p><em>Hmm. 14 MB is a big POST.</em></p><p>Occasionally a customer will push a bug into production that tries to DOS us in some way. Their traffic gets blocked at the firewall while we reach out. These large requests, however, were no accident: A new customer is monitoring an external SaaS service that happens to send along these large payloads. Out of the customer’s control, we had a clear choice: Figure out a way to handle these requests or lose their business.</p><p><strong>Take One.</strong></p><p>There are many ways to solve the problem but leaning on our engineering instincts we dismissed any solution that introduced new complexity. We had to solve the problem with the tools we already had.</p><p>There is obvious peril in simply increasing the <strong>client_max_body_size</strong> limit to 20 megabytes. Our traffic comes from machines and machines operate on a clock that is usually synchronized with NTP. Over a third of our pings are received in the first 10 seconds of each minute. Large request bodies passed down from kernel to web server to application server to application would have a certain impact on our throughput in those 10 seconds.</p><p>We tried it anyway. Our ping collectors average below 25% cpu utilization. The notion that tweaking a setting <em>might</em> buy us some time was too attractive to resist. I rolled it out to a hot-spare, then to a single production ping collector. I crossed my fingers and held my breath. I wasn’t holding it very long.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vtKab3rQfOLX82Bhrj_Hbg.png" /></figure><p><strong>Take Two.</strong></p><p>Later that day, with a little distance from the problem, I thought about how these large requests were wending thru our stack. Cronitor doesn’t use a single load balancer. Instead, we return a virtual IP from our equally-weighted Route53 DNS entries. The request is passed to an Nginx proxy server. Nginx routes ping requests to a tracking server built on the <a href="https://falcon.readthedocs.org/en/stable/">Falcon micro-framework</a>, and all other requests to a Django application.</p><p>After the TCP handshake completes, a client begins a POST request by sending the headers first and then the body. But HTTP is half-duplex and clients are going to send the entire body even if the <strong>CONTENT_LENGTH</strong> header exceeds the client_max_body_size. Knowing this we could trust that our performance issues were <em>after</em> Nginx: in the uWSGI server or our application. Reading the uWSGI docs helped connect the dots and before long I was deploying a test config to a hot-spare that included <strong>uwsgi_pass_request_body off</strong>. If we could drop the request body before the hand-off to the uWSGI upstream it <em>could </em>work. After rolling it out I tested a few simple POSTs and received 200’s. To simulate large requests I generated 30,000 kb of fake data and started posting it with Siege:</p><pre>$ dd if=/dev/zero of=/tmp/large.dat bs=1024 count=30000<br>$ siege -c 2 -t 30S -b “http://test.cronitor/d3x0 POST &lt; /tmp/large.dat”</pre><p>Watching CPU and IO performance, I was stoked. Success! Then I checked the error log.<em> [error] 23073#0: *60177384 upstream prematurely closed connection while reading response header from upstream.</em></p><p><em>Shit.</em></p><p><strong>Take Three.</strong></p><p>WSGI, <a href="https://www.python.org/dev/peps/pep-3333/">like the PEP it’s based on</a>, is a complex beast. I once fantasized about building our tracker on WSGI “bare metal” but breaking-trail proved painful and I played with Werkzeug before settling on Falcon. Stepping thru the Falcon code I came to the source of the problem:</p><pre>if self.content_length is not None:<br>    self.stream = helpers.Body(self.stream, self.content_length)</pre><p>The naive POST requests I sent worked because I hadn’t taken the time to pass a body. The CONTENT_LENGTH request header was 0 and this code path in Falcon was never hit. The problem arose when it received a non-zero CONTENT_LENGTH without a valid stream object. By dropping the body I introduced a mismatch between the CONTENT_LENGTH header and the actual content length. I needed to finish the job:</p><pre>uwsgi_pass_request_body off;<br>uwsgi_param CONTENT_LENGTH 0;</pre><p><strong>Success.</strong></p><p>After testing the build locally and rolling it out blue/green to our production environment we kept a close eye on performance. We’ve seen no measurable load impact and we’ve passed the good news on to our customer.</p><p>If you find yourself needing to handle large POST bodies at scale, our advice is to avoid passing the request body to your application server. If you can’t discard the body like we , consider parsing it locally as a <a href="http://wiki.nginx.org/HttpLuaModule#ngx.req.socket">ngx.req.socket</a> stream in the HttpLuaModule.</p><h4>Cronitor is a simple monitoring tool for scheduled jobs, periodic tasks, external SaaS tools, and almost anything else.</h4><h4><a href="https://cronitor.io/">Try Cronitor free for 2 weeks</a>.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d231dafb9b90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/crafting-cronitor/how-to-handle-large-requests-with-nginx-and-uwsgi-and-several-ways-not-to-d231dafb9b90">How to ignore large requests with Nginx and uWSGI (and several ways not to)</a> was originally published in <a href="https://medium.com/crafting-cronitor">Crafting Cronitor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>