<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mike Schroll on Medium]]></title>
        <description><![CDATA[Stories by Mike Schroll on Medium]]></description>
        <link>https://medium.com/@MikeSchroll?source=rss-bc88eac5202c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qZCgpYSGvvscwZVN.jpeg</url>
            <title>Stories by Mike Schroll on Medium</title>
            <link>https://medium.com/@MikeSchroll?source=rss-bc88eac5202c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:05:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@MikeSchroll/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why DNSFilter replaced InfluxDB with TimescaleDB for performance, ease of use]]></title>
            <link>https://medium.com/timescale/case-study-dnsfilter-replaces-influxdb-with-timescaledb-for-performance-ease-of-use-54b5381affe3?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/54b5381affe3</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[influxdb]]></category>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Fri, 16 Mar 2018 14:56:35 GMT</pubDate>
            <atom:updated>2018-08-30T17:02:17.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nLiCvxM_65sKOv9G7qEZCA.jpeg" /><figcaption>“I feel confident we can scale to 3B queries a day or more without much change — and look forward to continuing to grow with TimescaleDB.” — <a href="https://medium.com/u/bc88eac5202c">Mike Schroll</a>, CTO @DNSFilter</figcaption></figure><p>After using early iterations of InfluxDB and more recently InfluxDB 1.3, <a href="https://www.dnsfilter.com/">DNSFilter</a> ran into certain issues and we began to evaluate TimescaleDB as an alternative. <a href="https://blog.dnsfilter.com/3-billion-time-series-data-points-dnsfilter-replaced-influxdb-with-timescaledb-d9f827702f8b"><strong>Read why DNSFilter replaced InfluxDB with TimescaleDB</strong></a><strong>.</strong></p><p>Update! This post made it to Hacker News. <a href="https://news.ycombinator.com/item?id=16539317">Read the discussion</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6KhDAn8d9XUnWslNqVNpnQ.png" /><figcaption>A 5-day snapshot of Average CPU Load with InfluxDB 1.3 compared to TimescaleDB 0.8.</figcaption></figure><p><em>And if you’d like to learn more about </em><a href="http://www.timescale.com/"><strong><em>TimescaleDB</em></strong></a><em>, please check out their </em><a href="https://github.com/timescale/timescaledb"><em>GitHub</em></a><em> (stars always appreciated), and please </em><a href="/service/mailto:hello@timescale.com"><em>get in touch</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54b5381affe3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/timescale/case-study-dnsfilter-replaces-influxdb-with-timescaledb-for-performance-ease-of-use-54b5381affe3">Why DNSFilter replaced InfluxDB with TimescaleDB for performance, ease of use</a> was originally published in <a href="https://medium.com/timescale">Timescale</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards 3B time-series data points per day: Why DNSFilter replaced InfluxDB with TimescaleDB]]></title>
            <link>https://medium.com/@MikeSchroll/3-billion-time-series-data-points-dnsfilter-replaced-influxdb-with-timescaledb-d9f827702f8b?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9f827702f8b</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[internet]]></category>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Wed, 07 Mar 2018 20:00:02 GMT</pubDate>
            <atom:updated>2018-03-19T13:07:23.141Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Our results: 10x better resource utilization, even with 30% more requests</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cbHW_G1ok10DewDZtFqCEw.jpeg" /><figcaption><em>Our end results. Full explanation below</em></figcaption></figure><p><a href="https://www.dnsfilter.com/">DNSFilter</a> (a <a href="https://www.techstars.com/content/accelerators/introducing-techstars-class-117-techstars-anywhere-class-2018/">Techstars Anywhere 2018 company</a>) provides cloud-based content filtering and threat protection for IT Professionals. We accomplish this through DNS, and offer usage-based pricing.</p><p>In the early days of <a href="https://www.dnsfilter.com/">DNSFilter</a>, we spent some cycles thinking towards the future scalability of the business. I’m an advocate of early-stage startups not over-optimizing, but we also needed to understand the unit-economics of our business. That meant understanding how performant we could be, along with what bandwidth, storage and reporting costs would be.</p><p>A large component of our architecture is logging our customers’ requests. We are a usage-based service, and we provide analytics to our customers. This means we need to account for each DNS request, because any lost request is lost revenue, and lost data for our customers.</p><p>Back in late 2015 when we started the company, and into early 2016 as we began internally testing our MVP, we researched options in the market for storing and analyzing large sums of data. A new buzz-word at the time was ‘time series databases’. Our data seemed to fit the pattern, and I began working with the hot new startup InfluxDB.</p><h3>Challenges with InfluxDB</h3><p>After spending months experimenting, looking at their performance tests, implementing InfluxDB into query-processor, and setting up a fast and powerful server to run InfluxDB — There were still more hurdles. Unfortunately I wasted weeks of time on the learning curve of understanding in what ways InfluxDB differed from traditional SQL, limitations in their schema, query language, indexing, and more. I emailed influx, read forums, and github issues, and battled through issue after issue. But finally we had a working solution. For a while.</p><p>Unfortunately over time we ran into a number of issues with InfluxDB 1.3.</p><ol><li><strong>Cardinality.</strong> This was one of our biggest challenges. In order to provide customers with statistics about what domains are visited, we needed to ‘tag’ (index) this data. At the same time, there are millions of different domains out there, many of which our customers access each day. Influx was originally designed to handle only a certain threshold of unique data. We quickly exceeded it. What this meant for us was that to survive, we had to begin shaving off the outliers. We would only rollup entries with 20 or more accesses within a 15-min period; so that we reduced the overall number of unique domains.</li><li><strong>Rollup tables.</strong> In order to have aggregate stats which could be queried from our customer analytics dashboard, we needed to create rollup tables. 1 minute summaries, 15 minute, 1 hour, 1 day, etc. Some rollup tables depended on the completion of other rollup tables. Some were intermediary rollup tables just designed to serve a higher-level rollup table. A lot of storage and table creation went into these. I have a 5 stage, 3 page document outlining how to create our rollup tables.</li><li><strong>Ingestion performance vs query performance.</strong> Despite the touted performance of InfluxDB, we found that ingestion performance dropped significantly once you were also querying against that same data. At only a few hundred queries per second during the peak of our days, the logs would start alerting us to dropped queries. Again, this directly translated into lost revenue. We began planning for sharding of customer accounts across influxdb instances, further adding to the complexity of the solution.</li><li><strong>Resource utilization.</strong> At one point in time we had a 512GB ram server in order to try to ‘work-around’ the cardinality problem. <strong><em>Influx ground to a halt.</em></strong> We re-architected the entire schema with more rollup tables, shorter retention periods, and were able to run it for a few months on a 64GB server. It still used almost all the ram, and had a constant CPU load of over 50%. It felt like we had little room to grow.</li></ol><h3>Enter TimescaleDB.</h3><p>While in NYC, meeting up with one of our vendors, <a href="https://www.packet.net/">Packet.net</a>, I searched through LinkedIn and saw that an old friend, Ajay Kulkarni was now in New York, and working on an IoT startup. As we reconnected, I learned that they were in the process of pivoting the business based on a component they’d developed to process time series data; and store it in PostgreSQL. As I shared some of my challenges with InfluxDB, they shared their status with releasing <a href="http://www.timescale.com/">TimescaleDB</a>. I left New York that week excited and rejuvenated that maybe there was another option that could fit our needs.</p><p>Switching was not easy. Not because of TimescaleDB — but because we’d chosen to integrate so tightly with InfluxDB. Our query-processor code was directly sending data to our InfluxDB instance. We explored the option of sending data to both InfluxDB and TimescaleDB directly, but in the end realized an architecture utilizing Kafka would be far superior.</p><p>After learning the ropes of Kafka, setting it up, coding query-processor to send data to Kafka, writing our own golang Kafka consumer to process the Kafka queue, and send data to both InfluxDB and TimescaleDB — we were finally ready to compare the solutions.</p><p>What a difference! From the get-go, we noticed marked improvements:</p><ol><li><strong>Ease of change.</strong> Often a schema change with influxdb meant deleting everything and re-creating from scratch. TimescaleDB supports all standard PostgreSQL alter table commands, create index, and more — enabling fast changes as we improve or add features.</li><li><strong>Performance.</strong> We saw more efficient analytics querying, very low CPU on incoming data. Finally it felt like we had room to breathe and grow.</li><li><strong>No missing data. </strong>Comparing TimescaleDB to InfluxDB at the same time — <strong><em>we realized we were losing data</em></strong>. InfluxDB relied on precisely timed execution of rollup commands to process the last X minutes of data into rollups. Combined with our series of rollups, we realized that some slow queries were causing us to lose data. The TimescaleDB data had 1–5% more entries! Also we no longer had to deal with cardinality issues, and could show our customers every last DNS request, even at a monthly rollup.</li></ol><p>With upcoming GDPR regulations, it also seems TimescaleDB is better able to handle compliance as compared to InfluxDB. With InfluxDB, it is not possible to delete data, only to drop it based on a retention policy. TimescaleDB in comparison, offers a more efficient <a href="http://docs.timescale.com/v0.9/api#drop_chunks">drop_chunks()</a> function, but also allows for the deletion of individual records. This ability to delete particular records allows for compliance with deleting user data if requested.</p><p>The one area TimescaleDB natively lacked, was storage requirements. We saw that it was far less efficient as compared to InfluxDB. For us and our volume, this was not a big deal.</p><p>The current solution to this is filesystem level compression. The TimescaleDB folks recommend <a href="https://blog.2ndquadrant.com/pg-phriday-postgres-zfs/">ZFS compression</a>. After setting it up, we get about a 4x compression ratio on our production server, running a 2TB NVMe.</p><h3>Our results: 10x better resource utilization, even with 30% more requests</h3><p>Here’s a snapshot of system CPU load with InfluxDB from October 2017, compared to February 2018 with TimescaleDB. Also keep in mind that daily processed requests increased 30% between these time periods, and TimescaleDB is run on a slightly slower server.¹</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqkACHUj1uW8vNRQYE1tag.jpeg" /><figcaption>CPU Load Average comparison between InfluxDB and TimescaleDB for our use case</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cbHW_G1ok10DewDZtFqCEw.jpeg" /><figcaption>CPU Load Average difference charted</figcaption></figure><p><strong>As you can see from the charts, even while processing 30% more requests, we’re using 96% less CPU, with far lower maximum load spikes.</strong> If this scales perfectly, and the NVMe can keep up, this means we could potentially handle over 3B requests/day (CPU-wise… We’ll need to add a 2TB NVMe every 1B). This is relevant and helpful for cost-estimating, as we recently quoted a potential customer who would send us a little over 3B requests a day.</p><p><strong>The most surprising improvement in switching to TimescaleDB was in recovered development cycles. </strong>As myself and our backend developer were already very familiar with standard SQL, it has become faster to add new features, because it’s very clear-cut how to add columns, add indexes, and make the changes necessary to roll new features into production. We use a master-master architecture, with Kafka streaming to two production timescale instances. We can shift our API traffic between the instances when doing any database changes, so we can be sure we’re not impacting customers. Many PostgreSQL operations are non-blocking, so we wouldn’t even need to do this, but it’s also just great to have a hot failover.</p><p>We went from InfluxDB’s performance as a daily worry and development headache, to being able to treat TimescaleDB as a reliable component of our architecture which is a joy to work with.</p><p>I’m eternally thankful to the team at TimescaleDB for their help over the last year. Our business would not be the same today if it were not for our switch to TimescaleDB, which they helped make very painless. We’re doing 150M queries a day, and at less than 5% CPU usage. Our global anycast network currently has the capacity to process 20B queries a day if equally distributed. I feel confident we can scale to 3B queries a day or more without much change — and look forward to continuing to grow with TimescaleDB.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Flcontacts.herokuapp.com%2Fembed%2Fbutton%2Fwritercta%3FuserId%3D5aafb288ce3c95045ff59ab5%26mediumUserId%3D1b4c733590e984521ba122b614ae555e3c7eb863cd549d8cc6770f7b7c5a365ed&amp;src=https%3A%2F%2Flcontacts.herokuapp.com%2Fembed%2Fbutton%2Fwritercta%3FmediumUserId%3D1b4c733590e984521ba122b614ae555e3c7eb863cd549d8cc6770f7b7c5a365ed&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=lcontacts" width="480" height="350" frameborder="0" scrolling="no"><a href="https://medium.com/media/06c0709fb279793d91fe99348edf6b4e/href">https://medium.com/media/06c0709fb279793d91fe99348edf6b4e/href</a></iframe><p>[1] These are two different dedicated servers with two different providers. Both have high performance NVMe drives.</p><ul><li>InfluxDB: (<a href="https://serverscope.io/trials/D0GW">serverscope</a>) 48GB Ram, Intel Xeon E5–1650 v2 @ 3.50GHz <a href="https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-1650+v2+%40+3.50GHz">Passmark score: 12658</a></li><li>TimescaleDB: (<a href="https://serverscope.io/trials/erXo">serverscope</a>) 64GB Ram, Intel Xeon E3–1285 v6 @ 4.10GHz <a href="https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1285+v6+%40+4.10GHz&amp;id=3158">Passmark score: 11689</a>, zfs filesystem with lz4 compression</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9f827702f8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing a Dedicated Hosting Provider]]></title>
            <link>https://medium.com/mike-schrolls-blog/choosing-a-dedicated-hosting-provider-46a39a7b8b3a?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/46a39a7b8b3a</guid>
            <category><![CDATA[dedicated-server]]></category>
            <category><![CDATA[dedicated-hosting]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[dedicated-server-provider]]></category>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Tue, 11 Apr 2017 01:00:37 GMT</pubDate>
            <atom:updated>2017-04-11T12:50:29.326Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I was faced with finding a new dedicated hosting provider for <a href="https://dnsfilter.com/">DNSFilter</a>’s central infrastructure.</p><p>We previously used SoftLayer, as I was able to get some startup credits from them. Unfortunately they could not meet our needs for our global anycast network (spoiler alert: They don’t have helpful BGP Community strings).</p><p>They’re also ridiculously expensive. So, facing the reality of spending THOUSANDS per month on our dedicated server with them, I embarked on a mission to find an alternative. This would be a case of knowing how the sausage is made. I use to run a hosting company, with many racks in a datacenter. I’ve used dozens of dedicated hosting providers over the years. I’m an <a href="http://www.webhostingtalk.com/member.php?u=37934">active user</a> on the <a href="http://www.webhostingtalk.com/">WebHostingTalk</a> forum and I’ve co-located servers in a number of datacenters over the years (including the Frontier Datacenter here in Myrtle Beach, SC).</p><p>First, I set out with a requirements list. I needed two dedicated servers, for a core Ruby on Rails API backed by a postgresql database; and another for our InfluxDB / TimeScaleDB Postgresql instance. With the following specs:</p><ul><li>Xeon E3 or E5 CPU (prefer: Higher performing single cores)</li><li>32–64GB+ of RAM</li><li>SSD drive — 120GB minimum, more the better</li><li>1gbit uplink</li><li>KVM or IPMI</li><li>Good support (fast response on hardware failure, or routing issues)</li><li>Good peering (Wanted to see good latency and throughput from Myrtle Beach and east coast)</li><li>Bonus: Provider with multiple geographic locations around the US and globe, so as we expand, I don’t have to keep adding vendors.</li></ul><p>With that list in mind… I started narrowing down providers. Certainly there are MANY providers out there — just have a look at the <a href="http://www.webhostingtalk.com/forumdisplay.php?f=36">WebHostingTalk Dedicated Offers forum</a>. Having spent a lot of time there, I have a feel for who is a budget provider (Dedispec, Delimiter, ReliableHosting) and who are premium providers. For this instance, I wanted to avoid resellers, and focus on premium providers meeting my requirements. I also set a budget goal of ~$150/mo/server.</p><p>Here are the providers, in order of preference, with spec details, and notes on why they are where they are on the list:</p><figure><a href="https://tqhosting.com/"><img alt="" src="https://cdn-images-1.medium.com/max/296/1*0WRofP1mRSwk8gJL2tfKiw.png" /></a><figcaption>The provider we went with</figcaption></figure><ol><li><strong>NetActuate/HostVirtual/</strong><a href="https://tqhosting.com/dedicated"><strong>Tranquil Hosting</strong></a><br>Not only did they give me a great deal, but we already work with HostVirtual for our global anycast network, so it made sense to keep expanding with an existing vendor, and they have a LOT of global locations (though not necessarily instant-deploy dedicated gear everywhere). They have their own staff on-site in Dallas, where our servers are.<br><a href="https://tqhosting.com/checkout/dedicated/5366">The Hardware</a> (<a href="https://serverscope.io/trials/D0GW#system">benchmark</a>, <a href="https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-1650+%40+3.20GHz">passmark</a>):<br>* Dual Intel(R) Xeon(R) CPU E5–1650 v2 @ 3.50GHz<br>* 48GB RAM<br>* 500G SSD + 1TB SATA<br>* 5 IPs, 1gbps, 10TB<br>* IPMI<br>What really took the cake though, was the network performance: Near gigabit performance to Myrtle Beach, NYC, DC, Vultr in Dallas, and Linode in Dallas. Blew me away!</li><li><a href="https://www.totalserversolutions.com/">Total Server Solutions</a><br>Their offerings came recommended from my friend Joe at <a href="http://railsplayground.com/">RailsPlayground</a> who hosts with them. Much respect to Jason from TSS for taking the time to talk to me on Skype. Their offering was available in multiple datacenters in Central US and East coast. I did network testing from Atlanta, Dallas and Chicago, and it performed well. They were willing to work with me on pricing.<br>The Hardware:<br>* Intel Xeon E3–1241-v3 3.5ghz (<a href="http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1241+v3+%40+3.50GHz">passmark</a>)<br>* 32 GB DDR3-ECC-UDIMM<br>* 2 x 256 GB SSD HD<br>* 1IP, 1gbps port, 10 TB Bandwidth</li><li><a href="https://www.packet.net/">Packet.net</a><br>I’ve known these guys for a long time (I was a long-time customer of their previous dedicated host provider, Voxel). I love their offerings. Their network rocks! Their new instant-deploy model is very nice, with up-to-date web UI (web-based serial console!), and a powerful API. Their model is based on a small subset of hardware options. This was the only area where it fell short for me, for this use-case. With Tranquil Hosting I was able to get faster per-core performance, more ram, and greater storage, which I would not be able to get in a single server with Packet. We continue to use their <a href="https://www.packet.net/bare-metal/servers/type-0/">Type-0</a> servers as part of our global anycast network, and look forward to expanding with them. They were willing to work with me on pricing.<br>The Hardware: <a href="https://www.packet.net/bare-metal/servers/type-1/">Type 1</a></li><li><a href="https://phoenixnap.com/">PhoenixNAP</a><br>I’m a past customer, and they now have 7 datacenters around the globe. Their pricing came in a little high compared to the other offers, so I didn’t proceed with testing. From past experience, I was very pleased with their network and support.<br>The Hardware:<br>* Intel Xeon E3–1240v3 (4x 3.4GHz) 2-Bay Chassis<br>* 32 GB ECC DDR3<br>* 2x 240 GB SSD<br>* 5 IPs, 1 Gbps Uplink, 15 TB Transfer<br>* Location: Ashburn, VA</li><li><a href="https://www.psychz.net/">Psychz</a><br>I’ve known of them for years, but considered them a ‘budget’ provider, and I’ve heard tale of various spam and SEO farm bots being operated from their network. They now have a number of datacenter locations, and are working hard to repair their reputation, so I decided to give them a look. The pricing was very competitive, and they offered in Dallas, TX and Asburn, VA.<br>The Hardware:<br>* Quad Cores Intel E3–1230vX Series Server<br>* 32GB ECC DDR3 RAM<br>* 2x 256GB Pro Solid State Drive<br>The network performed well, except to Myrtle Beach… both Frontier and <a href="http://htcbluewave.com/">HTC</a> couldn’t get more than 200mbps. This is a tough one. We don’t have great peering here (<a href="http://mbixp.com/">I’m working on it!</a>), so it can be hit-or-miss. This immediately took them out of the running for me; as my every-day will be spent working on and transferring to these servers; and if I’m paying for gigabit fiber at home; I want to be able to use it to my core infrastructure! (Update: Psychz reached out, made some changes to their routing, and now I can get 450mbps)</li><li><a href="https://www.tier.net/">Tier.net</a> / <a href="https://www.iwfhosting.net/">iWFHosting</a><br>With multiple datacenters, one of which is DC74 in Charlotte, NC, I felt they may provide a great local connectivity option. They also have <a href="http://www.webhostingtalk.com/showthread.php?t=1640336">offers</a> <a href="http://www.webhostingtalk.com/showthread.php?t=1604539&amp;highlight=dc74">on</a> WHT, so the pricing was upfront, and fit in my budget.<br>Unfortunately, their network connectivity to everywhere I could test rarely exceeded 100mbps (but sufficiently exceeded it to indicate to me that it wasn’t limited by a 100mbit port). Very unfortunate, as I’ve heard good things about DC74, and thought it would be cool to have our servers in the Carolinas — Maybe I’d see them one day!</li><li><a href="https://turnkeyinternet.net/dedicated-servers/">Turnkey Internet</a><br>Upfront pricing, least expensive pricing by far, ticks many of the boxes. Only a single datacenter, and not a lot of paths out of their DC. Really only two: Cogent, and a mixed provider blend. I’m not a big fan of cogent bandwidth. They’re not considered a ‘premium’ bandwidth provider. I may use them for cheap compute in the future, but for customer-facing connectivity, I couldn’t justify having our core infrastructure there.</li><li><a href="http://wattaserver.com/dedicatedservers.php">WattaServer</a> / O’Neil Online<br>Found them through a <a href="http://www.webhostingtalk.com/showthread.php?t=1629512&amp;highlight=datacenters">WHT offer</a>. They had multiple DCs, but only in the US, Only a few matching my spec requirements, and on the east-coast. Pricing was good, but network performance was poor. Also felt like a reseller of sorts.</li></ol><p>You’ll note that I left specific providers pricing out, and what the final price was that I went with. Pricing changes so rapidly, it would immediately date this article. You’ll get much more accurate pricing estimates by just reaching out to any providers of interest from this list, and getting a quote yourself!</p><p>I actually considered this a short list, and it took me a little over a week go research all these providers, conduct speedtests, and make a decision. I’m pleased with my decision, but there are a lot of great providers out there!</p><p>Quick rant on ‘cloud’ providers, AWS, VMs: For most startups, AWS is cost-prohibitive in terms of pricing, performance, and knowledge overhead. It’s much easier to stand up a dedicated server, and grow with faster CPUs, more RAM, than to re-code your app to scale prematurely. VMs are great: If you know your constraints on performance… But once you start to get too big a VM; you might as well get a faster, more powerful, and dedicated server, where you do not share resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46a39a7b8b3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mike-schrolls-blog/choosing-a-dedicated-hosting-provider-46a39a7b8b3a">Choosing a Dedicated Hosting Provider</a> was originally published in <a href="https://medium.com/mike-schrolls-blog">Mike Schroll’s Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Live Patching the Linux Kernel]]></title>
            <link>https://medium.com/mike-schrolls-blog/live-patching-the-linux-kernel-4d48f877225?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d48f877225</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <category><![CDATA[patching]]></category>
            <category><![CDATA[ubuntu]]></category>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Mon, 10 Apr 2017 20:10:16 GMT</pubDate>
            <atom:updated>2017-04-10T20:27:17.161Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently came across <a href="http://tier.net/">Tier.net</a> offering <a href="http://kernelcare.com/">KernelCare</a> as an option when you purchase a dedicated server. As a result, I looked into the landscape of live kernel patching (upgrading the kernel without reboot); and here’s what I learned.</p><p>A few years ago efforts were made to integrate and merge a number of different projects into the upstream kernel. As of the 4.x (and specifically 4.4) kernel version, there is now code to perform live kernel patching. But it’s not used by every solution.</p><p>Players:<br><a href="https://www.cloudlinux.com/all-products/product-overview/kernelcare">KernelCare</a> (CloudLinux)<br><a href="http://www.ksplice.com/">Ksplice</a> (acquired by Oracle)<br><a href="https://auth.livepatch.canonical.com/">Canonical (Ubuntu) Livepatch</a><br><a href="https://www.suse.com/products/live-patching/">SuSE Live Patching</a></p><p>Canonical publishes <a href="https://pages.ubuntu.com/rs/066-EOV-335/images/20161017_LivePatching_DS_.pdf">a nice summary</a> of their product offering, and the marketplace, which I’ll use as a reference point.</p><p>Also big thanks to Alex Yevelev, VP of Sales at CloudLinux for contributing to this discussion.</p><p>First, lets start with support for distributions. Canonical supports only the generic kernel branch of 16.04 LTS. SuSE only supports their Enterprise Server 12, and charges $1,499/yr/server. Canonical provides your first 3 free, then charges $150/yr/server.</p><p>Ksplice has slightly broader support: Oracle Linux, Red Hat Enterprise Linux, Ubuntu Desktop and Fedora Desktop. The last two are free, and support a wider array:<br>* Fedora 24<br>* Fedora 25<br>* Ubuntu 12.04 LTS Precise<br>* Ubuntu 14.04 LTS Trusty<br>* Ubuntu 16.04 Xenial<br>* Ubuntu 16.10 Yakkety<br>* Ubuntu 17.04 Zesty (though at the time I wrote this, it 404s)</p><p>It appears as though for a commercial, supported version of Ksplice, you have to pay $2299/yr/server.</p><p>Finally, you have KernelCare.</p><p>They have the most reasonable pricing (after free), and support the largest number of distributions and kernels. I won’t go into everything, but it’s listed on their site, and depending on volume, and if you buy direct or through a reseller, it appears to only cost a few dollars per month, per server, (less than half the price of the next-cheapest: Canonical).</p><p>Here’s an in-depth dive into one of the technologies, kpatch:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVH9DCmeAI3k%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVH9DCmeAI3k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVH9DCmeAI3k%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8eb42a164141f178fe447a225dba6f90/href">https://medium.com/media/8eb42a164141f178fe447a225dba6f90/href</a></iframe><p>Now for those on a budget, dealing with VMs, etc. It does appear as though there’s a work-around to use the ‘free’ Ksplice version on a server (This likely violates Terms, Licenses, etc.).<br>It seems on a server, you can try to install the Desktop ksplice, it will fail due to a missing libgtk2-perl library. Install that. Do apt-get -f install to try to install dependencies again. It will have enough to complete the install of ksplice (ksplice-uptrack) enough to issue you an accesskey token for your machine, which can now be found in /etc/uptrack/uptrack.conf.</p><p>Then apt-get remove ksplice-uptrack, and follow the instructions <a href="https://wiki.ubuntu.com/Ksplice">here</a> to install the non-graphical uptrack. It will use the same config, with access key, and not require a GUI interface. Obviously you’re without support, and I can’t speak to whether the Ubuntu Desktop kernels match the Server Kernels, whether ksplice keeps them up to date, or how well it works.</p><p>I myself am sticking to a combination of KernelCare and Canonical for my server needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d48f877225" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mike-schrolls-blog/live-patching-the-linux-kernel-4d48f877225">Live Patching the Linux Kernel</a> was originally published in <a href="https://medium.com/mike-schrolls-blog">Mike Schroll’s Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Definitive Guide to Content Filtering Circumvention: And Lessons Learned in Prevention]]></title>
            <link>https://medium.com/@MikeSchroll/definitive-guide-to-content-filtering-circumvention-and-lessons-learned-in-prevention-c0f2daf2e104?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0f2daf2e104</guid>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[content-filtering]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[dns-filtering]]></category>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Tue, 17 Jan 2017 22:31:40 GMT</pubDate>
            <atom:updated>2017-01-17T22:31:40.160Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*ImX4Ed03Sm0OswmLbJEzgA.jpeg" /></figure><p>Prior to starting DNSFilter, I spent most of my career in Information Security, working for companies like ITA Software (acquired by Google) and Veracode, finding ways to get around normal system behaviors. As a result, I have a lot of experience in my DNA in getting around and through networks.</p><p>I’m going to step through various methods of getting around network restrictions, whether content filters, firewalls, captive portal, etc. I’ll explain the method, why it works, and what to do to prevent it.</p><h3><strong>Tunnel your traffic via VPN</strong></h3><p>This is probably the first method users think about using. It’s simple and effective; when it works. Negatives are that it adds latency to your traffic, as you’re going through the intermediary network, and the farther away it is from where you are, the slower your Internet usage will be.</p><p>Typical ports to firewall:<br> TCP 1723<br> UDP 500<br> TCP 1701<br> TCP 80<br> TCP 443<br> UDP 1194</p><p>These cover most <a href="https://thatoneprivacysite.net/">commercial VPN services</a>, but realistically, the VPN could be operating on any port., especially if you roll your own, on a VPS like <a href="http://lx.tc/digitalocean">DigitalOcean</a></p><h4>Prevention</h4><p>Basic prevention would be to block VPN ports which are not being used for other purposes (You can’t block 80 or 443 in this easy manner).</p><p>You can also block access to the providers websites (and VPN end-point hostnames) using <a href="https://dnsfilter.com/">DNSFilter</a>’s “<a href="http://help.dnsfilter.com/article/25-preventing-circumvention">Proxy and Filter Avoidance</a>” Threat Category</p><h3><strong>Tunnel your traffic via SSH SOCKS proxy</strong></h3><p>If you have a linux box on the net — whether an inexpensive VPS, or easy-to-use Provider like DigitalOcean, you should be able to ssh -D 8080 user@server and have a conveniently encrypted SOCKS 5 proxy listening on port 8080. Configure your browser of choice, and you’re off to the races.</p><h4>Prevention</h4><p>Most SSH daemons listen on the default port 22; though port 2222 is also becoming a popular alternative to avoid brute-force password attempts. Firewalling these outbound ports will eliminate most attempts; but keep in mind that SSHD can be configured to <a href="https://www.liquidweb.com/kb/changing-the-ssh-port/">listen on any port</a>.</p><h3>Using an HTTP or SOCKS Proxy</h3><p>There are many websites out there which will either list random, anonymous proxies (These come and go quickly!) or provide you a proxy service, similar to VPN providers. These typically provide less security protection than an ssh tunneled proxy; but are easier to set up, as they don’t require you to have a pre-configured server outside the network.</p><h4>Prevention</h4><p>Common TCP ports to firewall are: 8080, 1080, 3128, 8081, 843.</p><p>These can also be blocked with <a href="https://dnsfilter.com/">DNSFilter</a>’s “<a href="http://help.dnsfilter.com/article/25-preventing-circumvention">Proxy and Filter Avoidance</a>” Threat Category.</p><h3>Using a web-based Proxy site</h3><p>Quickly searching the Internet, you can come across websites which will proxy other website content for you. Alternative versions of this include <a href="http://archive.org/">archive.org</a>, google cache, and translation services.</p><p>This typically only works for text and image-based site content; and not interactive applications which require authentication.</p><h4>Prevention</h4><p>These services cannot be firewalled, as they operate on ports 80 and 443, but you can restrict access to some of them with <a href="https://dnsfilter.com/">DNSFilter</a>’s “<a href="http://help.dnsfilter.com/article/25-preventing-circumvention">Translation Sites</a>” Threat Category.</p><h3>DNS Tunnels, ICMP tunnels, etc.</h3><p>As a last-ditch effort on very restrictive networks, some folks will try to tunnel their traffic through other protocols not designed to handle traffic.</p><p>In my experience these are fun to play with, and you could leak small amounts of data, but they’re impractical for standard Internet use due to low bandwidth and high latency.</p><h4>Prevention</h4><p><a href="https://dnsfilter.com/">DNSFilter</a> detects and blocks DNS Tunneling attempts through our recursive resolvers.</p><h3>Summary</h3><p>Certainly there are other prevention measures, and other means of getting around protected networks, but this covers the most common methods. Keep in mind that Security is typically best in an onion-layers of security model: One prevention method will not be 100% effective 100% of the time, but you can increase the hurdles you put in front of malicious users or attackers by implementing some of these methods.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0f2daf2e104" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mission: SSH on all ports.]]></title>
            <link>https://medium.com/@MikeSchroll/mission-ssh-on-all-ports-3aa46d17379b?source=rss-bc88eac5202c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3aa46d17379b</guid>
            <dc:creator><![CDATA[Mike Schroll]]></dc:creator>
            <pubDate>Sun, 04 Dec 2016 15:22:38 GMT</pubDate>
            <atom:updated>2016-12-04T15:22:38.859Z</atom:updated>
            <content:encoded><![CDATA[<p>Mission: SSH on all ports.</p><p>This started out as a simple desire… Sometimes I’m at a location which has restrictive Internet, and I don’t trust it, and I would likely to securely tunnel my traffic. One of the easiest ways to do this is with SSH’s built in socks proxy:<br>ssh user@host -D 8080</p><p>This will setup a localhost:8080 SOCKS proxy which you can also tunnel DNS traffic through. This is easily configured independently in a browser such as firefox. You now have a secure browser whose traffic will appear to come from the remote server.</p><p>Now the problem is that some networks will block port 22. But perhaps they allow 53, or 443, or… some other of the 65535 ports, which I don’t know.</p><p>So; how do I figure out what port I may be permitted out? And even more so; since I cannot connect to my remote server; how do I reconfigure its SSH daemon to listen on that port?</p><p>This set me on a mission: Get OpenSSH to listen on EVERY port. I can then just nmap the host, find out which ports I can connect to, and go about my business of tunnel my traffic over whatever port I happen to find open; which I’m already listening on.</p><p>Well. Easier said than done.<br>First, when configuring ssh, you’ll come across this error:</p><p>too many ports</p><p>Not much documentation on this one; but it can be found in the sources <a href="https://github.com/openssh/openssh-portable/blob/85aa2efeba51a96bf6834f9accf2935d96150296/servconf.c#L1012">here</a>.</p><p>It is limited to 256 ports <a href="https://github.com/openssh/openssh-portable/blob/85aa2efeba51a96bf6834f9accf2935d96150296/servconf.h#L19">here</a>.<br>Once you’re past that hurdle, you’ll get errors about MAX_LISTEN_SOCKS being exceeded. Which is set to only 16 <a href="https://github.com/openssh/openssh-portable/blob/85aa2efeba51a96bf6834f9accf2935d96150296/sshd.c#L174">here</a>.</p><p>Clearly thats not all, as I still haven’t gotten it to work. After that, trying to bind to all 65535 ports, I ran out of ram (ok, it was a small, 256MB VM).</p><p>Trying to bind to ‘only’ 10,000 ports, ssh stops binding after a few hundred ports. No errors; and trying to connect just hangs.</p><p>Turns out this will be a bigger project than I figured. I can keep trying to modify the openssh source; or I could try a few different angles:<br>* port-forward with iptables<br>* use a different daemon to listen on all ports and redirect to openssh on a single port<br>* Use inetd to listen on all ports, then spin up openssh only when someone connects on that port.</p><p>I’ll update once I’ve succeeded!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3aa46d17379b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>