<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Making Instapaper - Medium]]></title>
        <description><![CDATA[Stories from the trenches - Medium]]></description>
        <link>https://medium.com/making-instapaper?source=rss----24982f151313---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Making Instapaper - Medium</title>
            <link>https://medium.com/making-instapaper?source=rss----24982f151313---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:47:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/making-instapaper" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Instapaper Outage Cause & Recovery]]></title>
            <link>https://medium.com/making-instapaper/instapaper-outage-cause-recovery-3c32a7e9cc5f?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c32a7e9cc5f</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 14 Feb 2017 09:08:32 GMT</pubDate>
            <atom:updated>2017-02-14T19:59:23.552Z</atom:updated>
            <content:encoded><![CDATA[<p>The Instapaper service experienced <a href="http://blog.instapaper.com/post/157027537441">an extended outage</a> between Wednesday, February 9 at 12:30PM PT through Thursday, February 10 at 7:30pm PT. We brought the Instapaper service back up with <a href="http://blog.instapaper.com/post/157045376396">limited access to archives</a> as a short-term solution while we worked to restore the service completely. Today, February 14, we completed the full recovery of the service.</p><p>The critical system that failed was our MySQL database, which we run as a hosted solution on Amazon’s <a href="https://aws.amazon.com/rds/">Relational Database Service (RDS)</a>. Here we’ll cover what went wrong, how we resolved the issue and what we’re doing to improve reliability moving forward.</p><h3><strong>Root cause</strong></h3><p>In short, the data failure was caused by a <a href="http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html">2TB file size limit</a> for RDS instances created before April 2014. On Wednesday, February 9 at 12:30PM PT our “bookmarks” table that stores articles saved by Instapaper users exceeded the 2TB file size limit. Subsequent attempts to insert new entries into the bookmarks table began to throw the following error:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9632f3e18d07b36f77ceaf65c4b6dc7a/href">https://medium.com/media/9632f3e18d07b36f77ceaf65c4b6dc7a/href</a></iframe><p>The reason this limitation exists is because MySQL RDS instances created before April 2014 used an <a href="https://en.wikipedia.org/wiki/Ext3">ext3 filesystem</a> which has a 2TB file size limit. Instances created after April 2014 are backed by an <a href="https://en.wikipedia.org/wiki/Ext4">ext4 filesystem</a> and subject to a 6TB file size limit.</p><h3><strong>Instapaper RDS history</strong></h3><p>In April 2013, <a href="https://marco.org/2013/04/25/instapaper-next-generation">betaworks acquired Instapaper</a> from Marco Arment. After the acquisition, we moved Instapaper from <a href="http://www.softlayer.com/">Softlayer</a> to <a href="https://aws.amazon.com/">Amazon Web Services</a> because all betaworks companies run on AWS so the engineers have expertise in that platform. To perform the migration, betaworks worked with two of their regular devops contractors. After the migration, operations were handed off to the newly-formed Instapaper team, and operating responsibilities fell on our director of engineering. After our director of engineering left the company in October 2014, I took over backend operations.</p><p>Our original instance, created in June 2013, was experiencing some performance issues during its backup window in early 2015. AWS support confirmed we were running on older hardware and an older version MySQL v5.6.18. If we upgraded to a more recent version (v5.6.19+), according to AWS, the issue would be resolved.</p><p>In March 2015, we created a read replica of our June 2013 RDS instance, upgraded the MySQL version and performed the cutover switch in the middle of the night with minimal downtime — roughly 5 minutes. Although this instance was created after April 2014, it was created as a read replica from the original June 2013 RDS instance, and, as such, inherited the same filesystem and 2TB file size limitation.</p><h3><strong>Prevention</strong></h3><p>Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue. As far as we can tell, there’s no information in the RDS console in the form of monitoring, alerts or logging that would have let us know we were approaching the 2TB file size limit, or that we were subject to it in the first place. Even now, there’s nothing to indicate that our hosted database has a critical issue.</p><p>If we ever had knowledge of the file size limit, it likely left with the 2013-era betaworks contractors that performed the Softlayer migration. As far as we can tell, there are only two ways that we, as RDS customers, could have prevented this issue.</p><p>First, we could have performed a complete dump of the database to disk, and then restored the database to a new RDS instance. In this scenario, we’d likely need to work directly with Amazon to set the new RDS instance as a read replica of the old one and then perform the cutover.</p><p>The other option would have been to set up a read replica running <a href="https://aws.amazon.com/rds/aurora">Amazon Aurora</a>, Amazon’s managed SQL-compatible database system. We’d previously considered migrating to Aurora (mostly because of the cost savings), however Aurora runs exclusively within a VPC, and Instapaper still runs on EC2-classic.</p><p>Ultimately, it’s unlikely that we would have performed either of those operations without knowledge of the limitation.</p><h3><strong>Backups</strong></h3><p>One of the great features of RDS is automated, daily backups of your database instances. We store 10 days of backups for our MySQL database. However, since those backups are all filesystem snapshots, they’re also subject to the 2TB file size limit.</p><h3><strong>Limited service restoration</strong></h3><p>We didn’t have a good disaster recovery plan in the event our MySQL instance failed with a critical filesystem issue that all of our backups were also subject to.</p><p>After a long phone call with AWS support and discussing the limitation with Pinterest’s Site Reliability engineers, we understood that our only path forward was to rebuild our 2.5TB database using a complete dump and restore. As quickly as possible, Pinterest’s SRE team guided us through dumping Instapaper’s production database to a 5.5TB RAID-0 disk provided by an i2.8xlarge instance.</p><p>When it became clear the dump would take far too long (first effort took 24 hours, second effort with parallelization took 10 hours), we began executing on a contingency plan to get an instance in a working state with limited access to Instapaper’s archives. This short-term solution launched into production after 31 hours of downtime. The total time to create that instance and launch it into production was roughly six hours.</p><p>Given that we didn’t have a plan for this type of event, we didn’t have a good understanding of the time required to dump and restore our database. Our initial estimate for the database dump was six to eight hours. However, we made that estimate using the number of rows, and later learned the first 25 percent of Instapaper bookmarks account for only 10 percent of overall data. We could have drastically cut our initial day of downtime if we understood that rebuilding the database would be a multi-day effort and that we needed to launch directly into limited service restoration.</p><h3><strong>Data restoration</strong></h3><p>Once the service was back up and the data dumps completed, the next step was to import all of the dumps into an instance that wasn’t subject to the 2TB file size limit. We worked closely with RDS engineers throughout the weekend to accomplish this with two parallel workflows:</p><ol><li>Set up an Aurora read replica of the old database. We agreed using Aurora was risky because we hadn’t thoroughly tested it against the Instapaper codebase, but it was very low friction to set up. The read replica completed in about 24 hours.</li><li>Create a new MySQL RDS instance, import all of the data without secondary indexes (8 hours) and create three secondary indexes after the data was imported (each secondary index took roughly 16 hours). At the time of this post, the last secondary index was still being created.</li></ol><p>After realizing that the secondary index creation was taking an unacceptably long time, one of the Amazon engineers mounted an ext4 filesystem to our failed production database and performed an rsync between the ext3 filesystem and ext4 filesystem. The rsync ran in roughly 8 hours, which ultimately got us a new, ext4-backed database instance with all of the data and indexing restored.</p><h3><strong>Syncing with temporary production database</strong></h3><p>Using the binary logs from the temporary production database (with limited archives), RDS engineers set up the new ext4-backed database with row-based replication to the temporary production database in order to sync up with the changes made between Thursday and Monday. The total time for this replication was roughly three hours.</p><h3><strong>Full service restoration</strong></h3><p>Once we had the new ext4-backed database with full data and indexes synced up to the temporary production database, the final step was to promote the new database to master and deploy the application code to point to the new database.</p><p>We performed the restoration without losing any of our users’ older articles, changes made to more recent articles or articles saved after recovering from the outage.</p><h3><strong>Reflections</strong></h3><p>This is any web application developer’s worst nightmare. A filesystem-based limitation we weren’t aware of and had no visibility into rendered not only our production database useless, but all our backups, too. Our only recourse was to restore the data to an entirely new instance on a new filesystem. This was further complicated by the fact that our only interface into the hosted instances is MySQL, which made filesystem-level solutions like rsync impossible without the direct assistance from Amazon engineers.</p><p>Even if we had executed perfectly, from the moment we diagnosed the issue to the moment we had a fully rebuilt database, the total downtime would have been at least 10 hours. Of course, that’s significantly less than the 31 hours of total downtime and five days of limited access, but we’d like to illustrate the severity of this type of issue even in a perfect world.</p><p>We’re firm in our belief that this issue was difficult to foresee and prevent, but the lack of a disaster recovery plan for this type of issue resulted in longer downtimes and recovery times than necessary. Additionally, there were several steps we could have taken from a communications perspective, both internally at Pinterest and with the Amazon Web Services team, in order to better leverage the resources at our disposal.</p><h3><strong>Action items</strong></h3><p>As part of our retrospective process at Pinterest, we’re defining a better workflow for system-wide Instapaper outages that escalate issues immediately to Pinterest’s Site Reliability Engineering team.</p><p>Additionally, we’re going to be more aggressive with testing our MySQL backups. We used to test backups every three months, and now we’ll test every month.</p><p>Neither of the above action items would have prevented this issue, but they’ll help accelerate our response times in the event of an outage and are good practices.</p><h3><strong>Relational database service</strong></h3><p>We’ll continue to use Amazon’s Relational Database Service for now. While it’s frustrating to experience an issue without warning or visibility, RDS has been a reliable and robust service for Instapaper for several years, and handled snapshots, failovers, read replication and other tasks without any engineering overhead from the Instapaper team.</p><p>Additionally, the RDS team has been incredibly helpful in expediting our full recovery. They even took a feature request from us with respect to adding some additional information about ext3-backed databases.</p><h3><strong>Accountability</strong></h3><p>I take full responsibility for the incident and the downtime. While the information about the 2TB limitation wasn’t directly available to me, it’s my responsibility to understand the limitations of the technologies I’m using in my day-to-day operations, even if those technologies are hosted by another company. Additionally, I take responsibility for the lack of an appropriate disaster recovery plan and will be working closely with Pinterest’s Site Reliability Engineering team to ensure we have a process in place to recover from this type of failure in the event it ever happens again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c32a7e9cc5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/instapaper-outage-cause-recovery-3c32a7e9cc5f">Instapaper Outage Cause &amp; Recovery</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rebuilding Instapaper Search]]></title>
            <link>https://medium.com/making-instapaper/rebuilding-instapaper-search-7fae1340699e?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/7fae1340699e</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 17 May 2016 00:37:46 GMT</pubDate>
            <atom:updated>2016-07-19T20:09:03.157Z</atom:updated>
            <content:encoded><![CDATA[<p><em>TL;DR: What follows is a long (good Instapaper material), highly-technical post about re-building Instapaper’s full-text search feature. I’d recommend at least a rudimentary understanding of Amazon Web Services before continuing. If you’d like to read about the product launch, which should be friendly for everyone, you can read the </em><a href="http://blog.instapaper.com/post/143899635741"><em>blog post on the Instapaper blog</em></a><em>.</em></p><p>I’ve been embarrassed about Instapaper’s full-text search for some time now.</p><p>Instapaper’s full-text search is available to Premium subscribers only, and it was originally set up as a Sphinx server to be used in conjunction with Instapaper’s MySQL database. The Sphinx server creates and manages full-text indexes over the MySQL search data, and Instapaper performs full-text search requests directly to the Sphinx server using a SQL-like syntax.</p><p>Instapaper’s full-text search box has been the most fragile and difficult to manage part of the system. Since making the transition to Amazon Web Services, Instapaper’s full-text search has run on a single m2.4xlarge EC2 instance, a memory-optimized instance with ~70GB of RAM. The Sphinx full-text indexes are stored in a 4TB mounted volume, which is a RAID10 array configured as 8 1TB EBS volumes.</p><p>Despite the search service being Instapaper’s largest single point of failure, we’ve run the service with almost 100% uptime over the past three years (with one brief outage a few months ago during scheduled maintenance). However, throughout 2015 we started to get an increasing number of complaints about the service being slow, requests timing out, and other intermittent reports and complaints. A quick inspection of the system revealed the issue:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*seqVhTfXPVvWJV5h2vIn8w.png" /></figure><p>The size of the indexes on disk had grown to 2.2TB, and a majority of the system’s memory was dedicated to holding data from the index files in memory. Because the size of the indexes had grown so large and only a small portion of the indexes could be cached in memory, queries on the indexes resulted in unacceptably long response times and, in some cases, timeouts for the end user.</p><h4>Picking a Solution</h4><p>An easy solution to the memory availability issue would be to set up a new Sphinx box running on an instance with a larger memory footprint. An r3.4xlarge instance would have increased the total system memory by 75% with only an 8% cost increase; however, increasing the memory footprint would have just masked the real issue.</p><p>The crux of the issue is that with the amount of data we have indexed, it’s no longer suitable to run our full-text search with a single machine. In order to continue scaling with the amount of indexed data, the indexes need to be spread across a cluster of search machines. Recreating our Sphinx server on a machine with a larger memory footprint would just be kicking the can down the road a bit further.</p><p>Picking a solution turned out to be easier than anticipated as Amazon Web Services had launched a hosted elasticsearch service in October 2015, which is designed for full-text search across distributed data and with high availability. Additionally, elasticsearch comes with a very simple REST API, excellent first-party libraries, great documentation, and a very active development community.</p><h4>Tuesday 4/19: Setting up the Cluster</h4><p>The goal for configuring the hosted elasticsearch cluster was to find the optimal point between storage, availability, and cost requirements. The parameters for setting up a cluster are number and type of data instances, number and type of master instances, and storage options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVVpJGENqC_iDvuqVeZVzg.png" /></figure><p>As a starting point, I wanted to ensure that the cluster had enough storage space in aggregate to store the existing 2.2TB of data with at least a year’s worth of additional capacity. I decided to create the cluster with 4TB of capacity for the following reasons:</p><ul><li>We currently index ~110GB of data for Instapaper Premium users each month.</li><li>Assuming the rate of data indexed remains constant and the size of the Sphinx indexes are equal to the elasticsearch indexes, that gives us 16 months of additional capacity ((4TB-2.2TB)/110GB=16 months).</li><li>4TB is the size of our existing search server’s storage.</li></ul><p>Each data instance can have an attached EBS volume with a maximum size of 512GB, so the cluster needed 8 data instances in order to achieve the 4TB capacity, and to ensure disk performance I chose provisioned IOPS SSDs with 1,000 IOPS.</p><p>For instance types, I decided on memory-optimized instances since these machines would have to hold a large amount of data in memory, and I settled on r3.large.elasticsearch instance types mainly due to reasonable pricing. The total cost of the cluster is roughly 40% cheaper than what it costs to run our existing search server.</p><p>In order to ensure high availability with fault tolerance, we set up dedicated, zone-aware master instances for the elasticsearch cluster. The default number of master instances suggested by Amazon is three, however, there is currently a 10 instance limit within a hosted elasticsearch cluster. Given that we needed 8 data instances to satisfy our 4TB capacity requirement, we could only proceed with two dedicated master instances.</p><h4>A Caveat on Cluster Size</h4><p>With the current 10 instance limit on hosted elasticsearch clusters, it wouldn’t be possible for us to expand capacity of the cluster without compromising its reliability (i.e. removing one of our two dedicated master instances). As there are storage-optimized instances that come with a 800GB drive attached, I briefly entertained setting up a cluster of five storage-optimized instances to create a 4TB cluster with the option to expand it to 6.4TB. Unfortunately, that would also increase the monthly cost of the service by 60%.</p><p>Essentially, I’m making a wager that it’ll be worth the cost savings over the next year to go with a cluster that is capped at 4TB. Hosted elasticsearch is a new service offered by Amazon and I’m betting that within the next 16 months, the arbitrary 10 instance limit will be lifted. Otherwise, I’ll have to re-create the cluster with the more expensive storage-optimized instances.</p><h4>Thursday 4/21: Indexing First Users</h4><p>After setting up our production cluster, I proceeded to create a development cluster with one instance to run against my development environment, and I wrote a script that would allow me to iterate over a user’s articles, parse each article, and index the result in elasticsearch.</p><p>After indexing the articles on my test account, I set up our front-end search handler to use elasticsearch if the current user is an admin and, otherwise, use the old Sphinx search.</p><p>The final step before going to production was to ensure that as Premium users saved and deleted articles the elasticsearch indexes were updated. When articles are saved or deleted in Instapaper, a number of celery-powered worker tasks are generated to handle additional work. Within the save and delete article code paths I created new worker tasks to update the index in elasticsearch, as well.</p><p>Finally, after deploying the new elasticsearch worker tasks into production, I back indexed all of the admin users articles so we could test the full-text searches against the new elasticsearch cluster. We demoed the performance differences during betaworks’ weekly all hands meeting and, needless to say, it performed way faster than our existing Sphinx setup.</p><h4>Tuesday 4/26: Back Indexing Premium Users</h4><p>The biggest challenge for rebuilding Instapaper’s full-text search was to index the 75 million articles saved by Instapaper Premium users.</p><p>The script I wrote to back index the three admin users’ articles executed linearly by iterating over each admin user, then all of the articles for each user, parsing each article, and adding the parsed text into the elasticsearch index. This process took a few hours for a few thousand articles, but would have taken the better part of a year to do all 75 million articles.</p><p>When writing the back indexing script, the goal was to complete the indexing as quickly as possible using our existing infrastructure without affecting the core service, so I decided the script would simply add jobs to a celery queue and the four worker machines would process those jobs to achieve parallelization.</p><p>To start, I set up a new celery queue <em>bulk_search</em> and created two jobs for that queue. The first job, <em>bulk_update_elasticsearch</em>, receives an article ID as parameter and indexes that article in elasticsearch. The second job, <em>bulk_add_articles_to_elasticsearch</em>, receives a user ID as a parameter, iterates over all of the user’s articles, and creates <em>bulk_update_elasticsearch</em> jobs for each article.</p><p>By creating a separate job queue for these tasks, I would ensure that jobs for our core service wouldn’t be affected by the influx of new jobs for back indexing into elasticsearch. I then configured the four worker machines to run 15 processes each to execute jobs from the <em>bulk_search</em> queue.</p><p>Lastly, I wrote the back indexing script to iterate over all of the Instapaper Premium users, queueing a <em>bulk_add_articles_to_elasticsearch</em> job for each user, and added a bit of code that would cause the script to sleep for 30 seconds if the <em>bulk_search</em> queue had more than 2,000 jobs in it.</p><p>I started running the back indexing script on the machine we use for scheduled tasks at around 4PM that afternoon, and continued monitoring the tasks until leaving work that evening. Here’s a backend diagram that illustrates the pieces of the system that handle our redis-backed celery jobs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7DvmnMzibv34013-P46EA.png" /><figcaption>Instapaper Worker Architecture. We use redis-backed celery to queue long running jobs that are processed by the workers. Search indexing jobs were added via our scheduler machine.</figcaption></figure><h4>Wednesday 4/27: Dedicated Search Workers</h4><p>At around 10AM I looked at the past 18 hours’ worth of data and found we had only back indexed just under a million articles, 1.3% of the total. At that rate, it would take roughly two months to index the remaining 74 million articles. I also found that the <em>bulk_search</em> job queue had over 2 million jobs in it that were slowly draining, which was causing the back indexing script to sleep for long periods of time.</p><p>In order to increase the indexing rate to something more acceptable, it was necessary to spin up dedicated worker instances that would only process jobs from the <em>bulk_search</em> queue. I decided to go with EC2 spot instances because I didn’t need these instances to stay around for long, if they were turned off it wouldn’t be critical, and it would be cost effective.</p><p>I placed four spot instance requests for dedicated workers, set up new configuration targets for the dedicated search workers, and configured the new targets to run 105 <em>bulk_search</em> processes per machine.</p><p>My major concern with scaling up the dedicated workers was causing overhead to our <a href="https://www.instaparser.com/">Instaparser</a> infrastructure, which is shared with Instapaper and Instaparser customers. Around 1PM, I added two of the dedicated search workers to bring the number of workers from 60 across four machines to 270 across six machines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVt4pW2hfOhDaVQI6VkzWQ.png" /><figcaption>Instapaper Worker Architecture after adding two dedicated search workers.</figcaption></figure><p>After adding the search workers, I continued to monitor the total time for Instaparser API calls:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*9-cRYbCA97o0BlE7-jFuxA.png" /><figcaption>Time for an API Call on Instaparser, y-axis is in milliseconds</figcaption></figure><p>From 1PM through 8PM there was a 33% increase in total time for processing Instaparser API calls, and I tracked the increase in time to parses on older URLs, whose domains which were taking 3–6 seconds to respond to requests. It did not appear that this increase affected the core Instapaper service, and I called it a day around 8PM.</p><h4>Thursday 4/28: Enterprise Instaparser</h4><p>When I arrived in the morning, I logged in to the Instaparser dashboard and I was happy to see we completed over 4 million parses on the 27th:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*qZyDP5sJe7WpaccqHp7hWw.png" /><figcaption>4.36M Parses on 4/27</figcaption></figure><p>Having dedicated workers increased the new rate of indexing from 1 million articles to 6.5 million articles every 20 hours. At the new rate, it would only take about 11 days to complete the remaining 68 million articles. However, I also found that the metrics around total parsing time had ballooned from an average of 1.1 seconds to over 1.5 seconds overnight:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*ul3pnzFyZWas7LF5yfT5Rg.png" /></figure><p>Due to concerns about increased parsing times affecting the main service and becoming a bottleneck for the back indexing process, I decided to create a dedicated Instaparser cluster that would only be used for the search back indexing. Effectively, this is a similar set up that we are offering for the Instaparser Enterprise plan.</p><p>Again, I used spot instances to set up another 8 servers: 3 to operate the Instaparser API which acts as a proxy to 5 internal parsers. Here is an illustration of the Instaparser architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xwxA8DwnnPcpCKZ3d-0Jtw.png" /><figcaption>Instaparser Architecture: API servers act as a proxy to the parsers, and perform some minimal business logic.</figcaption></figure><p>I set up “search” configurations for both the Instaparser and the Instapaper applications that had the same configurations as our normal production settings, except they were both configured to use the dedicated search infrastructure.</p><p>After completing the Instaparser Enterprise cluster and testing it, I set the dedicated search workers to use the new “search” configuration and restarted the processes, continuing to monitor the total time for API calls for our production Instaparser service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*dat_9vMjQ6N8vtvSjFIGUw.png" /></figure><p>And also the total time for API calls on our new “search” Instaparser Enterprise service:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*JlD9X0PnQY-0nRfmln5kFg.png" /></figure><h4>Friday 4/29: Limits of Scaling</h4><p>In the morning, I took another look at how many parses we were now able to do with the dedicated parsing infrastructure, and on the 28th we parsed over 8M articles, compared to 4M the day before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*d_wKFGrfu4AoWGGSyreAOw.png" /><figcaption>8.25M Parses on 4/28</figcaption></figure><p>The Instaparser Enterprise service allowed us to increase our rate of indexing by 20%, from 6.5 million to 7.8 million, in a 20 hour window. With the increase of rate in indexing, it would take another 8 days to parse and index the remaining 60 million articles.</p><p>Given that the search indexing and parsing were now running on totally dedicated services, I decided it would be good to add a third dedicated search worker to increase the rate of parsing even further. The additional machine would increase the number of worker processes by 40%, up to 375 from 230, and the configuration is illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPFewjUsV3Bvlv5322_q8g.png" /><figcaption>Instapaper Worker Architecture after adding a third dedicated search worker.</figcaption></figure><p>After adding the additional search worker machine, I began monitoring all of the systems when I noticed that we were starting to get a high volume of errors coming from the <em>bulk_update_elasticsearch</em> worker tasks:</p><blockquote>EsRejectedExecutionException[rejected execution (queue capacity 200) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction</blockquote><p>It appears that the additional search workers were now causing the hosted elasticsearch cluster to reject some of the index requests because we were exceeding the allowed queue capacity. In short, we were now parsing and indexing faster than our hosted elasticsearch could write the indexes, and some portion of indexed articles were failing.</p><p>Some light Googling showed that elasticsearch settings can be tweaked to increase the number of threads, queue size, etc. to alleviate the above issue. However, at this time those options are not available in the hosted elasticsearch settings. After determining that only .02% of indexed requests were failing and making sure that the error logs were available to re-index those articles, I decide to keep the third worker running over the weekend to accelerate the rate of indexing even further.</p><p>Ultimately, the limit to our scaling was the hosted elasticsearch cluster itself.</p><h4>Monday 5/2: Search Features</h4><p>Adding the third search worker machine on Friday increased our rate of indexing by an additional 18%, bringing us to an average 11 million articles parsed and indexed per day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*PxJ7SYCRzQ_tthMuJRRqNQ.png" /><figcaption>4/29–5/1 Parsed 11M Articles per day on Average</figcaption></figure><p>That brought the total number of indexed articles to 47 million, with only 28 million articles remaining to be indexed. With 11 million articles being parsed and indexed today, that meant we would finish back indexing all of the Premium users’ articles by mid-day Wednesday.</p><p>After updating the team with the timeframe, to completing this task, we set a tentative launch date for Thursday 5/5. Rodion Gusev, who heads up support, community, and copywriting, took the lead on drafting the launch blog post and Aaron Kapor, our design director, had been working on mockups for some additional search features made possible by elasticsearch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgJreOXS4CZsMM-pEUKz-g.png" /></figure><p>The main features we were planning on adding were paginated results (previously we only showed the most recent 50 matches), sorting options for relevance or chronology (previously only reverse chronological), and advanced filters for searching by author or domain.</p><h4>Thursday 5/5: Launch Day</h4><p>Over the previous few days we had finished editing the blog post, implemented all of the additional search features, and completed back indexing all of the Premium users’ articles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*HtkGqXQqH12y3hKkIY3k3Q.png" /></figure><p>We scheduled one of betaworks’ QA interns to come in Thursday morning and run through a checklist and attempt to find some corner case issues. After running through the checklist and only finding one minor issue, we decided to move ahead with the launch and address the issue after launching.</p><p>Here’s a graph showing the total time for search results before and after launching:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*8tMwzpLgyePpibssTSBP9Q.png" /><figcaption>Frontend Search Total Time, y-axis in milliseconds</figcaption></figure><p>Our search went from an average query time of 3 seconds (horribly embarrassing) to 250 milliseconds with several new features that really improved the quality of the full-text search product.</p><h4>Friday 5/6: Winding Down &amp; Learnings</h4><p>After launching on Thursday, I spent most of Friday cancelling our spot instance requests, winding down the workers and the Instaparser Enterprise cluster, and generally performing the rest of the clean up required after from scaling up for the back indexing of all 75 million of Instapaper Premium users’ saved articles.</p><p>Projects like this are part of the reason I really love working on Instapaper. I learned a ton about search software like Sphinx and elasticsearch. I went from not knowing anything about setting up an elasticsearch cluster to making cost and capacity decisions on a 4TB cluster, indexing over 2.2TB worth of data in the cluster, and leveraging a lot of elasticsearch features to set up new consumer-facing features for our search offering. And that was all in just 16 days!</p><p>I also learned a ton about the Instapaper system, how to scale up quickly, where the bottlenecks in the service are under load, and how to overcome those bottlenecks to push further performance in the system.</p><p>I’d love to hear your thoughts, questions, or criticisms in a response below, via email to <a href="/service/mailto:brian@betaworks.com">brian@betaworks.com</a>, or on <a href="http://twitter.com/bthdonohue">Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fae1340699e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/rebuilding-instapaper-search-7fae1340699e">Rebuilding Instapaper Search</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Instapaper Tweet Shots]]></title>
            <link>https://medium.com/making-instapaper/instapaper-tweet-shots-5df8587988e8?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/5df8587988e8</guid>
            <category><![CDATA[typography]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[twitter]]></category>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Fri, 27 Mar 2015 18:05:53 GMT</pubDate>
            <atom:updated>2015-03-27T19:49:56.775Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In 72 hours…</em></p><p>It’s amazing how a (somewhat) slight product change can define a whole new set of behaviors. On October 29th 2013, Twitter launched inline image previews directly in the timeline, allowing users to view images without clicking through a link. In the one year since inline images launched, there has been a trend of sharing screenshots of text (usually from mobile devices) to Twitter, which allows the user to share specific parts of an article without exceeding the 140 character limit.</p><p>At betaworks, we’ve been curious about this trend since it began and, by now, it has become pretty hard to miss if you’re working in technology. Some of the most prominent <a href="https://twitter.com/fmanjoo/status/535050216689659906">journalists</a> and <a href="https://twitter.com/cdixon/status/535245781000011776">venture capitalists</a> have been tweeting text shots, and with good reason. Look at the engagement difference between these two tweets about the same article:</p><h3>Chris Dixon on Twitter</h3><p>Nikola Tesla predicted the iPhone - &quot;A man will be able to carry one in his vest pocket&quot; http://cdixon.org/2014/11/19/a-man-will-be-able-to-carry-one-in-his-vest-pocket/ ...</p><h3>Chris Dixon on Twitter</h3><p>Nikola Tesla predicting today back in 1926 pic.twitter.com/5Air6JEULu</p><h3>Sunday, February 22nd</h3><p>I received a brief email from <a href="http://twitter.com/borthwick">John Borthwick</a> (CEO at betaworks):</p><blockquote>Could we add a share from Instapaper where I share an image of a highlight (however long it is) and we figure out the URL formatting etc. to make it clear that its an image of a highlight from Instapaper. Sharing would be to Twitter and Instagram.</blockquote><p>I had spoken with Instapaper designers Justin Van Slembrouck and Aaron Kapor in the past about doing this, but the priority was always lower than some other, more pressing task. However, with the rise of tweet shots, <a href="http://oneshot.link/">apps</a> being made to assist in the creation of those screenshots, and an imminent <a href="http://blog.instapaper.com/post/114681736471">Instapaper iOS release</a>, it made sense to take a day or so to try and build a tweet shot feature for Instapaper.</p><p>We started by looking at existing behaviors and identifying some obvious issues with the current process:</p><ul><li>It’s cumbersome: Creating a tweet shot takes about 10 steps.</li><li>Hard to read: At times, the original image is too large for Twitter, and the results are tiny font sizes, weird line heights, and too many or too few characters per line.</li><li>Not glanceable: Tweet shots often do not fit within the Twitter image preview, which is further complicated by the fact that the image previews are slightly different on desktop and mobile.</li><li>Clutters the camera roll: Because all of the existing solutions required a screenshot before image processing, the person’s camera roll becomes cluttered with screenshots.</li></ul><p>With the above in mind, we set out to make sure of the following for Instapaper tweet shots:</p><ul><li>Beautiful typography with an emphasis on readability that, whenever possible, wouldn’t require expanding the image.</li><li>No screenshots cluttering your camera roll.</li><li>No more than three taps to send to keep things really easy.</li></ul><h3>Monday, February 23rd</h3><p>The first thing we nailed down was the three-tap send, which could actually be done in just two taps by users that had already created a highlight of the text they wanted to share:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-2jI1O1zfICbk6TjP5kHbg.png" /></figure><p>We were able to achieve three-tap tweet shots by doing the following:</p><ul><li>The image is generated programmatically when you tap the “Tweet Shot” button. The app sets up the aspect ratio and type properties automatically rather than have the user crop and adjust a screenshot.</li><li>No “preview” version of the image. You’ll get a little thumbnail preview before you hit the “Post” button on the Twitter share sheet, but that’s it.</li><li>Use Instapaper’s built-in customization to generate the image using the current font and background color.</li></ul><h3>Tuesday, February 24th</h3><p>After some trial and error, we managed to finalize the layout of our tweet shots with the text in the user’s current font and article metadata &amp; attribution in HelveticaNeue:</p><h3>Test on Twitter</h3><p>pic.twitter.com/TilgAdhU9Y</p><p>Then, we started working through some edge cases…</p><p><em>What happens if you only select one sentence?</em></p><h3>Test on Twitter</h3><p>http://www.nybooks.com/blogs/nyrblog/2014/dec/03/weapon-for-readers/ ... But beyond these small technicalities pic.twitter.com/7hKLiu0ujl</p><h3>Test on Twitter</h3><p>http://www.nybooks.com/blogs/nyrblog/2014/dec/03/weapon-for-readers/ ... But beyond these small technicalities pic.twitter.com/N0tOu1XRPf</p><p><em>What happens if I choose a very long selection?</em></p><h3>Test on Twitter</h3><p>Such wall of text pic.twitter.com/FQzQg4YZ6b</p><p>Lastly, we started testing a ton of different font and background combinations to determine if there were any weird edge cases:</p><h3>Test on Twitter</h3><p>http://runemadsen.com/blog/on-meta-design-and-algorithmic-design-systems/ ... pic.twitter.com/bVkRQBcaOi</p><h3>Test on Twitter</h3><p>pic.twitter.com/wifywAbJ2C</p><h3>Wednesday, February 25th</h3><p>With the broad strokes finished, our designer Aaron and I embarked on the painstaking task of going through every font in Instapaper and adjusting line heights &amp; font size to make sure the images were perfectly set. After we completed all of the typography tweaks, I went through each of Instapaper’s 14 fonts, took screenshots for one sentence, two sentences, and a paragraph, and collected the generated image, the inline desktop view, and the inline mobile view for each screenshot in each font.</p><p>Once I collected all of the images, I sent them to Aaron for a final review. You can find the deliverable I sent him <a href="https://www.dropbox.com/s/vfu2fage1dzdkpm/TweetShots%20%2B%20Mobile.zip?dl=0">here if you’re interested</a>.</p><h3>Thursday, February 26th</h3><p>We shipped the release version of Instapaper 6.2 to our TestFlight testers (tweet shots were one of four major features in this release) and, after a week, submitted to the App Store for review. Here are a few of the earliest tweet shots from Instapaper testers:</p><h3>ryan on Twitter</h3><p>Testing @instapaper tweetshots. This makes my life much easier. http://en.wikipedia.org/wiki/Fenway_Park ... pic.twitter.com/yYRjU7yY92</p><h3>Rob Malanowski on Twitter</h3><p>Time sensitive Dropbox links http://plobo.net/time-sensitive-shared-links ... pic.twitter.com/YJduWwalws</p><h3>Joe Caiati on Twitter</h3><p>Thank you @bthdonohue! This is a real treat. cc @mgsiegler http://www.newyorker.com/magazine/2015/02/23/shape-things-come ... pic.twitter.com/UcMEJvkhl7</p><h3>Daniel H. on Twitter</h3><p>I do have a soft spot for Weightbot as well and hope @tapbots will finally have time. :-) http://tapbots.com/news/15-2-26-new-website-new-focus/ ... pic.twitter.com/0gWk5eX1ML</p><p>Daniel H., whose tweet is featured above, did something pretty interesting with Instapaper’s tweet shots. In reply to the above tweet, I wrote:</p><blockquote><a href="https://twitter.com/dahanbn"><em>@dahanbn</em></a><em> yes they are very handsome, lends itself to our strengths: simplicity &amp; beautiful typography.</em></blockquote><p>Daniel then saved that tweet to Instapaper, and created a tweet shot of my reply:</p><h3>Daniel H. on Twitter</h3><p>Even possible to tweetshot tweets via @instapaper. Well done, @bthdonohue! https://twitter.com/bthdonohue/status/571030105843245058 ... pic.twitter.com/Isylq9HD2D</p><p>Twitter recently reintroduced embedded tweets in the timeline, but as far as I can tell, this is a totally original use of tweet shots, and it’s always really exciting to see someone use your product in a way that wasn’t explicitly intended.</p><p>Also, it looks amazing inline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CG7FWCSV6xTdENdMjkYMtA.png" /></figure><p>If you’ve made it this far, you should follow the <a href="https://medium.com/making-instapaper">Making Instapaper</a> publication on Medium where we’ll be blogging regularly about the work we’re doing on Instapaper. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5df8587988e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/instapaper-tweet-shots-5df8587988e8">Instapaper Tweet Shots</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Instapaper Free]]></title>
            <link>https://medium.com/making-instapaper/making-instapaper-free-3844af1606ee?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/3844af1606ee</guid>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 24 Mar 2015 19:17:00 GMT</pubDate>
            <atom:updated>2014-10-10T18:23:25.932Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inshWfY7-6f8e0-Fq3bdFA.jpeg" /></figure><p>Toward the end of January 2014, <a href="https://twitter.com/mcandrew">Andrew McLaughlin</a> (CEO Digg &amp; Instapaper), <a href="https://twitter.com/borthwick">John Borthwick</a> (CEO betaworks), and I had our first meeting since my promotion from lead iOS developer to General Manager of Instapaper to discuss the company’s roadmap. Since joining the Instapaper team in June 2013, I had advocated internally to eliminate Instapaper’s paid download and, instead, focus on adding value to the subscription. At the time, Instapaper’s subscription was priced at $1 per month, and the primary feature offered was full-text search.</p><p>I put together a few slides for the meeting to give some structure &amp; narrative to my argument for going free, and it’s interesting to view those through the looking glass of the past year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxWeUtSkNfKuarnh2jqb_Q.jpeg" /><figcaption>iPhone App Revenue by Source (via TechCrunch)</figcaption></figure><p>Revenue from the iOS App Store is shifting away from paid-up-front applications to free applications with in-app purchases. The above chart shows a breakdown of revenues by source from the App Store, and it’s fairly easy to see where the growth in revenue is coming from. It’s definitely not app downloads.</p><p>The major issue with this breakdown is that it includes revenue from games, which are a major anomaly compared to other categories in the App Store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EG9ANHoUacLHwkPqtg2eSw.jpeg" /><figcaption>Growth in Freemium (via AppAnnie)</figcaption></figure><p>It’s tough to find vertical-focused analysis of App Store revenue, but the above slide separates revenue growth into “Games” and “Apps Excluding Games.” The data surrounding growth in games is huge (~125% YOY) and, given the huge growth in revenue for mobile games, I’m really excited for fellow betaworks company <a href="http://weplaydots.com/">Dots</a>. Perhaps less obvious, though, is that sometime in 2013, revenue from “other” mobile apps saw an inflection point between paid downloads and in-app purchases. 2013 marked the first time in the history of the App Store that in-app purchases generated more revenue than paid downloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G3aDyaE4SSC7V-dDFOUo3g.jpeg" /><figcaption>Top Grossing in News (via iTunes)</figcaption></figure><p>In an attempt to analyze revenue solely from the News category of the App Store, I did an analysis of the top grossing apps in the News category at the time. Of the twenty-two top grossing iPhone apps, 68% were free applications with in-app purchases. Looking only at the fifteen free apps in the twenty-two top grossing, 80% were Newsstand applications. The major difference between Newsstand apps and regular apps is that Newsstand apps all monetize on subscriptions from renewable in-app purchases. One possible conclusion here is that recurring revenue from renewable in-app purchases is driving the majority of grossing sales in the News category of the App Store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rck5HzcUrohg9DmYzD6UbQ.jpeg" /><figcaption>2012–2013 Monthly Downloads (source: AppFigures)</figcaption></figure><p>This is the most damning chart for any argument to keep Instapaper as a paid application. You can see a nose dive in Instapaper’s app downloads immediately following the launch of Pocket. <a href="http://twitter.com/marcoarment">Marco Arment</a>, founder of Instapaper, wrote a great blog post titled <a href="http://www.marco.org/2013/09/28/underscore-price-dynamics">Underscore Price Dynamics</a> back in September 2013, which brings up a lot of important points about the state of paid applications:</p><ol><li>Almost everyone, when presented with a paid-up-front app, will first seek a free alternative.</li><li>Customers hate the current method of paid “upgrades” (pulling the previous version from the store and putting up a new, separate paid-up-front app).</li><li>These objections don’t apply nearly as much to in-app purchase.</li></ol><p>What’s interesting to me from the above chart is the orange line, which represents Instapaper app updates. While the blue line shows a steep and steady decline of growth in app downloads, the orange line shows the size of Instapaper’s install base, which is significant. Unfortunately, there’s no great path to a paid upgrade if an app developer launches an update with a bunch of great features.</p><p>The only options for monetizing an existing install base are advertising (i.e. Facebook), creating a “new” app and converting your existing install base to purchase the “new” app (i.e. Tweetbot), or offering some type of subscription or consumable via in-app purchase (i.e. Evernote).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NzcDd1e8zThtSvIJK6Hcjw.jpeg" /><figcaption>Feature Rubic for Instapaper’s 2014 Roadmap</figcaption></figure><p>At the time, there were a lot of ideas floating around for Instapaper’s direction and which features to prioritize. In the interest of moving toward a more sustainable business, I advocated that we focus on adding as much value to our existing subscription model, with a short-term focus on features that fit the following criteria:</p><ol><li><strong>User Demand. </strong>We should be prioritizing the features our users are asking for.</li><li><strong>Competitive Advantage. </strong>I wanted to make sure we were leapfrogging the competition, not chasing them.</li><li><strong>Subscription Value. </strong>Potential for including the feature in the subscription package.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UYlcTD8xEeEtQ-hxH9SkLQ.jpeg" /><figcaption>Instapaper Premium proposal</figcaption></figure><p><a href="http://twitter.com/jvanslem">Justin Van Slembrouck</a> (designer Digg &amp; Instapaper) and I had previously discussed focusing on features for Instapaper’s subscription, and eventually increasing the price of the subscription and going free. Justin suggested that if we do that, we should rethink “Instapaper Subscription” and rebrand it to reflect to value of the new offering. We kicked around a few names, but settled on <a href="http://instapaper.com/premium">Instapaper Premium</a>, which just felt right. Instapaper Premium launched in September and contained a majority of the features from January’s proposal slide.</p><p>At the end of the meeting, we all agreed that this was the direction we needed to take to create a sustainable business for Instapaper. Months later at WWDC, I was able to catch up with Marco, and his response to our plan with Instapaper Premium was, “I think, ultimately, Instapaper will be the most successful as a freemium service.” Even though I advocated this decision and had the blessing of the betaworks higher-ups and Instapaper’s founder, making Instapaper free made me nervous. Revenue from paid downloads still made up a significant percentage of Instapaper’s quarterly revenue, but I knew we needed to forego short-term vanity metrics in the interest of creating a lasting, sustainable business. We owe that much to our users.</p><p>Within three weeks of launching Instapaper Premium, 4% of our subscribers have purchased at the new price point. As a way to show our appreciation for our existing subscribers, we grandfathered all of them at the previous price point ($1 per month), and it was great to see some support emails from people that wanted to cancel their original subscription in favor of purchasing Instapaper Premium at the new price point.</p><p>Our major challenge moving forward will be replacing revenue from app downloads with Instapaper Premium subscriptions, and growing the business beyond the revenue we’ve foregone. To do so, we’ve been executing a dead-simple strategy: implement the features our customers are asking for. You could practically recreate our roadmap by aggregating requests coming into Instapaper support and sorting by descending frequency. From highlights to text-to-speech, it’s worked incredibly well so far!</p><p>I’d love to hear your thoughts in the comments, or on <a href="http://twitter.com/bthdonohue">Twitter</a>!</p><p><em>Thanks to </em><a href="https://twitter.com/catzo"><em>Aaron</em></a><em>, </em><a href="http://twitter.com/andrewdumont"><em>Andrew</em></a><em>, </em><a href="http://twitter.com/loversmademen"><em>Rodion</em></a><em>, and </em><a href="https://twitter.com/_roySD"><em>Suman</em></a><em> for reading drafts of this post.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3844af1606ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/making-instapaper-free-3844af1606ee">Making Instapaper Free</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hacking TestFlight Invites]]></title>
            <link>https://medium.com/making-instapaper/hacking-testflight-invites-15cc96664b97?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/15cc96664b97</guid>
            <category><![CDATA[testflight]]></category>
            <category><![CDATA[beta]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 24 Mar 2015 15:40:41 GMT</pubDate>
            <atom:updated>2015-03-03T23:12:51.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YUI77blROymvCU5Mz6aBBw.png" /></figure><p>Beta distribution on iOS has always been a pain in the ass. At the end of 2010, TestFlight launched a solution that made iOS beta distribution considerably less painful. TestFlight provided a platform that allowed developers to manage a set of beta testers, upload beta versions of their apps, distribute those betas to the testers, and manage feedback from beta testers. TestFlight was a godsend for iOS developers.</p><p>Even with TestFlight, beta distribution on iOS was far from perfect. It was still cumbersome to manage testers’ devices properly, and developers were still subject to Apple’s 100 device per year limit. Apple obviously understood that beta distribution was still an area that needed lots of improvement because they acquired TestFlight in early 2014, and launched an official TestFlight integration in October 2014.</p><p>We beta tested <a href="https://itunes.apple.com/us/app/instapaper/id288545208?mt=8">Instapaper 6.1</a> using Apple’s TestFlight integration, and it’s incredibly well done. The new integration solves a lot of the pain points that the original TestFlight couldn’t address. Developers no longer have to manage a list of tester’s iOS devices, the 100 device limit was changed to a 1,000 user limit, and the entire process is more streamlined and straightforward. However, the old TestFlight did one thing much better than the new TestFlight integration: user invites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ya9nGBr3u88GkXY1BFq1A.png" /><figcaption>Old TestFlight recruitment page</figcaption></figure><p>With the old TestFlight you could manually add users via email, but TestFlight also gave you a URL that you could share anywhere in order to recruit beta testers for your app. With the new TestFlight you only have the option to add testers via email, or bulk inviting testers by uploading a CSV file with email addresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQweFCSCxSn_9B3VkKJtnA.png" /><figcaption>New TestFlight invite page</figcaption></figure><p>Munging CSV files and manually entering email addresses is a hassle, and a big step back from the days of “Want to beta test? Sign up <a href="https://testflightapp.com/join/4e94d263e2fcc0a131cd4cc715a402b1-MjE5ODU4/">here</a>!” I wanted to make recruiting beta testers for <a href="https://www.instapaper.com/">Instapaper</a> easy again by allowing people to visit a URL, sign up for beta testing, and have their information automatically entered into Apple’s new TestFlight solution.</p><h3>Research &amp; Implementation</h3><p>So I started by adding a tester in iTunes Connect and inspecting the page’s network requests to find the API call &amp; payload Apple is expecting when adding a new tester to TestFlight:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*3Y9GiwA184_V_EixxTF_EA.png" /></figure><p>Pretty straightforward. The next step was to figure out how to programmatically login to iTunes Connect, and manage the authorization cookies in the above request. Luckily, that problem had already been solved! I found a <a href="https://github.com/kirbyt/appdailysales/blob/master/appdailysales.py">great script</a> written by <a href="https://twitter.com/kirbyt">Kirby Turner</a> that programmatically downloaded app sales data from iTunes Connect by first logging the user in, and then performing the necessary web requests. Jackpot!</p><h3>Making it Usable</h3><p>Once I modified the appdailysales.py script to successfully add a new user to TestFlight, I basically just needed to set up a page for Instapaper users to submit their email address &amp; name (optional), and integrate my testflight_invite.py script into our code base. I’m pretty happy with the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJZH-GZ3T3d34jYI68sHJg.png" /></figure><p>I’ve put the <a href="https://github.com/Donohue/testflight_invite">code up on GitHub</a> in case you’re interested in playing with it or scaffolding your own TestFlight invite page.</p><p>Oh! One more thing! If you want to help beta test Instapaper on iOS you can <a href="https://www.instapaper.com/testflight">sign up here</a>!</p><p><em>Edit:</em> Within 5 hours we had 326 beta testers signup and the invite page is now closed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15cc96664b97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/hacking-testflight-invites-15cc96664b97">Hacking TestFlight Invites</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bookmarklets are Dead…]]></title>
            <link>https://medium.com/making-instapaper/bookmarklets-are-dead-d470d4bbb626?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/d470d4bbb626</guid>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 24 Mar 2015 15:40:26 GMT</pubDate>
            <atom:updated>2014-10-23T21:34:47.735Z</atom:updated>
            <content:encoded><![CDATA[<p>… we just don’t know it yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*oEzJGP6rR1dOUw4bEmQHLA.gif" /></figure><p>I first encountered the Content Security Policy HTTP header earlier this year when one of our users reported the <a href="https://www.instapaper.com/">Instapaper</a> bookmarklet wasn’t working on GitHub. Triggering the bookmarklet, and inspecting element on the page revealed the problem:</p><pre>Refused to load the script ‘https://www.instapaper.com/j/redacted?u=https%3A%2F%2Fgithub.com%2Fcocoapods%2Fcocoapods&amp;t=1414077850205&#39; because it violates the following Content Security Policy directive: “script-src assets-cdn.github.com collector-cdn.github.com”.</pre><p>A cursory glance at the Content Security Policy specification revealed the problems. Proposed as a solution to a suite of cross-site scripting (XSS) vulnerabilities, Content Security Policy allows web authors to specify the origin hosts of web content, this includes:</p><ul><li>JavaScript (script-src)</li><li>AJAX Calls (connect-src)</li><li>CSS (style-src)</li><li>Fonts (font-src)</li><li>Frames (frame-src)</li><li>Images (image-src)</li><li>Video &amp; Audio (media-src)</li><li>Flash (object-src)</li></ul><p>The crux of the issue for bookmarklets is that web authors can control the origin of the JavaScript, network calls, and CSS, all of which are necessary for any non-trivial bookmarklet. The W3C specification for Content Security Policy 1.0 has a specific clause to protect end-users from this type of authoritarian control:</p><blockquote>Enforcing a CSP policy should not interfere with the operation of user-supplied scripts such as third-party user-agent add-ons and JavaScript bookmarklets.<br>— <a href="http://www.w3.org/TR/CSP/#processing-model">W3C Content Security Policy 1.0, Section 3.3</a> November 15th, 2012</blockquote><p>The above clause treats browser extensions and bookmarklets as trusted components of the browser ecosystem, and intentionally prevents web authors from blocking the functionality provided by components specifically added and initiated by an end-user.</p><p><a href="https://www.w3.org/Bugs/Public/show_bug.cgi?id=23357#c0">Some have contended</a> that bookmarklets and add-ons can not be trusted and can be compromised by privilege escalation or other security vulnerabilities unrelated to cross-site scripting. These hypothetical, compromised add-ons could effectively negate any benefit of enforcing a Content Security Policy. As a result, the working draft of Content Security Policy 1.1 has been altered to:</p><blockquote>Note: User agents may allow users to modify or bypass policy enforcement through user preferences, bookmarklets, third-party additions to the user agent, and other such mechanisms.<br>— <a href="http://www.w3.org/TR/CSP2/#processing-model">W3C Content Security Policy 1.1 (Working Draft) Section 5</a><br>July 3rd, 2014</blockquote><p>The altered language significantly weakened the stance that browsers should bypass Content Security Policy for user-supplied add-ons, and suggests that providing the capability for users or add-ons to modify the Content Security Policy is optional. And it is.</p><p>Content Security Policy is currently being enforced by all major browsers, and is used by major websites like <a href="http://github.com/">GitHub</a>, <a href="http://twitter.com/">Twitter</a>, and <a href="http://medium.com/">Medium</a>. Support for modifying Content Security Policy is non-existent for users, tenuous for browser extensions, and impossible for bookmarklets. The end result is, unfortunately, an all too familiar story: we’ve sacrificed end-user freedom for the promise of additional security.</p><p>One of the HTML5 Design Principles, Priority of Constituencies, defines the course of action to take in the event of a conflict between stakeholders:</p><blockquote>In case of conflict, consider users over authors over implementors over specifiers over theoretical purity. In other words costs or difficulties to the user should be given more weight than costs to authors; which in turn should be given more weight than costs to implementors; which should be given more weight than costs to authors of the spec itself, which should be given more weight than those proposing changes for theoretical reasons alone. Of course, it is preferred to make things better for multiple constituencies at once.<br>— <a href="http://www.w3.org/TR/html-design-principles/#priority-of-constituencies">HTML5 Design Principles Section 3.2</a></blockquote><p>The ultimate catch-22 of the new Content Security Policy wording is that it’s intended to benefit the users, by providing additional security from hypothetical malicious add-ons on websites that enforce a Content Security Policy. In the end the bookmarklet has been relegated obsolete by the change, a casualty of one clause in one section of one web specification, and end-users and developers are the ones who will mourn its demise. The path to hell is paved with good intentions.</p><p>So what can we do? Well the deadline for comments on the working draft of Content Security Policy 1.1 was August 14th, 2014. Major browsers are enforcing Content Security Policy with varying strictness, the trend is that more web authors will start enforcing Content Security Policy, and there’s no user-based recourse for changing it.</p><p>I’d probably try to do more about it, but I’m too busy rewriting Instapaper’s bookmarklet into extensions for every major browser.</p><p><strong>RIP Bookmarklet <em>(1995—2014</em></strong>)</p><p><em>Thanks to </em><a href="http://twitter.com/loversmademen"><em>Rodion</em></a><em>, </em><a href="http://twitter.com/fjania"><em>Frank</em></a><em>, </em><a href="https://twitter.com/mephaust"><em>Paul</em></a><em>, </em><a href="http://twitter.com/myoung"><em>Mike</em></a><em>, and </em><a href="https://twitter.com/gooeyblob"><em>Kevin</em></a><em> for reading drafts of this post.</em></p><p><em>Sources &amp; Further Reading:</em></p><ol><li><a href="http://www.html5rocks.com/en/tutorials/security/content-security-policy/"><em>An Introduction to Content Security Policy</em></a></li><li><a href="https://blog.twitter.com/2013/csp-to-the-rescue-leveraging-the-browser-for-security"><em>CSP to the Rescue: Leveraging the Browser for Security</em></a></li><li><a href="http://www.w3.org/TR/CSP"><em>Content Security Policy 1.0</em></a></li><li><a href="http://www.w3.org/TR/CSP2"><em>Content Security Policy 1.1</em></a></li><li><a href="https://www.w3.org/Bugs/Public/show_bug.cgi?id=23357#c0"><em>Subverting CSP policies for browser add-ons (extensions)</em></a></li><li><a href="https://github.com/w3c/webappsec/commit/cbfaa8edfadebf21a9c7428242c12e45934d8c55"><em>CSP 1.1: Remove note about extensions.</em></a></li><li><a href="https://github.com/w3c/webappsec/commit/73963d509b20513a6f42b1e0839715aca8b578b0"><em>CSP 1.1: Add non-normative language for extensions</em></a></li><li><a href="http://lists.w3.org/Archives/Public/public-webappsec/2014Jul/0061.html"><em>[CSP] Request to amend bookmarklet/extensions sentence in CSP1.1</em></a></li><li><a href="http://lists.w3.org/Archives/Public/public-webappsec/2014Feb/0098.html"><em>Removal of the note about extensions</em></a></li><li><a href="http://lists.w3.org/Archives/Public/public-webappsec/2014Feb/0006.html"><em>CSP Formal Objection</em></a></li><li><a href="http://www.schemehostport.com/2011/10/priority-of-constituencies.html"><em>The Priority of Constituencies</em></a></li><li><a href="http://www.w3.org/TR/html-design-principles"><em>HTML5 Design Principles</em></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d470d4bbb626" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/bookmarklets-are-dead-d470d4bbb626">Bookmarklets are Dead…</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Instapaper Fragmentions]]></title>
            <link>https://medium.com/making-instapaper/instapaper-fragmentions-bb8159609d07?source=rss----24982f151313---4</link>
            <guid isPermaLink="false">https://medium.com/p/bb8159609d07</guid>
            <dc:creator><![CDATA[Brian Donohue]]></dc:creator>
            <pubDate>Tue, 24 Mar 2015 15:40:06 GMT</pubDate>
            <atom:updated>2016-05-23T18:01:10.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GE9xxFmnedPWxNTxLo_loQ.png" /></figure><h4>Linking to text from Instapaper</h4><p>When we built automatic sharing for Instapaper highlights, the team had several discussions about how to properly share a highlight link to another service. Here are two solutions we came up with:</p><ol><li>A page on Instapaper that showed the highlight, who shared it, and the original link to the article.</li><li>A page on Instapaper that embedded the original article in a full screen iframe, highlighted the correct text, and scrolled to the highlight.</li></ol><p>We knew neither of those solutions were great… As the launch got closer we were planning on punting on any special highlight link, and just share a link to the original article.</p><p>Two weeks before the big launch, <a href="http://twitter.com/borthwick"><strong>Borthwick</strong></a><strong> </strong>sent me a link to a recently published article titled “<a href="https://medium.com/p/41ef2be9953f"><strong>Fragmentions</strong></a>”.</p><h4><strong>Fragmentions</strong></h4><p><a href="https://twitter.com/kevinmarks"><strong>Kevin Marks</strong></a><strong>’ </strong>article about fragmentions is excellent, and I definitely recommend reading it. Here is the TL;DR summary of it:</p><ul><li>A fragmention references text in a webpage by appending the text to the end of the page’s URL.</li><li>Fragmentions are appended to the URL after a double number sign (“##”).</li><li>When a web page or browser detects a fragmention, it should highlight the corresponding text and scroll to it in the webpage. This behavior is similar to searching for text on a webpage.</li><li>An example fragmention: <a href="http://epeus.blogspot.com/2003_02_01_archive.html##annotate+the+web"><strong>http://epeus.blogspot.com/2003_02_01_archive.html##annotate+the+web</strong></a></li></ul><p>It’s an extremely elegant solution to the problem that we wrestled with when implementing highlight sharing in Instapaper: How do we link to a particular piece of text in a webpage? Kevin has proposed this as a web standard, and worked with <a href="https://twitter.com/jon_neal"><strong>Jonathan Neal</strong></a><strong> </strong>to create a <a href="https://github.com/chapmanu/fragmentions/blob/master/fragmention.js"><strong>script</strong></a><strong> </strong>that sites can use to support fragmentions.</p><h4><strong>Implementing Fragmentions in Instapaper</strong></h4><p>Fragmentions not only offer a much more elegant solution to referencing text in a webpage, but do so in a way that was <strong><em>much </em></strong>easier to implement than the solutions our team had initially proposed. Here’s a GitHub gist that shows the function from our Highlight class that generates the fragmention for a highlight: <a href="https://gist.github.com/Donohue/46b8f719f4d37f379f9f">https://gist.github.com/Donohue/46b8f719f4d37f379f9f</a></p><p>Just 8 lines of code to build a fragmention URL! When a highlight is created in Instapaper, and the user has automatic sharing enabled, the fragmention URL is posted to the appropriate services.</p><p>We also quietly rolled out an update to Instapaper’s <a href="https://chrome.google.com/webstore/detail/instapaper/ldjkgaaoikpmhmkelcgkgacicjfbofhh?hl=en-GB"><strong>Chrome</strong></a><strong> </strong>and <a href="http://www.instapaper.com/static/files/instapaper.safariextz"><strong>Safari</strong></a><strong> </strong>extensions that will detect fragmentions, highlight the text in the article, and scroll to the text in the page. The implementation is beautiful, simple, and seamless!</p><h4><strong>The Poet’s Problem</strong></h4><p>When creating a fragmention or a highlight, there’s a possibility that the text occurs more than once within the document. For instance in Edgar Allan Poe’s “<a href="http://www.heise.de/ix/raven/Literature/Lore/TheRaven.html"><strong>The Raven,</strong></a><strong>” </strong>the phrase “Quoth the raven” appears five times.</p><p>In a follow-up to his initial fragmentions post, entitled <a href="https://medium.com/p/fb8b535eb1f8"><strong>Fragmentions for Poets</strong></a>, Kevin outlines this problem and discusses solutions. The solution he recommends is to simply add more text in the fragmention such that the reference becomes unique. For instance “Quoth the raven” would become “I implore!’ Quoth the raven”.</p><p>Jonathan Neal has solved the problem by implementing an “<a href="http://indiewebcamp.com/irc/2014-04-22/line/1398233800"><strong>nth-occurrence” solution</strong></a><strong>. </strong>Essentially you can append “++” and a number to a fragmention that will reference a particular occurrence of the text. For example…</p><p><a href="http://www.heise.de/ix/raven/Literature/Lore/TheRaven.html##Quoth+the+raven++4"><strong>http://www.heise.de/ix/raven/Literature/Lore/TheRaven.html##Quoth+the+raven++4</strong></a></p><p>… would reference the 5th and last mention of “Quoth the raven”.</p><p>Interestingly enough, I entertained both of these solutions back in January when I started focusing on highlights for Instapaper. Do we try to clip additional text around the highlight to ensure we are highlighting the correct occurrence, or do we reference the nth-occurrence of that text in the document?</p><p>It seemed cleaner and more deterministic to implement the nth-occurrence solution for Instapaper, and as such we are adopting Jonathan’s solution for the Poet’s Problem in our fragmention implementations.</p><h4><a href="https://www.youtube.com/watch?v=rdciOXroU9o&amp;feature=kp"><strong>Things Done Changed</strong></a></h4><p>Another problem Kevin has responded to is what happens if the text changes as a result of an edit. It’s an extremely interesting problem, and one that we’ve discussed internally at <a href="http://betaworks.com/"><strong>betaworks</strong></a><strong>.</strong></p><p>Thinking about the problem in an abstract nature: What do we do with metadata when the data it references has changed? I’m not sure if this has been researched, but it would probably make for an interesting dissertation.</p><p>In Instapaper if we have a highlight that we cannot find in the article we silently fail and expect the user to handle that edge case. I think about this similarly to broken links on the web: If a link breaks, it’s up to the people that have linked to it to change those links, otherwise it remains broken. Similarly, if a highlight cannot be found in the document, it’s the user’s job to delete their old highlight and create a new one.</p><h4><strong>Closing Thoughts</strong></h4><p>Fragmentions, at least from Instapaper, are not always perfect. I’ve seen some instances where a website will use smart quotes (“…”) and the fragmention will have regular old quotes, which will fail to match. Also some services like Facebook, which parses canonical URLs, strip fragmentions altogether.</p><p>Having said that, fragmentions are an excellent and simple solution to referencing text in a webpage, and it’s a solution that Instapaper plans to implement and support even more robustly in the future.</p><p><a href="http://bthdonohue.tumblr.com/post/85643490431/instapaper-fragmentions">Originally published on my blog.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb8159609d07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/making-instapaper/instapaper-fragmentions-bb8159609d07">Instapaper Fragmentions</a> was originally published in <a href="https://medium.com/making-instapaper">Making Instapaper</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>