<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark Story on Medium]]></title>
        <description><![CDATA[Stories by Mark Story on Medium]]></description>
        <link>https://medium.com/@mark.story?source=rss-4b3e2df2a094------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W6Wsm70Z8f1uu1F4lO_mBg.png</url>
            <title>Stories by Mark Story on Medium</title>
            <link>https://medium.com/@mark.story?source=rss-4b3e2df2a094------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:50:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mark.story/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build Logs Available in Stickler-CI]]></title>
            <link>https://medium.com/@mark.story/build-logs-available-in-stickler-ci-ac778b033ec7?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac778b033ec7</guid>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[code-quality]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Sun, 26 Jul 2020 02:59:35 GMT</pubDate>
            <atom:updated>2020-07-26T02:59:35.687Z</atom:updated>
            <content:encoded><![CDATA[<p>Up until now if a build failed, or didn’t run correctly on <a href="https://stickler-ci.com/">stickler-ci </a>you had to rely on the build status to figure out what to try. While this mostly works when jobs outright fail, it isn’t great for jobs that succeed with the wrong or unexpected results. If you got incorrect or unexpected results, debugging what was going on was a tedious and time consuming process. Thankfully, those days are behind us now.</p><p>Going forward <a href="https://stickler-ci.com/">sticker-ci</a> check run results (but not reviews created with review comments) will contain build logs! Logs will be visible in both in <a href="https://stickler-ci.com/">stickler-ci</a> and more importantly in GitHub! In GitHub, build logs look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*jiE0YLg8rpF-CGyLSFmKSw.png" /><figcaption>Stickler build logs in GitHub Check Result</figcaption></figure><p>In <a href="https://stickler-ci.com/">stickler-ci.com</a>, you can find logs in the pull request list for your repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*dk23IIp70ufeMij6tpWJjQ.png" /><figcaption>Build log in stickler-ci.com</figcaption></figure><p>Inside the logs you’ll see:</p><ul><li>When the last build was run, and when it finished.</li><li>What commit was checked out.</li><li>The version of each linter stickler used to review your code.</li><li>The commands that stickler used to run each linter.</li><li>How many style violations or errors were collected from each linter.</li></ul><p>These additional details make it <strong>much</strong> simpler to debug builds. You can now see how stickler-ci is automating your code reviews and, why you got the results you did. I hope that this simplifies tweaking your configuration to get the results you’re looking for. Build logs are available in all plans and all platforms supported by stickler-ci. Happy coding, stay fresh and stylish.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac778b033ec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CakePHP Window functions and Common Table Expressions]]></title>
            <link>https://medium.com/@mark.story/cakephp-window-functions-and-common-table-expressions-e078aa112735?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/e078aa112735</guid>
            <category><![CDATA[window-functions]]></category>
            <category><![CDATA[common-table-expressions]]></category>
            <category><![CDATA[orm]]></category>
            <category><![CDATA[cakephp]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 02:19:50 GMT</pubDate>
            <atom:updated>2020-06-15T02:19:50.509Z</atom:updated>
            <content:encoded><![CDATA[<p>CakePHP 4.1.0 is going to be released soon. Thanks to <a href="https://github.com/othercorey">othercorey</a> and <a href="https://github.com/ndm2">ndm2</a> 4.1.0 adds powerful new features to the ORM — Window functions and Common Table Expressions.</p><h3>Window functions</h3><p>Window functions enable you to generate aggregations for ‘windows’ of your data. You define what the ‘window’ is by telling the database how to partition your data into frames around the current row, and perform aggregations across the rows in each frame.</p><p>Some example problems you could solve with window functions are:</p><ul><li>Getting the percentage of total views for each article an author has written.</li><li>Getting a running total for timeseries, or financial data.</li><li>Computing values based on data in adjacent rows. For example computing<br> time between train stations, or free time in a person’s calendar.</li></ul><p>A minimal example of window functions using the schema from the CMS tutorial would be:</p><pre>$query = $this-&gt;Articles-&gt;find();<br>$query-&gt;select([<br>    &#39;Articles.title&#39;,<br>    &#39;Articles.user_id&#39;,<br>    &#39;Articles.views&#39;,<br>    &#39;running_total_views&#39; =&gt; $query<br>        -&gt;func()<br>        -&gt;sum(&#39;Articles.views&#39;)<br>        -&gt;partition(&#39;Articles.user_id&#39;)<br>        -&gt;order(&#39;Articles.views ASC&#39;)<br>        -&gt;rows(0, null),<br>])<br>-&gt;orderDesc(&#39;Articles.user_id&#39;)<br>-&gt;orderDesc(&#39;Articles.views&#39;)<br>-&gt;disableHydration();</pre><pre>// Each row will have a `running_total_views` key.<br>debug($query-&gt;toArray());</pre><pre>// The output of the above would be:<br>[<br>    [<br>        &#39;title&#39; =&gt; &#39;First Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 1,<br>        &#39;running_total_views&#39; =&gt; 1,<br>    ],<br>    [<br>        &#39;title&#39; =&gt; &#39;Latest Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 4,<br>        &#39;running_total_views&#39; =&gt; 5,<br>    ],<br>]</pre><p>You could get the average views using a window function like this:</p><pre>$query = $this-&gt;Articles-&gt;find();<br>$query-&gt;select([<br>    &#39;Articles.title&#39;,<br>    &#39;Articles.user_id&#39;,<br>    &#39;Articles.views&#39;,<br>    // get average views<br>    &#39;average_views&#39; =&gt; $query<br>        -&gt;func()<br>        -&gt;avg(&#39;views&#39;)<br>        -&gt;partition(&#39;Articles.user_id&#39;),<br>])<br>-&gt;orderDesc(&#39;Articles.user_id&#39;)<br>-&gt;orderDesc(&#39;Articles.views&#39;)<br>-&gt;disableHydration();</pre><pre>// The output of the above would be:<br>[<br>    [<br>        &#39;title&#39; =&gt; &#39;First Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 1,<br>        &#39;average_views&#39; =&gt; 2.5,<br>    ],<br>    [<br>        &#39;title&#39; =&gt; &#39;Latest Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 4,<br>        &#39;running_total_views&#39; =&gt; 2.5,<br>    ],<br>]</pre><p>Window functions can be a bit opaque to work with at times, but we can use a <a href="http://twitter.com/GROUP_CONCAT">GROUP_CONCAT</a> to dump out the data each window is looking at:</p><pre>$query = $this-&gt;Articles-&gt;find();<br>$query-&gt;select([<br>    &#39;Articles.title&#39;,<br>    &#39;Articles.user_id&#39;,<br>    &#39;Articles.views&#39;,<br>    // Look at windows<br>    &#39;debug&#39; =&gt; $query<br>        -&gt;func()<br>        -&gt;aggregate(<br>            &#39;group_concat&#39;,<br>            [&#39;Articles.views&#39; =&gt; &#39;literal&#39;, &#39;,&#39;], <br>            [], <br>            &#39;string&#39;<br>        )<br>        -&gt;partition(&#39;Articles.user_id&#39;)<br>        -&gt;rows(0, null),<br>])</pre><pre>// The output of the above would be:<br>[<br>    [<br>        &#39;title&#39; =&gt; &#39;First Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 1,<br>        &#39;debug&#39; =&gt; &#39;1&#39;,<br>    ],<br>    [<br>        &#39;title&#39; =&gt; &#39;Latest Article&#39;,<br>        &#39;user_id&#39; =&gt; 1,<br>        &#39;views&#39; =&gt; 4,<br>        &#39;debug&#39; =&gt; &#39;1,4&#39;,<br>    ],<br>]</pre><h3>Common Table Expressions</h3><p>Common table expressions which are often abbreviated to ‘CTE’ enable you to create intermediate result sets and then use those tables to build queries on. They serve a similar purpose to database views or derived tables. However, they differ from derived tables and views a few ways. First you don’t have to maintain schema for a common table expression. Second, you can reference the results of a common table expression multiple times without incurring performance penalties, unlike derived table joins.</p><p>As an example, we want to fetch a list of tags, and the number of articles each one has.</p><pre>$query = $this-&gt;Articles-&gt;find();<br>$query-&gt;with(function (CommonTableExpression $cte, Query $query) {<br>    $q = $query-&gt;cleanCopy();<br>    $q-&gt;repository($this-&gt;Articles-&gt;Tags-&gt;junction());</pre><pre>$q-&gt;select([<br>        &#39;article_id&#39; =&gt; &#39;article_id&#39;,<br>        &#39;tag_count&#39; =&gt; $q-&gt;func()-&gt;count(&#39;*&#39;),<br>    ])<br>    -&gt;from(&#39;articles_tags&#39;)<br>    -&gt;group([&#39;article_id&#39;]);</pre><pre>return $cte<br>        -&gt;name(&#39;tag_counts&#39;)<br>        -&gt;query($q);<br>});</pre><pre>$query-&gt;select([<br>    &#39;title&#39;,<br>    &#39;tag_count&#39; =&gt; &#39;tag_counts.tag_count&#39;<br>])<br>-&gt;join([<br>    &#39;tag_counts&#39; =&gt; [<br>        &#39;table&#39; =&gt; &#39;tag_counts&#39;,<br>        &#39;type&#39; =&gt; &#39;LEFT&#39;,<br>        &#39;conditions&#39; =&gt; &#39;tag_counts.article_id = Articles.id&#39;<br>    ],<br>])<br>-&gt;disableHydration();</pre><pre>// The above would output<br>[<br>    [<br>        &#39;title&#39; =&gt; &#39;First Article&#39;,<br>        &#39;tag_count&#39; =&gt; 4,<br>    ]<br>]</pre><p>I have found common table expressions most useful in reporting queries when I need to break up a big complex query into easier to understand blocks, or merge the results of multiple aggregate queries together. The ORM will support common table expressions in <a href="http://twitter.com/SELECT">SELECT</a>, <a href="http://twitter.com/INSERT">INSERT</a>, <a href="http://twitter.com/UPDATE">UPDATE</a> and <a href="http://twitter.com/DELETE">DELETE</a> operations. Window functions, and common table expressions work with Postgres, SQLServer, SQLite and MySQL 8.0+. I’m really happy to see the CakePHP ORM continuing to evolve and support advanced SQL features. While my examples are simple they illustrate how straightforward the new APIs introduced in 4.1.0 are to use, and I hope they help you build even better applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e078aa112735" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Custom Builds for Stickler CI]]></title>
            <link>https://medium.com/@mark.story/building-custom-builds-for-stickler-ci-2d5cc02ba2a1?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d5cc02ba2a1</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[saas-tools]]></category>
            <category><![CDATA[eslint]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[code-review]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Sat, 11 Jan 2020 03:32:24 GMT</pubDate>
            <atom:updated>2020-01-11T03:32:24.050Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I shipped a new feature to <a href="https://stickler-ci.com/">Stickler CI</a> that enables users to extend Javascript and Python builds with additional packages. Maintaining review tool dependencies can be a drain on your team’s time. Stickler CI helps solve this problem, but used to come with a tradeoff of not being able to fully customize your style rules. While our default images come with many popular packages pre-installed, it is impossible to have every package for flake8 and eslint pre-installed.</p><h3>Enter Custom Builds</h3><p>After fielding a few package requests, I knew there had to be a better way. Stickler does all of its reviews inside docker containers with temporary filesystems. The solution seems simple enough. Before running each job find and install the additional packages required by the current repository. Then once the review is complete, docker could discard the container filesystem, leaving a pristine state for the next review.</p><p>This initially seemed simple enough. However, in addition to reviews, Stickler CI can also automatically apply any fixes from your linting tools. In order to avoid wasting a significant amount of time downloading and installing packages multiple times, I needed a way to share state between the fixer and review processes.</p><p>To solve this performance issue I chose to create temporary images. When a named container, docker will retain the container state. This state can then be turned into an image with docker commit. With a custom per-job image, I would be able to offer custom builds without wasting CPU and network. I chose to start with eslint as all of the additional package requests were for eslint.</p><h3>Installing Additional Packages</h3><p>Installing additional packages should be simple, but the real world is messy. As a GitHub App, Stickler is unable to access private NPM registries or pull dependencies from other git repositories. Furthermore, installing all of an application’s dependencies could be slow and expensive time wise. To workaround this only, the package installer searches for packages matching eslint-(config|preset). Only packages matching this pattern are installed. Thankfully most ESLint configuration packages are open-source and small enough that they don’t take long to download.</p><p>The end flow roughly looks like:</p><ol><li>Create a named container that uses a tool specific install script.</li><li>The install script uses regexp to find package names &amp; versions.</li><li>Each package is then installed into the image.</li><li>The resulting container filesystem is committed into a new image.</li><li>The fixer command is run in our new image, and any changed files are committed.</li><li>The check command is run and errors are reported to GitHub.</li><li>The image created in step 3 is deleted.</li></ol><p>Once I had this flow implemented for eslint I started beta testing it with customers. When a customer contacted me about additional eslint packages, I would give them instruction on how to use the plugin installer. I used this beta group to gain confidence in the approach, monitor how custom images operated, and fix a lingering bug or two.</p><p>Once I was happy with the solution, I implemented a similar flow for the flake8 integration. Finding which python packages should be installed is even more complicated than in Javascript. There are far too many places, and ways a Python application can store its dependencies. Because of this I chose to make the flake8 plugin installer require users to provide a list of plugins they use. Having an explicit list requires a bit more work from the user, but ensure that reviews are always deterministic and work the first time.</p><h3>Future improvements</h3><p>I’m quite happy with the solution I arrived at. It has been operating quite well throughout its beta and public releases. In the future, I would like to improve a few things. First, would be to keep a pool of frequently used custom containers around to help further improve review times. I’d also like to expand the number of linters that include custom builds. Ruby and PHPCS are the next likely candidates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d5cc02ba2a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons from a GitHub Application for Stickler CI]]></title>
            <link>https://medium.com/@mark.story/lessons-from-a-github-application-for-stickler-ci-f311871f2741?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/f311871f2741</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Tue, 21 Aug 2018 03:43:47 GMT</pubDate>
            <atom:updated>2018-08-21T03:43:47.124Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently built a GitHub Application for <a href="https://stickler-ci.com/">Stickler CI</a> and wanted to share what I learned along the way. While the documentation for GitHub Applications is pretty good there were a few things I struggled with.</p><h3>What is a GitHub App, why would I want to build one?</h3><p>In addition to OAuth integrations which have existed for almost a decade, GitHub now offers GitHub Applications. OAuth integrations allow an integration to request coarsely scoped access to a user’s account, and act as that user. For example, an OAuth integration cannot ask for write access to a few repositories. Instead it must ask for write access on all public and/or private repositories. This can make it harder to attract users as they don’t want to give you carte blanche with their data. If an OAuth application wants to act as a separate identity, then an additional GitHub account is required.</p><p>GitHub Applications by contrast can be ‘installed’ at the organization level, and only be given access to specific repositories. Furthermore, they can request more fine-grained permission controls at the repository level. For example, an application can request write access to statuses but only read access to the code. The end result of these finer grained permissions means that installing an integration for an open source project doesn’t expose your day job’s code. Once ‘installed’ an application acts on its own behalf, and doesn’t ever need to pretend to be a normal human. This removes the need to have a separate ‘bot’ account in GitHub.</p><h3>More complex data model</h3><p>GitHub Applications offer several benefits but they come with some drawbacks. The first is a far more complex suite of authentication mechanisms. The new conceptual model is that Applications are added to organizations creating ‘installations’. Installations gain access to an organizations repositories, and other organization resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*Y0kEna8T5h2Mzkyr.png" /><figcaption>Entities in a GitHub Application</figcaption></figure><h3>Three authentication modes</h3><p>The more complex model of GitHub Applications requires more authentication methods. In total GitHub Applications require three kinds of authentication credentials.</p><p>1. Private key signed JWT tokens. This authentication mode is required when acting as the ‘application’. You are acting as the application when you want to enquire about your installations, or interact with a specific installation.<br> 2. Installation Token. Installation tokens are created using a JWT. Once you have an installation token you can access the repositories enabled for that installation.<br> 3. OAuth Token — Applications can use OAuth to identify users. I personally didn’t end up using this mode as you can’t request additional scopes which makes it impossible to read a users organizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*6EX_REpqNgjJvQZw.png" /><figcaption>Resources available to each authentication type</figcaption></figure><p>The Installation and JWT based APIs require you to set an Accept header of application/vnd.github.machine-man-preview+json. However, once you have an installation access token you don’t need that header.</p><p>Knowing which authentication mode you have to use for which APIs requires reading the documentation carefully, as the required authentication mode is not highlighted. Any API endpoint that isn’t explicit about which kind of authentication it requires likely requires an installation token. Not all APIs are usable by GitHub Applications, so keep an eye out for the blue ‘i’ indicating an endpoint can be accessed via an installation token. If there is no ‘i’ that endpoint cannot be used by a GitHub Application.</p><p>Another wrinkle with GitHub Applications is how long tokens live for. JWT tokens live for a maximum of 10 minutes, while installation tokens live for 1 hour. This is very different than OAuth tokens, which live forever. The shorter token lifetime meant that I needed to ensure I created a new installation token when starting a review job, and that each web request created a new JWT token.</p><h3>Clunky OAuth mode</h3><p>I found the OAuth mode of a GitHub Application to be awkward. It doesn’t allow you to request additional scopes and can only be used to identify a user. Because of this, I wasn’t able to get a users’ organization list which made building the UI flow I wanted impossible.</p><p>I ended up needing to use both an OAuth integration and a GitHub Application. The OAuth integration allows me to access a user’s profile and organization data. With that I can use the Application JWT to discover a user’s installations. This combination allowed me to display organizations with and without Application installs. This approach was not documented, so I suspect it is not a recommended approach. However, it made migrating to a GitHub application easier, as my existing users can opt-in to installing the GitHub Application without disrupting their existing OAuth integration.</p><p>Once authentication modes were solved I had to change very little about my integration. One important thing I did have to change was how repositories are cloned. With an OAuth application a clone URL looks like:</p><pre>https://&lt;token-value&gt;:x-oauth-basic@github.com/user/repo.git</pre><p>While with a GitHub Application you must use.</p><pre>https://x-access-token:&lt;token-value&gt;@github.com/user/repo.git</pre><h3>In closing</h3><p>I found the process of implementing a GitHub Application straight forward once I grokked the new data model and authentication complexity. I’m glad to see GitHub making significant investments in their integration ecosystem that make it easier and better for users to adopt integrations, and offer new features like Checks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f311871f2741" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of Container Usage in Stickler CI]]></title>
            <link>https://medium.com/@mark.story/evolution-of-container-usage-in-stickler-ci-7366e4aff698?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/7366e4aff698</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[coding-style]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Fri, 11 May 2018 02:24:26 GMT</pubDate>
            <atom:updated>2018-05-11T02:24:26.783Z</atom:updated>
            <content:encoded><![CDATA[<p>Stickler CI is a software as a service application that automates a tedious part of code review; enforcing consistent style and preventing lint errors. By integrating with GitHub, Stickler checks each pull request for style errors and post review comments when an error is found. This helps your team align on coding standards and provide more valuable feedback. Stickler is free for public repositories; private repositories require a paid plan.</p><h4>Containerless prototype</h4><p>When I started building Stickler, the first implementation ran all linting tools on the host filesystem as a child process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*OP3k3Et_YE8k1lkCzwp-dg.png" /><figcaption>No containers to see here.</figcaption></figure><p>This design required having all linting tools installed on the host operating system. Getting all the linters to play together on the host operating system was challenging as a couple of them had esoteric and hard to build dependencies. This design could lead to secret information being accessed if someone managed to exploit a linter. A few supported linters allow users to provide their own rules. These linters posed the greatest risk and needed to have their custom rule capabilities disabled. I knew this was not an acceptable long term solution as it hobbled the product, and I wasn’t confident that all threats were mitigated. However, I wanted to make sure the product would be used before I invested in more complex solutions.</p><h4>Kitchen sink container</h4><p>After acquiring some users and gaining confidence that the product was useful, it became time to fix the prototype. In the next iteration, linter execution was moved into a docker container. Moving the linters simplified the reviewer dependencies. The new image included both parts of the Stickler application code and a sandbox to run linting tools in. The image used for this container was a ‘kitchen sink’ that contained all 13 linter tools Stickler supported. Getting the various dependencies installed resulted in a long Dockerfile. This approach resolved the obvious risks I knew about and allowed custom lint rules for PHPCS and ESLint to be enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*ianqOYkAeTebMq2V0q9AQg.png" /><figcaption>One container to rule them all.</figcaption></figure><p>As I continued to develop with this image some shortcomings became apparent.</p><ul><li>The long Dockerfile was creating a <strong>massive</strong> image. 3.5GB in total. This made updates to certain tools expensive as they required rebuilding multiple layers.</li><li>Deploying new review job code also required building new images. Thankfully the application code was the last layer, but needing to build images complicated the deploy process more than I’d like.</li><li>The massive image resulted in long startup times for containers.</li><li>Because the container contained application code and linters, I needed to drop privileges to mitigate exposing sensitive data stored in /proc. This added complexity to the code, and I wasn’t sure I had prevented every potential threat.</li></ul><h4>Tiny Containers</h4><p>As the workflow issues with the massive image became more annoying, I began to think of solutions. Given that many issues stemmed from the size and complexity of the Dockerfile and its resulting image, I wanted to try smaller images. These images would only contain the linter tools and no application code. The application code would be responsible for cloning the repository and using docker volumes to mount the cloned code into the containers. Each container would encapsulate only a one tool chain. For example, all the python2 tools.</p><p>My hypothesis was that single tool chain images would be smaller, easier to maintain, start faster and be safer to operate. In pursuit of smaller image sizes, I switched from Ubuntu to Alpine Linux shaving more megabytes off of each image. The architecture now looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*ISrDlsbnlKY48dN76hFeMQ.png" /><figcaption>Containers for every task</figcaption></figure><p>Adopting smaller images meant that I had to do changes to how reviews were run. Instead of running tools as simple sub-processes, each tool was run in a separate container. Having tools run in containers that only have access to the repository code fully isolated linter execution from the Stickler code and host operating system. Switching to smaller images was a huge change that required rebuilding all path handling throughout the review execution. With it complete, I’m very happy with the results. I’ve been able to reap a number of upsides from the adopting smaller images:</p><ul><li>Stickler is now able to offer both python 2 and python 3 linting</li><li>Each image is light, simple and contains no proprietary code.</li><li>Containers start up fast.</li><li>Application deployments don’t involve building and pushing docker containers anymore.</li></ul><p>As Stickler continues to grow, I’m sure there will be more opportunities to revisit this solution and improve it further.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7366e4aff698" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the CakePHP Authorization Plugin]]></title>
            <link>https://medium.com/@mark.story/introducing-the-cakephp-authorization-plugin-af80c06db3d3?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/af80c06db3d3</guid>
            <category><![CDATA[cakephp]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[authorization]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Tue, 03 Apr 2018 17:06:04 GMT</pubDate>
            <atom:updated>2018-04-03T17:06:04.350Z</atom:updated>
            <content:encoded><![CDATA[<p>In the next major release of CakePHP we’re going to be removing the AuthComponent. This component and its helpers have been part of CakePHP since the 1.2 days, but their time has come to an end. Over the years, AuthComponent has become a complex and difficult to extend piece of CakePHP. In its wake, we’re promoting two new plugins. First, <a href="https://github.com/cakephp/authentication">cakephp/authentication</a> will be replacing the identification parts of AuthComponent while <a href="https://github.com/cakephp/authorization">cakephp/authorization</a> handles access control and permissions. These plugins are intended to be used together, but can be used individually. They can also be used outside of CakePHP applications thanks to PSR-7. The new authorization plugin makes answering questions like ‘can the current user edit this record?’ and ‘what records can the current user see?’ simpler to answer from anywhere in your application, not just in controllers.</p><h4>Installation</h4><p>The authorization plugin is installed via composer. Currently it is in a beta release, but that will quickly change.</p><pre># For now run<br>composer require cakephp/authorization:1.0.0-beta4</pre><pre># Later on run<br>composer require cakephp/authorization:^1.0</pre><p>In your application’s bootstrap process you should add the authorization plugin to your application via Plugin::load(). This will enable bake to discover tasks. Next you’ll need to add the middleware to your application:</p><pre>// In your Application.php, import the class.<br>use Authorization\Middleware\AuthorizationMiddleware;</pre><pre>// inside your application’s middleware hook.<br>$middlewareStack-&gt;add(new AuthorizationMiddleware($this));</pre><h4>Concepts</h4><p>Authorization requires a few possibly new concepts.</p><ul><li><strong>Identity</strong> Is a person or agent acting in your application. Generally this will be a user or an API client.</li><li><strong>Resource</strong> A ‘thing’ that an <em>Identity</em> needs to access. For example: Articles, Tags, Comments.</li><li><strong>Action</strong> An operation that an <em>Identity</em> takes on a <em>Resource</em>. For example ‘update’, ‘delete’</li><li><strong>Policy</strong> The logic defining which <em>Actions</em> <em>Identities</em> can take on <em>Resources</em>.</li></ul><p>The authorization plugin provides abstractions for Identities, Actions and Policies. The Resources are the domain objects in your application.</p><h4>Identities</h4><p>The authorization plugin hooks into your application as middleware. It expects to find an authenticated user in the identity attribute of the request. By default, the authorization middleware will decorate this identity with the Authorization\IdentityDecorator. This interface defines three methods, and proxies all other methods, and properties to the decorated object. The decorator adds can() and applyScope() methods. These are the main methods that allow you to check permissions on your resources. Because all access controls are attached to the current user, you can more easily perform access control checks anywhere in your application:</p><pre>// Get the user from the request<br>$user = $request-&gt;getAttribute(‘identity’);</pre><pre>// Check a permission<br>if ($user-&gt;can(‘delete’, $article)) {<br> echo ‘User can delete this thing!’;<br>}</pre><p>While the decorator approach works, you can avoid any overhead/complexity it adds by implementing the Authorization\IdentityInterface in your existing user class. The <a href="https://github.com/cakephp/authorization/blob/master/docs/Middleware.md#identity-decorator">documentation has an example</a> of how to do this. Permission checks are always executed by <em>Policy</em> classes.</p><h4>Policies</h4><p>Policy classes contain the permissions checking logic in your application. Each resource that you want to check permissions for will need a policy class defined. Policy classes are mapped to the resources in your application by the ‘policy resolver’. For CakePHP applications you can define your policy resolver in your authorization hook method. If your application primarily works with ORM Entities, you can use the following:</p><pre>namespace App;</pre><pre>use Authorization\AuthorizationService;<br>use Authorization\Policy\OrmResolver;<br>use Cake\Http\BaseApplication;</pre><pre>class Application extends BaseApplication<br>{<br>  public function authorization($request)<br>  {<br>    $resolver = new OrmResolver();<br>    return new AuthorizationService($resolver);<br>  }<br>}</pre><p>If you have multiple kinds of domain objects to check, you can use the ResolverCollection to join resolvers for multiple datasources together.<br>Now that we have a resolver setup, we can create some policies. The simplest way to do this is with bake:</p><pre>bin/cake bake policy Project</pre><p>This will generate a class into <strong>src/Policy/ProjectPolicy.php</strong>. As an example, our application will only allow users to see/edit/delete their own projects. In our policy class we can enforce this logic with the following method:</p><pre>public function canUpdate(IdentityInterface $user, Project $project)<br>{<br>  return $user-&gt;id == $project-&gt;user_id;<br>}</pre><p>Policy methods use the convention of can and the action name. We could implement similar logic in our canDelete method too. To show a user their list of projects we can’t check records on individual rows as it would be inefficient to look at <em>all</em> projects to find only those the current user can see. Instead we want to push the access control conditions into the query that creates the project list. We call this a ‘policy scope’ as it scopes a query through a policy. Before we can apply scopes to our queries, we need to create a policy class for our Table class. We can create a table policy with bake:</p><pre>bin/cake bake policy --type Table Projects</pre><p>This will create <strong>src/Policy/ProjectsTablePolicy.php</strong>. In it we can add the following method:</p><pre>public function scopeIndex($user, $query)<br>{<br>  return $query-&gt;where([‘Projects.user_id’ =&gt; $user-&gt;id]);<br>}</pre><p>Then in our controller endpoints we can use our new policy:</p><pre>// In ProjectsController::index<br>$user = $this-&gt;request-&gt;getAttribute(‘identity’);</pre><pre>// Apply our policy scope<br>$query = $user-&gt;applyScope(‘index’, $this-&gt;Projects-&gt;find());</pre><pre>$this-&gt;set(‘projects’, $this-&gt;paginate($query));</pre><p>Policy scopes should return a new or mutated object. They are ideal to use with query builders and finder logic.</p><h4>Checking Permission</h4><p>Earlier we’ve seen a few examples of authorization checks in action. Once loaded the AuthorizationComponent can be used to simplify raising exceptions when authorization checks fail, and skipping authorization on actions that don’t require permissions to be checked.</p><pre>// In a controller<br>public function view($id)<br>{<br>  $project = $this-&gt;Projects-&gt;get($id)<br>  // Gets the ‘action’ from the controller action.<br>  // Will raise an exception if the check fails.<br>  $this-&gt;Authorization-&gt;authorize($project);</pre><pre>  // Rest of controller action.<br>}</pre><p>If your controller actions don’t require authorization, you can use the skipAuthorization() method to mark an action as not requiring further authorization. By default when authorization fails, or hasn’t been checked the AuthorizationMiddleware will raise an exception that your application can handle. If you’d prefer that authorization errors are converted into redirects, you can do that too. The <a href="https://github.com/cakephp/authorization/blob/master/docs/Middleware.md#handling-unauthorized-requests">documentation has more details</a>.</p><p>Hopefully this illustrates a bit of what the new authorization plugin can do and why we’re removing AuthComponent in favour of more powerful and flexible solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af80c06db3d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Stickler CI grew from a prototype to a product]]></title>
            <link>https://medium.com/hackernoon/how-stickler-ci-grew-from-a-prototype-to-a-product-6d2a58987ac1?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d2a58987ac1</guid>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[prototype-to-product]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Fri, 08 Dec 2017 17:08:56 GMT</pubDate>
            <atom:updated>2019-05-06T08:36:34.124Z</atom:updated>
            <content:encoded><![CDATA[<h3>About this article series</h3><p>In this three part series, I’m going to cover the evolution of Stickler CI in the past 2 years from the initial prototype to the present day. This specific article will cover how I brought Stickler CI from an unprofitable project to a revenue generating product and the growing pains surrounding that journey.</p><p><a href="https://stickler-ci.com/">Stickler CI</a> is a software as a service application that automates a tedious part of code review; enforcing style and checking lint errors. It integrates with GitHub and checks each pull request for code style errors. Stickler CI will leave review comments when someone on your team makes a style error. This saves you time, as feedback on style errors is handled automatically, and your team doesn’t have to read through build reports to find whitespace errors. Our business model provides access on public repositories for free, while private repositories require a paid plan.</p><h4>Real Customers, Real Problems</h4><p>In January of 2017 Stickler CI attracted its first paying customer. The technical challenges mounted as the year progressed and we started to acquire more customers. Some of our misadventures included problems around cancellations, <a href="https://hackernoon.com/tagged/security">security</a>, scaling up review jobs and handling the increased demand.</p><h4>Incomplete Cancellations</h4><p>In the first implementation, we didn’t build workflows to handle cancellations. We had assumed that with so few paying customers churn would not occur for some time. To our surprise, a customer upgraded and needed to cancel much faster than we had anticipated. This resulted in a manual cancellation via email, and the implementation of a proper cancellation workflow afterwards. In addition to voluntary cancellations, a well rounded billing integration needs to handle a cancellations due to non-payment. In our haste to get paid plans shipped, we also neglected to build this functionality. This resulted in a few former customers having access to private repositories after their subscription was cancelled. To correct this oversight, we implemented the <strong>customer.subscription.deleted</strong> and <strong>invoice.payment_failed</strong> webhooks that Stripe sends when a subscription misses a payment or is cancelled for missing payments. Today when those webhooks arrive, the subscription is cancelled in our local database, preventing <a href="https://hackernoon.com/tagged/future">future</a> reviews on private repositories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*-8lMpLol5QMyBEeJ." /><figcaption>Stripe Webhook workflow</figcaption></figure><h4>Safety Second</h4><p>Another issue with the initial offering was that we support linters that allow users to use linting rules they create with their own code. The ability to create custom rules allows users run whatever code they want on the Stickler CI servers. This presents a big security issue as malicious users could craft custom code to obtain API credentials or other private information. To remedy this gap, review jobs and linter execution was moved into a docker container, that has no access to the host filesystem containing secrets. After startup, the worker job drops privileges becoming an unprivileged user before running linters. Dropping privileges prevents custom linter code from accessing the environment variables the container was created with.</p><h4>Keeping workers available</h4><p>We cut a number of corners in how review jobs operated in order to release Stickler CI to market quickly. Some linters, like phpcs and eslint, support lint rules being defined in our user’s application code. These linters expose us to the risk that a nefarious user could put an infinite loop in one of their custom linter rules. An infinite loop would result in a locked review job worker, stalling reviews for all other users. This risk was mitigated by terminating review jobs that run longer than 10 minutes. We no longer have to worry about unpleasant users hogging our servers, with infinite loops, requiring us to provision more workers and impact Stickler CI’s profitability.</p><h4>Memory is a fickle thing</h4><p>As you may recall from the previous article, Stickler CI was originally deployed onto a single server. We didn’t anticipate how short lived that period would be.</p><p>Originally, the web-server had 1GB of RAM but soon became overloaded as review job traffic picked up. In response, that machine was upgraded to 2GB, but shortly after moving review jobs to docker, we started seeing various processes being terminated by my nemesis the OOMKiller (Out of Memory Killer). The system logs contained a number of ‘Killed process’ log messages. This told us that it was time to add more capacity, as our production environment was becoming over-capacity.</p><p>We quickly spun up a ‘worker’ machine by hacking together an Ansible playbook and some manual configuration. This configuration let us move docker and the review job processing to the new machine, freeing up precious RAM on the primary server. We also added small swap drives to both the worker and web-server hosts. This let us handle traffic spikes a bit and keep OOMKiller at bay. The resulting architecture of this change looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/0*Jx9DeO4KQPA9x1iC." /><figcaption>v2 of the Stickler CI architecture</figcaption></figure><p>This configuration lets the worker node connect to RabbitMQ on the web-server, but run review jobs on its own docker daemon. Logs from the worker are forwarded to the web-server enabling us to have centralized logging, which makes troubleshooting easier. By running docker and review jobs on the worker we can better manage and isolate the resources (time &amp; memory) that review jobs use and protect our web-server and database from expensive review job workloads. The docker container handles publishing review comments to GitHub, and allows us to re-queue jobs that fail with ease.</p><h3>Conclusion</h3><p>The end result of correcting these shortcuts was a more profitable, high performing, secure product. Stickler CI experienced the business impact of being able to take on more users, more reviews, and more revenue. The <a href="https://medium.com/@mark.story/evolution-of-container-usage-in-stickler-ci-7366e4aff698">next article</a> will investigate the fallout of our hastily built worker node solution and the nightmare of self-hosted email.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d2a58987ac1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/how-stickler-ci-grew-from-a-prototype-to-a-product-6d2a58987ac1">How Stickler CI grew from a prototype to a product</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I took Stickler CI from an idea into a side hustle]]></title>
            <link>https://medium.com/hackernoon/how-i-took-stickler-ci-from-an-idea-into-a-side-hustle-3a7287d953a5?source=rss-4b3e2df2a094------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a7287d953a5</guid>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[code-style]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[code-review]]></category>
            <dc:creator><![CDATA[Mark Story]]></dc:creator>
            <pubDate>Mon, 23 Oct 2017 16:08:28 GMT</pubDate>
            <atom:updated>2019-05-06T08:36:34.654Z</atom:updated>
            <content:encoded><![CDATA[<h3>About this article series</h3><p>In this three part series, I’m going to cover the evolution of <a href="https://stickler-ci.com/">Stickler CI</a> in the past 2 years from the initial prototype to the present day. This specific article will cover how I built the initial prototype and then added paid plans.</p><h3>My side project</h3><p>Stickler CI is a software as a service application that automates the more tedious parts of code review; enforcing style and checking lint errors. It integrates with your <a href="https://hackernoon.com/tagged/github">GitHub</a> repositories and checks each pull request for code style errors. Stickler CI will leave review comments when someone on your team makes a style error. This saves developer time as feedback on style errors is handled automatically, and your team doesn’t have to read through build reports to find whitespace errors.</p><p>I was inspired to build <a href="https://hackernoon.com/tagged/stickler">Stickler</a> CI after spending a lot of time giving feedback on style errors, as our team was refactoring older parts of our applications at FreshBooks. I thought there must be a better way to give feedback on code style &amp; formatting errors discovered in code review.</p><h3>The initial prototype</h3><p>The frontend application is built with CakePHP, while the backend portion of the application that performs code review is built with Python. Persistent data is stored in MySQL and RabbitMQ is used for the review job queue. After the initial stealth mode development phase over the course of a few months I had an initial prototype ready. The first release was deployed to a single 1GB DigitalOcean droplet. I started off with only a single user (me). The architecture looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/0*yhWCvDWYz7Gn8h15." /><figcaption>Initial application architecture</figcaption></figure><p>All the components of the application lived in a single box. My goal was to get the prototype in front of people with the smallest amount of expense and complexity. While the single server approach has limitations around capacity and scalability I wasn’t concerned about those issues at this time; I had only one user. Before making the deployment/infrastructure more resilient or scalable I wanted to see if anyone would use, and pay for this product.</p><p>The simple infrastructure enabled me to use the exact same Ansible playbooks to manage both my local and production environments. It also made it easy to scale ‘up’ as I could easily get a bigger instance from DigitalOcean. At this point, Stickler CI was only able to connect to public repositories as I wanted to charge for private repositories if the prototype proved successful.</p><p>I enabled Stickler CI on some of my open source projects to help contributors with their own lint errors and hopefully drum up some interest in the product. From my open source projects I attracted the very first users, a couple of users reached out wanting to use Stickler CI on their private repositories. This was enough of a positive signal to compel me to build private repository support and paid plans.</p><h3>Adding paid plans</h3><p>I chose to use a very simple subscription based business model that scales based on the number of private repositories a person wanted to enable. I chose private repositories as a differentiator because it was a similar billing model to Stickler CI’s competitors.</p><p>I used Stripe to build subscriptions. Stripe is a great product with a fantastic API and great documentation. Stripe handles card tokenization, plan listings, and subscription billing. I leveraged the metadata fields on plans to store the number of private repositories each plan provides. This removed the need to maintain a mapping of products and their repository limits in my application code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/0*SBBSaGnNQ8RTRNXe." /><figcaption>Plan metadata in Stripe</figcaption></figure><p>During an upgrade I can read the repositories metadata field and know how many private repositories to grant the user.</p><p>At this stage there were ~50 users and Stickler was processing about ~150 jobs a day. The addition of Stripe didn’t fundamentally change the architecture, but I was starting to get worried about some of the corners I cut to get the product out the door. I’ll detail those shortcuts and how I fixed them in my <a href="https://hackernoon.com/how-stickler-ci-grew-from-a-prototype-to-a-product-6d2a58987ac1">next article</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a7287d953a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/how-i-took-stickler-ci-from-an-idea-into-a-side-hustle-3a7287d953a5">How I took Stickler CI from an idea into a side hustle</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>