<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Andersen on Medium]]></title>
        <description><![CDATA[Stories by Jeff Andersen on Medium]]></description>
        <link>https://medium.com/@jeffandersen?source=rss-f9690c1e22b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JtkQhOF7qjGx2rYG7nzFoQ.jpeg</url>
            <title>Stories by Jeff Andersen on Medium</title>
            <link>https://medium.com/@jeffandersen?source=rss-f9690c1e22b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:52:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeffandersen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Compensation in the Halifax tech community, 2018]]></title>
            <link>https://medium.com/@jeffandersen/compensation-in-the-halifax-tech-community-2018-213bd0547e3b?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/213bd0547e3b</guid>
            <category><![CDATA[founders]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[startup-life]]></category>
            <category><![CDATA[community]]></category>
            <category><![CDATA[hr]]></category>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Tue, 30 Oct 2018 23:33:16 GMT</pubDate>
            <atom:updated>2018-10-30T23:33:16.648Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="https://halihax.com/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p3DiVslH758sLD5Y" /></a></figure><p><a href="https://halihax.com/">Halihax</a>, wordplay on Halifax and Hacking, is a community of tech minded individuals in Halifax, NS. It all started in 2014 as a meet-up series to engage the local developer community out of <a href="https://medium.com/u/473150029653">Volta Labs</a> with free beer, pizza and presentations from local gurus. We gave up on the live events pretty quickly.</p><p>Later, the brand was resurrected with the help of <a href="https://medium.com/u/3f6845de4e8c">Phil Renaud</a>, VP Engineering at <a href="https://medium.com/u/6ed0d0a087f7">Affinio</a>, as a Slack group so that the Halifax tech community could cross-pollinate and share ideas. Just this week Halihax surpassed 420 members, and represents a wide swath of the Halifax tech community.</p><p>One topic that keeps resurfacing in Halihax is job opportunities. We have a #jobs channel where each week local companies recruit for open positions. These postings often spawn discussions around salary expectations of the community, often from new founders but also from folks transitioning into the community from abroad.</p><p>Last year we ran an ad-hoc survey to gain insight into the active members of the group. Questions included job title, years of experience, base salary, annual bonus amount, whether they worked remotely, and other job related metadata. That survey had 40 respondents, a median base salary of $85,000 and median experience of 8 years.</p><p>We ran the survey again this year, incorporating some community feedback.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/0*Twcm42X7naSCSoT8" /></figure><h3>Survey results</h3><p>Community members wanted to know more about compensation perks like benefits and stock options, how long people had spent in their current role, whether people are considering changing jobs, and if members lived outside of Halifax itself. New questions for this year have been marked with an asterisk.</p><h4><strong>Who are we as a community?</strong></h4><ul><li><strong>73</strong> total <strong>respondents</strong> (↑ 33)</li><li><strong>94.5%</strong> of respondents are <strong>university or college educated</strong> (↑ 17%)</li><li><strong>24.7%</strong> of respondents <strong>work remotely</strong> (↓9.5%)</li><li><strong>83.6%</strong> of respondents touch <strong>code daily</strong> (↓ 6.4%)</li><li><strong>Industry</strong> experience, <em>median</em>, of<strong> 9.0 years</strong> (↑ 1.0 years)</li><li><strong>Role</strong> experience, <em>median</em>, of <strong>2 years</strong>*</li><li><strong>17.8%</strong> of respondents are <strong>actively seeking a new job</strong>*</li><li><strong>23.3%</strong> of respondents will <strong>seek a new job within 12 months</strong>*</li><li><strong>8.2%</strong> of respondents live <strong>outside of Nova Scotia</strong>*</li></ul><h4><strong>How is the community compensated?</strong></h4><ul><li><strong>Base salary</strong>, <em>median</em>, of <strong>$85,000 CAD</strong> (-)</li><li><strong>39.7%</strong> of respondents were given <strong>stock options</strong> (↓ 1.7%)</li><li><strong>93.2%</strong> of respondents are paid in <strong>Canadian dollars</strong>* (otherwise, USD)</li><li><strong>83.6%</strong> of respondents are given <strong>health related benefits</strong>*</li><li><strong>75.34% </strong>of respondents state they are <strong>happy with their compensation</strong>*</li></ul><h3><strong>Pretty charts and distributions</strong></h3><p>Click on any one of these charts to zoom in.</p><h4>Primary Programming Language</h4><p>If you code, what is your primary language?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*C5HixQg9G-KPceEz-ElJzA.png" /></figure><h4>Time Horizon of Seeking New Opportunity</h4><p>In what timeframe might you be seeking a new opportunity?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*MqwFO3rfU8U8vA6KVyM3nA.png" /></figure><h4>Salary vs Programming Language</h4><p>Median salary by programming language</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*x1kXl5yFHwkHOlSePfZtkg.png" /></figure><h4>Salary vs Industry Experience</h4><p>Annual base salary compared to years of industry experience</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*OB7-n4UxnOR1FnEQtPsIQg.png" /></figure><h4>Salary vs Remote Work</h4><p>Median salary of remote vs colocated workers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*v3y4cQ-ALx-HMkEL77PhMg.png" /></figure><h4>Salary vs Hours worked</h4><p>Annual base salary compared to reported work hours per week</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*Q4FKbqjcx9yDBJP6094Jkg.png" /></figure><h4>Secondary programming languages</h4><p>Which languages have you used in the last month?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*7F8Vg17PoooW3bP825O4Lw.png" /></figure><h4>Perceived In-Demand Skills</h4><p>What skills do you see as in-demand currently?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*T0R5ADVJz78ofgDyT5GDcg.png" /></figure><h4>What are you looking to get out of Halihax?</h4><p>Write in answers include: React or JS meetups, Information on salaries for negotiation purposes, and to support and be supported by peers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*tl2pL44WGkHGOHlO8119HA.png" /></figure><h4>Where are companies based?</h4><p>For those respondents who work remote, where is their company based?</p><ul><li>New York, United States</li><li>Haifa, Israel</li><li>Waterloo, ON</li><li>Ottawa, ON</li><li>Moncton, NB</li><li>Saint John, NB</li><li>St. John’s, NL</li><li>Wolfville, NS</li></ul><h4>General feedback about Halihax</h4><ul><li>Promote it more!</li><li>Run meet-ups!</li></ul><h3>Join our community</h3><p>We’ll let you draw your own conclusions on the data as the survey isn’t entirely scientific. Hopefully it gives you some insight into what the tech salary landscape is like in Halifax.</p><p>Anyone is welcome to join in on the action by applying through <a href="https://halihax.com/">our website</a>. We simply ask that members follow our <a href="https://gist.github.com/halihax/f5afa37e7d3152436b40a19ef5aefe47">community guidelines</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=213bd0547e3b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Manifold]]></title>
            <link>https://medium.com/hackernoon/getting-started-with-manifold-f0c2c5ab9e95?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0c2c5ab9e95</guid>
            <category><![CDATA[developer-marketplace]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[developer-services]]></category>
            <category><![CDATA[getting-started-manifold]]></category>
            <category><![CDATA[weekly-sponsor]]></category>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Fri, 22 Jun 2018 16:35:25 GMT</pubDate>
            <atom:updated>2019-05-22T08:20:30.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXfBAtGF5P6EupWw7tXcJQ.png" /></figure><blockquote>Disclosure: <a href="https://bit.ly/2lk2sCG"><strong>Manifold</strong></a><strong>, </strong>the developer marketplace, has previously sponsored Hacker Noon. <a href="https://bit.ly/2lk2sCG"><strong>Use code HACKERNOON2018 to get $10 off any service.</strong></a></blockquote><p><a href="https://bit.ly/2lk2sCG"><strong>Manifold</strong></a> is the marketplace for independent <a href="https://hackernoon.com/tagged/developer">developer</a> services. A place for you to find, organize, and connect the best cloud services directly to your team’s workflow, under a single account, with one bill at the end of the month.</p><p>We offer two different methods of <a href="https://dashboard.manifold.co/register">creating a Manifold account</a>: connect your GitHub account, or enter your email address and a password. If you sign up by email, you’ll need to verify your address to continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0GCf7SWnTLbPz6JW" /><figcaption>Create a project, add a resource or start a new team from the Manifold Dashboard.</figcaption></figure><h3>Create a team and collaborate</h3><p><a href="https://hackernoon.com/tagged/manifold">Manifold</a> allow you to organize all of your cloud services under a single account. Teams make it easy to share access to the service’s configuration and secrets, their web control panel, and aggregates everything on a single bill.</p><p>This means no more chasing down who owns which account, or who can give you the API keys you need to communicate to core services.</p><p>Teams also makes developer on-boarding super simple — invite a new person to your team and they gain access to exactly what they need.</p><h3>Start your first project</h3><p>Projects enable you to create logical groupings of services that belong to a single application, or component of your infrastructure.</p><p>Each project has a unique name, and can house as many services as you need. You then will be able to access the configuration for all the services in that project together. Because Projects aid in visualizing your stack, it simplifies on-boarding even further.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b_NNTfpLDQ9x-7Qv" /><figcaption>Choosing your first Service on Manifold.</figcaption></figure><h3>Discover services you need, add external configuration</h3><p>Your Manifold Dashboard starts by showcasing all the independent developer services that are available to be provisioned. All services can be filtered by Category so you can easily find the one you want.</p><p>There are three steps to getting started with your first service:</p><ol><li>Choose the product you want</li><li>Pick a plan</li><li>Select your customizations</li></ol><p>Manifold then takes care of provisioning exactly what you need from the provider.</p><p>If you have services which are not available through Manifold, or want to bring all of your configuration together in one project you can Bring Your Own Service. Using our encrypted key value store you can add any secrets or configuration you need to import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g4suU4Ax3LZf8U4x" /><figcaption>An IronCache instanced named cache-master part of the queue-service project.</figcaption></figure><h3>Configure your services</h3><p>Most services come with their own dashboards where you can view information about your usage and control your preferences. Manifold makes it easy to share access to these through Single sign-on. One click and you’ll be redirected, authenticated, to the service’s control panel.</p><h3>Integrate your development workflow</h3><p>Now that you have a project established, it’s easy to deliver your configuration and secrets directly to your application through one of our workflow integrations.</p><p>With development use the <a href="https://blog.manifold.co/manage-your-cloud-services-like-a-developer-with-our-cli-tool-f02c88d9a7fd">Manifold CLI</a> to inject your config as environment variables through <em>manifold run</em>.</p><p>When you’re ready to deploy to production you can use our <a href="https://blog.manifold.co/manifold-%EF%B8%8F-kubernetes-terraform-9dcf09703e5a">Terraform or Kubernetes integrations</a> to easily deploy your cloud services with your existing infrastructure.</p><h3>What are you waiting for?</h3><p><a href="https://bit.ly/2lk2sCG">Redeem the coupon<strong> <em>HACKERNOON2018</em> </strong>and receive $10 towards any service of your choice.</a></p><figure><a href="https://bit.ly/2lk2sCG"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__uElh8xLjsty7zqjqtHmg.png" /></a></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0c2c5ab9e95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/getting-started-with-manifold-f0c2c5ab9e95">Getting started with Manifold</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future of Cloud Services is Borderless]]></title>
            <link>https://medium.com/hackernoon/the-future-of-cloud-services-is-borderless-4710914f1b01?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4710914f1b01</guid>
            <category><![CDATA[future-of-cloud-services]]></category>
            <category><![CDATA[weekly-sponsor]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Mon, 22 Jan 2018 18:09:38 GMT</pubDate>
            <atom:updated>2019-05-22T08:21:01.923Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uP8Y_fZQIhm6xXgO2mKHXQ.png" /></figure><blockquote><em>Disclosure: </em><a href="https://goo.gl/BtueKD">Manifold</a><em>, the developer marketplace, has previously sponsored Hacker Noon. </em><a href="https://goo.gl/BtueKD">Use code HACKERNOON2018 to get $10 off any service.</a></blockquote><p><a href="https://hackernoon.com/tagged/cloud">Cloud</a> compute is a commodity. Containerization <a href="https://hackernoon.com/tagged/technology">technology</a> and platforms such as <a href="https://kubernetes.io/">Kubernetes</a> level the compute playing field to the point that it doesn’t matter where your VMs are running.</p><p>They remove the risk of vendor-specific implementations and platform quirks. You now orchestrate your infrastructure in a uniform manner across the board.</p><p>You no longer make one gargantuan decision to lock yourself in a single ecosystem. You mix and match, evaluating where you can get the best support and price for the capacity you require.</p><h3><strong>Falling for the single ecosystem trap</strong></h3><p>It used to be that you would get as many services as you could from your cloud compute provider. This was safe and easy because every deployment story began with selecting a compute solution that fit your workflow.</p><p>Next you would add the additional services you don’t want to manage yourself: DNS, monitoring and logging, email, database. Your cloud compute provider typically offered a “good-enough” version of these, causing you to double down on the ecosystem. It was easy, you just clicked a few buttons in the same place you managed every other service and you were done.</p><p>What made it even easier was cloud compute providers would sweeten the pot with large amounts of free credit, giving you the freedom to just keep piling services on.</p><p>This is where they set their trap.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FZ1LYiyIPhnG9O%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FZ1LYiyIPhnG9O%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FZ1LYiyIPhnG9O%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="233" frameborder="0" scrolling="no"><a href="https://medium.com/media/70277751880bd30b98841472cd8bcb33/href">https://medium.com/media/70277751880bd30b98841472cd8bcb33/href</a></iframe><p>Your monthly bill steadily climbed as you added new services, even ones you didn’t really need. Each service had its own pricing scheme and calculator, making it difficult to know exactly what you are signing on for.</p><p>Each of these “good-enough” services would have their own platform-specific quirks and implementation details. Only working together through proprietary APIs and integrations.</p><p>Soon the honeymoon period ends and the credits dry up. You would face the daunting reality that you had to pony up the cash to keep going because the effort to move was too daunting. You’re locked-in.</p><h3><strong>The perils of multi-ecosystem architectures</strong></h3><p>The alternative to locking yourself in to one provider’s ecosystem is to purchase the best tool for the job regardless of where you source it from, but this has its own problems.</p><p>Now you have multiple people adding services from several vendors to your application.</p><p>Your billing is fragmented with different pricing models and no longer comes on a single bill to the appropriate person at the end of the month.</p><p>You lose the ability to predict how your costs will scale as your application grows. You lose track of services that are dormant, but still being paid for. You have multiple payment methods and inevitably one will be rejected at some point in time causing downtime and confusion.</p><p>There is no single source of truth. You have no visibility into the breadth of services necessary to operate your application, complicating onboarding new developers and the auditing of who has access to what.</p><p>Developers have to go to multiple sources to obtain configuration necessary to run your application and one thing we all know is: developers can be lazy. Workflow fatigue results in less than ideal handling of your secrets (values in plain text, stored on disk, copied and pasted), which are critical to security and integrity.</p><p>It’s a mess, and annoying to work with.</p><h3><strong>How we’re able to embrace the chaos at Manifold</strong></h3><p>At <a href="https://goo.gl/BtueKD">Manifold</a> our team embraces the fragmentation of the multi-ecosystem infrastructure. We want to source the best tool for the job regardless of where it comes from. We want our developers to have the freedom to add services when they’re needed and not have to go through lengthy approval processes.</p><p>We have an ace in the hole.</p><p>We use our own product to manage our cloud services and configuration. <a href="https://goo.gl/BtueKD">Manifold’s marketplace</a> is a single location to find, buy and manage cloud services from multiple vendors that are the backbone of your applications.</p><p>We have a number of cross-discipline teams that work on multiple applications, they need to be able to focus on building our applications and not worry about obtaining configuration, secrets, or what service is managed by which team.</p><p>Every time a developer provisions a new service, they don’t have to worry about adding billing information. We get a single bill at the end of the month detailing all the services being used by the company.</p><p>We consolidate our cloud services, internal services and general configuration in a single location to make a seamless workflow regardless of which platform our application is deployed to.</p><p>Each application is grouped into projects, allowing us document exactly which services and configuration are required for which application. No more chasing down people from another team just to get an API token.</p><figure><a href="https://dashboard.manifold.co/register?utm_campaign=hackernoon_jan&amp;utm_source=hackernoon.com&amp;utm_medium=sponsorship&amp;utm_content=screenshot"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAAvJC55whmXWyZ7bw4Vxg.png" /></a><figcaption>A preview of the Manifold dashboard for ACME Corp</figcaption></figure><p>Sometimes the services we use aren’t available directly through the Manifold marketplace, this is where we add Custom Configuration resources. This lets us bring any external secrets into the Manifold ecosystem to live side by side with purchased services.</p><p>Similarly, sometimes we create our own internal services (workers, APIs, etc) that need to speak to each other, and Custom Configuration lets us bring those along too giving us a single source of truth.</p><p><strong>Configuration where you need it</strong></p><p>In development, all of our developers use the <a href="https://blog.manifold.co/manage-your-cloud-services-like-a-developer-with-our-cli-tool-f02c88d9a7fd">Manifold CLI</a> to seamlessly inject the required configuration to their application. No more config files, no more plaintext secrets. The configuration variables are securely delivered at runtime from Manifold’s API.</p><p>In production, we use Terraform to deploy our infrastructure from code. Using the <a href="http://github.com/manifoldco/terraform-provider-manifold">Manifold Terraform Provider</a> we’re able to define exactly which configuration needs to go where in our architecture, reconciling the most up to date values with every deploy. Our operations team no longer have to worry about having the correct keys for services another team provisioned.</p><p>With Kubernetes on the horizon for our stack, as we prepare for a true borderless ecosystem, we implement the same functionality using a <a href="http://github.com/manifoldco/kubernetes-credentials">Custom Resource Definition</a> that continuously reconciles the correct configuration and secrets with our cluster. This makes it even easier for us to migrate clouds when we want to.</p><h4><strong>A model for now and the future</strong></h4><p>Ecosystem lock-in is no longer the norm. Developers want choice, they want to be able to choose the best service for the job that affords them the best workflow.</p><p>As cloud computing becomes a commodity, the portability of your services and config becomes paramount. You need the flexibility to change where your applications are deployed without sinking huge effort into translating to a new platform.</p><p><a href="https://goo.gl/BtueKD">Manifold</a> takes the first steps towards modeling your services and config in such a way that you don’t need to think about which platform your application is being deployed and is continuing to evolve the multi-ecosystem workflow.</p><figure><a href="https://dashboard.manifold.co/register?utm_source=hackernoon.com&amp;utm_medium=sponsorship&amp;utm_campaign=hackernoon_jan&amp;utm_content=content"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzs9X5qdiZ3VbKAbBBeGig.png" /></a></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4710914f1b01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/the-future-of-cloud-services-is-borderless-4710914f1b01">The Future of Cloud Services is Borderless</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six things to consider when designing for the command line]]></title>
            <link>https://medium.com/manifoldco/six-things-to-consider-when-designing-for-the-command-line-2b9b9d6ad009?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b9b9d6ad009</guid>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[command-line]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Wed, 09 Nov 2016 15:17:15 GMT</pubDate>
            <atom:updated>2016-11-14T13:20:10.729Z</atom:updated>
            <content:encoded><![CDATA[<p>The nature of my work the last several months has immersed me on the command-line. We’re working on a secure, shared workplace for secrets. You can <a href="https://torus.sh/">check out what we’re up to here</a>, but let’s keep going.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4gPlpYNEK2PWPYjkbi6ug.png" /></figure><p>I’ve learned over the last few months that maintaining UX across an entire suite of commands is extremely difficult.</p><p>How are subcommands delimited? What happens when a required flag is missing — are they prompted for it? Lists or tables? Added whitespace or not? Is the language consistent? Let’s not forget success and error states.</p><p>Once you make a checklist, you realize you’re herding cats.</p><p>Most people wouldn’t associate user experience with the command line, but in many respects you need proper UX on the command line even more than web because you can’t rely on visuals to aid the user.</p><p>Just like you have a brand/style guide for your website, you have to establish similar constraints and norms for a command line tool.</p><blockquote><strong><em>Sidebar:</em></strong> At Manifold while hiring a designer we often joked that we needed to list the position as ASCII Designer. I’d say the position didn’t fall too far from that tree.</blockquote><p>So here are six things to keep in mind when you start building a CLI tool:</p><h4><strong>1. </strong>What is the barrier to entry with installation?</h4><p>We learned early on with Torus that installation flow is a huge part of UX. We wrote our initial prototype in Node.js, which required users to have that runtime installed (a relatively large barrier to entry). That is what pushed us to make the switch to Go, which enables us to be portable to any platform with a precompiled binary.</p><p>Your command has to be easily accessible at the point of need. If your target user is deploying Linux, you want to be available in the package manager that is applicable to that distribution. If it’s not available in their conventional source, then they’re going to look for alternatives to your tool.</p><h4>2. Are your commands intuitive?</h4><p>Every detail matters on the command line: Things aren’t just a click away, the user has to define what they are looking to do with their keyboard.</p><p>On one hand you have the benefit of a typical command-line user being familiar with reading help text, or man pages. If they can’t recite the commands as long as you’ve added documentation they’re likely to figure it out.</p><p>That’s no good, though. You want the experience to be intuitive.</p><p>You want a user to be able to understand how to navigate what you’ve built by learning the structure of a few base commands, and then extrapolate that everywhere else naturally.</p><h4>3. How annoying is it to type each command?</h4><p>Our initial prototype used colons to delimit subcommands <em>(e.g torus projects:create [name])</em>. We wrote the entire prototype this way, but we came to the realization that it was really awkward to keep reaching for shift just to enter a colon: we got rid of every colon and use spaces now.</p><p>Avoid inputs which are non-standard or cumbersome to enact.</p><h4>4. Are your error messages helpful?</h4><p>Your application is performing a million operations behind the scenes. That’s why the user is employing your tool in the first place: to reduce the complexity for <em>them</em>.</p><p>But what happens when a step fails? Your tool ideally does a good job at recovering from a situation, but in the event that it doesn’t you need to surface relevant content that will aid the user in solving the problem themselves. We were actually guilty of this and were actually <a href="https://github.com/manifoldco/torus-cli/issues/25">hiding context from the user</a> which made the experience confusing.</p><p>You want to surface clear and concise errors, which tell the user what action (if any) must be taken, so that the user doesn’t have to ask for help.</p><h4>5. Do you maintain consistent output?</h4><p>You’ve gone on a vision quest and reconciled differing opinions of spaces vs tabs, new lines or not, and how many emoji you’re comfortable scattering throughout. That sets you up to be consistent.</p><p>Command line tools need to be consistent, as they are often used in scripts.</p><p>Exit codes, table formatting, newlines and whitespace. All of these must remain consistent with the type of action you’re performing so that it makes it easy to understand and parse the output in a script.</p><h4>6. What is your automation strategy?</h4><p>We’re human, we’re error-prone.</p><p>We want to be able to move fast, continuously deploy and deliver new features to our customers. We can’t afford regressions.</p><p>At my last job (GoInstant/Salesforce, working on <a href="https://techcrunch.com/2011/09/13/goinstant-is-browser-sharing-with-no-downloads/">shared web</a>, and <a href="https://www.youtube.com/watch?v=bvBxWFpko3g">mobile screen sharing</a>) we focused a lot of resources on QA automation in multi-user/interactive applications. I learned a lot along the way about what it takes to test complex edge cases, and it’s something I’ve drawn on to apply to the command line.</p><p>You need to have an automation strategy so you can remove humans from the process of maintaining consistency, as we’re simply not wired for it.</p><p>With <a href="https://torus.sh/">Torus</a> we perform mostly manual QA, but are working towards fully automating that process. I’ll be writing another post specifically about our approach to automation soon.</p><p>Hopefully these lessons we have learned along the way will help you achieve success in building your own robust command line utility.</p><p>Read more about Torus at <a href="https://torus.sh,/">https://torus.sh</a>, or check out the code on <a href="https://github.com/manifoldco/torus-cli">Github</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*Jy3Y6m5iabk8oKdduknDPA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b9b9d6ad009" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manifoldco/six-things-to-consider-when-designing-for-the-command-line-2b9b9d6ad009">Six things to consider when designing for the command line</a> was originally published in <a href="https://medium.com/manifoldco">manifoldco</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Do you know where your app’s secrets are?]]></title>
            <link>https://medium.com/manifoldco/do-you-know-where-your-apps-secrets-are-e3e472bd9edb?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3e472bd9edb</guid>
            <category><![CDATA[developer]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[configuration-management]]></category>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Wed, 02 Nov 2016 09:01:48 GMT</pubDate>
            <atom:updated>2016-11-16T14:16:16.168Z</atom:updated>
            <content:encoded><![CDATA[<p>Development today relies on multiple teams, services, and environments all working in unison, and they’re difficult to orchestrate.</p><p>Mismanagement of secrets can land you on the front page of popular tech publications as part of an embarrassing data breach that results in loss of trust and future business. It happens to even the biggest of companies, for example: LinkedIn, Dropbox, Yahoo.</p><p>There is an ongoing tug of war between productivity and security which typically tips in favour of productivity. We take shortcuts and make mistakes resulting in compromised security posture.</p><p>You shouldn’t have to choose, so we’ve created <a href="https://torus.sh/"><strong>Torus</strong></a>.</p><p>Allow me to demonstrate how easy it is to take a project (in this case, a readily available open source one) and configure it with Torus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTQkDwABZ0UxSF7o5EM9Wg.png" /></figure><h3>Choose your project</h3><p>I need an application to start with, something that requires secrets.</p><p>I’ve chosen <a href="https://github.com/rauchg/slackin"><strong>SlackIn</strong></a>, which is a utility that provides self-service sign-up for Slack teams. First we clone the repository:</p><pre><strong>$ </strong>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:rauchg/slackin.git<br><strong>$</strong> cd slackin</pre><p>Install the required dependencies:</p><pre><strong>$ </strong>npm install</pre><p>And if you haven’t already, install <a href="https://torus.sh/"><strong>Torus</strong></a> and signup:</p><pre><strong>$</strong> npm install -g torus-cli<br><strong>$ </strong>torus signup</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DDgdm9i0uRfnECy5tGwYpA.png" /></figure><h3>Link it up</h3><p>We want to link our Torus account to the project using torus link<br>This allows the codebase to be physically linked to the Torus project.</p><pre><strong>$ </strong>torus link<br>✔ Select organization: skywalker<br>✔ Create a new project: slackin<br>Project slackin created.</pre><pre>This directory and its subdirectories have been linked to:<br>Org: skywalker<br>Project: slackin</pre><pre>Use ‘torus status’ to view your full working context.</pre><p>I now have a .torus.json file in the project directory. I’ll commit that to git:</p><pre><strong>$</strong> git add .torus.json &amp;&amp; git commit -m “Add .torus.json”</pre><p>By committing the .torus.json file, people who collaborate with me on this project will automatically be linked to the <em>skywalker</em> org and the <em>slackin</em> project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcDmnYMU0YHX64E6QacKlg.png" /></figure><h3>Secrets and where to store them</h3><p>SlackIn accepts four different environment variables as config:</p><ul><li>SLACK_SUBDOMAIN</li><li>SLACK_API_TOKEN</li><li>SLACK_CHANNELS <em>(optional)</em></li><li>SLACK_COC <em>(optional)</em></li><li>PORT</li></ul><p>We need to generate the two required SLACK items before going any further. We’re still in development, so let’s register a <a href="https://slack.com/">Slack team</a> specifically for test purposes, and once we’re logged in as admin we’ll <a href="https://api.slack.com/web">generate a new API token</a>.</p><p>For demonstration purposes my development team name and token will be:</p><pre>SLACK_SUBDOMAIN=skywalker-ranch-dev<br>SLACK_API_TOKEN=xoxp-8675309–9035768–1042534-r2d2</pre><p>Now that I have these values in plain text (taken from the Slack website output), I need to store them for use in my application.</p><p>Inside every org each member is given their own development environment. This allows me to experiment, and develop without impacting secrets shared across multiple teams or services.</p><p>First I set the port that I want the process to listen on, in development:</p><pre><strong>$</strong> torus set port 1337</pre><p>I set the subdomain and token value using torus set:</p><pre><strong>$ </strong>torus set slack_subdomain skywalker-ranch-dev<br>…<br>Credential slack_subdomain has been set at /skywalker/slackin/dev-jeff/default/*/*/slack_subdomain</pre><pre><strong>$ </strong>torus set slack_api_token xoxp-8675309–9035768–1042534-r2d2<br>…<br>Credential slack_subdomain has been set at /skywalker/slackin/dev-jeff/default/*/*/slack_api_token</pre><p>I’ll throw away the plaintext values, because I’ve now encrypted them inside Torus.</p><p>If I need to view the plaintext values, I can retrieve and decrypt the values using torus view.</p><pre><strong>$</strong> torus view<br>SLACK_SUBDOMAIN=skywalker-ranch-dev<br>SLACK_API_TOKEN=xoxp-8675309–9035768–1042534-r2d2<br>PORT=1337</pre><h3>Start your processes</h3><p>I mentioned earlier that SlackIn requires two environment variables in order to run, if we try running the process without those, it fails to run:</p><pre><strong>$</strong> ./bin/slackin<br>Usage: slackin [options] &lt;team-id&gt; &lt;api-token&gt;<br>…</pre><p>Because I’ve laid out what my secrets look like in Torus I can simply call torus run, specifying the necessary environment (defaulting to my personal dev environment when not provided).</p><p>This will inject the configured values into the application.</p><pre><strong>$</strong> torus run ./bin/slackin<br>fetching<br>listening on 0.0.0.0:1337<br>ready</pre><p>Here I can test to make sure it does everything I need. I can make changes to the project as I see fit without disrupting production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRDWMKXHF4C-vgl7bEvwpA.png" /></figure><h3>Sharing secrets</h3><p>Torus prides itself on being a tool that can be used in development and all the way to production.</p><p>I have a development application and now I want to deploy it to production. The application will need all of the same configuration, but in some cases we want environment-specific values.</p><p>First I create a production environment for my Torus org:</p><pre><strong>$ </strong>torus envs create production<br>✔ Org name: skywalker<br>✔ Project name: slackin<br>✔ Environment name: production</pre><pre>Environment production created.</pre><p>Next I need to set up my production Slack team just like I did earlier for development. My example team name and token for production will be:</p><pre>SLACK_SUBDOMAIN=skywalker-ranch<br>SLACK_API_TOKEN=xoxp-7779311–9035768–1042534-bb8</pre><p>Let’s take another look at what secrets I have set.<br>This time I’ll use torus ls, which allows me to explore the secrets across my environments by their path:</p><pre><strong>$ </strong>torus ls<br>Listing secrets within path: /skywalker/slackin/*/*/*/*<br>/skywalker/slackin/dev-jeff/default/*/*/port<br>/skywalker/slackin/dev-jeff/default/*/*/slack_api_token<br>/skywalker/slackin/dev-jeff/default/*/*/slack_subdomain</pre><p>Two of my secrets are environment specific. I don’t want to mix and match the slack values between environments, but PORT on the other hand can be the same across all environments.</p><p>Port is currently set at the following path:</p><pre>/skywalker/slackin/dev-jeff/default/*/*/port</pre><p>Notice the third segment (dev-jeff) — this is the environment value. I stored the port in my personal development environment, which makes it inaccessible to other environments.</p><p>To rectify this I can set a new port, 5000, across all environments using:</p><pre><strong>$ </strong>torus set port 5000 --environment “*”<br>…</pre><pre>Credential port has been set at /skywalker/slackin/*/default/*/*/port</pre><p>When I view the secret values again I see that port value is still 1337, why?</p><pre><strong>$</strong> torus view<br>PORT=1337<br>SLACK_API_TOKEN=xoxp-8675309–9035768–1042534-r2d2<br>SLACK_SUBDOMAIN=skywalker-ranch-dev</pre><p>Torus cascades the values depending on their specificity.</p><p>Originally I set a port in my personal development environment dev-jeff. This will override the globally specified portof 5000 with my personal value of 1337 because it is more specific.</p><p>I can switch to using the global value by unsetting the PORT in my development environment:</p><pre><strong>$</strong> torus unset port<br>You are about to unset “/skywalker/slackin/dev-jeff/default/*/*/port”.<br>This cannot be undone.<br>✔ Do you wish to continue? [Y/n] Y<br>…</pre><pre>Credential port has been unset at /skywalker/slackin/dev-jeff/default/*/*/port.</pre><p>Now we see 5000 as the PORT value:</p><pre><strong>$ </strong>torus view<br>PORT=5000<br>SLACK_API_TOKEN=xoxp-8675309–9035768–1042534-r2d2<br>SLACK_SUBDOMAIN=skywalker-ranch-dev</pre><p>I can now finish setting my production specific slack secrets:</p><pre><strong>$ </strong>torus set slack_api_token xoxp-7779311–9035768–1042534-bb8 -e production<br>…</pre><pre>Credential port has been set at /skywalker/slackin/production/default/*/*/slack_api_token</pre><pre><strong>$ </strong>torus set slack_subdomkain skywalker-ranch -e production<br>…</pre><pre>Credential port has been set at /skywalker/slackin/production/default/*/*/slack_subdomain</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7Qp_gdnSHyh_-5XX-bU1Q.png" /></figure><h3>Access control</h3><p>Thus far I have been using my personal Torus account to bootstrap the secrets. When we run this project in production I won’t want to use my personal credentials.</p><p>This is where Machines come in handy. We’re able to create an instance that acts like a user and interacts with configuration it is given access to.</p><p>We create a machine, placing it on a team for slackin instances. Torus suggests a name for this specific machine instance, which we will use.</p><pre><strong>$ </strong>torus machines create<br>✔ Org name: skywalker<br>✔ Create a new team: slackin<br>✔ Enter machine name: slackin-bb8r2d2</pre><pre>Machine team slackin created for org skywalker.<br>Creating machine “slackin-bb8r2d2”<br>…<br>Machine created.</pre><pre>You will only be shown the secret once, please keep it safe.</pre><pre>Machine ID: 04bvea[redacted]y91v8<br>Machine Token ID: 04c49f[redacted]hf4xm<br>Machine Token Secret: ZcA[redacted]sk1</pre><p>I save and put aside the Token ID and Token Secret values for deployment later.</p><p>I now have a machine called slackin-bb8r2d2, which is a member of the slackin team, but the slackin team has no access yet.</p><p>I need to tell the team what it can access using torus allow:</p><pre><strong>$ </strong>torus allow rl /skywalker/slackin/production/default/*/*/* slackin<br>Policy generated and attached to the slackin team.</pre><pre>Effect: allow<br>Action(s): read, list<br>Resource: /skywalker/slackin/production/default/*/*/*</pre><pre>Necessary permissions (read, list) have also been granted.</pre><p>I’ve given the machine rl (Read, List from the possible Create Read Updated Delete List) access to all secrets in my production environment, as it’s best practice not to give write access to deployments.</p><p>This will allow slackin-bb8r2d2 to read the values it needs to run SlackIn, but won’t allow anyone who may gain access to that server to overwrite or read anything outside of its scope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVxbfMPSXPb20qVSGxEZFw.png" /></figure><h3>Deployment</h3><p>Now that our Torus environment is bootstrapped, we can go ahead and deploy our application.</p><p>Once my application server’s environment is set up, and I’ve deployed the code, I can set up Torus to run the process. We’ll talk about more complex deployments in a future post.</p><p>I have a Machine Token ID and a Machine Token Secret.</p><p>In deployment to platforms like Heroku or running in CI/CD systems, where you are trusting the security of the operating environment, Torus accepts these values through environment variables.</p><pre><strong>$</strong> export TORUS_TOKEN_ID=04c49f[redacted]hf4xm<br><strong>$</strong> export TORUS_TOKEN_SECRET=ZcA[redacted]sk1</pre><blockquote><strong>Note: </strong>We will cover more complex deployment options which properly segregate your token ID and Secret from your application user in a future post.</blockquote><p>I also need to specify which Torus environment this machine belongs in:</p><pre>$ export TORUS_ENVIRONMENT=production</pre><p>Now the environment is setup to authenticate as our machine named slackin-bb8r2d2.</p><p>When I start my process, for example with <a href="https://github.com/Unitech/pm2">PM2</a>, the secrets are automatically read into the process environment:</p><pre><strong>$</strong> pm2 start torus -- run ./bin/slackin<br>[PM2] Applying action restartProcessId on app [torus](ids: 0)<br>[PM2] [torus](0) ✓<br>[PM2] Process successfully started</pre><p>I’m now running SlackIn for two environments: development and production using Torus to manage the secret configuration.</p><h3>Learn more</h3><p>You can read more about how Torus fits into your workflow in our <a href="http://torus.sh/docs">documentation</a>, as well as learn about the underlying <a href="https://www.torus.sh/docs/latest/internals/crypto">cryptography</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3e472bd9edb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/manifoldco/do-you-know-where-your-apps-secrets-are-e3e472bd9edb">Do you know where your app’s secrets are?</a> was originally published in <a href="https://medium.com/manifoldco">manifoldco</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Node.js REST API with Express (Part Two)]]></title>
            <link>https://medium.com/@jeffandersen/building-a-node-js-rest-api-with-express-part-two-9152661bf47?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9152661bf47</guid>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Sat, 31 Jan 2015 01:31:22 GMT</pubDate>
            <atom:updated>2015-02-18T13:41:58.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Node.js REST API with Express</h3><h4>Part 2, Making your API more robust</h4><h3>We have the foundation.</h3><h4>Let’s build on top of what we have learned in <a href="https://medium.com/@jeffandersen/building-a-node-js-rest-api-with-express-46b0901f29b6">Part </a>1.</h4><p>Our API has very basic functionality. We’re able to define routes, read information sent from a client, perform database queries and respond with a resource object.</p><p>How do we deal with malicious input? How do we handle returning subsets of our data? How do we accept multi-part forms (e.g file uploads)? When we’re ready to publish our API how do we ensure that we maintain existing functionality when making changes or adding new features?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsLdpjvsVZRFgk1Ef2Jvjw.jpeg" /><figcaption>The internet can be a scary place</figcaption></figure><h3>Validation</h3><p>We always want to make sure that data our API is processing is validated.</p><p>Data validation helps ensure that our handlers operate using only clean, correct and useful data. This helps maintain the health of your dataset, and improve the user experience of your API. Most importantly, it helps prevent the processing of malicious requests. We cannot trust the source of the request because any request sent to your server could be spoofed or modified in-flight before it gets to us.</p><p><strong>What’s the worst that could happen?</strong></p><ul><li>Wrong type of data stored in database, breaking future requests</li><li>Unintentional, or malicious access to data other than requested</li><li>Malicious code runs on your server exposing your database and codebase</li></ul><h4>How do we implement validation in Express?</h4><p>Validation can be done in the routes’ URI (against parameters), directly in<br>each of the routes’ handlers (<em>before using the data</em>), or by using middleware that can be applied to multiple routes (less duplication of code).</p><p>For our tutorial we’ll be implementing parameter validation in the URI, and data validation middleware with help from a validation library made specifically for Express.</p><h4>Parameter validation</h4><p>We have data entering our application two different ways currently. We have POST bodies providing payloads of data, and we have URI parameters defined in our routes (e.g <em>/photos/:id</em>).</p><p>Express provides native support for <a href="http://en.wikipedia.org/wiki/Regular_expression">Regular Expressions</a> (referred to as regexes) in URIs in order to do basic validation on a parameter. Regexes can become complex very quick, but we can keep them simple for parameter validation.</p><p>For our Photo resource we allow users to take actions based on providing an ID in the URI. In our database, this ID value is an integer. So for our routes, we want to ensure that the ID parameter that we accept is numeric, let’s first review our existing route, which would also match <em>/photo/random-text </em>:</p><pre>photoRouter.get(‘/:id’, lookupPhoto, function(req, res) {</pre><p>Express allows us to add a regex directly after the name of the parameter. We want to allow any length of number:</p><blockquote><em>([0–9]+)</em></blockquote><p>Where + indicates one or more of the preceding definition (a number 0–9).</p><pre>photoRouter.get(‘/:id([0-9]+)’, lookupPhoto, function(req, res) {</pre><p>Now if someone were to request <em>/photo/random-text</em> it will not be captured by this route and proceed to check the rest of the router for a match (resulting in <em>Cannot GET /photo/random-text</em>).</p><h4><strong>Data validation</strong></h4><p>Our API has two different routes which currently accept data: <em>create</em> (POST) and <em>update</em> (PATCH) of a resource. Both data objects should accept values for fields found in our Postgres database.</p><p>By using placeholders in creation of our SQL queries (e.g <em>WHERE id = $1</em>) we are protecting ourselves against SQL injection. But what if we weren’t using a library that took care of this for us? What if we were constructing the SQL manually and executing it?</p><blockquote><em>var sql = “SELECT * FROM photo WHERE id=“ + photoId;</em></blockquote><p>What if someone were to pass the following in place of an actual ID?</p><blockquote><em>0%20OR%20id%20&gt;%200</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoViYBq_M4X-fNt3MTD8BQ.png" /><figcaption>Example of SQL injection while constructing SQL queries manually without validation</figcaption></figure><p>I was able to return a record (in the screenshot above) because the data I supplied evaluated to an additional expression, circumventing the logic:</p><blockquote><em>SELECT * FROM photo WHERE id=0 OR id &gt; 0</em></blockquote><p>This particular attack would be mitigated by our parameter validation.</p><p><strong>It’s not all about security</strong></p><p>Data validation plays a large role in the user experience of your API, and health of your dataset. By validating incoming data you’re able to catch typos or mistakes by a client in constructing a request. This means that your final data set is predictable and clean.</p><p>Perhaps a client attempts to create a Photo resource object for an Album which does not exist yet? Or supplies a description which is longer than your application can support? You want to prevent these records from being stored in the database.</p><p>The data validation library we’ll be using is <a href="https://github.com/ctavan/express-validator">express-validator</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*rs2QTUeVeaUjFi2y7zu6xQ.jpeg" /></figure><p><strong>Why not write our own validation library?</strong></p><p>When you’re just learning about validation it can be good to attempt to write the validation yourself and learn the basics, but when you’re building a product you want to spend as much time as possible enhancing <em>your product</em> and not building bricks for the foundation.</p><p>Open-source software projects also carry the benefit of generally being built by developers who focus in that particular area of expertise. This results in their implementation being more comprehensive than something you may write yourself.</p><p><strong>What does our inbound request look like?</strong></p><p>For this section we’ll be using the <em>POST /photo </em>route as our guinea pig. When a client attempts to create a new Photo resource object there are three parameters which are required: description, filepath, and album_id.</p><p>What are our expectations of these values?</p><ul><li>description — should be text, of a certain length</li><li>album_id — should be numeric</li><li>filepath — we will return to this value later in the tutorial</li></ul><p>Let’s start by loading the Express Validator middleware to our <em>index.js </em>file:</p><pre>var express = require(‘express’);<br>var bodyParser = require(‘body-parser’);<br><strong>var expressValidator = require(‘express-validator’);</strong></pre><pre>var app = express();</pre><pre>app.use(bodyParser.json({ type: ‘application/json’ }));<br>// We add the middleware <em>after </em>we load the body parser<br><strong>app.use(expressValidator());</strong></pre><p>What this is doing is extending the standard Express request object with additional helper functions for validating post body, query or parameter data.</p><p>To enable us to re-use the same validation on the PATCH route, we need to make our own middleware that leverages the functions which Express Validator gives us.</p><pre>function validatePhoto(req, res, next) {<br>  req.checkBody(‘description’, ‘Invalid description’).notEmpty();<br>  req.checkBody(‘album_id’, ‘Invalid album_id’).isNumeric();</pre><pre>  var errors = req.validationErrors();<br>  if (errors) {<br>    var response = { errors: [] };<br>    errors.forEach(function(err) {<br>      response.errors.push(err.msg);<br>    });</pre><pre>    res.statusCode = 400;<br>    return res.json(response);<br>  }</pre><pre>  return next();<br> }</pre><p>So what’s happening here?</p><p>First we are using the <em>checkBody</em> helper function that is supplied from Express Validator to test the description (not empty), and the album_id (numeric). <em>checkBody</em> works by supplying the field name, and an error message. Then on the resulting object we can chain specific tests (in this case, notEmpty and isNumeric) on the values for those fields.</p><pre>req.checkBody(‘description’, ‘Invalid description’).notEmpty();<br>req.checkBody(‘album_id’, ‘Invalid album_id’).isNumeric();</pre><p><strong>Hold on, what is method chaining (cascading)?</strong></p><p>Method chaining is the process of calling one method after another on a particular object. In our example: we first created a validation object using the Express-Validator library (<em>checkBody)</em> then we chained the applicable tests. Here’s an example of chaining methods to affect a string:</p><pre>function MyString(value) {<br>  this.value = value;<br>  return this;<br>}</pre><pre>MyString.prototype.upperCase = function() {<br>  this.value = this.value.toUpperCase();<br>  // Because we return the appropriate context: this<br>  // We&#39;re able to continue calling methods<br>  return this;<br>};</pre><pre>MyString.prototype.addQuotes = function() {<br>  // Our mutated value can be updated again<br>  this.value = &#39;&quot;&#39; + this.value + &#39;&quot;&#39;;<br>  return this;<br>};</pre><pre>var str = new MyString(&#39;Hey, this is awesome!&#39;);<br>str.upperCase().addQuotes();<br>console.log(str.value); // &quot;HEY, THIS IS AWESOME!&quot;</pre><p>Pretty cool, right?</p><p>Next in our example we are checking whether the validation library detected any issues with the tests which were performed:</p><pre>var errors = req.validationErrors();</pre><p>From the <a href="https://github.com/ctavan/express-validator#validation-errors">documentation of Express Validator</a>, we know that errors is an array when something is wrong, and falsey if all tests pass. This allows us to return a specific error response (status code 400, Bad Request), or continue to the next middleware (the route handler).</p><p>To actually use this middleware we simply inject it before our route handler (just like our lookup function from <a href="https://medium.com/@jeffandersen/building-a-node-js-rest-api-with-express-46b0901f29b6">Part 1</a>):</p><pre>photoRouter.post(‘/’, validatePhoto, function(req, res) {</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C81KlC5oej-XLPbESB327g.png" /><figcaption>Request sent to our API without a description value</figcaption></figure><h3>Pagination</h3><p>When dealing with large amounts of data, it’s important to paginate. Pagination is useful from a performance perspective in that it decreases the amount of large queries against your data set, but it is also useful from the client’s perspective as they can request only the data they need.</p><p>Pagination is done in our queries by using the concepts of <em>OFFSET </em>and <em>LIMIT. </em>Offset specifies at which record in the list of results we want to start at, and Limit denotes the total number of records to be returned.</p><blockquote>SELECT * FROM table LIMIT 10;</blockquote><p>In this example, we are asking for 10 rows from the beginning of our table.</p><blockquote>SELECT * FROM table OFFSET 10;</blockquote><p>In this example, we ask for all rows from the table starting at record 10.</p><p>Combining these two we can retrieve any subset of records. If we wanted to retrieve 10 records after the first 10 we specify it as such:</p><blockquote>SELECT * FROM table OFFSET 10 LIMIT 10;</blockquote><p><strong>How do we apply this to our API?</strong></p><p>First we need the client to be able to tell us which records they want. This is implemented with <a href="http://en.wikipedia.org/wiki/Query_string">query strings</a>. We can allow the user to specify two parameters: Page and Limit.</p><p>Limit is the number of records to be returned, on which we will set a maximum and default. Page will begin at 1 and increase the offset of records returned.</p><p>Validation should be performed on the query parameters passed, and default values should be supplied (our default limit will be 10, maximum 50):</p><pre>// parseInt attempts to parse the value to an integer<br>// it returns a special &quot;NaN&quot; value when it is Not a Number.<br>var page = parseInt(req.query.page, 10);<br>if (isNaN(page) || page &lt; 1) {<br>  page = 1;<br>}</pre><pre>var limit = parseInt(req.query.limit, 10);<br>if (isNaN(limit)) {<br>  limit = 10;<br>} else if (limit &gt; 50) {<br>  limit = 50;<br>} else if (limit &lt; 1) {<br>  limit = 1;<br>}</pre><p>To apply these settings we first need to perform a query which tells us how many records there are. This sets us up to calculate the total number of pages.</p><p>For our tutorial, we have no <em>WHERE</em> clause so this query is simply counting all of the records in the table, but if you allowed filtering you would need to also specify those criteria here:</p><pre>var sql = &#39;SELECT count(1) FROM photo&#39;;<br>postgres.client.query(sql, function(err, result) {<br>  var count = parseInt(result.rows[0].count, 10);<br>});</pre><p>Because the count returned is a string, we must parse it to an integer in order to perform some basic calculations.</p><p>We now calculate the offset. The page number is reduced by one, and multiplied by the limit per page. This gives us the starting record for our query.</p><pre>var offset = (page - 1) * limit;</pre><p>We have our lower and upper bounds for our final query (<em>offset</em> and <em>limit</em>), let’s build the SQL to return our paginated results:</p><pre>sql = &#39;SELECT * FROM photo OFFSET $1 LIMIT $2&#39;;<br>postgres.client.query(sql, [offset, limit], function(err, result) {<br>  res.json(result.rows);<br>});</pre><p>So now we can start our server and test some requests. If you receive an empty result, simply create some sample data in your database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ajN-TISmjHjzn-9AkpXmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Utlr6uKuG8dq3LLuVuORBw.png" /><figcaption>If you look carefully, our page number has changed and a different result is returned</figcaption></figure><h3>Multi-part forms</h3><p>While not actually a concept of REST APIs, it’s important to understand the difference between types of forms and requests which are pointed at your API.</p><p>So far we have looked at the ability to send data over a POST body in a HTTP request, we have seen how to retrieve data from a URI by using parameters but one thing we haven’t touched on is how might someone send a file to our server.</p><p>Enter multi-part forms.</p><p>A multi-part form is a method of sending data from your browser to a remote server via the POST method in HTTP. To enable multi-part in a particular HTML form we simply specify an <em>enctype</em> (encoding type) attribute. Without this attribute the browser will not attempt to read in data from the local system to send to the server.</p><pre>&lt;form action=”/photos” method=”post”<br><strong>enctype=”multipart/form-data”</strong>&gt;</pre><p>By adding the <em>enctype </em>attribute to our form the browser now knows to look for inputs with the <em>file</em> type, read the contents of the file specified and send it <em>unencoded</em> to the server.</p><p>The <em>unencoded</em> part is key. If we were to submit our form, including file inputs, with no <em>enctype</em> attribute the values in the form are converted to ASCII (using the default <em>application/x-www-form-urlencoded</em> enctype). This will not allow your server to store the file in its original state.</p><p>We can modify our <em>POST / </em>route to receive the file upload. For testing purposes, we’ll include an HTML route in our app so we can try the upload (and would be removed later).</p><pre>var uploadRouter = express.Router();</pre><pre>app.set(‘views’, ‘./views’);<br>app.set(‘view engine’, ‘ejs’);<br>uploadRouter.get(‘/’, function(req, res) {<br>  res.render(‘form’);<br>});</pre><pre>app.use(‘/upload’, uploadRouter);</pre><p>A great module to use for parsing the multipart data is <a href="https://github.com/expressjs/multer">Express’ Multer</a> middleware. You simply supply some options to its middleware and it takes care of all of the parsing and saving of the files.</p><p>To receive/parse the file data we need to modify the <em>photoRouter</em> route for <em>POST /</em> which will allow us to save the file, then store it in the database.</p><p>We simply add the Multer middleware function before our photo validation function in the route definition:</p><pre>photoRouter.post(‘/’, multer(), validatePhoto, function(req, res) {</pre><p><strong>Why isn’t the Multer middleware the same as our validation one?</strong></p><p>There are different ways of approaching middleware, but the basic concept is ultimately the same: A function is returned which accepts a Request and Response object.</p><p>For our validation function we are simply allowing Express to invoke the function with the aforementioned arguments.</p><p>What Multer is doing is accepting an options argument which affects the function that is ultimately called by Express. Here is an example of a closure, using an option to deny a request to a particular URL:</p><pre>function (opts) {<br>  var denyUrl = opts.denyUrl;<br>  return function(req, res, next) {<br>    if (req.url === denyUrl) {<br>      return next(new Error(&#39;Bad URL&#39;));<br>    }</pre><pre>    return next();<br>  }<br>}</pre><p>To give you an idea: Multer is essentially receiving your options and then making assertions on the request object (such as the sizes of files being uploaded) and then continuing to the route handler if all checks pass.</p><p>By default the middleware is taking any number of files, of any size and storing them in its default destination directory: an <em>uploads</em> folder in your project’s root.</p><p>We learned about validation for request data earlier, but how can we apply this to file uploads?</p><p>We definitely don’t want to allow uploading of just any files, and we want to be able to limit the file sizes so it doesn’t fill up our hard drive.</p><p>By looking at the documentation for the Multer module we can see that they accept options by passing an object to the middleware function. The first thing we want to define is our destination directory, using the <em>dest</em> property:</p><pre>photoRouter.post(‘/’, multer({<br>  dest: &#39;./uploads/&#39;<br>}), validatePhoto, function(req, res) {</pre><p>The file that we receive from the client could be named anything, so Multer provides a <em>rename</em> option which accepts a function that receives two arguments: the field name (of the file input), and the original filename. We simply return the filename we ultimately want to store the file under:</p><pre>photoRouter.post(‘/’, multer({<br>  dest: &#39;./uploads/&#39;<br>  rename: function(field, filename) {<br>    filename = filename.replace(/\W+/g, &#39;-&#39;).toLowerCase();<br>    return filename + &#39;_&#39; + Date.now();<br>  }<br>}), validatePhoto, function(req, res) {</pre><p>We are able to remove any non-word characters using a regex, normalize the filename to lowercase, and append the date stamp for when the file was uploaded.</p><p>To prevent clients from eating up our server’s disk space we can specify a limit on both the number of files, and the individual file size:</p><pre>photoRouter.post(‘/’, multer({<br>  dest: &#39;./uploads/&#39;<br>  rename: function(field, filename) {<br>    filename = filename.replace(/\W+/g, &#39;-&#39;).toLowerCase();<br>    return filename + &#39;_&#39; + Date.now();<br>  },<br>  limits: {<br>    files: 1,<br>    fileSize: 2 * 1024 * 1024 // 2mb, in bytes<br>  }<br>}), validatePhoto, function(req, res) {</pre><p>One last modification we want to make is to ensure that the file has not been truncated. The <em>truncated</em> property will be True if the file violated the file size limit. We can check this value, as well as ensure the file field we’re looking for exists in our<em> validatePhoto</em> middleware.</p><pre>function validatePhoto(req, res, next) {<br>  // The file does not exist, something bad happened<br>  if (!req.files.photo) {<br>    res.statusCode = 400;<br>    return res.json({<br>      errors: [‘File failed to upload’]<br>    });<br>  }<br>  // The file was determined to be too large<br>  if (req.files.photo.truncated) {<br>    res.statusCode = 400;<br>    return res.json({<br>      errors: [‘File too large’]<br>    });<br>  }</pre><p>At this point the file is uploaded to our liking, but how to we track the file in our database? Multer appends a <em>files</em> attribute to the Request object, which is an object of the files uploaded (keyed by form field name).</p><p>Knowing our form fields, and that our field name is <em>photo </em>we’re able to modify the database query to now store the real filepath value in the database.</p><pre>var data = [<br>  req.body.description,<br>  req.files.photo.path,<br>  req.body.album_id<br>];<br>postgres.client.query(sql, data, function(err, result) {</pre><p>So let’s restart our server, load up <a href="http://localhost:3000/upload">http://localhost:3000/upload</a> and attempt to upload a file. If you inspect your project files after attempting to upload you should see an <em>uploads</em> directory which contains your files.</p><h3>Integration testing</h3><p>So you’ve written some basic REST API endpoints, and you’re going to keep building on top of them. How do you prevent yourself from accidentally breaking what you already have?</p><p>One way is to completely ignore it and hope for the best: you don’t make mistakes — Right (not recommended)? Another way is to exhaustively test each possible scenario manually when you change something (feasible, but prone to error).</p><p>Integration testing is a way for us to run checks against/through our API without having to actually test it manually. We can make changes to the code, run a set of defined tests and quickly know whether our changes had an effect on other areas of our codebase. The test is the same each time it runs, unlike a manual test which can have slight variations in order of operations and parameters.</p><p>Code testing itself can be daunting, it comes in many forms and there are many tools available to help you with it and that can make it confusing to get started.</p><p>The most popular testing framework for Node.js is <a href="http://mochajs.org/">Mocha</a>.</p><p>Mocha works by using <em>describe</em> statements to outline the functionality you’re testing, and then specify assertions in each of the callbacks to actually test your expected outcomes. Lets look an example of verifying <em>1+1=2.</em></p><p>First we install mocha globally with NPM, this allows us to use the mocha command in the CLI to run our test files:</p><pre>npm install -g mocha</pre><p>Then we create a test file, let’s put ours under the <em>test</em> directory as <em>example.js:</em></p><pre>// We&#39;re testing Math right now<br>describe(&#39;Math&#39;, function() {<br>  // And in Math, we have Addition<br>  describe(&#39;Addition&#39;, function() {<br>    // Then we provide specific test cases<br>    it(&#39;should return 2 when 1 is added to 1&#39;, function() {<br>    });<br>  });<br>});</pre><p>So we’ve now created the structure for a test about Math, specifically Addition, which verifies that when 1 is added to 1 it equals 2.</p><p>To actually verify this is the case, we use a built-in library called <a href="http://nodejs.org/api/assert.html">Assert</a>.</p><p>We require the module at the top of our file, then add assertions to the <em>it</em> blocks (our specific test cases):</p><pre>it(&#39;should return 2 when 1 is added to 1&#39;, function() {<br>  // Typically we would be running part of your application&#39;s code<br>  var result = 1 + 1;<br>  // Equal takes your test case, and the expected outcome<br>  assert.equal(result, 2);<br>});</pre><p>We can now run our tests from the command line using Mocha:</p><pre>&gt; mocha test/example.js</pre><pre>Math<br> Addition<br> ✓ should equal 2 when 1 is added to 1</pre><pre>1 passing (3ms)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*OQgKdfXF3qKeiJAh__mr7Q.png" /><figcaption>Of course, you don’t actually need to test basic Math functionality.</figcaption></figure><p>And we can introduce other sub-sections, or assertions (test cases) to expand our example test file.</p><p>In this example we’ve added a Subtraction section along with an extra test for Addition.</p><p>With integration testing you’re verifying that when pre-determined inputs enter your application, a specific outcome is achieved on the other end.</p><p>In our basic example we were testing code which only existed in the test file itself. Our REST API is an actual server, the code needs to be running for us to test particular outcomes.</p><p>We’re able to leverage a library built specifically for testing Express applications called <a href="https://github.com/tj/supertest">Supertest</a>. This library wraps an Express application, evaluating what happens when certain inputs (requests) are sent to the application.</p><p>We will still leverage Mocha for running our tests, but instead of doing simple math we will have Supertest activate our server and test a particular request.</p><p>Let’s start by outline what we actually want to test in describe statements:</p><pre>var assert = require(‘assert’);<br>var request = require(‘supertest’);</pre><pre>var app = require(‘../index’);</pre><pre>describe(‘Tutorial REST API’, function() {<br>  describe(‘Create photo’, function() {<br>    it(&#39;returns the created resource on success&#39;, function(done) {<br>    });<br>    it(&#39;returns 400, with error message on bad request&#39;, function(done) {<br>    });<br>  });<br>});</pre><p>The goal is to cover all of the potential outcomes, the more coverage you have the more robust/resilient your API will be (less prone to failure).</p><p>You’ll notice that the callback for our it statements now has an argument called<em> done. Done </em>is the callback function used by Mocha to know when you’ve completed running your test subject’s code. In this case, when Supertest has finished running its requests.</p><p>Now we can specify the Supertest itself:</p><pre>it(‘returns the created resource on success’, function(done) {</pre><pre>  var validPhotoResource = {<br>    description: ‘Photo created on ‘ + Date.now(),<br>    filepath: ‘/path/to/photo.jpg’,<br>    album_id: 1<br>  };</pre><pre>  request(app)<br>    .post(‘/photo’)<br>    .field(&#39;description&#39;, &#39;My photo description&#39;)<br>    .field(&#39;album_id&#39;, 1) <br>    .attach(&#39;photo&#39;, __dirname + &#39;/abe.jpg&#39;)<br>    .expect(201)<br>    .end(function(err, res) {<br>      if (err) {<br>        return done(err);<br>      }</pre><pre>      assert.equal(res.body.description, validPhotoResource.description);<br>      assert.equal(res.body.album_id, validPhotoResource.album_id);<br>      done();<br>    });<br>});</pre><p>So what is happening? First we are specifying what we believe is a valid POST body for creating a Photo resource object.</p><p>We then pass our <em>app</em> object to Supertest (named request). This allows Supertest to actually run our app code when we run our tests. Supertest then provides a number of chainable functions to both specify what the request should be, and what we expect the response to be.</p><p>Our first example shows us making a POST request to /photo with a <em>description</em> and an <em>album_id</em> property in the POST body.</p><pre>request(app)<br>    .post(‘/photo’)<br>    .field(&#39;description&#39;, &#39;My photo description&#39;)<br>    .field(&#39;album_id&#39;, 1)</pre><p>Our API expects a multi-part file to be uploaded along with the description and the album id fields. How do we simulate uploading a file?</p><p>Supertest has an <em>attach</em> method which allows us to simulate someone attaching a file to their form upload.</p><pre>request(app)<br>    .post(‘/photo’)<br>    .field(&#39;description&#39;, &#39;My photo description&#39;)<br>    .field(&#39;album_id&#39;, 1) <br><strong>    .attach(&#39;photo&#39;, __dirname + &#39;/abe.jpg&#39;)</strong></pre><p>The file has to exist inside of our test directory in order for the test to pass, since it needs a test subject to attempt to upload.</p><p>We then expect that the response will include all of our values supplied, an id property, and the HTTP status code 201. This signifies that the resource was created properly.</p><pre>.expect(201)<br>.end(function(err, res) {<br>  if (err) {<br>    return done(err);<br>  }</pre><pre>  assert.ok(res.body.id);<br>  assert.equal(res.body.description, validPhotoResource.description);<br>  assert.equal(res.body.album_id, validPhotoResource.album_id);<br>  done();<br>});</pre><p>If we go to the command line and run it we find that our test fails. We get a really long stacktrace and some red text. Why is that?</p><p>The problem is that when Supertest starts up our server it’s unable to connect to the database, because the database itself is initialized using the <em>bin/www</em> script in our repository.</p><p>We can solve this a few different ways, but the easiest and most simple for a beginner is to create a <em>test</em> database, and initialize the Postgres client before the tests run.</p><p>Mocha provides a number of helpful interfaces for setting up the environment before the tests actually run. One of these interfaces is <em>before</em>. Any code specified in a <em>before </em>block runs, you guessed it, before the tests in that context.</p><p>All we have to do is import our Postgres client code from <em>lib/postgres.js</em> and initialize it before the tests run:</p><pre>var pg = require(‘../lib/postgres’);</pre><pre>var DATABASE_URL = ‘postgres://username:password@localhost/test’</pre><pre>describe(‘Tutorial REST API’, function() {<br>  before(function(done) {<br>    pg.initialize(DATABASE_URL, done);<br>  });</pre><p>This ensures that there is a database connection available for the test cases to run against. (You can create your test database by just copying the existing api sample database).</p><p>Now when we run our tests we should see something like this:</p><pre>&gt; mocha -R spec test/api.js</pre><pre>Tutorial REST API<br> Create photo<br> ✓ returns the created resource on success</pre><pre>1 passing (35ms)</pre><p>We can develop further with confidence knowing that we will be notified of any breaking changes via running our tests.</p><h4>Where to go from here?</h4><p>Modern web APIs offer much more than we’ve accomplished so far. What are some things we’re missing that would be good to add?</p><ul><li>Authentication, <a href="https://www.digitalocean.com/community/tutorials/an-introduction-to-oauth-2">OAuth</a> or <a href="http://en.wikipedia.org/wiki/Basic_access_authentication">Basic Auth</a></li><li>Security features</li></ul><p>To view the code for our API thus far visit the <strong>part-two </strong>branch in the sample app’s repository: <a href="https://github.com/jeffandersen/node-express-sample/tree/part-two">https://github.com/jeffandersen/node-express-sample/tree/part-two</a></p><p>If you have any questions or comments don’t hesitate to contact me directly via Twitter <a href="https://twitter.com/jeffandersen">@jeffandersen</a>, or simply comment on this post.</p><p><em>This tutorial has been built specifically for purpose of exploring creation of a Node.js REST API with UIT Startup immersion students. Check out UIT at </em><a href="http://uitstartup.org/"><em>http://uitstartup.org</em></a><em> or on Twitter </em><a href="https://twitter.com/uitstartup"><em>@UITStartup</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9152661bf47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Node.js REST API with Express]]></title>
            <link>https://medium.com/@jeffandersen/building-a-node-js-rest-api-with-express-46b0901f29b6?source=rss-f9690c1e22b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/46b0901f29b6</guid>
            <dc:creator><![CDATA[Jeff Andersen]]></dc:creator>
            <pubDate>Fri, 30 Jan 2015 02:46:24 GMT</pubDate>
            <atom:updated>2015-02-10T16:58:56.578Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*k0u7bbZal3fIbVbRzDjhCA.jpeg" /><figcaption>I don’t recommend writing on your screen with a sharpie</figcaption></figure><h4>What is a REST API anyway?</h4><p>The de-facto way to construct a web API is to follow the <a href="http://en.wikipedia.org/wiki/REST">RESTful</a> approach (referred to as a REST API).</p><p>REST APIs are resource-based interfaces. On the web this means that data resources (typically formatted in JSON, or XML) are represented by URIs (paths) accessed via <a href="http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol">HTTP</a>.</p><p>Actions such as Create, Read, Update, and Delete (commonly referred to as <a href="http://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a>) are made against these resources using HTTP methods (otherwise referred to as verbs): POST, GET, PUT/PATCH, and DELETE.</p><p>Each request made to a REST API is stateless. This means that the server handling your request maintains no context between requests: each request is interpreted equally. Any context required to process the request must be provided with the request itself (for instance, an authorization token).</p><p>Good REST APIs maintain a uniform interface. Meaning that the same shape of request which is made against a particular resource can reasonably be expected to act the same way against another resource (for instance, the request modifying an attribute on a Photo resource is uniform to that of modifying a similar attribute on an Album resource).</p><p>Another characteristic of a REST API is that server responses are cacheable. This means that based on context returned with the request’s response by the server you can cache the item. The server may return explicit instructions for how long a resource may be cached for (perhaps it instructs not to cache, or provides a recommended length of time to cache before refreshing).</p><h2>Step by step tutorial</h2><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F118976690&amp;url=http%3A%2F%2Fvimeo.com%2F118976690&amp;image=http%3A%2F%2Fi.vimeocdn.com%2Fvideo%2Fdefault_640&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="640" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/b11b3beb97f923e1019c38be4cea8765/href">https://medium.com/media/b11b3beb97f923e1019c38be4cea8765/href</a></iframe><h3><strong>Setup</strong></h3><p>Lets start off by first cloning the sample repository to your local environment. You can find the base application at <a href="https://github.com/jeffandersen/node-express-sample">https://github.com/jeffandersen/node-express-sample</a></p><p>Our sample resources will be in a Postgres database. A sample database file named <strong>database.sql</strong> can be found in the project root<em>. </em>Simply import the SQL file into a new database, we’ll call ours <strong>api</strong>. You will need to edit /bin/www with your database connection details. Check out the video above for a walkthrough of this process.</p><p>Let’s try running our application for the first time by calling the command below from the project’s root.</p><pre>./bin/www</pre><p>If your terminal’s output matches the below, try loading your application in your web browser.</p><pre>&gt; ./bin/www<br>Example app listening at <a href="http://0.0.0.0:3000/">http://0.0.0.0:3000</a></pre><p>You should receive a message stating <em>Cannot GET /</em></p><p>We’re ready to start writing our API.</p><h3>Routing</h3><p>Express.js is great for constructing a REST API because it provides an easy interface to segregate your resources by both type and action.</p><p>If you’ve previously built a web application with Express, you’ll find that the steps for creating an API using the same framework are very similar.</p><p>Let’s open <em>index.js</em> in your code editor.</p><p>This file is as bare bones as you can get when defining an Express application. We have created an application object and exported it for use.</p><pre>var express = require(‘express’);<br>var app = express();</pre><pre>module.exports = app;</pre><p>The concept of Routes is used in Express for defining application behaviour to run when a particular request is received.</p><p>A route in Express is made up of three main parts:</p><ul><li>The HTTP method associated the request we want to capture.</li></ul><pre>app.<strong>get()</strong></pre><ul><li>The URI (or path) of the request we want to capture.</li></ul><pre>app.get<strong>(&#39;/&#39;)</strong>;</pre><ul><li>A handler function (which accepts a request and response object as args)</li></ul><pre>app.get(&#39;/&#39;<strong>, function(req, res) { }</strong>);</pre><p>This is an example of handling a GET request on <em>/</em> (in other words: what happens when someone loads your app’s index in their browser):</p><pre>var express = require(‘express’);<br>var app = express();</pre><pre>// Our handler function is passed a request and response object<br>app.get(&#39;/&#39;, function(req, res) {<br>  // We must end the request when we are done handling it<br>  res.end();<br>});</pre><pre>module.exports = app;</pre><p>After restarting our application, when we load <a href="http://localhost:3000/">http://localhost:3000</a> we receive a blank page instead of an error message.</p><p>For this tutorial we’ll be interacting with a database containing photos, and albums where an album can have many photos.</p><p>We first think about what functionality we want to provide our end-users. Should we wish to allow them to have full CRUD access we need to create routes for each of these actions for each of the resources (photos, albums).</p><h4><strong>Which HTTP method should we use?</strong></h4><p>When constructing a REST API each HTTP method corresponds to an action against a resource served by the API.</p><ul><li><strong>GET</strong> — retrieve a particular resource’s object or list all objects</li><li><strong>POST</strong> — create a new resource’s object</li><li><strong>PATCH</strong> — make a partial update to a particular resource’s object</li><li><strong>PUT</strong> — completely overwrite a particular resource’s object</li><li><strong>DELETE</strong> — remove a particular resource’s object</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F118976643&amp;url=http%3A%2F%2Fvimeo.com%2F118976643&amp;image=http%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F506137989_960.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="1152" height="720" frameborder="0" scrolling="no"><a href="https://medium.com/media/25c7c405f8deb404ed1776bf7f33f34d/href">https://medium.com/media/25c7c405f8deb404ed1776bf7f33f34d/href</a></iframe><p>We can start laying out our API by specifying the routes for a single resource. This draws from the uniform interface characteristic of REST APIs. We are able to define the routes for one resource, then essentially copy and paste them for additional resources.</p><p>To specify our routes we’ll use an <a href="http://expressjs.com/guide/routing.html">Express Router</a> for each resource. We create the router object, specify the routes we wish to reply to, and then attach the router for a particular path.</p><pre>var express = require(‘express’);<br>var app = express();</pre><pre>// Create the express router object for Photos<br>var photoRouter = express.Router();</pre><pre>// A GET to the root of a resource returns a list of that resource<br>photoRouter.get(‘/’, function(req, res) { });</pre><pre>// A POST to the root of a resource should create a new object<br>photoRouter.post(‘/’, function(req, res) { });</pre><pre>// We specify a param in our path for the GET of a specific object<br>photoRouter.get(‘/:id’, function(req, res) { });</pre><pre>// Similar to the GET on an object, to update it we can PATCH<br>photoRouter.patch(‘/:id’, function(req, res) { });</pre><pre>// Delete a specific object<br>photoRouter.delete(&#39;/:id&#39;, function(req, res) { });</pre><pre>// Attach the routers for their respective paths<br>app.use(‘/photo’, photoRouter);</pre><pre>module.exports = app;</pre><p>So now our API is ready to perform actions when we receive any of the following HTTP requests:</p><ul><li><strong>GET</strong> /photo — Retrieve all photos</li><li><strong>GET</strong> /photo/:id — Retrieve a photo by its ID</li><li><strong>POST</strong> /photo — Create a new photo</li><li><strong>PATCH</strong> /photo/:id — Update properties of a photo by its ID</li><li><strong>DELETE</strong> /photo/:id — Delete a photo by its ID</li></ul><p>This process can be repeated for the <em>album </em>resource.</p><p>If you restart your application and try to access /photo or /photo/123 you’ll find that the request hangs and your browser eventually times out. This is because the handler functions we specified do not end the request yet.</p><p>Your<em> index.js</em> file should now resemble the following code:</p><pre>var express = require(‘express’);<br>var app = express();</pre><pre>var photoRouter = express.Router();<br>photoRouter.get(‘/’, function(req, res) { });<br>photoRouter.post(‘/’, function(req, res) { });<br>photoRouter.get(‘/:id’, function(req, res) { });<br>photoRouter.patch(‘/:id’, function(req, res) { });<br>photoRouter.delete(‘/:id’, function(req, res) { });<br>app.use(‘/photo’, photoRouter);</pre><pre>var albumRouter = express.Router();<br>albumRouter.get(‘/’, function(req, res) { });<br>albumRouter.post(‘/’, function(req, res) { });<br>albumRouter.get(‘/:id’, function(req, res) { });<br>albumRouter.patch(‘/:id’, function(req, res) { });<br>albumRouter.delete(‘/:id’, function(req, res) { });<br>app.use(‘/album’, albumRouter);</pre><pre>module.exports = app;</pre><h4>Resource Lookup</h4><p>When we receive a request through a particular route we want to access that resource’s information from the database. This is so that we can return the data to the client performing the request.</p><p>Our code is getting verbose now and we want to reduce code duplication. This is where <a href="http://expressjs.com/guide/using-middleware.html">Express Middleware</a> comes into play. Middleware are functions which perform specific actions based on request information, but can be re-used throughout your routers. These functions get passed to the route after the path, and before the handler function, as such:</p><pre>req.get(‘/photo/:id’, <strong>lookupPhoto</strong>, function(req, res) { });</pre><p>In our case, an action we are going to be repeating quite frequently is looking up a particular resource object by its ID. So we can create a specific lookup function that is used on multiple routes.</p><p>The middleware function looks just like a route handler, but accepts a third argument (typically named <em>next</em>) which is the callback function to proceed to the route’s handler.</p><pre>function lookupPhoto(req, res, <strong>next</strong>) {<br>  // We access the ID param on the request object<br><strong>  var photoId = req.params.id;</strong></pre><pre>  // Build an SQL query to select the resource object by ID<br>  var sql = ‘SELECT * FROM photo WHERE id = ?’;<br>  postgres.client.query(sql, [ photoId ], function(err, results) {<br>    if (err) {<br>      console.error(err);<br><strong>      res.statusCode = 500;<br>      return res.json({ errors: [‘Could not retrieve photo’] });</strong><br>    }</pre><pre>    // No results returned mean the object is not found<br>    if (results.rows.length === 0) {<br>      // We are able to set the HTTP status code on the res object<br><strong>      res.statusCode = 404;<br>      return res.json({ errors: [‘Photo not found’] });</strong><br>    }</pre><pre>    // By attaching a Photo property to the request<br>    // Its data is now made available in our handler function<br><strong>    req.photo = results.rows[0];</strong><br>    <strong>next();</strong><br>  });<br>}</pre><p><em>Disclaimer: You want to introduce parameter/input validation before ever publishing such an API to the web.</em></p><p>Here’s a look at the lookup middleware function applied to our photo routes:</p><pre>var photoRouter = express.Router();<br>photoRouter.get(‘/’, function(req, res) { });<br>photoRouter.post(‘/’, function(req, res) { });<br>photoRouter.get(‘/:id’, lookupPhoto, function(req, res) { });<br>photoRouter.patch(‘/:id’, lookupPhoto, function(req, res) { });<br>photoRouter.delete(‘/:id’, lookupPhoto, function(req, res) { });<br>app.use(‘/photo’, photoRouter);</pre><p>Now each time a request comes through for any of those routes, the resource object will be retrieved from the database by its ID and passed via the request object to our handler function.</p><p>The same process can be duplicated for the other resources, and also for the handler functions of the <em>GET / </em>routes.</p><h4>Responding to requests</h4><p>When we try to list our photos by opening <a href="http://localhost:3000/book">http://localhost:3000/p</a>hoto in our browser, it still just sits there and doesn’t finish loading.</p><p>This is because our handler functions do not respond to the request yet. They’re empty.</p><p>When building a RESTful API there are a few concepts we want to be mindful of when responding with data:</p><ul><li>HTTP status codes should describe the response</li><li>Data returned should be formatted in JSON (preferred) or XML</li></ul><h4>HTTP status codes</h4><p>There are a multitude of HTTP status codes, here is just a few of the pertinent ones (which we will be using):</p><ul><li><strong>200 </strong>— OK, The request was successful</li><li><strong>201</strong> — CREATED, A new resource object was successfully created</li><li><strong>404 — </strong>NOT FOUND, The requested resource could not be found</li><li><strong>400 </strong>—BAD REQUEST, The request was malformed or invalid</li><li><strong>500 — </strong>INTERNAL SERVER ERROR, Unknown server error has occurred</li></ul><p>To return data with Express we can use the Response object’s <em>json</em> method. This method accepts a JavaScript object and will automatically convert it to JSON and end the request for us.</p><p>In the case of our <em>GET /photo/:id</em> route there is no additional processing we need to do in our handler function and we can simply return the data retrieved by our middleware:</p><pre>photoRouter.get(‘/:id’, lookupPhoto<strong>, </strong>function(req, res) {<br>  <strong>res.json(req.photo);</strong><br>})</pre><p>In the case of our <em>DELETE /photo/:id</em> route we need to process the removal of the record in our handler function (the middleware verifies that the resource object exists before we continue).</p><h4>How do we access the POST body?</h4><p>We want our clients to be able to create records in our database for resources. To do this we need them to supply us with data for that resource object.</p><p>The POST body is an object of data attached to the incoming request. Express requires additional middleware to be able to process these incoming values. Enter <em>body-parser.</em></p><p>The body-parser middleware has already been included in the sample project’s package. We need to import and attach the middleware to our Express application.</p><pre>var express = require(&#39;express&#39;);<br><strong>var bodyParser = require(&#39;body-parser&#39;);</strong></pre><pre>var app = express();<br><strong>app.use(bodyParser.json({ type: &#39;application/json&#39; }));</strong></pre><p>This enables our Express application to parse incoming JSON post bodies.</p><p>We have already set up a route at <em>POST /photo </em>for creation of a photo resource object, now we need to write the handler code.</p><p>The <em>body-parser</em> middleware that we have loaded automatically adds the <em>body</em> key to the Request object. This means we can read any property sent to use by simply doing:</p><pre>req.body.PROPERTY_NAME // e.g req.body.description</pre><p><em>Disclaimer: You always want to validate incoming data before trusting it. For sake of simplicity in the tutorial we are omitting this step</em></p><p>We can now simply write an insert query to run when a request is matched to our photo create route. In this case we return status code 201 (CREATED) and the resulting resource object (which should be identical to the GET of the same resource).</p><pre>photoRouter.post(&#39;/&#39;, function(req, res) {<br>  var sql = &#39;INSERT INTO photo (description, filepath, album_id) VALUES ($1,$2,$3) RETURNING id&#39;;<br>  // Retrieve the data to insert from the POST body<br>  var data = [<br>    req.body.description,<br>    req.body.filepath,<br>    req.body.album_id<br>  ];<br>  postgres.client.query(sql, data, function(err, result) {<br>    if (err) {<br>      // We shield our clients from internal errors, but log them<br>      console.error(err);<br><strong>      res.statusCode = 500;<br>      return res.json({<br>        errors: [&#39;Failed to create photo&#39;]<br>      });</strong><br>    }</pre><pre>    var newPhotoId = result.rows[0].id;<br>    var sql = &#39;SELECT * FROM photo WHERE id = $1&#39;;<br>    postgres.client.query(sql, [ newPhotoId ], function(err, result) {<br>      if (err) {<br>        // We shield our clients from internal errors, but log them<br>        console.error(err);<br><strong>        res.statusCode = 500;<br>        return res.json({<br>          errors: [&#39;Could not retrieve photo after create&#39;]<br>        });</strong><br>      }</pre><pre>      // The request created a new resource object<br><strong>      res.statusCode = 201;</strong></pre><pre>      // The result of CREATE should be the same as GET<br><strong>      res.json(result.rows[0]);</strong><br>  });<br>});</pre><p>The concepts we have applied for the POST route can also be applied to the PATCH route with basic changes to the SQL query.</p><p>Now, how do we test this?</p><p>You could us cURL and manually construct the HTTP request from scratch via the Terminal but that’s complicated for beginners. A cool tool for testing API requests is <a href="http://insomnia.rest/">Insomnia REST Client</a>. It’s a Google Chrome app, so you can run it from your Chrome browser.</p><h4>Testing requests with Insomnia.rest</h4><p>Once you’ve installed the Chrome app, launch it. You’ll have a screen that looks like this screen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*SL-eyr1y7s4lcRbtg7ls0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*5vrcVPtQE4zc22ZzR-S1Zg.png" /><figcaption>Creating a new request with Insomnia.rest API development tool</figcaption></figure><p>We simply enter a name for our first request, which I’ve named Create Photo, and click the Create button.</p><p>You’ll now be dropped into the request creation screen. It has a dropdown for the HTTP method/verb, the address and two big boxes. The left box is for your potential request body, the right box is to display the result of the request. To run the request you simply press the arrow on the right of the address bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2q7rxQko49BeK_eQLhB7A.png" /><figcaption>Create photo request for our tutorial API using Insomnia.rest</figcaption></figure><p>This is a great debugging tool for creating requests arbitrarily for different APIs (you can save them too, for future use).</p><h4>What about errors?</h4><p>Errors are just other forms of responses. These can be blank or data responses (depending on error type or API design).</p><p>Typical practice for JSON REST APIs is to return an <em>errors </em>array in the body of the response, for example if we were responding to a bad request to create a new photo we might do something like this:</p><pre>res.statusCode = 401;<br>return res.json({<br>  errors: [<br>    &#39;Missing description property&#39;,<br>    &#39;Missing album_id property&#39;<br>  ]<br>});</pre><p>The client will receive the 401 status code and the following JSON:</p><pre>{&quot;errors&quot;:[&quot;Missing description property&quot;,&quot;Missing album_id property&quot;]}</pre><p>This allows the client to predict how an error message manifests itself (by checking for the <em>errors </em>key in the response, and facilitates returning more than one error message at once.</p><p>We saw a very basic error response in our middleware example when the query cannot find the particular row by ID.</p><h4>You now have the basics</h4><p>So there we have it. We’ve created the basic functions of a REST API:</p><ul><li>Created routes</li><li>Looking up resources in the database based on parameters</li><li>Create resources in the database base on request body</li></ul><p>To view the code for our API thus far visit the <strong>part-one</strong> branch in the sample app’s repository: <a href="https://github.com/jeffandersen/node-express-sample/tree/part-one">https://github.com/jeffandersen/node-express-sample/tree/part-one</a></p><h4>What’s next?</h4><ul><li>Validation</li><li>Multipart post bodies</li><li>Pagination of results</li><li>Integration testing</li></ul><p>Ready for some more? <a href="https://medium.com/@jeffandersen/building-a-node-js-rest-api-with-express-part-two-9152661bf47">Check out Part 2</a></p><p>If you have any questions or comments don’t hesitate to contact me directly via Twitter <a href="https://twitter.com/jeffandersen">@jeffandersen</a>, or simply comment on this post.</p><p><em>This tutorial has been built specifically for purpose of exploring creation of a Node.js REST API with UIT Startup immersion students. Check out UIT at </em><a href="http://uitstartup.org/"><em>http://uitstartup.org</em></a><em> or on Twitter </em><a href="https://twitter.com/uitstartup"><em>@UITStartup</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46b0901f29b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>