<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Levvel on Medium]]></title>
        <description><![CDATA[Stories by Levvel on Medium]]></description>
        <link>https://medium.com/@GetLevvel?source=rss-fb16d70c66b9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*p-jT7RdRAzg4bsp71M5Ftg.png</url>
            <title>Stories by Levvel on Medium</title>
            <link>https://medium.com/@GetLevvel?source=rss-fb16d70c66b9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:39:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@GetLevvel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Three Approaches to Integrating Your Bank with Zelle]]></title>
            <link>https://medium.com/levvel-consulting/three-approaches-to-integrating-your-bank-with-zelle-1ce7cb755275?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ce7cb755275</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[zelle]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 25 Aug 2017 17:55:58 GMT</pubDate>
            <atom:updated>2017-08-25T17:55:58.499Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*UQNX9cANduWojnkUmsyW-w.jpeg" /></figure><p>Over the last month, the Levvel team has been writing about our experience helping banks onboard to the Zelle network. We have walked through what Zelle is and why it matters, dug deeper into how different banks are looking at Zelle as table stakes or a strategic differentiator, and now we’d like to discuss what we’ve found to be the three general approaches for banks to integrate with Zelle.</p><h3>Approach 1: Direct Integration with Zelle</h3><p>For most people, the roll-out of the Zelle brand is the first time they have heard of the functionality that has been quietly facilitating the peer-to-peer (P2P) payments offering at many major US banks for years. Previously operated as clearXchange, Zelle was built by Wells Fargo, Bank of America, JP Morgan Chase, and later expanded to include Capital One and US Bank, among others. For these banks that participated in the clearXchange launch or joined shortly thereafter, there was minimal additional work required to launch with Zelle. Most of these banks choose to continue with their direct integration approach. We feel that the most complex and ultimately strategic approach is for a bank to work directly with EWS to integrate into the Zelle directory and network, much like the large banks have already done.</p><p>Despite this, the decision to do so is more complex for middle-market banks. For many of these banks, technology dollars are tight, and the perceived lack of business case for P2P payments creates the need for additional scrutiny over the scope of a Zelle project. In our <a href="http://www.levvel.io/blog-post/Zelle-Table-Stakes-or-Strategic-Differentiator">last article</a>, we discussed several ways to build a business case around Zelle and how to position it as a strategic differentiator. This included future use cases like disbursements, bill pay, and broader money movement strategies. For a bank to be best positioned to take advantage of the newest features and uses cases made available by Zelle, they will need to work directly with the platform and often need a direct integration.</p><p>This may not be completely obvious, given that the sales materials from leading resellers promise the integration of all desired future functionality. However, we often find that some of what is promised has yet to be built. In contrast, banks that integrate directly with Zelle will have access to the features as they become available and can then begin building business models, products, and workflows utilizing the new features as soon as the specifications are released. We find this particularly useful for banks that have uses cases stretching across both the retail consumer and the commercial sides of the bank (since most vendor products do not stretch to both sides).</p><p>There are some downsides to this approach, including a higher initial cost of integration, time to market, the need for skilled internal labor, and potentially needing to hire an implementation partner. Ultimately, the direct approach seems best suited for banks looking to leverage the Zelle directory and network to build a market-differentiated approach to payments across different payment rails. Additionally, banks looking to reduce their platform vendor footprint in core business functions by bringing key functionality in-house will often find direct integration to be an optimal path.</p><h3>Approach 2: Leverage a Turnkey Solution From a Partner (Co-op, FIs, Fiserv, etc.)</h3><p>The above scenario works well for mid-sized to large banks with sufficient IT budget and business focus on the future of money movement capabilities. But what about the small regional banks and credit unions? We have found that many of those financial institutions take an approach on the opposite end of the integration spectrum by leveraging a “reseller,” such as Co-op, Fiserv, FIS, or Jack Henry. For the smaller banks, these partners often already operate the majority of the bank systems that serve as integration points for Zelle, including the mobile/online experience, core deposit system, ACH system, and bank messaging platform. Not only do these banks tend to lack the internal development resources needed to perform a Zelle integration, they also rely on vendors so heavily that building and operating a custom piece of software is often not an option.</p><p>One additional segment of banks we see leveraging resellers consists of those that were previously using Fiserv’s Popmoney, TransferNOW, or BillPay products for other money movement activities. For these banks, Fiserv presents an option that encapsulates most of the required Zelle integration points into already implemented interfaces the bank has with Fiserv. We have also seen this path become appealing to larger banks that have prioritized other efforts such as mobile wallets or API gateways and wish to be part of the Zelle movement, but with minimal cost.</p><p>While leveraging a provider is a perfectly acceptable solution for a wide range of banks, we advise that banks step into these partnerships with a clear understanding of what they will get from the vendor on day one and what is a future-state or road map feature. In some cases, the first bank to hire the vendor for this functionality will be the one working to build it, and only then will it become available to all other banks on the platform. This inherently means the bank will not be able to offer a differentiated product. This is nothing new, and most banks have come to accept that the large vendors provide features only after they are battle tested with other customers, but it does mean that it will be hard to differentiate amongst your competitors.</p><p>Additionally, leveraging “turnkey” solutions also means that a bank’s ability to leverage that integration across multiple lines of business or products is severely limited. This is often the result of tightly coupled experiences and services that can’t be customized or altered to facilitate incorporation of other vendors or homegrown products. Lastly, this type of integration can be somewhat challenging for banks leveraging multiple vendors for key components of the Zelle integration. As an example, a bank using a Hogan core with a FIS online banking interface and a Fiserv Pep+ ACH system is going to have a number of complex integration challenges hooking all of these products together and will likely need outside guidance.</p><h3>Approach 3: Combination of Custom Technology with Partner Integration</h3><p>So far, we have discussed the two opposite ends of a Zelle integration approach:building the integration in-house or utilizing a turnkey partner. For some banks, a direct-to-Zelle approach is too large of an effort while using a turnkey partner limits them in their desire for flexibility and how the product is implemented. For these banks, it is often best to approach Zelle with a hybrid model, choosing to leverage a vendor such as Icon, Co-Op, or FIS for key pieces of functionality while building other core components themselves. This approach is often ideal when the bank is also targeting the implementation of a new vendor product such as a payments hub, ACH system, or Bill Pay platform. In these cases, selecting a vendor that already has Zelle interfaces available as part of the core product offering means the bank is able to accomplish two objectives with one purchasing decision.</p><p>Additionally, this approach often means the bank is leveraging the APIs from their vendors and allows for a more loosely coupled integration of user experience and back-end functionality. As a result, the bank has more flexibility as new features become available to alter the experience or incorporate a new money movement rail or feature.</p><p>An additional area where this approach has become prevalent revolves around banks that are looking forward to the TCH Real Time Payments (RTP) network and are planning accordingly. For many of these banks, an API gateway such as Red Hat’s 3scale or Google’s Apigee platform, a payment hub such as D+H’s, or a payment framework such as Icon’s IPF provide a nice toolset for building future payment products, including Zelle and TCH RTP, while also allowing for bank-specific workflow customization and control. Most of these partners provide some Zelle interfaces out of the box, but they also integrate into customized bank logic and flows, ultimately providing the best of both worlds.</p><h3>Picking the Best Approach</h3><p>The approach that works for any given bank depends on a number of factors, including budgets, internal IT capabilities, and current vendor relationships. While many larger banks will continue down their traditional path of either building internally or outsourcing, the real opportunity lies with the mid-market banks who have a chance to start punching above their weight by building the right features that create value and resonate with their customers.</p><p>Over the next several years, the combination of Zelle and TCH RTP will allow banks of all sizes to create new business cases and market opportunities if they form the right partnerships and build the right technology. Levvel is actively working with a number of US banks to help inform these decisions and ultimately design and build the integrations needed to ensure success with these products.</p><p>Interested in learning more? Contact us at <a href="/service/mailto:payments@levvel.io">payments@levvel.io</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ce7cb755275" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/three-approaches-to-integrating-your-bank-with-zelle-1ce7cb755275">Three Approaches to Integrating Your Bank with Zelle</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Insights on Zelle from the Levvel Payments team]]></title>
            <link>https://medium.com/levvel-consulting/insights-on-zelle-from-the-levvel-payments-team-35b497d701f5?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/35b497d701f5</guid>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[rtp]]></category>
            <category><![CDATA[zelle]]></category>
            <category><![CDATA[p2p]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Tue, 22 Aug 2017 18:44:43 GMT</pubDate>
            <atom:updated>2017-08-22T18:44:43.816Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*m_pKjPUq3f9c15Eg3WNe7g.png" /></figure><p>Zelle, a new peer-to-peer (P2P) payments service backed by over 30 major U.S. banks, is generating a lot of buzz as of late. But what exactly is Zelle, and how is it any different from Venmo and other P2P payment services?</p><p>As a veteran of the financial services and payments industries, I recently <a href="https://letstalkpayments.com/what-is-zelle-why-does-it-matter/">wrote an article</a>for the Let’s Talk Payments website to answer these and other common questions on Zelle, such as:</p><ul><li>Is this the end of Venmo and Square Cash?</li><li>What technology is powering Zelle’s engine?</li><li>What is Zelle’s impact on P2P and real-time payments?</li><li>How does Zelle affect my business? Why should I consider (or reconsider) it?</li><li>How might Zelle be used in the future?</li></ul><p><a href="https://letstalkpayments.com/what-is-zelle-why-does-it-matter/">Read the full article</a> on the Let’s Talk Payments website to learn more about how Zelle is disrupting the payments industry and the effects it will have on your business and your relationship with your customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35b497d701f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/insights-on-zelle-from-the-levvel-payments-team-35b497d701f5">Insights on Zelle from the Levvel Payments team</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Challenges and Opportunities in B2B ePayment Adoption]]></title>
            <link>https://medium.com/levvel-consulting/challenges-and-opportunities-in-b2b-epayment-adoption-2b676d61e8f4?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b676d61e8f4</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[payments]]></category>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[b2c]]></category>
            <category><![CDATA[e-payment]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Mon, 03 Jul 2017 14:19:11 GMT</pubDate>
            <atom:updated>2017-07-03T14:19:11.234Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*CSwnKJ0xnfvUc6a_YBYiqg.png" /></figure><p>It’s an exciting time in the B2C (business to consumer) payments space as adoption of <a href="http://www.levvel.io/blog-post/top-payments-trends-in-2017">emerging capabilities</a> grows, bolstered by success stories and lots of interest from developers. But it’s a much different story in the B2B (business to business) payments space. According to the<a href="https://www.paystreamadvisors.com/resource/2017-electronic-payments-report/"> 2017 Electronic Payments Report</a> by PayStream Advisors, many organizations continue to rely on check and ACH. This is true despite the growing availability of electronic payments (ePayments) solutions designed especially for the B2B environment.</p><p>There are a few fairly obvious factors that contribute to the lag in ePayments adoption. B2B buyers, especially in large, widespread organizations, must adhere to rigid approval hierarchies and sometimes extremely complex regulations. They also tend to favor a net-30 payment term and the associated float time. This makes implementation of ePayments tools more difficult to implement and scale than in a typical B2C environment.</p><p>There are additional nuances that explain how payment methods are selected and implemented in the B2B space. All payment methods come with their own unique characteristics and processing challenges. These can be training-related challenges, like a lack of understanding about the correct method to use. They can also involve hard costs when they lead to missed discounts or duplicate payments.</p><p>As the basis of the 2017 ePayments report, PayStream surveyed hundreds of B2B professionals across several industries about their attitudes and usage of payments tools. PayStream’s research found that payments challenges are more common for heavy check users, or “strong proponents” of checks. Strong proponents of check payments are much more likely to report hard losses like missed discounts and late payments than strong proponents of other payments types (see Figure 1 below). They were also more likely to experience payment fraud. Among respondents with no payment challenges, none were strong proponents of checks.</p><p>Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*s_X9w5GmC7q3Jct3K8XN2A.png" /></figure><p>Of course, payment challenges vary depending on industry and market segment. For example, in a highly regulated environment like healthcare, payment costs tend to be higher. This isn’t surprising, since complex process flows and approval hierarchies are common in healthcare payables. The result is overall longer payment cycles that demand extra time and resources.</p><p>PayStream also found that when ePayments tools are introduced to a company’s payments management process, both costs and headaches are reduced. Organizations that adopt commercial cards reported improvements in convenience, costs, and rebate capture, see Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2rv4HSmNwOl1r2Df_4Gfsg.png" /></figure><p>ePayments tools help speed up cycle times and boost security and compliance. Organizations that use them are better able to capture early payment discounts and achieve card-based rebates. Speeding up payment cycle times also improves relationships with suppliers — when suppliers are paid faster, less time and energy goes into responding to calls and emails about payment delays. The secure data conduit provided by these tools can be especially helpful for B2B organizations with high volumes of cross-border supplier payments.</p><p>Making the transition to ePayments tools is often a daunting process for B2B buyers and suppliers. However, it’s crucial to explore all available ePayments opportunities for the success of back-office operations, the bottom line, and long-run competitive advantage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b676d61e8f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/challenges-and-opportunities-in-b2b-epayment-adoption-2b676d61e8f4">Challenges and Opportunities in B2B ePayment Adoption</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Get Started with Apache Mesos, Marathon, and ZooKeeper]]></title>
            <link>https://medium.com/@GetLevvel/how-to-get-started-with-apache-mesos-marathon-and-zookeeper-24fb72d76cf9?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/24fb72d76cf9</guid>
            <category><![CDATA[mesos]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[marathon]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 18:59:43 GMT</pubDate>
            <atom:updated>2018-08-10T14:06:02.207Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQW1WVcjMVPMF9knRPzLOQ.png" /></figure><p>One of the larger problems companies have when deciding to Dockerize their applications is that they often continue to think of managing their compute, memory, and CPU resources in a traditional Hypervisor-centric model.</p><p>The benefits of Docker in production can be more fully realized when the speed of infrastructure provisioning matches or exceeds the speed of application development, application containerization, and application release. In short, coupling the rapid application spin-up time of a Docker container or cluster of Docker services should be easily abstracted through a similarly rapid and simple datacenter interaction.</p><blockquote>Imagine if you could program against your entire datacenter as if it were a single pool of resources.</blockquote><h3>Apache Mesos</h3><p>Enter Apache Mesos, a distributed systems kernel with built-in support for Docker containers, an HTTP API, a Web UI, true high availability, fault tolerance, and is designed natively for the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*WHAIZSWmRN6DIXxoJHLLnw.jpeg" /></figure><p>Following along with the <a href="https://open.mesosphere.com/advanced-course/">Advanced Mesos Tutorial</a> provided by Mesos, let’s experiment with a simple Vagrant VM running Mesos.</p><h4>First Things First</h4><p>To begin, you’ll want to Install <a href="https://www.vagrantup.com/">Vagrant</a> and <a href="https://www.virtualbox.org/">VirtualBox</a>. For ease of starting, you may download this simple <a href="https://github.com/xacaxulu/mesos_vagrant">GitHub starter repo</a>. Vagrant will use the <strong>ansible_local</strong> provider to install Ansible on the guest VM and configure it with the necessary packages to run Mesos (PR’s accepted for those Ansibulls out there with better ideas for the playbook). Once you’ve cloned the repository, you can CD into it and run: <strong>$ vagrant up</strong></p><p>Per the Mesos instructions, our <em>Vagrantfile</em> and Ansible <em>playbook.yml</em> will execute tasks which perform the following:</p><ul><li>Install a simple Centos-7.1 Vagrant VM</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*r3jLkixeOA5kHhX4FgG4wg.png" /></figure><ul><li>Configure your /etc/hosts file to have the correct hosts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*dgWGbkePQLZUkzp6HawWfw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*iZm02CPWkqN2AMAyEHnauQ.png" /></figure><ul><li>Add the Mesosphere RPM repository</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*vpPZjoIJkQWR_2cjTQtk_Q.png" /></figure><ul><li>Install the Mesos and Marathon RPMs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*owlArI8HYpjdKZFSOv9j8Q.png" /></figure><p>Remember, you can use <strong>vagrant halt</strong>, <strong>vagrant destroy</strong>, and <strong>vagrant up</strong> to rebuild the VM at will.</p><p>Continuing on, the Ansible playbook installs Zookeeper, which Apache Mesos uses as the master election service.</p><p>1. Install the necessary ZooKeeper RPMs &amp; install it<br>2. Initialize the zookeeper-server to use ID of 1<br>3. Start ZooKeeper</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*vpPZjoIJkQWR_2cjTQtk_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Vy9UvXYpUMsnNEgqGBMyIA.png" /></figure><p>Next, our Ansible playbook will start the Mesos Master and Slave processes as well as starting Marathon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*keTSDfO6Fr2BbNUyOpbVkA.png" /></figure><p>Let’s run <strong>$ vagrant ssh</strong> to get into our Vagrant VM and test out a few things.<br><strong>$ vagrant ssh</strong><br><strong>$ sudo netstat -nlp | grep mesos</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Agu1qoo_osnnRI7P5QwbvQ.png" /></figure><p>Now, you should be able to access the Mesos user interface with your browser at 192.168.33.10:5050</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hqdJZyY_Wqrqi-STtIv08Q.png" /></figure><p>You can test the <em>mesos-execute</em> command via the following:<br><strong>$ export MASTER=$(mesos-resolve `cat /etc/mesos/zk` 2&gt;/dev/null)</strong><br><strong>$ mesos help</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3fe4eanmtLvrpZcsEePCzA.png" /></figure><h3>Using Marathon</h3><p>Marathon is a production-proven Apache Mesos framework for container orchestration, providing a REST API for starting, stopping, and scaling applications. Written in Scala, Marathon can run in highly-available mode by running multiple copies. The state of running tasks gets stored in the Mesos state abstraction.</p><p>Marathon is a <em>meta framework</em>, meaning you can start other Mesos frameworks such as Chronos or Storm with it to ensure they survive machine failures. It can launch anything that can be launched in a standard shell. In fact, you can even start another Marathon instances via Marathon.</p><blockquote>Mesos can launch anything that can be launched in a standard shell.</blockquote><p>Analogous to running <strong>$ sudo service marathon start</strong>, we’ll use Ansible’s service module to enable and start marathon. If you’ve ran <strong>$ vagrant up</strong>, this task has already been completed for you.</p><p>Visit the Marathon GUI by clicking 192.168.33.10:8080/</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Frp2O9jnKESLdKn41BoHDg.png" /></figure><p>Returning to the Vagrant VM CLI, you can test installing an application that will be run by Marathon.</p><p><strong>$ python -m SimpleHTTPServer</strong></p><p>Now visit 192.168.33.10:8000 and you should see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*uc3qXxnM72wTsuS8JGV3Mw.png" /></figure><p>Feel free to CTRL+Z this process and <strong>$ exit</strong> out of the Vagrant VM.</p><h3>Marathon GUI</h3><p>Now, let’s start using the Marathon GUI located at: 192.168.33.10:8080/ui/#/apps</p><p>Click on <strong>Create Application</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*deeNbh4XugZ_iPG6yzwjtg.png" /></figure><p>Here, we’re presented with a window in which we can name and configure an application to run in Marathon. In this case, we’ll use the same simple Python webserver:</p><p>For name: <strong>mypythonserver</strong></p><p>For command: <strong>python -m SimpleHTTPServer $PORT</strong></p><p>The $PORT ENV variable will dynamically assign this to a port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*FB1RnfuXpyUPrBlXx5JErw.png" /></figure><p>Click <strong>Create Application</strong> and we’re off. We now see the status go from deploying to running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*o_EuY2DHogjsvkirpTfCDQ.png" /></figure><p>We can discover details about this instance running and find the port to access it from a web browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*_q38j8n71wvDU1Ea1nQ1QA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*tGUrSqLs5cy52oDf7uuDEg.png" /></figure><p>In this case, the random port assigned is 31377, which can be accessed in a browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*VlULnXVr2H3tE_mPJUGSVQ.png" /></figure><p>We can now test the scaling ability with Marathon. I’ve set the instance configurations for mypythonserver to be 0.1 for CPU and 32 for Memory. Click on Scale Application, select 4 as the number of instances to scale to, and watch Marathon go to work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cLbnU8vHiP297QblyAaMvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*51cEV4Mx847MOToBr1ssdw.png" /></figure><p>This is just the tip of the iceberg for using Mesos to automate your datacenter. We’ll move onto more complex deployments in the next installment. Until then, happy hacking!</p><p><em>For more on DevOps, visit the </em><a href="https://levvel.io/blog"><em>Levvel Blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24fb72d76cf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HIPAA, PHI, and Caching, Part 2]]></title>
            <link>https://medium.com/levvel-consulting/hipaa-phi-and-caching-part-2-9a194eb13c09?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a194eb13c09</guid>
            <category><![CDATA[hipaa]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 18:20:14 GMT</pubDate>
            <atom:updated>2017-01-06T20:26:40.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*M478_MVMLENZXlwOVdmJuQ.jpeg" /></figure><p>by <a href="http://www.levvel.io/people/ben-vandgrift">Ben Vandgrift</a></p><p>In our <a href="http://www.levvel.io/blog-post/hipaa-phi-software/">previous article</a>, we introduced you to HIPAA, PHI, and provided an overview of what implications they might have for the implementation of software solutions. Now we’ll talk more specifically about what this means for caching PHI using the <a href="http://redis.io/">Redis in-memory data store</a>. Given the compliance requirements discussed earlier, implementing a caching solution that satisfies those requirements may not be straightforward. Redis does not include a “secure” configuration out of the box.</p><p>The solution that follows (inspired by an <a href="http://bencane.com/2014/02/18/sending-redis-traffic-through-an-ssl-tunnel-with-stunnel/">article</a> from Benjamin Cane, and adjusted to meet our needs) was built on behalf of a healthcare client in need of a HIPAA-compliant caching solution deployed to a public cloud infrastructure. Their application had grown to the point where caching was required, and no off-the-shelf solution was available. We’ll start with documenting what needs to be accomplished.</p><h3>Goals</h3><p>Here’s what we require in order to ensure HIPAA compliance of our cache that contains PHI:</p><ul><li><strong>Secure the environment:</strong> Our cache must run with dedicated tenancy, and be accessible via network only from application servers and bastion hosts.</li><li><strong>Tightly control access:</strong> Our cache should only be accessible by our application, and only with correct credentials. We’ll accomplish this by establishing a password for Redis, and allowing access to the host only with via ssh key.</li><li><strong>Encrypt in flight:</strong> PHI exchanged between the application and the cache, if moving across a network, must be encrypted. We’ll set up a secure tunnel between client and server.</li><li><strong>Encrypt at rest:</strong> Any PHI stored to disk by the caching mechanism should be encrypted. We’ll handle this by not allowing Redis to save data.</li><li><strong>Disable live configuration:</strong> In order to protect the other security measures in place, a client application must not be able to update the configuration after Redis starts.</li><li><strong>Document our decisions:</strong> This includes anticipated risks and their mitigations.</li></ul><p>We don’t need to account for the following:</p><ul><li><strong>Securing the cache itself:</strong> individuals don’t access a cache directly; the software uses the cache as a facilitation tool. As such, we don’t need to provide separate credentials for each user when accessing the cache.</li><li><strong>Ensuring integrity:</strong> since the PHI in the cache is intended to be transient and can be regenerated on demand, ensuring the integrity of the data isn’t necessary. It is worth ensuring that when a value is updated, it is updated correctly, but that is the responsibility of the application, not the cache itself.</li><li><strong>Auditing:</strong> Since data is being accessed and modified at the application level, we’ll focus our auditing efforts there, and not at the caching level.</li></ul><h3>Environment</h3><p>The solution we’re describing has been implemented on an AWS EC2 instance running Ubuntu 14.04 within a VPC with dedicated tenancy. If you’re deploying to a different Linux version, adjust these steps as appropriate.</p><h3>The Network</h3><p>The first line of defense is the network. When using redis as an application cache, you must ensure that the outside world can’t access the instance. In our Virtual Private Cloud, cache instances run in their own private subnet, accessible from the DMZ and the bastion host only. In AWS, this can be accomplished via an Access Control List configured to only allow access to the caching subnet from those two CIDR ranges, and a Security Group that denies all traffic except the following: Allow application hosts (or CIDR block) to access port 6379 (redis) Allow bastion hosts to access port 22 (ssh).</p><p>Also, recall that HIPAA requirements dictate that each person signing into a host should do so using their own (auditable) credentials. In general, HIPAA compliance is best served by using thorough network and host security practices.</p><h3>Manual Server Configuration</h3><p>We should install the server first. We’ll install the appropriate packages, configure redis, generate an x509 certificate, and configure stunnel.</p><h3>Package Installation (Server)</h3><p>As an administrative user (i.e., with sudo access), install the redis-server and stunnel4 packages on the redis host:</p><pre>apt-get install redis-server stunnel4 update-rc.d redis-server enable</pre><p>If it’s not already present on the system, openssl should be required by stunnel4. That’s it.</p><h3>Redis Configuration (Server)</h3><p>After installing the redis package, you’ll need to edit its config file, which can usually be found in /etc/redis/redis.conf. In particular, we will be adding a passphrase, disabling writes to disk, and renaming commands which cause insecurities. In the section of redis.conf dealing with persistence, comment (or remove) all the lines starting with save, or replace them with:</p><pre>save &quot;&quot;</pre><p>We also want to disable problematic commands. Include the following, selecting a new config command:</p><pre>rename-command CONFIG &lt;NEW_CONFIG_NAME&gt; rename-command SAVE &quot;&quot; rename-command BGSAVE &quot;&quot; rename-command BGREWRITEAOF “” rename-command DEBUG &quot;&quot;</pre><p>You can disable as many of redis’s standard commands as you like. That’s a basic list, but there are other things that are worthy of exploration (SHUTDOWN, FLUSHDB, and FLUSHALL come immediately to mind).</p><p>Finally, we want to ensure that applications (and users) connecting to redis require an additional layer of authentication. Add the following to redis.conf:</p><pre>requirepass &lt;SOME-PASSWORD&gt;</pre><p>By default, redis will run on port of 6379 of localhost. Review the redis.conf file to ensure this is true. Applications should not be able to connect from outside of the host without going through stunnel4.</p><p>Before we can configure stunnel4, however, we need an x509 certificate we can share between the server and any clients.</p><h3>Certificate Generation</h3><p>An x509 certificate can be purchased or generated on a host you control, as long as you have access to openssl. First you’ll generate a key, then a new certificate from that key:</p><pre>openssl genrsa -out /etc/stunnel/key.pem 4096 openssl req -new -x509 -key /etc/stunnel/key.pem -out /etc/stunnel/cert.pem -days 1826</pre><p>The -days flag specifies the number of days the certificate will remain valid; 5 years seems like a reasonable duration. You’ll also need to fill in a number of fields to complete the certificate. When complete, merge the key and certificate into a single .pem file, and change the permissions:</p><pre>cat /etc/stunnel/key.pem /etc/stunnel/cert.pem &gt; /etc/stunnel/private.pem chmod 640 /etc/stunnel/key.pem /etc/stunnel/cert.pem /etc/stunnel/private.pem</pre><p>This file will need to be present on the server (we’ll assume that the server is where it was generated), and copied to all clients that intend to connect.</p><p>Having generated the certificate, we’re ready to configure stunnel.</p><h3>Stunnel Configuration (Server)</h3><p>Configuring stunnel requires enabling the service and creating a tunnel file; each tunnel runs in its own process and connects to a single IP address and port. We’ll be setting up a redis-specific configuration.</p><p>In /etc/default/stunnel4, enable the service by setting ENABLED=1. When enabled, you can create a tunnel configuration in the /etc/stunnel directory. For redis, create /etc/stunnel/redis-server.conf:</p><pre>cert = /etc/stunnel/private.pem pid = /var/run/stunnel.pid [redis] accept = external-host-ip:6379 connect = 127.0.0.1:6379</pre><p>That’s it! Restart the redis and stunnel services, and move on to the client.</p><h3>Manual Client Configuration</h3><p>Having configured the server, you’ve already taken most of the steps you’ll need for a client configuration. You’ll need to repeat these steps on each client intending to access redis. The stunnel service runs transparently; as long as the stunnel service is configured and the service is running, it will appear to any client that redis is running locally, and accessible from 127.0.0.1 and port 6379. We’ve offloaded securing the connection from a client application to a system service for a couple of reasons.</p><p>First, stunnel is very fast, and will likely be faster than any client when handling encrypting and decrypting the connection to redis: that’s the only thing it’s built for.</p><p>Second, it allows us to create a wall between the application and the credentials necessary to access our redis cache. While the application will need to know the password we chose earlier, it won’t be able to view the x509 certificate. As long as your application is running as a user without administrative privileges, you’ll be able to sleep a little better at night.</p><h3>Package Installation (Client)</h3><p>We’ll need to install and configure stunnel4 on the client as well as the server; optionally, you may install redis-tools if you want to test the connection from the client before you trust your application with it.</p><pre>apt-get install stunnel4 [redis-tools]</pre><p>Before we can configure stunnel, we need a copy of the x509 cert generated on the server. For simplicity, drop it into the same place:</p><pre>/etc/stunnel/private.pem</pre><h3>Stunnel Configuration (Client)</h3><p>As before, we’ll need to set ENABLED=true in /etc/default/stunnel4, and create a tunnel file in /etc/stunnel. On the client, a couple of things are switched — in /etc/stunnel/redis-client.conf:</p><pre>cert = /etc/stunnel/private.pem client = yes pid = /var/run/stunnel.pid [redis] accept = 127.0.0.1:6379 connect = ip-of-redis-server:6379</pre><p>Restart the stunnel service, and you’re good to go. Yes, that’s all there is to it.</p><h3>Follow-up Questions</h3><p>Let’s talk about the questions that are likely on your mind before setting this up, namely: security in practice, trade-offs, scalability, and automation.</p><h3>How Secure Is It?</h3><p>Beginning with the understanding that there is no such thing as perfect security, this system provides a solid defense — with a few conditions. It only works if you’re a responsible administrator, and you’ve developed a secure application.</p><p>As we’ve defined it above, in order to compromise redis, an attacker must have access to the x509 certificate to make the connection, and must know the redis password defined in its config file. These are easy to achieve if an attacker can compromise the server or client hosts.</p><p>By keeping security patches current on the host, limiting access to the host via the network and firewall rules, and limiting the number of people able to sign into a user account on the host, your bases are largely covered. Remain vigilant and responsible.</p><p>Likewise, if an attacker can compromise a client application that connects to redis, the information contained therein is compromised. It’s likely they’ll also have access to other data sources in that case. Just as you need to remain vigilant to protect your network and hosts, use good security practices and tools when developing or integrating the client software.</p><p>Remember, however, that our goal is not to create a perfectly impregnable environment, but to ensure that we’re meeting HIPAA compliance recommendations and providing our users and customers with as much risk mitigation as we can, but subject to our chosen trade-offs. We’re also scoping this to the connection between a client application and a redis datastore running remotely. From that point of view, this arrangement is sufficiently secure.</p><h3>What are the Trade-offs?</h3><p>All technical decisions have trade-offs. In this case, we’re trading a certain amount of performance (sending and receiving data via stunnel carries an execution cost) for the ability to run a cache remotely, likely because it needs to be accessible from a number of application hosts.</p><p>We’re also creating a certain amount of complexity: an additional service must be configured and running on both client and server in order for the system to communicate. This is exactly the kind of thing one might neglect to implement in a test environment, so there is the potential for differences in performance and function between test and production. This risk can be mitigated by ensuring the test environment and the production environment match configurations.</p><h3>Does it Scale?</h3><p>Finally, because we’re choosing a simple method and a single redis server, our ability to scale our caching layer is limited. This is perhaps the greatest weakness of this system; however, for most applications, by the time you’ve reached the need for automatically-scaling distributed caching, you’ll need to re-evaluate your overall caching strategy anyway. This solution is most appropriate for applications that have grown sufficiently enough to need a caching layer but have not yet undergone a major performance optimization.</p><h3>What’s Next?</h3><p>With our <a href="http://www.levvel.io/blog-post/hipaa-phi-software">first article</a> in this series, we provided you with an overview of HIPAA compliance and PHI. Here, we’ve provided a straightforward lockdown of a Redis instance that satisfies all HIPAA compliance requirements. While we didn’t address documentation requirements or personnel responsibilities in this article, they will mirror those present in the rest of your application environment.</p><p>In this article, we configured the system manually–there is value in understanding all the steps to be performed to make this (relatively simple) lockdown a reality. In Levvel’s practice, we try to automate provisioning whenever possible. We’re also firm believers in open-source solutions. We’ll be following up this article with two additional libraries: a chef cookbook and puppet module, each of which implements the solution as described above for easy integration into your application infrastructure.</p><p>As in all cases of implementation, there are details that will vary between our system and your setup, and it should be easy to adjust accordingly. We hope you’ve gained another technique for building secure and performant software. Please feel free to <a href="/service/mailto:hello@levvel.io">contact us</a> with any questions.</p><p><em>For more on healthcare, visit the </em><a href="http://levvel.io/blog"><em>Levvel Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*i18EEda1Pmq5qB7w092obQ.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a194eb13c09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/hipaa-phi-and-caching-part-2-9a194eb13c09">HIPAA, PHI, and Caching, Part 2</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CloudBees + Docker]]></title>
            <link>https://medium.com/levvel-consulting/cloudbees-docker-7285eda10e06?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7285eda10e06</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloudbees]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 15:58:04 GMT</pubDate>
            <atom:updated>2017-01-06T20:27:53.672Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*73mj2xR9HgDb-oNW.png" /></figure><p><em>by </em><a href="http://www.levvel.io/people/jay-johnson"><em>Jay Johnson</em></a><em>, Senior Consultant at </em><a href="https://medium.com/u/fb16d70c66b9"><em>Levvel</em></a></p><p>At Levvel, we are always looking to improve how we build, test and deploy. It’s why we love DevOps. At its core, DevOps is a balancing act between cost and effort. Today, we’ll share an approach that is easy and cost-effective.</p><h3>A Jenkins Groovy Docker Container Pipeline</h3><p>Today’s post is about setting up your own DevOps artifact pipeline for docker containers running on <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition">CloudBees Jenkins Enterprise (CJE)</a>. For those who are new to the <a href="https://jenkins.io/solutions/pipeline/">CloudBees Pipeline-as-code</a>, it is software for defining your own custom build lifecycle and deployment actions. With just a few lines of Groovy code, you can be building and pushing your own tested Docker containers to your container registry.</p><h3>Keeping Things Under Your Control</h3><p>As a developer, I want a Docker container pipeline for iterating on my <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx">Django + nginx + Slack + Sphinx docs</a> stack that deploys using Docker Compose. This repository holds two containers (<a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/tree/master/django">Django</a> and <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/tree/master/nginx">nginx</a>) and I want to have a build tool that can: build both containers, run some container tests, and then push them to a container registry using source code that I can manage. Eventually, I’ll want to validate that the Docker Compose deployment passes integration tests before pushing to my registry, but that is out of scope for today’s post.</p><p>While there are many tools available for building containers, the advantage of CJE is that we have control of where the build environment is running and can scale it with <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform">CloudBees Jenkins Platform (CJP)</a>. This is a win for organizations looking to keep their builds on-premises or within their own infrastructure. The advantages of using Jenkins free versus CloudBees Jenkins Enterprise are discussed below.</p><h3>Getting Started</h3><p>This section provides a detailed tutorial for how to setup a Docker Container Pipeline with CJE. If you’re just interested in the benefits of CJE and how it differs from the community version, <a href="http://www.levvel.io/blog-post/building-devops-artifact-pipeline-for-docker-containers#taking-it-to-the-next-levvel">skip ahead</a>.</p><h3>Helpful Links</h3><p>If you are new to Jenkins Pipeline or Groovy, these are some helpful primers:</p><ul><li><a href="https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-workflow.html#workflow">Pipeline Plugin Suite</a></li><li><a href="https://jenkins.io/doc/pipeline/steps/">Pipeline Steps Reference Guide</a></li><li><a href="https://www.cloudbees.com/blog/top-10-best-practices-jenkins-pipeline-plugin">Pipeline Best Practices</a></li><li><a href="https://github.com/cloudbees/jenkins-scripts">CloudBees Jenkins Script Examples</a></li></ul><h3>Building the Docker Container Pipeline with CloudBees Jenkins Enterprise</h3><p>We will be using the <a href="https://github.com/cloudbees/docker.git">CloudBees Jenkins Enterprise Docker Repository</a> for this post and following the <a href="https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-docker-workflow.html">CloudBees Docker Workflow</a>.</p><ol><li>Start the CloudBees Jenkins Enterprise Docker container.</li></ol><pre>$ docker run -p 8080:8080 -p 50000:50000 -d --name cje -v /var/run/docker.sock:/var/run/docker.sock -v /usr/local/bin/docker:/usr/local/bin/docker cloudbees/jenkins-enterprise</pre><p>2. Open the Jenkins Credentials page.</p><p>Login to the CJE instance by browsing to: <a href="http://localhost:8080/">http://localhost:8080/</a> and then register for a free trial. Once logged in, click on Credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AN_oEyd_XC2lEb5n.png" /></figure><p>3. Setup your Docker Hub (or private registry) credentials.</p><p>Here you can add your docker credentials to CJE. I am currently housing my containers in Docker Hub: <a href="https://hub.docker.com/r/jayjohnson/django-slack-sphinx/">Django image</a> and <a href="https://hub.docker.com/r/jayjohnson/django-nginx/">nginx image</a> so I added my login credentials and named it <strong>jayjohnson-DockerHub </strong>. This will become the logical name associated to my user credentials for build jobs and items to use in the future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fz7-e6NLzqEd6nMt.png" /></figure><p>4. Click OK, then click the <strong>Manage Jenkins </strong>option to install Jenkins plugins. Once it is open, click on <strong>Manage Plugins</strong> .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aQI3AYlkvpt8CKkp.png" /></figure><p>5. Update the <strong>CloudBees Docker Pipeline </strong>plugin.</p><p>Update the <a href="https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Pipeline+Plugin">CloudBees Docker Pipeline plugin</a>; restart Jenkins after it finishes installing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FdLUzpQzI2D6DZ-0.png" /></figure><p>6. Add the <a href="https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/">Available Pipeline Utility Steps plugin</a> and restart Jenkins again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uVa72bwWd5Xu9j7-.png" /></figure><p>7. Create a New Pipeline Item.</p><p>Once Jenkins restarts, click on <strong>New Item</strong> and enter a name, then select it to be a <strong>Pipeline </strong>type.</p><p>Click the OK button once you are done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AR3wT01x29r_ibEy.png" /></figure><p>8. Paste the Jenkinsfile Contents into the Pipeline Groovy Section.</p><p>Per CloudBees best practices, the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/Jenkinsfile">Jenkinsfile is stored in the repository</a>. In the future, we can include this Jenkinsfile automatically with the <strong>Pipeline script from SCM </strong>option. For now, just copy the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L3-L158">highlighted lines in this link</a> into the <strong>Pipeline Definition </strong>text box.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VnsmjDtHOJsDeeBM.png" /></figure><p>9. Start the Build.</p><p>Now that our new Groovy Pipeline is ready to build and push the containers, we can kick off the Pipeline. Click <strong>Build Now </strong>to initiate the Pipeline job. This starts the Django and nginx container builds that auto-push to the Docker Hub registry. (Note: The build will fail since I am not distributing my Docker Hub credentials for this demonstration.)</p><p>It may take a few minutes to download and install the containers from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4cAgEju3DQJ4PruT.png" /></figure><p>10. Verify the images and tags were pushed to the registry.</p><p>Open the container registry and verify the <strong>testing </strong>tag was pushed. For this demo, I pushed these containers to Docker Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xx60--WNgJkDNoTJ.png" /></figure><h3>How the Groovy Pipeline Works</h3><p>The repository uses <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L3-L158">this Jenkinsfile</a> in the root of the repository. For now, it is using a single node to run the Groovy code, which means it will only run on one node/slave. Below is a breakdown of what the Groovy code is doing. Each major section is broken up by a <strong>stage </strong>declaration that makes it easier to debug when looking at the Stage View screen. In this sample code, each of the green boxes seen in the image from <strong>Step 9 </strong>above is part of a defined Pipeline <strong>stage </strong>declaration.</p><ol><li><a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L5-L8">Set up auth credentials</a></li></ol><p>This section allows you to configure the registry, login, and default build tag.</p><pre>registry_url = &quot;https://index.docker.io/v1/&quot; // Docker Hub<br>docker_creds_id = &quot;jayjohnson-DockerHub&quot; // name of the Jenkins Credentials ID<br>build_tag = &quot;testing&quot; // default tag to push for to the registry</pre><p>2. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L10-L11">Define the build repo</a></p><p>This section defines the GitHub repo to target during the build.</p><pre>stage &#39;Checking out GitHub Repo&#39;<br>git url: &#39;https://github.com/jay-johnson/docker-django-nginx-slack-sphinx.git&#39;</pre><p>3. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L13-L14">Target the Docker Registry for the Django container</a></p><p>This section uses the CloudBees Docker Pipeline plugin to target a container registry (Docker Hub) and the ID for my Jenkins Credentials for the registry (jayjohnson-DockerHub). Learn more about <a href="https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs">Injecting Secrets with Jenkins Builds</a>.</p><pre>stage &#39;Building Django Container for Docker Hub&#39;<br>docker.withRegistry(&quot;${registry_url}&quot;, &quot;${docker_creds_id}&quot;) {</pre><p>4. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L16-L33">Build the Django Docker container</a></p><p>Assign the container maintainer and the container name, then read in the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/testing.env">testing Docker env file</a>, then kick off the build. A nice feature of the CloudBees Docker Pipeline is that it allows a <strong>Dockerfile </strong>in a repository’s subdirectory. This is why the <strong>django </strong>directory is added as an argument in the docker.build() method. View the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/tree/master/django">Django subdirectory</a>, which holds the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/django/Dockerfile">Django on CentOS 7 Dockerfile</a>.</p><pre>// Set up the container to build <br>maintainer_name = &quot;jayjohnson&quot;<br>container_name = &quot;django-slack-sphinx&quot;<br>docker_env_file = &quot;testing.env&quot;<br><br>// Read testing environment file:<br>docker_env_values = readProperties file: &quot;./${docker_env_file}&quot;<br><br>// Assign variables based off the env file<br>default_root_volume = &quot;${docker_env_values.ENV_DEFAULT_ROOT_VOLUME}&quot;<br>doc_source_dir = &quot;${docker_env_values.ENV_DOC_SOURCE_DIR}&quot;<br>doc_output_dir = &quot;${docker_env_values.ENV_DOC_OUTPUT_DIR}&quot;<br>static_output_dir = &quot;${docker_env_values.ENV_STATIC_OUTPUT_DIR}&quot;<br>media_dir = &quot;${docker_env_values.ENV_MEDIA_DIR}&quot;<br><br>stage &quot;Building&quot;<br>echo &quot;Building Django with docker.build(${maintainer_name}/${container_name}:${build_tag})&quot;<br>container = docker.build(&quot;${maintainer_name}/${container_name}:${build_tag}&quot;, &#39;django&#39;)</pre><p>5. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L40">Start the Django container</a></p><p>The code below starts the Django server by using the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/testing.env">testing Docker env file</a>, which <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/testing-docker-compose.yml">testing-docker-compose.yml</a> targets for integration testing using the <a href="https://hub.docker.com/r/jayjohnson/django-slack-sphinx/tags/">testing</a> tag. Using the environment file keeps our Groovy code a little cleaner than the original version, and we can run this Django server in <strong>DEV </strong>mode for curl testing later in our container validation steps. Being able to specify environment variables to run containers via the <a href="https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-docker-workflow.html#docker-workflow-sect-run">withRun</a> method is great for validating the container works as expected with Docker Compose.</p><p>Previously, <a href="http://blog.levvel.io/blog-post/a-docker-container-pattern-compose-configuration/">we talked about</a> how much easier Docker development is when using Docker Compose to drive the container’s configuration. In this repository, I am gluing a python Django server together with an nginx server through the use of a shared volume exposed from the host. This is handy for serving static assets (css, javascript, images, etc.) with a proven load balancer like nginx. It also allows for defining environment specific resources, such as <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/c819693d6a007528ca8ca96c744c577a933f002e/django/containerfiles/django/wsgi/server/webapp/api.py#L40-L48">where to post exceptions into Slack</a>, without changing any code or the container. The values are also defined in the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/docker-compose.yml">docker-compose.yml</a> and <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/testing-docker-compose.yml">testing-docker-compose.yml</a> (which uses <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/master/testing.env">testing.env</a>) files.</p><pre>// Run the container with the env file, mounted volumes and the ports:<br>docker.image(&quot;${maintainer_name}/${container_name}:${build_tag}&quot;).withRun(&quot;--name=${container_name} --env-file ${docker_env_file} -e ENV_SERVER_MODE=DEV -v ${default_root_volume}:${default_root_volume} -v ${doc_source_dir}:${doc_source_dir} -v ${doc_output_dir}:${doc_output_dir} -v ${static_output_dir}:${static_output_dir} -v ${media_dir}:${media_dir} -p 82:80 -p 444:443&quot;)  { c -&gt;</pre><p>6. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L42-L63">Wait for the Django container to start</a></p><p>Once the container starts running, we need to wait for it to initialize the Django server process. This can take a few seconds, which is why the <a href="https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-unstash-code-restore-files-previously-stashed">waitUntil</a> method is very helpful. This code block will continue retrying itself using an exponential backoff retry timer until it returns true. By using this block, we can ensure that the container is running and that the internal Django server is ready to respond to HTTP requests.</p><pre>// wait for the django server to be ready for testing<br>// the &#39;waitUntil&#39; block needs to return true to stop waiting<br>// in the future this will be handy to specify waiting for a max interval: <br>// https://issues.jenkins-ci.org/browse/JENKINS-29037<br>waitUntil {<br>  sh &quot;docker exec -t ${container_name} netstat -apn | grep 80 | grep LISTEN | wc -l | tr -d &#39;\n&#39; &gt; /tmp/wait_results&quot;<br>  wait_results = readFile &#39;/tmp/wait_results&#39;<br><br>  echo &quot;Wait Results(${wait_results})&quot;<br>  if (&quot;${wait_results}&quot; == &quot;1&quot;)<br>  {<br>      echo &quot;Django is listening on port 80&quot;<br>      sh &quot;rm -f /tmp/wait_results&quot;<br>      return true<br>  }<br>  else<br>  {<br>      echo &quot;Django is not listening on port 80 yet&quot;<br>      return false<br>  }<br>} // end of waitUntil</pre><p>7. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L68-L73">Begin Django container tests</a></p><p>For simplicity, I limited this example to three container tests. By default, the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L75">expected result</a> for each test expects a 0 returned, but each test can control this expected value.</p><pre>// this pipeline is using 3 tests <br>// by setting it to more than 3 you can test the error handling and see the pipeline Stage View error message<br>MAX_TESTS = 3<br>for (test_num = 0; test_num &lt; MAX_TESTS; test_num++) {     <br>   <br>    echo &quot;Running Test(${test_num})&quot;</pre><p>8. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L76-L81">Test the Django container shows the home page</a></p><p>From outside the container, run a docker exec to issue a curl command. This command will return the contents of the home page, then count the number of “Welcome” occurrences and trim any newline characters out before writing the cropped output to a temporary file.</p><pre>if (test_num == 0 ) <br>{<br>    // Test we can download the home page from the running django docker container<br>    sh &quot;docker exec -t ${container_name} curl -s http://localhost/home/ | grep Welcome | wc -l | tr -d &#39;\n&#39; &gt; /tmp/test_results&quot; <br>    expected_results = 1<br>}</pre><p>9. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L82-L89">Test the Django container is configured to listen on Port 80</a></p><p>This is a redundant test of the curl command above, but I wanted to show how to utilize <a href="https://docs.docker.com/engine/reference/commandline/inspect/">docker inspect</a> from a Groovy script. This is helpful when you need to verify that Docker Compose deployed the composition correctly before promoting the container to production. Like the test above, this counts the occurrences of port 80 being open externally and trims the results to a temporary file.</p><pre>else if (test_num == 1)<br>{<br>    // Test that port 80 is exposed<br>    echo &quot;Exposed Docker Ports:&quot;<br>    sh &quot;docker inspect --format &#39;{{ (.NetworkSettings.Ports) }}&#39; ${container_name}&quot;<br>    sh &quot;docker inspect --format &#39;{{ (.NetworkSettings.Ports) }}&#39; ${container_name} | grep map | grep &#39;80/tcp:&#39; | wc -l | tr -d &#39;\n&#39; &gt; /tmp/test_results&quot;<br>    expected_results = 1<br>}</pre><p>10. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L90-L95">Test Django does not have an ESTABLISHED connection on Port 80</a></p><p>This is another demonstration of how to log into the container and verify the internal Django process is running correctly. Since nginx is not being deployed at this time and there are no incoming connections with this test, Django should be in a LISTEN state without any ESTABLISHED connections on port 80. In the future, an integration test could verify the deployed composition successfully established connectivity from ngninx to the Django server during normal operation. As above, this counts occurrences of ESTABLISHED connections and writes the trimmed output to a temporary file.</p><pre>else if (test_num == 2)<br>{<br>    // Test there&#39;s nothing established on the port since nginx is not running:<br>    sh &quot;docker exec -t ${container_name} netstat -apn | grep 80 | grep ESTABLISHED | wc -l | tr -d &#39;\n&#39; &gt; /tmp/test_results&quot;<br>    expected_results = 0<br>}</pre><p>11. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L96-L102">Exit and log an error for any unsupported tests</a></p><p>After a bit of debugging, this code allows the Jenkins Pipeline to stop running tests immediately when it executes. The code will auto-exit with an error message if you <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L68-L70">increase the MAX_TESTS</a> to something more than 3.</p><pre>else<br>{<br>    err_msg = &quot;Missing Test(${test_num})&quot;<br>    echo &quot;ERROR: ${err_msg}&quot;<br>    currentBuild.result = &#39;FAILURE&#39;<br>    error &quot;Failed to finish container testing with Message(${err_msg})&quot;<br>}</pre><p>12. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L104-L113">Check that the test results match the expected results</a></p><p>Each test will run this block and fail testing if the test results do not match the expected results. Make sure to remove that temporary file from the host afterwards to prevent test results overlapping by accident.</p><pre>// Now validate the results match the expected results<br>stage &quot;Test(${test_num}) - Validate Results&quot;<br>test_results = readFile &#39;/tmp/test_results&#39;<br>echo &quot;Test(${test_num}) Results($test_results) == Expected(${expected_results})&quot;<br>sh &quot;if [ \&quot;${test_results}\&quot; != \&quot;${expected_results}\&quot; ]; then echo \&quot; --------------------- Test(${test_num}) Failed--------------------\&quot;; echo \&quot; - Test(${test_num}) Failed\&quot;; echo \&quot; - Test(${test_num}) Failed\&quot;;exit 1; else echo \&quot; - Test(${test_num}) Passed\&quot;; exit 0; fi&quot;<br>echo &quot;Done Running Test(${test_num})&quot;<br><br>// cleanup after the test run<br>sh &quot;rm -f /tmp/test_results&quot;<br>currentBuild.result = &#39;SUCCESS&#39;</pre><p>13. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L117-L121">Stop testing for an exception</a></p><p>This code allows the Jenkins slave/executor to stop running tests if there is an exception (which is helpful when debugging complex Pipeline tasks).</p><pre>} catch (Exception err) {<br>    err_msg = &quot;Test had Exception(${err})&quot;<br>    currentBuild.result = &#39;FAILURE&#39;<br>    error &quot;FAILED - Stopping build for Error(${err_msg})&quot;<br>}</pre><p>14. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L123-L126">Push the Django container to Docker Hub</a></p><p>If testing passes, this code will push the container image to the registry (Docker Hub) under the <strong>testing </strong>tag.</p><pre>stage &quot;Pushing&quot;<br>container.push()<br><br>currentBuild.result = &#39;SUCCESS&#39;</pre><p>15. <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L129-L146">Test and push the nginx container to Docker Hub</a></p><p>This section builds the nginx container from the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/tree/master/nginx">nginx subdirectory</a> and then pushes the built image to the registry (Docker Hub) under the <strong>testing </strong>tag. It is a lightweight example for building a simple docker container pipeline.</p><pre>stage &#39;Building nginx Container for Docker Hub&#39;<br>docker.withRegistry(&quot;${registry_url}&quot;, &quot;${docker_creds_id}&quot;) {<br>    <br>    // Set up the container to build<br>    maintainer_name = &quot;jayjohnson&quot;<br>    container_name = &quot;django-nginx&quot;<br> <br>    stage &quot;Building Container&quot;<br>    echo &quot;Building nginx with docker.build(${maintainer_name}/${container_name}:${build_tag})&quot;<br>    container = docker.build(&quot;${maintainer_name}/${container_name}:${build_tag}&quot;, &#39;nginx&#39;)<br>   <br>    // add more tests<br>    <br>    stage &quot;Pushing&quot;<br>    container.push()<br>    <br>    currentBuild.result = &#39;SUCCESS&#39;<br>}</pre><h3>Enhancing the Pipeline</h3><p>I want to parallelize this build to run faster. To do this, I will perform the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L13-L127">Django build</a> step and the <a href="https://github.com/jay-johnson/docker-django-nginx-slack-sphinx/blob/51a698b64eb7d088910c6541b829c0522ed6acae/Jenkinsfile#L129-L146">nginx build</a> step using the <a href="https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins">native parallel support</a> to put the build tasks onto two slaves or multiple executors. This speeds up the builds and at the same time increases the build’s complexity for the upcoming Docker Compose integration tests I want to run. The build currently uses the slave host’s docker engine to perform the build which, if parallelized, could end up running on two separate hosts in an environment running more than one Jenkins slave/executor.</p><h3>Taking it to the Next Levvel</h3><p>This fun little demo demonstrates Jenkins Pipeline’s power. However, while it works in the simple, one-developer working off one branch scenario, it breaks down when utilized across a team running multiple concurrent builds using a git flow multiple Pull Requests development model. Numerous potential bottlenecks around building and running containers on one Jenkins host exist. When I want to scale this to multiple slaves, I have to first look at where my containers are housed before trying to deploy with Docker Compose to run my integration tests. Making Docker container builds work seamlessly across large organizations is a much larger task, and some organizations will need help in preventing those bottlenecks that causes teams to lose time waiting on builds and deployments.</p><h3>A Solution that Scales</h3><p>Levvel has partnered with <a href="https://www.cloudbees.com/">CloudBees</a> to help our clients adopt DevOps practices that save them time and money. They are a great option to consider when looking to prevent Jenkins development and deployment bottlenecks.</p><h3>Why Should I Pay for Something that is Free?</h3><p>A fair question, so let’s look at the DevOps landscape and how we got here.</p><p>Jenkins has massive adoption spanning multiple verticals and solves numerous DevOps use cases. Before Pipelines, Jenkins was already a successful continuous integration tool and a friend to many, many developers. With Pipelines, Jenkins is positioned to tackle continuous deployments at scale. When setup correctly, Pipelines are a powerful tool that will bring even more users to Jenkins. Being able to handle artifact deployments and post-build actions empowers organizations with the tools to get features in front of customers faster.</p><p>Inevitably, it is this simple and powerful combination that can lead to issues. There are <a href="https://wiki.jenkins-ci.org/display/JENKINS/Consideration+for+Large+Scale+Jenkins+Deployment">inherent complexities when running scaled-out multiple Jenkins master environments</a>. To prevent these kinds of issues and overhead, CloudBees launched the <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform">CloudBees Jenkins Platform (CJP)</a>, which offers two variants: <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition">Enterprise Edition</a> and <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform/private-saas-edition/features">Private SaaS Edition</a> (private cloud deployments on AWS or OpenStack with Jenkins running on Mesos for <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition/features/high-availability-plugin">High Availability</a>). By just starting with CJP, your organization gets:</p><ul><li>Benefits of administration for <a href="http://pages.cloudbees.com/rs/cloudbees/images/Jenkins-Operations-Center-by-CloudBees.pdf">Multiple Jenkins Masters with Jenkins Operation Center (JOC)</a>;</li><li><a href="https://go.cloudbees.com/docs/cloudbees-documentation/cje-user-guide/chapter-ha.html#">Resiliency;</a></li><li><a href="https://wiki.jenkins-ci.org/display/JENKINS/Multi-Branch+Project+Plugin">Multi-Branch Project</a> support for Bitbucket and Github;</li><li><a href="https://github.com/jenkinsci/github-organization-folder-plugin">Building all GitHub Organization Repositories;</a></li><li>Security, including <a href="https://www.cloudbees.com/products/cloudbees-jenkins-platform/team-edition/features/role-based-access-control-plugin">Role-Based Access Controls;</a></li><li><a href="https://go.cloudbees.com/docs/cloudbees-documentation/cjoc-user-guide/analytics.html">Customizable Elasticsearch and Kibana Analytics</a>.</li></ul><p>All this, plus access to the best-in-the-Jenkins-business gurus for supporting your organization’s continuous integration and continuous deployment capabilities with proven tools such as the Jenkins Pipeline.</p><p>Where CloudBees provides the enterprise with a comprehensive DevOps management platform, Levvel helps organizations:</p><ul><li>Migrate, setup and install CloudBees Jenkins Platform (Enterprise Edition or Private SaaS);</li><li>Deploy and scale large multiple Jenkins master environments that are managed with CloudBees Jenkins Operation Center;</li><li>Customize their Kibana Analytics dashboard;</li><li>Lock down their Jenkins environments.</li></ul><h3>What Comes Next</h3><p>I hope to hear your thoughts about this DevOps artifact pipeline for docker containers running on CloudBees Jenkins Enterprise. I have been looking for a simple way to control where my Docker builds were running in-house. The code within this blog post is already building my containers, and I hope it helps you get started with your Jenkins Pipeline.</p><p>Going forward, I plan to add Docker Compose integration tests to validate that my deployed composition is ready for primetime. Eventually, I could see the CloudBees Docker Pipeline wanting this supported natively out of the box. Docker Compose is already an established part of docker ecosystem and is super helpful when organizing and deploying complex topologies. It just makes good sense to vet the Compose deployment works prior to opening it up to production traffic.</p><p>If your organization would like assistance building your DevOps artifact pipeline using Jenkins, adopting CJP and CJE, Docker development or production strategy, please <a href="http://www.levvel.io/contact">reach out to us</a>.</p><h3>Extra Reading</h3><h4>How to stop, clean up, and troubleshoot your environment.</h4><p>Stopping the CloudBees Jenkins Enterprise Container</p><p>Stop the container with:</p><pre>$ docker stop cje<br>cje<br>$</pre><p>Remove the container (which will delete your Pipeline) with:</p><pre>$ docker rm cje</pre><h4>Troubleshooting</h4><p>From the command line, I had to set the permissions for the Jenkins container to access the docker socket:</p><pre>$ sudo chmod 777 /var/run/docker.sock</pre><p>If you want to build the CloudBees Jenkins Enterprise Docker container:</p><pre>$ docker build -t jenkins-enterprise .</pre><p>Modifying the docker engine to make sure it utilizes the correct services:</p><p><a href="https://docs.docker.com/engine/admin/systemd/">https://docs.docker.com/engine/admin/systemd/</a></p><p>I run a local docker swarm so I had removed the docker socket at /var/run/docker.sock from the systemd service file. So I had to re-enable it:</p><p>Then restart the docker engine with the following start command:</p><pre>sudo service docker restart</pre><p>Looking at the previous JenkinsCI work for help with Groovy I found this example helpful for getting the Pipeline working: <a href="https://github.com/jenkinsci/docker-workflow-plugin/blob/master/demo/repo/flow.groovy">https://github.com/jenkinsci/docker-workflow-plugin/blob/master/demo/repo/flow.groovy</a></p><p><em>For more content from Levvel, visit </em><a href="http://levvel.io/blog"><em>levvel.io/blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*i18EEda1Pmq5qB7w092obQ.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7285eda10e06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/cloudbees-docker-7285eda10e06">CloudBees + Docker</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HIPAA, PHI, and Software, Part 1]]></title>
            <link>https://medium.com/levvel-consulting/hipaa-phi-and-software-f672a37386cb?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f672a37386cb</guid>
            <category><![CDATA[health-technology]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[phi]]></category>
            <category><![CDATA[hipaa]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 15:05:26 GMT</pubDate>
            <atom:updated>2017-01-06T20:30:26.338Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v81uZARkPfadLrKSVkkDbg.jpeg" /></figure><p><em>by </em><a href="http://www.levvel.io/people/ben-vandgrift"><em>Ben Vandgrift</em></a></p><p>The software that we deliver for several of our clients contains medical information. As soon as we brush up against the words “medical” or “health,” we have to consider building in facilities to comply with the <a href="https://www.gpo.gov/fdsys/pkg/PLAW-104publ191/html/PLAW-104publ191.htm">Health Insurance Portability and Accountability Act of 1996</a> (HIPAA), the Health Information Technology for Economic and Clinical Health (HITECH) provisions in the <a href="https://www.gpo.gov/fdsys/pkg/PLAW-111publ5/html/PLAW-111publ5.htm">American Recovery and Reinvestment Act of 2009</a>, and the <a href="https://www.gpo.gov/fdsys/pkg/FR-2013-01-25/pdf/2013-01073.pdf">HIPAA Omnibus rule</a>. These three pieces of legislation tend to be referred to collectively as HIPAA. The part that gets the most attention is the “Privacy Rule” covering Private Health Information (PHI), and that’s what we’ll primarily concern ourselves with here.</p><p>Much of this legislation was written by lawmakers possessing limited knowledge on how software works, so complying with these regulations can feel more like interpretive dance and less like checking boxes on a solid list of rules. Even so, there are many ways in which a client (or an implementer) can get into trouble by failing to comply, and we like to avoid that kind of trouble. Reading all of the relevant regulations, expert recommendations, and the plethora of disjointed advice that may or may not apply to your situation can be daunting. It can help to focus on a small piece of the puzzle with big implications. In the next article in this series, we’ll focus on caching — more specifically, caching in a key-value store (we’ll be using Redis), deployed into a cloud environment (AWS, in our case).</p><p>Before we tackle that, we’ll dive into HIPAA a bit further. If you’re already familiar with HIPAA, you can <a href="http://www.levvel.io/blog-post/hipaa-phi-caching-with-redis">skip ahead to the tech</a>.</p><h3>What is PHI?</h3><p>The HIPAA Privacy Rule concerns itself with PHI, or “protected health information.” This consists of anything that connects health-related information and the person to whom it belongs. According to the U.S. Department of Health and Human Services (HHS), this is “all individually identifiable health information held or transmitted by a covered entity or its business associate, in any form or media, whether electronic, paper, or oral.” This includes, but is not limited to, names, email addresses, phone numbers, record numbers, photographs, and imaging of records that include PHI. Anything that can place a person next to their data is subject to the type of risk mitigation required by HIPAA. As software professionals, we tend to care about the Security Rule, which governs electronic PHI (e-PHI).</p><p>Obviously, the record of a visit to your physician or specialist would constitute PHI: It contains your name, address, phone, as well as the nursing assessment (heart rate, blood pressure, etc.), diagnosis, prescriptions, and so on. Less obviously, a pre-rendered and subsequently cached piece of view logic containing a patient name is also PHI, as is every record of a pivot table in a database that links a patient record identifier to a medical device identifier. The usual approach to dealing with PHI is to secure everything. It tends to be more time-efficient to treat everything in the system as PHI rather than identify what is and isn’t PHI and then building parallel systems to handle each.</p><p>The HIPAA regulations are written to cover “(1) health plans, (2) health care clearinghouses, and (3) health care providers who electronically transmit any health information in connection with transactions for which HHS has adopted standards”[1].</p><p>Large organizations with many concerns, some of which are health-related, can designate themselves as Hybrid Entities[2]. In that case, only the parts of the entity described as health-care components are subject to HIPAA. Entities “conducting certain functions on behalf of a covered entity” such as a software as a service provider that stores or manages PHI are covered as Business Associates. A covered entity is allowed to disclose PHI to a business associate, provided a Business Associate Agreement (BAA)[3] exists between the two parties. Under these circumstances, a covered entity can be indemnified from mishandling of the PHI by the Business Associate. The services covered under the Business Associate Agreement — and how they’re covered within that agreement — can help define who is responsible for securing the PHI addressed by the Privacy Rule.</p><p>Third-party organizations developing software for use by the public largely qualify as health care clearinghouses or business associates. As an example, AWS has a standard BAA that covers nine services. AWS ensures that HIPAA compliance is possible for those services under the shared responsibility model, though the specifics of the implementation are left to the service consumer.</p><p>Short answer: if you handle medical information that is coming from or headed to a health care provider, a health insurance company, or anyone in communication with either, you should seriously consider whether or not HIPAA applies to you.</p><h3>What—specifically—is required?</h3><p>HIPAA is not quite as broad as “it depends on who you ask,” but it doesn’t mandate specific measures either. While the Privacy Rule is pretty clear on what and who is covered, the Security Rule is somewhat less specific. It defines three classes of protection: administrative, technical, and physical. We’ll summarize them, but to give you an example, here is the text of §164.312(a)(2)(iv): “Encryption and decryption (Addressable). Implement a mechanism to encrypt and decrypt electronic protected health information.” This helpful little requirement doesn’t talk about when or where the PHI needs to be encrypted, so most implementers elect to cover their bases: encrypt the disk, encrypt the database record, encrypt the transport channel. However, the motivation behind the requirement may have been to encrypt data leaving the system, and to decrypt data entering the system. So instead of talking about that level of detail, let’s get to the higher-level summaries.</p><p>The administrative security requirements dictate a minimum level of bureaucracy necessary to document procedures and risk analysis, regularly conduct audits, identify appropriate user roles, and choose a responsible party for ensuring compliance with the Security Rule.</p><p>The physical security requirements are primarily organizational. They deal with access to facilities, proper (physical) security on workstations, and disposal of decommissioned hardware. Both physical security and administrative security are almost entirely outside the scope of this post.</p><p>The technical security mandates, our primary concern, include:</p><ol><li>Access control via unique user identification, authentication authorization, and system sign-off</li><li>Procedures for accessing PHI in an emergency (and presumably disaster recovery) scenario</li><li>Encryption “at rest” and “in flight”</li><li>Protection from unauthorized modification and destruction</li><li>Auditing</li></ol><p>There is also the Final Rule[4], 138 pages of fine print that set out clear dates, penalties, and clarifies the other rules. The National Institute of Standards and Technology has provided several products and documents to help figure out the details with specifics, but the goal of all of these measures is to make it as difficult as possible for unauthorized persons to view protected health information.</p><h3>You mentioned faxes?</h3><p>Yes, doctors still use faxes, and they spend thousands of dollars per year on fax machines, fax paper, telephone lines, and long distance bills to enable it. Why? Two reasons: First, despite the allocation of $30 billion dollars in the HITECH Act of 2009 to subsidize the digitization of health records, the cost of developing general electronic health record (EHR) software that is compliant with HIPAA’s restriction is high. That cost gets passed on to the hospitals and doctors’ offices that use the software. Second, efforts to standardize EHR for interoperability are still in the ‘efforts’ stage. That is, even if your general practitioner’s office buys a fancy HIPAA-compliant software system for storing electronic health records, there’s no guarantee that when you send that record to the hospital on behalf of your patient, their software will be capable of understanding it.</p><p>In order to create a true Health Information Exchange (HIE), a number of standards have to be in place, including but not limited to:</p><ul><li>The healthcare vocabulary must be standardized, so that words and acronyms mean the same in electronic health records across the board.</li><li>PHI stored electronically (<em>e.g.</em>, EHR) must have a consistent structure, so that it can be understood by different systems.</li><li>Communication channels between systems must be secured, whether via email or through a third-party exchange system or authenticated drop box.</li></ul><p>As you can imagine, building a set of standards robust enough to account for all EHR requirements and formats is complex, and would require the kind of consensus rarely seen in both government and non-governmental organizations (NGOs). Implementing those standards would be expensive, leading to expensive software products.</p><p>Faxes? Faxes work everywhere, and they work everywhere right now, so doctors, nurses, insurance companies, and many other organizations which handle PHI still use them.</p><p>How can this fundamentally insecure technology be HIPAA-compliant? Procedures surrounding the sending and reception of faxes stand in for technological solutions, since the technology was in place prior to HIPAA. When developing a new technology or software solution, the full scope of HIPAA applies. Because of this, it’s useful for us as technologists to understand HIPAA as best we can.</p><h3>What’s next?</h3><p>With this article we’ve introduced the concepts of HIPAA and PHI, and reviewed the broad brush strokes of compliance. We’ve also hinted at the difficulty inherent in developing software that is HIPAA compliant, especially if it needs to communicate with external systems. In the first follow-up, we’ll speak specifically about what this means for caching data using Redis, and describe how to implement a HIPAA-compliant Redis server.</p><p>[1] <a href="https://privacyruleandresearch.nih.gov/pr_06.asp">https://privacyruleandresearch.nih.gov/pr_06.asp</a><br>[2] <a href="https://privacyruleandresearch.nih.gov/pr_06.asp#6b">https://privacyruleandresearch.nih.gov/pr_06.asp#6b</a><br>[3] <a href="http://www.hhs.gov/hipaa/for-professionals/covered-entities/sample-business-associate-agreement-provisions/index.html">http://www.hhs.gov/hipaa/for-professionals/covered-entities/sample-business-associate-agreement-provisions/index.html</a><br>[4] <a href="https://www.gpo.gov/fdsys/pkg/FR%E2%80%932013%E2%80%9301%E2%80%9325/pdf/2013%E2%80%9301073.pdf">https://www.gpo.gov/fdsys/pkg/FR–2013–01–25/pdf/2013–01073.pdf</a></p><p><em>Continue on to </em><a href="https://medium.com/levvel-consulting/hipaa-phi-and-caching-part-2-9a194eb13c09#.v7grf07cp"><em>Part 2</em></a><em> of this topic.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*i18EEda1Pmq5qB7w092obQ.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f672a37386cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/hipaa-phi-and-software-f672a37386cb">HIPAA, PHI, and Software, Part 1</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sales Forecasting with Hadoop & R, Part 2]]></title>
            <link>https://medium.com/levvel-consulting/sales-forecasting-with-hadoop-r-part-ii-2a7bee3c82c8?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a7bee3c82c8</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[sales-forecasting]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[sales]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 14:48:39 GMT</pubDate>
            <atom:updated>2017-01-06T18:23:25.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*invywkgbwLMHazqQ-PUrkg.png" /></figure><p><em>by </em><a href="http://www.levvel.io/people/keith-laforce"><em>Keith LaForce</em></a><em>, Director at </em><a href="https://medium.com/u/fb16d70c66b9"><em>Levvel</em></a></p><p>This post is the second post in a series on data science. To see the first part, read <a href="http://www.levvel.io/blog-post/applying-data-science/">here</a>.</p><p>If you are following along with the source code, see <a href="https://github.com/GetLevvel/datascienceblogseries/tree/master/part2">here</a>.</p><p>In our first post of this series, we demonstrated how to perform a naive sales forecast for Initech’s CFO by using linear regression in Excel and R. However, this framework for forecasting will not scale once the IT department computes the sales data for hundreds of locations and departments. Manually performing sales forecast in Excel for hundreds of locations and aggregating the results into a report could potentially takes weeks of effort. How can we programmatically perform sales forecasts for the CFO at scale?</p><p>Hadoop is one of the premier tools for analyzing big data. According to the Hadoop website, “The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.”</p><p>Hadoop and MapReduce may seem like complicated concepts, but at its core MapReduce is a simple concept. MapReduce references two core components that Hadoop can leverage to perform a task. The first component is the Mapper function. The Mapper utilizes an input data set and converts it into another data set of keys and values. You can think of the Mapper as setting the stage for the Reducer function. The second component, the Reducer function, utilizes the output from a Mapper function as input and aggregates those key/values into another set of key/values.</p><p>Hadoop seems like the perfect tool to allow us to analyze and perform sales forecasts on the big data set IT is currently working to produce for the Initech CFO. There are a variety of methods to install Hadoop on your machine, but one method can be found <a href="http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/">here</a>.</p><p>Once Hadoop is installed, we need to set up our file system with our sales forecast data. Hadoop has its own distributed file system called HDFS. The distributed file system can be accessed via Hadoops hdfs command. Using the hdfs command will allow you to place your data on the distributed file system. Run the following commands (adjusted for your install) to copy our sample data set to the distributed file system:</p><pre>hdfs dfs -mkdir /user hdfs dfs -mkdir /user/hduser hdfs dfs -mkdir /user/hduser/salesforecastsimple hdfs dfs -put /Users/Keith/Projects/datascienceblogseries/part2/smalldataset.csv /user/hduser/salesforecastsimple/salesdata-in hdfs dfs -ls /user/hduser/salesforecastsimple</pre><p>The first three commands will setup your folder structure on HDFS. The next command will copy the smalldataset.csv to the HDFS using the -put switch of the hdfs dfs command. Finally, listing the contents of the directory to confirm that everything is configured properly.</p><p>Now that HDFS is configured, we can begin running Hadoop jobs to perform our sales forecast. Currently, we only have one file of sales data for testing purposes, but in our next post we will scale this forecast model to hundreds of files and Hadoop will begin to shine as a big data analysis tool.</p><p>Hadoop has two execution modes for data scientists to leverage. The first method of execution can be performed by creating a jar file (using any JVM based language) and running a hadoop command to execute the JAR. This is the normal mode of execution, but Hadoop also has a Streaming API that allows MapReduce jobs to run in any language or shell that can read STDIN and STDOUT.</p><p>For our sales forecasting, we will continue to use the R language to integrate with Hadoop’s Streaming API. For more details on the Streaming API see <a href="https://www.mapr.com/blog/how-using-non-java-programs-or-streaming-mapreduce-jobs">here</a>.</p><p>The streaming API works slightly different than the JAR based execution mode, but the key to integrating with the Steaming API is to adhere to the MapReduce contracts for STDIN and STDOUT. For example, the output of your Mapper must be your Key, followed by a Tab, the Value, and ending with a newline. When looking at our mapping code for our sales forecast you can see this <a href="https://github.com/GetLevvel/datascienceblogseries/blob/master/part2/mapper.R#L8">implemented</a> via the cat function in R.</p><p>In order to perform our Sales Forecast for Store 1/Department 1 we need to create executable scripts by running the following commands:</p><pre>chmod a+x ./hadoopForecast.sh chmod a+x ./mapper.R chmod a+x ./reducer.R ./hadoopForecast.sh</pre><p>The hadoopForecast script will kick off the MapReduce job utilizing our Mapper and Reducer scripts built in R. The output of this script will be the final sales forecast results for Store 1/Department 1 of Initech.</p><p>In our example for Initech, we will need to build a Mapper function that takes all Sales data and maps it into a format that can be used by the Reducer to perform the actual sales forecast. For our <a href="https://github.com/GetLevvel/datascienceblogseries/blob/master/part2/smalldataset.csv">sample data set</a> the data is already somewhat organized into keys and values. The Store and Department can be considered a key, while the sales number can be considered the value. Thus, our Mapper function will be trivial to massage the data into the appropriate format for the Reducer. We simply need to create key/value pairs for our sales data entries. In this case, a natural key will be the combination of Store Number and Department Number. We will want to keep track of the Sales and the Day the Sales occurred, so that is a natural fit for the value portion of the mapper. The code for the mapper can be seen in the repository, but ultimately all it does is output the data in the format StoreNumber$DepartmentNumber [tab] DayNumber$SalesAmount [newline].</p><p>To see what the output looks like before it goes to the reducer, you can run:</p><pre>cat smalldataset.csv | ./mapper.R</pre><p>The mapper takes our input data and maps it into key/value pairs that the Hadoop ecosystem will understand and be able to distribute across multiple nodes in parallel.</p><p>The more complex function in our sales forecast example will be our <a href="https://github.com/GetLevvel/datascienceblogseries/blob/master/part2/reducer.R">Reducer function</a>. When utilizing the JAR based Hadoop execution model, Hadoop is responsible for managing all the keys and files, but when using the Streaming API, your Reducer function must be aware of when Key changes occur and handle it appropriately. Thus, our reducer function has some bookkeeping code to track which Key we are currently on.</p><p>For quick reference, here is our Reducer example script:</p><pre>library(&#39;forecast&#39;) ForecastNextSale &lt;- function (salesData){ fit &lt;- ets(salesData) print(forecast(fit)) } mapOut &lt;- read.table(&quot;stdin&quot;,col.names=c(&quot;mapkey&quot;,&quot;mapval&quot;)) CurrKey &lt;- as.character(mapOut[1,]$mapkey) CurrVal &lt;- &quot;&quot; sales &lt;- numeric() for(i in 1:nrow(mapOut)){ Key &lt;- as.character(mapOut[i,]$mapkey) Val &lt;- as.character(mapOut[i,]$mapval) DataVal &lt;- unlist(strsplit(Val,&quot;\\$&quot;)) if (identical(Key,CurrKey)){ CurrVal = paste(CurrVal, Val) sales &lt;- c(sales, as.numeric(DataVal[2])) } else { ForecastNextSale(sale) CurrKey &lt;- Key CurrVal &lt;- Val sales &lt;- numeric() sales &lt;- c(sales, DataVal[2]) } } ForecastNextSale(sales)</pre><p>Hadoop is responsible for sorting the data that is fed to the reducer by Key so that we are guaranteed the correct order of data. The reducer code for our sales forecasting will simply keep track of the Sales Amount in a temporary vector until the <a href="https://github.com/GetLevvel/datascienceblogseries/blob/master/part2/reducer.R#L23">key changes</a>. Once the key changes we know that all data has been processed and we can perform our final forecast.</p><p>Sales forecasting has a number of valid techniques from linear regression to exponential smoothing to time series analysis. Exponential smoothing is the technique we have chosen for performing the sales forecast for Initech. The details behind exponential smoothing can be found <a href="https://www.otexts.org/fpp/7/1">here</a>. However, we do not want to reinvent the wheel, so we have chosen to leverage the R forecast package to perform the heavy lifting. The R forecast package allows for forecasting to be done quickly and effectively. The ets function has a number of options, but for purposes of demonstration we have chosen the most basic <a href="https://github.com/GetLevvel/datascienceblogseries/blob/master/part2/reducer.R#L6">use case</a>. The default R forecast ets function will generate output that contains the forecast for the next 10 cycles and options for an 80% and 95% confidence interval. Simply pass ets a vector of numbers, and it will handle the rest for you.</p><p>In order to run all of our code through Hadoop, run the following command ./hadoopForecast.sh</p><p>The following demonstrates the output from our Hadoop job:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*zbnzcoaPReHnRbhk.png" /></figure><p>Today we have covered a simple example of utilizing data science to give Initech’s CFO a sales forecast for one specific department in one store using the exponential smoothing forecasting method. In our next post we will have gathered all the sales data from the IT department and fully leverage Hadoop to help us perform sales forecasts on a big data set at scale. This post will leverage Amazon’s EMR to help perform our sales forecast quickly and cost effectively.</p><p><em>For more Levvel content, check out </em><a href="http://levvel.io/blog"><em>levvel.io/blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a7bee3c82c8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/sales-forecasting-with-hadoop-r-part-ii-2a7bee3c82c8">Sales Forecasting with Hadoop &amp; R, Part 2</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Your App at Risk of Getting Kicked out of the App Store?]]></title>
            <link>https://medium.com/levvel-consulting/is-your-app-at-risk-of-getting-kicked-out-of-the-app-store-88bc0c2058ca?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/88bc0c2058ca</guid>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[app-store]]></category>
            <category><![CDATA[apps]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Thu, 05 Jan 2017 21:10:43 GMT</pubDate>
            <atom:updated>2017-01-05T21:10:43.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*06yAKqu2W2eReEZa.jpeg" /></figure><p><em>by </em><a href="http://www.levvel.io/people/steve-moser"><em>Steve Moser</em></a><em>, Senior Consultant at </em><a href="https://medium.com/u/fb16d70c66b9"><em>Levvel</em></a></p><p>While developers are thinking about Swift 3, Xcode 8, and support for iOS 10, app owners are considering whether it is the time to update their years-old iOS apps. Apple has just given app owners a great reason to consider updating their apps, <a href="https://developer.apple.com/support/app-store-improvements/">announcing</a> that it is “cleaning up” the App Store by removing old apps that haven’t been updated in some time and will thus fail to pass the new app review process. If your app becomes non-compliant, Apple is giving you 30 days to update it. This process has the potential to affect a large number of apps, as <a href="https://david-smith.org/blog/2016/09/03/a-sizeable-cleanup/">roughly half</a> of the apps in the App Store have not been updated in the last year. This post will walk you through the main points to consider when updating your app for iOS 10.</p><p>When you first open your app in Xcode 8, you’ll be prompted to update the version of Swift your app uses. For the first time, you have two options: you can either make a smaller update to Swift 2.3 or leap all the way forward to Swift 3.0. By updating to Swift 3.0, you’ll not only be more prepared for the next version of Swift but you’ll also have access to Xcode 8’s new <a href="https://developer.apple.com/videos/play/wwdc2016/412/">thread sanitizer</a> and <a href="https://developer.apple.com/videos/play/wwdc2016/410/">visual memory debugger</a>. The thread sanitizer helps surface difficult to find concurrency issues. The visual memory debugger graphs object ownership in your app, which can help you detect memory leaks. However, updating to Swift 3.0 is a more difficult update than moving to 2.0 for many reasons, one of them being the pervasive syntax changes. You’ll also have to update your dependencies for Swift 3.0. Hopefully, the Swift version migrator runs flawlessly but, if it doesn’t, you can rerun the migrator. For a sufficiently large Swift project, however, you’ll probably have to manually update your code, which is beyond the scope of this post.</p><p>Since Objective-C hasn’t had any major changes in the past couple of years, you should be able to build and run your app in Xcode 8 without any errors, for the most part. However, you’ll probably see new warnings about nullability annotations and lightweight generics. <a href="https://developer.apple.com/swift/blog/?id=25">Nullability annotations</a> let the compiler and other developers know which pointers may or may not be null. This also allows your swift code to use optionals for pointer types defined in your Objective-C code that are annotated as nullable. Lightweight generics allows you to specify the type of objects inside of a collection. Adopting both of these changes can be tedious but they should be a priority — not only because they allow your Swift code to better interoperate with your Objective-C code — but because they will also make your code more explicit and safer.</p><p>Your next decision when upgrading your app is which minimum iOS version you want to target. This is different from the base SDK, which should be set to ‘Latest’ and, consequently, will be iOS 10 in Xcode 8. Because iOS users update their devices so quickly, the general consensus and Apple’s recommendation is to target the last minor version of the previous major release. Since iOS 10.1 is out, that means your app’s deployment target should be 9.3. With the new deployment target set, be sure to remove code that references previous iOS versions through system version checks. While we’re on the subject of system version checks, make sure your code doesn’t check for the system version like this:</p><pre>#define IsIOS7 ([[[[UIDevice currentDevice] systemVersion] substringToIndex:1] intValue]&gt;=7)</pre><p>This will return false for iOS 9 but true for iOS 10, because it only compares the first character of the returned system version. This is apparently as <a href="https://github.com/search?q=%255B%255B%255BUIDevice+currentDevice%255D+systemVersion%255D+substringToIndex%253A1%255D&amp;type=Code&amp;utf8=">widespread</a> as the <a href="http://www.pcworld.com/article/2690724/why-windows-10-isnt-named-9-windows-95-legacy-code.html">Windows system version check</a> looking for Windows 95 or 98.</p><h3>Find and Fix New Warnings</h3><p>When you compile and run your app, you’ll most likely see some new warnings about deprecated methods, but hopefully your app runs the first time. If it exits on launch, check the console for a line similar to this:</p><pre>The app&#39;s Info.plist must contain an NSPhotoLibraryUsageDescription key.</pre><p>Usage description keys have been around awhile but, starting with iOS 10, Apple requires all systems prompts for access to the user’s data to include a usage description. If your app doesn’t include the corresponding description for a system prompt, it will exit on launch. Simply add a usage description key for each framework that accesses user data to your app’s info.plist file. If you already included usage description keys in the past, then be sure to also check out the <a href="https://developer.apple.com/library/content/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html">new required usage description keys</a>. Note that your app might not exit on launch if a dependency in your library links against one of these frameworks, but you still need to include a usage description; otherwise, your app will not pass validation when uploading to iTunes Connect.</p><p>On the topic of dependencies, now would be a good time to update those as well. If you are using CocoaPods, update your Podfile with your deployment target and then run ‘pod update’ to fetch updates for your pods.</p><p>Before you start working your way down the list of new warnings, first update to the recommended Xcode project settings by selecting ‘project’ in the sidebar, then in the menu select Editor → Validate Settings…, then Perform Changes. Among other things, this will turn on new compiler checks that will generate new warnings. After you have fixed the resulting warnings, go ahead and run the static analyzer on your project, since it includes new checks in Xcode 8.</p><p>In addition to fixing deprecated methods, I ran into some other warnings when updating my iOS 7 app to iOS 10, such as a crash when presenting UIActivityViewController on iPad. If you try to present a UIActivityViewController without specifying a popover presentation controller source view in iOS 8 or newer, your app will crash and produce the following log within the console:</p><pre>Your application has presented a UIActivityViewController (&lt;UIActivityViewController: 0x7ffd88c6c730&gt;). In its current trait environment, the modalPresentationStyle of a UIActivityViewController with this style is UIModalPresentationPopover. You must provide location information for this popover through the view controller&#39;s popoverPresentationController. You must provide either a sourceView and sourceRect or a barButtonItem. If this information is not known when you present the view controller, you may provide it in the UIPopoverPresentationControllerDelegate method -prepareForPopoverPresentation.</pre><p>Fixing this is usually as simple as what the log suggests. In my case, I set my UIActivityViewController popoverPresentationController’s sourceView to the button that calls up the Activity View Controller.</p><p>Another issue I encountered related to launch storyboard requirements. iOS 8 introduced launch storyboards, which allow you to specify how your launch image displays across all supported iOS device sizes with a single adaptive storyboard. This will save time in the long run as Apple introduces new device sizes, but if you are not ready to make that investment you can opt out of it on iPad-only apps with the ‘UIRequiresFullScreen’ info.plist key.</p><p>Once you’ve updated your deployment target, updated your dependencies, and fixed your warnings, you’re ready to upload your app to iTunes Connect. I would suggest, however, that you take the time to go beyond SDK compatibility and strive for taking advantage of the updated SDK.</p><h3>iOS 8 SDK</h3><p>The biggest change that the iOS 8 SDK brings to the table is adaptivity. Adaptivity allows developers to specify how their app should best fill a certain screen size through AutoLayout and new adaptivity APIs handle the rest. Developers no longer have to maintain different files for iPhone and iPad layout. While the new adaptivity APIs are powerful, they can take a significant amount of time to adopt — especially if your app doesn’t use AutoLayout yet. Before iOS 8, the only screen size change introduced to iOS came as a result of the extra height of the iPhone 5 and iPhone 5s devices. These were relatively simple to support since most content on iOS scrolls vertically. When the iPhone 6 and 6 Plus devices arrived, they changed not only the height but also the width of the screen. Now developers have to consider how their layout and especially their text layout expands and truncates with the different sizes. Designers also have to get involved by cutting new 3x assets for iPhone devices with 5.5-inch Retina HD Displays. Note that Apple still accepts iPhone 5-only sized apps in the App Store.</p><p>If you’re not ready to take on this change, your app will simply expand to fill the screen. However, users will notice that you are not taking full advantage of their screen size especially when they try to type on the blown up keyboard they are unaccustomed to seeing in other appropriately sized apps. Also, no one knows how much longer Apple will allow updates for iPhone 5 only sized apps in the App Store.</p><h3>iOS 9 SDK</h3><p>The iOS 9 SDK continued with the theme of adaptivity from the iOS 8 SDK by bringing support to iPad. Since iOS 9 allows iPad users to use apps in split screen and slide-over modes, developers must allow their app to gracefully fill these new sizes. Like before, Apple will still accept apps that don’t support these new modes, but no one knows for how long.</p><p>3D Touch was another enhancement that Apple introduced with iOS 9. It allows for new peek and pop pressure sensitive options. Thankfully, these are relatively easy to support for vanilla UIKit applications.</p><h3>iOS 10 SDK</h3><p>This year’s iOS 10 SDK is relatively easy to support, especially since no new device screen sizes were introduced with it, though you do have to contend with App Transport Security (ATS) starting in the new year. ATS is a security measure by Apple to ensure that your app communicates securely with its back end. In order to meet its security requirements, your app must use HTTPS and TLS version 1.2 with forward security. ATS was easily ignored when it was introduced in iOS 9 by setting the NSAllowsArbitraryLoads key to YES under the NSAppTransportSecurity dictionary key. This prevents the alert below from popping up. However, starting January 1, 2017, all apps submitted to the App Store are required to use ATS. If you foresee this being an issue for your app, you’ll want to get your app update out before this requirement hits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/0*pHRbmyv8ytF_t-nj.png" /></figure><h3>iTunes Connect</h3><p>Before you submit your update to the App Store via iTunes Connect, you might need to guard against roadblocks, such as code signing. If it has been a year since your last update, then your app signing setup is probably expired. Thankfully, Apple has further improved its ‘Fix Issue’ button by adding Automatic Code Signing with Xcode8. Previously, this button might overwrite a previous provisioning profile that was correctly set up on a coworker’s machine and thus render the coworker’s code signing setup useless. This brought on so much frustration that a developer made an Xcode plugin to disable this button entirely in order to prevent wasted afternoons of resetting of provisioning profiles. With Xcode 8, Automatic Code Signing instead uses a dedicated profile managed by Xcode, which means you can turn on Automatic Code Signing without worrying about messing up your coworker’s signing setup after checking in your project settings file. Just go to your project’s target and tick on the Automatic Code Signing checkbox shown below and then click ‘Enable Automatic’ to accept the changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/0*q712Rfz9-x0cSVGB.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*rZpyYesKQigdKmoZ.png" /></figure><p>With code signing fixed, you can now create a new version of your app in iTunes Connect, build an archive, and upload your app. If everything went okay, then you’ll be ready to submit your app for review. If your app uploaded to iTunes Connect but you don’t see the build, be patient. If you still don’t see your build after 30 minutes, check the developer email address associated with your app. When I tried this with my app, I discovered that one of my dependencies had linked against a framework that requires a usage description key for privacy purposes. If you run into issues like this, you’ll need to increase your app’s build number (not its version number) after you fix the issue and upload it again in order for iTunes Connect to see the new version of your app.</p><p>Even now, just as you’re ready to hit the ‘Submit for Review’ button, you might hit one or two more snags. In recent years, Apple has reduced the number of allowed special characters in your app description. Simply remove those restricted characters and move on to the screenshots, if this applies. If you’re submitting an iPad compatible application, iTunes Connect needs an iPad Pro screenshot. This shouldn’t take long to do well, but if you’re in a real hurry, Apple will accept blown up versions of your existing iPad screenshots.</p><h3>Conclusion</h3><p>Getting your app up and running with the iOS 10 SDK might take a little more time and require extra attention, but it’s worth it. Not only will you get to keep your app in the App Store, but users will appreciate the speed and reliability that comes with building your app against the latest SDK.</p><p><em>For more on applications, visit </em><a href="http://levvel.io/blog"><em>levvel.io/blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88bc0c2058ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/is-your-app-at-risk-of-getting-kicked-out-of-the-app-store-88bc0c2058ca">Is Your App at Risk of Getting Kicked out of the App Store?</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[API Management and Perimeter Security for COTS Applications]]></title>
            <link>https://medium.com/levvel-consulting/by-robert-c-broeckelmann-jr-principal-consultant-at-levvel-691cd2aa300?source=rss-fb16d70c66b9------2</link>
            <guid isPermaLink="false">https://medium.com/p/691cd2aa300</guid>
            <category><![CDATA[cot]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kronos]]></category>
            <category><![CDATA[consulting]]></category>
            <dc:creator><![CDATA[Levvel]]></dc:creator>
            <pubDate>Thu, 05 Jan 2017 20:37:19 GMT</pubDate>
            <atom:updated>2017-01-06T14:56:40.518Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c_wZyHdR48GgjnfX.png" /></figure><p><em>by </em><a href="http://www.levvel.io/people/robert-c.-broeckelmann-jr."><em>Robert C. Broeckelmann Jr.</em></a><em>, Principal Consultant at </em><a href="https://medium.com/u/fb16d70c66b9"><em>Levvel</em></a></p><p>Kronos provides a suite of workforce management solutions, including a mobile application for activities like clocking in and out of work, staff scheduling for shift management, and vacation requests. Our client previously deployed the Kronos Workforce Central server software to handle time tracking for hourly employees and other employee-related activities. Recently, the client decided to deploy the Kronos Workforce Mobile application to give managers access to shift and schedule data and approval functionality without having to be in front of a corporate workstation. The Kronos mobile app was ready to solve an important business problem, but an early architecture and security review of the proposed system uncovered an issue: perimeter security.</p><p>Most security integration engagements in the mobile space that I’ve been involved with have a custom client in the form of a native mobile application. If any commercial-off-the-shelf (COTS) solutions are used, the COTS component(s) are typically on the server-side. This project was somewhat unique in that both the mobile app (API consumer) and Workforce Central app servers (API provider) were part of a COTS solution — from the same vendor.</p><p>Most of the time, enterprise integration projects have COTS components from multiple vendors and the security integration mechanisms must map the front-end security model to the back-end security model. The goal is to have well-defined, flexible front-end security model and a flexible back-end security model that supports a variety of potential situations. This typically involves token acquisition, mediation, credential mapping, and caching some information (not the original password or similarly sensitive information, however) about these to facilitate efficient transformation between the security models. In this case, the Kronos mobile app has its own end-to-end security mechanism, but we had to layer on an additional piece to satisfy the client’s information security standards.</p><p>The original Kronos Workforce Central deployment was purely an internal system that communicated with punch clocks at various company sites and provided a web application for administrators. When the mobile initiative came along, this internal system now had to be exposed to the Internet so that mobile devices could hit the API with which the mobile app interacts. Exposing the system to the Internet could be easily done using a reverse web proxy such as F5 APM to move the traffic into the internal network where the app servers are deployed. However, this particular organization is a large, security-conscious Fortune 100 organization whose Information Security department maintains a standing requirement for all Internet-facing systems. It states that no unauthenticated traffic is allowed into the internal corporate network. InfoSec needs the perimeter network (the DMZ) to pre-authenticate the user before any traffic enters the internal network segments. This provides the ability to enforce various access controls for users at the entry point of each network segment.</p><p>This problem had several possible solutions, including the following:</p><ul><li>Move the application server infrastructure to a DMZ. The client didn’t have the infrastructure to support moving Kronos infrastructure to the DMZ, however. It wasn’t a pattern that had been used before.</li><li>Deploy a reverse proxy or application security gateway technology such as F5 APM or TAMeb/iSAM in the DMZ. None of these technologies were available or had limited abilities to implement custom security integrations. Moreover, breaking the out-of-the-box UI is always a bad idea for COTS applications, especially when they are mobile applications that are built often. The last thing you want is a company-specific version leading to all sorts of support and backwards compatibility issues.</li><li>Grant a security exception and let the initial traffic enter the corporate network unauthenticated. I reject this option on principal; so did they.</li><li>Use a Service or API Gateway such as CA’s API Management solution, Apigee Edge, or WebSphere DataPower/IBM API Connect to integrate the proprietary API Security model that exists between the Kronos Mobile App and Kronos Workforce Central server with a standard enterprise security model that can provide the perimeter security required. This is the approach that was taken.</li></ul><p>The Integration and API Management layers this organization has deployed are shown in the diagram below. Each actor in the processing pipeline is required to authenticate all traffic that comes into it. The standard enterprise API security model uses JSON Web Tokens (JWT) to describe the end-user or system actor (it could be either, depending on the requirements) to propagate identity and address authentication at each layer. Authorization decisions for API requests are performed at the API Gateway in the cloud and/or at the internal ESB. All API traffic uses TLS v1.2.</p><p>The client had created a well-defined enterprise API Security model during the initial API Management project, one designed with flexibility and integration with COTS, third-parties, and SaaS solutions in mind. Plus, they were already an Apigee Edge Public Cloud and WebSphere DataPower customer. Apigee Edge Public Cloud acts as a cloud-hosted API Gateway that provides a common security model, statistics gathering, monitoring, common error handling, along with other benefits. WebSphere DataPower is acting as the traffic ingress point for API traffic to the internal network. Frankly, it is playing a similar role to what Apigee plays out in the cloud, but it is protecting the corporate datacenter.</p><p>Recent versions of Kronos support Azure Active Directory <a href="https://azure.microsoft.com/en-us/marketplace/partners/kronos-incorporated/kronos/">logins</a>. This organization is not yet on the latest and greatest; moreover, given the previous discussion about where the Kronos application servers are located on the network, this feature doesn’t get us any closer to the perimeter security requirement.</p><p>The Kronos Mobile application API traffic exchanges XML messages with the Workforce Central application server. This API is called the <a href="https://www.scribd.com/doc/65571304/Kronos-WFC-Developers-Toolkit-Reference-Timekeeping-v6">Workforce Central XML API</a>. It uses a proprietary login mechanism that puts the username and password in the XML message body of an API call. The Kronos Workforce Central application server can validate those credentials via an LDAP bind against an LDAP server or Active Directory; this organization does LDAP binds against Active Directory. After authentication and authorization of the mobile user’s session, a JSESSIONID cookie is returned to the mobile app. All subsequent API calls use this JSESSIONID cookie to track the security session.</p><p>The existing OAuth2-based Enterprise Security Model uses Azure Active Directory as the Identity Provider (IdP). All features of the OAuth2 spec that are supported by Azure Active Directory are available to us. Since this organization doesn’t replicate user passwords from the internal Active Directory (AD) to Azure Active Directory (AAD), it was necessary to use another federation server that does have access to the internal Active Directory domain and is federated with Azure Active Directory. The organization uses Active Directory Federation Server v2.0 as its on-premise federation solution for Web Application SSO and WS-Trust Security Token Service.</p><p>The set of steps involved in authenticating an end user are:</p><ol><li>Mobile app collects the end user credentials and sends them to the Kronos server via API gateway.</li><li>API Gateway intercepts the call, extracts the username and password, and in turn makes a ws-trust authentication request to the internal federation server. Now, the end user is authenticated at the gateway and no traffic has traversed the corporate network.</li><li>The Security Gateway and ESB validate the JWT token that was included (by Apigee) in the API request’s Authorization header to authenticate each request.</li><li>The call is let through to the Kronos server with the end user info, where it does the LDAP based login on the backend. (As far as the Kronos flow goes, we did not change a thing.)</li></ol><p>Given this set of technologies, we had to authenticate the user via a WS-Trust call to ADFS to obtain a SAML2 bearer token using the credentials extracted from the login API call. That SAML2 assertion was then placed into an OAuth2 call — as defined in the “Security Assertion Markup Language (SAML) 2.0 Profile for OAuth 2.0 Client Authentication and Authorization Grants” specification — to Azure Active Directory to obtain an OAuth2 access token (JWT) that could be used to interact with the other layers of the infrastructure.</p><p>Admittedly, this was more complex than it could’ve been given a different configuration; if the password hashes had been replicated to Azure Active Directory, then a resource authorization grant call could have been made directly to AAD. There are security implications and other considerations to putting your organization’s password hashes in AAD or any cloud-hosted, third-party IdP, however. The resource authorization grant is the appropriate choice in such a situation because it is a system component making these calls and this grant provides an non-interactive login. But, we were required to go the circuitous route to get the access token we needed to route API traffic through the rest of the infrastructure.</p><p>We decided to implement the perimeter security using the Apigee Edge API Gateway in the public cloud because the distributed cache it provides between its nodes (Message Processors, or MPs) allowed the solution to work when requests were routed to different MPs. The Apigee Edge caching facility was used to create an authentication cache that kept track of the OAuth2 access token that came from the OAuth2 call during the initial login. The cache key for the authentication cache was the JSESSIONID cookie value that the Kronos App server assigned to the mobile user’s session. During response processing of the initial login request, the API Gateway will have access to the JSESSIONID value and the access token that must be included in the request for downstream processing at the Security Gateway and ESB. The security session on the Kronos application server is only valid for five minutes, so the Apigee Edge authentication cache was also configured with a 300 second expiration timeout. Whether Apigee Edge or the Kronos App Server (or another layer) returns a 401 Not Authenticated error, the mobile app responds the same way, forcing the client to enter credentials again.</p><p>Once a JWT token has been obtained that describes the authenticated user and has the proper audience for the Kronos API, the role information (relevant to the Kronos application) is available to the authorization processing mechanism used on the API Gateway. If the user attempting to login does not have the proper role information, the API Gateway rejects the login attempt. Normally, Apigee Edge would return a 403 Not Authorized error, but analyzing the behavior of the Kronos API, we found that the mobile app client did not react properly to a 403 error or other return codes it wasn’t expecting. So, we changed all error responses generated by Apigee to be a 500 Internal Server Error.</p><p>Once the request has been properly authenticated and authorized at the API Gateway layer, the request is sent to the corporate data center. The Security Gateway in the DMZ uses transport layer security with client authentication and only allows the certificate belonging to the API Gateway to connect. This satisfies the perimeter security requirement.</p><p>By routing the API traffic through Apigee Edge, the standard statistics logging/reporting, monitoring, error handling, and other features that the API Gateway provides are available to the Kronos API traffic.</p><p>The technologies described in this post can be used to address a wide range of security integration needs. API Management solutions like Apigee Edge are a versatile tool that, when combined with the integration expertise and knowledge of information security practices, can produce elegant, low-cost solutions.</p><p>Do you have a custom or COTS mobile application that must be integrated with various backend components, each having their own security concerns that are not compatible out of the box? <a href="/service/mailto:hello@levvel.io">Contact us</a>.</p><p><em>For more on this topic, visit </em><a href="http://levvel.io/blog"><em>levvel.io/blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=691cd2aa300" width="1" height="1" alt=""><hr><p><a href="https://medium.com/levvel-consulting/by-robert-c-broeckelmann-jr-principal-consultant-at-levvel-691cd2aa300">API Management and Perimeter Security for COTS Applications</a> was originally published in <a href="https://medium.com/levvel-consulting">Levvel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>