<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gazal Gafoor on Medium]]></title>
        <description><![CDATA[Stories by Gazal Gafoor on Medium]]></description>
        <link>https://medium.com/@gazal.k?source=rss-27ac4324fcee------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*33PqEn_Ux1ggugqG.</url>
            <title>Stories by Gazal Gafoor on Medium</title>
            <link>https://medium.com/@gazal.k?source=rss-27ac4324fcee------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:51:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gazal.k/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bolstering Security & Automating Management of Target Australia’s EKS clusters]]></title>
            <link>https://medium.com/@gazal.k/bolstering-security-automating-management-of-target-australias-eks-clusters-705c6abf5b74?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/705c6abf5b74</guid>
            <category><![CDATA[karpenter]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[bottle-rocket]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 01:13:32 GMT</pubDate>
            <atom:updated>2023-07-13T01:17:30.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DA6FdmW--_USPbwLtKh45g.png" /></figure><p>To keep pace with Australia’s growing eCommerce market, Target needed to enhance its online offering by developing an engaging, reliable, and cost-effective eCommerce platform for consumers who expect personalised and seamless online shopping experiences. Using AWS, we have modernised the online platform at Target. (check the story of our cloud transformation journey <a href="https://medium.com/devops-dev/migrating-the-target-online-platform-to-aws-f490295ca29b">here</a>)</p><p>As of around 2017, Target’s online platform was composed primarily of the core eCommerce system, a search solution and a couple of middleware systems that helped integrate with other enterprise software systems like the merchandising system, supply chain systems, finance systems etc. At about that time we started “strangling” out capabilities into smaller software systems; microservices. We used a serverless pattern leveraging AWS Lambda, API Gateway etc to create these new microservices. And then in 2020, we modernised our long running workloads that don’t fit the Function as a Service mould. For those applications, which are long running, we decided to containerise them and use AWS Elastic Kubernetes Service (EKS) as the container orchestration platform. Since running all of the software systems that constitute the online commerce experience on AWS, we have been able to scale our applications to meet increased demand during sales periods and new product launches. Since our initial adoption of EKS, we have made significant progress in bolstering security and automating management of our clusters leveraging Bottlerocket, Fargate &amp; Karpenter.</p><h3>Enhancing the Container Host OS</h3><p>At Target, when we started using EKS, we were using EKS optimised Amazon Linux as the host OS for the worker nodes in our clusters. The base AMIs, provided by AWS, are built on top of Amazon Linux 2 (<a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html">https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html</a>). It is of course a general purpose OS, with plenty of binary dependencies that are unnecessary for the workload compute layer in an EKS cluster.</p><p>Here’s an approximation of the various layers in the host OS we use that is based on Amazon Linux.</p><figure><img alt="AL2 Layers" src="https://cdn-images-1.medium.com/max/571/1*4827zV9x3VHLk2PKromsqA.png" /></figure><p>And we were not using the provided EKS optimised AL2 AMIs as is, we were baking custom AMIs to ensure they were CIS hardened. And we put it through the CIS benchmark for Amazon Linux 2 which is perhaps not appropriate for usage within a container orchestrated environments. Some of the CIS hardening rules impacted k8s networking and we had to disable some of those to ensure our clusters were functional.</p><figure><img alt="AMI Bake Pipeline" src="https://cdn-images-1.medium.com/max/1024/1*cETmCjS74kSVlXXMIBnQIg.png" /></figure><p>Maintenance of the custom AMI pipeline started becoming more and more challenging. Sometime last year we tried to run newer m6i instances and something related to our customisation was preventing those instances from joining our clusters. We couldn’t quite figure out what it was, but it was a major hassle.</p><p>We wanted to get away from the chore of “baking” our own AMIs. We seemed to just add to the “bloat” of a general purpose OS with our custom “baking”.</p><h4>Bottlerocket OS</h4><figure><img alt="Bottlerocket OS" src="https://cdn-images-1.medium.com/max/1024/1*HW6XPDP_MfRx6vdq7kCLLg.png" /></figure><p>Since around end of 2020, when Bottlerocket became GA (or perhaps before), we had our eye on it. Bottlerocket was purpose built for running containers.</p><p>Compared to the Amazon Linux based host OS, Bottlerocket is so much leaner which meant a thin surface of attack.</p><figure><img alt="Bottlerocket Layers" src="https://cdn-images-1.medium.com/max/561/1*WAOh7XPyvIbl9_2LPzbn3Q.png" /></figure><p>It does not have the bells and whistles (in the context of container orchestration) associated with a general purpose OS, like compilers, package managers, SSH server, not even interactive shells.</p><ul><li>Instead of having a package manager, Bottlerocket uses an update mechanism with partition flips of the entire filesystem image. (more on that here: <a href="https://github.com/bottlerocket-os/bottlerocket#updates">https://github.com/bottlerocket-os/bottlerocket#updates</a>).</li><li>There is no need for an SSH server. To access hosts we can use the more secure AWS SSM Session Manager, which is part of a “control container”.</li><li>Instead of having interactive shells like bourne shell, bash, zsh etc, Bottlerocket’s “control container” can be used to introspect hosts and make API calls to manage hosts.</li></ul><p>Having fewer software components of course means fewer components that contribute to security vulnerabilities. Aside from having a very thin surface of attack, Bottlerocket was designed from the ground up to be security hardened. The following are some of the key security features of Bottlerocket:</p><ul><li>Automated security updates: A k8s operator is available for in-place updates to bottlerocket hosts.</li><li>Immutable root filesystem backed by <a href="https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/verity.html">dm-verify</a>: So, userspace processes cannot modify rootfs protecting against container escape vulnerabilities.</li><li>Bottlerocket uses <a href="https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt">tmpfs</a>, memory-backed filesystem, for /etc: So modification of system config is not supported. OS Config changes can be done through an API or through containers written to implement CNI or CSI.</li><li>No shell or interpreters installed: That’s no SSH nor python. So that’s something else we don’t need to worry about. For troubleshooting, an <a href="https://github.com/bottlerocket-os/bottlerocket-admin-container">admin container</a> (disabled by default) will have to be created that will provide shell access and has an Amazon Linux 2 base. The act of logging into an individual Bottlerocket instance is intended to be an infrequent operation for advanced debugging and troubleshooting</li><li>Executables built with hardening flags</li><li>SELinux enabled in enforcing mode: No option to disable it.</li></ul><h4>Rolling out Bottlerocket</h4><p>The lean nature of Bottlerocket is not without challenges. We had to make the following changes:</p><p>1. Used Container Storage Interface (CSI) drivers for any persistent storage used by our application workloads:</p><ul><li>For standard block storage, instead of the “in-tree” storage drivers, we started using <a href="https://github.com/kubernetes-sigs/aws-ebs-csi-driver">https://github.com/kubernetes-sigs/aws-ebs-csi-driver</a>.</li><li>For network shared block storage, we were using the now archived <a href="https://github.com/kubernetes-retired/external-storage/tree/master/aws/efs">efs-provisioner</a>. Bottlerocket, by design, does not have the binary dependencies to run it. We started using <a href="https://github.com/kubernetes-sigs/aws-efs-csi-driver">https://github.com/kubernetes-sigs/aws-efs-csi-driver</a>.</li></ul><p>2. Updated configuration of some auxiliary applications to support containerd as opposed to docker. (this would have had to happen on any host OS eventually with dockershim deprecation, but Bottlerocket for k8s defaulted to containerd ahead of that)</p><ul><li>Log forwarding configuration of our observability solution had to be tweaked.</li><li>Switched to the CRIO/ContainerD version of a DaemonSet for our Vulnerability Management agent (Cloud Workload Protection / CWP).</li></ul><p>3. Switched to an entirely new Endpoint Detection &amp; Response solution, one that is more cloud native. Using their eBPF based agent so that it can support Bottlerocket without having to build kernel modules.</p><p>4. We had some applications for which we needed to safe-list some networking related sysctl in kubelet config (<a href="https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/">https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/</a>). Bottlerocket has a different mechanism for settings using TOML syntax; <a href="https://github.com/bottlerocket-os/bottlerocket#settings">https://github.com/bottlerocket-os/bottlerocket#settings</a>.</p><h4><strong>Outcomes from switching to Bottlerocket</strong></h4><p><strong>Security</strong></p><p>A key metric for measuring how secure Bottlerocket is when compared to our previous host OS was to check vulnerabilities identified. We compared reports from our vulnerability management solution and found some very favourable results.</p><figure><img alt="Vulnerabilities comparison" src="https://cdn-images-1.medium.com/max/1024/1*aK0Ommo-jwHrPj7ApqmmGg.png" /></figure><p>With a simple weight multiplier applied to the vulnerabilities, *4 for Critical, *3 for High, *2 for Medium &amp; *1 for Low, our vulnerability score used to be a big fat <strong>804</strong>. With Bottlerocket, we brought it down to just <strong>3</strong>.</p><p><strong>Performance​</strong></p><p>We did expect improvement in performance, but the results from our performance tests significantly exceeded our expectations. Some rejigging of security tooling may also have contributed to the results, especially the new eBPF based EDR agent.</p><ul><li>30.6–41.7 % response time reduction in customer facing applications.​</li><li>33 % speedup in cart calculation logic.​</li><li>Increased order placement throughput.​</li><li>16.2 % response time speedup in store employee facing applications.​​</li></ul><p><strong>Cost</strong>​</p><ul><li>40 % reduction in compute capacity requirement.​</li></ul><p>This lowered our AWS infra cost, especially during peak trade.​</p><p><strong>Stability</strong>​</p><p>We used to observe hosts becoming unhealthy, sporadically.​ We suspected that the causes for instability were​:</p><ul><li>Using the incorrect CIS benchmark for baking the Amazon Linux 2 AMIs.​</li><li>Legacy security tooling that did not support container orchestration environments.​</li></ul><p>We have not observed these issues since rolling out Bottlerocket.​</p><h3>Uplifting Compute Provisioning</h3><p>For provisioning / scaling compute in EKS clusters, we, like most, were using cluster-autoscaler (CA) <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler">https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler</a>. The provisioning mechanism can be best described as:</p><ul><li>Watch for any pending, non schedulable Pods.</li><li>If there are any, interact with one of the AutoScaling Groups (ASG) / node-groups associated with the cluster to increase its size.</li><li>Kubernetes scheduler (part of the control plane) assigns the pending Pods to the newly provisioned Node (EC2) once it joins the cluster.</li></ul><figure><img alt="How Cluster Autoscaler Provisions Compute" src="https://cdn-images-1.medium.com/max/655/1*67sCSV55HFIkCmVQgqj3iw.png" /></figure><h4>Limitations with Cluster Autoscaler</h4><p>Cluster Autoscaler works quite well for the most part, but has some limitations.</p><p><strong>Cumbersome security updates</strong></p><p>The update process for our nodes were tricky, because they are part of an ASG. The patching strategy was to bake a new AMI with all the updates, then create a new ASG and delete the old one.</p><figure><img alt="Update strategy with ASGs / node-groups" src="https://cdn-images-1.medium.com/max/752/1*gwNBXkJryND0pm0PWMDgmw.png" /></figure><p>With bottlerocket, there is also the option of running the <a href="https://github.com/bottlerocket-os/bottlerocket-update-operator">https://github.com/bottlerocket-os/bottlerocket-update-operator</a> that supports in-place updates. But the in-place updates mean that any new instances that are spun up, would be instantiated using the version specified in the ASG before being updated by the operator.</p><p><strong>Nodes within a node-group must be homogeneous</strong></p><p>Cluster Autoscaler does not support heterogenous nodes in a node-group. Say, nodes with 8 vCPUs and 4 vCPUs in the same node-group / ASG.</p><p><strong>Not Availability Zone aware</strong></p><p>We would need node-groups per availability zone (AZ) if we have applications with AZ affinity. e.g., if we have applications that use EBS volumes.</p><p><strong>Nondeterministic spread of applications across AZs</strong></p><p>The node groups themselves can be spread across availability zones, but without significant additional configuration, applications need not be spread across availability zone. In the example below we have nodes across 3 AZs, but an application (in yellow) incidentally in just a single AZ. An AZ outage can cause some disruption to such an application.</p><figure><img alt="Non deterministic spread of application with cluster autoscaler" src="https://cdn-images-1.medium.com/max/971/1*SVaS_FUtMEwrYckSB0ZFGQ.png" /></figure><p><strong>Instance distribution / capacity type configuration for cost optimisation applies more broadly</strong></p><p>Here is an example ClusterConfig (for use with <a href="https://eksctl.io/">eksctl</a> of-course).</p><pre>apiVersion: eksctl.io/v1alpha5<br>kind: ClusterConfig<br><br>metadata:<br>  name: test<br>  version: &quot;1.24&quot;<br><br>nodeGroups:<br>- name: node-group-1<br>  privateNetworking: true<br>  amiFamily: Bottlerocket<br>  minSize: 1<br>  maxSize: 10<br>  instancesDistribution:<br>    instanceTypes:<br>    - m6i.2xlarge<br>    - m5.2xlarge<br>    - m5a.2xlarge<br>    - m5ad.2xlarge<br>    - m5d.2xlarge<br>    onDemandBaseCapacity: 3<br>    onDemandPercentageAboveBaseCapacity: 25</pre><p>The instance distribution applies across workloads. To get more granular control, like having specific applications use on-demand capacity instead of spot will require more complex config using labels and taints. And to fallback to using on-demand capacity when spot capacity is limited or unavailable, we would need separate ASGs for spot and on-demand capacity as opposed to mixing them.</p><h4>Karpenter</h4><figure><img alt="Karpenter" src="https://cdn-images-1.medium.com/max/318/1*G0lJGIezR1wN0Ex_YH5GCA.png" /></figure><p>When Karpenter became generally available, we found the solution to most of the limitations we were observing with Cluster Autoscaler. The concept of “groupless” worker nodes seemed perfect. Roughly, here is what compute provisioning with Karpenter looks like:</p><figure><img alt="How Karpenter Provisions Compute" src="https://cdn-images-1.medium.com/max/660/1*6VCLVIRzgMdT8iQGjUldSg.png" /></figure><p>Instead of interacting with ASG APIs, Karpenter interacts with EC2 APIs. In practice this has meant faster compute provisioning for Target’s EKS clusters.</p><h4>How we adopted Karpenter</h4><p>There’s an excellent document on Karpenter’s official pages for migrating from Cluster Autoscaler to Karpenter: <a href="https://karpenter.sh/docs/getting-started/migrating-from-cas/">Migrating from Cluster Autoscaler</a>. Here is what we did:</p><p><strong>Prerequisite infrastructure</strong></p><p>There were some AWS resources we needed to create before actually installing Karpenter on our clusters.</p><ul><li>Node IAM Role and Instance Profile.​</li><li>SQS queue that subscribes to EC2 event bridge rules for spot interruption, health, rebalance and instance state changes.​</li><li>IAM Role for Service Account (IRSA) for Karpenter controller.​</li><li>Fargate profile for running Karpenter controller.​</li></ul><p>For this prerequisite infrastructure, we added a CDK stack. We like how concise and testable our infrastructure as code is with CDK.</p><p><strong>Installing Karpenter</strong></p><ul><li>Add the Node IAM Role we created for Karpenter to the aws-auth ConfigMap. We have a custom pipeline to update this ConfigMap in an idempotent manner. [We are closely following an issue​ on the AWS Containers Roadmap which should make this a lot easier: <a href="https://github.com/aws/containers-roadmap/issues/185">https://github.com/aws/containers-roadmap/issues/185</a>]</li><li>Install Karpenter controller using helm charts.​ (there are 2 charts, a main chart for Karpenter and another for managing the CRDs)</li><li>Install Provisioners (+ AWSNodeTemplate)​. Right before this step would be a good time to uninstall Cluster Autoscaler.</li><li>Delete the node groups.​ Using eksctl will help ensure the old nodes are cordoned and drained allowing for a disruption free migration, if your applications have appropriate Pod Disruption Budgets (<a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/">https://kubernetes.io/docs/tasks/run-application/configure-pdb/</a>).</li></ul><h4>Fargate Usage</h4><figure><img alt="Fargate" src="https://cdn-images-1.medium.com/max/761/1*tmHdftlLvJzW2VYV6ftpCQ.png" /></figure><p>Yes, we do use Fargate to run Karpenter controller itself. How the compute for the compute provisioner (Karpenter controller) is provisioned is something of a chicken and egg problem.</p><p>The other option was to create a node-group just for the Karpenter controller. This seemed counter intuitive to us. This would be a very simple fixed size ASG, but we would have still had to use our old update strategy on this ASG. Fargate seemed like an excellent fit because host updates are automated along with provisioning. So, having a Fargate profile for Karpenter controller (selecting karpenter namespace) seemed like a good solution to us, until / hopefully Karpenter comes out of the box in the EKS control plane; <a href="https://github.com/aws/containers-roadmap/issues/1792">https://github.com/aws/containers-roadmap/issues/1792</a>.</p><h4>Outcomes from adopting Karpenter</h4><p><strong>Security</strong></p><p>We have automated security updates to the host OS using Karpenter. We leverage the ttlSecondsUntilExpired setting provided by Karpenter for upgrading nodes. And since we use Bottlerocket and not a custom AMI, Karpenter uses the latest version of Bottlerocket when new nodes are provisioned. So, new nodes provisioned to scale out and / or to replace existing nodes that expire would be running a newer host OS with security patches.</p><p>Be careful with the ttlSecondsUntilExpired setting though. Make sure your applications have the right Pod Disruption Budgets; <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/">https://kubernetes.io/docs/tasks/run-application/configure-pdb/</a></p><p><strong>Cost</strong></p><p>With Karpenter we get much more reliable mix of spot and on-demand capacity without too much trouble. Before, we were setting base on-demand capacity to ensure some applications were not disrupted. Now we can either use nodeAffinity or nodeSelector for applications that need on-demand capacity. Here is an example:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      ...<br>      nodeSelector:<br>        karpenter.sh/capacity-type: on-demand</pre><p>So, most of our clusters use just spot instances now.</p><p>Karpenter also has a consolidation feature to achieve better bin packing and hence reduce unused compute capacity.</p><pre>apiVersion: karpenter.sh/v1alpha5<br>kind: Provisioner<br>metadata:<br>  name: default<br>spec:<br>  consolidation:<br>    enabled: true<br>...</pre><p>There was a demonstration of this feature at AWS re:Invent 2022. The demonstration used <a href="https://github.com/awslabs/eks-node-viewer">https://github.com/awslabs/eks-node-viewer</a> to measure and showcase the impact of consolidation. Here’s a link to that demonstration:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5B4-s_ivn1o%3Fstart%3D2173%26feature%3Doembed%26start%3D2173&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5B4-s_ivn1o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5B4-s_ivn1o%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef1f5fc6e44793998bc20faca8d26e65/href">https://medium.com/media/ef1f5fc6e44793998bc20faca8d26e65/href</a></iframe><p><strong>Availability</strong></p><p>With Karpenter, we can also easily configure our applications to achieve more deterministic spread across availability zones (AZ). Here is a sample Deployment with the right topologySpreadConstraints:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      ...<br>      topologySpreadConstraints:<br>      - labelSelector:<br>          matchLabels:<br>            app: my-app<br>        maxSkew: 1<br>        topologyKey: topology.kubernetes.io/zone<br>        whenUnsatisfiable: ScheduleAnyway</pre><p>We also don’t need the added complexity of multiple ASGs / node-groups per AZ to ensure any applications that use EBS volumes are scheduled.</p><p><strong>Flexibility</strong></p><p>Before Karpenter, we used to carefully plan out the EC2 instance types we used in our clusters to ensure our applications “fit” well in our node groups. Now, we let Karpenter to do all the math to find the right sized nodes / EC2s.</p><pre>apiVersion: karpenter.sh/v1alpha5<br>kind: Provisioner<br>metadata:<br>  name: default<br>spec:<br>  requirements:<br>  - key: karpenter.k8s.aws/instance-category<br>    operator: In<br>    values:<br>    - c<br>    - m<br>    - r<br>    - t<br>  - key: karpenter.k8s.aws/instance-cpu<br>    operator: Gt<br>    values:<br>    - &quot;3&quot;<br>  - key: karpenter.k8s.aws/instance-cpu<br>    operator: Lt<br>    values:<br>    - &quot;9&quot;<br>  - key: karpenter.k8s.aws/instance-hypervisor<br>    operator: In<br>    values:<br>    - nitro<br>...</pre><p>Another benefit is that we don’t need a separate DaemonSet or Deployment for interruption handing anymore. We were using <a href="https://github.com/aws/aws-node-termination-handler">https://github.com/aws/aws-node-termination-handler</a> before, but Karpenter incorporates interruption handling too, if you configure an SQS queue for it; <a href="https://karpenter.sh/v0.29/concepts/deprovisioning/#interruption">https://karpenter.sh/v0.29/concepts/deprovisioning/#interruption</a>.</p><h3>Conclusion</h3><p>Amazon Elastic Kubernetes Service makes it easier for organisations to adopt Kubernetes. And adopting Bottlerocket, Fargate &amp; Karpenter has helped Target Australia bolster our security posture and simplify our usage of Kubernetes. We hope that sharing this account of our experience would be useful to the community.</p><p><em>The views and opinions expressed in this document are solely those of the original author. These views and opinions do not necessarily represent those of Target Australia.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=705c6abf5b74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling for PS5 sales - The Target eCommerce way]]></title>
            <link>https://medium.com/@gazal.k/scaling-for-ps5-sales-the-target-ecommerce-way-dbd36c52e02e?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbd36c52e02e</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Wed, 15 Jun 2022 07:29:41 GMT</pubDate>
            <atom:updated>2022-06-15T07:29:41.424Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wo0REOhk8eebGoireSjgZA.png" /></figure><p>Since Oct 2020, the Target eCommerce Platform has been running on AWS (check the story of our cloud transformation journey <a href="https://medium.com/@gazal.k/migrating-the-target-online-platform-to-aws-f490295ca29b">here</a>). We use EKS to manage our containerised applications. This gives us the capability to scale on demand with quality of service.</p><p>But as with anything in technology (or in life), there’s always room for improvement. Target Online sells the elusive PS5 from time to time.</p><p><a href="https://www.target.com.au/playstation-5">PlayStation 5 | Target Australia</a></p><p>Considering the stock shortages for PS5; <a href="https://www.makeuseof.com/ps5-shortage-continues-2022/">https://www.makeuseof.com/ps5-shortage-continues-2022/</a>, whenever it is back in stock, it is met with enthusiastic demand. In 2021 we observed that although Target’s Online platform can keep up with organic spikes in traffic, it wasn’t quite enough to keep up with the kind of surge to sell PS5s.</p><p>We made a number of changes to help our customers have a seamless experience when buying those PS5s online.</p><h4>Health check changes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*3JE5GLl0wwdZ3fH84g9A6w.png" /></figure><p>Kubernetes helps us ensure that application instances (Pods) stay healthy. To do this, we get to define probes. The two main ones are:</p><ul><li>Readiness Probe: used to check if a Pod is ready to serve traffic.</li><li>Liveness Probe: used to check if a Pod is alive.</li></ul><p>Readiness Probe is essentially the load balancer health check. It is what determines which Pods are healthy to serve traffic. Ideally Pods become ‘ready’ after their usual startup time and may occasionally become ‘unready’ if they are overwhelmed by traffic. If that happens the load balancer suspends traffic to unready/unhealthy Pods until they become healthy again. The temporary suspension of traffic gives those Pods a breather.</p><p>Liveness Probe on the other hand is used to detect if application Pods have gone into a faulty state. At this point, Kubernetes (kubelet) will restart the faulty Pod to try to restore the application to a healthy state.</p><p>What was hurting us was that our Liveness Probe configuration was a bit too eager. During huge surges in traffic, this is what was happening:</p><ul><li>Increase in traffic meant more load on Pods currently serving traffic.</li><li>Horizontal Pod Autoscaler (<a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a>) triggers the startup of more Pods to serve traffic.</li><li>While new Pods were starting up, existing Pods (due to the eager liveness probe setting) were being detected as faulty and were being restarted.</li><li>This means even more load on remaining Pods leading to a cascade of failures.</li></ul><p>So instead of quickly scaling to meet demand, we were shooting ourselves in the foot and restarting existing Pods.</p><p>Here’s an excellent blog post that we stumbled upon that explains the problem in more detail: <a href="https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/">https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/</a></p><h4>Database Optimisations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*GGUjhAqXuFKf3jopRu8l7A.png" /></figure><p>Before we moved to AWS, we were using SQL Server RDBMS on-prem. On AWS we use Aurora (the MySQL compatible variant). On any database, tuning your SQL and using the right indexes can significantly reduce the load exerted by your application. But something we realised the hard way was that Aurora (MySQL) was more unforgiving of suboptimal SQL.</p><p>We identified suboptimal SQLs using our APM and using <a href="https://aws.amazon.com/rds/performance-insights/">AWS RDS Performance Insights</a>. We added the right indexes for some and refactored others to make them more performant.</p><p>Another bottleneck that we had to resolve was around SYNCH wait events. This AWS reference: <a href="https://aws.amazon.com/premiumsupport/knowledge-center/aurora-mysql-synch-wait-events/">https://aws.amazon.com/premiumsupport/knowledge-center/aurora-mysql-synch-wait-events/</a> helped us identify the parameter to tweak. We bumped up the setting for innodb_sync_array_size.</p><h4>Non Blocking IO</h4><p>Our storefront application uses <a href="https://www.baeldung.com/jgroups">jGroups</a> for messaging between Pods. We were using TCP for transport protocol and <a href="https://github.com/jgroups-extras/jgroups-kubernetes/">KUBE_PING</a> for discovery protocol. Regular TCP was the blocking IO variant. We observed that this was causing thread locking issues. The number of busy threads were hitting the limit (200) across multiple Pods when this happens and they all go into ‘faulty’ status resulting in restarts.</p><p>We decided to switch to jGroups TCP_NIO2 for transport. Along with this we also changed the tomcat connector from NIO(1) to NIO2. This resulted in average busy threads in the application going down from ~10 to 0 and more significantly, we almost never see the busy threads hit the limit anymore. This means the application is more stable in general, especially during traffic surges.</p><h4>Pre-Warming</h4><p>Finally, this is an operational process change we have implemented to ensure a smooth PS5 sale event. Although Target’s Online platform has amazing scalability now, it still unfortunately has limits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZE2nn4QNat2jAjzH4ZSEeA.png" /></figure><p>The above graph shows how the traffic surged during a recent PS5 sale event. And this was one of the quieter ones because it was held at around 07:30. The traffic increased by about 500%. Some team members have joked that it is similar to a DDoS attack :).</p><p>What we do now to deal with this is, just before the sale we increase the minimum number of Pods in anticipation of the traffic surge.</p><h4>Conclusion</h4><p>We have come a long way to ensure our customers have a great experience buying PS5s when they come back in stock. The only reason they should be disappointed now is if they were too late to checkout. And PS5s sell like hot cakes, they are gone within about 5 minutes.</p><p>We won’t stop here. There’s always room for improvement.</p><p><em>The views and opinions expressed in this document are solely those of the original author. These views and opinions do not necessarily represent those of Target Australia.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbd36c52e02e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Target eCommerce Platform — our Cloud Transformation]]></title>
            <link>https://blog.devops.dev/migrating-the-target-online-platform-to-aws-f490295ca29b?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/f490295ca29b</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Tue, 31 May 2022 11:38:55 GMT</pubDate>
            <atom:updated>2022-06-14T08:02:57.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cB7YU0H6jkuYssQ5" /></figure><p>We migrated our core online applications to AWS in 2020. I say core, because we already had some cloud native micro-services on AWS.</p><h3>Before</h3><p>We were running long running workloads in the online platform on virtual and some bare metal machines in private data centres. Here’s a simplified view of how things were in our private data centres:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*INWj9IPbkzXfVkqOgHzwbw.png" /></figure><p>Some of the constraints we had:</p><ul><li>Fixed / non-scalable compute resources.</li><li>Fixed / non-scalable database servers.</li><li>Legacy load-balancers with limited capability for automation.</li><li>Legacy firewalls with limited capability for automation.</li><li>Ageing hardware.</li></ul><p>This is by no means a generalisation of private data centres, but the situation as it was for Target’s online platform infrastructure.</p><h4>Development &amp; Continuous Integration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNV_Ki2VuuyxX0MH-4tzMw.png" /></figure><p>Our continuous integration also had limitations before we moved to AWS.</p><ul><li>Did not have easily replicable test db instance.</li><li>Did not have an easily replicable app build instance.</li><li>PRs were built one by one from a queue.</li></ul><p>This meant if there were a lot of PRs on the same application, developers often had to wait hours before they find out whether all tests and code quality checks pass for their PR.</p><h4>Continuous Deployment &amp; Releases</h4><p>Our test environments did not have high availability, so deployment of new application versions would cause disruption. So, deployments to our test environments were scheduled to execute overnight. If there were urgent merges requiring “midday” deployments, we had to check with all of the team and announce that there would be a disruption.</p><p>Deployment to production was also not easy and involved a lot of manual steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgXIPY4fQITfyCJ8F2522A.png" /></figure><p>Since the steps had to be done server by server, the whole process used to take around 6 hours.</p><p>On to the good stuff now. Let’s look at how we transformed our infrastructure and application deployment process.</p><h3>To The Cloud!</h3><p>The first step in migrating to AWS was changing how/what is deployed. We knew that the old “copying build artefacts as compressed archives” is not the way to go. We’d been tinkering with containerising our applications for a while.</p><h4>Containerisation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*P5yEoYLJgAhfTliKZZrmzw.png" /></figure><p>Containerisation is of course a very lightweight form of virtualisation. With containers, we get pretty much all of the benefits of virtualisation, such as:</p><ul><li>partitioning of hardware resources</li><li>isolation</li><li>encapsulation</li></ul><p>but without the overhead of guest OSs as with hypervisor based virtualisation.</p><p>And containerisation of course became really simple with Docker. We can define our applications as container images that encapsulate our app binaries along with all of its dependencies. And all that in a neat little text file; <em>Dockerfile</em>.</p><ul><li>For our UI applications we create a container image that starts with the base image of a web server and we copy in the UI assets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*utq3Y7tKlJ9UKccTCH3fbQ.png" /></figure><ul><li>For our Java applications we start with a base image with the right JVM and add any other libraries we may need and then our application build artefacts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*qftNmgqwneI58IFyGPCRnA.png" /></figure><p>In short, our applications are no longer just compressed archives o that we copy to a server to run in. Instead we are defining the “server” itself that runs the application process within it.</p><p>So, now that we have made neat little containers of our applications, we need a way to “orchestrate” it.</p><h4>Container Orchestration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*SPmT04VmQveuVZfOAOqM-Q.png" /></figure><p>Running our applications as containers brings a lot to the table. But we need something to manage the environment in which our application containers execute. This becomes especially relevant when you have a lot of application containers and you want to manage their resource allocation, lifecycle, scale them on demand etc. That’s what container orchestration is for. We chose Amazon Elastic Kubernetes Service (EKS).</p><p>Some of the capabilities of EKS, or Kubernetes(k8s) in general are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nk4c8r_9ILlWUBmD" /></figure><ul><li>Efficient capacity utilisation: k8s ensures our compute resources are efficiently utilised.</li><li>Scalability: with k8s there’s declarative ways to scale our applications to adapt to demand.</li><li>Keep applications healthy: k8s provides a very resilient platform. It ensures applications keep running and stay healthy.</li><li>And best of all, the k8s application metadata can be expressed as readable yaml files in source code.</li></ul><p>There’s a really nice comic illustration of what kubernetes is and how it works: <a href="https://cloud.google.com/kubernetes-engine/kubernetes-comic/">https://cloud.google.com/kubernetes-engine/kubernetes-comic/</a>.</p><h4>Kubernetes Ecosystem</h4><p>The open source nature of Kubernetes and the community around it has lead to a plethora of tools to extend and compliment it. Some of those tools that we use are:</p><ul><li><a href="https://helm.sh/">Helm</a>: a package manager for k8s. All our applications are installed and managed using helm. Also other applications we need in our k8s cluster, like: <a href="https://github.com/kubernetes-sigs/metrics-server">metrics-server</a>, <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler">cluster-autoscaler</a>, <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/">aws-load-balancer-controller</a>, <a href="https://github.com/kubernetes-sigs/external-dns">external-dns</a>, etc.</li><li><a href="https://github.com/roboll/helmfile">Helmfile</a>: a spec to manage helm packages on our clusters.</li><li><a href="https://cloud.google.com/tekton">Tekton</a>: A k8s native framework for CI/CD.</li><li><a href="https://jenkins-x.io/">Jenkins X</a>: Uses the above tools to provide <a href="https://www.cloudbees.com/gitops/what-is-gitops">GitOps</a> based CI/CD.</li></ul><h3>Target Online Platform on AWS</h3><p>The environment looks quite different from how it did on-prem. We now use the following:</p><ul><li><a href="https://aws.amazon.com/elasticloadbalancing/application-load-balancer/">ALB </a>for load balancing &amp; layer 7 routing.</li><li><a href="https://aws.amazon.com/eks/">EKS</a> to manage our containerised applications.</li><li><a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora RDS cluster</a> instead of MS SQL Server.</li><li><a href="https://aws.amazon.com/elasticache/redis/">ElastiCache</a>(redis) for session storage.</li><li><a href="https://aws.amazon.com/s3/">AWS S3</a> for media assets storage.</li></ul><p>Our application workloads are now <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Pods</a>, managed using other higher level k8s resources such as <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployments</a>, <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Services</a> etc. To scale on demand we use <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">horizontal pod autoscaler</a> and <a href="https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html">cluster autoscaler</a>. To integrate our applications with AWS load balancers, we use <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> resources.</p><p>Here’s a simplified view of what the online platform looks like on AWS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5yKJMEOhwW8eguVob-Tug.png" /></figure><h3>Continuous Integration &amp; Deployment</h3><p>Using Jenkins X, we’ve also transformed how we deploy our applications.</p><p>For each application, we have webhook based triggers for Pull Requests and when code gets merged into main branch of the application.</p><ul><li>For PRs, we run tests and code quality analysis to ensure that each change when merged to main would be production ready.</li><li>When a PR is merged into main, the release pipeline is triggered.</li></ul><h4>GitOps</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nryXaqiCd9vXlp2cxbAROw.png" /></figure><p>The release pipeline for each application does the following:</p><ul><li>Create the container image for the application and push it to <a href="https://aws.amazon.com/ecr/">AWS Elastic Container Registry</a>.</li><li>Create a helm chart and publish it to <a href="https://chartmuseum.com/">ChartMuseum</a>.</li><li>Promote the new release to GitOps based environments.</li><li>Every environment is defined using a git repository. (duh!)</li><li>The test environments would be defined as what is called “Auto” promotion environments. Every application release would get automatically promoted to such environments.</li><li>The higher environments, like the ones for User Acceptance testing and the Production environment itself are defined as “Manual” environments. So new releases won’t be promoted automatically through the release pipeline of an application to those environments. Instead we need to update the git repository for those environments deliberately as part of a production release cycle.</li></ul><p>The promotion process itself, either “Auto” or “Manual” is essentially about applying a set of application references to our Kubernetes cluster.</p><p>The environment repositories use <a href="https://github.com/roboll/helmfile">helmfiles</a> like the following to declaratively specify applications:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/074cae50113626bbe2e4c000a3513181/href">https://medium.com/media/074cae50113626bbe2e4c000a3513181/href</a></iframe><p>An application called <a href="https://github.com/jenkins-x/jx-git-operator">jx-git-operator</a> polls the GitOps repository for a particular environment and syncs the versions of applications as specified.</p><h4>Continuous Deployment Transformation</h4><p>What all of this enables is a very powerful, yet simple process to manage how our applications are deployed. Almost every aspect of deploying a new version of the application is automated.</p><p>On-prem, we had to disconnect servers from load balancers to deploy a new version, start it back up and then reconnect it to load balancers. Kubernetes automates all this for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLchVFGvp9fFeTK3XWa5mA.png" /></figure><ul><li>New application pods are started alongside old ones.</li><li>The old ones still serve users.</li><li>Once the new pods are “Ready” to serve users, the load balancer automatically and seamlessly diverts traffic to the new pods.</li><li>In the off chance the new application has some issue, they would not get “Ready”, so the old pods continue serving traffic and the new version is rolled back.</li></ul><p>On-prem, we also had to drain customer sessions. We don’t need to worry about this anymore because we now use an AWS ElastiCache (redis) cluster to store customer sessions. So we can update our application, scale it up or down without losing customer sessions. Here’s a story detailing how we use the session store: <a href="https://medium.com/swlh/use-redis-as-a-session-store-for-hybris-commerce-29bea8d498f1">https://medium.com/swlh/use-redis-as-a-session-store-for-hybris-commerce-29bea8d498f1</a></p><p>Our storefront application is built on top of SAP Commerce and needs to run a “typesystem” update before we release a new version. We automate this using <a href="https://helm.sh/docs/topics/charts_hooks/">Helm Chart Hooks</a>. Here’s a little write-up delving into more detail about this: <a href="https://medium.com/@gazal.k/hybris-type-system-update-on-kubernetes-791d75e94917">https://medium.com/@gazal.k/hybris-type-system-update-on-kubernetes-791d75e94917</a></p><p>Updating binary dependencies are taken care of as part of the application container images.</p><p>Updating other infrastructure settings such as, say load balancer settings, is also managed in code as k8s resources in application helm charts.</p><p>The deployment process that used to take about 6 hours is now just a matter of:</p><ul><li>Create a PR to GitOps repository for prod.</li><li>Pass automatic verification (<a href="https://helm.sh/docs/helm/helm_lint/">helmfile lint</a>).</li><li>Get approvals from the team.</li><li>Merge</li></ul><h3>Conclusion</h3><p>Before the migration to AWS, we had to use mechanisms like virtual waiting rooms for Black Friday to ensure the high traffic did not overwhelm the platform. We have modernised the platform, made it scalable and resilient to ensure our customers have a great shopping experience online.</p><p><em>The views and opinions expressed in this document are solely those of the original author. These views and opinions do not necessarily represent those of Target Australia.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f490295ca29b" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/migrating-the-target-online-platform-to-aws-f490295ca29b">Target eCommerce Platform — our Cloud Transformation</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Journey with Jenkins X]]></title>
            <link>https://medium.com/@gazal.k/our-journey-with-jenkins-x-f1f51c462c3e?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1f51c462c3e</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[jenkins-x]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[tekton]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Wed, 20 May 2020 01:47:56 GMT</pubDate>
            <atom:updated>2020-05-20T01:47:56.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K9zesREkDaIMHdKkfQ4p6g.png" /></figure><p>A couple of years back we began to re-evaluate our CI/CD tooling. We were using good old Jenkins. And we weren’t even using Jenkinsfile for a lot of our jobs.</p><p>Some of the problems we had with our existing tooling was:</p><ul><li>Our build jobs were not defined using code.</li><li>We had to manage all the pre-requisites for our Jobs; JDK installations, NodeJS installations, git, artifact repositories etc.</li><li>We did not have adequate mechanisms to scale out the build infrastructure.</li></ul><p>Some of us wanted to get away from Jenkins altogether because it seemed to carry a lot of legacy baggage. But it also had it’s benefits of having been around so long. We liked some of the newer tooling out there, especially those that leveraged containers for build execution. We looked at quite a few alternatives:</p><ul><li>Bitbucket Pipelines</li><li>AWS CodeBuild/CodeDeploy etc.</li><li>Circle CI</li></ul><p>This is when one of us stumbled upon Jenkins X. We liked it from the get go because it seemed to use Jenkins(although this would change soon) but re-purposed it in a way that was very new, cloud-native some would say.</p><h3>Using Jenkins X for just CI</h3><p>We started on Jenkins X during the very early days of the project. They offered two variants:</p><ul><li>Jenkins Static Master</li><li>Server-less</li></ul><p>At the time I think the server-less option was using Prow as the pipeline execution engine.</p><p>We used Jenkins X with Jenkins static master. We tried how it works for our Continuous Integration and the results were an obvious improvement over what we had. Since it was using Jenkins under the hood, it was a pretty easy transition as well. The team was already comfortable with how to check build logs, test reports etc on Jenkins. I’d go as far as to suggest that some of us probably didn’t even notice the change in tooling.</p><p>What we essentially did was:</p><ul><li>Created an AWS EKS Cluster.</li><li>Installed Jenkins X using <a href="https://jenkins-x.io/docs/guides/managing-jx/old/install-on-cluster/#installing-jenkins-x-on-a-cloud">jx install</a> (this is deprecated now, but was the only option at the time)</li><li>While installing the wizard prompts whether we need just CI capability or CI + CD. Although we intended on leveraging only CI at the time, we chose CI + CD to keep an eye on what CD on JX looks like.</li><li>We ran <a href="https://jenkins-x.io/commands/jx_import/">jx import</a> to generate Jenkinsfile, Dockerfile and <a href="https://helm.sh/">helm</a> charts for our applications.</li><li>We commented out the steps for CD.</li></ul><p>We couldn’t start using Jenkins X for Continuous Deployments though. This was because JX was very much laser focused on Kubernetes applications and our applications, well, they weren’t deployed on Kubernetes (yet 😃).</p><h3>CD with Jenkins X</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*_zXtG67th6jZrLNAOP728w.png" /></figure><p>After a while of using Jenkins X just for CI, a long proposed cloud migration project started picking up within our organisation. Jenkins X wasn’t just another CI/CD platform, it thrust us into the Kubernetes ecosystem. The fact that it used Kubernetes as the infra platform on which it was installed meant that we were getting familiar with K8s and it’s ecosystem through Jenkins X.</p><p>So when it came time to migrate our platform to the cloud, Kubernetes was the obvious choice.</p><p>K8s and it’s ecosystem can be quite overwhelming at first. What JX seems to do for that is; sort of wade through the K8s ecosystem and present an opinionated collection of K8s tools that we can leverage.</p><p>So, we started migrating our applications to K8s. It was very much powered by Jenkins X. We already had the CD steps(commented), Dockerfile and helm charts although we weren’t using them. We went ahead and started customising them for our applications and that formed the foundation of our migration.</p><h3>Jenkins X Evolution</h3><p>While we were using Jenkins X, installed using jx install and using a static master Jenkins, JX was evolving at a rapid pace.</p><ul><li>the installation of JX itself started leveraging GitOps principles; <a href="https://jenkins-x.io/docs/getting-started/setup/boot/">https://jenkins-x.io/docs/getting-started/setup/boot/</a></li><li>the old jx install and related CLI commands were getting deprecated.</li><li>Using static master Jenkins itself was getting the axe.</li><li><a href="https://cloud.google.com/tekton">Tekton</a> was becoming the preferred pipeline execution engine.</li><li><a href="https://jenkins-x.io/docs/reference/components/lighthouse/">Lighthouse</a> was becoming the preferred webhook &amp; <a href="https://jenkins-x.io/docs/guides/using-jx/faq/chatops/">ChatOps</a> handler.</li><li><a href="https://github.com/jenkins-x/bucketrepo">bucketrepo</a> is getting built to replace <a href="https://github.com/jenkins-x-charts/nexus">nexus</a> and <a href="https://github.com/helm/charts/tree/master/stable/chartmuseum">chartmuseum</a>.</li></ul><h4>Adopting Tekton</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgBhdNtesjsw6DD2IT05Vw.png" /></figure><p>The first thing we had to deal with was to ditch static Jenkins and start using Tekton. There were a few hurdles that prevented us from easily adopting Tekton:</p><ul><li>At least one of our applications needed to run a Database as a sidecar container for running tests during CI.</li><li>The same application, which is the core of our platform, produced build output outside of the source directory. With static Jenkins this was OK because the whole pipeline was executed essentially by one container and a few sidecars (one for JNLP and more for any DB sidecars). With tekton, each step was a separate container and the only data that persists between steps would be in attached volumes.</li></ul><p>We did work our way around both hurdles. For the sidecar container issue, what we did was to create a composite container, i.e. we installed the DB into the builder image we were using. I know, not exactly best practise for containers. But it works for now, until sidecar containers in JX is resolved; <a href="https://github.com/jenkins-x/jx/issues/619">https://github.com/jenkins-x/jx/issues/619</a></p><h4>Helm 3</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_A0J-b9flCPBrhr1lH5cg.jpeg" /></figure><p>JX predates helm 3 and hence was built to use helm 2. There’s an excellent blog post on some of the shortcomings of using vanilla helm 2: <a href="https://jenkins-x.io/blog/2018/10/03/helm-without-tiller/">https://jenkins-x.io/blog/2018/10/03/helm-without-tiller/</a></p><p>This means JX has had to provide a wrapper around helm 2 to make it play nice. The wrapper does have some limitations though:</p><ul><li>We can use the templating features of helm, but not the release management features.</li><li>The wrapper does not provide complete feature parity with helm install commands. For example, we have an application that needs to perform a DB schema update before deploying the new version of the application. Helm provides <a href="https://helm.sh/docs/topics/charts_hooks/">Chart Hooks</a>; so you can run a job as a pre-install and/or pre-upgrade hook. But the helm 2 wrapper that JX provides does not wait for the pre-* hooks to finish before applying the other chart resources.</li></ul><p>With some further customisation of the helm wrapper in JX, the second issue can probably be addressed. But that probably wouldn’t be of much long term value considering Helm 3 is GA now.</p><h4>Multi-cluster deployment</h4><p>For our cloud migration we needed to setup multiple clusters;</p><ul><li>production cluster</li><li>performance tests cluster</li><li>user acceptance testing cluster</li><li>dev/quality assurance cluster etc.</li></ul><p>By default environments in JX are handled as namespaces within the same cluster. Which brings us to the wildly experimental Jenkins X Labs project.</p><h3>Jenkins X Labs: JXL</h3><blockquote>Fair warning: the Jenkins X team have closed work on JXL. The JX team plans on incorporating some of that work into JX stable. But how much of the work done as part of JXL makes it into JX is unclear and JXL is not officially supported.</blockquote><p>JXL offered the two things we were looking for:</p><ul><li>Multi-cluster deployments</li><li>Helm 3 support</li></ul><p>For now we don’t see much of an option but to use JXL while keeping an eye out for any multi-cluster related work that is getting incorporated into JX stable. We plan on switching to JX stable when it has multi-cluster and helm 3.</p><p>Some of the things we liked about the work that went into JXL:</p><ul><li>The use of OOTB helm 3 instead of a wrapped version.</li><li>The use of OOTB <a href="https://github.com/roboll/helmfile">helmfile</a>.</li><li>GitOps environments are described almost entirely as helmfiles; <a href="https://github.com/jenkins-x/default-environment-helmfile">https://github.com/jenkins-x/default-environment-helmfile</a>, <a href="https://github.com/jenkins-x/default-environment-helmfile-local">https://github.com/jenkins-x/default-environment-helmfile-local</a></li></ul><h3>How we installed Jenkins X (Labs version)</h3><p>Official JXL docs; <a href="https://jenkins-x.io/docs/labs/">https://jenkins-x.io/docs/labs/</a></p><p>We can discuss some extra steps we undertook to suit our needs.</p><h4>Pre-requisite cloud infrastructure</h4><p>First we created AWS EKS clusters. Other cloud resources that need to be setup are: S3 buckets for logs, reports etc and <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IAM Roles for Service Accounts</a> for JX helm charts.</p><h4>JXL Boot</h4><ul><li>JXL has a <a href="https://jenkins-x.io/docs/labs/boot/getting-started/repository/">wizard</a> to create the GitOps repo. This wizard also tries to create any pre-requisite cloud infra. You can skip this using the --terraform flag; <a href="https://jenkins-x.io/docs/labs/boot/getting-started/repository/">https://jenkins-x.io/docs/labs/boot/getting-started/repository/</a></li><li>At this point you can customise the configuration: <a href="https://jenkins-x.io/docs/labs/boot/getting-started/config/">https://jenkins-x.io/docs/labs/boot/getting-started/config/</a></li><li>Then setup secrets for the installation; <a href="https://jenkins-x.io/docs/labs/boot/getting-started/secrets/">https://jenkins-x.io/docs/labs/boot/getting-started/secrets/</a></li><li>Then we run the boot process itself; <a href="https://jenkins-x.io/docs/labs/boot/getting-started/run/">https://jenkins-x.io/docs/labs/boot/getting-started/run/</a></li></ul><h4>Remote Environments</h4><p>Adding remote environments is how we do multi-cluster deployments in Jenkins X. We had to tweak this a little bit to get it working just right. (when muti-cluster eventually makes it’s way into stable JX, this may not apply)</p><p>For each cluster where you want to deploy applications to, you need at least the following system apps and Jenkins X apps in jx-apps.yml:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1ab8d310c4ba05e6f2d2e356841268c/href">https://medium.com/media/b1ab8d310c4ba05e6f2d2e356841268c/href</a></iframe><p>The primary cluster is where all CI would happen and CD will be triggered for any “Automatic” environments. Where the CD for each environment is executed depends on whether an environment is local or remote.</p><ul><li>If it is local, the deployment will be handled by the Jenkins X apps in the primary cluster</li><li>Otherwise it will be handled by the Jenkins X apps on another cluster.</li></ul><p>We setup 2 clusters: ci &amp; dev with the following GitOps repos:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80debe253fadfb360db1a9cfc6a77dd3/href">https://medium.com/media/80debe253fadfb360db1a9cfc6a77dd3/href</a></iframe><p>We created a separate environment-dev-base repo for the JX installation and general applications like <a href="https://hub.helm.sh/charts/stable/metrics-server">metrics-server</a>, <a href="https://hub.helm.sh/charts/stable/cluster-autoscaler">cluster-autoscaler</a> etc. so that we can segregate that from actual ‘deployment’ environments. We don’t mention this repo in environment-ci-base.</p><p>environment-dev-auto* environments are referenced in the <a href="https://jenkins-x.io/docs/labs/boot/getting-started/config/">jx-requirements.yml</a> file of environment-dev-base repo under environments array. Referencing environment-dev-auto* repos in environment-ci-base as remote environments don’t work as intended. The JX installation in the primary cluster sets up GitHub webhooks to itself for these repos if we do that. But the CD for auto1 &amp; auto2 environments should be done by the JX installation in the dev cluster (environment-dev-base repo).</p><p>To link auto1 &amp; auto2 to the JX installation in the primary CI cluster, we added a custom chart in environment-ci-base:</p><pre>remote-envs/<br>  templates/<br>    env-auto1.yaml<br>    env-auto2.yaml<br>  Chart.yaml</pre><ul><li>Chart.yaml</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ae4048359c91dbb141dfe29ff09e693/href">https://medium.com/media/0ae4048359c91dbb141dfe29ff09e693/href</a></iframe><ul><li>env-auto1.yaml</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63231fee7bafa9d1d42cd922aa9a166e/href">https://medium.com/media/63231fee7bafa9d1d42cd922aa9a166e/href</a></iframe><h4>Some extras for EKS</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*cgkU6oBaTpQfdFpbbB8Kag.png" /></figure><p>Here’s some docs on the Apps framework based on helmfile and helm 3: <a href="https://jenkins-x.io/docs/labs/boot/apps/">https://jenkins-x.io/docs/labs/boot/apps/</a>. We recommend adding the following charts to jx-apps.yml:</p><ul><li><a href="https://hub.helm.sh/charts/stable/cluster-autoscaler">cluster-autoscaler</a></li><li><a href="https://hub.helm.sh/charts/stable/metrics-server">metrics-server</a></li></ul><p>JX uses <a href="https://hub.helm.sh/charts/jetstack/cert-manager">cert-manager</a> for TLS on ingress. We decided to go another way. With version 1.15 EKS now supports TLS termination for Network Load Balancers.</p><ul><li>Setup Route53 to setup a top level domain for your cluster; <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html">https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/getting-started.html</a></li><li>Request a public certificate using AWS ACM; <a href="https://aws.amazon.com/certificate-manager/">https://aws.amazon.com/certificate-manager/</a></li><li>Make sure you have bitnami/external-dns chart in your jx-apps.yml file and have set the following in jx-requirments.yml:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b780c55ecf5a7bb96291f6a48fc584ab/href">https://medium.com/media/b780c55ecf5a7bb96291f6a48fc584ab/href</a></iframe><ul><li>Notice that we do enable TLS settings here, but we don’t install <a href="https://hub.helm.sh/charts/jetstack/cert-manager">cert-manager</a>. JX applications would reference TLS secrets and that’ll log some warnings, but we are ignoring that for now.</li><li>Create a file apps/nginx-ingress/values.yaml.gotmpl with the following content:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1adbf2567c691dfd2444ea3c97b1dd3/href">https://medium.com/media/a1adbf2567c691dfd2444ea3c97b1dd3/href</a></iframe><p>You can also setup a safe list of CIDR ranges for your NLB as shown above. Make sure GitHub webhook IP ranges are definitely present; <a href="https://api.github.com/meta">https://api.github.com/meta</a>. (assuming your repositories are in GitHub of course)</p><h4>Version Stream</h4><p>Here’s some docs on the version stream: <a href="https://github.com/jenkins-x/jx-docs/blob/master/content/en/about/concepts/version-stream.md">https://github.com/jenkins-x/jx-docs/blob/master/content/en/about/concepts/version-stream.md</a></p><p>JXL uses it’s own Version Stream repo; <a href="https://github.com/jenkins-x-labs/jenkins-x-versions">https://github.com/jenkins-x-labs/jenkins-x-versions</a>. Since this is no longer kept up to date, you can maintain your own fork of that or it’s base. You can specify which version stream to use in jx-requirments.yml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7584c8bb68a5fa48a60c2bc1a2cd8826/href">https://medium.com/media/7584c8bb68a5fa48a60c2bc1a2cd8826/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1f51c462c3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybris Type System Update on Kubernetes]]></title>
            <link>https://medium.com/@gazal.k/hybris-type-system-update-on-kubernetes-791d75e94917?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/791d75e94917</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[hybris]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Mon, 16 Mar 2020 00:28:34 GMT</pubDate>
            <atom:updated>2020-05-03T00:22:05.891Z</atom:updated>
            <content:encoded><![CDATA[<p>Hybris provides a mechanism to rolling update the database schema without downtime. Hybris uses metadata tables that form what is called the type system to manage multiple versions of database schema. When updating your hybris application to a new version you can:</p><ul><li>Create a new type system (copying from current type system)</li><li>And update the new type system</li></ul><p>without affecting the type system used by the active application. Once this is done, we can start hybris servers with the new version of the application referencing the new type system.</p><p><a href="https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/6.7.0.0/en-US/8c455268866910149b25f7b53d1af3e1.html">SAP Help Portal</a></p><h3>On our data centre with fixed capacity</h3><p>On our current data centre where we run hybris, we have 2 limitations we need to work with:</p><ul><li>We have a fixed number of servers to work with. We can’t provision more easily and certainly not in an automated manner.</li><li>We don’t have a centralised session store. User sessions are stored in the hybris server nodes where the user is connected through to from the load balancer. To ensure the user’s session is maintained across requests, we use sticky sessions and the load balancer will always direct a user to the same server they interacted with before.</li></ul><p>So, with those limitations the process by which we currently update hybris is:</p><ol><li>We drain the sessions on one of the servers to make sure customers are not directed to that server anymore.</li><li>We take that server off the load balancer.</li><li>We copy the existing type system to create a new type system.</li><li>Update the new type system.</li><li>Deploy the new version of the application referencing the new type system on that server.</li><li>Add the server back on the load balancer.</li><li>We repeat steps 1, 2, 5 and 6 for the remaining servers.</li></ol><p>This process takes time and is hard to automate.</p><h3>On Kubernetes in public cloud</h3><p>When running hybris on Kubernetes in public cloud the limitation on the number of servers is gone out of the box. We also mitigated the session store issue by using a redis cluster as a centralised session store. With those 2 limitations out of the way, our strategy to run type system update on Kubernetes is:</p><ol><li>Create a Job with the new version of the application (container image).</li><li>The Job creates a new types system and updates it.</li><li>Once the Job is complete, the k8s Deployment object is updated to use the new version of the application (container image). The Deployment takes care of starting Pods with the new container image and kill old Pods once the new ones are ready.</li></ol><p>This process is entirely automated using helm. We use helm chart hooks</p><p><a href="https://helm.sh/docs/topics/charts_hooks/">Chart Hooks</a></p><p>The Job that is responsible for creating and updating a new typesystem is defined to run as a pre-install and pre-upgrade hook.</p><h3>Tweak to type system update procedure</h3><p>The hybris type system uses a bunch of meta data tables; AtomicTypes, ComposedTypes, MapTypes, CollectionTypes, EnumerationValues, AttributeDescriptors etc. When we copy an existing type system to create a new one, those tables are duplicated and tables such as AtomicTypes0, ComposedTypes0, MapTypes0 etc are created. Every new type system gets their own copy of these tables. This means any downstream system that may need to reference type system tables needs to identify the version of type system tables currently in use.</p><p>To avoid this problem:</p><ul><li>Use a fixed reference type system, say DEFAULT or something like ‘MASTER’.</li><li>Start by updating the reference type system.</li><li>Then create a copy for the new version being deployed.</li></ul><p>Most existing rolling update scenarios should be OK with this strategy. Also, any downstream systems can easily reference the type system tables for DEFAULT or MASTER which does not change.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=791d75e94917" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Redis as a session store for Hybris Commerce]]></title>
            <link>https://medium.com/swlh/use-redis-as-a-session-store-for-hybris-commerce-29bea8d498f1?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/29bea8d498f1</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[spring-session]]></category>
            <category><![CDATA[hybris]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Mon, 30 Dec 2019 06:07:37 GMT</pubDate>
            <atom:updated>2020-01-10T00:15:49.237Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*ycZwBbzZ2OHchGS0zSeqjQ.png" /></figure><p>When running Hybris Commerce (or any web application for that matter) in cloud infrastructure, we need to cater for the fact that our application containers or VMs may fail/stop at any time. This needn’t be because of an error scenario; it can even happen when the infrastructure is scaled out or scaled down.</p><p>The traditional/default approach of maintaining sessions in the server’s memory with sticky sessions won’t be enough. Which is why it is one of the <a href="https://12factor.net/processes">tenets</a> of The Twelve Factors App.</p><h3>Hybris Session Failover Support</h3><p>To create a centralised session store, we looked at what Hybris Commerce offers out of the box. This is what we found:</p><p><a href="https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/6.7.0.0/en-US/5da5660b2a3f4263ac3095d900d07f88.html">SAP Help Portal</a></p><p>The mechanism does use spring-session, but the backing store is the database which of course may not be the most performant solution.</p><p>We initially looked at how we can extend the same capability and use Redis for storage instead of the database because that seemed like the easiest way to accomplish what we need. But that ended up being a bit of a rabbit hole.</p><h3>Spring Data Redis</h3><p>We eventually settled on using <a href="https://spring.io/projects/spring-data-redis">spring-data-redis</a>. Since Hybris Commerce already uses a lot of spring modules, using another community supported solution seemed like the best way forward.</p><p>We’ll look at how we add a session store to our storefront extension.</p><h4>Adding the dependencies</h4><p>First step we need to add relevant dependencies. Since we are currently using Hybris 6.7.x, the latest compatible versions of libraries we needed were:</p><ul><li>spring-data-commons-1.13.23.RELEASE.jar</li><li>spring-data-keyvalue-1.2.23.RELEASE.jar</li><li>spring-data-redis-1.8.23.RELEASE.jar</li><li>jedis-2.9.0.jar</li></ul><p>Hybris already packs spring-session, so we don’t need to add that again.</p><h4>SessionStoreConfiguration</h4><p>We need to create a spring configuration class to enable redis backed http sessions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c50374a738af256144c2046bf7d198bb/href">https://medium.com/media/c50374a738af256144c2046bf7d198bb/href</a></iframe><p>This configuration class needs a couple of things to work:</p><ul><li>It needs a ConfigurationCondition; SessionStoreCondition to evaluate to true</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/942fb9860db13a7f50e08982638166bc/href">https://medium.com/media/942fb9860db13a7f50e08982638166bc/href</a></iframe><ul><li>And for the property <em>spring.sessionstore.hostnam</em>e to have the url to the redis server/cluster.</li></ul><p>Now we need to add the configuration class to our extension’s spring context. Best practise for hybris extensions is to segregate beans by functionality in spring context xmls. So let’s create one for spring session:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/978fb927f7e197b83376b0a3ee0ca6af/href">https://medium.com/media/978fb927f7e197b83376b0a3ee0ca6af/href</a></iframe><p>Now import this into the main web-app-config.xml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3496bd60f9bad330f58305b8d795f14d/href">https://medium.com/media/3496bd60f9bad330f58305b8d795f14d/href</a></iframe><p>Once all of this has been done, a new filter bean; <em>springSessionRepositoryFilter</em> will be available in the context. Add the filter to the <em>storefrontFilterChain</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9470bdb15ac8c2a6b82facbf50f0816a/href">https://medium.com/media/9470bdb15ac8c2a6b82facbf50f0816a/href</a></iframe><p>With these changes we have all the ground work to have our sessions stored centrally using a redis server/cluster. We’ve introduced two properties to configure it:</p><ul><li>spring.sessionstore.enabled</li><li>spring.sessionstore.hostname</li></ul><h4>NO-OP SessionRepositoryFilter</h4><p>When <em>spring.sessionstore.enabled</em> is set to false, our <em>storefrontFilterChain </em>still references <em>springSessionRepositoryFilter</em> which can prevent our application from starting up without errors. To workaround this, create a NO-OP SessionRepositoryFilter:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c0f4f25bd446db7cd9d39b1a1a244e9/href">https://medium.com/media/5c0f4f25bd446db7cd9d39b1a1a244e9/href</a></iframe><p>The filter depends on <em>SessionStoreConditionNot</em> which is the inverse of <em>SessionStoreCondition</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e6f44efc5f7caae4b12dea9ffc1b549/href">https://medium.com/media/3e6f44efc5f7caae4b12dea9ffc1b549/href</a></iframe><h4>Ensure Session is Serializable</h4><p>Storing sessions in Redis using spring-session requires that sessions be serializable. Hybris Commerce does have some inbuilt capability to verify session serialization. Please check the documentation here:</p><p><a href="https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/6.7.0.0/en-US/5da5660b2a3f4263ac3095d900d07f88.html#loio7060f9504fe444b286a2578633ea0412">SAP Help Portal</a></p><h4>Enable session.replication.support</h4><p>If you are relying on attributes in the <em>JaloSession</em>, you’ll need to set <em>session.replication.support</em> to <em>true</em>.</p><p>Hybris’s <em>SessionFilter</em> adds <em>JaloSession</em> as an attribute to the servlet container’s session. Every attribute added to session is serialised and saved into the session store.</p><p>But since <em>JaloSession</em> itself is a store for other attributes, mutations to <em>JaloSession</em> can’t automatically be persisted in the session store.</p><p>When <em>session.replication.support </em>is true, Hybris’s <em>SessionFilter</em> removes and adds the <em>JaloSession</em> attribute again to session after executing the filter chain so that any changes to <em>JaloSession</em> are persisted to the session store.</p><h4>Watch out for cookie manipulation</h4><p>This is probably not relevant to everyone who tries to use Redis to back session storage of their Hybris Commerce solution, but we had code that specifically rewrote the JSESSIONID cookie to the response. The problem was that the filter that performed this operation relied on session.getId() and expected it to be JSESSIONID. But once we add <em>springSessionRepositoryFilter</em> to the filter chain, <em>session.getId()</em> will return the id for the wrapped spring session object.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29bea8d498f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/use-redis-as-a-session-store-for-hybris-commerce-29bea8d498f1">Use Redis as a session store for Hybris Commerce</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Amazon S3 for Hybris Commerce Media]]></title>
            <link>https://medium.com/@gazal.k/using-amazon-s3-for-hybris-commerce-media-2f803fc65e5a?source=rss-27ac4324fcee------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f803fc65e5a</guid>
            <category><![CDATA[hybris]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Gazal Gafoor]]></dc:creator>
            <pubDate>Tue, 24 Dec 2019 08:17:07 GMT</pubDate>
            <atom:updated>2020-03-13T06:29:21.975Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Amazon S3 for SAP Hybris Commerce Media Storage on Kubernetes (EKS)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*YrJ1FK18N1bYyrahcTdFfg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*FjtxYKwiYnV2fs5fSuFjlg.png" /></figure><p>Hybris 6.x+ already has an out of the box extension to store and retrieve media data from S3 buckets.</p><p>The official documentation details how that is done; <a href="https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/6.7.0.0/en-US/8c79f86786691014bd22c0377274d396.html">https://help.sap.com/viewer/d0224eca81e249cb821f2cdf45a82ace/6.7.0.0/en-US/8c79f86786691014bd22c0377274d396.html</a>. We wanted to use this as part of migrating our platform to AWS, but we wanted a more secure way than using an access key pair in hybris configuration. To do this, we went about customising the hybris extension.</p><h3>IAM Roles for Service Accounts</h3><p>We were using Kubernetes(<a href="https://aws.amazon.com/eks/">EKS</a>) for container orchestration; so <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html">IRSA</a> seemed like the obvious choice to secure access from the application to an S3 bucket.</p><p>To use IRSA we had to upgrade to newer versions of AWS SDK jars; <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html">https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html</a></p><ul><li>aws-java-sdk-core-1.11.665.jar</li><li>aws-java-sdk-kms-1.11.665.jar</li><li>aws-java-sdk-s3–1.11.665.jar</li><li>aws-java-sdk-sts-1.11.665.jar</li></ul><h3>Custom Extension</h3><p>So we ended up creating a new hybris extension; customamazoncloud with the new SDK jars and amazoncloudserver.jar; the jar from the OOTB amazoncloud extension in hybris.</p><p>We copied most of the content from the resources folder from amazoncloud extension (localisation, beans.xml, items.xml, spring.xml)</p><ul><li>We extended the DefaultS3StorageServiceFactory class so that we can override how the AmazonS3 client is created.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/431c13d38c1ef679b5df4336e575be4f/href">https://medium.com/media/431c13d38c1ef679b5df4336e575be4f/href</a></iframe><ul><li>Out of the box S3MediaStorageStrategy uses a slightly different naming convention for the tenant root folder compared to LocalFileMediaStorageStrategy. The location property stored for a media object is also different between the two. Since we were migrating our existing media to an S3 bucket, we wanted backward compatibility. To do that we extended S3MediaStorageStrategy as follows:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e253c75bf23d7090c92654437532e23b/href">https://medium.com/media/e253c75bf23d7090c92654437532e23b/href</a></iframe><ul><li>Update the spring bean definitions</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bfc5707ef9eb0163db2f2c0f2fb29e3/href">https://medium.com/media/0bfc5707ef9eb0163db2f2c0f2fb29e3/href</a></iframe><h3>Kubernetes Service &amp; Ingress Objects</h3><p>Now that we’ve setup the storage to an S3 bucket, we can setup Kubernetes Service and Ingress objects to route traffic to it.</p><p>We use <a href="https://jenkins-x.io/">Jenkins X</a> for CI/CD and our applications are packaged using <a href="https://helm.sh/">Helm</a>. To route traffic to our S3 bucket, we create a Service object of type <a href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname">ExternalName</a>. Jenkins X’s <a href="https://github.com/jenkins-x/exposecontroller">exposecontroller</a> creates the Ingress object for us based on the annotations with prefix fabric8.io. So the service object we created for routing traffic to our S3 bucket looked like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd048b952b22e11d344af2775e27e994/href">https://medium.com/media/bd048b952b22e11d344af2775e27e994/href</a></iframe><p>And for the Service to securely access the S3 bucket, we ensured only traffic from within the VPC of the Kubernetes cluster has access to it. To do this:</p><ul><li>Create a VPC endpoint for your S3 bucket; <a href="https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/">https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/</a></li><li>Create a bucket policy to allow access from the VPC endpoint</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dceffab02ac408d9ce38a13bd0f9837/href">https://medium.com/media/2dceffab02ac408d9ce38a13bd0f9837/href</a></iframe><p>That was it. We were able to store and retrieve Hybris media from an S3 bucket.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f803fc65e5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>