<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anuradha Prasanna on Medium]]></title>
        <description><![CDATA[Stories by Anuradha Prasanna on Medium]]></description>
        <link>https://medium.com/@anuradhaneo?source=rss-2689aaaafbf0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*aDZe92VMvF3huKW2TN9j_g.jpeg</url>
            <title>Stories by Anuradha Prasanna on Medium</title>
            <link>https://medium.com/@anuradhaneo?source=rss-2689aaaafbf0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:35:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anuradhaneo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Common EC2 key pair for all EC2 instanced in multiple AWS regions]]></title>
            <link>https://anuradhaneo.medium.com/common-ssh-key-pair-for-all-multi-region-aws-ec2-instances-311798eb48f6?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/311798eb48f6</guid>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[keypair]]></category>
            <category><![CDATA[ssh-keys]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Wed, 11 Nov 2020 05:52:00 GMT</pubDate>
            <atom:updated>2020-11-11T05:52:00.121Z</atom:updated>
            <content:encoded><![CDATA[<p>In some deployment architecture scenarios with multi-region high available application deployments or SaaS solutions, it may be required that you as the AWS or SaaS platform admin you have to access the AWS EC2 instances deployed in different regions. It is handy and convenient to have same set of EC2 SSH key pairs across the regions so that you can connect to any particular EC2 resource across different regions. This can be applicable for deployment architectures simplified as below for the purpose of explaining this SSH key usecase as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*lWOC44Qetvr0hLYmK-SeUg.jpeg" /></figure><p>How AWS key pairs work ?</p><p>Key pairs are a bundle of public and private keys. AWS stores the public key portion of the key pair so that it can decrypt the SSH traffic generated by the authentic private key which should be privately and securely held in your possession.</p><p>In order to have same set of key pairs deployed in all AWS regions what we have to do is</p><ol><li>Generate a key pair in of the the regions and download the key pair to your PC/Mac</li><li>Export/Extract the public key to your local disk</li><li>Upload/Import this public key portion to all AWS regions you will be working with.</li></ol><p>Lets see how this can be done.</p><p>First, lets generate one key pair using the AWS console and store it locally on your machine. You can skip to next step if you already have a key pair generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZDIK7TJw7-fCBy4AoLD_Q.png" /></figure><p>Secondly, lets extract the public key of this key pair you downloaded in previous step with below command.</p><blockquote>&gt; ssh-keygen -y -f MyAWSVMKP.pem &gt; ./MyAWSVMKP.pub</blockquote><p>This .pub file contains the public key component of the key pair which is something like this</p><blockquote>ssh-rsa KKKKKB3NzaCUYD&amp;*YUIDAQABAAAkndfod8FDUrwXPKdosG3M8LcsaZJWZxZPxk98SjpF9XVW8JuGK9Ef2NKCN897enjkHhmDiMhaIcx6bYY370MZ9ZQZqLgSW6Kk+BUTjewNTC1d0cMsJND&amp;^ThbhjdlUO+vC/WTlO9Ugl6u2NSrNovI8+Qsr8mjnfds7cwYeI/LSKJND88+9Ct0PK6xiwnVrQjfyokF5Sf7+Y3RHa52evvtsBrXSvAmBULgwBgKVBONT+f3cPYfjd97kGPKA/cBp3pCX1+1JN5jhf7sd8VhP&amp;hk8+McHTJ/G/jh86d7sD+G6noJ/2d2i8njk2nL7i5Gu/4Wo3cd9diu</blockquote><p>Finally, You can import this public key to all your regions as shown below</p><ol><li>Change to the AWS region you need to have the key pair imported</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*ftViXVGZ5j_BWGH0pksMHg.png" /></figure><p>2. Go to EC2 Console, search EC2 and click to open EC2 console view</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ep4OLCRBjlzztw34qtYvdg.png" /></figure><p>3. Go to KeyPairs section under Networking &amp; Security items in left side menu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4CM5AuJPQXvw2M04Ab8vSQ.png" /></figure><p>4. Click on the Import Key Pair from the Actions menu at top-right</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*1XcKlIt5Ef9ZyR45LJccMQ.png" /></figure><p>5. Upload the .pub file generated previously and give your key pair a name. Alternatively you can copy the contents from the .pub file in to the text box too</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmIEOPRRcu_Y5v8Zxi3cqg.png" /></figure><p>6. Hit import key pair button</p><p>7. You should now seethe key pair entry is generated</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzDdp2vu6Y1RtzvcJPRRgQ.png" /></figure><p>We’re Done !</p><p>With this now you can use the same private key to SSH into the EC2 VMs you have been using in some other region with the SSH command you use something like below.</p><blockquote>&gt; ssh -i MyKeyPair.pem ec2-user@&lt;server_IP&gt;</blockquote><p>Hope this helps you guys and learnt something good even if its a small thing !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=311798eb48f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How important is monitoring to DevOps]]></title>
            <link>https://anuradhaneo.medium.com/how-is-monitoring-important-to-devops-3d4669f84fc4?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d4669f84fc4</guid>
            <category><![CDATA[application-monitoring]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Thu, 28 May 2020 11:13:15 GMT</pubDate>
            <atom:updated>2020-05-28T11:13:15.606Z</atom:updated>
            <content:encoded><![CDATA[<p>DevOps process/culture can be simply visualised as below cyclic diagram !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*LQ1ZJRrVZ10_4c2I3e9Pvg.jpeg" /></figure><p>As you can see Monitoring is a key step within the DevOps cycle, and surrounded by live application in behind and planing in forward. Lets break down this link and understand it,</p><p>When an application is deployed to production its operated serving the clients doing various transactions and operations exchanging digital messages additionally across various south-bound systems as a part of the business use cases.</p><p>This application operations leaves below traces</p><ol><li>infrastructure resource utilisation (CPU, vCPUs, memory, disk I/O etc.)</li><li>application runtime resource utilisation (virtualisation resources, threads, memory allocation/utilisation, disk pressure, active instances etc.)</li><li>application transaction/operational use and patterns</li><li>service health history</li><li>information feeds to analytical/data warehouse systems</li></ol><p>What values does above information leaves us with ?</p><ol><li>action in to environment/runtime configurations (eg. expand memory allocation, adjust heap memory)</li><li>help triage incidents and bugs</li><li>generate alerts based on set threshold limit based event triggers</li><li>and there can be more uses custom to operations teams</li></ol><p>What is the missing thing teams were ignoring historically ? and ideally suggested by DevOps</p><ul><li>Use the monitoring insights to identify the bottlenecks within the performance of the code</li><li>how to optimise the releases based on application usage patterns</li><li>automate incident reporting and assignment to development teams</li><li>use the monitoring information/reports for the development sprint planning sessions (specially after new releases)</li></ul><p>This is the closure of the feedback loop from the production application operations to refined/prioritised development process (through backlogs)</p><p>This post production feedback loop is very important to run naturally as a practice. By doing so over time it makes sure the software customers get quality and working software customised to their demands and market trends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*ffxkpcygPsl4qejMjkO0FQ.png" /></figure><p>Know know your application behaves in production with enabling visibility to all vital information to make sure the road for application’s operation is clear without leading the potential to catastrophic or silent failures that can cost in big dollar values</p><p>Let’s enforce our teams and determine to build monitoring platforms parallel to our applications so that from the day one of the application operation there is a mechanism and a safety net available to better cater the customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d4669f84fc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DORA for DevOps]]></title>
            <link>https://anuradhaneo.medium.com/dora-quadrant-for-devops-80f8526a2eb3?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/80f8526a2eb3</guid>
            <category><![CDATA[application-performance]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-deployment]]></category>
            <category><![CDATA[dora]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Thu, 28 May 2020 03:52:40 GMT</pubDate>
            <atom:updated>2020-05-28T03:52:40.456Z</atom:updated>
            <content:encoded><![CDATA[<h3>DORA Quadrant for DevOps</h3><p><strong>DORA</strong> is a research program investigates the capabilities that drive software delivery performance and stability in organisations across the globe.</p><p>One of the wonderful analytical tools DORA provides if the DORA Quadrant. This analytical tool provides a view of the current status of an organisation’s DevOps maturity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*EB8jUcmRud7qv3_3fIHFag.png" /><figcaption>DORA DevOps Maturity Quadrant</figcaption></figure><p>This tool indicated four key measurements, and they are</p><ol><li>Deployment frequency</li><li>Lead time for changes</li><li>Time to restore service</li><li>Change failure rate</li></ol><p>Now lets look at the definitions of these four measurements</p><p><strong>Deployment frequency</strong></p><p>For the primary application or service you work on, how often does your organisation deploy code to production or release it to end users ?</p><p><strong>Lead time for changes</strong></p><p>For the primary application or service you work on, what is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production) ?</p><p><strong>Time to restore service</strong></p><p>For the primary application or service you work on, how long does it generally take to restore service when a service incident or a defect that impacts users occurs (e.g., unplanned outage or service impairment) ?</p><p><strong>Change failure rate</strong></p><p>For the primary application or service you work on, what percentage of changes to production or released to users result in degraded service (e.g., lead to service impairment or service outage) and subsequently require remediation (e.g., require a hotfix, rollback, fix forward, patch) ?</p><p>These four measurements identifies two aspects ,</p><p><strong>DevOps Performance</strong></p><blockquote>by Deployment frequency and Lead time for changes</blockquote><p><strong>Application Stability</strong></p><blockquote>by Time to restore service and Change failure rate</blockquote><p>DORA also provides the details to compare your application or organisation with their global studies DORA is performing every year.</p><p>You can see the summary of the study DORA has done in 2019</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*Dg9DKKSyk5FlmTXoE7izAw.png" /><figcaption>source: DORA Accelerate State of DevOps 2019 report</figcaption></figure><p>You can compare in which category(s) your application/team/organisation is in by comparing your rating to these industry benchmarks.</p><p>If you have multiple applications to assess you can measure these values individually against applications and average to derive the final quadrant.</p><p>Altogether, in summary this quadrant depicts how mature is your DevOps process by identifying the quality of the application code and how frequently you can move that code to production !</p><p>So this is a great tool for any organisation to use and this must be calculated periodically to track improvements or any deviations from the targets set.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80f8526a2eb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Kafka” is not the best anymore, Meet “Pulsar” !]]></title>
            <link>https://anuradhaneo.medium.com/kafka-is-not-the-best-anymore-meet-pulsar-9eb435c9fc0b?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9eb435c9fc0b</guid>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[pulsar]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[apache]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Wed, 26 Jun 2019 19:47:25 GMT</pubDate>
            <atom:updated>2019-06-26T19:47:25.916Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka is not the best anymore, Meet Pulsar !</h3><p>Apache Kafka created by Linkedin in 2011 was the long standing decoupled messaging power house that was the only option for most performance critical large scale messaging requirements that needs to pass millions of messages per day, well thats a lot (average tweets per day in Twitter 500 millions per day in 2018 for average number of 100 million users per day). Back in those days there we no such MOM system to handle large streaming capabilities to a large subscriber base. So Kafka was the survivor for most big names Linkedin, Yahoo, Twitter, Netflix and Uber.</p><p>Now the things are much more changed in 2019 and daily messaging counts go up in billions, so as platform support needs to scale accordingly to support the continued growing demand. So in order to do this messaging systems need to scale continuously and seamlessly without impacting the customers. Kafka has lot of problems in scaling and it is difficult system to manage. Guys who love Kafka will feel bad about telling this, but its nothing personal, I too a Kafka fan, but as world innovating new tools which as much convenient in comparison it is obvious that we tend to feel the previous things are more difficult to manage and has problems to deal with. Its the nature.</p><p>So a new guy comes to the town is called “<strong>Apache Pulsar</strong>” !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-PCGcR-kqbhIy5Js_0IqMw.jpeg" /></figure><p>Pulsar is created by Yahoo back in 2013 and donated to Apache in 2016. Pulsar is now an Apache top level project with proven success. Yahoo and Twitter both use Pulsar where in Yahoo 100 billion messages were sent per day with over 200 million messaging topics. Well its unbelievably large.</p><p>Lets see what are the problems with Kafka and how Pulsar takes those burden off your shoulders.</p><ol><li>Scaling Kafka is difficult, this is due to the way Kafka stores data within the broker as distributed logs that stores as messaging persistence store. Spinning off another broker means it has to replicate the topic partitions and replicas completely which takes time to complete the broker spin off.</li><li>Changing partition sizes to allow more storage can mess the message order by conflicting the message indexes. This is critical if the message order is a prime concern.</li><li>Leader election can go crazy if the partition replicas are not in ISR (In Sync) state. Basically there should be at least one ISR replica to be elected as the leased when the original leader partition fails, but this cannot be guaranteed. there is a setting to disable this but then if enabled a non-ISR replica will get elected as the leader which is much worse since that impose a worse situation than a service outage without a leader partition.</li><li>Ideally you must plan and calculate number of brokers, topics, partitions and replicas in first place (that fits planned future usage growth) to avoid scaling problems, but now adays with unpredictable traffic demands and spike its is hard to plan.</li><li>Kafka cluster rebalancing can impact the performance of connected producers and consumers</li><li>Kafka topics can lose messages in failure scenarios (specially in point 3)</li><li>Working with offsets is a pain because Kafka is dumb (don’t be angry for saying that, because the “dumb” comes from the architectural concept “dumb broker and intelligent client”)</li><li>Old messages has to be deleted pretty soon if the usage is high to avoid disk space problems</li><li>Kafka’s native across region geo replication mechanism (MirrorMaker) is notoriously problematic, even within just two data centers. Because of this even Uber has created another solution to overcome this called <a href="https://eng.uber.com/ureplicator/">uReplicator</a> as an example</li><li>You must use another realtime event analyzer tool such as Apache Storm, Apache Heron or Apache Spark if you need to do so, also that should be in a strength to support the incoming traffic rates</li><li>No native multi-tenancy capabilities with complete isolation of tenants . It is done with the use of security features such as topic authorization</li></ol><p>There are ways architects and engineers solve above problems in their production environments of course, but there it costs some hairs in the platform/solution or site reliability engineer’s head and some headaches for sure, because these are not simple fixes like we fix a logic in a code logic and then deploy the packaged binaries to production.</p><p>OK ,,,,</p><p>Now lets talk about out “guy”, Pulsar. Because he leads the game.</p><h3>What is Apache Pulsar ?</h3><p>“Apache Pulsar is an open-source distributed pub-sub messaging system originally created by Yahoo” as per the website’s intro. Essentially its something like Kafka for those who understands Kafka :)</p><h3>Pulsar Performance !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMx8WQqpN19shIc3sV0Meg.jpeg" /></figure><p>Best highlight is performance, Pulsar is a much faster than Kafka, and it is proven by a performance comparison done by a Texas based technology research and analysis firm called <a href="https://gigaom.com/">GigaOm</a>.</p><p>Approximately Pulsar is <strong>2.5 times faster</strong> and <strong>40% less latency</strong> than Kafka. Isn’t that great ? Of course. (source is <a href="https://streaml.io/pdf/Gigaom-Benchmarking-Streaming-Platforms.pdf">here</a>)<br>Note this performance comparison is on 1 topic over 1 partition with 100-byte messages where Pulsar maxed to a 220,000+ messages per second as shown below .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*9kxNX_RnAg69g6A9.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*UNMbYJMktvV7nLLx.jpg" /></figure><p>WOW, thats really awesome !</p><p>Well its the definite point to move away from Kafka and embrace Pulsar, and this will be supported by below facts I have explained.</p><h3>Apache pulsar advantageous and features</h3><p>Pulsar can function both like Kafka’s offset based topic reader consumption method as well as traditional pub-sub topic message consumption which is a great thing.</p><p>Pulsar has a different architecture for data persistence. Rather than using log files in the local broker as in Kafka, Pulsar stores all the topic data in a specialized data layer powered by Apache Bookkeeper as data ledgers. Simply said Bookkeeper is a highly scalable, fault-tolerant and low-latency storage service optimized for real-time durable data workloads. So bookkeeper guarantees the availability of the data unlike the issues possible to arise with Kafka log files which resides in individual brokers as well as for catastrophic server failures. Due to this guaranteed persistence layer, Pulsar brings another advantage where “brokers are made stateless”, yes you read it correctly. Thats a big difference compared to Kafka. Now the real advantage is Pulsar brokers can seamlessly scale horizontally to cater the growing demands by spinning new Pulsar brokers very quickly as there is no topic partition data to load nor synchronize such as Kafka does when spinning up new brokers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*ArZPH_pvvhR4xk8FHrpLUQ.jpeg" /></figure><p>What if a Pulsar broker goes down ? Topics will be reassigned immediately to another broker, and this is possible because no topic data persists in the broker’s disk and service discovery will handle the publishers/consumers.</p><p>Unlike in Kafka where you need to purge old data to make disk space available for Kafka, Pulsar store its topic data in a tiered structure where you can connect additional disks or Amazon S3 to extend and offload the topic data storage literally unlimitedly. The cool thing is Pulsar shows data seamlessly to the consumers as if they are coming from a single drive. <br>Another valuable use case is since you never need to purge old data is configured you can use these organized Pulsar topics as a “<strong>Data Lake</strong>”. <br>But of course you can set Pulsar to purge old data if needed.</p><p>Pulsar natively supports multi-tenancy with data isolation at the topic namespace level where such isolation is not possible with Kafka. <br>On top of that to make Pulsar applications more solid and secure, Pulsar supports fine grained access control capabilities.</p><p>Pulsar has multiple client libraries available for languages Java, Go, Python, C++ and WebSocket.</p><p>Pulsar has Function as a Service (FaaS) support natively which is a very cool feature which is similar to Amazon Lambda functions where realtime data streams can be analyzed, aggregates or summarized in realtime. This is a great advantage when compared to Kafka where you need a stream processing system such as Apache Storm which is an additional cost and pain to maintain. As of now Pulsar functions support Java and Python and other languages will be supported in later releases. <br>Example use cases of Pulsar functions are content based routing, aggregations, message formatting, message cleansing.<br>Below shows a sample function calculates word count,</p><pre>package org.example.functions;</pre><pre>import org.apache.pulsar.functions.api.Context;<br>import org.apache.pulsar.functions.api.Function;</pre><pre>import java.util.Arrays;</pre><pre>public class WordCountFunction implements Function&lt;String, Void&gt; {<br>    // This is invoked every time messages published to the topic<br>    <a href="http://twitter.com/Override">@Override</a><br>    public Void process(String input, Context context) <br>        throws Exception {</pre><pre>        Arrays.asList(input.split(&quot; &quot;)).forEach(word -&gt; {<br>            String counterKey = word.toLowerCase();<br>            context.incrCounter(counterKey, 1);<br>        });<br>        return null;<br>    }<br>}</pre><p>Pulsar supports number of data sinks for routing processed messages for leading products such as Pulsar topics itself, Cassandra, Kafka, AWS Kinesis, Elastic search, Redis, Mongo DB, Influx DB and many more. <br>Also there is ability to persist processed messages stream to disk files.</p><p>Pulsar enables querying past messages with a SQL style way with Pulsar SQL, which is very efficiently query Bookkeeper data efficiently with the use of Presto engine. Presto is a high performance, distributed SQL query engine for big data solutions that allows query data from multiple data sources within a single query. Below is a sample Pulsar SQL query,</p><pre>show tables in pulsar.&quot;public/default&quot;</pre><p>Another very important feature is built in robust geo-location replication mechanism that synchronizes messages instantly across different clusters in different regions to maintain the message integrity. When messages are produced on a Pulsar topic, they are first persisted in the local cluster, and then forwarded asynchronously to the remote clusters. Geo-replication must be enabled on a per-tenant basis in Pulsar. Geo-replication can be enabled between clusters only when a tenant has been created that allows access to both clusters.</p><p>For messaging channel security TLS based and JWT token based authorization mechanisms are supported natively. So you can specify who can publish or consume messages from which topics. Also for additional security Pulsar Encryption allows applications to encrypt all messages at the producer and decrypt at the consumer passing encrypted messages through Pulsar. Encryption is performed using the public/private key pair configured by the application. Encrypted messages can only be decrypted by consumers with a valid key. But remember this comes with a performance hit since every message needs to be encrypted and decrypted for processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*_lS1lJRl_U6fxf3rjCW8bw.jpeg" /></figure><p>Anyone who is using Kafka wants to migrate to Pulsar has a big relief because Pulsar natively support to work directly with Kafka data through a connector OR you have the option to import existing Kafka application data to Pulsar very easily.</p><p>It is also need to note that both Kafka and Pulsar both have enterprise support providers and they are Confluent and Streamlio respectively which provides production support and consultations if you prefer.</p><h3>Summary</h3><blockquote>So this is not to say Kafka is bad and the only option is Pulsar when it comes to a large scale messaging platform, but what is says is that some of the pain points that exists in Kafka is already handled by Pulsar for us, which is a great thing for any engineer or architect. Other most important point is due to the architectural aspects Pulsar is much much faster in large messaging solutions, as Yahoo and Twitter (and many others) already using the production environments message loads which is a proof that its stable and production ready for any business. But there is a little learning curve to move in and to mindset shift from Kafka (because we all were very proud to say that we are Kafka guys) to move to Pulsar based solutions, but there is a definite ROI for that !</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gFfCjTX12MM41E_qVpRpZg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9eb435c9fc0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Content based API authorization in 
WSO2 APIM]]></title>
            <link>https://anuradhaneo.medium.com/content-based-api-authorization-in-wso2-apim-c40238f9a43c?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c40238f9a43c</guid>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[content-based]]></category>
            <category><![CDATA[wso2-api-manager]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Fri, 21 Jun 2019 19:19:19 GMT</pubDate>
            <atom:updated>2019-06-22T10:53:44.845Z</atom:updated>
            <content:encoded><![CDATA[<h3>Content based API authorization with <br>WSO2 APIM</h3><h3>Requirement</h3><p><em>A public API exposed from WSO2 APIM was needed for an existing back-end API.</em></p><h3>Problem</h3><p><em>Existing backend API is a multi purpose core API that provides different services based on the request payload. This means anyone who has credentials to access this API will have access to all the functions it supports which can be pretty dangerous (Eg. if this is a banking API). So this API requires different authorization to access for different payloads. So a content based authorization model is required.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5-ZoxkJu2z0PZNfT" /></figure><h3>Solution !</h3><p><em>Validate the payload contents and few parameters in it and decide whether the calling party is allowed to access the API or not.</em></p><h3>Before dive in to solution design,</h3><p>First we must understand what is the problem in detail.</p><p>This API supports below operations</p><ul><li>Credit card settlement</li><li>Get credit card statement</li><li>Get account details</li><li>Get last 10 account transactions</li><li>Open new savings/current account</li></ul><p>These operations will be mapped to different users with the role</p><ul><li>Credit card settlement — Credit card user role</li><li>Get credit card statement — Credit card user role</li><li>Get account details — Bank account user role</li><li>Get last 10 account transactions — Bank account user role</li><li>Open new savings/current account — Bank officer role</li></ul><p>Below are the users of each role</p><ul><li>Credit card customer role — Tim, Gina</li><li>Bank account customer role — Ricky, Ellen</li><li>Bank officer role — Tony</li></ul><p>So you can see these different roles should be able to access different functionalities only permitted to their role. If you enable this API to the third-parties directly any user who has access credentials to the API can do anything. A credit card customer can create bank account. To make things worse lets think for now this API can also do bank transfers, hows that ? :)</p><h3>How did I implement the solution !</h3><p>For the simplicity and the time saving I created content based authorization policies in a WSO2 carbon registry entry so I can refer that in the “In mediation sequence” of the API to decide whether the API access if authorized or not based on the user calls the API.</p><p>I could also use a custom API handler to do the same thing(see how custom handlers are created in WSO2 documentation <a href="https://docs.wso2.com/display/AM260/Writing+Custom+Handlers">here</a>) but creating Java classes and deploying makes things little painful if you have to modify the policy logic where in a API mediation “In sequence” you can change logic on the fly, at least in a development/test environment (even though you wouldn&#39;t do that in a production environment) which makes my life easy and makes things simple.</p><p>Firstly I must say above methods are two ways to implement the solution, and this solution does not necessarily have to be the best or one of the best solutions. There are different other ways to implement this solution such as use a in-memory dat store such as Redis or Infinispan to store the authorization policies which makes the API response time much lesser compared to registry based policy solution. <br>Another solution is to define the content based authorization policies using XACML (eXtensible Access Control Markup Language). Deciding which method to use depends on how much time you have, what kind of a serious integration solution are you implementing or what are the quick technology skills you have handy to start off.</p><p>So let me explain the solution the way I did.</p><p>See below image that I have added extra two steps to the API manager request pass through process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G38rdVtb6wTIb441" /></figure><p>So they are in detail,</p><p><strong>Identify the payload —</strong></p><p>Extract the identifier properties from the payload so that I know how to handle it. In below example payloads shown in the image its the “type” property in the request payload</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*VVv9VeM8WHh0uXDqXZzurw.png" /><figcaption><strong>Tony’s Request</strong></figcaption></figure><p><strong>Validate the policy —</strong></p><p>This is basically check whether the identified request payload (&quot;type&quot;: &quot;open_new_account&quot; in above example) and see if that is allowed for the client who is calling the API. In this client (here the client can be a human user, mobile app, back office system or something else too) is mapped to the OAuth token because you don’t share a OAuth token with multiple parties if you do things right. But one party can have multiple OAuth tokens and this is not a problem. Tokens are always generated against an Application in API manager. So knowing the token from the incoming request means related Application is known.</p><blockquote>Client/User &gt; OAuth token &gt; Application</blockquote><p>Now the next step is to create authorization policy that maps to an Application, so with this it can be visualized in sequence as</p><blockquote>Client/User &gt; OAuth token &gt; Application &gt; Custom Authorization Policy</blockquote><p>Now if we define what is permitted or what is not allowed in this policy we are good to go with the solution.</p><p>Now lets try to “exampalize” this,</p><p>Tony is a bank officer who is in the “Bank Officer” role and access the API with the API token 3849–3be4–9322–612a04ab4 and this token is issued to the Application canned “BankOfficerAPIApp” and now above sequence becomes</p><blockquote>Tony &gt; 3849–3be4–9322–612a04ab4 &gt; BankOfficerAPIApp &gt; to below</blockquote><pre>&lt;Authorization&gt;<br>  &lt;Policy&gt;<br>    &lt;App&gt;BankOfficerAPIApp&lt;/App&gt;<br>    &lt;AllowedRequestTypes&gt;<br>      &lt;Type&gt;open_new_account&lt;/Type&gt;<br>    &lt;/AllowedRequestTypes&gt;<br>  &lt;/Policy&gt;<br>&lt;/Authorization&gt;</pre><p>now this says “Tony” can access the API with the request body with type=open_new_account</p><p>One important thing to remember — Custom policy name must be same as Application name (BankOfficerAPIApp in here).</p><p>Lets implement the solution,</p><blockquote><em>Note I am using WSO2 API Manager version 2.6.0</em></blockquote><p>Lets create the API in WSO2 APIM Publisher portal,<br>Below shows the complete API implementation steps and code</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C29GKEoBRlpWsORnaeu92Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Goy7wXr_ycdhY9MaM7UZRQ.png" /></figure><p>Firstly, Upload the code below in order to select the in flow sequence in above screen “BankCoreAPI_ValidatePayloadMethodSeq”</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;sequence name=&quot;BankCoreAPI_ValidatePayloadMethodSeq&quot; trace=&quot;disable&quot; xmlns=&quot;<a href="http://ws.apache.org/ns/synapse">http://ws.apache.org/ns/synapse</a>&quot;&gt;<br>    &lt;enrich description=&quot;Store original payload for later use&quot;&gt;<br>        &lt;source type=&quot;body&quot;/&gt;<br>        &lt;target property=&quot;request_payload&quot; type=&quot;property&quot;/&gt;<br>    &lt;/enrich&gt;<br>    &lt;property expression=&quot;//type&quot; name=&quot;request_type&quot; scope=&quot;default&quot; type=&quot;STRING&quot;/&gt;<br>    &lt;property expression=&quot;get-property(&#39;registry&#39;,&#39;conf:/resources/policies/banking_api_auth_policies.xml&#39;)&quot; name=&quot;resultOM&quot; scope=&quot;default&quot; type=&quot;OM&quot;/&gt;<br>    &lt;property expression=&quot;$ctx:resultOM//Authorization&quot; name=&quot;RootOM&quot; scope=&quot;default&quot; type=&quot;OM&quot;/&gt;<br>    &lt;enrich&gt;<br>        &lt;source clone=&quot;true&quot; property=&quot;RootOM&quot; type=&quot;property&quot;/&gt;<br>        &lt;target type=&quot;body&quot;/&gt;<br>    &lt;/enrich&gt;<br>    &lt;xslt key=&quot;conf:/resources/xslt_processors/policy_processor.xslt&quot;&gt;<br>        &lt;property expression=&quot;$ctx:api.ut.application.name&quot; name=&quot;api_application&quot;/&gt;<br>        &lt;resource key=&quot;conf:/resources/xslt_processors/policy_processor.xslt&quot; location=&quot;policy_processor.xslt&quot;/&gt;<br>    &lt;/xslt&gt;<br>    &lt;property expression=&quot;//App&quot; name=&quot;application&quot; scope=&quot;default&quot; type=&quot;STRING&quot;/&gt;<br>    &lt;property expression=&quot;//AllowedRequestTypes/Type&quot; name=&quot;allowed_types_collection&quot; scope=&quot;default&quot; type=&quot;STRING&quot;/&gt;<br>    &lt;filter xpath=&quot;get-property(&#39;application&#39;)=$ctx:api.ut.application.name&quot;&gt;<br>        &lt;then/&gt;<br>        &lt;else&gt;<br>            &lt;payloadFactory media-type=&quot;xml&quot;&gt;<br>                &lt;format&gt;<br>                    &lt;Fault&gt;<br>                        &lt;detail&gt;<br>                            &lt;ServerFault&gt;<br>                                &lt;error_code&gt;500900&lt;/error_code&gt;<br>                                &lt;error_message&gt;The client not configured for API authorization policies&lt;/error_message&gt;<br>                            &lt;/ServerFault&gt;<br>                        &lt;/detail&gt;<br>                    &lt;/Fault&gt;<br>                &lt;/format&gt;<br>            &lt;/payloadFactory&gt;<br>            &lt;property name=&quot;HTTP_SC&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;501&quot;/&gt;<br>            &lt;property name=&quot;RESPONSE&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;true&quot;/&gt;<br>            &lt;header action=&quot;remove&quot; name=&quot;To&quot; scope=&quot;default&quot;/&gt;<br>            &lt;property name=&quot;messageType&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;application/json&quot;/&gt;<br>            &lt;respond/&gt;<br>        &lt;/else&gt;<br>    &lt;/filter&gt;<br>    &lt;filter regex=&quot;true&quot; source=&quot;fn:contains(get-property(&#39;allowed_types_collection&#39;),get-property(&#39;request_type&#39;))&quot;&gt;<br>        &lt;then/&gt;<br>        &lt;else&gt;<br>            &lt;payloadFactory media-type=&quot;xml&quot;&gt;<br>                &lt;format&gt;<br>                    &lt;Fault&gt;<br>                        &lt;detail&gt;<br>                            &lt;ServerFault&gt;<br>                                &lt;error_code&gt;500950&lt;/error_code&gt;<br>                                &lt;error_message&gt;Client request is not authorized to call the API&lt;/error_message&gt;<br>                            &lt;/ServerFault&gt;<br>                        &lt;/detail&gt;<br>                    &lt;/Fault&gt;<br>                &lt;/format&gt;<br>                &lt;args/&gt;<br>            &lt;/payloadFactory&gt;<br>            &lt;property name=&quot;HTTP_SC&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;401&quot;/&gt;<br>            &lt;property name=&quot;RESPONSE&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;true&quot;/&gt;<br>            &lt;header action=&quot;remove&quot; name=&quot;To&quot; scope=&quot;default&quot;/&gt;<br>            &lt;property name=&quot;messageType&quot; scope=&quot;axis2&quot; type=&quot;STRING&quot; value=&quot;application/json&quot;/&gt;<br>            &lt;respond/&gt;<br>        &lt;/else&gt;<br>    &lt;/filter&gt;<br>    &lt;enrich description=&quot;Restore original payload&quot;&gt;<br>        &lt;source clone=&quot;false&quot; property=&quot;request_payload&quot; type=&quot;property&quot;/&gt;<br>        &lt;target type=&quot;body&quot;/&gt;<br>    &lt;/enrich&gt;<br>&lt;/sequence&gt;</pre><p>You can quickly create a backend service mock to test this with any mock framework you like, I like Mockoon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fXpjLrv9l-P5G98lmlmI2Q.png" /></figure><p>Save and Publish the API.</p><p>When the API is created it can be accessed with below API URI <a href="https://localhost:8243//services/bankingcore/v1">https://localhost:8243/services/bankingcore/v1</a>/</p><p>Now,</p><p>Lets create the necessary registry resources which are dependencies for the “BankCoreAPI_ValidatePayloadMethodSeq” in flow sequence we used</p><p>create the file, banking_api_auth_policies.xml with below content, this specifies which Applications/roles are allowed to access which request types</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;localEntry key=&quot;MyAPI_Auth_Policies&quot;&gt;<br> &lt;Authorization&gt;<br>     &lt;Policy&gt;<br>         &lt;App&gt;BankOfficerAPIApp&lt;/App&gt;<br>         &lt;AllowedRequestTypes&gt;<br>             &lt;Type&gt;open_new_account&lt;/Type&gt;<br>         &lt;/AllowedRequestTypes&gt;<br>     &lt;/Policy&gt;<br>     &lt;Policy&gt;<br>         &lt;App&gt;BankAccountCustomerAPIApp&lt;/App&gt;<br>         &lt;AllowedRequestTypes&gt;<br>             &lt;Type&gt;account_details&lt;/Type&gt;<br>             &lt;Type&gt;last_ten_transactions&lt;/Type&gt;<br>         &lt;/AllowedRequestTypes&gt;<br>     &lt;/Policy&gt;<br>     &lt;Policy&gt;<br>         &lt;App&gt;CreditCardCustomerAPIApp&lt;/App&gt;<br>         &lt;AllowedRequestTypes&gt;<br>             &lt;Type&gt;creditcard_settlement&lt;/Type&gt;<br>             &lt;Type&gt;creditcard_statement&lt;/Type&gt;<br>         &lt;/AllowedRequestTypes&gt;<br>     &lt;/Policy&gt;<br> &lt;/Authorization&gt;<br>&lt;/localEntry&gt;</pre><p>and <br>create the file policy_processor.xslt with content below</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;xsl:stylesheet version=&quot;1.0&quot; xmlns:xsl=&quot;<a href="http://www.w3.org/1999/XSL/Transform">http://www.w3.org/1999/XSL/Transform</a>&quot;&gt;<br> &lt;xsl:param name=&quot;api_application&quot;/&gt;<br> &lt;xsl:template match=&quot;/&quot;&gt;<br>  &lt;root xmlns=&quot; &quot;&gt;<br>        &lt;xsl:for-each select=&quot;//Policy&quot;&gt;<br>   &lt;xsl:if test=&quot;App=$api_application&quot;&gt;<br>           &lt;xsl:copy-of select=&quot;.&quot;/&gt;<br>   &lt;/xsl:if&gt;<br>        &lt;/xsl:for-each&gt;<br>      &lt;/root&gt;<br> &lt;/xsl:template&gt;<br>&lt;/xsl:stylesheet&gt;</pre><p>Upload both of these files to paths in registry</p><blockquote>banking_api_auth_policies.xml &gt; /_system/config/resources/policies</blockquote><blockquote>policy_processor.xslt &gt; /_system/config/resources/xslt_processors</blockquote><p>Finally,</p><p>Below screen shows how the API Application is created, Notice the Application name is “BankOfficerAPIApp”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2b2s3diM8CLoHUXq03RChQ.png" /></figure><p>Now generate OAuth tokens for this Application.</p><p>Subscribe the new API BankingCoreAPI to the new Application BankOfficerAPIApp.</p><p>You can create different API Applications like above steps shows for the other client roles “BankAccountCustomer” and “CreditCardCustomer”</p><p>Now, to test</p><ol><li>Access the API with Tony’s request with the OAuth token generated for the “BankOfficerAPIApp”, this becomes successful !</li><li>Access the API with Tony’s request with the OAuth token generated for “BankAccountCustomer”, this becomes failed and you will get a HTTP 401 error because the token for is not allowed to call the requests with type “open_new_account”</li></ol><p>This concept can be applies to any such high level requirement with slight modifications to correctly identify which users authorized to access which content and operations.</p><p>Hope you have learnt something useful !!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyD9uAvCfOPL3ixwqVGtgQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c40238f9a43c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Sustaining Engineering Culture]]></title>
            <link>https://anuradhaneo.medium.com/a-sustaining-engineering-culture-72ad0d32caf9?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ad0d32caf9</guid>
            <category><![CDATA[employees]]></category>
            <category><![CDATA[success]]></category>
            <category><![CDATA[sustainability]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[engineer]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Sat, 23 Feb 2019 20:43:11 GMT</pubDate>
            <atom:updated>2019-02-24T11:40:27.028Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been working in multiple engineering organizations and they were small to global billion dollar (covering about 75% of a spread across the globe) sized enterprises. Point is I have been a part of various successful and failed engineering cultures. So in this post my experience talk !!!</p><p>Most of us think “working cultures” as an unclear, difficult to understand, difficult to control or fix and/or manipulated phenomenon that exists within any work environment. But one thing is definite, Its all about people. Humans define any culture.</p><p>I will be mainly talking about IT/Software based company culture because this is my experience area, but this can be easily converted and interpreted for any business organization culture as you read through because the concepts are the same for any organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_CvaDPz3XF-tMUXKNSdnw.png" /></figure><p>A good culture is like a rich soil bed for growing seeds. It helps the final outcome of company which are products and services delivered quickly meeting the customer expectation (or exceeding). But in any organization people come from different backgrounds and from diverse corporate cultures. Unless driven and influenced towards a better unified culture it will always be a subjective and group centric mixed culture which does not help company grow towards its corporate goals in most cases. <br>But this may be false in some cases, for an example employees in some countries/regions are trained to focus towards productivity, efficiency, adapting to best practices, collaborate, innovate, enjoy the work from their early education life which defines cultures nothing but to seek and improving above properties. When there is a common focus in above said properties and when it becomes an integral piece of corporate culture it becomes a cultural driver too and influence others inwards the same.</p><p>In most of the situations regulations, policies and mandatory processes generate a negative thought process, discouraged employees and impact job satisfaction. By experience, this is specially true for organizations with professionals are employed.</p><p>If above methods do affect an unsustainable and deteriorating culture what options do we have in front of us to redesign a sustaining office culture ?<br>Remember, once again culture is all about people.</p><p>There are many methods we can incubate in an organization</p><ul><li><strong>Cultivate a critical thinking environment</strong><br>Define an innovative thinking culture so that engineers themselves become architects of their work. Create thought processes in cross-cutting aspects and make different set of employees own it, in software companies we can think of this layer as engineers with DevOps, CloudOps , security and Database Admins skills etc. When established these cross skilled matrix of employees working together as a team who are innovative thinkers rather than those who think of deliver what ever assigned to them with what they know so far only becomes and R&amp;D (research and development) enabled, organizational best practice defining delivery team who wants bring nothing but top of the game success because the always want to deliver best and want to hold the ownership of the quality of the work they have done.<br>Make sure to select the teams carefully to add around a minimum of 50% of engineers of this category including the leads so that they will train the young engineers in the team in same way and influence them continuously.</li><li><strong>Energize the working environment</strong><br>Make the environment happening and fun to work. This works back and forth when properly cultivated. Use gamification to get engineers to achieve targets within reachability. Do not set gamification in a way that always a genius or high performing set of people only win and that itself act as a demotivating factor to the 90% of the engineering members. <br>Initiate programs out of the practices they are in such as pet projects, CSR projects, fun event organizing so that people become energetic and love to work with different members across organization.<br>Don’t be shy and scared to do crazy and out of the box things, employees define the level of fun they need in the working environment. Now the management should look at this in a positive manner and should be thinking constructively and support it by not being too traditional.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZQmZ2Z4qEYilkf_p4tJAw.png" /></figure><ul><li><strong>Build relationships</strong><br>First of all employees and engineers should feel comfortable and they are being cared in the work place. There should not be senior level people that scares and command below levels. In fact it is unnecessary and lead to frustrations and failures. Talk to engineers and juniors after you getting in to their shoes. Always talk like a team, use “we” instead of “I” or “You”. Check the work in progress and help and guide them if necessary. Make others feel that you are on their side while you know you are on the correct path.</li><li><strong>Seed a learning environment</strong><br>Build an eager to learn and knowledge sharing environment. This can be initiated with few people but need consistent expansion. Make the thirst to learn a contagious thing. Start influencing engineers to R&amp;D, BYOT (Bring your own technology), bring best practices, positively debating discussions, objective setting to get exams/certifications done etc. Make sure you use the engineers who follow this to be used as the ambassadors and evangelists to promote and keep the learning happening consistently and increasing fashion. Few people with genuine passion and motivation can drive this behind the scene in the long go.<br>Start a reward program to recognize their achievements. Generate opportunities to use the learnings and give chances to engineers to solve problems with their new methodologies and best practices they’ve learnt.<br>Seeding and forming a learning environment builds a sustaining learning culture.</li><li><strong>Flattern the employee structure and communication layers</strong><br>Unless in the projects where some level of a structure is required due to the responsibility assignment, Think of flat employee structure in terms of relationships and interactions so that even the CXO levels can talk to interns and vice versa. This also helps building trust and relationships between different levels and teams and also help to get insights of smallest problems that are sometimes not visible to the directly involved team so that necessary remedies can be found and applied without its too late.</li><li><strong>Stop managing or bossing around, but start guiding and helping achieving duties and deliveries<br></strong>To help you understand this let me add a real conversation I had with a engineer named “Chum” falling under my mentorship (mind this is a technical conversation)<br><em>Myself</em>: Chum, I have this great idea that I want to explore this new tool called <a href="https://wso2.com/api-management/api-microgateway/">WSO2 API MicroGateway</a> and to be the first to do that in the company, its a cool tool that fits the modern day high scalable application requirements which is the most clients requirements as of today. I need to do a PoC (Proof of Concept) to explore how it works and do a comparison of it with other similar technologies we use. You know what, I also thought I am going to involve you too to get your help on doing this because I think together <strong>we</strong> can do this better. What do you think ?<br><em>Chum</em>: Wow, I am very interested to be a part of this. Tel me how do we do this ?<br><em>Myself</em>: Ok, I have already done some some basic tests around it and I can share my knowledge, what you need to do is that to the next level and explore inside out and see how you can run a complete end to end solution as a PoC so that we know how it works and how we can configure the product. Then I will do a performance test of the product to see its performance statistics, as next step we can deploy that in a <a href="https://kubernetes.io/">Kubernetes</a> environment see how it behaves in a high client traffic situation in a production environment.<br><em>Chum</em>: Ok, I can finish my assigned work quickly and start tomorrow itself.<br><em>Myself</em>: I will make sure that you will not get loaded with other work and also I will make sure to get a chance for you to present a report of the findings to the management so that you get a recognition for the work.<br><em>Chum</em>: Thanks a lot and I am exited to work on this !<br><br>So this is a classic example that you can team together with below level employees and get them to trust you and work as a flat team to achieve short team goals while you guide them.</li><li><strong>Empower employees</strong><br>Empower engineers always to do their next level things. For an example <br>~ Trust and give chances to own assignments and deliver them<br>~ Enable to deal with customers directly<br>~ Present research reports to senior management<br>~ Act as the next level senior members in case of their absence</li><li><strong>Allow freedom and authority to take necessary technical decisions</strong><br>Empower and delegate engineers and teams to take technical decisions what is best for their team, their customer and to use the latest and leading technologies rather than a organizational level super-engineering-authority decide something they don’t have the context of and which consumes time to gather around the panel.</li><li><strong>Celebrate success together</strong><br>Think of celebrating even the smallest success. Don’t think of avoiding small things you can organize with minimum to no-budget. Think of a successful completion of an <a href="https://www.atlassian.com/agile/scrum/sprints">Agile Sprint</a> by taking a call with the customer and give a round of applause to the hard work of the team which is just a simple no-budget approach. Or go for a lunch out to a near by restaurant which you can do with minimum budget because you’d anyway buy lunch.</li><li><strong>Appreciate achievements and contributions</strong><br>Recognize employees who contribute in various ways to the success of the organization and appreciate and reward them accordingly.</li><li><strong>Break employee silos and diversify employee engagements</strong><br>Always-together teams or practice silos are the reason behind corporate politics sometimes. Also this defines clear separations of teams and disconnect of the culture to form mini-cultures which is a very unpleasant thing. Mixed engineer teams in assignments where possible to break the silo effect and this is also good from another perspective which is an opening to the room to learn from different engineers and share knowledge.</li><li><strong>Own your subordinates and help them win but don’t act as they should working under your command</strong><br>Think your subordinates or mentees and your children and help them learn and train them so that even exceed you at someday. <br><em>“You are not a good leader until you make leaders who exceed you”</em></li><li><strong>Provide enough transparency</strong><br>Have clarity on all aspects to all employees and effectively communicate organizational news and changes and enable a forum to provide feedback and ask further questions to avoid rumors, individual employee assumptions and incorrect individual employee decisions</li></ul><p>To do all above, it is essentials that the full understanding and blessing of the senior management and decision makers of how a healthy and positively growing organizational culture can bring immense success to the business and business values with sustaining long term employee and customer satisfaction. This requires employee empowerment, some initiatives and perhaps budget allocations as well. Most importantly decision makers should have a penetrating/transparent vision to place right people (hire if not exist) at right roles.</p><p>Truth is, In any organization there are always employees who change their employment regularly because they are never satisfied with the opportunity and the perks they get, so they keep on changing jobs continuously and going after this un-quenching thirst. Also there are some employees who are aggressive and resistant to new changes and adaption to new methodologies and accepting new challenges because of their nature. Organizations should recognize the group of engineers other than these two groups to initiate and cultivate a sustaining successful culture to innovate on their products and services and to help their customers succeed.</p><p>On the other hand, employees should always think of improving and innovating what they do and assigned for to bring the best to the clients, organization and at last but not definite the least to take their skills to the next levels and to enhance their career too.</p><p>Further, employees and management should look at matters from different angles in contrast to usual and traditional ways of thinking because that could provide different and unseen insights and understanding to take correct decisions and improvements in timely fashion.</p><h3><strong>Schneider Model for Culture Assessment</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*mfoZabua4qpl6BN2" /><figcaption>Schneider Model for Culture Assessment</figcaption></figure><h3>4 Culture Types</h3><p>Control cultures (COMPANY/REALITY oriented) <br>Process-driven; the org’s success depends on data, processes, etc. Many energy, aviation, defense companies are control cultures. Control cultures want no competition — they want to be the only game in town. Leaders manage the work; control cultures are command-and-control, hierarchical.</p><p>Collaboration cultures (PEOPLE/REALITY oriented)<br>People work together toward a shared goal. Here you’ll find “working managers”. Everyone rolls up their sleeves. Relationships are key to getting things accomplished.</p><p>Cultivation Cultures (PEOPLE/POSSIBILITY oriented) <br>Often centered around a greater mission; examples include religious organizations, non-profits, social impact organizations. Leaders remove obstacles that impede attaining the org’s mission.</p><p>Competence Cultures (COMPANY/POSSIBILITY oriented) <br>Innovative (possibility) and are utilize the best talent to bring ideas to bear. This culture is about being the best and winning — best product, most market share, etc. Leaders are results-driven and will hire and fire to do what needs to be done.</p><blockquote><strong>Advisory Tip</strong></blockquote><blockquote><strong>If your organization is in top-right cell (Control cultures), strategically plan to either move to bottom-right (Competence) or to top-left (Collaboration) for a sustainable culture !!!</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ad0d32caf9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate the CI/CD Automation with Jenkins (Part 2)]]></title>
            <link>https://anuradhaneo.medium.com/automate-the-ci-cd-automation-with-jenkins-part-2-fcd987456ba8?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcd987456ba8</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[automate]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[job-dsl]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Mon, 17 Sep 2018 09:08:12 GMT</pubDate>
            <atom:updated>2018-09-17T09:08:12.235Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have not read the first part of this article you can read from <a href="https://medium.com/@anuradha.neo/automate-the-ci-cd-automation-with-jenkins-part-1-623a2c532848">here</a></p><p>Ok now with part one read, I believe you have a good understanding around the concepts and Jenkins Job DSL plugin installed on your Jenkins instance.<br>Lets talk about the steps on how to use the plugin to automate the Jenkins jobs and pipelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yvddO8NuvVKCAeKF7gQazg.png" /></figure><p>In this article we are going to talk about the “Seed Jobs”.<br>Seed job is the top level Jenkins job that will generate the usual Jenkins CI/CD jobs. In other words, Seed job will run Groovy based Job DSL scripts and will generate all the CI/CD pipelines and jobs that you need to have for your projects. In Job DSL scripts you need to specify how the CI/CD jobs will look like.</p><p>We are going to generate a Jenkins job that when run builds a sample Java Springboot app and upload the artifact to a Nexus artifact repository (<a href="https://www.sonatype.com/nexus-repository-oss">https://www.sonatype.com/nexus-repository-oss</a>). Job configuration will look like this. You can create this job by hand using the UI, but thats not the way to do it. The correct way to create this job is to write this job as a Job DSL script that will generate the below job when run. i.e. you run the job DSL script and then you’ll get the below Jenkins job automatically generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGWoXw9v-RWEUAm_hrDi6g.png" /><figcaption>image 1</figcaption></figure><p>Important points in above sample Jenkins Job.</p><ul><li>Clone a Java project from github (we use a sample <a href="https://github.com/anuradhaneo/springboot-api-demo">https://github.com/anuradhaneo/springboot-api-demo</a>)</li><li>Fetch the code from github every day</li><li>Build the java project as a Maven job with goals “clean” and “install”</li><li>With Maven build there will be a artifact generated named target/SpringBootRestApiExample-1.0.jar</li><li>Upload the build artifact to a locally hosted Nexus OSS repository</li></ul><p>Think this is one of our CI/CD jobs for one of our projects, and we don’t want to create this job by hand manually. We need to use Job DSL tool to automate the Jenkins job creation.</p><p>Lets create the Seed Project fist.</p><p>Create a new Freestyle project, and lets call it “Seed Project”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjhCKwbVJyewCqdii5RVLg.png" /><figcaption>image 2</figcaption></figure><p>Open the new project and go to “Configure” section from the menu.</p><p>Just scroll down to the “Build” section and add a “Build Step” called “Process Job DSLs”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*g6RMNh_tRdq35kTDdUgYSA.png" /><figcaption>image 3</figcaption></figure><p>Now select the “Use the provided DSL script” option and type the DSL commands as shown below in the large textbox that appear. Don’t worry about the commands and we will talk about them in a minute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95Kjg7_tZSekEA-KtQO3mQ.png" /><figcaption>image 4</figcaption></figure><p>Save the project, thats it nothing more. WOW, its that simple to generate a seed job.</p><p>Below is the DSL Groovy commands used in the above image.</p><pre>// Jenkins Job DSL definition file for generating a Job for <br>// https://github.com/anuradhaneo/springboot-api-demo<br><br>String basePath = &#39;springboot-api-demo&#39;<br>String gitRepository = &#39;anuradhaneo/springboot-api-demo&#39;<br>String buildBranch = &#39;*/master&#39;<br>String projectName = &#39;Build Project&#39;<br>String projectDisplayName = &#39;Springboot API Sample&#39;<br>String projectDescription = &#39;This example shows basic folder/job creation&#39;<br>String credentialIDGithub = &#39;github-anuradhaneo&#39;</pre><pre>String artifactGroupID = &#39;org.spring.boot.sample&#39;<br>String artifactID = &#39;SpringBootRestApiExample&#39;</pre><pre>final String HTTP = &#39;http&#39;<br>final String HTTPS = &#39;https&#39;</pre><pre>// root folder creation<br>folder(basePath) {<br>    displayName(projectDisplayName)<br>    description(projectDescription)<br>}</pre><pre>// job definition<br>mavenJob(basePath + &#39;/&#39; + projectName) {<br>    description(&#39;Build the Java roject: &#39; + gitRepository)<br>    scm {<br>        git {<br>            branch(buildBranch)<br>            remote {<br>                github (gitRepository, HTTPS, &#39;github.com&#39;)<br>                credentials(credentialIDGithub)<br>            }<br>        }<br>    }<br>    triggers {<br>        scm(&#39;<a href="http://twitter.com/daily">@daily</a>&#39;)<br>    }<br>    wrappers {<br>        goals(&#39;clean install&#39;)<br>    }<br>}</pre><p>If you are familiar with Java or Groovy the line 1 to 14 are nothing but variables with values.</p><p>Line 16 to 19 defines a folder in Jenkins to organize the jobs and we will be placing our Jenkins job (when generated after running Seed job) inside a folder named “Springboot API Sample” which is defined with the line number 17.</p><pre>folder(basePath) {<br>    displayName(projectDisplayName)<br>    description(projectDescription)<br>}</pre><p>Line 22 to 39 defines the Maven build steps</p><p>Line 22 specifies the name of the project, <br>note in below we also specify the full path to the project including the parent folder we created earlier</p><pre>// set the project name<br>mavenJob(basePath + &#39;/&#39; + projectName) {</pre><p>Line 23 sets the description</p><pre>description(&#39;Build the Java project: &#39; + gitRepository)</pre><p>line 26 sets the branch in the github project</p><pre>branch(buildBranch)</pre><p>Line 28 specifies the github.com repository name</p><pre>github (gitRepository, HTTPS, &#39;github.com&#39;)</pre><p>Line 33 specifies the build frequency</p><pre>scm(&#39;@daily&#39;)</pre><p>Line 37 specifies the Maven goals to be executed</p><pre>goals(&#39;clean install&#39;)</pre><p>Use the Job DSL API to learn more about the commands</p><p><a href="https://jenkinsci.github.io/job-dsl-plugin/">https://jenkinsci.github.io/job-dsl-plugin/</a></p><p>Now if you run this Seed Project it will generate a project with configuration as below</p><ol><li>Parent folder</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*73i_WSrZ_5ntHlP86qqedA.png" /><figcaption>image 5</figcaption></figure><p>2. Actual job inside the above shown folder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3sEcJu_SG6VZGlkOqHvB_g.png" /><figcaption>image 6</figcaption></figure><p>3. with job configuration as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ptcaViiJGF60-LA2gyWnTg.png" /><figcaption>image 7</figcaption></figure><p>So as you see it is very simple to create a Seed Job with a simple DSL script to generate a single Jenkins Job.</p><p>Now lets look at a extended version of the above example code</p><pre>// Jenkins Job DSL definition file for generating a Job for <br>// https://github.com/anuradhaneo/springboot-api-demo<br><br>String basePath = &#39;springboot-api-demo&#39;<br>String gitRepository = &#39;anuradhaneo/springboot-api-demo&#39;<br>String buildBranch = &#39;*/master&#39;<br>String projectName = &#39;Build Project&#39;<br>String projectDisplayName = &#39;Springboot API Sample&#39;<br>String projectDescription = &#39;This example shows basic folder/job creation&#39;<br>String credentialIDGithub = &#39;github-anuradhaneo&#39;</pre><pre>String artifactGroupID = &#39;org.spring.boot.sample&#39;<br>String artifactID = &#39;SpringBootRestApiExample&#39;</pre><pre>final String STATUS_SUCCESS = &#39;SUCCESS&#39;<br>final String HTTP = &#39;http&#39;<br>final String HTTPS = &#39;https&#39;</pre><pre>String nexusOSSURI = &#39;localhost:8081&#39;<br>String nexusOSSVersion = &#39;nexus3&#39;<br>String nexusOSSRepositoryName = &#39;SpringBootRestApiProject&#39;<br>String nexusCredentialsID = &#39;NexusRepoCredentials&#39;</pre><pre>// root folder creation<br>folder(basePath) {<br>    displayName(projectDisplayName)<br>    description(projectDescription)<br>}</pre><pre>// job definition<br>mavenJob(basePath + &#39;/&#39; + projectName) {<br>    description(&#39;Build the Java roject: &#39; + gitRepository)<br>    scm {<br>        git {<br>            branch(buildBranch)<br>            remote {<br>                github (gitRepository, HTTPS, &#39;github.com&#39;)<br>                credentials(credentialIDGithub)<br>            }<br>        }<br>    }<br>    triggers {<br>        scm(&#39;<a href="http://twitter.com/daily">@daily</a>&#39;)<br>    }<br>    wrappers {<br>        goals(&#39;clean install&#39;)<br>    }<br>    postBuildSteps(STATUS_SUCCESS) {<br>        // upload the generated artifact to Nexus OSS repo<br>        nexusArtifactUploader {<br>            nexusVersion(nexusOSSVersion)<br>            protocol(HTTP)<br>            nexusUrl(nexusOSSURI)<br>            groupId(artifactGroupID)<br>            version(&#39;${POM_VERSION}&#39;)<br>            repository(nexusOSSRepositoryName)<br>            credentialsId(nexusCredentialsID)<br>            artifact {<br>                artifactId(artifactID)<br>                type(&#39;jar&#39;)<br>                classifier(&#39;&#39;)<br>                file(&#39;target/&#39; + artifactID + &#39;-${POM_VERSION}.jar&#39;)<br>            }<br>        }<br>    }</pre><pre>}</pre><p>We have added a new section to the code</p><pre>postBuildSteps(STATUS_SUCCESS) { .... }</pre><p>This adds a post-build step which is similar to the below shown UI action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-QJOl_n3J_-xM45Rr7AAg.png" /><figcaption>image 7</figcaption></figure><p>Lets explore the code within the “postBuildSteps” tag.</p><pre>nexusArtifactUploader { ... }</pre><p>Above tag specifies the scope for the required parameters for artifact uploading to Nexus artifact repo with as shown below. I have a locally running Nexus OSS server exposed via port 8081.</p><pre>-- sets the version of the Nexus OSS software<br>nexusVersion(nexusOSSVersion)</pre><pre>-- set the access protocol<br>protocol(HTTP)</pre><pre>-- set Nexus server URI<br>nexusUrl(nexusOSSURI)</pre><pre>-- set uploading artifact group ID<br>groupId(artifactGroupID)</pre><pre>-- set artifact version<br>version(&#39;${POM_VERSION}&#39;)</pre><pre>-- set the repository name within Nexus<br>repository(nexusOSSRepositoryName)</pre><pre>-- credentials configuration ID<br>credentialsId(nexusCredentialsID)</pre><pre>-- actual upload artifact file details<br>artifact {<br>  artifactId(artifactID)<br>  type(&#39;jar&#39;)  <br>  classifier(&#39;&#39;)<br>  file(&#39;target/&#39; + artifactID + &#39;-${POM_VERSION}.jar&#39;)<br>}</pre><p>See <a href="https://github.com/jenkinsci/nexus-artifact-uploader-plugin">https://github.com/jenkinsci/nexus-artifact-uploader-plugin</a> for more details if you are interested.</p><p>So if you run this Seed job the result will be something very similar to the Jenkins job I have shown in “image 1”.</p><p>Now lets see how we get rid of defining the Job DSL scripts inline in the build step of the seed project and get that code from a SCM repository such as github.</p><p>I have created a github repository with the full working code in below URL <a href="https://github.com/anuradhaneo/jenkins-job-automation-sample">https://github.com/anuradhaneo/jenkins-job-automation-sample</a></p><p>We are going to use this git repository as the source for the DSL code in out seed project.</p><p>Step1.<br>Open the seed project configuration again and set the project as a GIT based one as shown below. Fill the values as shown below.</p><p>Note you need to manually create the credential sets in Jenkins in all cases since this is something that shouldn’t be automated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V8Aqae86C1494ZEIuDCYLg.png" /><figcaption>image 8</figcaption></figure><p>Step2.<br>Select the “Look on the Filesystem” option from the “Process Job DSLs” build step options. And type the path to the script file from the git checkout folder.</p><p>Note: we can use a relative path here</p><p>In our example its “<em>job1-spring-boot-api-sample/main.groovy”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIgqPzlijEFHxcnTPiiT9w.png" /><figcaption>image 9</figcaption></figure><p>So when you run the Seed project main.groovy file will be executed.</p><p>Save and exit the Seed project and you will see the project in “image 1&quot;.</p><p>So thats all and you can see how easy it it to create Seed projects and generate your whole Jenkins CI/CD pipelines.</p><p>All you need to master here is how to read and apply the properties defined in Job DSL API documentation.</p><blockquote>&lt;Happy Coding&gt;</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcd987456ba8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate the CI/CD Automation with Jenkins (Part 1)]]></title>
            <link>https://anuradhaneo.medium.com/automate-the-ci-cd-automation-with-jenkins-part-1-623a2c532848?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/623a2c532848</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[automated]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Thu, 13 Sep 2018 19:11:21 GMT</pubDate>
            <atom:updated>2018-09-17T09:11:06.804Z</atom:updated>
            <content:encoded><![CDATA[<p>Lets cut the crap and break it down real quickly !</p><p><strong>CI/CD</strong> is Continuous Integration and Continuous Delivery.</p><p>CI — Do frequent builds and run unit tests to make sure new changes wont break anything.</p><p>CD — Do regular frequent deployments and run tests to QA/UAT/PreProd and Prod.</p><p>CI/CD Automation is essentially automate this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*xr6dyv8H2Jfrjq_y.png" /></figure><p><strong>How ?</strong> Using a tool.</p><p><strong>Like what ?</strong> Like Jenkins, which is the most popular (of course there is other plethora of tools available like Travis, Circle CI, Bamboo, TeamCity etc etc…)</p><p><strong>Then what ?</strong> Then every time (or any other criteria like time based builds) someone make a change to the code CI process will run. And every time you create a new version it will be deployed to UAT and automated tests will be running. <br>* Well for those of you who do not know we don’t deploy to production just like that directly as there are processes and approval steps that comes in between, but yet this can be fully automated too as a part of CD.</p><p><strong>But how with Jenkins ?</strong> You create Jenkins jobs for each and every process you want to automate.<br>Example: create a Jenkins job that listens to GitHub.com repo, ans whenever there is a commit to the repo it will trigger a build , lets say a Maven build. Also there are all different kind of other jobs for deployments to environments and run tests, copy build artifacts to different locations, send build artifacts for static code analysis (Eg. SonarQube, Codacy) or even integrate to services like AWS services to do various other things.</p><p><strong>So you have to create the jobs right ?</strong> Yes, of course.<br><strong>By hand ?</strong> Of course, with Jenkins UI, easy-peasy right ? Ummm… not really.</p><p>Have you ever worked in a large scale project (playing a DeOps role) for a serious client that had number of deployment environments, with multiple analytical stops for the tests and analysis of the code/functionality, different authorization and approval steps to perform for the deployments ? <br>How many jobs would you need to create for all this ?<br>Oh, I forgot to mention about the branching strategy in a typical multi developer project (SCM branching), heard about that branches like development, master, hotfix, release and various dozens of feature branches ? well this would multiply the number of jobs in another multiple folds.</p><p>So let me ask the magic question again.</p><p><strong>So you have to create the jobs right ?</strong> Yes, of course.<br><strong>By hand ?</strong> Hell NO, don’t even think about it.</p><p>Let me bring in another more point to make that “Hell No” a double.<br>How about accidental deletion of a Job, or you have messed with the configuration and you don’t know how to correct that ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/0*3b3bl3b57OoyA3Gx.png" /></figure><p>OK, so looks like when it comes to real projects and when you want to become a real DevOps guy that manual Jenkins job creation does not work at all.</p><p>But then what is the solution for this ?</p><p>Ladies and Gentleman ! “Jenkins Job DSL plugin”</p><p>How can this guy become a remedy ? well, it lets you write Jenkins jobs and pipelins as code (<em>Configuration as code</em> — CaC), so its a CaC tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/0*mXtT9vQlCvkDAnFC.jpg" /></figure><p>Jenkins DSL plugin is a open sourced Apache-2.0 licensed project available in github (<a href="https://github.com/jenkinsci/job-dsl-plugin">https://github.com/jenkinsci/job-dsl-plugin</a>)</p><p>Anything that you create from UI can be written as a Groovy language based code and this code when ran generates the Jenkins Jobs/Pipelines etc.</p><p>Whats so cool about this tool ?</p><ul><li>Write CaC scripts once, generate configurations many time you need in a single click.</li><li>You never make mistakes like you mess with working jobs in to a faulty ones.</li><li>You never make mistakes with production jobs that can do catastrophic failures in production environment or deploy wrong versions.</li><li>You can version control the CaC code (well, the Jenkins jobs in fact), so you know the full history of changes.</li><li>This tool comes with a cool authorization policy, that when ever a configuration code is changed it will need a manual approval of the new code to be executed.</li><li>you can reuse CaC code in many projects with only minor modifications.</li><li>Accidental deletions of Jenkins jobs, not a problem anymore.</li><li>These CaC code is human readable and easy to understand , specially for this who are familiar with Java and YAML.</li></ul><p>So why did I call this post “Automate the CI/CD Automation”, <br>“Automation” twice is a mistake ? nope (actually some one I did a session to asked me this question personally).<br>Well, Jenkins is a tool that automates the DevOps CI/CD process, how as automated jobs and pipelines. So when you automate the creation (actually automation and standardization) of this in fact it becomes “automate the automation”.</p><p>Ok, now,</p><p>How to use this plugin or CaC tool and take the full advantage ?</p><p>First lets look at how to install the plugin to Jenkins.</p><p>Download the latest version of the plugin from <br><a href="https://updates.jenkins-ci.org/latest/job-dsl.hpi">https://updates.jenkins-ci.org/latest/job-dsl.hpi</a></p><p>Go to Manage jenkins &gt; Manage Plugins section</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*zcLiIqLWhwzn-hIeXAR1JQ.png" /></figure><p>Go to Advanced tab, <br>Upload the plugin file (job-dsl.hpi) from the “Upload Plugin” section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4AzaGxJrH091gpB11UjTg.png" /></figure><p>Go to “Installed” tab in the same page and search for “DSL” in searchbox, you will see the latest version installed (v1.70 is the latest as of the time of writing)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cF4hZ7EK2GLpqxijk7VK1Q.png" /></figure><p>Installation is Done !</p><p>I will scope out the usage of the plugin (Part 2 of this post), i.e. how to write Jenkins config as DSL code in to a different post because I think that way the readers with basic knowledge in Jenkins and those who are eager to directly jump in to usage can use that post instead directly.</p><p><em>I will be posting the full technical steps on how to use the tool to fully automate the Jenkins CI/CD automation pipelines within coming two days and post the link to the new post here. <br></em><strong><em>UPDATE: Part two can be read at </em></strong><a href="https://medium.com/@anuradha.neo/automate-the-ci-cd-automation-with-jenkins-part-2-fcd987456ba8"><strong><em>here</em></strong></a></p><p>So await for the best part !!</p><p>But until than you can have a quick glance at the Job DSL scripting API documentation in below location.</p><p><a href="https://jenkinsci.github.io/job-dsl-plugin/">https://jenkinsci.github.io/job-dsl-plugin/</a></p><p>You can open it via localhost also after you install the plugin to Jenkins.</p><p><a href="http://localhost:8090/plugin/job-dsl/api-viewer/index.html">http://localhost:8080/plugin/job-dsl/api-viewer/index.html</a></p><p>UPDATE: Part two can be read at <a href="https://medium.com/@anuradha.neo/automate-the-ci-cd-automation-with-jenkins-part-2-fcd987456ba8">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=623a2c532848" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which software architecture and when ?]]></title>
            <link>https://anuradhaneo.medium.com/what-software-architecture-and-when-c3556f20ef48?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3556f20ef48</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[soa]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Tue, 14 Aug 2018 12:08:05 GMT</pubDate>
            <atom:updated>2018-08-14T16:41:04.741Z</atom:updated>
            <content:encoded><![CDATA[<p>When I started my professional career as a software engineer I saw that it was the era of 3-tier architecture which was a yet a new thing and was the successor of monolithic software applications which was the obvious choice in the initial era of the software world. Those days software architects had very little problems to worry about due to the simplicity of the business requirements and to the extent the businesses had embraced the technology. Then came the MVC architecture and N-tier architecture which is also can be seen as some form of multi tier layered architectures to support better separation of application responsibility as well as to support various different types of clients emerged such as different lightweight yet powerful web browsers as well as mobile devices such as smartphones and notebook devices. The advantage of all these was to separate the UI logic and rendering separated from business logic and domain model so the implementation layers of business logic and data persist persistence can be made fat services and multiple view/presentation servers can be deployed to handle traffic. But yet these layers are very much coupled with each other which was a problem when in operation and extending for the future requirements.</p><p>As a solution then came the Service Oriented Architecture(SOA) which is a more decoupled in terms of the components separated as services and clients. This architecture became more and more popular and adopted the emerging of RESTful web services and JSON format rapidly. The clear separation between service layer and the client layers brought great advantages in terms of scaling the applications horizontally rather than vertically depending on the client demands for the application services. Say for an example if there is a higher load forecasted for next year, it is possible to deploy two more web servers to host services as well as few more client applications where they are fronted with the same existing load balancer so the clients are seamlessly connected without noticing any service lags nor outages. Further it is also possible to add new different services to extend the new business services as well as upgrade the existing services to new versions.</p><p>SOA is still is used widely and is architecturally totally fine with many of the small to medium scale businesses. But SOA becomes a problematic for large scale businesses and as well as non-large scale businesses that has a very dynamic traffic loads varying from average to very high loads and vice versa more frequently. SOA architecture limitations mainly raised with the problems related to the service availability, scalability and stability. Lot of things become challenging as the inbound traffic load increases, I mean architecturally. If we use SOA for a this kind of scenario, the backend business services implemented in a giant monolithic service module you have to scale the service layer to cater the increasing high demand and thus you are necessarily replicating this monolithic web service module which requires much higher resources to host them and as well as affect the dynamic scaling nature due to much higher startup time when spinning up new servers dynamically. In other words say that there is a web server that hosts shared common admin services, financial services, HR services together as a monolith and imagine a situation where there is a considerable or a rocketing demand for financial services at some point of time, how would you scale the web services to face the incoming load will be by scaling the web servers horizontally. But now the problem here is that as you scale the service layer all other non required services too get scaled which makes the dynamic nature of the scaling process slow as well as the underlaying infrastructure needs to be made more resourceful. This becomes a large waste of resources as well as money for the business which can be if we could only scale the necessary services but not the others.</p><p>Above situation requires a architectural paradigm shift from the SOA. Essentially the idea is to further break the service layer modules to relevant smaller modules that would interact with each other to expose final services to the client applications layer. This is the simple idea of Micro Services Architecture (MSA), architecture the applications with micro sized services which are logically decoupled to work together to form a service cloud to handle the increasing client requests.</p><p>Sounds simple right ? No, it is not. Every luxury comes with its own set of risks and complexity to deal with. With the emergence of the MSA there are new problem to deal with.</p><ul><li>How to manage scalability and availability of individual micro services</li><li>In order to dynamically scale, how is the underlaying infrastructure layer support that ?</li><li>How to expose services as APIs at different layers both internally and to the public</li><li>How to integrate different services exposed with different protocols (eg. HTTPS and JMS and then with SOAP)</li><li>How to handle message flows and message delivery resilience</li><li>How to monitor the service calls which hops from service to service</li><li>How to manage deployments w/o affecting the service availability</li><li>How to manage upgrades and patch deployments</li><li>How to monitor the overall system health and statistics</li><li>How to debug and troubleshoot the applications</li><li>How to implement a common layer of shared services which most of the other micro services will consume and how to manage the vast inward traffic</li></ul><p>Above are just a few of a large set of complexities that needs to be addressed due to the MSA based software applications and deployments. And solving them are definitely not easy as you may already have probably felt by looking at the above list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*8u3YmjoGqYynWY6p" /></figure><p>Further going in to a MSA you need to educate the customer for the implications in terms of implementation/maintenance complexity as well as additional resources/tools/persons needed and hence costs that may be incurred going forward. I must say you cannot practically implement a MSA based application/system without the proper adoption of the Agile principles for the project governance. It is essential to build the individual services incrementally and integrate with each other in a very Agile way.</p><p>So the advice;</p><p>Do not follow the trend blindly, every architectural decision needs to have a reasoning behind it.</p><ul><li>If the customer business and/or the scope of the application is so small and have budget constraints go for a monolithic application, and why not ? of course its still possible and is the right decision.</li><li>If the customer says he has a plan to expand everything rapidly after 5 years, don’t build a complex system now for a business that you nor the customer has already not seen. Its worth build something simple to cater the current business requirements and build something advances later as the business expands and architecture/design the future system based on the then actual situation. Who knows how businesses and technology changes in 5 years. Sometimes you will have to convince this to customer.</li><li>If the customer business has decent amount of clients /client applications now or coming up in a short time period consider architecture the applications using SOA unless otherwise cannot be done with MSA.</li><li>Consider incorporating Message Oriented architectural aspects if the software application requires Quality of Services (QOS) attributes.</li><li>If the traffic is very high or the traffic pattern if very dynamic MSA can be considerable but it has to be done with the proper analysis of the implications and with the customer awareness is given too. It is important to make the customer awareness set correctly so that the customer will support and understand the technical complexity and challenges in MSA.<br>Make sure there is a agile plan from the architecture itself to deliver the system step by step.</li></ul><p>Note that there is no any proper criteria to say above points will be the only (or most prominent) consideration when coming up with a architecture. There are some situations that being small or being a startup you just have to work on a architecture which is very scalable and highly available in multiple regions in the world from the inception due to the nature of the business, market, clients/customers or the vision. Example is Twitter as a social media platform had to work on a excellent architecture being a startup back then because of the above reasons and in fact they ended up defining some of the MSA best practices and building tools and frameworks which did not exist to support the platform implementation. <br>Hence selecting the right architecture becomes very objective and should be done with a proper analysis of the problem as well as involves lot of thinking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3556f20ef48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a serverless microservices infrastructure for Docker and Kubernates with AWS Fargate]]></title>
            <link>https://anuradhaneo.medium.com/build-a-serverless-microservices-infrastructure-for-docker-and-kubernates-with-aws-fargate-6ab7d9ad3c98?source=rss-2689aaaafbf0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ab7d9ad3c98</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[fargate]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Anuradha Prasanna]]></dc:creator>
            <pubDate>Thu, 26 Apr 2018 20:22:37 GMT</pubDate>
            <atom:updated>2018-04-26T20:22:37.162Z</atom:updated>
            <content:encoded><![CDATA[<p>As technology scavengers we always seek better technologies and new avenues as well as evangelists we keep on promoting and talking about them and discuss on the same to spread the word as well as to gain feedback and to gather ideas. What we are always looking for is better ways of doing things in more efficient, streamlined, maintainable and cost effective manners. As one of the best examples for that we can talk about is how the technological trend and thus ourselves as well moving from bare metal servers to virtual machines to containers to serverless infrastructure. I think there are better ways or technologies are yet to be invented which the future generations are responsible for, but whats we see as the cutting edge technologies as of today are really superior related to the current time.</p><p>Serverless, What is serverless and why is it important ? Well, to answer that lets recap how things had worked (and is still in use as common majority) before serverless. You would purchase and spin-up a VM or a containerized instance to host your application and make it run, may be in long-term or shut it down after sometimes when your work is fully finished. In the case of running containers you would always need to have a VM to as the underlying infrastructure layer. Having to maintain a host or a VM meaning maintenance, resolving of issues as well as costs that may incur which are relatively higher, specially when the requirement is to run them in long-term. Example may be version upgrades, patch application, backups and log cleanups etc.</p><p>How about thinking of a hollow placeholder (such as a runtime environment) instead of a VM that is capable of running specific container/function when the container/function is injected to it and the lifetime of the placeholder exists only for the duration of the execution of the functionality ? In other word this placeholder may provide an environment to run a container. To customize this according to the need can be done with a config specification attached to the placeholder and you only have to pay for that spec as well as to the run time (time of existence) of the placeholder. You don’t need to provision VMs or manage them, and hence no recurring charges for running VMs contentiously.</p><p>If you are aware of AWS Lambda which is a serverless computing platform to run functions on-demand that does not require you to have infrastructure, AWS Fargate is also something very similar in concepts where a Docker container is run instead of a functional code which AWS Lambda is configured to run. This is what AWS Fargate does and offers for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hkMCtBCsNfbpMBWaK4Y1JA.jpeg" /></figure><p><strong>Lets talk about AWS Fargate technical aspects in detail;</strong></p><p>It lets you specify the technical configuration for this placeholder which in Fargate is called as Task Definition to allocate necessary CPUs and the Memory to run the container. <br>Pls note the CPUs allocated are virtual CUPs (vCPUs).</p><p>In below screenshot shows Task Definition creation screen that specifies 0.5Gb memory and 0.25 vCPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*eF-h60lRapd922JsCsKKCg.png" /><figcaption>AWS Fargate “Task Definition‘” screen used to specify vCPUs and Memory</figcaption></figure><p>Now how do you say what task or functionality to inject in to the placeholder. Or in other words we need to have some task/function to run which the Task Definition will help to run. We call this functionality specification as “Container Definition” in Fargate. This specifies the docker image that is started up as the functionality. Currently AWS Fargate (as well as <a href="https://aws.amazon.com/ecs">ECS</a>) supports Docker container only, mind that Docker is not the only container implementation technology but there are others too such as <a href="https://linuxcontainers.org/">LXC</a>. As the docker image for the container to be running, you can specify a path in <a href="https://hub.docker.com/">Docker Hub</a> repository or you can use a image hosted in AWS <a href="https://aws.amazon.com/ecr">ECR</a> (Elastic Container Registry). But if later case you need to manually upload your preferred custom docker image to <a href="https://aws.amazon.com/ecr">ECR</a> manually before using. As an example we can think of a <a href="http://tomcat.apache.org/">Apache Tomcat</a> based docker container that&#39;s serving a web application once run.</p><p>Pricing based on requested vCPU and memory resources for the Task and billed per second. Price per vCPU is $0.00001406 per second ($0.0506 per hour) and per GB memory is $0.00000353 per second ($0.0127 per hour).</p><p>For an example if you run a single vCPU and 1GB sized container for 1 hour, your cost would be $0.0506 + $0.0127 = $0.0633</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*ogrDpFgmSHJuvMIjPagFlA.png" /><figcaption>“Container Definition” in AWS Fargate used to specify Docker image</figcaption></figure><p>Below if the structural illustration of the different configuration definitions or layers that is used to build a AWS Fargate service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*q8KilBVcuKKWW9v3W6Vd_g.jpeg" /><figcaption>Layers within a Fargate based service implementation</figcaption></figure><p>Service definition defines how may simultaneous docker instances need to run and how to load-balance the incoming traffic. For an example think of a webservice that you need to be served with multiple instances to handle the ingress load. <br>In below sample screenshot shows only one task is assigned to the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*qR0CK3eQBfq79AUWz2HQYA.png" /><figcaption>Fargate Service definition that asks how many task definitions to run</figcaption></figure><p>A cluster is nothing but a logical grouping of related set of services and hence a set of tasks. This can be done based on your requirement on grouping services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*DrLKLHiPLSRECir59R1xpw.png" /><figcaption>Image Source: <a href="https://aws.amazon.com/fargate/">https://aws.amazon.com/fargate/</a></figcaption></figure><p><strong>True benefits of using serverless container services</strong></p><ul><li>No EC2 hosts to manage, hence you can focus on application rather than infrastructure. When you don’t have hosts to manage you do not have to restart/stop them, upgrade resources manually, apply security patches or OS updates, health checks on the OS which is a great relief as a infrastructure/system admin where heavy-lifting is handled automatically by AWS.</li><li>Simple and cheap cost model since you only have three things to consider CPU size, Memory size and the number of seconds of run-time.</li><li>Various vCPU &amp; memory combinations to select just the right amount for your application. You can allocate smallest resources to high power capabilities which is very flexible. Also you have the flexibility to change the resources allocated by upgrading or downgrading.</li><li>Seamless auto scaling containers against the load unlike you have to manually add ECS instances to handle the load as an example.</li><li>Ease of use/ administration due to the lack of hosts to manage as well as due to central management capabilities via AWS Web Console or CLI.</li></ul><p><strong>AWS EKS</strong></p><p>Amazon Elastic Container Service for <a href="https://kubernetes.io/">Kubernetes</a> or <a href="https://aws.amazon.com/eks">Elastic Kubernates Service (EKS)</a> is a fully certified Kubernates implementation in AWS cloud that is a fully managed service.</p><p>EKS is currently (Apr, 2018) in preview mode users (you have to <a href="https://pages.awscloud.com/amazon-eks-preview.html">register</a> separately to get access to evaluate EKS which is subject to AWS approval based on your usecases). EKS will be generally available with support for Kubernetes v1<strong>.</strong>10. This service will be released as a Fargate version which you will no need to manage Kubernates nodes but will be similar to <a href="https://aws.amazon.com/ecs">ECS</a> version of Fargate with different vCPU and Memory configuration to choose that reflects the size of the workload for each Kubernates pod.</p><p>So what are the true benefits of using serverless Kubernates service</p><ul><li>Eliminate the use of hosts (VMs) as Kubernates nodes so you do not have to manage them.</li><li>Complexity in managing the Kubernates cluster is taken off from you and is internally managed by AWS cloud for you.</li><li>No Kubernates masters to manage. Highly available provisioned master nodes cluster will be handling all that work in background.</li><li>Built upon vanilla Kubernates but not on a AWS version of Kubernates, hence any application working on standard plain Kubernates is compatible with EKS.</li><li>Significant cost reduction due to no infrastructure to manage.</li></ul><p><strong>Summary</strong></p><blockquote>So why is serverless technologies are getting more and more attention and platform migrations these days ?</blockquote><blockquote>Simply since it is because of the convenience to manage the applications and cost effectiveness. You know you have less (of may be 0%) maintenance work to do with not having virtual hosts and you also know you only pay absolutely for what you use only since these serverless charging model work for resource usage parameters only.</blockquote><p><strong>How AWS takes a lead as a service provider</strong></p><blockquote>AWS has two Docker container based clustered environment capabilities called ECS and EKS where fist is AWS own implementation of cluster management/orchestration framework and EKS is a Kubernates based implementation. <br>They both offer serverless versions (EKS version will be released in future within 2018) named AWS Fargate where underneath infrastructure management is fully managed and scaled by AWS. All you have to do is prepare the Docker image that contains your applications and to deploy them on Fargate.<br>Best of all these services are seamlessly integrated to other AWS tools and service where you can take advantage of the robust AWS features such as <a href="https://aws.amazon.com/iam">IAM</a>, <a href="https://aws.amazon.com/s3">S3</a>, <a href="https://aws.amazon.com/rds">RDS</a>, <a href="https://aws.amazon.com/cloudwatch">CloudWatch</a> and various <a href="https://aws.amazon.com/products/networking/">other tools</a>.</blockquote><blockquote>Fargate is a forerunner technology and is still evolving and rolling out in the AWS Cloud. As of today (Apr 2018) Fargate technology is only available for ECS platform and yet only within <a href="https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/"><em>Northern Virginia</em></a><strong> </strong>region which will be available to other regions in future as well as to EKS. It is definitely beneficial to be aware and exposed to this technology as there are insights and tendency to migrate future workloads to Serverless platforms and Fargate which I believe will be the most prominent and powerful of all.</blockquote><p>** Await more articles on this topic in future !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ab7d9ad3c98" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>