<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leah Tarbuck on Medium]]></title>
        <description><![CDATA[Stories by Leah Tarbuck on Medium]]></description>
        <link>https://medium.com/@leahtarbuck?source=rss-b387d766fe3f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*B2C-cdoHm8RBV-1OUCeBSw.png</url>
            <title>Stories by Leah Tarbuck on Medium</title>
            <link>https://medium.com/@leahtarbuck?source=rss-b387d766fe3f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 19:37:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leahtarbuck/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS Glue vs EMR]]></title>
            <link>https://medium.com/swlh/aws-glue-vs-emr-433b53872b30?source=rss-b387d766fe3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/433b53872b30</guid>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Leah Tarbuck]]></dc:creator>
            <pubDate>Wed, 02 Sep 2020 10:10:43 GMT</pubDate>
            <atom:updated>2020-09-04T14:13:09.882Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.investopedia.com/articles/investing/011316/what-amazon-web-services-and-why-it-so-successful.asp#:~:text=AWS%20is%20so%20large%20and,and%20Google%20Cloud%20at%206%25.">Amazon Web Services</a> provide two service options capable of performing ETL: <a href="https://aws.amazon.com/glue/?whats-new-cards.sort-by=item.additionalFields.postDateTime&amp;whats-new-cards.sort-order=desc">Glue</a> and <a href="https://aws.amazon.com/emr/?whats-new-cards.sort-by=item.additionalFields.postDateTime&amp;whats-new-cards.sort-order=desc">Elastic MapReduce</a> (EMR). If they both do a similar job, why would you choose one over the other? This article details some fundamental differences between the two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*gKJD6O_6TU7eT2P-TUsYPA.png" /></figure><p>AWS Glue is a pay as you go, server-less ETL tool with very little infrastructure set up required. It automates much of the effort involved in writing, executing and monitoring ETL jobs. If your data is structured you can take advantage of Crawlers which can infer the schema, identify file formats and populate metadata in <a href="https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html">Glue’s Data Catalogue.</a> Based on your specified ETL criteria, Glue can automatically generate Python or Scala code for you and provides a nice UI for job monitoring and scheduling.</p><p>In comparison, EMR is a big data platform designed to reduce the cost of processing and analysing huge amounts of data. It is a managed service where you configure your own cluster of EC2 instances. You have complete control over the configuration and can install <a href="https://aws.amazon.com/emr/features/hadoop/">Hadoop ecosystem components</a>, which makes EMR an incredibly flexible and complex service. Its use cases are vast. Data scientists can use EMR to run machine learning jobs utilising the <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-tensorflow.html">TensorFlow</a> library, analysts can run SQL queries on <a href="https://medium.com/r?url=https%3A%2F%2Fdocs.aws.amazon.com%2Femr%2Flatest%2FReleaseGuide%2Femr-presto.html">Presto</a>, engineers can utilise EMR’s integration with streaming applications such as <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-kinesis.html">Kinesis</a> or <a href="https://aws.amazon.com/emr/features/spark/">Spark</a>… the list goes on!</p><p>You could replace Glue with EMR but not vice versa, EMR has far more capabilities than its server-less counterpart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TS4uny9AuS6wELHu5al93g.png" /></figure><p>Another thing to consider when choosing between these tools is cost. <a href="https://www.knowledgenile.com/blogs/aws-glue-vs-emr/#:~:text=Typically%2C%20AWS%20Glue%20costs%20you,per%20day%20for%20similar%20configurations.">Glue is more expensive than EMR when comparing similar cluster configurations</a>, probably because you’re paying for the server-less privilege and ease of set up.</p><blockquote><a href="https://aws.amazon.com/blogs/big-data/how-drop-used-the-amazon-emr-runtime-for-apache-spark-to-halve-costs-and-get-results-5-4-times-faster/#:~:text=We%20found%20that%20AWS%20Glue,time%20of%207%E2%80%938%20minutes.">Drop’s Data Lake solution</a> found a reduction in cold start time and an 80% reduction in cost when migrating from Glue to EMR.</blockquote><p>There are currently only <a href="https://docs.aws.amazon.com/glue/latest/dg/add-job.html">3 Glue worker types</a> available for configuration, providing a maximum of 32GB of executor memory. This restriction may become problematic if you’re writing complex joins in your business logic. If the join isn’t optimised for performance then executor memory can quickly be consumed and the job may fail. The same can occur if you have to unpack a very large zip/gzip file, <a href="https://luminousmen.com/post/spark-tips-dont-collect-data-on-driver">all of the data will be held on one node</a> (such is the workings of Spark!).</p><p>In contrast to this, EMR has a plethora of<a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-supported-instance-types.html"> supported Instance Types</a> to choose from! <em>(although you’d still want to optimise joins to improve performance 😃 and ideally avoid zip and gzip formats!)</em></p><p>One advantage of using AWS Glue, is that it automatically sends logs to <a href="https://aws.amazon.com/cloudwatch/features/">CloudWatch</a>, which is very handy if your architecture uses multiple AWS services — providing you with one centralised location for monitoring and alerting. EMR on the other hand, sends logs to S3 by default — although you can install the CloudWatch agent via EMR’s bootstrap configuration.</p><p>In conclusion, if your workforce is new to AWS configuration and you only wanted to execute simple ETL, Glue might be a sensible option. However if you wished to leverage Hadoop technologies and perform more complex transformation, EMR is the more viable solution.</p><p>Thank you for reading! 😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=433b53872b30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/aws-glue-vs-emr-433b53872b30">AWS Glue vs EMR</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating and Load Testing Custom CloudWatch Alarms]]></title>
            <link>https://medium.com/@leahtarbuck/creating-and-load-testing-custom-cloudwatch-alarms-cda2c37658b8?source=rss-b387d766fe3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cda2c37658b8</guid>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[cloudwatch-custom-metrics]]></category>
            <category><![CDATA[ec2-instance]]></category>
            <category><![CDATA[cloudwatch-agent]]></category>
            <dc:creator><![CDATA[Leah Tarbuck]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 09:01:01 GMT</pubDate>
            <atom:updated>2020-08-07T09:01:01.011Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I’ve seen a few different articles detailing how to create custom EC2 instance metrics using the CloudWatch Agent, but none suggesting how to test these alarms once they are created. How do you check whether they actually work? <em>🙃</em>. This article details installation <strong>and</strong> testing, if you’re interested in the latter, skip ahead to the stress-ng section!</blockquote><p>Perhaps you’re using Amazon Elastic Container Service or Elastic MapReduce and have multiple EC2 instances running in a cluster and you want to be alerted if one particular instance is struggling for memory, disk space or CPU…</p><p>The metrics generated out-of-the-box for these services often provide you with the <em>average</em> <em>usage across the cluster, </em>which in itself is useful<em> </em>but runaway processes on individual instances can still go <strong>undetected </strong>😫</p><p>The AWS CloudWatch agent allows you to collect and publish these system level metrics, as well as logs from EC2 instances to CloudWatch. You can easily create alarms using these metrics which will trigger if a specified threshold has been breached.</p><p>You can <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent-New-Instances-CloudFormation.html">install the agent using CloudFormation</a> or <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-commandline.html">command line.</a></p><p>In our case we used EMR’s bootstrap configuration to install the agent with a few lines of code:</p><pre># Install the CloudWatch Agent for log publishing<br>sudo rpm -Uvh <a href="https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm">https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm</a></pre><pre># Download the configuration file<br>sudo -u root aws s3 cp s3://${bootstrap_bucket}/amazon-cloudwatch-agent.json /opt/aws/amazon-cloudwatch-agent/etc/</pre><pre># Start the CloudWatch Agent<br>sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s</pre><p>You need to provide the agent with a configuration file that specifies what metrics and logs the agent is to collect. As part of our terraform deployment, this was loaded into an S3 bucket: “<em>${bootstrap_bucket}</em>” 🙂</p><blockquote>Just a heads up here: AWS recommend that you name the configuration file “/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json” on a Linux server. I did run into some issues with the agent not recognising the configuration when using an alternative file name and directory.</blockquote><p>You can create the configuration file <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html">manually</a> or by using the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html">Configuration Wizard</a> which gets installed along with the agent. I would recommend using the Wizard first initially to get an idea of the expected format, then tweaking to fit your needs!</p><p>An example configuration is below:</p><pre>{<br>   &quot;agent&quot;: {<br>      &quot;metrics_collection_interval&quot;: 60<br>   },<br>   &quot;metrics&quot;: {<br>      &quot;append_dimensions&quot;: {<br>         &quot;InstanceId&quot;: &quot;${aws:InstanceId}&quot;,<br>         &quot;InstanceType&quot;: &quot;${aws:InstanceType}&quot;<br>      },<br>      &quot;metrics_collected&quot;: {<br>         &quot;cpu&quot;: {<br>            &quot;measurement&quot;: [<br>               &quot;cpu_usage_active&quot;<br>            ],<br>            &quot;metrics_collection_interval&quot;: 60,<br>            &quot;resources&quot;: [<br>               &quot;*&quot;<br>            ]<br>         },<br>         &quot;disk&quot;: {<br>            &quot;measurement&quot;: [<br>               &quot;disk_used_percent&quot;<br>            ],<br>            &quot;metrics_collection_interval&quot;: 60,<br>            &quot;resources&quot;: [<br>               &quot;/&quot;<br>            ]<br>         },<br>         &quot;mem&quot;: {<br>            &quot;measurement&quot;: [<br>               &quot;mem_used_percent&quot;<br>            ],<br>            &quot;metrics_collection_interval&quot;: 60<br>         }<br>      }<br>   }<br>}</pre><p>AWS provides some detailed documentation of the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html">available fields to use</a> in the configuration file. Here i’m publishing the total CPU aggregated across all cores as well as individual thread CPU usage, total memory metrics and disk space usage of root. The agent automatically resolves the variables ${aws:InstanceId} and ${aws:InstanceType} for you. By default the unit for these metrics is Percent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLrSE2qhKBtmsNqV-gPs6w.png" /></figure><p>After deploying the EMR cluster with the bootstrap configuration, we can now see metrics being published to CloudWatch within a “CWAgent” namespace. You can create associated alarms easily <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ConsoleAlarms.html">via the AWS console</a> or deployment using <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_metric_alarm">IaC tool configuration</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qzb4Ps__jD10XqyNhv8U1g.png" /></figure><p>… but how do we know these alarms ‘work’ as expected? Perhaps you have an action on the alarm which when triggered sends an email to users <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html">via SNS Topic Notification</a>?</p><p>You could try to cheat the system and ‘tweak’ the threshold temporarily -decreasing it’s value to force a notification… but does our alarm monitor system load as expected? have we configured our agent to publish the appropriate metrics?</p><p>Ideally we need a workload generator to check this!</p><h3><strong>Stress-ng</strong></h3><p>To load test CPU and Memory usage alarms, I installed the <a href="http://manpages.ubuntu.com/manpages/bionic/man1/stress-ng.1.html">stress-ng tool</a> on the EC2 instances:</p><pre>sudo yum install -y stress-ng</pre><p>I used very simple alarm configurations, if the total CPU or Memory usage of an EC2 instance exceeded 80% for 1 datapoint within 1 minute, the alarm would change state to “<strong><em>in alarm</em></strong>”.</p><h4>CPU Load Testing</h4><p>…was performed using:</p><pre>stress-ng --cpu 4 --cpu-load 95% --timeout 2m --metrics --times --verify &amp;</pre><p>As the command suggests, stess-ng will start 4 processes with a 95% load for 2 minutes. For obvious reasons your percentage here needs to exceed your threshold to ensure the alarm is triggered.</p><p>Stress-ng measures a stress test “throughput” using “bogus operations per second”, the “‘--metrics” option outputs the number of bogus operations in total performed by the stress processes.</p><p>“--times” show the cumulative user and system times of the child processes at the end of the stress run.</p><p>You can also verify that the computations are correct using “--verify”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0l26_lIfGlm593pra9nhPw.png" /><figcaption>Example stack trace output for a CPU load test</figcaption></figure><p>The number of processes you should load test with would depend on how your instances are configured at launch time. Testing with less processes than threads used on your instance is unlikely to trigger the alarm (unless you’ve used a very small threshold of course 🙃).</p><p>You can execute the following AWS CLI command to view the CPU options for your instance/s:</p><pre>aws ec2 describe-instances --instance-ids &lt;your_instance_id&gt;</pre><p>Within the returned output information, check your CoreCount and ThreadsPerCore, e.g.</p><pre>&quot;CpuOptions&quot;: {<br>       &quot;CoreCount&quot;: 2,<br>       &quot;ThreadsPerCore&quot;: 2<br>               }</pre><p><em>Find out the supported number of vCPU’s (threads) for your instance type </em><a href="https://aws.amazon.com/ec2/instance-types/"><em>here</em></a><em>.</em></p><p>For my m4.xlarge instance, the physical core count is 2, but I’m using 2 threads per core, so 4 stress-ng threads (“--cpu 4”) would be necessary to trigger the alarm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rB1o-ICONjmG_Y97a5iHXA.png" /></figure><p>You can see in the CloudWatch Metrics console the CPU usage for each thread is published, all contributing to the total amount (labelled “<em>cpu-total</em>”).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pev_rINulR_CZc14HcGBjA.png" /></figure><p>Once the stress work has stopped (or been terminated), the alarm will return to an OK state.</p><h4>Memory Load Testing</h4><p>… was performed using the following command:</p><pre>stress-ng --vm 1 --vm-bytes 99% --timeout 2m --vm-keep --metrics --times --verify &amp;</pre><p>“--vm” starts workers which write to memory.</p><p>I needed to add in the “--vm-keep” flag, which keeps the memory usage constant and prevents memory reallocating.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WujrDFPf5na7Ee2x5QvlTQ.png" /><figcaption>Example stack trace output for a memory load test</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1gz-zIcBBG5xza9CTui6w.png" /></figure><h4>Disk Space Load Testing</h4><p>You can view available disk space by using “df -hT”, which displays the file system in a human readable format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*LHN0RDsCl5Or0tSyTTtZKQ.png" /></figure><p>Triggering the disk space alarm is easily performed using the “<em>fallocate</em>” command, which creates a file of a specified size. For example:</p><pre>sudo fallocate -l 100GB file.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mmWL4uimWnILhF68MU5_w.png" /></figure><blockquote>Just remember to remove the file so the alarm can go back to an “OK” state 😇</blockquote><p>I hope this has helped others struggling to load test their EC2 instance alarms. Thanks very much for reading! 👋🏻😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cda2c37658b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An overview of Apache NiFi and toolkit cli deployments]]></title>
            <link>https://levelup.gitconnected.com/an-overview-of-apache-nifi-and-toolkit-cli-deployments-785978dbce3b?source=rss-b387d766fe3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/785978dbce3b</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[apache-nifi]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[nifi]]></category>
            <dc:creator><![CDATA[Leah Tarbuck]]></dc:creator>
            <pubDate>Wed, 29 Jul 2020 14:58:48 GMT</pubDate>
            <atom:updated>2020-12-15T15:12:25.767Z</atom:updated>
            <content:encoded><![CDATA[<h3>An overview of Apache NiFi and Toolkit CLI deployments</h3><blockquote>Apache NiFi is a tool used to create Data Pipelines via a drag/drop interface and is designed to automate the flow of data between systems. It fits into the ‘<em>no code/low code</em>’ category of tools, primarily geared towards companies who feel less comfortable writing and managing code or building a solution that requires a significant amount of engineering effort.</blockquote><h3>What can NiFi be used for?</h3><p>NiFi is good at reliably transferring data between platforms e.g. Kafka to ES and allows you to perform some lightweight ETL during the process. It can enrich and prepare data, perform conversion between data formats, change named fields and filter/route data to different locations.</p><p>However, it should not be used for complex event processing, distributed processing, joins, rolling windows or aggregation (it is not Apache Spark and doesn’t claim to be! 😎). Therefore it shouldn’t be compared to big data processing tools like AWS EMR or more flexible Open Source ETL Tools like Talend. A more reasonable comparison would be between <a href="https://statsbot.co/blog/open-source-etl/">Apache NiFi and StreamSets</a>.</p><p>NiFi can integrate with some AWS services, such as S3, SNS, Lambda, SQS and DynamoDB.</p><p>If you need to integrate with streaming services, it’s worth noting that there is more support for the fellow Apache product Kafka, than there is AWS Kinesis. There is currently no ability to read from Kinesis Data Streams, but there is an <a href="https://issues.apache.org/jira/browse/NIFI-2892">open PR</a> for the change!</p><h3>NiFi Terminology</h3><p>Below are some useful terms specific to constructing NiFi flows which will help in understanding the rest of this article:</p><p><strong>FlowFile</strong> — is the data, it consists of content (the data) and attributes (metadata key value pairs e.g. ‘creation date’)</p><p><strong>Processor</strong> — applies a set of transformations and rules to the FlowFiles and generates new FlowFiles. Apache NiFi boasts there are 280+ processors to choose from! Examples include getKafta, getFTP, putKinesisStream, putHDFS, convertRecord, replaceText, validateCSV etc.</p><p><strong>Connector</strong> — a queue of all the FlowFiles that are yet to be processed by the next processor. Which defines rules about how FlowFiles are prioritised (which ones are first, or filtered out e.g. FIFO). Connectors can be used for Load balancing and re-directing ‘failed’ FlowFiles.</p><p><strong>Controller Service</strong> — a shared service used by processors e.g. DB connections, AWS credentials, Writers and Readers that you can use within conversion processors (to name just a few).</p><p><strong>Process Group</strong> — acts as an abstraction layer, generally good practice to use these as a way of breaking up large data flows.</p><p><strong>Templates</strong> — can be used if you have a collection of processors performing some logic you’d like to repeat and re-use. e.g. common components you wish to share with others.</p><p><strong>Input/Output ports</strong> — allow you to transfer data in or out of remote process groups.</p><h3>Installation</h3><p>Installation is very simple, you can download the nifi-&lt;version&gt;-bin.tar.gz via this <a href="https://nifi.apache.org/download.html">link </a>or pull, build and run the <a href="https://hub.docker.com/r/apache/nifi/">docker image</a>.</p><h3>UI and NiFi Flows</h3><p>The UI is very intuitive and simple to use. You drag different components from the top banner onto the canvas and configure accordingly. You can also write labels within the code, a bit like JavaDoc!</p><p>An image of the UI is below, you can see a processing group in the centre of the canvas which you can right click and ‘enter’ into to display the flow within.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxUvya4JgfS0BIHwEtfB1Q.png" /><figcaption>NiFi Canvas with a process group</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6e8L5tsqchlaMmZnqPVEmA.png" /><figcaption>An example NiFi flow: reads from S3, converts to JSON and writes to Kinesis Firehose</figcaption></figure><p>Here you can see how processors are used in conjunction with connectors to handle the flow of data. Failed files can be redirected to other pathways and handled/logged accordingly. No code is needed, this is all done through configuration. This flow lists CSV files in an S3 bucket (to achieve this you need a combination of ListS3 and FetchS3Object processors), logs any issues via a LogAttribute processors, converts the files to JSON and puts on to a Kinesis Firehose Stream.</p><p>Within your root installation NiFi directory (e.g. ‘/Applications/nifi-&lt;version&gt;’), state is stored within a ‘/state’ folder (unless you run as a cluster, then Zookeeper stores this). Logs are stored under ‘/logs’, this directory is utilised by the LogAttribute processor. By default the flow is saved in ‘/conf/flow.xml.gz’, but you can override this configuration along with many others by editing the ‘/conf/nifi.properties’ file’.</p><p>NiFi directs you to documentation (if available) via the small book icon, for example the View usage option below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*MGcmBsZmZVmxBW0VjKzAeQ.png" /></figure><p>This is also available when adding controller services to a process group.</p><h3>AWS Connectivity</h3><p>You can create an AWSCredentialsProviderControllerService (set at process group level) to re-use your AWS credentials within multiple components of your flow. Specify either the location of a credentials file, which states the AWS Access Key and Secret Access Key or set these key’s directly within the properties of the controller service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8wUk0GWpv_5H-UPjVxPpw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*s4S9mAWzpjWjEzlh3RroHw.png" /></figure><p>Then you can reference this controller service within the configuration of processors that require AWS connection e.g. the PutKinesisFirehose processor properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74CVSIALCsRX5dH_dQvbfg.png" /><figcaption>PutKinesisFireshose processor configuration</figcaption></figure><p>Running processors need to be stopped to apply any changes such as new controller services or other property modifications.</p><h3>NiFi Registry</h3><p>The NiFi Registry is an Apache NiFi sub project and aims to answer people’s deployment and version control needs! It provides a central location to store resource group flows. You have to use the registry UI to create a ‘bucket’ first (which are used to organise the version controlled flows) before adding the new registry client in your local NiFi flow. This article: <a href="https://bryanbende.com/development/2018/01/19/apache-nifi-how-do-i-deploy-my-flow">NiFi-how-do-i-deploy-my-flow?</a> has some nicely written steps detailing how to do this.</p><blockquote>You can download the registry locally via <a href="https://nifi.apache.org/registry.html">this link</a> or use <a href="https://hub.docker.com/r/apache/nifi-registry/">Docker</a>.</blockquote><h3>NiFi Toolkit and Deployments</h3><p>The toolkit provides several command line utilities to help setup NiFi in single and clustered environments. Whilst investigating deployment options, I used the cli tool to connect to the NiFi Registry.</p><blockquote>You can download the toolkit locally via <a href="https://nifi.apache.org/download.html">this link</a> or use <a href="https://hub.docker.com/r/apache/nifi-toolkit">Docker</a>.</blockquote><p>Within your downloaded toolkit bin directory (e.g. /Applications/nifi-toolkit-version/bin) there is a ‘cli.sh’ script which can be used to launch an interactive shell.</p><p>For my deployment to an AWS account, I used terraform to create two EC2 instances (<a href="https://www.batchiq.com/nifi-selecting-an-aws-instance-type.html">using the m4.large instance type</a>) running within the same VPC, one was installed with NiFi and the other with NiFi Registry. The simple user_data configuration for NiFi is below:</p><pre>#!/usr/bin/env bash <br>curl -fsSL https://get.docker.com get-docker.sh | sh <br>sudo docker pull apache/nifi:latest <br>sudo docker build -t apache/nifi:latest . <br>sudo docker run --name nifi -p 8080:8080 -d apache/nifi:latest</pre><p>…. and NiFi Registry:</p><pre>#!/usr/bin/env bash <br>curl -fsSL https://get.docker.com get-docker.sh | sh <br>sudo docker pull apache/nifi-registry:latest <br>sudo docker build -t apache/nifi-registry:latest . <br>sudo docker run --name nifi-registry -p 18080:18080 /<br>-d apache/nifi-registry:latest</pre><p>Both shell scripts install docker, pull, build and run the latest images and publish the default container ports to the instance.</p><blockquote>For my ami_id I used a community edition of ubuntu-focal-20.04-amd64-server — this allows you to install docker on the instance.</blockquote><p>I had already created a bucket (<em>“leah’s_bucket”</em>) in the remote NiFi Registry and committed a flow (“<em>product_flow</em>“). Now I wanted to see how to deploy this to the remote NiFi instance…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVOGoKIc4Sul_vvqXwLz3w.png" /><figcaption>Local version controlled process group</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKHFGdzbnrHNsajeZmvx4A.png" /><figcaption>Remote registry with committed versioned flow</figcaption></figure><p>Once you’ve launched the interactive shell, you can deploy and start a versioned flow with just a few cli commands:</p><p><strong>First you need to create a new registry client in the remote NiFi instance, e.g:</strong></p><pre>nifi create-reg-client --baseUrl http://&lt;public_ip_of_nifi&gt;:8080 \<br>--registryClientUrl http://&lt;private_ip_of_registry&gt;:18080 \<br>--registryClientName leah&#39;s_client</pre><p>The baseUrl is the url to execute the command against (as I’m running these cli commands on my local machine, the baseUrl is the public IP address of the NiFi EC2 instance — because I’m connecting over the internet).</p><blockquote>The EC2 instance Security Group was modified to allow traffic from my local machine</blockquote><p>The registryClientUrl can refer to the private IP of the NiFi Registry — as these two EC2 instances are within the same VPC in AWS so can talk to one another!</p><p>The registryClientName is whatever name you want this registry to be called within the NiFi UI.</p><p><strong>Next we need to create a process group by importing a versioned flow from the registry, e.g:</strong></p><pre>nifi pg-import --bucketIdentifier 0455d64e-ea40-4213-bbe4-359985de09f9 --flowIdentifier 6df2b6c1-249b-4af3-8f84-b57f32f6c5aa \<br>--flowVersion 1 --baseUrl http://&lt;public_ip_of_nifi&gt;:8080</pre><p>The bucketIdentifier and flowIdentifier can both be found in the screenshot of the committed versioned flow above, these are unique.</p><p>This pg-import command returns a unique id for the process group (processGroupId) — which you need for the next steps!</p><p><strong>Enable any controller services you are using, e.g:</strong></p><pre>nifi pg-enable-services --processGroupId 94a93dc0-0173-1000-fee3-442454b1ecd6 --baseUrl http://&lt;public_ip_of_nifi&gt;:8080</pre><p><strong>Then you can start the process group, e.g:</strong></p><pre>nifi pg-start --processGroupId 94a93dc0-0173-1000-fee3-442454b1ecd6 --baseUrl http://&lt;public_ip_of_nifi&gt;:8080</pre><p>You should be able to see the newly deployed process group running in the UI!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*aPeYEw6D6j_9x1EFQgTHBw.png" /></figure><p>These commands could be scripted to provide a more automated way of deployment.</p><blockquote>If you need to set variables in your deployed NiFi flow, check out<a href="https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/"> this link</a> for more details.</blockquote><h3>Things to consider when using Apache NiFi</h3><ul><li>Unit testing — with most “low code“ solutions unit testing is never easy and often goes overlooked 😢. NiFi have provided a way to test individual processors using a <a href="https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing">TestRunner class</a>, however no way to test a process group. <a href="https://touk.pl/blog/2019/01/10/testing-nifi-flow-the-good-the-bad-and-the-ugly/">This article</a> discusses the difficulties in testing the application and introduces a library they created for testing a full flow.</li><li>Reviewing the code — as the flow code format is xml, it would be hard to detect small but potentially impactful (e.g. connector load balance changes) modifications to the flow file via a code review tool. You’d need a fairly strict review process here and consider looking through the Flow Configuration History in the UI (images below). The registry provides a way of version controlling the code, so you could review the configuration locally, before promoting it to another environment.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_SP6ABlVgeQ4zdqfiarLQ.png" /><figcaption>Flow Configuration History</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jrx_y3__7cV85LEbh12TEg.png" /><figcaption>Flow Configuration Action Details for a component</figcaption></figure><p>Thanks for reading my whistle-stop tour of Apache NiFi! I hope you’ve found it useful 😊</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=785978dbce3b" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/an-overview-of-apache-nifi-and-toolkit-cli-deployments-785978dbce3b">An overview of Apache NiFi and toolkit cli deployments</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The ‘Small Files Problem’ in AWS Glue]]></title>
            <link>https://medium.com/@leahtarbuck/the-small-files-problem-in-aws-glue-49f68b6886a0?source=rss-b387d766fe3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/49f68b6886a0</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Leah Tarbuck]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 15:11:20 GMT</pubDate>
            <atom:updated>2020-07-24T09:16:52.214Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>AWS Glue is a pay-as-you-go serverless extract, transform and load (ETL) tool, using Apache Spark under the covers to perform distributed processing.</blockquote><p>AWS Glue takes care of the provisioning and management of the resources required to perform the ETL for you,<em> </em>so you only really need to worry about the code… right? That being said… when you see nasty out of memory exceptions thrown by the driver, you need to understand a bit about how Spark works in order to resolve them.</p><p>In this very simple scenario, imagine we’ve written some ETL code in a Glue job which reads files from an S3 bucket, performs some file conversion and writes files back out to S3. We execute the job…</p><p>The Glue console provides you with running metrics and memory profiles for each job you create and execute, which helps you spot job abnormalities and performance issues like the one below!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*J0srvKurn394_A-Ink9GFw.png" /></figure><p>You can see the memory of the driver (blue line), exceeds the threshold of 50 percent and once it reaches 100, the job fails with a OOM and is killed. The executors (green line) haven’t started to use any memory yet.</p><p>In this instance, the transformation performed in the ETL isn’t the issue here. There is no data movement and the tasks haven’t been distributed to the executors yet. So what’s going on?</p><blockquote><strong>Spark Driver</strong></blockquote><blockquote>The Spark Driver is the heart of a spark application, it runs the main method, creates the SparkContext, converts the user program into units of physical execution (tasks) and distributes/schedules these across the executors. It does this by initially creating a DAG (directed acyclic graph) or execution plan (job). It also communicates with the Cluster Manager (e.g. YARN) and requests for resources to launch the executors.</blockquote><p>The main problem here is that Spark will make many, potentially recursive, calls to S3’s list(). This method is very expensive for directories with a large number of files. The driver keeps track of metadata for each file it reads and <strong>keeps this in memory.</strong></p><p>In our case, we had millions of small files sitting in our S3 bucket waiting to be read, this resulted in the driver quickly exhausting it’s 10G of memory (which is the maximum you can provide the driver when using Standard Worker nodes, more on this below!).</p><h3>Solutions?</h3><p>We had hit the ‘Small Files Problem’ in AWS Glue in a very big way! Here are a few things we investigated to try to rectify this issue.</p><h4>Glue’s Dynamic Frames</h4><p>A recommendation by AWS (documented <a href="https://docs.aws.amazon.com/glue/latest/dg/monitor-profile-debug-oom-abnormalities.html"><strong><em>here</em></strong></a><strong><em>) </em></strong>was to use Glue’s Dynamic Frame grouping option on the Spark read. This grouping is automatically enabled when you’re reading more than 50,000 files from S3. It allows the driver to keep track of a group of files, rather than recording memory for each individual file. All implementation examples we found online were written in python, we used the following to create Dynamic Frames in Scala:</p><pre>glueContext<br>  .getSourceWithFormat(<br>    connectionType = &quot;s3&quot;,<br>    options = JsonOptions(<em>Map</em>(&quot;paths&quot; -&gt; s3Paths, &quot;groupFiles&quot; -&gt; &quot;inPartition&quot;, &quot;useS3ListImplementation&quot; -&gt; <strong>true</strong>)),<br>    format = &quot;xml&quot;,<br>    formatOptions = JsonOptions(<em>Map</em>(&quot;rowTag&quot; -&gt; &quot;our-row-tag&quot;))<br>  )<br>  .getDynamicFrame()</pre><p>Where ‘s3Paths’ were a list of S3 Object keys to iterate though.</p><p>When you set ‘useS3ListImplementation’ to True, Glue doesn’t cache the list of files in memory all at once, instead it caches in batches. So the driver is less likely to run out of memory.</p><p>However, with further reading, we realised this would not be suitable for the majority of our datasets:</p><blockquote>“With AWS Glue grouping enabled, the benchmark AWS Glue ETL job could process more than 1 million files using the standard AWS Glue worker type.”</blockquote><p>Therefore this would still not provide a long-term solution for us, we were exceeding this for many of our datasets, in some cases we had over 20 million files to process!</p><h4>Changing the Glue Worker Types</h4><p>Another option, was to provide the driver with much more memory. Currently, Glue provides three different worker types:</p><ul><li><strong>Standard — </strong>the default worker type, maps to 1 DPU (Data Processing Unit). A Standard worker has 16 GB memory (for the executors), 4 vCPUs of compute capacity and 50GB of attached EBS storage with two Spark executors. The maximum amount of driver memory you can provide is 10GB.</li><li><strong>G.1X — </strong>maps to 1 DPU. This worker consists of 16 GB memory, 4 vCPUs, and 64GB of attached EBS storage with one Spark executor. The maximum amount of driver memory you can provide is 10GB.</li><li><strong>G.2X — </strong>maps to 2 DPUs and allocates twice as much memory, disk space, and CPUs as G.1X worker type with one Spark executor. The maximum amount of driver memory you can provide is 20GB.</li></ul><p>In this case, we’re not interested in what memory the executors have, only the driver, so the only option available which could provide some benefit was the G2.X worker.</p><p>It is worth noting that running a G.2X worker costs twice as much as a Standard worker (providing you run the jobs for the same amount of time and are using the same number of executors). As AWS charges $0.44 per DPU per hour (with a 10 minute minimum charge) and a G.2X worker uses 2 DPU’s compared to a Standard worker’s 1 DPU. (Documentation <a href="https://aws.amazon.com/glue/pricing/">here</a>).</p><p>Although this provided a short term fix, here we’re configuring vertical scaling, not horizontal, which <a href="https://medium.com/@arslan70/horizontal-vs-vertical-scaling-e15bdbf9e7e"><strong>goes against the principle of distributed processing</strong></a>.</p><h4>Concatenating the Files</h4><p>A more optimal solution (and one which we opted for), was to write another process which concatenated the small files into larger ones before they reached the source S3 bucket. To stay consistent with the serverless architecture approach, concatenation was performed via a Lambda before Glue processed the files. The workflow was orchestrated by an AWS Step Function.</p><p>There were multiple advantages to using this final approach:</p><ul><li>Fewer writes/reads to S3 resulting in a substantial reduction in cost.</li><li>The driver would not need to keep track of so many small files in memory, so no OOM errors!</li><li>Reduction in ETL job execution times (Spark is much more performant when processing larger files).</li></ul><p>This provided a more long-term solution compared to increasing the driver’s memory or using Glue’s Dynamic Frames.</p><p>Although AWS Glue hides a lot of Spark’s processing complexity, sometimes you have to go through the pain of understanding what’s hidden in order to reap its full potential.</p><p>I hope this has helped others in their development! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49f68b6886a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>