<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Campaign Monitor Engineering - Medium]]></title>
        <description><![CDATA[Insights from the Campaign Monitor Engineering team - Medium]]></description>
        <link>https://medium.com/cm-engineering?source=rss----f08afbb26c30---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Campaign Monitor Engineering - Medium</title>
            <link>https://medium.com/cm-engineering?source=rss----f08afbb26c30---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:09:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cm-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Observability at Campaign Monitor]]></title>
            <link>https://medium.com/cm-engineering/observability-at-campaign-monitor-22796334958?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/22796334958</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[apm]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <dc:creator><![CDATA[Pranavi Chandramohan]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 23:12:03 GMT</pubDate>
            <atom:updated>2020-07-15T23:12:03.532Z</atom:updated>
            <content:encoded><![CDATA[<p>These are exciting times for engineers at CM as many features from our core app (a monolith) are being refactored into microservices. A system which had a request path like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*c8X9nKaEce1b5MRV8D0XjQ.jpeg" /></figure><p>now looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*_UjDpVVn5s9D7CoihCIr6g.jpeg" /></figure><p>When a system starts to scale in terms of the number of microservices, mapping a request’s end to end journey becomes harder. Unless you have strict documentation procedures, maintaining architecture diagrams in a wiki is hard as they can become outdated quickly and are then more misleading than useful.</p><p>For the Site Reliability Engineering (SRE) team who are responsible for the reliability of the platform and emergency response, this means more complexity and chaos.</p><p>I joined the SRE team at a time where engineers with the most knowledge about the monolith had moved on, so not only did we have to support the newly added microservices but also the giant monolith! We realised that we needed the system to answer a very important question: <em>what is broken and why?</em></p><p>For the last year, our main goal has been to make our system answer this question. In this post, we will look at the ways we went about accomplishing this.</p><h4>What is observability?</h4><blockquote>Observability is a measure of how well internal states of a system can be inferred by knowledge of its external outputs. — wikipedia</blockquote><p>If we can answer questions about an application without having prior knowledge about it, then we can say that the application is observable. This is the key to helping us resolve production issues for our system. To make it observable, the app has to be instrumented to expose relevant data.<br>Let’s go through all the different kind of data we expose from our application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7HuFR1r3TNiIfxC15XOIsg.jpeg" /><figcaption>Source: <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">https://www.elastic.co/blog/observability-with-the-elastic-stack</a></figcaption></figure><h3>Logs</h3><p>Extensive logging has always been practised by developers at CM but, without structure, it is hard to find and analyze relevant logs. We live in an age of distributed microservices and unstructured logs just don’t cut it. Logs need structure, and by that I mean they need fields and values so that we can search and aggregate on those field values. There are libraries we use to provide structured logging for eg SLF4J for Java, Serilog for C# and Zerolog for GO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7XfdI2GL_KRg75q4AWO-Q.png" /></figure><p>It is not possible to change all the applications at once but every time we touch a part of a system we ensure the right logging libraries and right formats are used. Organically our log data has become significantly more useful!</p><h4>Tools</h4><p>We send all our application logs to <a href="https://www.graylog.org/">Graylog</a> which stores the data in an Elasticsearch (ES) cluster. Graylog is quite a powerful tool with options to manage ES indexes, alerting, dashboards and many more cool features.</p><h3>Metrics</h3><p>Our engineers use metrics heavily throughout the application. Different metric types like counts, histograms and gauges are used to give information about a particular process or activity over time. However, these were not standardised.</p><p>Every application has inputs and outputs like: Amazon Simple Queue Service (SQS), RabbitMQ, Kafka, HTTP/gRPC endpoints etc via which data is ingested, processed and passed on to another application or stored for persistence. For each of these inputs and outputs, it is important to measure all the key metrics. This will determine if the issues are with the application itself or its dependencies.</p><p>What are the key metrics we look at?</p><ul><li>Traffic / Usage</li><li>Latency</li><li>Errors</li><li>Saturation of resources (CPU / Memory/ IO)</li></ul><p>An application could use many services like gRPC, Kafka, Redis etc. In order to standardise the metrics we collected for each of these services, we created a list of key metrics. We verified that every application or feature which is ready to be released exposed these key metrics. This gave the team confidence in being able to find the right metrics to help troubleshoot the issue.</p><h4>Tools</h4><p>We use <a href="https://docs.datadoghq.com/metrics/">Datadog</a> for our metrics and monitors and PagerDuty for event management. We integrated Datadog monitors with PagerDuty, to ensure any metric breaching threshold(s) alerts an engineer.</p><h3>Why not just be happy with logs and metrics?</h3><p>Logs and metrics are great tools for observability. However, they have gaps like:</p><ul><li>With logs, we often struggle with not knowing what log messages to search for (due to the sheer volume of log data).</li><li>With metrics, the data is aggregated so it is helpful for a birds-eye view but not to drill down to specific parameters.</li><li>With Datadog, adding <a href="https://docs.datadoghq.com/developers/metrics/">custom metrics</a> helps in performing deeper analyses but that comes with a huge price tag <strong>$$$$</strong></li></ul><p>We add metrics and logs where we think something might go wrong but what about the unknown unknowns? What about understanding the end to end journey of a request?</p><h3>Traces</h3><p>Datadog’s <a href="https://docs.datadoghq.com/tracing/">APM</a> has been a game-changer for us in terms of observability.<br>We started with a basic set up following the guidelines <a href="https://docs.datadoghq.com/tracing/setup/dotnet-framework/?tab=environmentvariables">here</a>. With just the initial traces for HTTP requests, SQL queries and Redis operations we were able to derive more insights into our application. The visual appeal of tracing for me was a clear win over logs and metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpaXVI4lNBAQT-azMYx52Q.png" /><figcaption>Source: <a href="https://docs.datadoghq.com/tracing/visualization/">https://docs.datadoghq.com/tracing/visualization/</a></figcaption></figure><p>APM made the dependencies between various services visible with <a href="https://docs.datadoghq.com/tracing/visualization/services_map/"><em>services map</em></a>. <em>Watchdog</em> is another cool feature that will automatically detect anomalies in traces and provide alerts on system performance. Datadog also provides <a href="https://docs.datadoghq.com/tracing/guide/metrics_namespace/">Trace Metrics</a>, which are the following aggregate statistics over all the traces instrumented:</p><ul><li>Total requests and requests per second</li><li>Total errors and errors per second</li><li>Latency</li><li>Breakdown of time spent by service/type</li><li><a href="https://docs.datadoghq.com/tracing/faq/how-to-configure-an-apdex-for-your-traces-with-datadog-apm/">Apdex score</a> (web services only)</li></ul><p>In order to customise some of the functionality, we created our own wrapper around Datadog’s library for each language we support: C#, Java and Go. This gave us more control over creating traces and how we wanted to serialise the span context. We trace distributed calls by adding the span context (<em>traceId</em> and <em>spanId</em>) as a header value from the client. In the server, we could then use that span context as the parent giving visibility into the journey of a request across microservices.</p><p>Datadog samples the traces that are stored in the system. It is important to understand the <a href="https://docs.datadoghq.com/tracing/guide/trace_sampling_and_storage/?tab=java">sampling rules and storage</a> of traces in APM to help set the right sampling priority, especially when dealing with distributed tracing.</p><h3>Baking in APM</h3><p>Tracing does involve adding more code to the application, this could be a blocker for widespread adoption. In order to make it easier for our engineers, we created libraries that are easy to plug into existing services and in some cases, we added tracing in existing libraries. As of now, services like MSSQL, SQS, Couchbase, Cassandra, Kafka, Redis, gRPC are traced. This has significantly increased our visibility into the performance of our entire system.</p><h3>Conclusion</h3><p>Datadog has been key to our observability journey. We did consider Newrelic, Honeycomb and a few other options but Datadog has achieved a happy medium in terms of features and cost.</p><p>Knowing that if we get paged in the middle of the night, we have the right information to deal with the issue has helped us sleep better ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22796334958" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/observability-at-campaign-monitor-22796334958">Observability at Campaign Monitor</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multiple Spark Streaming Jobs in a Single EMR Cluster]]></title>
            <link>https://medium.com/cm-engineering/multiple-spark-streaming-jobs-in-a-single-emr-cluster-ca86c28d1411?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/ca86c28d1411</guid>
            <dc:creator><![CDATA[Adel Helal]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 22:01:02 GMT</pubDate>
            <atom:updated>2020-02-13T22:01:01.969Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/519/0*YOj3j5sne9MPYAUn.png" /></figure><p>A managed Spark cluster can always save time on infrastructure maintenance. <a href="https://aws.amazon.com/emr">EMR</a> (Elastic Map Reduce) is a tool that empowers data engineers to focus on building jobs to run on Spark rather than spending days managing the architecture of the cluster. However, that doesn’t come without its shortfalls.</p><p>As you may already know, you can only run one job at a time in an EMR cluster. This may not be too much of an issue when running a finite Spark job or batched process since there is a completion time, however that is not the case with Spark’s <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Structured Streaming</a> because there is no end time!</p><p>In a self-managed vanilla Spark cluster, it is possible to submit multiple jobs to a YARN resource manager and distribute the CPU and memory allocation to share its resources even when the jobs are running Structured Streaming.</p><p>Unfortunately submitting a job to an EMR cluster that already has a job running will queue the newly submitted job. It will remain in a pending state until the previous job completes, which never does in a streaming scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*emnVR6QZIX-rxsu1.png" /></figure><p>Having one EMR cluster for every job wouldn’t be the best solution however, because AWS infrastructure costs can grow pretty quickly, especially when consulting the <a href="https://aws.amazon.com/emr/pricing/">EMR pricing</a> page. The main cost issues revolve around having a master node created for every EMR cluster which only passes job requests to the core nodes, and those cores may not even have their resources fully utilized for smaller streaming tasks.</p><p>This means rethinking the strategy of writing streaming jobs for use within an EMR cluster in order to work around this limitation.</p><p>For this we need to look into improving the application layer.</p><h3>A Typical Structured Streaming Application</h3><p>The primary reason a data engineer will require the use of Structured Streaming is for consuming data from an event source, transforming it based on specific business rules or for the purpose of aggregation, and then finally persisting the result of the transformation in a data store. Typically the Scala code would require three main steps.</p><ol><li>Reading from the event payload</li></ol><pre>def <em>getEmailOpenedEvent</em>(rawEvent: Array[Byte]): List[EmailAggregateEvent] = {<br>  <strong>EmailOpened</strong>.parseFrom(rawEvent)<br>    .map(msg =&gt;<br>      EmailAggregateEvent(<br>        msg.getEmailId.toByteArray,<br>        msg.getSubscriberId.toByteArray,<br>        msg.getOpenedDateUtc.toTimestamp<br>        )<br>    ).toList<br>}</pre><p>Firstly, as shown in the above code, we will need the ability to parse the payload from the event stream. The EmailOpened type is generated based on a standard byte array format (e.g. <a href="https://github.com/protocolbuffers/protobuf">Protobuf</a>, <a href="https://avro.apache.org/">Avro</a> or <a href="https://thrift.apache.org/">Thrift</a>).</p><p>2. Transforming the messages into aggregated hourly buckets</p><pre>def <em>transformEmailOpened</em>(ds: Dataset[EmailAggregateEvent])<br>  : Dataset[EmailAggregateData] = {<br>  import ds.sparkSession.implicits._</pre><pre>  ds.withWatermark(&quot;timestamp&quot;, &quot;24 hours&quot;)<br>    .groupBy(<br>      col(&quot;emailId&quot;)<br>      window(col(&quot;timestamp&quot;), &quot;1 hour&quot;)<br>    )<br>    .agg(count(&quot;*&quot;).as(&quot;eventCount&quot;))<br>    .withColumn(&quot;timestampBucket&quot;, col(&quot;window.start&quot;))<br>    .as[EmailAggregateData]<br>}</pre><p>As shown above, a typical solution on how to work with streams is by aggregating the number of events based on a timestamp and maintaining an hourly bucket, which is then parsed as an EmailAggregateData type. For more information, the online programming guide on <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time">window operations in Structured Streaming</a> can provide further detail on this type of implementation.</p><p>3. Starting the process</p><pre>def <em>process</em>(topic: String, sql: String): Unit = {<br>  import spark.implicits._</pre><pre>val stream = SparkSession.builder.getOrCreate<br>    .<strong>readEventStream</strong>(config.brokers, topic)<br>    .flatMap(<em>getEmailOpenedEvent</em>(_))</pre><pre>  val query = <em>transformEmailOpened</em>(stream)<br>      .writeStream<br>      .foreach(<strong>databaseWriter</strong>(sql))<br>      .trigger(Trigger.ProcessingTime(&quot;5 minutes&quot;))<br>      .start</pre><pre>  query.awaitTermination<br>}</pre><p>Lastly, we will need to start the process of consuming and writing to our data store. The readEventStream() function is an abstraction of SparkSession.readStream() to consume from an event source (either <a href="https://kafka.apache.org/">Apache Kafka</a>, <a href="https://aws.amazon.com/kinesis">Amazon Kinesis</a>, or <a href="https://azure.microsoft.com/en-au/services/event-hubs">Azure Event Hub</a>). The databaseWriter() function creates a JDBC connection to execute SQL upsert commands.</p><p>There is an issue with this approach however.</p><p>If it is not already obvious, this code can quickly grow. Similar code will need to be duplicated for every event to consume from and aggregate on. Moreover, this code would require separate EMR clusters, so the code will need to be consolidated.</p><h3>Consolidating repeated code using generics</h3><p>Since Scala is flexible in providing a hybrid of both functional programming and Java&#39;s familiar object-oriented concepts, leveraging the use of base classes can provide great advantages.</p><pre>class Event extends Product {<br>  // base functions<br>}</pre><pre>case class EmailAggregateEvent(<br>  emailId:      Array[Byte],<br>  subscriberId: Array[Byte],<br>  timestamp:    Timestamp<br>) <strong>extends Event</strong> {<br>  // override functions<br>}</pre><p>There is one important note to consider about the above inheritance. Although Scala is flexible in enabling object-oriented programming techniques, we are still dealing with the caveats of Spark, namely serializable objects required for passing information between executors. Attempting to use a trait will actually throw an error:</p><pre>Error: scala.ScalaReflectionException: &lt;none&gt; is not a term</pre><p>This is because it simply cannot serialize a non-concrete type, and the Spark runtime is not (yet) clever enough to understand inheritance, so it will attempt to serialize the actual trait itself, which is impossible.</p><p>Moreover, when using a concrete class as the base class it will need to extend off the Product trait. This is because a case class automatically extends the Product trait which allows Spark to recognizes complex types at runtime for serialization.</p><p>Inheritance will enable us to achieve the following:</p><pre>case class Job[T &gt;: Event, U &gt;: Data](<br>  topic:     String,<br>  consume:   Array[Byte] =&gt; List[T],<br>  transform: Dataset[T] =&gt; Dataset[U],<br>  sql:       String<br>)</pre><p>The Job type can now help substitute any job-specific implementation in the process() function as follows:</p><pre>def <em>process</em>[T &gt;: Event, U &gt;: Data](<strong>job</strong>: Job[T, U]): Unit = {<br>  import spark.implicits._</pre><pre>val stream = SparkSession.builder.getOrCreate<br>    .readEventStream(config.brokers, <strong>job</strong>.topic)<br>    .flatMap(<strong>job</strong>.consume(_))</pre><pre>  val query = <strong>job</strong>.transform(stream)<br>      .writeStream<br>      .foreach(databaseWriter(<strong>job</strong>.sql))<br>      .trigger(Trigger.ProcessingTime(&quot;5 minutes&quot;)))<br>      .start</pre><pre>  query.awaitTermination<br>}</pre><p>Unfortunately this will still not be enough.</p><p>In the original transformEmailOpened() function above, there are implicit conversions which can no longer work when using inheritance. The Spark API provides an easy to use property called as[T] which returns a new Dataset where each field is mapped to columns of the same name of the given type T. During runtime Spark will need a little help to understand what type is required explicitly, so the transformEmailOpened() function will need to change as follows:</p><pre>def <em>transformEmailOpened</em>(ds: Dataset[EmailAggregateEvent])<br>  : Dataset[EmailAggregateData] = {<br>  import ds.sparkSession.implicits._<br>  ds<br>    // explicitly map to concrete type<br>    <strong>.map(row =&gt; EmailAggregateEvent(<br>      row.emailId,<br>      row.subscriberId,<br>      row.timestamp<br>    ))</strong><br>    .withWatermark(&quot;timestamp&quot;, &quot;24 hours&quot;)<br>    .groupBy(<br>      col(&quot;emailId&quot;),<br>      window(col(&quot;timestamp&quot;), &quot;1 hour&quot;)<br>    )<br>    .agg(count(&quot;*&quot;).as(&quot;eventCount&quot;))<br>    .withColumn(&quot;timestampBucket&quot;, col(&quot;window.start&quot;))<br>    // replace as[T] with explicit map<br>    <strong>.map(row =&gt; EmailAggregateData(<br>      row.getAs(&quot;emailId&quot;),<br>      row.getAs(&quot;timestampBucket&quot;),<br>      row.getAs(&quot;eventCount&quot;)<br>    ))</strong><br>}</pre><p>Inheritance can now work at runtime, and will enable our code to scale when more events are required for consumption. Adding multiple jobs can now be relatively easy.</p><p>However, we cannot simply just write a jobs.foreach() implementation and expect it to work since it will never iterate past the first streaming job. This is where concurrency plays an important role in the overall solution.</p><h3>Concurrency</h3><p>Enabling parallel programming in Scala is very simple. The .par() function enables parallel processing of a collection in an extremely concise way. We can take advantage of this by wrapping the entire functionality of the process() function into a parallel foreach() loop as follows:</p><pre>jobs.<strong>par</strong>.foreach(job =&gt; {<br>  process(job)<br>})</pre><p>There is one thing that hasn’t been considered with the above code though. The SparkSession that is used within the process() function is a static object and only one can be active per Spark cluster. However that doesn&#39;t mean we cannot start multiple streaming jobs using the same SparkSession! The SparkSession.readEventStream() functionality won&#39;t be a problem running in parallel even though the function is being called multiple times against the same static Spark session.</p><p>In terms of concurrency though, the .par() functionality is a multi-threaded implementation. In the JVM framework each thread runs on one CPU unit (either a physical processor or hyper-threaded processor). This means a multi-threaded approach would only work if the number of jobs required is less than the number of processors in the EMR cluster. To find out how many cores (and essentially how many separate threads the job can run on) the following value can be outputted:</p><pre>Runtime.getRuntime.availableProcessors</pre><p>The problem with this solution is cost. When considering that each job may only utilize 5% or less of the CPU units, we may have more cores in the EMR cluster than is necessary. As an example, running 12 jobs concurrently means the next instance size up to support this is m4.4xlarge which has 16 cores. That is a total of 4 cores that will sit idle never to be used. Moreover, each used core may also be underutilized.</p><p>Considering asynchronous programming rather than a multi-threaded approach suddenly becomes more relevant, and Scala&#39;s Futures can come in handy with only a few simple changes:</p><pre>val executionContext = ExecutionContext.fromExecutorService(<br>  Executors.newFixedThreadPool(maxConcurrentJobs)<br>)</pre><pre>val tasks: List[Future[Unit]] = jobs<br>  .map(job =&gt;<br>    <strong>Future</strong> {<br>      process(job)<br>    }(executionContext)<br>  )</pre><pre>Await.result(Future.sequence(tasks)(implicitly, executionContext), Duration.Inf)</pre><p>With this minimal code change we can have massive benefits in cost reduction. By default the thread pool count (i.e. the maximum number of concurrent runs) is actually the number of processors, just like the multi-threading approach. This is where the executionContext comes in handy to explicitly set the number of concurrent executions with the maxConcurrentJobs variable. This number can be as large as necessary to handle as many job across the entire EMR cluster regardless of how many cores are provisioned. The limit can now be based on the exact number of cores and the resource allocation required per individual job.</p><h3>Conclusion</h3><p>Even with the limitations of EMR clusters allowing only one job to run at any given time, as well as considering that only one static SparkSession object exists per Spark job, there are still efficient ways to work around these limitations.</p><p>Jobs can be refactored to enable consolidation of code repetition which in turn enables the possibilities of concurrent processing. This can significantly reduce cost of infrastructure and keep a clean code base that can be extensible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca86c28d1411" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/multiple-spark-streaming-jobs-in-a-single-emr-cluster-ca86c28d1411">Multiple Spark Streaming Jobs in a Single EMR Cluster</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 tips  in writing a spark job in Scala]]></title>
            <link>https://medium.com/cm-engineering/10-tips-in-writing-a-spark-job-in-scala-cc837149a173?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/cc837149a173</guid>
            <category><![CDATA[bts]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Binzi Cao]]></dc:creator>
            <pubDate>Tue, 14 Jan 2020 21:01:01 GMT</pubDate>
            <atom:updated>2020-01-14T21:01:01.154Z</atom:updated>
            <content:encoded><![CDATA[<h3>10 tips of writing a spark job in Scala</h3><p>At Campaign Monitor, our data team has been working on many <a href="https://spark.apache.org/"><strong>Spark</strong></a> jobs. <a href="https://www.scala-lang.org/"><strong>Scala</strong></a> has been adopted as one of our main programming languages for big data development. We use both <strong>Scala</strong> and <strong>Python</strong> in our projects. To decide which language to use is case by case and can be influenced by the following considerations:</p><ul><li>For machine learning related projects , we tend to choose <strong>Python</strong> because we can easily integrate many ML libraries written in <strong>Python</strong>.</li><li>For ETL, Streaming Application and Data Management, we use <strong>Scala</strong>.</li></ul><p>In this article we will focus on Scala.</p><h3>Why Scala?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*gn2p6bYk7HmWCWDGWtN0Sg.png" /><figcaption><a href="https://www.scala-lang.org/"><strong>Scala</strong></a></figcaption></figure><ul><li>Scala is one of the primary languages for many big data frameworks like <a href="https://spark.apache.org/">Spark</a> , <a href="https://kafka.apache.org/">Kafka</a>, <a href="https://flink.apache.org/">Flink</a> and so on.</li><li>Scala supports object oriented programming. Developers from a Java background or other similar frameworks can easily learn and start using it productively.</li><li>Scala supports functional programming. The concepts in functional programming such as immutable data structure, pure functions and APIs like map, flatMap, filter are very common in big data frameworks. So the language itself can help data engineers understand and learn big data concepts in a very natural and easy way.</li><li>Scala can also reuse a lot of existing Java frameworks but provides a less boiler-plate code.</li></ul><p>However, writing a spark job in Scala can be quite a learning curve for beginners, so in this blog we’d like to share our best practices and tips at Campaign Monitor.</p><h3>1. Make Master optional</h3><p>When we deploy a spark job to a cluster, <strong>spark-submit</strong> command sets the <strong>Master</strong> for the job directly. So you don’t need to set <strong>Master</strong> in the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/0*77u3Ki8x5CRa5NYN" /><figcaption><a href="https://aws.amazon.com/emr/">EMR</a></figcaption></figure><p>However, if we want to run the job via <strong>sbt</strong> locally, we do need to set the Master to <strong>local</strong> explicitly. A small trick is making the <strong>Master</strong> an optional configuration or parameter like below:</p><pre>def getSparkSession(<br>        master:  Option[String],<br>        appName: String<br>    ): SparkSession = {</pre><pre>val builder =<br>        SparkSession.builder<br>          .appName(appName)</pre><pre>master match {<br>        case Some(m) =&gt; builder.<strong>master(m)</strong>.getOrCreate<br>        case _       =&gt; builder.getOrCreate<br>      }<br>}<br>def main(args: Array[String]): Unit = {</pre><pre>val spark: SparkSession =<br>        Spark.getSparkSession(args.headOption, &quot;my spark job&quot;)<br>.....<br>}</pre><p>So to run the job locally, you can just use <strong>sbt &#39;run local[*]&#39;</strong>, while in the cluster side, <strong>spark-submit</strong> still works without any code change or configuration updates.</p><h3>2. Use type-safe configurations</h3><p>A spark job may have a lot of configuration, using <a href="https://github.com/lightbend/config">typesafe</a> configuration is always a good choice. The <a href="https://pureconfig.github.io/">PureConfig</a> library supports environment variables and java system properties to override the configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*c6ltEewCqityxu0xQZ6dlg.png" /><figcaption><a href="https://github.com/pureconfig/pureconfig">PureConfig</a></figcaption></figure><p>The configuration is based <a href="https://github.com/lightbend/config/blob/master/HOCON.md">HCON</a> file as below:</p><pre>dataFolder = “/tmp/data” <br>dataFolder = ${?DATA_FOLDER}</pre><pre>env = “dev”<br>env = ${?ENVIRONMENT}</pre><p>Loading the configuration is as simple as a couple of lines of code:</p><pre>object Settings {</pre><pre>   //Use CamelCase configuration field names <br>   implicit def hint[T] =<br>      ProductHint[T](ConfigFieldMapping(CamelCase, CamelCase))</pre><pre>   val settings = <strong>loadConfigOrThrow</strong>[Settings]<br>  }</pre><p>If you have a configuration file application.conf, you can run the job in many different ways</p><ul><li>To run the job in local</li></ul><pre>sbt &#39;run local[*]&#39;</pre><ul><li>To run the job in a cluster with spark-submit</li></ul><pre>spark-submit --deploy-mode cluster \<br>--class  mysparkjob.main  \<br>--conf &quot;-DdataFolder=s3://mybucket/data -Denv=prod &quot; \<br>--conf &quot;-DdataFolder=s3://mybucket/data -Denv=prod&quot; \<br><a href="http://nexus/yourapplication.jar">http://yourrepo.com/</a>mysparkjob<a href="http://nexus/yourapplication.jar">.jar</a></pre><ul><li>To use environment variables in local</li></ul><pre>DATA_FOLDER=/tmp/bucket ENVIRONMENT=prod sbt run</pre><ul><li>To use environment variables in EMR cluster</li></ul><pre>{<br>      &quot;Classification&quot;: &quot;spark-env&quot;,<br>      &quot;Configurations&quot;: [<br>        {<br>          &quot;Classification&quot;: &quot;export&quot;,<br>          &quot;Properties&quot;: {<br>            &quot;DATA_FOLDER&quot;: &quot;s3://mybucket/data&quot;, <br>            &quot;ENVIRONMENT&quot;: &quot;prod&quot;<br>          }<br>        }<br>      ],<br>      &quot;Properties&quot;: {}<br>  }</pre><h3>3. Build common file system APIs</h3><p>A spark job is normally deployed in a remote cluster in production, which is <strong>EMR</strong> in our case. If there is code that needs to read or write a file in <a href="https://aws.amazon.com/s3/"><strong>S3</strong></a>, it can become difficult for local development or when adding unit tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/0*VfwX_Ldnu8dMW7wG" /><figcaption><a href="https://aws.amazon.com/s3/">S3</a></figcaption></figure><p>A good practice would be to build some common file system APIs , which supports multiple file systems such as <strong>Local FS, S3</strong>, and <strong>HDFS</strong> .</p><pre>object FileSystem {</pre><pre>  def apply(path: String): FileSystem = {<br>      val uri = new URI(path)</pre><pre>       uri.getScheme match {<br>        case &quot;s3&quot; =&gt; S3FileSystem<br>        case &quot;hdfs&quot; =&gt; S3FileSystem<br>        case _    =&gt; LocalFileSystem<br>      }<br>     }</pre><pre>}</pre><pre>trait <strong>FileSystem</strong> {<br>    def stringContent(path: String): String</pre><pre>    def exists(path:  String): Boolean<br>    def copy(srcPath: String, destPath: String): Unit</pre><pre>    def copyToLocal(srcPath:  String, destPath: String): Unit<br>    def copyToRemote(srcPath: String, destPath: String): Unit</pre><pre>    def delete(paths:       Seq[String]): Seq[(String, Boolean)]<br>    def listFiles(path:     String):      List[String]<br>    def getSubFolders(path: String):      List[String]</pre><pre>    def close(): Unit</pre><pre>}<br>//Implement each filesystems<br>case class <strong>LocalFs</strong>(path:String) extends FileSystem {...}<br>case class <strong>HdfsFs</strong>(path:String) extends FileSystem {...}<br>case class <strong>S3Fs</strong>(path:String) extends FileSystem {...}</pre><h3>4. Accelerate the sbt build</h3><p>Building an application jar file could be very slow with <strong>sbt assembly</strong>. It needs to download all the dependencies to the machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/0*50LUVj2aFOfGPETq" /><figcaption><a href="https://www.docker.com/">Docker</a></figcaption></figure><p>To make the build faster, we created a task in our pipeline, which generates a <strong>docker image. </strong>The image includes all the dependencies for the application.</p><p>The docker image improved the build time from <em>20 minutes</em> to <em>1 minute</em><strong>.</strong> The docker file to create the build image is like below:</p><pre>FROM Your-docker-with-sbt:latest<br>COPY . /src</pre><pre>WORKDIR /src</pre><pre>RUN sbt assembly</pre><p>Every time you add a new dependency to your <strong>build.sbt</strong> file, just publish a new version of this docker image.</p><h3>5. Manage library dependencies</h3><p>We always use sbt assembly to build a fat jar which includes all the required third party dependencies. There are some basic steps to follow when you manage the dependencies in build.sbt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*d5Y_D2fRgTY_5IVtOq7oiA.png" /></figure><ul><li>Set all dependencies which exist in EMR as provided, for example:</li></ul><pre>&quot;org.apache.spark&quot; %% &quot;spark-core&quot; % sparkVersion % &quot;provided&quot;,<br>&quot;com.amazonaws&quot; % &quot;aws-java-sdk-s3&quot; % awsSDKVersion % &quot;provided&quot;,</pre><ul><li>Make test libraries as test only</li></ul><pre>&quot;org.scalatest&quot; %% &quot;scalatest&quot; % &quot;3.0.4&quot; % Test,</pre><ul><li>Shade some libraries if they have some conflicts</li></ul><pre>assemblyShadeRules in assembly := Seq(<br>    ShadeRule.rename(&quot;com.google.common.**&quot; -&gt; &quot;repackaged.com.google.common.<a href="http://twitter.com/1">@1</a>&quot;).inAll,<br>    ShadeRule.rename(&quot;com.google.protobuf.**&quot; -&gt; &quot;repackaged.com.google.protobuf.<a href="http://twitter.com/1">@1</a>&quot;).inAll,<br>    ShadeRule.rename(&quot;org.json4s.jackson.**&quot; -&gt; &quot;repackaged.org.json4s.jackson.<a href="http://twitter.com/1">@1</a>&quot;).inAll<br>  )</pre><ul><li>Exclude some jars if you still see any errors in sbt assembly</li></ul><pre>assemblyExcludedJars in assembly := {<br>    val cp = (fullClasspath in assembly).value<br>    val excludedJars = List(<br>      &quot;protobuf-java-3.2.0.jar&quot;,<br>      &quot;grpc-protobuf-lite-1.8.0.jar&quot;,<br>      &quot;grpc-protobuf-1.8.0.jar&quot;,<br>      &quot;grpc-core-1.8.0.jar&quot;,<br>      &quot;grpc-context-1.8.0.jar&quot;,<br>      &quot;hadoop-yarn-api-2.6.5.jar&quot;<br>    )<br>    cp filter { x =&gt;<br>      excludedJars.contains(x.data.getName)<br>    }<br>  }</pre><h3>6. Run with provided dependency</h3><p>If you have a spark job using <a href="https://www.scala-sbt.org/"><strong>sbt</strong></a>, the spark related dependencies should always be set to provided, to run the job in local via sbt run, it will complain about the dependency missing. A small tip to make it work is to add the below lines into your build.sbt file</p><pre>(run in Compile ) :=<br>  Defaults.runTask(<br>         fullClasspath in Compile,<br>         mainClass in (Compile, run),<br>         runner in (Compile, run)</pre><pre>).evaluated</pre><h3>7. Publish the application</h3><p>To run the spark job in a cluster in EMR, you’ll need to publish the jar in a common place that the EMR instances could access it. Two typical ways are as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/0*U_WI6SaUroFdoOns" /></figure><ul><li>S3</li></ul><pre>s3://my-release-bucket/my-spark-job/my-spark-1.1.1.jar</pre><ul><li>Maven</li></ul><pre>https://my-maven-server/repo/spark/my-spark-1.1.1.jar</pre><p>Using your own<strong> Maven repository</strong> might be easier as you don’t have to keep application binaries in multiple places.</p><h3>8. Performance tracking</h3><p>In the spark job, we always do performance analysis to help us to identify which code takes too long.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/0*QcDUPhnhiri3Sn5T" /></figure><p>A very simple function we can use is as below:</p><pre> def time[R](info: String)(block: =&gt; R): (R, Long) = {<br>      Log.info(s&quot;$info started..&quot;)</pre><pre>      val t0     = System.nanoTime()<br>      val result = block // call-by-name<br>      val t1     = System.nanoTime()</pre><pre>      val duration = (t1 - t0) / 1000000<br>      Log.info(s&quot;$info - Elapsed time: $duration ms&quot;)<br>      (result, duration)<br> }<br>time(&quot;Loading data from s3 bucket&quot;)(spark.read.parquet(&quot;s3://my-bucket/data&quot;))</pre><p>You can integration this function into your dashboard such as <a href="https://www.datadoghq.com/"><strong>Datadog</strong></a></p><h3>9. Unified deployment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*Jda6ajnXkOX-PyDYwyq-3w.png" /><figcaption><a href="https://airflow.apache.org/">Apache Airflow</a></figcaption></figure><p>We use <a href="https://www.terraform.io/"><strong>Terraform</strong></a> and <a href="https://airflow.apache.org/"><strong>Airflow</strong></a> as our deployment pipeline. The deployments in these systems could be very different. So we decided to wrap the <strong>terraform</strong> commands into Airflow DAG steps, by doing this, we have:</p><ul><li><em>One </em>way to create a cluster and deploy the job from local, <a href="https://www.jetbrains.com/teamcity/">TeamCity</a> or Airflow</li><li>All configurations or parameter changes in source code</li><li>Minimize the configuration complexity in different systems</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*ij7a4F156bwbBXXGnKAsZw.png" /><figcaption>Airflow DAG flow</figcaption></figure><p>The deployment is always triggered by <strong>terraform</strong> and we only have one single variable release_version for different version deployment.</p><pre>terraform apply -var &#39;release_version=1.1.1&#39;</pre><h3>10. Some useful libraries</h3><p>There are several good open-source libraries we want to share:</p><ul><li><a href="https://github.com/pureconfig/pureconfig"><strong>PureConfig</strong></a>: A boilerplate-free library for loading configuration files</li><li><a href="https://github.com/lloydmeta/enumeratum"><strong>enumeratum</strong></a>: A type-safe, reflection-free, powerful enumeration implementation for Scala with exhaustive pattern match warnings and helpful integrations.</li><li><a href="https://github.com/pathikrit/better-files"><strong>better-files</strong></a>: A dependency-free <em>pragmatic</em> thin Scala wrapper around Java NIO.</li><li><a href="https://github.com/nscala-time/nscala-time"><strong>nscala-time</strong></a>: A new Scala wrapper for Joda Time</li><li><a href="https://github.com/circe/circe"><strong>circe</strong></a>: A JSON library for Scala</li></ul><p>Thanks for reading and stay tuned, we’ll share our tips for <strong>Python </strong>development in <strong>Spark</strong> and <strong>ML</strong> in a separate blog soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc837149a173" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/10-tips-in-writing-a-spark-job-in-scala-cc837149a173">10 tips  in writing a spark job in Scala</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we built the first spark streaming application at CM]]></title>
            <link>https://medium.com/cm-engineering/how-we-built-the-first-spark-streaming-application-at-cm-3e9db846bf0f?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/3e9db846bf0f</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[emr]]></category>
            <dc:creator><![CDATA[Binzi Cao]]></dc:creator>
            <pubDate>Wed, 22 May 2019 01:28:19 GMT</pubDate>
            <atom:updated>2019-05-22T01:28:19.155Z</atom:updated>
            <content:encoded><![CDATA[<p>Building a real time streaming application with large volumes of data is always challenging, especially when starting from scratch. At Campaign Monitor, I was lucky enough to work with a group of talented people to design and implement our first spark structured streaming application. So are you ready to take 5 minutes to read our story?</p><h3><strong>The problem</strong></h3><p>At Campaign Monitor, we have more than 250,000 clients all around the world. As an email marketing company, we send over 100 million emails for our clients every day.</p><p>When a user <strong>opens </strong>or <strong>clicks </strong>an email from us, we receive an event in our system. It turns out that our system receives more than 30 million opens every day. What’s more, this number continues to grow.</p><p>So what problem did we want to resolve?</p><p><strong><em>Find the total opens for a campaign within any date range in real time.</em></strong></p><p>Looks simple? What options do we have ?</p><p><strong><em>Monolith</em></strong></p><p>We do have a big monolith code base and database, and generating data for real time would mean having to query a db. Our db has more than 100 billion records to query with<strong> group by</strong>. This will lock our db and stop any other service from running :(</p><p><strong><em>Microservice</em></strong></p><p>Microservices have been widely adopted at CM and we also have all the events published to Kafka topics. A key thing to note is that microservices do not scale well with data growing , and as a result, are unable to perform real time data aggregation efficiently.</p><p>So it didn’t take very long before we reached the design with <strong>spark structured streaming</strong> via Kafka, and below is the design</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*L2ff3xawAeGXS2L8Hc1ufg.png" /></figure><p>What does our spark streaming job do? A streaming job basically loads all the events from Kafka, aggregates data in real time and writes data to the database. Here is a flow of spark micro batch jobs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QczlRMgdLAJ5GxgkiZV9IA.jpeg" /></figure><p>It did not take us very long to finish writing all the spark jobs because writing the spark job in Scala is super simple. We’d love to share some challenges we faced during the project and how we resolved them one by one.</p><h3><strong>Challenges</strong></h3><p><strong>Offset Management</strong></p><p>Each streaming job running in <a href="https://aws.amazon.com/emr/">EMR</a> (Elasic Map Reduce) has a checkpoint folder in our S3 bucket. After we finished the implementation of all the jobs , we started thinking about <strong>disaster recovery</strong> when we lose a checkpoint folder. Of course , this is not likely to happen, but it is better to be safe than sorry.</p><p>Managing offsets could be complicated and there isn’t a proper way to record the offsets in a streaming job in a db. So we approached it in a different way: <strong><em>why don’t we rely on timestamps rather than offsets?</em></strong></p><p>An event in Kafka event has 2 timestamps: <strong>event(data) timestamp</strong> and <strong>Kafka timestamp</strong>. The event timestamp is normally earlier than the Kafka stamp.</p><p>Our solution was that when a job needed to restart without a checkpoint folder, it always starts from an <strong>event timestamp</strong> (of course we know when our job failed), and uses the timestamp to calculate the <strong>Kafka offsets</strong> to start with. We start the job from these offsets and <strong>filter out old events</strong> before the timestamp in the query, then the job starts writing to db and will catch up with the latest offsets eventually.</p><p>So basically our job restarting flow looks like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8R6QIWSKiVSnLbRdi5Z3Bg.jpeg" /></figure><h3>Checkpoint management</h3><p>It turned out that a streaming job can have a huge amount of checkpoint files, which introduced a cost to us. We started using <strong>HDFS</strong> instead of <strong>S3</strong> for our checkpoint folder, however, that is not easy in AWS. Check out this blog by my colleague about the story:</p><p><a href="https://cm.engineering/using-hdfs-to-store-spark-streaming-application-checkpoints-2146c1960d30">Using HDFS to store Spark Streaming application checkpoints</a></p><h3>Data back-filling</h3><p>One of the most challenging things for staring a streaming job is how to back-fill historical data, especially for more than 30 billion historical events.</p><p>To make the back-filling work run smoothly, we exported the database data to <strong>CSV</strong> files into S3, and started a spark job to convert the data to <strong>parquet</strong> (for other data analytics), and loaded parquet data to our new Aurora DB.</p><p>The workflow is as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tiA3eeqOy8diiuLanpz5Q.png" /></figure><h3>Other learnings</h3><h4>Job Retry</h4><p>A streaming job is supposed to run 24x7 while it can still fail for reasons such as db connection issues and S3 API failures . So having some backing off logic to restart your <strong>whole streaming job</strong> is necessary.</p><h4>EMR/Aurora Resizing</h4><p>One beauty of having Spark jobs and DBs using AWS infrastructure is that you can always resize your spark cluster based on your volume of data to achieve the best cost-effective infrastructure.</p><h4>Kafka with transactional producers</h4><p>Kafka introduced transactional producer and we started using it for a while. With transactional producer, a topic could have some special offsets for transaction. Unfortunately it was not supported in the Kafka consumers in spark before version 2.4, and we had to wait for spark 2.4 to run our streaming jobs with our Kafka topics</p><h3>Other blogs by my awesome colleagues at CM</h3><ul><li><a href="https://cm.engineering/using-hdfs-to-store-spark-streaming-application-checkpoints-2146c1960d30">Using HDFS to store Spark Streaming application checkpoints</a></li><li><a href="https://cm.engineering/automating-deployments-for-sparks-structured-streaming-on-aws-emr-cf940d6a588a">Automating Deployments for Spark’s Structured Streaming on AWS EMR</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e9db846bf0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/how-we-built-the-first-spark-streaming-application-at-cm-3e9db846bf0f">How we built the first spark streaming application at CM</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using HDFS to store Spark Streaming application checkpoints]]></title>
            <link>https://medium.com/cm-engineering/using-hdfs-to-store-spark-streaming-application-checkpoints-2146c1960d30?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/2146c1960d30</guid>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Mikael Paladiants]]></dc:creator>
            <pubDate>Tue, 30 Apr 2019 01:01:00 GMT</pubDate>
            <atom:updated>2019-12-01T02:12:15.186Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*MiTwWHQS39AQqlwye-HVag.png" /></figure><p>When you want to run a Spark Streaming application in an <a href="https://aws.amazon.com/emr/">AWS EMR</a> cluster, the easiest way to go about storing your <a href="https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing">checkpoint</a> is to use <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-fs.html">EMRFS</a>. It uses S3 as a data store, and (optionally) DynamoDB as the means to provide <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-consistent-view.html">consistent</a> reads. This approach allows you to freely destroy and re-create EMR clusters without losing your checkpoints. Also, enabling consistent reads is simply a matter of checking the corresponding checkbox:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*Y5me1KkhYT8pujkb" /><figcaption><strong>Fig. 1.</strong> Enabling EMRFS consistent view</figcaption></figure><h3>The problem</h3><p>But this convenience comes at a price, literally. Here in the Insights team at Campaign Monitor, we found that the cost of using EMRFS to store the checkpoints of our Spark jobs constituted about 60% of the overall EMR costs. The cost distribution was: S3–80%, DynamoDB — 20%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*B-J4R9mkUGYiKJR9" /><figcaption><strong>Fig. 2.</strong> An EMR cluster with EMRFS (S3 + DynamoDB) as a checkpoint store</figcaption></figure><p>One of the reasons of cost increase is the complexity of streaming jobs which, amongst other things, is related to:</p><ul><li>the number of Kafka topics/partitions read from</li><li><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking">watermark</a> length</li><li><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers">trigger</a> settings</li><li>aggregation logic</li></ul><p>More complex jobs have larger checkpoints. Some our checkpoints contain more than 25,000 files of total size of around 1Gb, and get updated every 90 seconds. This generated a lot of S3/DynamoDB requests.</p><p>It hadn’t been too long until we decided to cut that cost down.</p><h3>Our solution</h3><p>Here is the diagram of the solution we have gone with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*dILZ7ZP6UfK6xGpL6-sb8w.png" /><figcaption><strong>Fig. 3.</strong> HDFS replaces EMRFS</figcaption></figure><p>Let’s walk through it quickly. HDFS is a part of your EMR cluster and is available to it out of the box. That is where we store checkpoints now. Not only does this not incur any extra cost, but it also gets you more performant reads and writes since you don’t have to go through two separate AWS services (S3 and DynamoDB).</p><p>The downside, of course, is that once you destroy your cluster, HDFS and checkpoint data are also gone. So we need a way around that. This is where the Cron+S3+TeamCity trio comes into play. There is now a Cron job running on the master node of every EMR cluster we create. Its responsibility is to back up checkpoints to S3 (currently, once every 3 hours). First, we copy checkpoints from HDFS to the local disk by using this command:</p><pre>hdfs dfs -copyToLocal hdfs:///[checkpoint-dir] /tmp/[checkpoint-dir]</pre><p>Then, from local it goes to a time-stamped folder in S3, courtesy of the following command:</p><pre>aws s3 cp --recursive /tmp/[checkpoint-dir] s3://[destination-path]</pre><p>The process takes around 4 minutes for our busiest Spark job. Note that we don’t want to copy from HDFS directly to S3 as this would use the cluster’s computing resources.</p><p>TeamCity is useful when we want to re-provision a cluster. As before, running a build creates a new or replaces the existing cluster. But now, additionally, it looks up the latest checkpoint backup by timestamp and <a href="https://aws.amazon.com/premiumsupport/knowledge-center/copy-s3-hdfs-emr/">copies</a> it to the newly provisioned cluster using <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html"><em>s3-dist-cp</em></a>.</p><p>This gives us a fully automated way of re-provisioning EMR clusters, storing checkpoints and their backups, all at a fraction of the original cost and with increased performance — a good reason to say <em>“win-win”</em>.</p><h3>Conclusion</h3><p>Using EMRFS as a checkpoint store makes it easier to get started with AWS EMR, but the cost of using it can get high for data-intensive Spark Streaming applications. Opting for HDFS with a little bit of extra work will rid you of most of that cost.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2146c1960d30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/using-hdfs-to-store-spark-streaming-application-checkpoints-2146c1960d30">Using HDFS to store Spark Streaming application checkpoints</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently Compiling Spark Jobs Built in Scala and SBT on Build Machines]]></title>
            <link>https://medium.com/cm-engineering/efficiently-compiling-spark-jobs-built-in-scala-and-sbt-on-build-machines-f5deab75b2e5?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/f5deab75b2e5</guid>
            <category><![CDATA[spar]]></category>
            <category><![CDATA[bts]]></category>
            <category><![CDATA[teamcity]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Adel Helal]]></dc:creator>
            <pubDate>Mon, 21 Jan 2019 23:31:00 GMT</pubDate>
            <atom:updated>2019-01-21T23:31:00.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*1g707ErKIn3IAZ-p.jpg" /></figure><p>Deploying a Spark job can be challenging, especially considering that one Spark job is never the same as another. Deployment times can vary significantly depending on a wide range of factors. Finding ways to make the deployment efficient can greatly improve the process, and that can be achieved with a few simple strategies.</p><p>Consider the following steps required to deploy a Spark job that is built in scala and compiled using sbt:</p><ol><li>Get latest project files <em>(requires </em><em>git or </em><em>subversion etc.)</em></li><li>Download dependencies <em>(requires </em><em>sbt)</em></li><li>Compile Spark application <em>(requires </em><em>sbt)</em></li><li>Run unit test <em>(requires </em><em>sbt)</em></li><li>Package Spark application to a .jar file <em>(requires </em><em>sbt)</em></li><li>Upload .jar file to an accessible location (e.g. s3) <em>(requires </em><em>aws-cli)</em></li><li>Spark-submit using the s3 file to the master node <em>(requires </em><em>spark)</em></li></ol><p>This can be achieved using a deployment pipeline created in a CI tool like TeamCity or Jenkins.</p><p>Usually build machines used for a deployment pipeline are provisioned so they are lightweight with very few applications installed. In fact, if the only application installed on a build machine is docker, then that is enough.</p><p>Since our build machines don’t have sbt installed, a docker image with scala and sbt is required to run the sbt assembly command. There are many that can be found on docker hub such as <a href="https://hub.docker.com/r/spikerlabs/scala-sbt/">spikerlabs/scala-sbt</a> which can achieve this. However, running sbt assembly using this docker image will take a significantly long time to complete, sometimes as long as 30 minutes! This is because all the necessary dependencies for your spark job will need to be downloaded before compiling your Spark application.</p><p>Performing this operation every time you need to deploy your Spark job is costly. So in order to improve the efficiency of builds to prevent these download times, the dependencies first need to be downloaded onto a specific sbt docker image that is tailored for your Spark job. This can then be used as part of the deployment pipeline.</p><p>The following steps will need to be carried out for this to be achieved:</p><p>1. cd into the project folder locally.</p><p>2. Run the following docker command to start a spikerlabs/scala-sbt container in interactive mode.</p><pre>docker run -i -v &quot;/$(pwd)/&quot;:/app -w &quot;//app&quot; --name &quot;my-scala-sbt-container&quot; &quot;spikerlabs/scala-sbt:scala-2.11.8-sbt-0.13.15&quot; bash</pre><p>Note that a specific scala and sbt version will alway be required otherwise the &quot;latest&quot; tag could fail during compilation with any breaking changes.</p><p>3. Once in interactive mode within the docker container, run the following commands within the container.</p><pre>&gt; sbt assembly #to download dependencies, compile, &amp; package the job<br>&gt; exit #once the packaging is complete</pre><p>4. This will create a docker container called my-scala-sbt-container which will need to be exported, then imported as an image, as follows:</p><pre>docker export --output=&quot;./my-scala-sbt.tar&quot; &quot;my-scala-sbt-container&quot;<br>docker import &quot;./my-scala-sbt.tar&quot; [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]</pre><p>Where [INTERNAL_REPOSITORY_URL] is a company wide docker repository location e.g. like <a href="https://www.sonatype.com/nexus-repository-sonatype">nexus</a>, and [VERSION_NUMBER] needs to be bumped up from a previous version.</p><p>Note that the import allows a container that has all the necessary dependencies to be converted into an image called [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER].</p><p>This will be needed when docker push is executed. Unfortunately the docker API for a pushcommand requires the image <em>name</em> to be exactly the same as the <em>docker repository url</em> which seems a little non-intuitive.</p><p>5. To publish the the local docker image to the internal docker repository, run the following docker push command:</p><pre>docker image push [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]</pre><p>Where [INTERNAL_REPOSITORY_URL] is a company wide docker repository location e.g. like <a href="https://www.sonatype.com/nexus-repository-sonatype">nexus</a>, and [VERSION_NUMBER] is the same as the previous step.</p><p>This can now help in the CI deployment pipeline, where the step to run sbt assembly can be done with the docker run command as follows:</p><pre>cat &lt;&lt;EOF &gt; assembly.sh <br>#!/usr/bin/env bash <br>set -ex <br>sbt assembly <br>EOF</pre><pre>docker run \<br>    --rm \<br>    -v &quot;/$(pwd)/..&quot;:/app \<br>    -v &quot;/$(pwd)/assembly.sh&quot;:/assembly.sh \<br>    -w &quot;//&quot; \<br>    --entrypoint=sh \<br>    [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER] \<br>    //assembly.sh</pre><p>That should improve the build timings by removing up to 30 minutes off the deployment pipeline, depending on how many dependencies are required for the Spark application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5deab75b2e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/efficiently-compiling-spark-jobs-built-in-scala-and-sbt-on-build-machines-f5deab75b2e5">Efficiently Compiling Spark Jobs Built in Scala and SBT on Build Machines</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating Deployments for Spark’s Structured Streaming on AWS EMR]]></title>
            <link>https://medium.com/cm-engineering/automating-deployments-for-sparks-structured-streaming-on-aws-emr-cf940d6a588a?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf940d6a588a</guid>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Adel Helal]]></dc:creator>
            <pubDate>Tue, 02 Oct 2018 00:01:02 GMT</pubDate>
            <atom:updated>2018-10-02T00:01:02.256Z</atom:updated>
            <content:encoded><![CDATA[<p>Anyone who’s familiar with submitting a Spark job will know it’s as simple as running a single command, so long as the Spark CLI is installed on the machine running that command:</p><pre>spark-submit --master ${MASTER_ENDPOINT} \<br>    --deploy-mode cluster \<br>    --class com.cm.spark.SparkApp \<br>    s3://bucket/jobs/${APP_JAR}</pre><p>This is easy enough when working with a self-managed vanilla Spark cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*Y_N3tyvEYVzLshokfV4NTQ.png" /></figure><p>AWS also provides a service called EMR (Elastic Map-Reduce) which offers hosting of Spark clusters, so the AWS CLI provides an equally effortless way to submit Spark jobs:</p><pre>aws emr add-steps cluster-id j-UJODR7SZ6L7L steps \<br>    Type=Spark,<br>    Name=&quot;Aggregator&quot;,<br>    ActionOnFailure=CONTINUE,<br>    Args=[<br>        --deploy-mode,cluster,<br>        --class,com.cm.spark.SparkApp,<br>        s3://bucket/jobs/${APP_JAR}<br>    ]</pre><p>This is essentially a wrapper around the spark-submit command. The <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html">AWS documentation</a> is relatively detailed in providing information for adding additional parameters to the command line arguments in order to provide all the features found in the spark-submit command. This makes it incredibly simple to create a deployment step in your CI pipeline.</p><p>Even if the build machines don’t have AWS CLI installed, the commands can be wrapped around a docker run call using an image called garland/aws-cli-docker which can be found on <a href="https://hub.docker.com/r/garland/aws-cli-docker/">Docker Hub</a>:</p><pre>cat &lt;&lt;EOF &gt; submit.sh<br>    #!/usr/bin/env bash<br>    set -ex<br>    <strong>aws emr add-steps cluster-id j-UJODR7SZ6L7L steps ...</strong><br>EOF</pre><pre>docker run \<br>    --rm \<br>    -e AWS_DEFAULT_REGION=${AWS_REGION} \<br>    -e AWS_ACCESS_KEY_ID=&quot;${AWS_ACCESS_KEY_ID}&quot; \<br>    -e AWS_SECRET_ACCESS_KEY=&quot;${AWS_SECRET_ACCESS_KEY}&quot; \<br>    -v &quot;/${APP_JAR}&quot;:/${APP_JAR} \<br>    -v &quot;/$(pwd)/submit.sh&quot;:/submit.sh \<br>    -w &quot;//&quot; \<br>    --entrypoint=sh \<br>    <strong>garland/aws-cli-docker:latest</strong> \<br>    //submit.sh</pre><p>The beautiful thing about docker is that the build machines only ever need one prerequisite: docker. Anything else that needs to be installed for deployment can instead be found as a docker image on <a href="https://hub.docker.com/">Docker Hub</a>. In this case the aws emr add-steps command is executed within a docker container and the build machines don&#39;t need to know anything about it!</p><p>However, there is a catch. This is only straightforward if you are under the assumption that your Spark job eventually completes, whether it returns a <strong>success</strong> or <strong>failure</strong> status. In fact, that is exactly the same assumption that the AWS documentation makes. Once a job completes, running the same command will add a new step on the cluster and the step will run without any issues.</p><p>But what about <em>Structured Streaming</em>? An ongoing Spark job with an open connection to listen to an event stream suddenly makes one simple command slightly more complicated when deployment of a code change is required.</p><p>This would require stopping a currently running step before adding a new step. <a href="https://docs.aws.amazon.com/cli/latest/reference/emr/cancel-steps.html">AWS documentation</a> has another command that can be useful in this scenario:</p><pre>aws emr cancel-steps<br>    --cluster-id j-UJODR7SZ6L7L<br>    --step-ids s-2V57YI7T5NF42</pre><p>Unfortunately the cancel-steps command can only remove a <strong>pending</strong> step i.e. one that isn&#39;t in a <strong>running</strong> state. In fact there is no API to terminate a running step at all and the only solution found in <a href="https://aws.amazon.com/premiumsupport/knowledge-center/cancel-emr-step/">AWS documentation</a> is to do the following:</p><ol><li>Log into the EMR master node (which will need a secure .pem file)</li><li>Run <strong>yarn application -list</strong></li><li>Find the application ID of the currently running <strong>job</strong></li><li>Run <strong>yarn application -kill ${APP_ID}</strong></li></ol><p>Obviously this is not a viable solution for automation since we will have to deal with ssh and passing .pem files around on build machines in order to achieve this process. No doubt your security team would have their own opinion on this approach.</p><h3>Rebuilding the cluster as part of the job deployment</h3><p>Alternatively, there is another <a href="https://docs.aws.amazon.com/cli/latest/reference/emr/terminate-clusters.html">AWS API call</a> that may be a useful workaround:</p><pre>aws emr terminate-clusters --cluster-ids j-UJODR7SZ6L7L</pre><p>This could achieve what we are looking for, if the following automation steps are executed:</p><ol><li>Terminate EMR cluster</li><li>Recreate EMR cluster using terraform or cloud formation</li><li>Add step to newly created cluster</li></ol><p>Terminating an entire cluster just to deploy a job may seem like overkill, however in the new world of infrastructure-as-code, it isn’t quite as insane as one might think. After consulting AWS experts and Spark specialists alike, both have agreed that these days it is actually encouraged to dedicate one spark job to a single Spark cluster.</p><p>The benefits are quite significant:</p><ul><li>Dedicated clusters mean each job can only affect itself without consuming resources from other jobs</li><li>If there are any fatal errors or memory leaks other jobs aren’t affected</li><li>The entire cluster’s resources can be specifically tailored for a single job, rather than maintaining resource allocation for many jobs which will require significant monitoring and any necessary resizing will affect all currently running jobs with downtime</li></ul><p>However, the downfalls are also notable:</p><ul><li>Rather than one single master with many shared cores, there will be a master for each job which will obviously mean more EC2 costs, <em>however</em> masters can be kept small since they’re only required to receive requests and distribute compute to slaves</li><li>Deployment pipelines will be slower since recreating clusters for every application code update will add significant time to the build process</li></ul><p>These downfalls may be deal breakers for some use cases, and it has given enough attention to the AWS experts after consultation to at least rethink the API strategies they’ve provided for a scenario such as Structured Streaming. The initial assumption that a Spark job is a short lived process is simply not enough.</p><h3>Using YARN’s REST API</h3><p>A better solution for now would actually be to utilise the YARN API instead. Since YARN is the orchestrator of the Spark nodes within the cluster, it provides many features in which to manage both the cluster and its jobs. There is <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">comprehensive documentation</a> on every command available via HTTP REST API calls. That means the build machines can avoid having to ssh into master nodes using .pem files and can send an HTTP request instead (so long as the inbound firewall permissions of the Spark cluster allow the build machines access to it - and that is a simple terraform or cloud formation configuration).</p><p>That means the following automation steps can achieve a successful deployment pipeline:</p><pre><strong><em># 1. Get a list of active jobs</em></strong><br>APPS=$(curl -s -X GET -H &quot;Content-Type: application/json&quot;<br>    &quot;http://${MASTER_ENDPOINT}/ws/v1/cluster/apps&quot;)</pre><pre><strong><em># 2. Parse the response payload and return the id of the running job</em></strong><br>APP_ID=$(echo $APPS | jq &#39;.apps.app[0].id&#39;)</pre><pre><strong><em># 3. Send a kill request to the currently running job</em></strong><br>curl -X PUT -H &quot;Content-Type: application/json&quot;<br>    -d &#39;{&quot;state&quot;: &quot;KILLED&quot;}&#39;<br>    &quot;http://${MASTER_ENDPOINT}/ws/v1/cluster/apps/${APP_ID}/state&quot;</pre><pre><strong><em># 4. Add a new step to submit the new job</em></strong><br>aws emr add-steps cluster-id j-UJODR7SZ6L7L steps<br>    Type=Spark,<br>    Name=&quot;Aggregator&quot;,<br>    ActionOnFailure=CONTINUE,<br>    Args=[<br>        --deploy-mode,cluster,<br>        --class,com.cm.spark.SparkApp,<br>        s3://bucket/jobs/${APP_JAR}<br>    ]</pre><p>This removes all the downfalls because it creates a separation of concerns by providing a specific CI pipeline for job deployment while the cluster can be resized and rebuilt when necessary. The benefits of keeping one dedicated cluster per Spark job can still be maintained with this solution, only that the cluster doesn’t have to be destroyed every time there is an iteration required on the job itself.</p><p><em>Originally published at </em><a href="http://adelhelal.blogspot.com/2018/09/automating-deployments-for-sparks.html"><em>adelhelal.blogspot.com</em></a><em> on September 26, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf940d6a588a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/automating-deployments-for-sparks-structured-streaming-on-aws-emr-cf940d6a588a">Automating Deployments for Spark’s Structured Streaming on AWS EMR</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transient timeouts and the retry rabbit hole (.Net 4.5)]]></title>
            <link>https://medium.com/cm-engineering/transient-timeouts-and-the-retry-rabbit-hole-net-4-5-f406cebbf194?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/f406cebbf194</guid>
            <dc:creator><![CDATA[Tulara Webster]]></dc:creator>
            <pubDate>Mon, 25 Jun 2018 23:11:01 GMT</pubDate>
            <atom:updated>2018-06-25T23:11:01.121Z</atom:updated>
            <content:encoded><![CDATA[<p>A month ago I would have told you retrying on timeouts in c# was as simple as chucking <a href="https://github.com/App-vNext/Polly">Polly</a> at the thing. oh, to be young and naïve and using ambiguous language!</p><p>Turns out, there are some easy mistakes to make, and here is a post summarising my learnings. All code samples shown are runnable in <a href="https://www.linqpad.net/">LinqPad</a>.</p><p><strong>tldr;</strong> If you want to retry an HTTP call, HttpClient’s <a href="https://docs.microsoft.com/en-us/aspnet/web-api/overview/advanced/http-message-handlers">Delegating Handlers</a> don’t let you distinguish between external task cancellations and actual request timeouts. And if you don’t use them, you have to be careful to make sure your request and its content hasn’t been disposed when you retry. Skip to the last attempt to see how we solved this.</p><h3><strong>Task</strong>:</h3><p>Campaign Monitor has a high-performance in-house file storage service we use for things like storing campaign content.<br>The mission in this instance was to add retries, and a circuit breaker, to the client library that makes (a lot of) http calls to this service.</p><h3>Attempt 1: Polly + DelegatingHandler = easy retries! right?!</h3><p>Ingredient 1: Polly.</p><p>I like Polly. It’s a resilience and transient-fault-handling library that allows for configuring such wonders as retries and circuit breakers in a fluent manner. It handles async and is thread safe. It’s so wonderful that <a href="https://blogs.msdn.microsoft.com/webdev/2018/02/28/asp-net-core-2-1-preview1-introducing-httpclient-factory/">it’s been integrated into .Net Core 2.1</a>. However, alas, some of our code is still operating in .Net 4.5 so here we’ve had to put the pieces together ourselves.<br> <br>You can use it for anything that might throw an exception at you (in the real world, you’ll most likely doing this when that failure is transient):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e5b3bc1ca97ef5342869df5962a1116/href">https://medium.com/media/7e5b3bc1ca97ef5342869df5962a1116/href</a></iframe><p>Ingredient 2: .Net’s Delegating Handler.</p><p>This guy has been around since .Net 4.5 and sits in between the HttpClient and its built-in HttpClientHandler a là:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*B8AZVlQmu6uC21kM-MveKA.png" /></figure><p>You can read more about it <a href="https://www.thomaslevesque.com/2016/12/08/fun-with-the-httpclient-pipeline/">here</a> and <a href="https://docs.microsoft.com/en-us/aspnet/web-api/overview/advanced/http-message-handlers">here</a> but the essence of it is: do something fun like log or retry on any request the client issues. In code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19a754dd309080c74b0f7220b30ceb48/href">https://medium.com/media/19a754dd309080c74b0f7220b30ceb48/href</a></iframe><p>Put the two ingredients together, et voila! Neat little request client with retries.</p><p>Only… There’s a problem.</p><p><strong>Exhibit A. Timeouts.<br></strong>HttpClient bubbles timeouts up as task cancellations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/099d968d9406518cfbc89fcbfb653ef7/href">https://medium.com/media/099d968d9406518cfbc89fcbfb653ef7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2-A43ey5K50RF_LtfvxqA.png" /><figcaption><strong>resulting in a TaskCanceledException</strong></figcaption></figure><p><strong>Exhibit B. External Task Cancellations</strong></p><p>If someone closes a connection (e.g. gives up on a browser tab), the client will cancel all the outstanding requests, for say things like images. These are passed down as task cancellations.</p><p>The following shows <strong>instances of “System.Threading.Tasks.TaskCanceledException: A task was canceled”</strong>.<br>A peak of 2000 requests that have been cancelled in the space of a minute? Yep, that’ll break your circuit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6BROUqgHbPxyRyD4WCzjUA.png" /></figure><p>Sure enough, <strong>instances of “Polly.CircuitBreaker.BrokenCircuitException: The circuit is now open and is not allowing calls”:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KqnJObT4tn-ZlCSKvSZVYQ.png" /></figure><p>A retry handler like that in Attempt 1 was responsible for sending these requests. It&#39;s now attempting to retry requests that no-one will ever see the result of.</p><p>What a sad and lonely thought. 😓</p><p><strong>Problem: <br> The cancellation token can’t tell us where the cancellation came from.</strong></p><p>If we are outside the httpClient we could just catch the exception and ask the cancellationToken if it was cancelled. True = someone outside my method cancelled the request. False = timeout (because the httpClient must have cancelled it!).<br>The thing is, you can’t distinguish exhibit A from exhibit B once you’re inside a DelegatingHandler. The first thing the HttpClient does when it gets your request is link your cancellation token together with its own. It is the combined token that is passed down through the handlers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-BBX2THDRmaL-B3t7qAug.png" /><figcaption><strong>HttpClient.SendAsync</strong></figcaption></figure><p>The cancellation token given to your Delegating handler will show IsCancellationRequested == true for both an external task cancellation request and an http client timeout:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cb9636f252f34985bc207102cc19f63/href">https://medium.com/media/7cb9636f252f34985bc207102cc19f63/href</a></iframe><p>Resulting in:<br>False <br> True</p><p><strong>That’s a bummer… what does it mean?</strong></p><p><strong>1st Lesson: </strong>If we want to retry a TaskCanceledException caused as a result of a transient timeout, but not as a result of a cancellation request by the caller, we can’t use a <strong>DelegatingHandler</strong>.</p><h3>Attempt 2: Differentiate timeouts and task cancellations</h3><p>Alright, fine. We’ll retry the request from outside the httpClient call. Who needs you anyway DelegatingHandler.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/14a2894bd6b34a6ad0fac97aba1d9429/href">https://medium.com/media/14a2894bd6b34a6ad0fac97aba1d9429/href</a></iframe><p>ha, and I still got to use a Polly retry policy 😉 Let’s just execute that…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IXQXOm7qlXhrFzVwk8Mudg.png" /></figure><p>oh… hmm. You win this round HttpClient. But, I can totally just recreate this request!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3af52fbfa327672629a42279ae5d4eb/href">https://medium.com/media/e3af52fbfa327672629a42279ae5d4eb/href</a></iframe><p>Unit test the new retry handler. Manually test the integration with a few Fiddler-motivated retries. Ready to go!</p><p><strong>2nd Lesson: </strong>To recreate HttpRequestMessage, you have to recreate the HttpContent object (and test retries on your goddamn writes!)</p><p>Otherwise you might end up bringing down a production API for half an hour… 😬</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SKzQjJnXNjLWEh5jfv7JkA.jpeg" /><figcaption><strong>Snippet of a log from the resulting mayhem</strong></figcaption></figure><p>From here, I solicited advice from prolific C sharpers (my colleagues) and we came up with the following options:</p><ul><li>clone http content using a memory stream</li><li>create an undisposable http request object</li><li>stop using httpClient’s Timeout property and time out the request using our own cancellation token</li></ul><p>So in the pursuit of simplicity I opted for option 3.</p><p>It’s worth noting that this has been rectified in dotNet Core; <a href="https://github.com/dotnet/corefx/pull/19082">httpClient will no longer dispose of the request</a>.</p><h3>Attempt 3: HttpClient.Timeout you’re no friend of mine.</h3><p>Moving back to the delegating handler model described in approach one. This time we have another handler which will manage a cancellation token coordinating the request timeout. And let’s actually throw a relevant exception while we’re at it.</p><p>Note the linkedToken; this is what httpClient was doing under the hood. Now we have out own, unpolluted token which we can check for cancellation. If either the inherited one or our timeout token are cancelled, that is passed through to httpClient via the linked token.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e602528b05eadda21cb50027a8c1a3c/href">https://medium.com/media/0e602528b05eadda21cb50027a8c1a3c/href</a></iframe><p>Throw that together with the appropriate RetryHandler (catching timeouts only):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/220020049373f90f7da6e0fa3a149e25/href">https://medium.com/media/220020049373f90f7da6e0fa3a149e25/href</a></iframe><p>So now, if we cancel the request via the token from outside the client request we won’t see a retry.<br><strong>One more gotcha</strong> — the default Timeout on HttpClient is 100 seconds. If you make your timeout configurable (like we have outside these examples) some crazy person might set this to a timespan greater than the default and we would lose our retries!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df3b9a120cd476b0668005b323f6a476/href">https://medium.com/media/df3b9a120cd476b0668005b323f6a476/href</a></iframe><h3>Success!</h3><p>Retries are happening (this is the last day of data):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*wHsA7oM3uG0PaKjhjco61g.png" /></figure><p>at a healthy rate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*uJcKHGCxbtuZyjFOx3GxXg.png" /></figure><p>And the ones we’ve logged are retrying for the right reasons:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BMj9VjBiyK250vumYTrQA.png" /></figure><p>🙌🙌🙌🙌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f406cebbf194" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/transient-timeouts-and-the-retry-rabbit-hole-net-4-5-f406cebbf194">Transient timeouts and the retry rabbit hole (.Net 4.5)</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to test accessibility of emails]]></title>
            <link>https://medium.com/cm-engineering/how-to-test-accessibility-of-emails-b68fed03f5f4?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/b68fed03f5f4</guid>
            <category><![CDATA[email-marketing]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[accessibility]]></category>
            <dc:creator><![CDATA[Samantha Connelly]]></dc:creator>
            <pubDate>Tue, 05 Jun 2018 00:22:14 GMT</pubDate>
            <atom:updated>2018-06-05T00:22:14.093Z</atom:updated>
            <content:encoded><![CDATA[<p>Testing for accessibility in your emails doesn’t have to be a chore. Making accessible and inclusive emails will increase your target audience (<a href="https://www.and.org.au/pages/disability-statistics.html">1</a>, <a href="https://nfb.org/blindness-statistics">2</a>, <a href="http://www.colourblindawareness.org/colour-blindness/">3</a>) and will have other unintended benefits. These other benefits are known as the <a href="https://ssir.org/articles/entry/the_curb_cut_effect">Curb-Cut effect</a>. When people first introduced curb cuts (i.e. sloped ramps) at intersections for wheel chair riders they discovered that many other people benefited too; i.e. people pushing strollers/shopping trolleys, people with deliveries and people with walkers all used these new curb cuts to help navigate their urban environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MyrTq2gkQPO-0fpl3JJbjQ.jpeg" /><figcaption>A curb cut from <a href="https://commons.wikimedia.org/wiki/File:Pram_Ramp.jpg">WikiCommons</a>, <a href="https://99percentinvisible.org/episode/curb-cuts/">here</a> is a podcast on the history of these ramps.</figcaption></figure><p>Over 4 million people in Australia live with some form of disability. That’s 1 in 5 people (<a href="https://www.and.org.au/pages/disability-statistics.html">1</a>). Over my lifetime I’ve experienced a number of health issues that have impacted my abilities temporarily. I might not be living with an officially diagnosed disability today but I frequently use features such as the closed captions on YouTube and Audio Books when I feel like changing up how I consume information.</p><p>It’s easy to get overwhelmed when thinking about accessibility testing, there’s many different levels of accommodations to consider (<a href="https://services.anu.edu.au/human-resources/respect-inclusion/different-types-of-disabilities">7</a>). Here is just a sample to consider when sending emails;</p><ul><li>Vision Impairment (Can people use screen readers on your email?)</li><li>Hearing Impairment (does any linked video content have closed captions?)</li><li>Mental health/Intellectual/Autism spectrum (Does your content overload people’s processing ability? Is it easy to read and follow?)</li></ul><p>It’s no wonder people often put this type of testing in the too-hard basket. However if you start with testing with vision impairments in mind you should be able to get a good return on investment towards a more inclusive experience.</p><h3>Screen Reader technology</h3><p>Most vision impaired users will use some sort of screen reader technology to browse the web. Even I use this technology when I feel like listening to a news article online instead of reading. 2.3% of the US population have some sort of vision impairment (<a href="https://nfb.org/blindness-statistics">2</a>). This statistic is likely to grow with an aging population. The growth of voice interfaces will also grow the use case for this technology. I prefer using <a href="https://www.apple.com/au/accessibility/iphone/vision/">iOS’s VoiceOver</a> technology but there are many other tools out there; Windows comes with <a href="https://support.microsoft.com/en-au/help/22798/windows-10-narrator-get-started">Narrator</a> and Android has <a href="https://support.google.com/accessibility/android/answer/6283677?hl=en">TalkBack</a>, there’s also the paid software <a href="http://www.freedomscientific.com/products/blindness/jawsdocumentation">JAWS</a>.</p><h3>Using VoiceOver with Email</h3><p>On my iPhone 6 running iOS 11.3.1, I can go into Setting &gt; General &gt; Accessibility &gt; VoiceOver</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*IYaw9ka0xly4wuL1WuvMHw.png" /><figcaption>iOS’s accessibility options view</figcaption></figure><p>I can switch it on here but this can be hard to switch off with VoiceOver if you aren’t used to interacting with mobile technology this way. To help with this, I enable accessibility shortcuts on my mobile device. When I press my home button 3 times, I can easily switch accessibility features on and off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*6U8Hf2jmR0uQj5ZwFHJa9Q.png" /><figcaption>My iOS accessibility shortcuts, I have VoiceOver and colour filters enabled</figcaption></figure><p>When I test emails using VoiceOver;</p><ul><li>First I’d navigate to the email I’d like to test</li><li>Then enable VoiceOver using my home button shortcut</li><li>Finally flick/scroll two fingers up the screen</li></ul><p>This enables the screen reader to read everything on the screen from top to bottom and I can test if the flow makes sense. There are plenty of other guides out there for using VoiceOver (<a href="https://www.imore.com/how-use-voiceover-iphone-and-ipad">12</a>). I suggest switching on a screen reader on your mobile device and getting familiar with the technology. As a test, can you figure out how to take a photo on your phone using screen reader technology?</p><h3>Alternative text for images</h3><p>Screen readers will read out the alternative text for images for vision impaired users who can’t see your images. Alt text is an optional field for HTML image tags. For example, using <a href="https://www.campaignmonitor.com/resources/guides/alt-text-in-email/">Campaign Monitor’s Email Builder</a> I have included an alt text tag with my image to convey the message of the image for screen readers. Here is the HTML tag for the image;</p><pre>&lt;img src=&quot;darth_vader.jpg&quot; alt=&quot;Meme; Come to the dark side ... we have cookies&quot;&gt;</pre><p>This is what this email looks like with images;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lGKw5aC_5No_mrP9jomfJA.png" /><figcaption>Email view with images</figcaption></figure><p>I used Firefox’s <a href="https://addons.mozilla.org/en-US/firefox/addon/web-developer/">web developer</a> tools to replace images with their alt text to see this next view of the same email;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhYi7y9A9DttGzQl8isfmA.png" /><figcaption>Email view with alt text replacing the images</figcaption></figure><p>It’s considered a web best practice to include alt text (<a href="https://support.siteimprove.com/hc/en-gb/articles/115000013031-Accessibility-Image-Alt-text-best-practices">15</a>). You can also read more about the <a href="https://www.w3.org/WAI/">web accessibility initiative at w3</a>.</p><h3>Tips for Alt text</h3><ul><li>Check you have alternative text for your images and that it makes sense</li><li>Check that you have blank strings or no alt text options for decorative images so that screen reader don’t try to read the image or the file name</li><li>Don’t include words like “image of” in your alt text because screen readers will announce it’s an image and then announce the alt text. You will get a double whammy “image. image of blah” from your screen reader, this is super annoying</li><li>Use tools that display the alt text or replace images with their alt text to help test this</li></ul><h3>Test the plain text version of your emails</h3><p>Plain Text emails are important, and even though people say they prefer image based emails the statistics show they are more engaged with plain text emails (<a href="https://litmus.com/blog/best-practices-for-plain-text-emails-a-look-at-why-theyre-important">17</a>, <a href="https://blog.hubspot.com/marketing/plain-text-vs-html-emails-data">18</a>, <a href="https://www.smartinsights.com/email-marketing/email-deliverability/plain-text-vs-html-emails-2/">19</a>);</p><ul><li>Tech savvy users can default their email clients to use text only emails</li><li>Plain text is more accessible (there’s no pesky images or HTML layout to contend with)</li><li>It’s more private because image based tracking doesn’t work</li><li>They are a good fall back option when people are concerned about bandwidth consumption, for example when traveling overseas</li><li>Spam email will fall back to the text only version because email clients may think the images and attachments are dangerous</li></ul><p>Most Email Service Providers (ESPs) should have how to guides for testing plain text emails. For example our guide is <a href="https://help.campaignmonitor.com/plain-text-versions-of-html-emails">here</a>. When I test for plain text emails, I use my own web hosting service that has a web-mail client that defaults to plain text. When I access these test emails they often look like this;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwezdZr9BfAoZ6ymcX74uA.png" /><figcaption>a plain text email view using hostpapa and horde webmail client</figcaption></figure><p>A screen reader is going to have an easier time with a plain text email compared to a mostly image based email.</p><h3>Test your images for colour blindness</h3><p>About 4.5% of the British population experience some form of colour blindness (<a href="http://www.colourblindawareness.org/colour-blindness/">3</a>). To help test this I have a grey scale shortcut button set up on my iPhone. On a side note, setting up grey scale for your phone helps make it less addictive too (<a href="https://www.youtube.com/watch?v=NUMa0QkPzns">21</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*YzMAMIyK6589WavBRiKksA.jpeg" /><figcaption>This is what my phone looks like with grey scale enabled, you can also see what the accessibility shortcuts look like</figcaption></figure><h3>Putting this into practice</h3><p>By doing VoiceOver and grey scale testing on an email I sent out for a recent bugbash, I discovered a few issues with my email. First the yellow highlight that I used for the “consent2track v3” text didn’t have enough contrast under a greyscale view to make it stand out. Secondly I didn’t update the alternative text for the image, I had an old bug bashes alt text attached to the image 😢. If I had done this testing before sending out this email I would have caught these issues. Here are the two views of the email side by side;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2MWNhapspbrZcnhVldIwgg.png" /><figcaption>Side by Side comparison of full colour and grey scale view of an email</figcaption></figure><p>Hopefully you now realize that testing for accessibility in your emails isn’t that hard. If a screen reader works with your email, you have a plain text fall back option and your email is still readable using grey scale you will be most of the way there in providing an awesome, inclusive experience with your emails.</p><h3><strong>References</strong></h3><ol><li><a href="https://www.and.org.au/pages/disability-statistics.html">https://www.and.org.au/pages/disability-statistics.html</a></li><li><a href="https://nfb.org/blindness-statistics">https://nfb.org/blindness-statistics</a></li><li><a href="http://www.colourblindawareness.org/colour-blindness/">http://www.colourblindawareness.org/colour-blindness/</a></li><li><a href="https://ssir.org/articles/entry/the_curb_cut_effect">https://ssir.org/articles/entry/the_curb_cut_effect</a></li><li><a href="https://commons.wikimedia.org/wiki/File:Pram_Ramp.jpg">https://commons.wikimedia.org/wiki/File:Pram_Ramp.jpg</a></li><li><a href="https://99percentinvisible.org/episode/curb-cuts/">https://99percentinvisible.org/episode/curb-cuts/</a></li><li><a href="https://services.anu.edu.au/human-resources/respect-inclusion/different-types-of-disabilities">https://services.anu.edu.au/human-resources/respect-inclusion/different-types-of-disabilities</a></li><li><a href="https://www.apple.com/au/accessibility/iphone/vision/">https://www.apple.com/au/accessibility/iphone/vision/</a></li><li><a href="https://support.microsoft.com/en-au/help/22798/windows-10-narrator-get-started">https://support.microsoft.com/en-au/help/22798/windows-10-narrator-get-started</a></li><li><a href="https://support.google.com/accessibility/android/answer/6283677?hl=en">https://support.google.com/accessibility/android/answer/6283677?hl=en</a></li><li><a href="https://www.freedomscientific.com/products/blindness/jawsdocumentation">https://www.freedomscientific.com/products/blindness/jawsdocumentation</a></li><li><a href="https://www.imore.com/how-use-voiceover-iphone-and-ipad">https://www.imore.com/how-use-voiceover-iphone-and-ipad</a></li><li><a href="https://www.campaignmonitor.com/resources/guides/alt-text-in-email/">https://www.campaignmonitor.com/resources/guides/alt-text-in-email/</a></li><li><a href="https://addons.mozilla.org/en-US/firefox/addon/web-developer/">https://addons.mozilla.org/en-US/firefox/addon/web-developer/</a></li><li><a href="https://support.siteimprove.com/hc/en-gb/articles/115000013031-Accessibility-Image-Alt-text-best-practices">https://support.siteimprove.com/hc/en-gb/articles/115000013031-Accessibility-Image-Alt-text-best-practices</a></li><li><a href="https://www.w3.org/WAI/">https://www.w3.org/WAI/</a></li><li><a href="https://litmus.com/blog/best-practices-for-plain-text-emails-a-look-at-why-theyre-important">https://litmus.com/blog/best-practices-for-plain-text-emails-a-look-at-why-theyre-important</a></li><li><a href="https://blog.hubspot.com/marketing/plain-text-vs-html-emails-data">https://blog.hubspot.com/marketing/plain-text-vs-html-emails-data</a></li><li><a href="https://www.smartinsights.com/email-marketing/email-deliverability/plain-text-vs-html-emails-2/">https://www.smartinsights.com/email-marketing/email-deliverability/plain-text-vs-html-emails-2/</a></li><li><a href="https://help.campaignmonitor.com/plain-text-versions-of-html-emails">https://help.campaignmonitor.com/plain-text-versions-of-html-emails</a></li><li><a href="https://www.youtube.com/watch?v=NUMa0QkPzns">https://www.youtube.com/watch?v=NUMa0QkPzns</a></li></ol><h3>About the Author</h3><p><em>Sam is a quality coach at Campaign Monitor, focusing on teaching software engineers how to be better bug hunters. She can be described as a motorbike riding &amp; beer brewing girl geek.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b68fed03f5f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/how-to-test-accessibility-of-emails-b68fed03f5f4">How to test accessibility of emails</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My time at CSSConfAU 2018]]></title>
            <link>https://medium.com/cm-engineering/my-time-at-cssconfau-2018-ba6458e80827?source=rss----f08afbb26c30---4</link>
            <guid isPermaLink="false">https://medium.com/p/ba6458e80827</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[css]]></category>
            <dc:creator><![CDATA[Mitchell Busby]]></dc:creator>
            <pubDate>Tue, 08 May 2018 01:51:20 GMT</pubDate>
            <atom:updated>2018-05-08T02:05:55.488Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-0wFoND11k_hzabgqHfIg.png" /></figure><p>This year’s CSSConfAU was my first developer conference. I woke up on Tuesday morning all fresh-eyed and bushy tailed, ready to learn from pros and meet new people.</p><p>What a <em>phenomenal</em> experience it was! I met a bunch of friends (both old &amp; new), enjoyed the unlimited coffee ☕ 👌, and learnt a tonne of new stuff. Oh, and I got the full conference experience by getting con flu. 🤒 😄</p><p>It dawned on me as I left CSSConf that the landscape around CSS is changing even more rapidly than ever before — and that’s without even looking at CSS-in-JS! With CSS grid, Houdini’s browser-breaking abilities, and the maturation of organisational design systems, there’s heaps to learn and develop skills in.</p><h4>Crazy-fast CSS animations (David Khourshid) — <a href="https://www.youtube.com/watch?v=zVbLR7rI8ZU">video</a></h4><ul><li>CSS animations have an inherent cost, even though they’re more performant than the jQuery-powered animations of old</li><li>Useful animations that don’t drop the frame rate of your web app are fundamental to matching the experience of a native application</li><li>Many size-changing CSS animations suffer from performance issues — his <a href="https://slides.com/davidkhourshid/illusions">slides</a> outline some techniques for preventing frame-dropping when performing these types of animations</li><li>CSS animations are something I need to get better at — I’m keen to play around with some of the stuff explored in his keynote</li></ul><h4>CSS Grid today (Brenda Storer) — <a href="https://www.youtube.com/watch?v=fVX0xscGP9o">video</a></h4><ul><li>CSS Grid is truly <strong>production ready</strong> today (with graceful fallbacks for IE, of course)</li><li>Allows you to eschew the constraints of display:table; and the float:left; buggery</li><li>Fun tip: You can specify widths by character — perfect for paragraph text. I was told <strong>73 character widths</strong> is the ideal width in most cases. You should give it a go in your next project</li></ul><p><em>If you want to take a closer look, my fellow Campaign Monitor employee, </em><a href="https://twitter.com/cwrightdesign"><em>Chris Wright</em></a><em>, did an amazing introduction to CSS Grid </em><a href="https://cm.engineering/getting-to-know-css-grid-layout-818e43ca71a5"><em>here</em></a><em>. </em><a href="https://twitter.com/iamryanyu"><em>Ryan Yu</em></a><em>, another CM’er, has a stellar and constantly growing collection of CSS Grid resources </em><a href="https://medium.com/@iamryanyu/an-ultimate-collection-of-css-grid-a105e88e7153"><em>here</em></a><em>.</em></p><h4>Colour systems and you (Diana Mounter) — <a href="https://www.youtube.com/watch?v=AbpS3Z3xXE0">video</a></h4><p>Another highlight was Diana Mounter’s talk on structuring a colour system. The key takeaways for me included the value of <strong>incremental correctness</strong> (bridges and shims from your old system and new are okay!) and signs that your colour system needs to be reworked:</p><ul><li>Variable names that represent component colours, not systematic colours</li><li>Colours that don’t “scale well” — reusability is minimal, and colour “steps” are not included in the system</li><li>Poor colour contrast in the palette, leading to reduced accessibility when colours are combined</li></ul><p>Diana’s talk focussed on how she rolled out a new colour system at GitHub, and it was rad!</p><h4>Magic tricks with Houdini (Sam Richard) — <a href="https://www.youtube.com/watch?v=FYWZrLpN5Po">video</a></h4><p>The most intriguing thing to come out of the web standards organisation W3C in recent years is the Houdini task force. It is basically aimed at allowing <strong>performant CSS polyfilling</strong>. So it was with great excitement that I watched Sam Richard’s talk on what Houdini has brought to the table so far.</p><p>The speaker heavily emphasised that Houdini specs are still very much in development, with APIs breaking constantly. But the core ideas will likely persist to the final specs implemented by browser vendors:</p><ul><li><strong>Worklets</strong>, little threads that allow running arbitrary JS code that is plugged into the CSS rendering engine</li><li><strong>TypedDOM</strong> and typed CSS values, to allow for type safe manipulation and operations on the CSSDOM</li><li><strong>Custom</strong> CSS variables for dynamic values to be utilised in CSS, and custom type definitions (<strong>you can define your own variable type syntaxes</strong>!)</li></ul><p>It was really cool to see examples of things like worklets with actual working code. The code shown got a bit gnarly at times — after all, you’re literally writing CSS rendering rules from scratch. Sam demonstrated a bunch of neat conceptual examples that you can implement with the current spec, like:</p><ul><li>Canvas drawing abstractions as CSS properties</li><li>Twitter’s animated sticky header replicated as a CSS property</li><li>Performant masonry built into the browser rendering engine(!)</li></ul><h4>Can you see that OK? CSS tips for low-vision accessibility (Julie Grundy) — <a href="https://www.youtube.com/watch?v=Wq2-cZ-vZBc">video</a></h4><p>This talk by Julie Grundy had some actionable insights we can use every day when building user interfaces. And it’s important too; as Julie noted:</p><blockquote>“…at 9% of the Australian population, there are more people with vision impairments visiting your site than Internet Explorer users”.</blockquote><p>Here’s some tips I got out from the talk:</p><ul><li>Horizontal scrolling is quite painful for users with arthritis, so avoid where possible;</li><li>Responsive design <em>can</em> help with accessibility. Thanks to zoom modes generally emulating lower-density screens you’ll fall into a mobile-like experience. <em>However</em>, text-only zoom (which some users require) requires variable height containers so that the containers don’t cut off larger text</li><li>Check for inaccessible colour schemes really quickly without any tools by using body { filter: grayscale(100%); }</li><li>Don’t rely solely on colour to distinguish between menu items that are active and not active. Use shape changing as well so you don’t disadvantage low-contrast users</li><li>Links only indicated via colour don’t work; you’re going to need to use underlines. For nicer flow, there’s also a CSS <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/text-decoration-skip-ink">draft spec</a> that doesn’t put the underline over descender letters (p/q/j/g)</li><li>When working with form elements, add a 1px solid transparent border around the inputs and modals. High-contrast rendering modes in the browser respect this and make the boundaries of inputs obvious where your drop shadows aren’t very visible. These sorts of standards are great for a design system to roll out to consumers</li><li>Julie suggested the book “<a href="https://abookapart.com/products/color-accessibility-workflows">Colour Accessibility Workflow</a>s” for further reading on accessibility</li><li>I also intend to watch the <a href="https://www.youtube.com/watch?v=pvZiB7NkT8M&amp;t=21749s">JSConf ’18 video</a> on forging a path forward for accessibility in legacy applications and applications that may be quite behind in accessibility support — check it out</li></ul><h4>Speedy font loading (Jeremy Wagner) — <a href="https://www.youtube.com/watch?v=iEinIwNppW4">video</a></h4><p>We’ve all visited a website where the text content didn’t load until <em>seconds</em> after the page layout finished loading. There’s a bunch of things in this talk that will allow you to optimise for speedy web font loading, and optimise the user experience for when font loading doesn’t go so well.</p><p>I strongly recommend watching the video in its entirety, but here’s a few neat things I got out of it:</p><ul><li>Host your web fonts on your own domain — with HTTP/2 you’ll reduce the load time, as the same handshake can be used for multiple resources loading simultaneously</li><li>If you can’t, tack on rel=”preconnect” (<a href="https://caniuse.com/#search=preload">caniuse</a>) or rel=”dns-prefetch” (<a href="https://caniuse.com/#search=dns-prefetch">caniuse</a>) to decrease the blocking cost of that handshake</li><li>If you host your own fonts, you can do things like subsetting to optimise your font bundle size for your use cases</li><li>font-display: swap; is great! If you have a font-stack defined, it will display the first locally-installed font, then go up the stack once your preferred font is loaded</li><li>Icon fonts aren’t the only way to effectively deliver icons to your users, <strong>and maybe you shouldn’t<em> </em>use them</strong> — check out <a href="https://css-tricks.com/icon-fonts-vs-svg/">this post</a> for more</li></ul><p>A lot of the talk corroborated with stuff I had found from this <a href="https://meowni.ca/posts/web-fonts/">article</a>. Give it a read if you want some actionable stuff you can do on your website <em>now</em>.</p><p>Overall my experience was overwhelmingly positive — the atmosphere was warm and welcoming, and the nice touches that come with a community conference <strong>designed by and for the people</strong> were really appreciated. I look forward to my next web dev conference!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba6458e80827" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cm-engineering/my-time-at-cssconfau-2018-ba6458e80827">My time at CSSConfAU 2018</a> was originally published in <a href="https://medium.com/cm-engineering">Campaign Monitor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>