<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Denise Schlesinger on Medium]]></title>
        <description><![CDATA[Stories by Denise Schlesinger on Medium]]></description>
        <link>https://medium.com/@deniseschlesinger?source=rss-5be4408574c4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yBfWL4HPtuvNeGHH-UZctw.jpeg</url>
            <title>Stories by Denise Schlesinger on Medium</title>
            <link>https://medium.com/@deniseschlesinger?source=rss-5be4408574c4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:31:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deniseschlesinger/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streaming data into Real-time Dashboards with Azure]]></title>
            <link>https://medium.com/@deniseschlesinger/streaming-data-into-real-time-dashboards-with-azure-81de3ea2ebf7?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/81de3ea2ebf7</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Wed, 01 Feb 2023 19:45:48 GMT</pubDate>
            <atom:updated>2023-03-01T11:56:39.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Photo by Jachan de Vol from Unsplash" src="https://cdn-images-1.medium.com/max/1000/0*iHBEAdVfSpL9x6Vy.jpg" /></figure><p>Sometimes organizations are required to respond to events in real-time, allowing them to detect and respond to changes such as security threats, customer-impacting operational issues, business KPIs, system monitoring, etc. Using real-time dashboards can help them make data-driven decisions fast.</p><p><strong>What is a real-time dashboard?</strong><br>A real-time dashboard is a type of visualization that is automatically updated with the most current data available.</p><p><strong>Which data sources can be used for real-time dashboards?</strong><br>The sources of streaming data can be IOT sensors, social media posts, monitoring metrics, or any other time-sensitive data being collected.</p><p><strong>Which technology can process streaming events?</strong><br>There are many technologies that allow producing and consuming streams of events, such as Apache Kafka, Spark and others. <br>We will be using Azure technologies that will help us hit the road running, in no-time.</p><p>In this post I will show you how to create synthetic data, stream it using Azure Events hub, process it with Azure Stream Analytics and showcase it in Power BI in a live dashboard using “streaming datasets”.</p><h3><strong>Let’s start</strong></h3><h4>What are we building?</h4><p>We will:<br>- Run a python script creating synthetic events and sending them to Azure Events hub<br>- Use Azure Stream Analytics to consume the events from Events hub and sending them to Power BI streaming dataset<br>- Visualize in Power BI the events in real time with a dashboard.</p><p><strong>Here is the architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*Ht4f8a0Vgobhn98Zqp82Rw.png" /></figure><h3>The recipe</h3><p>I assume you already have an azure account <a href="https://azure.microsoft.com/en-us/free/search/?&amp;ef_id=CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE:G:s&amp;OCID=AIDcmm81syc84i_SEM_CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE:G:s&amp;gclid=CjwKCAiAuOieBhAIEiwAgjCvcjllhUzymVpLAwj0UYrCWrW7qTfm1MbvvwujlSC0FqAHaY5qkQxTWRoCDU0QAvD_BwE">(click here if you don’t)</a></p><p><strong>Create an events hub</strong><br>In the azure portal create an Events Hub</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*MVE3binZta60bhDUR5z-WQ.png" /></figure><p><strong>Click on Create</strong><br><strong>Fill all the relevant fields<br>Click “review and create”<br>Click “create”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*peBtlJ6bJyAmVXh9WSembQ.png" /></figure><p><strong>Go to your newly created events hub</strong><br><strong>Create a Shared Access Policy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*quoK9OIt-svn-OspTs7luA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*x9pvEiNyjGiXDBBJJ-CSYw.png" /></figure><p><strong>Copy the Shared Access Policy primary key and the events hub name and paste it into the python script you downloaded from (also see below) </strong><a href="https://github.com/denisa-ms/public-snippets.git"><strong>[github]</strong></a></p><pre>import datetime<br>import time<br>import random<br>import socket<br>import struct<br>import json<br>from azure.eventhub import EventHubProducerClient, EventData<br>import psutil<br>import pyodbc<br>import pandas as pd<br>eventHubConnString = &quot;&lt;paste here the events hub shared access signature primary key&gt;&quot;<br>eventHubName = &quot;&lt;paste here the events hub name&gt;&quot;<br>producer = EventHubProducerClient.from_connection_string(conn_str=eventHubConnString, eventhub_name=eventHubName)<br>hostname = socket.gethostname()<br>def generateRandomIP():<br> ip = socket.inet_ntoa(struct.pack(&#39;&gt;I&#39;, random.randint(1, 0xffffffff)))<br> return ip<br>def getRandomBrowser():<br> browsersList = [&quot;Edge&quot;, &quot;Chrome&quot;, &quot;Safari&quot;, &quot;Firefox&quot;]<br> browser = random.choice(browsersList)<br> return browser<br>def getRandomMethod():<br> methods = [ &quot;GET&quot;, &quot;POST&quot;, &quot;PUT&quot;, &quot;DELETE&quot; ]<br> method = random.choice(methods)<br> return method<br>def getRandomURL():<br> urls = [&quot;/page1.html&quot;, &quot;/page2.html&quot;, &quot;/page3.html&quot;, &quot;/page4.html&quot;, &quot;/page5.html&quot; ,&quot;/page6.html&quot; ]<br> url = random.choice(urls)<br> return url<br>def getCurrentDate():<br> today = datetime.datetime.now()<br> current = today.strftime(&quot;%Y-%m-%d&quot;)<br> return current<br>def getCurrentTime():<br> today = datetime.datetime.now()<br> current = today.strftime(&quot;%H:%M:%S&quot;)<br> return current<br>def getRandomTemperature():<br> return random.randint(0,99)<br>def generateClickEvent():<br> click_event = {<br> &quot;tableName&quot; : &quot;clicks&quot;,<br> &quot;date&quot; : getCurrentDate(),<br> &quot;time&quot;: getCurrentTime(),<br> &quot;ip&quot;: generateRandomIP(),<br> &quot;browser&quot;: getRandomBrowser(),<br> &quot;method&quot;: getRandomMethod(),<br> &quot;url&quot; : getRandomURL(),<br> &quot;productID&quot;: getRandomProductId()<br> }<br> return click_event<br>def getRandomProductId():<br> return str(random.randint(0,99999))<br>def sendToEventsHub(jsonEvent):<br> eventString = json.dumps(jsonEvent)<br> print(eventString)<br>event_data_batch = producer.create_batch() <br> event_data_batch.add(EventData(eventString)) <br> producer.send_batch(event_data_batch)<br>def generateClicks():<br> try:<br> while True:<br> clickEvent = generateClickEvent() <br> sendToEventsHub(clickEvent)<br> # time.sleep(10)<br>except KeyboardInterrupt:<br> producer.close()<br> conn.close()<br>if __name__ == &quot;__main__&quot;:<br> generateClicks()<br></pre><p><strong>Create a stream analytics job</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*yMZdDBIsIvBYE1txvgqOAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*Fzb01mUIG1kE_7LZAWz2Eg.png" /></figure><p><strong>Fill the details and click review and create and then Click create<br>Click on Inputs — Add stream input</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*UPNO2x4E8V8mZzsLp1VPSA.png" /></figure><p><strong>Choose your subscription, the events hub we created above, and settings as shown below</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*V2Y4iVhp9mXQtXMw0p8oGw.png" /></figure><p><strong>Click Outputs and add a Power BI output</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*w5ZBr4P6HKrfvFUIQz9PAQ.png" /></figure><p><strong>Fill all the details as shown below</strong><br>Please notice that this will authorize the stream analytics job to access your power BI account (I recommend for demo purposes to use the same account as the one logged in into the Azure Portal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*eYv8xnuZEFgRwr-rZqnX-A.png" /></figure><p><strong>Click on Query</strong><br>Copy and paste the query snippet below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*9-BRhuw0xohU6_ytxldLTw.png" /></figure><pre>SELECT <br> DateAdd(s,-3,System.TimeStamp) AS windowstart,<br> System.TimeStamp AS windowend,<br> browser,<br>COUNT(*) AS entries<br>INTO<br>[powerbi]<br>FROM<br>[bddemo-clicks] TIMESTAMP BY EventProcessedUtcTime<br>GROUP BY browser, TumblingWindow(s, 3)</pre><p><strong>Save the query</strong><br><strong>Create a real time dashboard in Power BI<br>Login to power BI with the same credentials you provided the stream analytics job above<br>Create a dashboard</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*Dr434gQEcNNLGaEp7q5GSw.png" /></figure><p><strong>Add a tile to your dashboard</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*qNl-rhkAsBGSQPK7tiTYMA.png" /></figure><p><strong>Click on Custom Streaming Data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*hxsMaEN9tI_Ouv85XrXKDQ.png" /></figure><p><strong>Choose the stream we created in the stream analytics job</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*UwpYzit1Xn62ALoCj2im6g.png" /></figure><p><strong>Choose the visualization type</strong><br>We will choose the Line chart and fill the details</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*vo4muMqC4QiaNCBiY9k_EQ.png" /></figure><p><strong>Choose a tile name and click Apply</strong></p><h3><strong>We are done preparing everything!</strong></h3><h3><strong>Let’s run our demo</strong></h3><p><strong>Run the python script provided</strong><br>Make sure you are authenticated to run your local code and connect to Azure<a href="https://learn.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-9.3.0"> (if not click here and install powershell)</a></p><p><strong>Perform Azure authentication</strong></p><pre>$ az login</pre><p><strong>Run the python script</strong><br>python streamingClickEvents.py</p><pre>python streamingClickEvents.py</pre><p>Now our code is running and sending events to the events hub</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*bZcMmsAqyJuimG-_A32mug.png" /></figure><p><strong>Start the stream analytics job</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*Mc3FOpM4xk93Oy-HeFn2dQ.png" /></figure><p><strong>After 1–2 minutes the job status will be “running” and the events will start flowing</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*h6_ll483XmzH0TNe6IStDg.png" /></figure><p><strong>In Power BI you will see the events flowing in your dashboard</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*lZvCeugIAwRuUZbDN9eGMQ.png" /></figure><h3><strong>MAKE SURE to shut down your script and stop the stream analytics job to avoid unnecessary costs.</strong></h3><p>That’s all folks!</p><p>I hope you enjoyed this tutorial.<br>Cheers!<br>Denise</p><p>Photo provided by:</p><p><a href="https://unsplash.com/@jachan_devol?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jachan DeVol (@jachan_devol) | Unsplash Photo Community</a></p><p><em>Originally published at </em><a href="https://dev.to/denisams/streaming-azure-data-pipelines-into-real-time-dashboards-532b"><em>https://dev.to</em></a><em> on February 1, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81de3ea2ebf7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building resilient Distributed Systems at scale — Part 1]]></title>
            <link>https://medium.com/@deniseschlesinger/building-resilient-distributed-systems-at-scale-part-1-710c4c70d74c?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/710c4c70d74c</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Sat, 10 Aug 2019 09:04:37 GMT</pubDate>
            <atom:updated>2019-08-10T11:05:53.515Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building resilient Distributed Systems at scale — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tT71Rnit0w4zJFLt.jpg" /><figcaption>Bend but don’t break — like Bamboo</figcaption></figure><p>In this brave new world of distributed systems, we are entrusted with keeping the infrastructure up and running. <br>The source of the challenge is to monitor the services themselves and the space in between.</p><p>We face non-determinism, sometimes we can’t tell if our system is up, down, or partially working, and every failure is a task for Sherlock Holmes.</p><p>At <a href="https://www.novarize.com/">Novarize</a> we collect more than 2 billion data points a day. We have to develop services in a continuous iterative process, where we design for observability and resilience. Predictability and transparency are our primary goals.</p><p>We use continuous delivery so we need to find which tools and processes will help us tackle the complexities of deploying microservices at scale and make them resilient.</p><h3><strong>How can we build resilient systems?</strong></h3><p>Resilience is all about preventing faults turning into failures.</p><ul><li><em>Fault is an incorrect internal state in your system.</em></li><li><em>Failure is an inability of the system to perform its intended job.</em></li></ul><p>“Being available” is crucial, so we need to build resilience from ground up. Building resilience in distributed systems communicating with each other is difficult.</p><blockquote><em>The best way to write reliable and secure applications is write no code at all — write nothing and deploy nowhere</em></blockquote><blockquote><em>Kelsey Hightower</em></blockquote><p>We need to know what happens everywhere all the time, this is defined as <strong>observability</strong>.</p><blockquote><strong><em>Observability</em></strong><em> is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.</em></blockquote><p>We need to be able at all times to infer the state of every piece of our architecture so when something goes wrong, we can diagnose and correct the failure as fast as possible.</p><p>There are 3 pillars of observability:</p><ul><li>Logs</li><li>Metrics</li><li>Tracing</li></ul><p><strong>Logs:</strong> tell the story of the flow in our microservices and if implemented correctly we can look at them as in one of the Matrix movies and find the pattern in a mass of letters and numbers to understand what is really going on.</p><figure><img alt="Finding the pattern in logs" src="https://cdn-images-1.medium.com/max/1000/0*puBJvg9FBdr1V8y7.jpg" /><figcaption>Understanding the system by looking at the logs</figcaption></figure><p><strong>Metrics:</strong> are numeric values, measured over a period of time. Metrics are comprised of attributes like name, timestamp, value, and more. For example, system metrics can be: average CPU, disk space or kubernetes metrics such as memory usage, number of pods running/terminated, etc. Application metrics can be the number of collected/processed items over a period of time.</p><p><strong>Tracing:</strong> is a representation of a series of causally related distributed events that encode the end-to-end request flow through a distributed system. Traces show the path of a request flow across the system.</p><h3>The tools we use</h3><p><strong>Logs</strong></p><p>We need logs to debug on production, to understand what happened and why. We use centralized logging since our services are autoscaling all the time making log files volatile, so we send them to a single persisted location.</p><p>We use the ELK stack (elasticsearch, logstash, kibana) for logs.</p><p>We found that with the increase in data being processed we surpassed the write limit for ElasticSearch and got failures all the time. Increasing the size of our ElasticSearch cluster was one of the options but we decided it was not a long term cost-effective solution, so we introduced Kafka (we love Kafka).</p><p>We use Kafka to manage and buffer massive volumes of data ingested, in this case our logs and application metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpfZsm4Rj5MCtHsHj66Ghw.png" /><figcaption>Centralized logging — before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKh--Eh0CWB7CeFAtr_f8Q.png" /><figcaption>Centralized logging — after</figcaption></figure><p><a href="http://kafka.apache.org/">Apache Kafka</a> is a high throughput message bus. By introducing Kafka, we are now able to scale our data collection services indefinitely without risking losing logs.</p><p>We decided to use Filebeat , Kafka and Logstash.</p><p><strong>Filebeat: </strong><a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a> is a log data shipper for local files. Filebeat agent is installed on the servers, then filebeat monitors all the logs in the log directory and forwards them to a Kafka topic.</p><p><strong>Logstash: </strong><a href="https://www.elastic.co/products/logstash">Logstash</a> is used to collect the data from disparate sources (Kafka topic) and normalize the data into the destination of our choice, ElasticSearch in our case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*TmBNze6mtEd9P3_B.jpg" /><figcaption>Kibana dashboards</figcaption></figure><p><strong>Metrics</strong></p><p><strong>System metrics:</strong> We gather system metrics from AWS Cloudwatch into Grafana and Kubernetes metrics into Prometheus.</p><p>We monitor among other things:</p><ul><li>Total and used cluster resources: CPU, memory, filesystem.</li><li>Total cluster network I/O pressure.</li><li><a href="http://kubernetes.io/docs/user-guide/pods">Kubernetes pods</a> usage: CPU, memory, network I/O.</li><li>Containers usage: CPU, memory, network I/O.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/0*kMl8lWgBw4qYzDsT.png" /><figcaption>Grafana showing system metrics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/0*_wbfA_4_iGqCBLDu.png" /><figcaption>Prometheus sample metric</figcaption></figure><p><strong>Application metrics</strong>: We send application metrics using Filebeat to a Kafka topic and then we stream them into <a href="https://druid.apache.org/">Druid</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZaSiOBDL20q0Lrgc0heJQg.png" /><figcaption>Application metrics at scale with Kafka and Druid</figcaption></figure><p><strong>Tracing</strong></p><blockquote>A <em>trace</em> is a representation of a series of causally related distributed events that encode the end-to-end request flow through a distributed system.</blockquote><p>Traces look almost like event logs. A single trace can provide visibility into both the path traversed by a request as well as the structure of a request.</p><blockquote>Traces are used to identify the amount of work done at each layer while preserving causality by using <em>happens-before</em> semantics. A trace is a directed acyclic graph (DAG) of <em>spans</em>, where the edges between spans are called <em>references</em>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/0*8wX4wjExaoCqdB2R.png" /><figcaption>Flow of a single request through a distributed system</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FglB7EOTzdi7mDIh.png" /><figcaption>The various components of a distributed system touched during the lifecycle of a request, represented as a directed acyclic graph</figcaption></figure><p>When a request begins, it is assigned a globally unique ID, that is passed as a parameter to all the requests being sent afterwards, allowing to trace the flow on the same request. Reconstructing a flow of execution enables us to better understand the lifecycle of a request.</p><p>For example — understanding the entire request lifecycle allows us to debug requests across multiple services to pinpoint the source of latency or resource utilisation.</p><p>Two of the most popular open source distributed tracing solutions are <a href="https://zipkin.io/">Zipkin</a> and <a href="https://www.jaegertracing.io/">Jaeger</a>.</p><p>Adding tracing into an existing infrastructure is hard because every component in the path of a request needs to be modified to propagate tracing information. Using open source frameworks extensively might require additional instrumentation.</p><p>@Novarize we decided to inject a request identifier in the logs and manually trace the logs for understanding the requests flow without implementing a tracing framework.</p><h3>Conclusion</h3><p>To achieve resilience in distributed systems we need observability.</p><p>Logs, metrics and tracing help us achieve transparency and allows us to debug in production.</p><p>We have created a core library to be used in all services to send logs and metrics.</p><p>We have set a pre-defined structure with naming conventions for the logs and metrics to create a system “language” for debugging.</p><h3>Coming next:</h3><p>Part 2 of this article will tackle the architectural decisions to be made for resilience in distributed systems — Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=710c4c70d74c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data consistency across Microservices]]></title>
            <link>https://medium.com/@deniseschlesinger/data-consistency-across-microservices-4f768b253816?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f768b253816</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[event-driven]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Sat, 28 Jul 2018 08:58:33 GMT</pubDate>
            <atom:updated>2018-07-28T12:56:08.909Z</atom:updated>
            <content:encoded><![CDATA[<p>We were told a monolith is evil and microservices are the answer. What nobody told us is that microservices come with many pain points deriving from its distributed nature.</p><p>In the past, we built an application connected to one database where normalized data was queried using “joins”. Then came: big data, big traffic and with that: big latency. We needed to solve query latency where no cache would help us, the data was too big.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jnIdMYy5SNAMPs4R.jpg" /></figure><p>Microservices came to the rescue, we needed to break our big application into small pieces. Scaling our servers only in the required areas, allowing teams to work in different parts of a system without deploying the whole application, enabling autonomy.</p><p>We all jumped and started writing microservices, not always understanding the implications. We read blog posts on big companies and their hundreds of microservices; how they scaled their architecture, seeing what they built, but sometimes without understanding the “how”.</p><blockquote>“People try to copy Netflix, but they can only copy what they see. They copy the results, not the process” — Adrian Cockcroft, former Netflix Chief Cloud Architect</blockquote><p>To build microservices we needed to remove dependencies across domains and we found that this was easier said than done. The basic idea was to build services that were responsible for their own logic and wrote to their own DB.</p><p>Distributed services dictated distributed data and this came with a price, we gained autonomy and lost ACID transactions, a single source of truth and “joins” between entities. In order to reconcile these we needed to find a way to communicate across services.</p><h3>Microservices communication patterns</h3><ul><li><strong>Service-to-service communication:</strong> each service implements the logic it needs, gathering data from multiple services.</li></ul><p><strong>Pros</strong>: each service gathers all the data it needs, so the business logic is developed only in the corresponding service.</p><p><strong>Cons</strong>: creates dependencies between services, coupling them for deployment and maintenance. This results in a “distributed monolith”. Each service will need multiple hops to get all the data it needs and then will need to “join” it using business logic instead of queries in the DB.</p><ul><li><strong>API gateway: </strong>a single service that acts as the backend entry point.</li></ul><p><strong>Pros</strong>: Clients don’t need to understand the backend architecture. Reduces the number of requests from the clients.</p><p><strong>Cons</strong>: Every API needs to be developed twice, once in the gateway and once in the service itself. We still don’t have joins in the DB, we do it programmatically in the gateway. The gateway service gets all the load. We get latency since the gateway hops to many services and “joins” the data.</p><ul><li><strong>Aggregate data at the client side: </strong>clients perform different requests to all services in the backend and then aggregates the data.</li></ul><p><strong>Pros</strong>: services are autonomous and can be deployed separately without inter-dependencies at the backend.</p><p><strong>Cons</strong>: clients need to store all the gathered data and “join” the sources, sometimes requiring business logic. In mobile apps this approach is not acceptable, since it will require the app to demand updates for every change in business logic.</p><ul><li><strong>Events bus:</strong> each service sends events for its changes and the “interested” parties listen and save the info as needed.</li></ul><p><strong>Pros</strong>: Separation of concerns, low latency.</p><p><strong>Cons</strong>: Data duplication that carries storage costs, the need to write failover procedures, the risk of data inconsistency across services.</p><p>Choosing each one of these communications patterns demands: monitoring, alerts, data-modeling for consistency, transactional updates across domains and rollback capabilities.</p><p>You need to understand that writing distributed systems is hard (microservices are after all distributed systems).</p><p>In my case, I am a fan of the “lean and mean” approach and architecture evolution.</p><p>In every new project, I start with the simplest approach that can be built with a small team and as the application and the organization scales, the architecture evolves. Remember:</p><ul><li>Never compromise quality, test everything. Integration and end-to-end tests are your best friends, then you can refactor as you go without risking instability and regressions.</li><li>Build for the actual requirements of scale you have at the moment. Measure everything to know when to refactor and improve. Over-architecture is your enemy. Complicated technologies will stall your team’s progress.</li><li>Use decoupled architecture and define APIs for future refactoring, defining clear domains and responsibilities (for example: separate APIs for users, orders, stock, etc. that can be separated in the future)</li><li>Build everything using backwards compatibility where there are no breaking changes, neither in APIs nor in the DB. DB tables/columns are always added, never removed or changed.</li></ul><p>Keep in mind that “There is no such a thing as a free lunch”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f768b253816" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Querying our Data Lake in S3 using Zeppelin and Spark SQL]]></title>
            <link>https://medium.com/@deniseschlesinger/querying-our-data-lake-in-s3-using-zeppelin-and-spark-sql-be50c3b7a613?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/be50c3b7a613</guid>
            <category><![CDATA[apache-zeppelin]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Tue, 26 Dec 2017 16:43:34 GMT</pubDate>
            <atom:updated>2017-12-28T19:47:12.535Z</atom:updated>
            <content:encoded><![CDATA[<p>Until recently, most companies used the traditional approach for storing all the company’s data in a Data Warehouse.</p><p>The internet growth caused an increase in the number of data sources and the massive quantities of data to be stored, requiring scaling these Data Warehouses constantly. They were not designed to handle petabytes of data, so companies were driven into using big data platforms (such as Hadoop) capable of transforming big data into actionable insights at<strong> high speed</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*tlRhRoXZ42KSoy2E." /></figure><p>Hadoop traditionally coupled storage and computation together, so this was relatively expensive. Each node on a cluster wasn’t just a storage system, it was a server with processors and memory resources. The more data you keep, the more compute resources you needed to pay for, even though you didn’t really need that extra computational power to help you analyze your data. As a result, compute and storage were supposed to be scaled together and the clusters were persistently on, otherwise the data becames inaccessible.</p><p>This is the force that is driving the separation of storage from compute, to be able to scale them separately. This is how the term <a href="http://www.gartner.com/it-glossary/data-lake/">Data Lake</a> emerged, being defined as: “a collection of storage instances of various data assets additional to the originating data sources.”</p><p>A Data Lake is a repository capable of storing vast quantities of data in various formats. It can contain data from webserver logs, application databases, and third-party data. Data can flow into the Data Lake by either batch or real-time processing of streaming data.</p><p>The reason to build a Data Lake is to provide with the ability to query all data sources combined in a single place, providing “Data and Analytics as a Service” for business users such as data scientists and business analysts.</p><p>The Data Lake offers an approach where compute and storage can be separated, in our case, S3 is used as the object storage, and any processing engines (Spark, Presto, etc) can be used for the compute. This means we can scale separately depending on the business needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*fsGK2WOKNoF5WlsP.png" /></figure><p>As I explained in <a href="https://medium.com/@deniseschlesinger/building-a-data-lake-in-s3-using-an-event-driven-serverless-architecture-dd4d32dc285">one of my previous posts, </a>we decided to build our data lake based on Amazon Simple Storage Service (S3) to combine private and third party data in order to enable users to answer their most interesting business questions.</p><p>We use AWS Kinesis Firehose to push data into S3, AWS lambda functions for some pre-processing and Spark for our data pipelines.</p><p>In order to query S3 we needed a querying engine that allowed us to use SQL to provide access to TBs of structured and semi-structured data for quick analysis. We evaluated Presto, Athena and Spark SQL.</p><p>Out data is arranged in S3 in the following format (hadoop like partitions):</p><p>&lt;bucket_name&gt;\processed\&lt;event_type&gt;\dt=&lt;date&gt;\hh=&lt;time&gt;\*.json</p><p>In Presto and Athena (managed version of Presto by AWS) we encountered errors either when we created the tables with HIVE or when querying these tables. We found that both Presto and Athena do not “like” data types that change over time (for example: timestamp was string and then bigint) or any special characters at the beginning of the field names (_data_type). We corrected the data, using Spark and the pre-processing lambdas, so we thought this will suffice, but this was not enough, we had very bad performance when querying small JSON files. We knew that Parquet will perform much better, but the partitions dictated smaller files. So what did we do?</p><p>We thought to try Spark SQL as our query engine. <a href="https://spark.apache.org/sql/">Spark SQL</a> provides the capability to expose the Spark datasets over JDBC API and allow running the SQL like queries on Spark data using traditional BI and visualization tools.</p><p>On top of this, our analysts and data scientist needed strong visualization tools to find data patterns and correlations.</p><p>Apache Zeppelin was our choice. Zeppelin is a Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala, Python and more. It provides analysts a great way to create interactive web notebooks to write the queries and visualize the results. The notebooks can be shared in real time. The queries can be scheduled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qsecvGIrkByaToMN.png" /></figure><h3>Tutorial</h3><p>1- Create a cluster in AWS EMR with Spark and Zeppelin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*doO40ghqlC1oUUq2mn2_7g.png" /></figure><p>2- Click on the link “Zeppelin” after the cluster was provisioned to access Zeppelin UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Su2SW10m6TdyTxJqlG3drg.png" /></figure><p>3- Download sample data for the San Francisco Traffic accidents data. (it can be found at: <a href="https://data.sfgov.org/api/views/vv57-2fgy/rows.csv?accessType=DOWNLOAD">https://data.sfgov.org/api/views/vv57-2fgy/rows.csv?accessType=DOWNLOAD</a>).</p><p>4- Place it in an S3 bucket , for example: “test-zeppelin-ni”.</p><p>Now we can access and query the data using Spark SQL and Zeppelin.</p><p>We will write some Scala code inside Zeppelin to visualize this CSV file and extract information contained on it. In order to view the contents of this file and manipulate the data, we will create a dataframe, bind it to a schema, create a temporary table and then use SQL for queries.</p><p><em>“A powerful feature of Spark SQL is that you can programmatically bind a schema to a Data Source and map it into Scala case classes which can be navigated and queried in a typesafe manner.”</em></p><p><strong><em>import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType};<br>import org.apache.spark.sql.{DataFrame, Dataset, Encoders, SparkSession}<br>import org.joda.time.DateTime<br>case class TrafficData(IncidntNum: String, Category: String, Descript: String, DayOfWeek: String,<br> Date: String, Time: String, PdDistrict: String,Resolution:String,<br> Address:String,<br> X:Double, Y:Double, Location:String, PdId:String)</em></strong></p><p><strong><em>val schema: StructType = Encoders.product[TrafficData].schema</em></strong></p><p><strong><em>val df = sqlContext.read.schema(schema).csv(“s3a://test-zeppelin-ni/*”)<br>df.printSchema()</em></strong></p><p>Here is the second paragraph:</p><p><strong>df.registerTempTable(“TrafficTable”)</strong></p><p>Here is the third paragraph:</p><p><strong><em>%sql</em></strong></p><p><strong><em>select * from TrafficTable</em></strong></p><p>As you can see, after loading the data into Spark , we created a temp table and then we can query it using regular SQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tteSafALX9grN6ahAzcvtQ.png" /></figure><h3>Wrapping it up</h3><p>I hope this article gave you a good starting point to learn about how to use Spark and Zeppelin.</p><p>In this article, we have learned how to read a large dataset from a S3 public bucket and perform SQL queries on it. This combination is one of the most common used setups for Machine Learning projects running data at scale.</p><p>Note: Remember that the changes you make on Zeppelin only persist as long as the EMR cluster is running.</p><p>If you want to know more about Zeppelin<a href="https://youtu.be/J6Ei1RMG5Xo">, watch this video</a> by Moon soo Lee, who created Zeppelin, speaking in Amsterdam at Spark Summit Europe 2015.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be50c3b7a613" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Big Data is pushing us towards Machine learning]]></title>
            <link>https://medium.com/@deniseschlesinger/why-big-data-is-pushing-us-towards-machine-learning-105a8712aed0?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/105a8712aed0</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Sat, 14 Oct 2017 10:57:18 GMT</pubDate>
            <atom:updated>2017-10-15T14:41:54.425Z</atom:updated>
            <content:encoded><![CDATA[<p>As an Engineer Manager with more than 20 years of experience I have seen many changes that completely disrupted different areas: “Web 2.0”, “Cloud computing”, “Mobile-first”, “Big Data”, etc. The new kid on the block is “Machine learning” and it is definitely at its peak, one example is perfectly described by <a href="https://www.cbinsights.com/blog/funding-ai/">CB Insights</a> on how startups coined the terms related to Machine learning to raise their valuation: <em>“If you want to drive home that you’re all about that AI, use terms like machine learning, neural networks, image recognition, deep learning, and NLP. Then sit back and watch the funding roll in.”</em></p><p><a href="https://www.forbes.com/sites/louiscolumbus/2016/08/21/gartner-hype-cycle-for-emerging-technologies-2016-adds-blockchain-machine-learning-for-first-time/#d5e16933f82e">In 2016 Gartner’s famous Hype Cycle of Emerging Technologies</a> added the term “machine learning”, since it is being used everywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/0*rri_Gx4QYs_ky8Y0.jpg" /></figure><p>But let’s put things in perspective and try to understand why is this happening now.</p><p>Huge amounts of data are being streamed from phones, computers, TVs and IoT devices. Every day the equivalent of<a href="http://www.northeastern.edu/levelblog/2016/05/13/how-much-data-produced-every-day/"> 530,000,000 million digital songs or 250,000 Libraries of Congress</a> worth of data are being created globally.</p><p>The data gathered grows exponentially, creating a paradigm shift on how we store and process large data sets. This affects the data infrastructure and long-term devops strategic decisions we need to make in order to support the increasing demand for scalability and concurrency.</p><p><strong>But… It is not the <em>quantity</em> of data that is revolutionary but, that we can <em>do</em> something with the data.</strong></p><p>For most organizations leveraging massive data sets is a problem since not everyone knows how to deal with terabytes of data. It takes highly specialized teams to analyze and process insights. When the data is huge, it is not humanly possible to understand which variables affect each other.</p><p>This is where Machine Learning fits in and why it will ultimately change the way we handle data. Regardless of the amount, researchers need to ask the right questions, design a test, and use the data to determine whether their hypothesis is right.</p><h3>Let’s tidy things up</h3><p>Here is an interesting Venn diagram on machine learning and statistical modeling in data science (Reference: SAS institute)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/0*OI7KBzHqDfV00yji.png" /></figure><p><strong><em>Artificial Intelligence — </em></strong>a term coined in 1956, refers to a line of research that seeks to recreate the characteristics possessed by human intelligence.</p><p><strong><em>Data science — </em></strong>uses automated methods to analyze massive amounts of data to extract valuable knowledge and insight.</p><p><strong><em>Machine Learning — </em></strong>is a recent development that started in the 1990s when the availability and pricing of computers enabled data scientists to stop building finished models and train computers instead.</p><h3>Building data science teams</h3><p>Most organizations create heterogeneous teams that include three primary data-focused roles: data scientists, data analysts and data engineers.</p><p>Here are the key differentiators between the data-focused roles:</p><p><strong>Data Engineer — </strong>basically these are developers that know how to handle big data. In general they would have majors in: Computer science and engineering.</p><p><strong>Data Analyst — </strong>they translate numbers into plain English. A <strong>data analyst’s</strong> job is to take <strong>data</strong> and use it to help companies make better business decisions. In general they would have majors in: Business, economics, statistics.</p><p><strong>Data Scientist — </strong>they combine statistics, mathematics and programming. They have the ability to find patterns by cleansing, preparing, and aligning data. In general they would have majors in: Math, applied statistics, operations research, computer science, physics, aerospace engineering.</p><h3>Wrapping up</h3><p>Let me play with <a href="https://twitter.com/danariely/status/287952257926971392?lang=en">Dan Ariely</a>’s quote on big data as others already did:</p><p><em>“Machine Learning is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…”</em></p><p>So, can everybody use machine learning to increase ROI? probably not and I don’t think that in the near future all our jobs will be automated by bots. But, we definitely need to be able to massively process data at scale using machine learning in order to provide solutions whether it is to increase ROI in our organizations or to help solve global urgent problems like cancer research, natural resources scarcity and more.</p><p>Make sure you hire the right team and ask the right questions, hopefully your organization’s data + data science will provide you with the answers you are looking for.</p><p>Let me leave you with one last quote:</p><p><em>“We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten. Don’t let yourself be lulled into inaction.” — Bill Gates</em></p><p>So, start learning and researching so you can find the right answer for you and your data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=105a8712aed0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a data lake in S3 using an event-driven serverless architecture]]></title>
            <link>https://medium.com/@deniseschlesinger/building-a-data-lake-in-s3-using-an-event-driven-serverless-architecture-dd4d32dc285?source=rss-5be4408574c4------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd4d32dc285</guid>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Denise Schlesinger]]></dc:creator>
            <pubDate>Fri, 06 Oct 2017 16:02:56 GMT</pubDate>
            <atom:updated>2017-10-09T05:51:17.217Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is about the journey from:</p><ul><li>A data warehouse to a data lake.</li><li>Batch to near real-time processing.</li><li>Availability to query all data from the same repository (raw and processed).</li></ul><p>Let’s start by understanding some terms.</p><p>Wikipedia defines <a href="https://en.wikipedia.org/wiki/Data_warehouse">Data Warehouses</a> as:“…central repositories of integrated data from one or more disparate sources. They store current and historical data and are used for creating trending reports for senior management reporting such as annual and quarterly comparisons.”</p><p>And the term “<a href="https://en.wikipedia.org/wiki/Data_lake">data lake</a>” in Wikipedia states: “…A data lake is a method of storing <a href="https://en.wikipedia.org/wiki/Data">data</a> within a system or repository, in its natural format, that facilitates the collocation of data in various schemata and structural forms, usually object blobs or files. The idea of data lake is to have a single store of all data in the enterprise ranging from raw data (which implies exact copy of source system data) to transformed data …”</p><p>Until recently, the de-facto standard for handling big data would include several technologies that can cope with huge amounts of data such as it the technologies on the <a href="https://hadoopecosystemtable.github.io/">Apache Hadoop ecosystem</a> (deployed and managed with Cloudera or Hortonworks, for example). These are very powerful tools but they require lots of devops maintenance and tuning.</p><p>For most of the use cases, you really can’t go wrong with them performance-wise, but there can be a big difference between the ongoing operational burden of running your own infrastructure and letting AWS/Google/Microsoft do it for you using their managed solutions. Having said that, if your team requires special tuning and portability or on-premise installation, you should stick to the non-managed solution.</p><p>For the ones that can use managed solutions: AWS, Google and Microsoft have come to the rescue, offering many data services as PaaS (Platform as a service), that can provide the required functionality and scale without the complexity of maintaining the infrastructure associated with it.</p><p><strong>Our needs</strong></p><p>At <a href="http://www.naturalint.com/">Natural Intelligence</a> we manage and analyze data being delivered from our comparison sites. A critical part of our day-to-day is the ability to store and query it from the data warehouse.</p><p>Our ETLs crunch the data in batches every couple of hours at the moment, so the data availability and the monitoring capabilities are limited. We wanted to be able to get insights and alerts in near real-time.</p><p>When we started analyzing the use cases we encountered that our needs required the ability to query:</p><ul><li>Raw unfiltered and unstructured data (near real-time analytics, alerts on anomaly detection, machine learning).</li><li>Structured and optimized data (dashboards, analytics, daily aggregations)</li></ul><p><strong>Our Data Platform Architecture</strong></p><p>We use AWS all over and Redshift <a href="https://aws.amazon.com/data-warehouse/">as our data warehouse</a> with Tableau as the visualization tool, so it was only natural to look into what AWS has to offer us to solve our new challenges.</p><p>For the unstructured data we decided to build our data lake in S3 and query it using <a href="https://aws.amazon.com/redshift/spectrum/">some Distributed SQL engine for big data (for example: Redshift Spectrum</a>, <a href="https://aws.amazon.com/athena/">AWS Athena</a> or <a href="https://prestodb.io/">Presto</a>). <a href="https://aws.amazon.com/kinesis/firehose/">Using AWS Kinesis Firehose</a> for streaming data and <a href="https://aws.amazon.com/lambda/details/">AWS Lambda</a> for serverless processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4F-p0cBEapCWg_9mq7a_A.jpeg" /></figure><p><strong>S3 for storage</strong></p><p>Amazon S3 is <a href="https://aws.amazon.com/what-is-cloud-object-storage/">an object storage</a> built to store and retrieve any amount of data from anywhere . It is designed to deliver 99.999999999% durability, and stores data for millions of applications.</p><p><strong>AWS Kinesis Firehose and AWS lambda for data ingestion</strong></p><p>Kinesis Firehose pairs Kinesis streams with out-of-the-box Lambda functions, so it can deliver streams of data to pre-defined destinations such as S3, Redshift and ElasticSearch. Kinesis streams allow for real-time data processing continuously collecting data as it is generated.</p><p>AWS Lambda provides a strong FaaS solution. FaaS stands for “function as a service” which means that your code is executed in containers that are invoked in response to events like an http call to a REST API or a file created in an S3 bucket.</p><p><strong>A distributed SQL engine for big data</strong></p><p>We are evaluating 3 options at the moment.</p><p><a href="https://aws.amazon.com/athena/">Amazon Athena</a> is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.</p><p><a href="https://aws.amazon.com/redshift/spectrum/">Amazon Redshift Spectrum</a> enables you to run Amazon Redshift SQL queries against exabytes of data in <a href="https://aws.amazon.com/s3/">Amazon S3</a>. With Redshift Spectrum, you can extend the analytic power of Amazon Redshift beyond data stored on local disks in your data warehouse to query vast amounts of unstructured data in your Amazon S3 “data lake”.</p><p><a href="https://prestodb.io/">Presto</a> is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.</p><p><strong>Our Solution</strong></p><p>We use S3 as our data lake, storing both the raw and unstructured data and querying it using some distributed SQL query engine for big data.</p><p>Our applications send events to AWS Kinesis Firehose. Firehose writes the events into files located in S3 buckets. We use AWS Lambda functions written in python for processing the streaming events in the S3 files.</p><p>We will continue using ETLs to process the raw data into structured formats and aggregations that are visualized using Tableau.</p><p>We are now in the process of implementing <a href="https://aws.amazon.com/redshift/spectrum/">Redshift Spectrum</a> on top of S3 data lake that is showing great performance so far (more on this on the coming part of this series). We will be doing benchmarking on other distributed SQL query engines for big data and keep you posted.</p><p><strong>Next steps</strong></p><ul><li>Rollout the Redshift Spectrum implementation to be queried by relevant users.</li><li>Implement alerts and monitoring over the streaming data for near real-time processing using anomaly detection.</li><li>Create a near real time dashboard.</li><li>Tie it all together as a robust solution.</li></ul><p>I will be posting more articles and “how to’s” and “lessons learned” topics in here, so stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd4d32dc285" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>