<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vítor Resing Plentz on Medium]]></title>
        <description><![CDATA[Stories by Vítor Resing Plentz on Medium]]></description>
        <link>https://medium.com/@vrplentz?source=rss-1be461175f3f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lw6bc7r6d9zSskzOW7uMMg.jpeg</url>
            <title>Stories by Vítor Resing Plentz on Medium</title>
            <link>https://medium.com/@vrplentz?source=rss-1be461175f3f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 12:17:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vrplentz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Moving abroad —  Decision and Companies hiring.]]></title>
            <link>https://medium.com/@vrplentz/moving-abroad-decision-and-companies-hiring-cb49d238dd26?source=rss-1be461175f3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb49d238dd26</guid>
            <category><![CDATA[expat]]></category>
            <category><![CDATA[relocation]]></category>
            <category><![CDATA[work]]></category>
            <category><![CDATA[work-abroad]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Vítor Resing Plentz]]></dc:creator>
            <pubDate>Thu, 15 Dec 2022 19:37:22 GMT</pubDate>
            <atom:updated>2022-12-15T19:37:22.343Z</atom:updated>
            <content:encoded><![CDATA[<h3>Moving abroad — Decision and Companies hiring.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bPPSt--5ksLXZdId" /><figcaption>Photo by <a href="https://unsplash.com/@purzlbaum?utm_source=medium&amp;utm_medium=referral">Claudio Schwarz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Hello, “Ich bin Vitor. Ich kommen aus Brasillien” (“I’m Vitor, I’m from Brazil” — in German). Nice to meet you.</blockquote><p>This is my new introduction phrase since I moved to Germany to work as a Senior Data Engineer for Delivery Hero (2 months ago).</p><p>This is the first article from a series of articles about moving to work abroad.</p><p><strong>In this one we’re going to talk about:</strong></p><ul><li>Motives to do it;</li><li>When does it make sense for companies to hire and bring you from abroad;</li><li>Which kinds of companies are doing this</li></ul><h4>But Vitor, why are you writing about this?</h4><p>The simple answer is, It has been 2 months since I moved here and I already had <strong>several calls from old friends asking me how I managed to do it</strong>.</p><p>Also, when I was in the process, <strong>I wish I had all this info in one place</strong>.</p><h3>First thing, have the motive to go and the timing to do it:</h3><p>There always will be things you love back at home, your family, your friends, your pet, and yes the local food.</p><p><strong>If you want to move abroad you’ve to be decided to do it, and hopefully with a good reason.</strong></p><p>My motives (<em>besides the wish that most people have</em>) were <strong>mainly a strong need for a new personal and professional challenge and the search/need for a work-life balance oriented culture</strong>.</p><p>Also, to be fair I was in a good position to take this decision, not many responsibilities to attach me to staying back in Brazil, and I had the money to stay alive for a few months if everything goes wrong. <strong>That’s when the motive matched the timing for me.</strong></p><h3>Cutting to the chase. When does it make sense to do it (career-wise)?</h3><blockquote>Maybe the question is “When does the companies will see enough value to bring you from abroad?”.</blockquote><p>Nowadays many companies are bringing talent from abroad, giving relocation bonuses, and even Visa sponsorship to guarantee to have the best employees.</p><p>Usually those companies that are paying and assisting you to come will ask for a few things:</p><ul><li><strong>Experience that is worth bringing:</strong> Usually you will see Senior roles for relocation;</li><li><strong>Profile:</strong> They will try to evaluate if this person has the profile to handle the pressure of working in a different culture, away from everything they are used to. Mainly <strong>understanding their motivation behind choosing of moving abroad;</strong></li><li><em>Maybe they will require a College Diploma: Depending on the country you’re going to, having an IT Degree will help you in regards to the working Visa, usually it’s faster than the common ones, since that’s a field with high demand.</em></li></ul><p>They are asking all this stuff because they want to guarantee that they are minimizing the risk of bringing someone from abroad. If you think about it, they are spending a lot of resources to get you.</p><p>For example (this may vary):</p><ul><li>HR interviews (at least 1 month);</li><li>Visa Requirements (my case 3 months);</li><li>Visa and Housing assistance;</li><li>Relocation package (bonus for you to come).</li></ul><p><em>Disclaimer: It’s not impossible to get Junior-Midior roles, but you probably will find fewer open positions.</em></p><h3>The kind of company that is getting people from abroad.</h3><p><strong>Start-ups/Scale-ups:</strong> companies that recently got VC investment and need talent.</p><p>Examples:</p><ul><li>Tickets — Netherlands;</li><li>OneFootball;</li><li><em>Go and find them…</em></li></ul><p><strong>Large Companies:</strong> Companies that one day were startups, but today are gigantic.</p><ul><li>Google — Worldwide;</li><li>AWS — Worldwide;</li><li>DeliveryHero — Germany;</li><li>Booking — Netherlands;</li><li>HelloFresh — Germany;</li><li>Adyen — Netherlands;</li><li><em>Many others;</em></li></ul><p><strong>Consultancy companies:</strong></p><ul><li>Deloitte — US;</li><li>McKinsey — I guess worldwide;</li><li>Google — Depending on the department — Worldwide probably.</li><li><em>Some others.</em></li></ul><p>As you may notice, I have much more examples for large companies. But I don’t believe this is a coincidence, from my experience those are the kind of companies you will find the most open positions to apply for.</p><h3>What comes next? 👀</h3><p>I know, I know, there is still a lot to cover. 😅</p><p>In the next article, I’ll bring more information on the process itself, and how I got prepared for it.</p><p>Have any questions? Ask in the comments.</p><p>Thanks for reading!! 🤙</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb49d238dd26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Have you heard of Data Trusts? !]]></title>
            <link>https://medium.com/@vrplentz/have-you-heard-of-data-trusts-d90bff6dc7a7?source=rss-1be461175f3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d90bff6dc7a7</guid>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Vítor Resing Plentz]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 00:12:38 GMT</pubDate>
            <atom:updated>2022-08-19T00:12:38.854Z</atom:updated>
            <content:encoded><![CDATA[<h3>Have you heard of Data Trusts? !</h3><p>Hello, fellow Data Engineers,</p><p>I have to start with a question for you… Have you heard of Data Trusts?</p><p>In this post we will be discussing, What are Data Trusts, examples of utilization of this concept, and also what are some of the challenges when creating a Data Trust Platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mKaPhq7YQeCNW8fG" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_source=medium&amp;utm_medium=referral">NASA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>What is a Data Trust:</h4><p>Imagine you’re working in an industry, and you’re assigned a new project that wants to get data from your customers databases to be able to give Stakeholders access to accurate market data.</p><p>Your stakeholders want to get regular information about how our industry products are doing in the customer stores.</p><p>So the first checkbox you’ve got is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*Thk3qC84j_bp8GhnS2Wb0g.png" /></figure><p>But when deeply think about it, you wonder,</p><blockquote>“Isn&#39;t this the data that every single industry wants to have access to?</blockquote><blockquote>Why does each company have to create this similar data platform every single time?</blockquote><blockquote>There should be a company organizing all this data and doing the sharing governance between partners. Then we could just focus on the Business Value.”</blockquote><p>That’s the concept and the why of Data Trusts!!</p><p>To be more precise, accordingly to <a href="https://hbr.org/2020/11/data-trusts-could-be-the-key-to-better-ai">George Zarkadaskis on HBR</a>, Data Trusts are:</p><blockquote>“A data trust that is set up as a fiduciary for the data providers could make it much easier for firms to safely share data by instituting a new way for governing the collection, processing, access, and utilization of the data.</blockquote><blockquote>That legal and governance setup obliges the data trust administrators (the “fiduciaries”) to represent and prioritize the rights and benefits of the data providers when negotiating and contracting access to their data for use by data consumers, such as other private companies and organizations.”</blockquote><h4>Benefits of the sharing model:</h4><p>With this model is possible to:</p><ul><li>Democratize access to data, making data accessible to businesses and people;</li><li>Create governance and ensure compliance on data;</li><li>Standardize access to data: imagine having most of your partners data integrated as one large source, not tons of small datasets;</li></ul><h4>OK, got my interest. But what exactly are the challenges we face when building such a type of solution??</h4><p>Here are some of the challenges involved:</p><p><strong>Challenge 1: SPONSORSHIP!!! 💰 💸</strong></p><p>Lifting a Data Trust Platform is no easy thing (I’ve been there), and it’s not cheap.</p><p>To be able to start such a platform, you have to be able to crack a data marketplace (chicken and egg problem) between the customers that want to share data (producers) and the customers that want to access data (consumers).</p><p>In the beginning, you probably have to find partners (mostly consumers) that want to help build such an “ecosystem” and can invest in the Trust Platform, giving some kind of benefit back to the data producer.</p><p><strong>Challenge 2: Data Acquisition!! 💻</strong></p><p>Let’s say you have a wallet full of cash and can start the Trust Platform.</p><p>You still will have to manage data variety, connect with many kinds of sources, talk to customers proprietary systems to gather data, and have enough people to create new connectors (or create a queue for integrations 😅).</p><p><strong>Challenge 3: Managing Data Contracts: 📎</strong></p><p>When contacting the proprietary systems, a bunch of different types of integrations will appear, you better have a contract with the provider to get some guarantee on schema changes or updates.</p><p>In my experience, you should expect to have many different versions of integrations with the providers. Different customers will have different versions, even with the same provider (your job will be minimizing those).</p><p><strong>Challenge 4 Standardizing Data and adding governance: 🛂</strong></p><p>The consumers customers will expect to have access to data in some format (so their applications can connect), and the producers customers will want to have their data protected and shared upon agreements.</p><p>To do so with such a variety of sources, your team will have to be able to map those into a specific schema and understand those datasets to make them compliant.</p><h4>Challenge 5: Creating Catalogs: 📅</h4><p>When you create a data trust, you want to be sure that every new customer integration can “talk” to the current Data Trust Database.</p><p>For example, you have to be able to know that an Xbox Series S is an Xbox Series S in every Retailer customer in your Trust.</p><p>To do so, expect to create Catalogs to act like connectors between different customers. These Catalogs will be using Global unique identifiers, such as GTINs, and Zip Codes.</p><p>Some companies have products only for this.</p><ul><li>Bluesoft Cosmos (<a href="https://cosmos.bluesoft.com.br/">https://cosmos.bluesoft.com.br/</a>);</li><li>GS1 (<a href="https://www.gs1br.org/">https://www.gs1br.org/</a>).</li></ul><h4>Examples of Data Trusts:</h4><p><strong>Retail — Skupos:</strong></p><p><a href="https://www.skupos.com/">Skupos</a> connects convenience Retailers to enterprise Brands, giving benefits to Retailers while giving market insights to Brands.</p><p>It claims to have connected more than 15000 customers and over 8M transactions per day.</p><p><strong>Healthcare — IQVIA:</strong></p><p><a href="https://www.iqvia.com/insights/the-iqvia-institute/available-iqvia-data">IQVIA has a lot of datasets</a> on Healthcare, allowing healthcare companies to innovate with data.</p><p>It claims to have more than 1.2B non-identified patient records, over 150000 data suppliers, and 85% of the global sales pharma tracked.</p><h4>Conclusion</h4><p>By now you should be familiar with Data Trusts and their challenges.</p><p>Feel free to comment.</p><p>THANKS for the reading!! 😁</p><h4>Resources:</h4><ul><li><a href="https://hello.elementai.com/rs/024-OAQ-547/images/Data_Trusts_EN_201914.pdf">Element AI on Data Trusts</a>;</li><li><a href="https://hbr.org/2020/11/data-trusts-could-be-the-key-to-better-ai">HBR on Data Trusts</a>;</li><li><a href="https://medium.com/@anoukruhaak/data-trusts-why-what-and-how-a8b53b53d34">Medium: Data Trusts Why, What and How?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d90bff6dc7a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replacing Confluent Wikipedia Tutorial Processing Stack]]></title>
            <link>https://medium.com/@vrplentz/replacing-confluent-wikipedia-tutorial-processing-stack-71a04864305c?source=rss-1be461175f3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/71a04864305c</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[confluent]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[schemaregistry]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Vítor Resing Plentz]]></dc:creator>
            <pubDate>Sat, 16 Jul 2022 15:23:40 GMT</pubDate>
            <atom:updated>2022-07-16T15:23:40.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*3ayZgx52Yp5tCpMwtlIEow.jpeg" /><figcaption>PySpark Structured Streamming + Confluent Kafka + Confluent Schema Registry</figcaption></figure><p>I was studying about Kafka using <a href="https://developer.confluent.io/learn-kafka/">videos and tutorials from Confluent</a>, and eventually, when digging around their tutorials and videos I’ve got into the <a href="https://github.com/confluentinc/cp-demo">Wikipedia repository</a>.</p><p>This repository lifts a Docker Confluent Kafka Infrastructure (as the image bellow). With Kafka Connect sending Wikipedia data to topics, real-time processing done by KSQL and a KStream App, and lastly the transformed data is sent to other topics which are consumed by ElasticSearch Sink Connector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZj9ChgkvOu447Aiv_Y4vQ.png" /><figcaption>Wikipedia Confluent Kafka Architecture. Image from <a href="https://github.com/confluentinc/cp-demo">https://github.com/confluentinc/cp-demo</a></figcaption></figure><p>When I looked at this I thought <em>“What if I transform all this data using PySpark Structured Streamming instead of KStream and KSQL? This seems a nice home project to do!!“</em>… and that’s when the the story begins.</p><p>This article shows how to connect the PySpark Structured Stremming with Kafka with Confluent Schema using the Wikipedia Confluent Repository as base.</p><h3>Approach the project changes:</h3><p>The first thing was making a list of things I should do:</p><ul><li>Make the project run as is;</li><li>Understand the schemas ;</li><li>Removing old processing stack;</li><li>Read data from Kafka using PySpark Structured Streaming;</li><li>Parse the value data from messages ;</li><li>Transform the data to replace a real-time transformation;</li><li>Load the data back to Kafka.</li></ul><h3>Running the project as is and understanding the schema:</h3><p>Running the project run was the most easy part I just followed the <a href="https://docs.confluent.io/platform/current/tutorials/cp-demo/docs/on-prem.html">tutorial</a> and run the start script ./scripts/start.sh.</p><p>With the project running with Docker, I’ve opened the Confluent Central (localhost:9091) to check the schema and actually see the data flowing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5uvdsAR-8YuRzAT2vvPIQ.png" /><figcaption>Data flowing from Wikipedia Connector to wikipedia.parsed Topic.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NGvO_JquHdnPBTl4DU4DA.png" /><figcaption>Checking the wikipedia.parsed topic schema.</figcaption></figure><ul><li>Make the project run as is; ✅</li><li>Understand the schemas ✅</li></ul><h3>Removing old processing stack:</h3><p>After checking the data Schema, checking it’s fields, message value type (Avro), and verifying that everything was working as supposed, I finally could start to modify the Infrastructure so it doesn’t automatically start processing data using the KStream App or the KSQL queries.</p><p>To do so I’ve modified the many scripts:</p><ul><li>docker-compose removed the KStream App container;</li><li>Removed kstreams-app directory;</li><li>Removed the scripts/ksqlDB directory (it contains the KSQL transformations);</li><li>Modified scripts/start.sh , removing KStream App container init and KSQL queries execution.</li></ul><p>If you want to go deeper on this step you can take a look into my <a href="https://github.com/vplentz/kafka_wikipedia_spark_processing">fork repository.</a></p><ul><li>Removing old processing stack ✅</li></ul><h3>Reading data from Kafka using PySpark Structured Streaming:</h3><p>So now I thought “I can just do as the PySpark + Kafka Documentation says connect to Kafka and then parse the data using the Avro Schema”, and that’s what I’ve done with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e746b8c1a3123245101258068495e969/href">https://medium.com/media/e746b8c1a3123245101258068495e969/href</a></iframe><p>And it worked, I could now see the data flowing into the PySpark Application (check PySpark container console).</p><ul><li>Read data from Kafka using PySpark Structured Streaming ✅</li></ul><h3>Parsing the value data from messages;</h3><p>So even thought I could read data from the Kafka topic, my code couldn&#39;t actually understand the messages values, since it wasn&#39;t connected to Confluent Schema Registry.</p><p>Even if you try to read it with your own Avro schema by comparing to the Confluent Central, the deserialization wont work as expected.</p><p>To be able to deserialize the DataFrame value column with Confluent Avro schema, you will have to connect to Confluent Schema Registry, ask for the schema and then parse the column.</p><p>To do so with PySpark I’ve found the <a href="https://github.com/AbsaOSS/ABRiS"><strong>Abris library</strong></a><strong> that allows pain free integration between Spark and Kafka with Schema registries.</strong></p><p>Unfortunately the integration was not 100% pain free, I had to work trough the examples and Confluent Docs to be able to use it with PySpark and Confluent Schema Registry with users credentials as SSL.</p><p>To do so you will have to create a wrapper for the Java/Scala functions and then create a dict of configs to be used to connect to the Schema Registry (getting this right was the hard part).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbd9ac5d4a69fe0fbdd16b56fd7d8765/href">https://medium.com/media/cbd9ac5d4a69fe0fbdd16b56fd7d8765/href</a></iframe><h4>Explaining the configs:</h4><p>URL config:</p><ul><li>schema.registry.url : is the URL from the Confluent Schema Registry;</li></ul><p>SSL Configs:</p><ul><li>schema.registry.ssl.truststore.location: Used for HTTPS. Location of the trust store. Required only to authenticate HTTPS clients;</li><li>schema.registry.ssl.truststore.password: Truststore password;</li><li>schema.registry.ssl.keystore.location:Used for HTTPS. Location of the keystore file to use for SSL;</li><li>schema.registry.ssl.store.password : Keystore password;</li></ul><p>Auth Configs:</p><ul><li>basic.auth.credentials.source : USER_INFO for user password credentials;</li><li>basic.auth.user.info : User:Password String.</li></ul><p><strong>To check more about those settings go to:</strong></p><ul><li><a href="https://github.com/AbsaOSS/ABRiS#schema-registry-security-and-other-additional-settings">Abris Schema Registry Additional Settings</a>;</li><li><a href="https://docs.confluent.io/platform/current/schema-registry/installation/config.html#ssl-truststore-location">SSL Configs</a>;</li><li><a href="https://docs.confluent.io/platform/current/security/basic-auth.html#schema-registry">Auth Configs</a>.</li></ul><p>If you rundf.printSchema() to check the topic received data, you will get something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*KVwqH6v3-cmuKjaI3rOhuQ.png" /><figcaption>DataFrame Schema before parsing the value column.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*yJZI6LBsxiBdy_Fi8LhRtQ.png" /><figcaption>DataFrame schema after running the decode Confluent Avro Schema code.</figcaption></figure><ul><li>Parse the value data from messages; ✅</li></ul><h3>Transforming the data to replace a real-time transformation:</h3><p>For this article I’ll replace only the KSQL No Bot Filter SQL (<em>to access the other transformations go to the </em><a href="https://github.com/vplentz/kafka_wikipedia_spark_processing"><em>repository code</em></a>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3ddbf37fde9933b8b5e1405b6c6b536/href">https://medium.com/media/a3ddbf37fde9933b8b5e1405b6c6b536/href</a></iframe><p>This KSQL query creates a Stream processing filtering data generated by bots and creating a new column with the byte changes between the new and the old Wikipedia post lengths.</p><p>Replacing it with PySpark:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6d8b8a5d4352f58cfe20d7041dc26d7/href">https://medium.com/media/a6d8b8a5d4352f58cfe20d7041dc26d7/href</a></iframe><ul><li>Transform the data to replace a real-time transformation;✅</li></ul><h4>Loading the data back to Kafka.</h4><p>The last step is loading back the data to a Kafka topic called “WIKIPEDIANOBOT”, to do so you have to create the subject with the schema into the Schema Registry, and then you should be able to load the data with schema guarantees from the Schema Registry.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7ed2b514d6e81736b9eaa44d835fc53/href">https://medium.com/media/d7ed2b514d6e81736b9eaa44d835fc53/href</a></iframe><p>And then you can start writing back to Kafka:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6293faedf28e35929d6f2f25ae37356d/href">https://medium.com/media/6293faedf28e35929d6f2f25ae37356d/href</a></iframe><ul><li>Load the data back to Kafka. ✅</li></ul><h4>Conclusion + Repository with example:</h4><p>This was a fun project to do, and I hope that with this knowledge more people can use PySpark Structured Streaming with Confluent Kafka + Schema Registry without much pain.</p><p>If you want to go deeper on how to actually run this, go to my <a href="https://github.com/vplentz/kafka_wikipedia_spark_processing">forked repository.</a></p><p>To check an example of a full code in a Jupyter Notebook, go to this <a href="https://github.com/vplentz/kafka_wikipedia_spark_processing/blob/7.1.0-post/pyspark_structured_streaming/jupyter_sandbox/bot_filter.ipynb">link</a>.</p><p>Thanks for the reading. 😁</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71a04864305c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Architectures Explained]]></title>
            <link>https://medium.com/@vrplentz/data-architectures-explained-b788cdbef687?source=rss-1be461175f3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b788cdbef687</guid>
            <category><![CDATA[kappa-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[lambda-architecture]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Vítor Resing Plentz]]></dc:creator>
            <pubDate>Fri, 08 Jul 2022 17:39:47 GMT</pubDate>
            <atom:updated>2022-07-08T17:39:47.846Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BXvV_23EPYTxy6rX" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Overview of Data Architectures</h3><p>So you’ve started to search about Data Architectures and heard a bunch of different (awkward) keywords… Traditional Data Lakes, Lambda and Kappa Processing Architectures. What does that even mean? 👀</p><p>Those are Data Architectures that we’re exploring in this article!! The idea is to explain in a simple way how they work , why they were created and give an example of each one! 😁</p><p>Before we start, check if you’re familiar with the following concepts (if not, start there):</p><ul><li><a href="https://databricks.com/discover/data-lakes/introduction">What’s a Data Lake</a>;</li><li><a href="https://www.confluent.io/learn/batch-vs-real-time-data-processing/#:~:text=Batch%20processing%20is%20when%20the,data%20flows%20through%20a%20system.">Batch and Streaming Processing</a>.</li></ul><p><strong>Spoiler alert 🚨</strong>, it’s all an concept evolution!!</p><h4>Traditional Data Lakes and Data Warehouses:</h4><p>Imagine you’re an PM (product manager) and needs to be able to understand how users have been using the product you’ve been working into to present in a important meeting. To do so, every week you ask an engineer from your team to create a report about some metrics over the last year data alongside some other sources information.</p><p>Unfortunately over and over again the reports just take too long to be done, and often other stakeholders also have their own reports, but with different results from yours.</p><p>In this context Traditional Data Lakes and Warehouses were created. Using them as a central source of truth, companies can minimize the work/time to get reports in hands (get advanced analytics), democratize access to information (avoid non-matching reports about KPIs) and be able to use the power of Machine Learning.</p><p>Traditional Data Lakes are usually a Blob Storage (S3, GCS, etc) or HDFS with batch loaded (from time to time) and a Batch Processing tool (usually Spark) that transforms data and send it to Data Lake zones.</p><p>Usually Data Lakes use the following name conventions for each zones (but in practice you may find many customized DLs):</p><ul><li>Landing Zone <em>(AKA Raw Zone or Bronze Zone)</em>: All data loaded as is;</li><li>Staging Zone (AKA Trusted Zone or Silver Zone): Mainly cleaned data, data delivered as agreements on data quality;</li><li>Analytics Sandbox <em>(AKA Refinement Zone, Silver Zone)</em>: Area where Data Engineers, Data Scientists and Data Analytics can discover, explore and experiment;</li><li>Curated Data Zone (AKA Refined Zone, Gold Zone): All processed data, usually available for consuming by other applications and data professionals;</li></ul><p>Data Warehouses are high volume analytical databases (BigQuery, RedShift, SnowFlake), with easily queryable data, Data Visualiztion tools are commonly seem connected to them. Usually inside DWs, data is formatted into dimensional schemas such as Star and Snowflake Schema.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*mn8onrGXu9VvoeXMKgPD0A.jpeg" /><figcaption>Traditional Data Lake Example.</figcaption></figure><h4>Lambda Architecture:</h4><p>So we’ve our Traditional Data Lakes, and now we are finally able to discover look at the past and being able to understand what happened and generate business value on it. But what if you don’t want to wait until the next batch jobs run to get value of data? EG: Stakeholders don’t wanna see info on the user behavior from yesterday, they want metrics on the last 10 minute.</p><p>That’s why the Lambda architecture has been proposed, the goal of this architecture is to be able to use a mix of Batch and Streaming Processing techniques.</p><p>The idea is to get fast access to what is happening now trough Streaming Processing and also be able to leverage historic data with Batch Pipelines.</p><p>Usually the architecture has the following concepts:</p><ul><li>Batch Layer: A layer with that process data in a batch fashion, can be structured and processed in a similar way to Traditional DLs;</li><li>Speed Layer: A streaming storage and processing layer. Data is usually stored inside Kafka Topics and processed/transformed with Kafka Streaming, KSQL, or other Streaming Processing Tool and written back to another topic;</li><li>Serving Layer: A layer where data is ready for consumption. It may have datasets/views from the Batch Layer, the Stream layer and also a combination of both. There are many DBs used in this layer, such as BigQuery, Cassandra, DynamoDB, etc.</li></ul><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*zPNDQXuNSzLU_BD-ljpJ7A.jpeg" /><figcaption>Lambda Architecture Example. Edges are ingestion/transformation processes.</figcaption></figure><p><em>Note that</em> f<em>rom time to time the data in the topics are send to Batch Layers.</em></p><h4>Kappa Architecture:</h4><p>So your Stakeholders are finally happy that they can use both company historical data and real time data to achieve better results, but as you were implementing the Lambda Architecture you’ve noticed that you had a bunch of “duplicated code”.</p><p>After taking a deeper look, you found that you’ve two codebases with similar transformations, one for the Batch Layer and one for the Speed Layer. Day-to-day maintenance to both types of processing techniques is getting harder and harder, matching the outputs between the shared views using those two layers, and finding engineers to keep the two necessary infrastructure is not an easy task too.</p><p>So what know? Data Engineers are we deemed to be firefighters? 🚒</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b6EpdyA97E_mJiWZ_EVYKg.png" /><figcaption>Lambda is hard to maintain!! Image from: <a href="https://www.slideshare.net/HostedbyConfluent/big-data-kappa-mark-senerth-the-walt-disney-company-dmed-data-tech">https://www.slideshare.net/HostedbyConfluent/big-data-kappa-mark-senerth-the-walt-disney-company-dmed-data-tech</a></figcaption></figure><p>That’s why the Kappa Architecture was designed, to be able to handle both streaming and batch pipelines using the same codebase and infrastructure.</p><p>In the Kappa Architecture all data passes trough the Streaming Layer (probably Kafka) and it’s processed as it arrives inside the Streaming Layer. So as each message arrives in the Message System, your consumers (data processors) are collecting it and doing it’s own transformations.</p><p>If you’re wondering, what about the past data? In case you need to process all historical data, your consumers should use data from the topic beginning (first message).</p><p>So by having all historic data available in the topics, you’re able to use only one data processing tool, one code, one infrastructure to make data available in the Serving Layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*B2IHdNWBHn_Bl9DVn4FRMQ.jpeg" /><figcaption>Kappa Architecture Example</figcaption></figure><h4>Conclusion. What’s next?</h4><p>Now that you’re familiar with some of the most common Big Data Architectures, my suggestion (and maybe future articles) is to go deeper on those Architectures:</p><ul><li>Search use cases of each architecture;</li><li>Understand when should you use each;</li><li>Understand what tech stack you could be using to implement them;</li><li>Understand the complexity of creating newer architectures.</li></ul><p>Also other architecture/organization topics that you should take a look:</p><ul><li><a href="https://developer.confluent.io/learn-kafka/data-mesh/intro/">Data Mesh</a>;</li><li><a href="https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf">Lakehouse Architecture</a>.</li></ul><p>Do you have any comments or suggestions, feel free to comment them!!</p><p><strong>References:</strong></p><ul><li>Manoj Kukreja, <a href="https://www.amazon.com/Engineering-Apache-Spark-Delta-Lakehouse-ebook/dp/B098X63L4V">Data Engineering with Apache Spark, Delta Lake, and Lakehouse,</a> 2021;</li><li>Mark Senerth, <a href="https://www.confluent.io/events/kafka-summit-americas-2021/big-data-kappa/">Kafka Summit Americas</a>, 2021;</li><li>Andrew Carr, <a href="https://blog.scottlogic.com/2021/05/28/what-actually-is-a-data-mesh-and-is-it-really-a-thing.html">What actually is a Data Mesh? And is it really a thing?</a>, 2021;</li><li>Martin Kleppmann, <a href="https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/">Tuning the database inside out with Apache Samza</a>, 2015;</li><li>Data Bricks, <a href="https://databricks.com/discover/data-lakes/introduction">Introduction to Data Lakes</a>, 2021;</li><li>Data Bricks, <a href="https://databricks.com/glossary/lambda-architecture">Lambda Architecture</a>, 2021;</li><li>Irene Mikhailouskaya, <a href="https://www.scnsoft.com/blog/data-lake-implementation-approaches">Alternative Approaches to Implementing Your Data Lake</a>, 2018;</li><li>Data Lakers, <a href="https://medium.com/datalakers-blog/o-que-s%C3%A3o-as-zonas-de-um-data-lake-5c00af387a4e">O que são as zonas de um Data Lake?,</a> 2021.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b788cdbef687" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing Pelotas citizens requests to the city hall.]]></title>
            <link>https://medium.com/@vrplentz/analyzing-pelotas-citizens-requests-to-the-city-hall-9596842e8a92?source=rss-1be461175f3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9596842e8a92</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pelotas]]></category>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Vítor Resing Plentz]]></dc:creator>
            <pubDate>Mon, 19 Feb 2018 16:55:59 GMT</pubDate>
            <atom:updated>2018-11-30T13:27:37.911Z</atom:updated>
            <content:encoded><![CDATA[<p>The contributors to this article are <a href="https://medium.com/u/db88c1ac021e">Lorenzo F. Antunes</a>, Nelson Dutra and I.</p><p>In this project we choose to investigate the citizens request to the city hall.</p><p>The dataset is available at <a href="http://www.pelotas.com.br/portal-de-dados/">http://www.pelotas.com.br/portal-de-dados/</a> and in the git repository is <a href="https://github.com/vplentz/pelotasTextAnalysis">here</a>.</p><p>We choose two approaches, the one of them is an text analysis, and the other one it’s a try to find insights from all the data.</p><h3>The first approach (Portuguese text analysis):</h3><p>In the dataset there is a column that describes peoples request as text, this text follows no specific structure as shown bellow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o47Atj3Kk1lg8aVWuZjEBw.png" /><figcaption>Peoples requests sample.</figcaption></figure><p>After reading some of the requests, we wanted to get the whole context from the words, try to find some patterns using only the text decryption. To achieve this we followed some steps:</p><h4>Step 1: Make text easier to analyze.</h4><ul><li>Make all text words lowercase;</li><li>Transform the texts in a list of words, so it’s possible to make a frequency analysis;</li><li>Remove portuguese stop-words (stop-words are usually words connectives such as “e”, “ou”, “mas”, …);</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*6jgWHc5gR9JoASaJ1mKN0Q.png" /><figcaption>This is how we did the steps described above.</figcaption></figure><h4>Step 2: Applying Frequency distribution in our list of words, and analyzing the 100 most used words.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RliphZV2A5V480gwdp2lgQ.png" /><figcaption>WordCloud of the most used words</figcaption></figure><p>The list of the 100 most used words can be found in the <a href="https://github.com/vplentz/pelotasTextAnalysis/blob/master/text_analizis.ipynb">notebook</a>, decided not to post it here, to keep this article readable.</p><p>Analyzing the words, we found that even removing the stopwords, much of the words shown didn’t give us much of the context we wanted. To achieve that we filtered the words to have a minimum size of 4 and frequency above 200.</p><h4>Step 3: Re-analyzing the filtered content:</h4><p>To have a better view of the word frequency we’ve plotted a bar graphic (also can be found in the <a href="https://github.com/vplentz/pelotasTextAnalysis/blob/master/text_analizis.ipynb">Notebook</a>, same reason). Taking a look at the most used words in the texts we found 2 things:</p><ul><li>Some of the city neighborhoods were in the list (Centro, Fragata, Laranjal, Areal and Três Vendas);</li><li>Some of the words pointed to the same context, so we decided to group some of them in 4 groups “saneamento”, “limpeza”, “poda” and “iluminação” (which are sanitation, cleaning, pruning and lighting).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*4yWfxGNEqMSLJ5fq_FJISA.png" /><figcaption>Grouped frequency</figcaption></figure><p>We found that the most common problems are related to the groups we found and the neighborhoods that complain most also were present in the text:</p><h3>Step 4: Verifying what we found:</h3><p>To verify what we found we matched it with other columns content;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*efhtlAMcwGxNh4eC5GNUPA.png" /><figcaption>Verify step</figcaption></figure><h3>Conclusion of first approach:</h3><p>Using the requests decryption we were able to find out some of the city problems and some neighborhoods that are requesting improvements to the town hall. Haply our analysis match with the other columns in the dataset content, taking a look of the request classification (shown as “tipo”) and our problem grouping it’s noticeable that they are very close to each other, and the 5 most cited neighborhoods (shown as “descricao”) match perfectly with the 5 most cited in our frequency analysis.</p><h4>Our try to find insights using all available data is <a href="https://medium.com/@lorenzofantunes/analise-de-dados-abertos-da-prefeitura-de-pelotas-rs-cbca40f456f6">here</a> (in portuguese).</h4><p>Thanks, for the reading!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9596842e8a92" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>