<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Seleme Topuz on Medium]]></title>
        <description><![CDATA[Stories by Seleme Topuz on Medium]]></description>
        <link>https://medium.com/@seleme.topuz?source=rss-4bc5eeb71e6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4SEZN8JlL7GF4p9S.jpg</url>
            <title>Stories by Seleme Topuz on Medium</title>
            <link>https://medium.com/@seleme.topuz?source=rss-4bc5eeb71e6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:51:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@seleme.topuz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Picnic 10 Years: 2021 — Expanding into France, and beyond]]></title>
            <link>https://medium.com/picnic-engineering/picnic-10-years-2021-expanding-into-france-and-beyond-694ac400bcde?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/694ac400bcde</guid>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Tue, 15 Jul 2025 12:32:13 GMT</pubDate>
            <atom:updated>2025-07-15T12:32:13.985Z</atom:updated>
            <content:encoded><![CDATA[<h3>Picnic 10 Years: 2021 — Expanding into France, and beyond</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*elYLCfdXxLezQHVL" /></figure><p>In our seventh year, Picnic launched in France — a milestone not just in expansion, but in rethinking how we scale supplier integrations globally.</p><p>By 2021, seven years after launching in the Netherlands, and two after entering Germany, the business proposition proved strong to continue unlocking Picnic’s untapped potential in France. It’s time to say <em>Bonjour!</em></p><h3>We’ve done this not once, but twice! Copy, paste, rinse, and repeat… or not?</h3><p>Opening a new market requires an articulate and compelling plan, mindful of many aspects of our organization such as recruitment, equipment, legislation, and technical system support. As engineers, we’re fortunate to work alongside great colleagues at Picnic, allowing us to focus on the latter. Many teams from different domains need to be involved. In<a href="https://blog.picnic.nl/picnic-10-years-2018-hello-germany-39ced9051905"> the year 4 edition</a> of this blog series, we explained the technical challenges we faced launching in Germany. You can also find details about our internal Product Information Management system and how it benefited from using object-oriented design to accommodate a new market, which we’ll revisit later.</p><p>Picnic sits at the end of the product supply chain, delivering groceries directly to customers’ doors. As part of this chain, it’s crucial for our commercial business to partner with hundreds of product suppliers. These suppliers typically operate within specific countries.</p><p>Picnic negotiates agreements with selected suppliers, who then deliver products to our warehouses. As an industry practice, the supplier and buyer communicate master data, product orders, invoices, stock information, price updates via a technical interface. Without it, processes fall back to the good old emails and excel sheets, prone to human error and security flaws.</p><p>So, to launch in a new market, we first need to:</p><ul><li>Find suppliers in the market</li><li>Collaborate with them to specify integration requirements</li><li>Track progress through regular check-ins with suppliers</li><li>Build and connect to the supplier’s API</li></ul><p>Our team already had experience with this from our prior launch in Germany. From an infrastructure point of view, our systems are single-tenant, running separately for each market. Thanks to France being our third market, <strong>we could copy the deployments of systems with configuration change, allowing the team focus on supplier API and master data concerns</strong>. A few tech teams got involved in this project, ticking off milestones one by one, leading to our successful launch in France with a partnered supplier!</p><p><em>Bonjour!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mrHZhlsDr8nNBKKf" /><figcaption><em>Picnic launches in Valenciennes, France in 2021</em></figcaption></figure><p>This is not where the blog ends, but takes off!</p><p>Amidst all the croissant and baguette emojis flying around in our celebratory Slack posts 🥖🥐, one couldn’t help but wonder: what’s next? We were live in three countries. What happens when we’re in 100? We were working with a handful of suppliers via direct API connections. What would happen when we have 1000?</p><p>Would our hacky scripts and cron jobs written in different programming languages be sustainable then? Could we maintain consistent domain terminology across markets? Could we afford slow issue resolution? What was the projection of engineering hours needed — not only to launch a new market but also to maintain 100 existing ones?</p><p>Copy, paste, rinse, and repeat could only take us so far. There’s a certain level of engineering needed to come up with a strategy to invest into development of a brand-new future-proof software architecture. <strong>As with most software, our solutions needed to evolve alongside business success</strong>.</p><p>Not long after the France launch, Picnic had doubled down on growing operations in Germany. At that time, our integration only worked in a specific geographical area — the Rhein-Ruhr region. Expanding to the rest of Germany meant scaling our integrations. So, those earlier questions about sustainability and scalability had just become more relevant than prophetic.</p><p>Thankfully, decision-making informed by current business priorities, future goals, and technical excellence allowed us to find brilliant answers. This enabled us to mobilize towards innovative solutions.</p><h3>Master Data Integration Challenges</h3><p>We recognized the value of modelling our internal product master data early on at Picnic. The data resides within our<a href="https://blog.picnic.nl/the-art-of-master-data-management-at-picnic-48b5cf978221"> MDM system, aka PIM (Product Information Management)</a>. Using an object-oriented design, over time we developed a consistent model for all master data components required by our operations and software.</p><p>The data that fuels our business can be summarized in three parts:</p><ul><li>Information about the <strong>consumer product</strong> itself. Users want to see ingredients, allergens, country of origin for products they inspect in the app.</li><li>Information about the <strong>product supplier</strong>. The commercial team needs to know availability and pricing.</li><li>Information about <strong>supply chain logistics</strong>. This covers packaging forms — crates, pallets — and how many consumer products Picnic receives when ordering these forms.</li></ul><p>This robust model allowed us to <strong>tap into a new supplier’s master data model and convert it into our internal format</strong>. This is the principle behind our master data integrations. Their role in our software landscape is to connect to supplier APIs, transform data into the canonical model and ingest the information into the internal PIM API in a country-agnostic format.</p><p>However, certain problems in our master data solutions hindered the addition of new integrations. Some issues were known at the time; others discovered later:</p><ul><li><strong>Scattered implementations:</strong> Legacy and shortcut solutions were written in various programming languages, making it difficult to operate and consolidate them.</li><li><strong>No central ownership:</strong> Different teams owned separate integrations for different countries and suppliers.</li><li><strong>Long lead time for every new supplier integration</strong>: Every new supplier required a carefully tailored approach to integrate the master data into Picnic systems.</li><li><strong>Shadow IT:</strong> As this was before our<a href="https://blog.picnic.nl/turning-shadow-it-into-sunshine-it-262e5e17e323"> internal edge systems platform</a> came into existence, some solutions didn’t use platform tooling and didn’t meet the quality standards raised back then. No observability, no alerting, no clear process.</li><li><strong>Inability to meet certain supplier requirements:</strong> Advanced features like pub/sub interfaces, caching older data, early reaction to faulty data, etc., couldn’t be supported by quick scripts.</li><li><strong>Inability to meet internal requirements:</strong> Our data warehouse lacked raw supplier master data analytics, blocking data-driven decision-making.</li></ul><p>We were confident we weren’t suffering from “not-invented-here” syndrome. It was an informed decision to tailor our solution to our own and our suppliers’ needs, giving us full control over the pipeline.</p><p>This is where I personally come into the picture. Thanks to amazing<a href="https://blog.picnic.nl/a-decade-of-scaling-real-time-analytics-and-master-data-at-picnic-ae672f21ee0f"> domain leadership</a> that took responsibility for large-scale plans, elevated personal and business ambitions, extended help whenever needed, brought clarity, and acted with optimism and humility, I was able to step into technical leadership at Picnic. Transitioning from software engineer, I was excited and determined to make this case a success.</p><p>We mobilized a team of Python engineers, blending internal developers from various teams with new hires. Our task was clear: we were the suppliers’ master data gateway into PIM. Our goal was to deliver secure, robust, scalable, and country-agnostic master data integration software, featuring traceable and observable data pipelines.</p><p>By adopting a shift-left strategy — investing in master data integrations early in our software landscape — we could frontload critical checks and successful interpretation of supplier master data, reducing friction downstream in PIM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QFv0m5NA4LUTTjx1" /><figcaption><em>You know there was a very productive brainstorm when even the whiteboard doesn’t know what’s going on anymore</em></figcaption></figure><p>Revamping the entire master data landscape required a strong R&amp;D culture and led to a greenfield project. Office brainstorm meetings in front of a whiteboard became our bread and butter. We ran proofs of concept to decide the best path forward. Pros and cons of alternative routes were clearly laid out, aiding objective decision-making within the team. We knew the timeline demanded kinetic action, so we started developing early. In an iterative fashion, we paved the way one brick at a time — sometimes removing a brick only to replace it with a different color, which is what working agile entails.</p><h3>Re-architecting Master Data Integrations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*77dKPFuB5Rwt5s8i" /></figure><p>At the time, functional requirements were fairly well understood and can be summarized as follows:</p><p>Data transport protocols:</p><ul><li>Suppliers use varying APIs, e.g. HTTP, pub/sub, SFTP.</li><li>Data can be pushed from suppliers on updates or pulled regularly by us.</li><li>Complete product data may come in multiple payloads, not just one.</li><li>Certain product information requires prior context.</li><li>Older updates might arrive after newer ones.</li></ul><p>Data structure:</p><ul><li>Data arrives in different raw formats, e.g. JSON, XML, CSV.</li><li>Supplier’s data schema is out of our control. Detect changes early.</li><li>Updates may be atomic and sparse, not containing all information.</li></ul><p>Data integrity:</p><ul><li>Sensitive commercial information must be encrypted.</li><li>Support for non-ASCII characters is essential. After having to deal with the German ß or ü, we found even more of them in our French data 😅</li></ul><p>Previous master data solutions provided some guidance on how to design the system. This helped us optimize the architecture early — but not prematurely. We could generalize the data model early, but not hastily. Quick proofs of concept boosted confidence, lifted spirits up and helped prevent sunk-cost fallacy by guiding informed investment decisions.</p><p>A key breakthrough was cementing <em>country-agnostic data modelling</em> at the architecture’s core. Essentially, data exists either in the supplier’s raw model or in our own country-agnostic canonical model. Since supplier representations aren’t always in an operable format, we convert them into JSON format. Then the canonical model is reformatted again before loading into Salesforce, powering our Product Information Management system. <strong>Such a supplier-agnostic canonical model enables consistency across markets, simplifies the core master data system, and protects Picnic data from supplier modelling choices.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MZTz2-egM3fYBGNs" /></figure><p>This breakthrough helped fix clear domain boundaries, each of which eventually turned into a loosely coupled microservice:</p><ul><li><strong>Raw master data consuming:</strong> Connect to supplier API.</li><li><strong>Raw master data preprocessing:</strong> Convert to JSON and optionally apply further preprocessing like caching.</li><li><strong>Master data canonicalization:</strong> Transform data into the country-agnostic canonical model.</li><li><strong>Master data materialization:</strong> Update master data records with atomic sparse updates, ensuring idempotency and that only newer updates override older ones.</li><li><strong>Master data loader:</strong> Push master data to Salesforce.</li><li><strong>Master data analytics:</strong> Propagate raw and canonical master data to the data warehouse, empowering analysts in decision-making.</li></ul><p>The system uses an event-driven design where master data flows across microservices. Each microservice is based on a shared framework, implemented in several Python packages. They adhere to AMQP and utilize RabbitMQ as the messaging broker to consume, process, and produce messages containing master data.</p><p>This divide-and-conquer approach made the task more manageable and allowed us to deliver an MVP early on. Each microservice had its own backlog with prioritized tasks. The MVP also gave us the chance to test resilience in unexpected scenarios:</p><ul><li>Are failed messages alerted on?</li><li>What happens to messages on failure?</li><li>Can messages be replayed from the archive back into the message broker queue?</li></ul><p>Months passed as we developed the system from scratch. Meanwhile, we connected our development/testing environments to suppliers’ testing environments in Germany (with a pub/sub API) and began processing test master data for additional German regions, such as North and South-West Germany, alongside Rhein-Ruhr.</p><h3>The Moment Theory Became Practice</h3><p>While the greenfield project progressed as planned and timelines to launch Picnic in new German territories seemed feasible, a sudden new business priority emerged: integrate <strong>a new supplier in France within 6 weeks</strong>. The new system wasn’t fully built yet. We had to decide whether to build this supplier integration in the legacy way or the new system.</p><p>Truthfully, having such a “big project” with a short timeline might feel daunting. However, the team’s sentiment was <strong>the opposite</strong>. This was our moment to reap the benefits of the new architecture and make a business impact — starting in France, rather than Germany as planned! We planned sprints ahead, shifted focus to the new French supplier, and said <em>Bonjour</em> once again 🥖🥐!</p><p>Thanks to most of the system operating on canonical data, we only needed to:</p><ul><li>Create a new Master Data Consumer service to collect data from the new supplier’s SFTP server.</li><li>Convert the raw CSV (including French non-ASCII characters) to JSON.</li><li>Transform only the minimally required data to canonical form.</li></ul><p>This meant most functionalities came out of the box, <strong>dramatically reducing lead time</strong> for the new supplier integration.</p><p>Together with other teams redesigning their supplier integration solutions and tackling one of the biggest bottlenecks in master data ingestion, we succeeded — delivering the French supplier integration two days before the deadline. Adding master data from a new supplier into PIM and extending purchase ordering and invoicing to cover that new supplier is a complex task. Yet, users could enjoy a quality assortment, high availability, and completeness in their orders without major interruption.</p><p>The following year, we wrapped up the integration of a new German supplier and pushed the big red button to launch the new system in Germany as well. We consolidated all active legacy integrations into the new system and eventually retired all legacy solutions. Today, all master data ingestion at Picnic flows through a single system.</p><h3>Conclusion</h3><p>The year 2021 marked Picnic’s launch in France and the birth of a new idea that grew into a dedicated system and team overseeing master data integrations.</p><p>In our CTO’s<a href="https://blog.picnic.nl/10-lessons-from-10-years-of-innovation-and-engineering-at-picnic-870ef4730955"> blog post</a> kicking off this series, he mentions embracing the <em>“slow down to speed up”</em> principle. This project is a perfect example where adhering to that principle paid off. Investing in a greenfield project gave us full control of supplier integrations by centralizing and consolidating them in a single framework — significantly reducing lead times for every new supplier integration.</p><p>After leading this team and managing the system for 2.5 years, I transitioned into a new leadership role focused on warehouse systems. My new team now benefits from the high-quality data provided by the master data integration system! This work wouldn’t have been possible without the team’s relentless curiosity, resilience under pressure, and cross-domain collaboration.</p><p>Although this blog post concludes here, it’s comforting to know that Picnic’s master data integration platform is processing important updates even as you read this!</p><p><strong>TL;DR</strong></p><ul><li>🇫🇷 Launched in France by reusing and adapting Germany’s playbook</li><li>🔄 Rebuilt supplier integration from scratch for scale</li><li>🧱 Microservice architecture powers all supplier master data ingestion</li><li>📊 Enabled real-time, reliable, and country-agnostic data pipelines</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=694ac400bcde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/picnic-engineering/picnic-10-years-2021-expanding-into-france-and-beyond-694ac400bcde">Picnic 10 Years: 2021 — Expanding into France, and beyond</a> was originally published in <a href="https://medium.com/picnic-engineering">Picnic Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code in Unix while using Windows]]></title>
            <link>https://medium.com/picnic-engineering/code-in-unix-while-using-windows-75556abe3281?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/75556abe3281</guid>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[unix]]></category>
            <category><![CDATA[picnic]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[macos]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Thu, 18 Feb 2021 10:08:49 GMT</pubDate>
            <atom:updated>2021-02-18T10:29:20.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mOezJCEskmOa6307PUcYpQ.png" /></figure><p>You have probably wondered at some point why most developers choose Unix based operating systems such as Linux distributions or macOS over Windows. The root cause of the issue can be simply reduced to the availability of the development technologies, meaning that Windows does not and cannot offer some features that are vital to some development processes. If this is the fact, does Windows acknowledge this issue, and what do they do about this? At Picnic, as an OS-agnostic company, how do we resolve the problems arising from using multiple operating systems for development?</p><p>The differences in operating systems from both architectural and technical points of view result in countless problems in software engineering. In Picnic, we have managed to standardize the Python development environment for both Unix based Linux/macOS and Windows users. This unified working experience would not be possible if it were not for WSL, Windows Subsystem for Linux. In this article, I will dive into the Windows development environment in which various Unix based tools are installed.</p><h3>Unix and Windows</h3><p>You probably read or heard that someone considers a problem “Windows related”. It can be in the changelog of a tool, in a bug report, or as simple as someone reacting to a problem you have in your Windows machine. What does that mean? Some games are only released for Windows. Some software technologies are only available in Unix based operating systems. These situations occur because of various reasons. The underlying technologies these OS’es are built upon are some of them. One can think of them as “magical components” that make up an operating system such as the file systems, kernels, drivers, pre-built compilers, binaries, etc. Another reason is the target audience for which the organizations build their operating systems. An average Windows user can be considered as a consumer that seeks an end-user ready product. Windows hides away the underlying technology and polish their UI for this reason. On the other hand, the most popular UNIX based operating systems like Ubuntu and Arch Linux are more direct, give more freedom, and expose underlying technology.</p><p>In Picnic, we have a broad project called Edge Systems where the business analysts, who strive for the initiative, have the opportunity to develop cronjob-like services and use tools to enhance and deploy them automatically. This enables them to reach their maximum potential as analysts by increasing their technical capacity and automating repetitive tasks. There are enabler tools that we develop internally varying from templates, code linters to CICD pipelines. There are also some tools over which we don’t have full control such as <a href="https://github.com/pyenv/pyenv">pyenv</a>, <a href="https://github.com/pypa/pipenv">pipenv</a> and <a href="https://www.python.org/">Python</a> itself. All these tools are developed in Unix environments. The core problem of not being able to support everyone arises from this point because Picnic is an OS-agnostic workplace. Most analysts use Windows whereas most software engineers prefer Unix based operating systems. Due to the nature of the difference in operating systems, the tools cannot behave the same in different operating systems. This always results in problems, support tickets, questions about tools acting differently in multiple operating systems.</p><p>This is where WSL comes into play.</p><h3>The solution: WSL</h3><p>Windows Subsystem for Linux (WSL) is a technology being developed by Microsoft that lets you create a Linux VM that is perfected to work within Windows 10. While doing so, Microsoft admits the problem they have and addresses the issues mentioned above by developing WSL, essentially enabling users to use Unix based technologies. The following is from <a href="https://docs.microsoft.com/en-us/windows/wsl/faq">the official WSL FAQ</a>:</p><blockquote>This is primarily a tool for developers — especially web developers and those who work on or with open source projects. This allows those who want/need to use Bash, common Linux tools (<em>sed</em>, <em>awk</em>, etc.) and many Linux-first tools (Ruby, Python, etc.) to use their toolchain on Windows.</blockquote><p>It is worth denoting that some controversies imply the development of WSL is a way for Microsoft to <a href="https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguish">embrace, extend, and extinguish</a> Linux.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qNdPgpa75B-0608q" /><figcaption>Ubuntu WSL running on Windows</figcaption></figure><p>One can think of WSL as a daemon process always running in the background. It offers not a GUI but command-line access in which Linux commands can be run. Should someone want to use some technology only available in Linux, they can install it on WSL via this command-line interface. Moreover, it has a direct connection to the Windows 10 file system and vice versa. This direct and fast connection between Windows and WSL allows developers to build programs on the Windows side that execute commands on the WSL side and get its output.</p><p>There are two WSL major releases, WSL 1 and WSL 2. The latter offers more features and optimizations whereas the former has the edge in being more performant across file systems. One big advantage of WSL 2 that piques most developers’ interest is its built-in support for Docker. This means that if you have Docker Desktop installed on Windows, it can be available on the WSL side as well without any further effort. Read more about how Docker developers explain their WSL integration<a href="https://docs.docker.com/docker-for-windows/wsl/"> here</a>.</p><p>The lack of GUI can be interpreted as a prime problem as it makes it harder to write code. The only option seems to use command-line tools such as vi/m , nano , etc. However, it is not the case. Like mentioned before, Unix commands can be called in WSL from the Windows side. This enables IDEs to support WSL in a way that you can continue writing code via your IDE on the Windows side and run it on the WSL side. Furthermore, the Windows team is working on <a href="https://devblogs.microsoft.com/commandline/whats-new-in-the-windows-subsystem-for-linux-september-2020/">GUI support</a> for applications residing in WSL.</p><p>Now that the importance of WSL and how effective it can be in a developer’s arsenal are clear, we can explore two widely used IDEs and their support in WSL, namely PyCharm and VSCode. Let us dive into this research topic that is carried out in the scope of migration to WSL.</p><h3>Pycharm</h3><p>Pycharm is one of the most popular IDEs. With its features specifically tailored for Python development which come out of the box without any extra installations, it provides every need of a Python developer to them. Due to these builtin features, it can be slower than its alternatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*zmoLWTb5ZtCAvqQ6" /><figcaption>Pycharm WSL integration</figcaption></figure><p>The latest version of Pycharm (2020.2 as of the writing) supports WSL integration only to some extent. There are many open issues and undergoing developments. The integration does not seem to reach its maturity and it is easy to find bugs while trying it out.</p><ul><li>Pycharm allows you to select a Python interpreter that is in WSL. The interpreter is immediately used for file indexing which is relatively slow and can sometimes take approximately 10 minutes.</li><li>Pycharm requires the project files to be on the Windows side to view and edit them. For this reason, git needs to be installed on the Windows side.</li><li>To run or debug a piece of code, Pycharm generates a shell command to tell WSL to run a Python command on the files in Windows. That is possible as WSL has access to the Windows file system. However, we discovered some bugs in some of these commands such as wrong character escaping and misplacement of environment variables in the command.</li><li>The Docker GUI in Pycharm uses Docker from Windows.</li><li>At Picnic, we use some tools that expect the project files to stay in the same environment as the Python interpreter resides in. Therefore, the code repository is also cloned on the WSL side which results in a particularly hard problem to overcome. The code is replicated and the changes on both sides should be git-tracked and synchronized constantly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-9jb6eFAHaxJ66ml" /><figcaption>Running a Pycharm project with a Python interpreter from WSL</figcaption></figure><p>Moreover, Pycharm offers this integration only in their professional edition which is not free. Nonetheless, if you do not encounter bugs or work around them, it works.</p><p>To sum up, since Pycharm does not support remote development and due to its communication design with WSL, you can only do so much. Even though the development is still doable, it is definitely worth looking into its alternatives.</p><h3>VSCode</h3><p>With the increasing number of developers that adapt to work with VSCode, the IDE gains more recognition in the Python community each day. VSCode is a lightweight code editor that is aimed at no specific programming language. Extensions can be installed to equip the editor with more features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*zE-a6wH3rrP_yeCS" /></figure><p>VSCode has an interesting and efficient integration design for WSL. Probably thanks to the fact that both VSCode and WSL are being developed by Microsoft, it is no wonder that they go hand in hand.</p><ul><li>VSCode has an<a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl"> extension</a> (“Remote — WSL”) to enable WSL support. Once you run code on the WSL side, it installs a server so that the VSCode on the Windows side can communicate with its WSL counterpart. After the server is installed, the VSCode on the Windows side is only for UI and code editing purposes. It can be considered as a window into the VSCode on the WSL side. For instance, when installing further extensions such as Python or Docker, they are installed not on the Windows side but on the WSL side.</li><li>VSCode, in contrast to Pycharm, can use the file system on WSL, open a project based in WSL and edit them. <em>code .</em> command can be run in the WSL terminal to open a project in a new VSCode tab on the Windows side.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eGSotDLHo0X0F5uw" /><figcaption>Opening a VSCode editor on the Windows side from WSL</figcaption></figure><ul><li>Development is quite fast including indexing speed as the actual VSCode server resides on the WSL side.</li><li>VSCode offers a series of quality-of-life improvements that automate some processes such as automatic Python interpreter discovery and fast debugger.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CEdBj6LhJW8A6mvI" /><figcaption>Running a VSCode project with a Python interpreter from WSL</figcaption></figure><p>In conclusion, VSCode seems to master the integration with WSL thanks to its superior architecture. It would not be wrong to say the integration will bear more fruits in future as WSL improves. VSCode makes the developer feel a seamless Unix development experience in Windows! For all these reasons, we opted for VSCode in our Windows development environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*f3j3y7FRi488rd0yKf_Xag.png" /><figcaption>Comparison of Pycharm and VSCode in terms of WSL support</figcaption></figure><p>The ability to use Unix tools smoothly in Windows is the perfect solution. Once we discovered what WSL has to offer and its integration with popular IDEs, it was inevitable to continue researching further. And the research demonstrated that we can set up all the tools required in Windows!</p><h3>Python Development Environment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*LYwPX_G-m9i442Lt" /></figure><p>So far, we have talked about the problems, solutions, and trade-offs to make some tools and technologies to work in Windows to be able to develop with Python. So, what kind of tools and technologies are we using?</p><ul><li>Python: We always use the latest versions of Python!</li><li>pyenv: This is how we manage different Python distributions.</li><li>pipenv: Virtual environment, dependency management, and dependency locking.</li><li>git: The one and only source control.</li><li>Docker: The one and only containerization tool.</li><li>Picnic Tools: Custom tools that we internally develop that serve various purposes such as automation, linting, private packages, etc.</li></ul><h3>Some Learnings</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*LdYf5H9nd3X_D1ErcHDaJA.png" /></figure><p>I was prejudiced against using an IDE other than Pycharm, namely VSCode. However, when I researched VSCode to determine its WSL capabilities I got surprised with its overall abilities. Consequently, I switched from Pycharm to VSCode and have been using it since then. Be open to change and do not judge without trying.</p><p>Picnic is a scale-up where engineers are constantly working to simply make things better. They are given opportunities to research cutting-edge technologies and come up with solution proposals to existing problems even though it may lead to company-wide adoption. We have found out that WSL along with the other tools mentioned solve a lot of our problems. Do not hesitate to research and evaluate new technologies.</p><h3>Conclusion</h3><p>The added value of WSL is not measurable since it offers something new that we use to achieve the standardization of tools and unification of programming in various operating systems. Moreover, development tools like Docker, Pycharm, VSCode are constantly working to integrate their systems with it. It may transform how people think about developing in Windows. Although in Picnic we use it to set up a Python development environment, its reach can easily be extended by simply installing other tools just like one could install in Linux.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75556abe3281" width="1" height="1" alt=""><hr><p><a href="https://medium.com/picnic-engineering/code-in-unix-while-using-windows-75556abe3281">Code in Unix while using Windows</a> was originally published in <a href="https://medium.com/picnic-engineering">Picnic Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real Time Anomaly Detection with AWS]]></title>
            <link>https://medium.com/data-science/real-time-anomaly-detection-with-aws-c237db9eaa3f?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c237db9eaa3f</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Mon, 27 May 2019 11:29:55 GMT</pubDate>
            <atom:updated>2019-05-27T13:58:18.205Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The more recent data is, the more actionable therefore valuable it is. That is the underlying logic of the real time data analytics.</blockquote><p>There might be some users performing suspicious activity, or abusing your discounts. On the other hand, you may as well just observe client behavior and take action in the act. To achieve this goal, I will go through how to create a real-time data analytics app using AWS tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*ZukumHQqW8TF1wHLyQLetQ.png" /><figcaption>Picture taken from <a href="https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sqlrf-random-cut-forest.html">AWS Docs</a></figcaption></figure><p>Imagine the last thousand data points you want to observe in real time is drawn on 2D graph like above. Our app will be able to capture those <em>red </em>points which imply an outlier, an anomaly. On top of that, we also build an alerting system that is going to be triggered whenever a <em>red</em> point is present.</p><h3>Pipeline</h3><h4>Input</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/1*rVImEYDpojHzOPWs7LXIZA.jpeg" /></figure><p>The input of our app actually<em> can be any streamed data. </em>However, we need it to be actionable data in order for our anomaly detection app to make sense. Method logs in the backend, click/touch events from mobile applications or database change events will do fine due to their nature giving us the ability to observe client actions.</p><h4>Stream</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*-y41FB3oKZH4KbJz4fpedg.png" /></figure><p>How do we ingest and stream the data we have into our app ? <a href="https://aws.amazon.com/kinesis/">AWS Kinesis</a> is a very handy tool which is easy to use, easy to configure and self-managed. Frankly, Kinesis consists of 3 different services; for streaming, we will be using <a href="https://aws.amazon.com/kinesis/data-streams/">Kinesis Data Streams</a>.</p><p>In order to create a Kinesis Data Stream, you need to give it a name, and select shard count for it. Shards are processing units of Kinesis streams, as of the time I am writing this story, a shard can ingest up to 1MB/sec and 1000 records/sec, and emit up to 2MB/sec. For most basic applications, 1 shard is enough to process your input data.</p><h4>Ingestion</h4><p>Now that we have created the stream, we can ingest the live data. There are official libraries to use AWS tools, for example:</p><ul><li><strong>boto </strong>for <em>Python 2</em></li><li><strong>boto3 </strong>for <em>Python 3</em></li><li><strong>aws-sdk </strong>for <em>node.js</em></li></ul><p>We only need the name of the stream that we want to put the records, and permission to write to that stream.</p><h4>Data Analytics App</h4><p>To process the data and capture anomalies from our live data, we will use Kinesis’ another service called <a href="https://aws.amazon.com/kinesis/data-analytics/">Kinesis Data Analytics</a>. This is a very flexible tool which gives us the ability to preprocess the data before being processed by the app, then process the data with enriched SQL engine, and postprocess the data captured by the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*0i3pBB8u3coL-n0rzOkAPw.jpeg" /></figure><ul><li>Preprocessor is a AWS Lambda function that is triggered by Kinesis Data Streams, which may enrich the data or clean the data. If we want to use some other data in the app, we may enrich our input with that data. Also, we might want to clean the input by doing type casting or remove unused fields.</li><li>Anomaly Detection is the name of our Kinesis Data Analytics app. It is written with SQL which is enriched with special analytics functions. I will go further into this in the next part.</li><li>Postprocessor is a AWS Lambda function that is triggered by Kinesis Data Analytics for each of its results. You may do whatever you want to do with the result. You might attach it to an alerting system, or you might call an endpoint, or you might want to train a model with that data. The flexibility is there.</li></ul><h4>Alert</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*mnZ-03eEMoFUYSmX57k2MA.png" /></figure><p>AWS serves this tool called <a href="https://aws.amazon.com/sns/">Simple Notification Services</a> which you use to create alerting systems. In our case, we send the results from anomaly detection app to an SNS <em>topic</em>. Then every user that subscribed to that topic will be notified with a service of our selection, let’s say, an email.</p><p>As a result, our pipeline is like below. You might disconnect Lambda functions, or SNS, and replace with another service you want. This approach offers flexibility while it keeps self-management and durability thanks to AWS tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*aiOhneYOxoFIk304gSo16Q.jpeg" /></figure><h3>Data Analytics App</h3><p>Creating a Data Analytics app in Kinesis is fairly easy:</p><ul><li>We select the app engine, either SQL or Apache Flink. We will use SQL for our case.</li><li>We select a data source, either Kinesis stream or Kinesis Firehose delivery stream. We will use Kinesis stream for our case.</li><li>A lambda function can be created to preprocess the data beforehand.</li><li>Since we are building our app with SQL, our data needs to be strictly typed meaning that a schema is needed. Kinesis Data Analytics have a feature to infer the schema automatically, then you can edit the schema for your need.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYAukPE4q1RLG51_P9fG7A.png" /><figcaption>An example schema enforced data analytics app input format</figcaption></figure><ul><li>We can connect another data source called reference data to use in application. We will pass over it for now.</li><li>We can select a destination for the results among a Kinesis stream, a Kinesis Firehose delivery stream or a Lambda function. In our case, a postprocessor Lambda function to send the data to SNS to create alerts is selected.</li><li>As for the code of the app, one needs to read the <a href="https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/analytics-sql-reference.html">AWS Kinesis data analytics SQL reference documentation</a>. The SQL for data analytics app is enriched with analytical purpose functions and techniques such as streams, pumps and windows. I will slightly talk about them in the next chapters.</li></ul><h4>Overall Logic</h4><ul><li>There are <em>streams</em> like tables with data having TTL (time-to-live).</li></ul><pre>CREATE OR REPLACE STREAM &quot;DESTINATION_SQL_STREAM&quot; <br>(<br>&quot;field1&quot; DOUBLE, <br>&quot;field2&quot; INTEGER, <br>&quot;ANOMALY_SCORE&quot; DOUBLE<br>);</pre><ul><li>And there are <em>pumps</em> that enters real time data into streams.</li><li>You can access your input stream in the nameSOURCE_SQL_STREAM_001 .</li><li>The output of the code should be a stream. You need to pump the results into that stream.</li></ul><pre>CREATE OR REPLACE PUMP &quot;STREAM_PUMP&quot; <br>AS INSERT INTO &quot;DESTINATION_SQL_STREAM&quot;</pre><pre>SELECT &quot;field1&quot;, &quot;field2&quot;, ANOMALY_SCORE<br>FROM TABLE (RANDOM_CUT_FOREST(CURSOR(</pre><pre>SELECT STREAM &quot;field1&quot;, &quot;field2&quot; <br>FROM &quot;SOURCE_SQL_STREAM_001&quot;<br>WHERE &quot;field1&quot; &gt; 0 AND &quot;field2&quot; &gt; 0<br>ORDER BY STEP(&quot;TEMP_STREAM&quot;.ROWTIME BY INTERVAL &#39;10&#39; SECOND);</pre><pre>)));</pre><blockquote>The code pieces above actually constructs a whole real time data analytics app code. We do read <em>field1</em> and <em>field2</em> from the stream in windows, which I will talk about after this, and apply them to <a href="https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sqlrf-random-cut-forest.html"><em>RANDOM_CUT_FOREST</em></a> built-in SQL function that uses machine learning in the background. The model is trained incrementally and can say whether a data point is an anomaly or not after some data points are entered for the model having enough data to train. The results of the random cut forest functions are pumped into destination stream.</blockquote><h4>Windows</h4><p>Windows provide us to perform aggregation functions on our data. For example, to perform COUNT on real time data, we need to have boundaries for our data.</p><p>There are different techniques for windows, but I will only go through most used windows: tumbling and sliding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FhEgj3Bq1ByYyDlhLluVQA.png" /></figure><ul><li>Tumbling windows separates the data in batches. Boundaries for these batches are commonly selected on time basis and rarely data count basis. The underlying reason we may want to use tumbling windows is that we do not use same data point twice.</li></ul><pre>GROUP BY field1,<br>STEP(&quot;SOURCE_SQL_STREAM_001&quot;.ROWTIME BY INTERVAL &#39;60&#39; SECOND);</pre><ul><li>Sliding windows is more useful when doing time basis aggregations. There is a fixed sized window that moves to the right incrementally. You may want to use it to observe a peak point.</li></ul><pre>WINDOW W1 AS (    <br>PARTITION BY field1     <br>RANGE INTERVAL &#39;1&#39; MINUTE PRECEDING);</pre><p>You can also build custom windows or use staggering windows where a window is created whenever a new data point arrives.</p><h4>Advanced Usage</h4><p>Kinesis Data Analytics SQL engine gives you the flexibility to create more complex logic to handle harder situations.</p><ul><li>What if we only want to send the records with anomaly scores bigger than a specific value ? Or, what if that score needs to be aggregated with some other reference data ? To solve these, a second stream can be defined in the app. The results of anomaly detection algorithm should be pumped into this new temporary stream instead our destination stream. Then we have another window to make new aggregations on our result data. The outcome of this new aggregations now can be pumped into our destination stream.</li></ul><p>As you can see from the above example, there can be multiple streams with different window types.</p><h3>Conclusion</h3><p>I have gone through the flexible structure, powerful SQL engine with fulfilling streaming techniques, self-managed services of AWS. We have created an example real time data analytics application that detects anomalies in the data. We took advantage of AWS Lambda to pre and post process the data before and after hand data analytics app. We used AWS SNS to create an alert system that will notify us whenever an anomaly is present in real time.</p><h4>For interested</h4><p><a href="https://youtu.be/PvxlF3A-Res?t=1211">Zynga tech team presents their usage of Kinesis Data Analytics and how they used it to solve the complex situations.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c237db9eaa3f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/real-time-anomaly-detection-with-aws-c237db9eaa3f">Real Time Anomaly Detection with AWS</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Development Gets Easier with AWS EMR]]></title>
            <link>https://medium.com/@seleme.topuz/spark-development-gets-easier-with-aws-emr-f5fb02f545bb?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5fb02f545bb</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Tue, 14 May 2019 08:17:49 GMT</pubDate>
            <atom:updated>2019-05-14T08:17:49.428Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Before I started to get my hands dirty and learn all these I am going to mention, I researched and wanted a beginner’s guide for EMR, could not find what I am exactly trying to find. Therefore, I wanted to write this guide to help those just like me.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*Vt2EwsL3RE-po6iBS6c0Jw.png" /></figure><p>Apache Spark is a wonderful technology which should be in toolbox of every big data engineer. Apache Spark excels when it is running on top of Apache Hadoop. There is also another tool Apache Livy that lets you submit Spark jobs to your cluster from an endpoint. You develop, analyze and visualize your Spark applications through Apache Zeppelin.<em> I am not even done yet, there are tens of products related to Apache Spark somehow.</em> Each of them should be connected to another one with some clunky settings. You need to take care of your master/slave node relations, scaling, maintenance, settings, etc. Or instead of doing all these, you can just use this very cool product EMR (Elastic MapReduce) which AWS serves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WF7YzQir1WSP2VhJvoHQg.png" /></figure><p>As of version 5.23.0, EMR offers these technologies which you can deploy by only one click, and not worry about the rest.</p><p>In this story, I am going to focus on how I use EMR for both development and production level purposes.</p><h3>Development</h3><h4>Cluster Creation</h4><ul><li>First, we create the cluster: AWS EMR -&gt; Cluster -&gt; Create Cluster .</li><li>There are bunch of settings in which I am not interested for the sake of this guide.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gk6V3ISRMGPezD-K5C1ESA.png" /></figure><ul><li>You have got to select tech stack with Spark, Hadoop. YARN, Ganglia and Zeppelin in ‘<em>Software Configuration</em>’ settings.</li><li>You can either set stuff like number of worker instances or leave them be in default.</li></ul><p>Frankly, you are done. There is not much to cover in <em>‘Quick Options</em>’ when creating EMR cluster. I, personally create my cluster from ‘<em>Advanced Options</em>’ because I use Python 3 for Spark, therefore I may need several other options under<em> Advanced Options</em>.</p><ul><li>To inform Spark, while submitting new PySpark jobs, about that you want to use Python 3 instead of Python 2, you have got to set spark-env.sh . Instead, from <em>Configurations</em> tab, you can submit your configurations about Hadoop, Spark, etc. The tab has ability to read from S3. I did upload configurations.json which contains following JSON in S3.</li></ul><pre>[<br>    {<br>        &quot;Classification&quot;: &quot;spark-env&quot;,<br>        &quot;Properties&quot;: {},<br>        &quot;Configurations&quot;: [<br>            {<br>                &quot;Classification&quot;: &quot;export&quot;,<br>                &quot;Properties&quot;: {<br>                    &quot;PYSPARK_PYTHON&quot;: &quot;python36&quot;<br>                },<br>                &quot;Configurations&quot;: []<br>            }<br>        ]<br>    }<br>]</pre><ul><li>Another thing you might want to have an eye on is that the <em>Bootstrap Actions </em>under the tab<em> General Cluster Settings</em>. The script you submit will be run on every node instance that is going be up. This enables the nodes install Python 3, and the packages you need.</li></ul><blockquote>In bootstrap actions<em>, </em>I install <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">boto3</a><em> </em>which is not present on nodes for some reason whereas boto for Python 2 is there. Come on AWS, you can do better.</blockquote><ul><li>The last thing I want to mention is to set security and key pair settings as you do not want to mess up your AWS key pairs.</li></ul><h4>Dynamic Port Forwarding</h4><p>After your cluster is up and running, you should connect to a proxy in order to connect to your cluster via your browser. AWS gives you <a href="https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-proxy.html">instructions</a> to do that. Simply, you install <a href="https://chrome.google.com/webstore/detail/foxyproxy-standard/gcknhkkoolaabfmlnjonogaaifnjlfnp?hl=en">FoxyProxy Standard</a> extension to Chrome, then create a file named foxyproxy-settings.xml with the contents:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;foxyproxy&gt;<br>   &lt;proxies&gt;<br>      &lt;proxy name=&quot;emr-socks-proxy&quot; id=&quot;2322596116&quot; notes=&quot;&quot; fromSubscription=&quot;false&quot; enabled=&quot;true&quot; mode=&quot;manual&quot; selectedTabIndex=&quot;2&quot; lastresort=&quot;false&quot; animatedIcons=&quot;true&quot; includeInCycle=&quot;true&quot; color=&quot;#0055E5&quot; proxyDNS=&quot;true&quot; noInternalIPs=&quot;false&quot; autoconfMode=&quot;pac&quot; clearCacheBeforeUse=&quot;false&quot; disableCache=&quot;false&quot; clearCookiesBeforeUse=&quot;false&quot; rejectCookies=&quot;false&quot;&gt;<br>         &lt;matches&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;*ec2*.amazonaws.com*&quot; pattern=&quot;*ec2*.amazonaws.com*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot; /&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;*ec2*.compute*&quot; pattern=&quot;*ec2*.compute*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot; /&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;10.*&quot; pattern=&quot;http://10.*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot; /&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;*10*.amazonaws.com*&quot; pattern=&quot;*10*.amazonaws.com*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot; /&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;*10*.compute*&quot; pattern=&quot;*10*.compute*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot; /&gt; <br>            &lt;match enabled=&quot;true&quot; name=&quot;*.compute.internal*&quot; pattern=&quot;*.compute.internal*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot;/&gt;<br>            &lt;match enabled=&quot;true&quot; name=&quot;*.ec2.internal* &quot; pattern=&quot;*.ec2.internal*&quot; isRegEx=&quot;false&quot; isBlackList=&quot;false&quot; isMultiLine=&quot;false&quot; caseSensitive=&quot;false&quot; fromSubscription=&quot;false&quot;/&gt;	  <br>	   &lt;/matches&gt;<br>         &lt;manualconf host=&quot;localhost&quot; port=&quot;8157&quot; socksversion=&quot;5&quot; isSocks=&quot;true&quot; username=&quot;&quot; password=&quot;&quot; domain=&quot;&quot; /&gt;<br>      &lt;/proxy&gt;<br>   &lt;/proxies&gt;<br>&lt;/foxyproxy&gt;</pre><p>Then you import the file as settings to FoxyProxy in its options. Now that you are doing port forwarding, you should be able to see the following screen with the ‘<em>Connections</em>’ available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*izB_minvKom_U9ShF3LeNw.png" /></figure><h4>Coding</h4><p>Let us get to real business, coding. By clicking <em>Zeppelin</em>, you go to 8890 port of your master node in which Zeppelin web server is served.</p><p>Apache Zeppelin gives you the ability to develop your program just like Jupyter. Moreover, it submits your Spark jobs to the cluster and return the results to you on Zeppelin notebook.</p><p>While writing your code in notebook in Zeppelin, you select a ‘mode’ for the code you are going to be writing.</p><p>For example, thanks to dep mode, you can install the Maven packages or local jars before you start to use Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zN86aIcRmA_YeH9HS0C6Wg.png" /></figure><p>Now that you are done with everything you should have considered before doing some coding, I leave you the power of Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhCjxz7ZOfNqIAd6G-V91A.png" /></figure><p>The rest, you can discover on Zeppelin.</p><h4>Monitoring</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0F-31Y-flUn7NRzzPRBz4Q.png" /><figcaption>Ganglia</figcaption></figure><p>There are some predefined chart that you can see overall cluster perfomance, and KPI’s like CPU, memory and network user over <a href="http://ganglia.sourceforge.net/">Ganglia</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pttV9Etu1XvL7C6cCjky9w.png" /><figcaption>YARN</figcaption></figure><p>YARN’s web UI is available on <em>‘Resource Manager’ </em>link. Here, you can see your Hadoop jobs, their states, statuses, logs, allocations, etc.</p><h4>Production</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*F9zQPC3mnA66hKSkP3Nzag.png" /></figure><p>First things first, your EMR cluster creation was kind of painful. And I give the good news: you will not have to be again, as you can export the AWS CLI command for EMR creation.</p><p>Another thing I want to mention that you can treat both HDFS and S3 file paths as local file paths in AWS EMR. For example, when you want to run a code, the code in s3 is accessible from command line in EMR and you can just say python s3://bucket/code.py .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*NlblaiB53xtq8wdxmDKLKg.jpeg" /><figcaption>Production Use Case</figcaption></figure><p>We use AWS Lambda to trigger an EMR cluster creation whenever we need. In our case, Lambda is triggered periodically to create the cluster and ‘add steps’ to the cluster. Steps imply the ‘spark-submit’ commands we submit to EMR from AWS web interface. The cluster gets up, finish the steps and terminates itself.</p><p>There is not much to cover apart from these as I have come to the end of this guide. AWS EMR saves you time, gives you a nice UI, many shortcuts. You would have struggled with settings, configurations to integrate big data tools otherwise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5fb02f545bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time Real Estate Posting Notifier]]></title>
            <link>https://medium.com/@seleme.topuz/real-time-real-estate-posting-notifier-e2a3d2a50709?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2a3d2a50709</guid>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[life-hacking]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Mon, 13 May 2019 13:27:15 GMT</pubDate>
            <atom:updated>2019-05-13T13:27:15.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xiXj8KoPQ5O-qQPeRUCRRQ.png" /></figure><p>Ever finding yourself spending hours in front of a computer in the hope for the perfect apartment to rent ? Well you don’t have to do at as we do it for you. Ok, that was quite unnecessary and cliche. Well, here you go yet another real estate web scraper.</p><p>Just like as I said, I found my self staring at my laptop in the bed, letting my precious weekend morning to waste. As soon as I realized what I was doing, I started to plan and design what I was going to build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*vvlcSGiL2liH07vHjvTMjQ.jpeg" /></figure><p>As simple as it seems, the program took almost an hour to build. An example use cycle of the program would be:</p><ul><li>The script is ran with some arguments such as URL to scrape from, email credentials, etc.</li><li>With <a href="https://2.python-requests.org/en/master/">requests</a> library, website HTML content is loaded into memory.</li><li>The contents are processed with scraping library <a href="https://www.crummy.com/software/BeautifulSoup/">BeautifulSoup4</a>.</li><li>All real estate postings in the page are extracted out from the page and filtered by last posting that is kept in an OS file.</li><li>With <a href="https://docs.python.org/3/library/smtplib.html">smtplib</a>, newer postings are emailed to your choosing of email.</li></ul><p>The code is available in my <a href="https://github.com/slmtpz/real-estate-scraper">GitHub repository</a>, although it will not give you a lot of options for email services or specific webpages to scrape from, it is usable for your gmail account and sahibinden.com.</p><p>Well there is this piece of code that I use on my many mini-projects in order to mail if any is interested:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/277b08ef4abbef4498bafcca840898a2/href">https://medium.com/media/277b08ef4abbef4498bafcca840898a2/href</a></iframe><p>Before using the code, you need to set the security settings for ‘Less secure app access’ on in Gmail from <a href="https://myaccount.google.com/lesssecureapps?utm_source=google-account&amp;utm_medium=web">here</a>. The setting will let the script be able to login to your Gmail account. Gmail prevents it in order to block malicious stuff, as I recommend change the setting back to off after you are done with the script.</p><p>The program is designed to be run periodically so that it can notify you as soon as possible. It is not quite competitive in rentals but you don’t wanna miss an excellent opportunity. After all, time is everything. I recommend using the old, traditional way of crontab. These are the steps documenting how you deploy the program:</p><ul><li>Open an instance, e.g <em>AWS EC2</em>, <em>Google Computing Engine</em>.</li><li>Connect through <em>SSH</em>.</li><li>Be a nice guy and create virtual environment.</li><li>Install <em>Git</em> to clone the repository. Install <em>Python 3</em> and install packages that the repository needs, in <em>requirements.txt</em>, with <em>pip</em>.</li><li>Edit <em>crontab</em> with crontab -e. */10 * * * * indicates your script is ran every 10 minutes. Save the crontab and you are done.</li></ul><p>Hopefully, you get the results in your mailbox. If you aren’t, then either you did something wrong, or there is no result for your URL.</p><p>Here is how you will see the results in your mail. You might guess what each row imply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujXGyW1GHpBK47jCtST8cA.png" /><figcaption>URL, type, description, m2, rooms, rent amount, time, location</figcaption></figure><p>Another problem is automated, now I’m getting feed of the newer postings of rentals in my area. Frankly, I do not recommend any code-writers to use this project or any other life-hack projects as I think you should get your hand dirty and do it yourself. These little works can teach you a lot of stuff.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2a3d2a50709" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Depths of Big Data Platform at Getir: Streaming]]></title>
            <link>https://medium.com/getir/depths-of-big-data-platform-at-getir-streaming-4138eb5611f8?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4138eb5611f8</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Mon, 13 May 2019 07:07:58 GMT</pubDate>
            <atom:updated>2019-05-14T10:48:39.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*2VwP9MpAykjj-WL5DLeOJg.jpeg" /></figure><p>The project reflects MongoDB changes on Redshift in the big data platform in near real-time.</p><h3>Phases</h3><h4>1) Data Ingestion</h4><ul><li>MongoDB, after version 3.6, released a future called <a href="https://docs.mongodb.com/manual/changeStreams/">change streams</a> enabling you to acquire all the changes that happens in a collection, database or cluster. You can imagine this as the modern version of tailing <a href="https://docs.mongodb.com/manual/core/replica-set-oplog/">oplogs</a> from MongoDB. The <a href="https://docs.mongodb.com/manual/reference/change-events/index.html">change events</a> can be streamed through the <a href="https://docs.mongodb.com/manual/reference/method/db.watch/index.html">watch</a> function that is available on MongoDB client.</li><li>A producer actually consumes Mongo’s change streams and produce entries for AWS Kinesis. Now that change events are in Kinesis right now, one can write a consumer to read those changes. We have consumer triggering a AWS Lambda function to save those change events to AWS S3.</li><li>Our data ingestion pipeline can both vertically and horizontally scale to keep up with the changes.</li></ul><h4>2) Data Transformation</h4><ul><li>Periodically a streaming job runs and collects data in s3 in minibatches.</li><li>Data in S3 is in JSON format which should be converted to Redshift SQL format.</li><li>Some metadata like schemas and table names had been written by ETL job. These metadata is used by streaming job to shape the JSON data into SQL.</li></ul><h4>3) Load</h4><ul><li>Data in minibatches being very small, can be written to Redshift in a short time period which indicates the whole process is said to be close real time.</li></ul><h3>In the pipeline</h3><ul><li>After change events are being written to S3, the streaming job gets those change events, using a forced-schema model to transform those to Redshift-acceptable Spark SQL format. Then it updates Redshift with the data.</li><li>Even though Hadoop’s big data support is not required in this step, Spark is needed and Hadoop YARN’s Web UI gives us a nice job-overlooking system. We can see which jobs have failed and why.</li></ul><h3>Transformation of the data</h3><ul><li>There is an advantage in making streaming application in the form of minibatches. Say, the streaming job runs every 3 minutes. A document in MongoDB can be update many times in that period of time. Instead of updating the corresponding row in Redshift many times, I simply ignored the former ones and used only the latter change event which gave me the last updated version of the document.</li><li>My other approach is that I put the change events of ‘insert’, ‘replace’ and ‘update’ types in the same bucket whereas I put ‘delete’ events are in another bucket. That is because all ‘insert’, ‘replace’ and ‘update’ events gives me the whole document, and should be considered the same for data ingestion.</li><li>Another notable point that I discovered was the difference of BSON types. In ETL, <a href="https://docs.mongodb.com/spark-connector/master/">mongo-spark connector</a> converted MongoDB’s BSON typed objects into SparkSQL representatives by using a mapping in Scala. There are a lot of differences in streaming case because the change events are in JSON format, and the BSON types are converted to a special JSON alternative. For example ‘Date’ BSON typed objects are converted to a JSON like {“$date”: &lt;timestamp&gt;}. In order to overcome this problem, I had to write a recursive function to iterate through all data schemas and finds these BSON type conversion and updates them with what we want.</li></ul><h3>Streaming Benchmark</h3><p>During streaming, we do bunch of different transformations than ETL. That naturally may cause data to be altered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*eiC525zAfB10HxtvINCcEw.jpeg" /></figure><p>Streaming benchmark, written with Python 3 just like many other parts of Big Data Platform, prepares a set of ‘id’s that are ensured to be the documents which had been moved to Redshift in both ways of ETL and streaming so that we could not miss streaming documents as they may be potentially smaller in size compare to those moved with ETL.</p><p>We have set up performance criteria that we consider in benchmark results. Those are computed after each MongoDB and Redshift sourced data are transformed and being prepared for the comparison part.</p><p>The streaming benchmark should also be run in an environment with Spark. However, Hadoop is not required in this case due to the small size of data that are compared. And, we do not require any distributed computing along the way. Spark dependency is due to the MongoDB transformer which requires Spark SQL engine to extract DataFrames out of MongoDB data.</p><h3>Recovery System</h3><p>During development of the streaming, I discovered this annoying problem where the type of a field changes, and the type of streamed field does not follow the schema generated by ETL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*HKK-fOiQsJ1N187KinWu4Q.jpeg" /></figure><p>This problem can be overcome by running ETL again since it would infer the schema types over again, and resolve it with the correct field type.</p><p>For example, ETL can label the type of a field as ‘integer’. However, MongoDB is a NoSQL database, which allows double typed values in that field. When the double typed value is present in the streaming job, the job fails during transforming process.</p><p>The recovery system automatically detects if a streaming job fails, and runs ETL job, then starts rerunning streaming jobs. This may not be a perfect solution, and the data will not be updated in longer time.</p><p>Streaming is the modern way to moving data in near real-time, compared to mass ETL. However, there are still many problems emerging from the underlying understanding of streaming the data. One may state that streaming is not as secure and guaranteed as the ETL. In my opinion, in contrast, the risk can be reduced and the overhead is worth it as you can have your data in near real-time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4138eb5611f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/getir/depths-of-big-data-platform-at-getir-streaming-4138eb5611f8">Depths of Big Data Platform at Getir: Streaming</a> was originally published in <a href="https://medium.com/getir">Getir</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Depths of Big Data Platform at Getir: ETL]]></title>
            <link>https://medium.com/@seleme.topuz/depths-of-big-data-platform-at-getir-etl-1227d82a6087?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1227d82a6087</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Mon, 13 May 2019 07:01:46 GMT</pubDate>
            <atom:updated>2019-05-13T07:38:45.037Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*8s8UDbjVssgfWrX--_Z5hA.jpeg" /></figure><p>The project is the generic implementation of Mongo-to-Redshift ETL in our big data pipeline. Inferring automatically from NoSQL format to SQL format and leveraging the power of PySpark, whole databases can be moved to Redshift in minutes.</p><h3>Phases</h3><h4>1) Extract</h4><ul><li>Using <a href="https://www.mongodb.com/products/spark-connector">Mongo-Spark connector</a>, internally using workers to query, aggregate and infer data, we get the data to memory.</li><li>The connector uses a sample of the data to infer the Spark SQL schema of each collection.</li><li>These schemas are uploaded to S3 to be used later during streaming application.</li></ul><h4>2 ) Transform</h4><ul><li>The schemas are in Spark SQL format. AWS Redshift does not accept some data types in Spark SQL like ArrayType or NullType.</li><li>The nested structure of NoSQL is converted to SQL format by flattening the keys of each leaf in nested structure.</li><li>The array type fields are moved out by creating a new table for each of them so that no data will be lost.</li><li>All these are done by recursive functions, hence the application does not require a mapping.</li><li>Table names created in these steps are uploaded to S3 to be used later during streaming application.</li></ul><h4>3) Load</h4><ul><li>Using <a href="https://github.com/databricks/spark-redshift">Spark-Redshift connector</a>, we load the data to Redshift.</li><li>The way the connector does this is through S3.</li><li>The data first is loaded to S3 in a temporary bucket.</li><li>Then COPY command is run in AWS Redshift to get the data from S3 into Redshift.</li></ul><h3>In the pipeline</h3><ul><li>ETL job, queries bulk data, transform it and load it to somewhere. This is a heavy job and take potentially hours for big databases. This is intended to be a reset function to our data platform. While streaming, if streaming fails, ETL should be rerun to clear the corrupt data.</li><li>Mongo-Redshift ETL can be triggered through an AWS Lambda function, creating new EMR cluster, adding the ETL job as a step to EMR. After EMR runs ETL, it will terminate.</li><li>This is a database-level implementation of ETL. For each database, we should run 1 EMR job.</li></ul><h3>Transformation of the data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uaMFzz_xl4P051NP5065Ww.png" /><figcaption>How data is being stored in MongoDB</figcaption></figure><p>A JSON object can be saved in MongoDB just like as it is. MongoDB dynamically set the BSON types of every field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLJ5iHBTME1HSK2AHYtQJQ.png" /><figcaption>Mongo Spark Connector’s mapping</figcaption></figure><p><a href="https://www.mongodb.com/products/spark-connector">Mongo-Spark connector</a> uses a mapping from these BSON types to Spark SQL types. It does do the conversion in all worker nodes in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*r9ustLGGMJxnzQ4BvwJKwg.png" /><figcaption>A SparkSQL schema</figcaption></figure><p>However, the problem here occurs during loading to Redshift since Redshift does not support some field types such as NullType or ArrayType in Spark SQL. The pain does not end here as Redshift do not support nested structure.</p><p>So, to achieve converting the data from NoSQL to Redshift’s SQL format, we first converted the data to Spark SQL format. Then the data flows through a <strong>generic and recursive flattening algorithm</strong> before going into loading phase.</p><pre>func flatten(prefix):<br>    for each field in the schema =&gt;<br>       if type of the field is SparkSQL.StructType =&gt;<br>            call flatten(prefix + path of the field)<br>       if type of the field is SparkSQL.ArrayType =&gt;<br>            store the array fields path<br>       if this is the leaf =&gt;<br>            store the leaf fields path</pre><p>The algorithm iterates through SparkSQL schema, and finds the leaf nodes. If an ArrayType is confronted, do nothing and store that field. The whole point is convert data in type</p><p>{“field1”: {“field2”: 5}}</p><p>to something like</p><p>{“field1.field2”: 5}</p><p>while not losing the array fields by keeping them. What happens to array fields is now that they go through another generic and recursive function <strong>in order to create a new whole table for each of them</strong>.</p><pre>func process array type:<br>    new data = data [path of the field with array type]<br>    exploded data = change each column of the array data to rows<br>    leaf fields, array fields = flatten (exploded data)<br>    result table = exploded data [leaf fields]<br>    process array type (each array field)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*h66rVab9CL9ZIgHOlMJrjw.png" /></figure><p>What the algorithm does is to convert array typed fields into a new whole table with <em>arrays elements are represented as rows</em>. We keep positions in the array as another column in order not to lose any semantics from original data. For example data in a shape of</p><p>{“arrayfield1”: [{“field2”: 5}, {“field2”: 6}]}</p><p>is converted to</p><p>{“arrayfield1.field2”: 5, “arrayfield1.field2.position”: 0}</p><p>{“arrayfield1.field2”: 6, “arrayfield1.field2.position”: 1}</p><p>As you can see the each array fields own data, is ‘exploded’ into rows instead of elements, then it is being treated as a new whole input data by flattening. This generic and recursive approach enables us to convert ‘any’ shaped data into some collection of tables which shall keep all the semantics of the original data.</p><p>Along with these changes, table names are also designed in a way that it gives you information about which fields represents array fields. Imagine a structure like:</p><pre><em>{t: {x: {y: [{z: {m: [1,2,3]}}, {z: {m: [4,5,6]}}]}}}</em></pre><p>The table names derived from this structure are: t, t_x__y, t_x__y_z__m which indicate that by only looking at the last table name we can see ‘y’ and ‘m’ fields represents array fields, therefore stores a ‘position’ field inside those tables with names ‘t_x__y_pos’ and ‘t_x__y_z__m_pos’ respectively. Hence, we do not lose any semantics about the exact location in JSON schema any data is from.</p><p>During the development of the ETL, I embraced more of heuristic approaches that turned our to be perfectly analytical solution that caused no loss in semantics of the data. We potentially may have thousands of tables in Redshift after performing ETL on all our MongoDB databases. The project, Mongo-to-Redshift ETL, should be constantly worked on and gotten improvements until it becomes error-prune and more scalable as the data size goes huge. As much as it sounds to be the traditional way of moving data, ETL is still a secure and guaranteed way to move the data. That’s why we have robustness systems that do run ETL whenever streaming fails.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1227d82a6087" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Presenting: Big Data Platform at Getir]]></title>
            <link>https://medium.com/@seleme.topuz/presenting-big-data-platform-at-getir-68503981d8d7?source=rss-4bc5eeb71e6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/68503981d8d7</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Seleme Topuz]]></dc:creator>
            <pubDate>Mon, 13 May 2019 06:56:38 GMT</pubDate>
            <atom:updated>2019-05-13T06:56:38.781Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4K_2B2QXi-HkR1BixFnEw.png" /></figure><p>Getir is an on-demand delivery app that can get thousands of market items and restaurant foods in minutes. We are growing every day, naturally so our data. To keep up with the data size, we needed our scalable data platform. We needed to start building our this big data platform at some point, and this is how we are doing it.</p><blockquote>For those who are asking ‘What is this big data thing anyways?’, big data platform enables analytics team to access data in shorter time by simplifying querying the data. It enables our system to store much more data for analytics purposes. It will present a code layer for data engineers to clean the data before processing it. It will present a cluster computing engine for us to use in an easy way on all of our data. It will collect all the data from different places into one place.</blockquote><h3>The Problem</h3><p>As size of data increases day by day, it gets harder to maintain speed of querying databases which is not meant for analytical purposes. We face this exact problem at Getir. We use MongoDB for backend data storage. Our analytics team often complain about its slow speed as they see their scheduled tasks getting slower. Another downside is that MongoDB is NoSQL yet most data analysts get used to tinker with SQL oriented databases.</p><blockquote>So yes, we are talking about querying MongoDB for analytical purposes, and writing custom ETL pipelines as we go.</blockquote><p>Getir did not have the dire need for data engineers as you can see our data pipeline is primal, and the data analysts are building their our ETL pipelines for the data they require. This may quite likely arise problems as we scale, potentially endangering whole system. I approach this problem and find a solution by creating a new whole data platform that any person can use for analytics.</p><h3>The Approach</h3><p>While designing the platform, I worked closely with our DevOps team, kudos to Ufuk Dag, who helped me dearly envisioning the future use of this platform. As an R&amp;D engineer, it is my job to evaluate varying technologies and select a tech stack for our need. Hence, I constantly try different technologies and weigh their pros and cons in the architecture. By building demo-like prototypes which gives the desired outcome, we can see how a technology such as libraries, frameworks, services, providers, techniques are performing. For example, I tried to get stream data directly from Kinesis into Spark which I did not choose over doing it with mini-batches, as it had been resulting in many bugs, feeling the libraries quite underdeveloped.</p><blockquote>While choosing a technology, we try to select an AWS-alternative service when available since we, as Getir, are quite fond of AWS overall.</blockquote><p>One of main criteria while choosing a technology is to see if it can scale both horizontally and vertically inside our stack. Where we cannot use an AWS service, I especially focus on this since most AWS services are designed in this way.</p><p>The big data platform consists of two main and two supporting projects:</p><ol><li>An <strong>ETL pipeline </strong>that can move entire databases to a warehouse as many companies also did with their data platform. However and moreover, we have a requirement for supporting big data implying that we need a distributed system that can hold up to petabytes of data and make transformation processes over it, and it also needs to be done in shorter time.</li><li>A <strong>streaming application</strong> that will reflect every change in our core database on our data warehouse. The design here is a much harder issue as you have to research on how you can do it since the most companies are not here yet.</li><li>A <strong>streaming benchmark</strong> developed in order to test performance of our streaming application by comparing data in both data source and sink.</li><li><strong>Recovery system </strong>which can react to failures in the pipeline by fixing automatically to ensure robustness of the whole pipeline.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OS2xcgj1s3uDDQyRrUQ68Q.jpeg" /><figcaption><strong>Notable technologies we use for Big Data Platform</strong></figcaption></figure><p>Our end-game goal is to keep all the data in<strong> AWS Redshift</strong> knowing that it is a PostgreSQL structured data warehouse with AWS backing up and maintaining. It can scale up, return answers to analytical queries in much shorter time for much bigger data.</p><p>Next, we have our distributed system provider, <strong>Apache Hadoop </strong>as I think this tech is the only tech I could not give up as it is the backbone of all of big data stuff. Hadoop lets you work on clusters of computers, storing data and computer in all of them for which it gives you a single entrance.</p><p><strong>Apache Spark </strong>is another powerful tech here that I chose early as I knew it it is a modern, cutting edge in-memory cluster computing tool that supports working top of Hadoop. <strong>Spark SQL </strong>is an SQL engine that Spark offers, and we use it to transform data.</p><p>When using Spark and Hadoop at the same time, Spark needs a resource management system that will allocate resources from cluster for Spark, or submit its jobs on Hadoop, or gives you UI to access all Spark jobs you run on the cluster. I go with Hadoop’s default resource negotiator: <strong>YARN</strong>.</p><blockquote>The good thing is, AWS, of course, offers a service called <strong>AWS EMR</strong> that contains Hadoop, Spark, any many other big data tools.</blockquote><p><strong>AWS S3</strong> is used to keep metadata, and put some output files. It is also used, while sinking data into Redshift, as a temporary storage.</p><p><strong>MongoDB </strong>stores the data we want to move from.</p><p><strong>AWS Kinesis</strong> is the heart of our streaming pipeline. It enables us to ingest the change data from MongoDB to our big data platform in real time as well as it can support multiple consumers which any programmer can build to read from change events.</p><p>To trigger creation of EMR clusters and jobs to run in it, we use <strong>AWS Lambda.</strong></p><p>While selecting programming language, I had Scala first in mind since I knew it is a popular and common language in big data world. However, due to my extensive knowledge and its developer-friendly and general purpose structure, <strong>Python 3 </strong>was the way to go. However, we left the option to use Scala whenever we needed it as we also knew, Python in some big data frameworks was not supported. Though, we never used it however it would be a better choice in many places.</p><p>Other misc tools are:</p><ul><li><strong>Apache Zeppelin, </strong>is a notebook provider, similar to Jupyter, but differentiated by its availability to support submitting Spark jobs on cluster from notebook UI. This is used as a development tool.</li><li><strong>Ganglia</strong>, is a monitoring tool that we are using to see cluster performance KPIs. Frankly, I used it just because EMR offered it.</li></ul><h3>Using Big Data Platform</h3><p>There are different use cases for interacting with the big data platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*7Re78LzI644eVvYThr-1Fw.jpeg" /></figure><ul><li>Data analysts may use <strong>Jupyter</strong> to connect to Redshift and make their analysis through programming languages Jupyter offer such as Python, R. They can us <strong>DataGrip</strong> to dive into the data, and make deductions out of it.</li><li>Data scientists may use <strong>PyCharm</strong> to build machine learning applications accessing and querying Redshift.</li><li>Big data engineers should interacts with <strong>Hadoop YARN</strong>’s web UI to track Spark jobs, or <strong>Ganglia</strong> to see cluster performance. They may use <strong>Zeppelin</strong> to build new jobs and submit to Hadoop cluster.</li><li>DevOps engineers may use to strengthen the whole pipeline by building alert or robustness systems using <strong>AWS Lambda </strong>and many other AWS services.</li></ul><h3>Conclusion</h3><p>Now that we have the big data platform that can give fast data access to data team or BI tools, this work and Getir have a lot more future works on its way. By streaming the live data into Redshift in near real-time, I think someday, we might serve our all internal and many external services using the data Big Data Platform offering.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68503981d8d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>