<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Elyadata on Medium]]></title>
        <description><![CDATA[Stories by Elyadata on Medium]]></description>
        <link>https://medium.com/@elyadata?source=rss-48d50203581------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ApyWdV8texFELzG2-yLKyQ.png</url>
            <title>Stories by Elyadata on Medium</title>
            <link>https://medium.com/@elyadata?source=rss-48d50203581------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:39:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@elyadata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Flat and Nested Arabic Named Entity Recognition]]></title>
            <link>https://insights.elyadata.com/flat-and-nested-arabic-named-entity-recognition-2de66c1fd583?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/2de66c1fd583</guid>
            <category><![CDATA[linguistics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 12:01:25 GMT</pubDate>
            <atom:updated>2024-01-09T12:25:28.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTBvPn0LBJlYgycFxOlnkg.png" /></figure><p>Recently, <strong>Elyadata</strong> [1] participated to the <strong>Wojood</strong> Arabic Named Entity Recognition (NER) Shared Task. <strong>Wojood</strong> was organized as part of the first ArabicNLP conference (link). The shared task targets both flat and nested Arabic NER.</p><p>Below we describes <strong>Elyadata </strong>submissions that achieved <strong>first</strong> and <strong>third</strong> place respectively for nested and flat NER!</p><h3>Wojood dataset</h3><p>Wojood [2] is a dataset for Arabic NER. It contains 16817, 3133 and 5990 sentences in both Modern Standard Arabic (MSA) and dialectal Arabic for train, development, and test, respectively. It is manually annotated and consists of 21 entity type and 75k entity mentions with nested entities constituting 22.5% of the total.</p><h3>Flat versus Nested NER</h3><p>NER is the task of identifying entities such as currencies, dates and time, people, names, locations and geo-political entities in text.</p><p>Named entities can be subdivided into two types: flat and nested named entities. Flat entities are linear spans of contiguous tokens that do not overlap with each other.</p><p>In many practical applications (i.e. biomedical data), an entity could contain other entities or be a part of other entities. As in<em> Bank of America</em> and <em>University of Washington</em>, both organizations contains nested locations (<em>America</em> and <em>Washington)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grmZ02mrj5J8XD2vnScLUA.gif" /><figcaption>Example of Nested entities in Arabic sentence</figcaption></figure><h3>Experiments and Results</h3><p>We experimented using both data- and model-centric approaches. Data-centric involving dataset re-sampling and pre-processing, while model-centric approach employing various neural network model architectures.</p><h4>• Data-Centric approche</h4><p>We used four re-sampling strategies in order to mitigate the inherent unbalance of the NER dataset due to the over-representation of the outside &lt;O&gt; token.</p><h4>• Model-Centric approches</h4><p>As regards the model-centric approches, we fine-tuned and evaluated two state-of-art nested and flat models: <em>PIQN</em> and <em>DiffusionNER</em>. Better results were obtained using the <em>DiffusionNER </em>architecture<em> </em>presented below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tES5CDJqca16_3oh0GGkQA.png" /><figcaption><em>DiffusionNER model </em>architecture</figcaption></figure><p>AS presented in the figure above, <em>DiffusionNER </em>is based on a Denoising Diffusion Implicit Models. The architecture is composed of two main blocks: the Sentence Encoder and an Entity Decoder. <em>DiffusionNER</em> samples <strong>K</strong> noisy spans at each time-step then performs denoising by updating the spans boundaries and classification.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YSMt2aSCy_aMqKwldd72g.png" /><figcaption>Elyadata results for Wojood Shared Task.</figcaption></figure><p>Our systems achieved top performance in nested NER (93.93% F1-score) and ranks third in flat NER (91.92% F1-score) among participants.</p><h3><strong>Code and models</strong></h3><p>All our the results are reproducible using the code available here: <a href="https://github.com/elyadata/NER_shared_task_2023">https://github.com/elyadata/NER_shared_task_2023</a></p><h3>References</h3><p>[1] <a href="https://dblp.org/pid/362/4747.html">Imen Laouirine</a>, <a href="https://dblp.org/pid/351/5876.html">Haroun Elleuch</a>, <a href="https://dblp.org/pid/75/9232.html">Fethi Bougares</a>. <strong>ELYADATA at WojoodNER Shared Task: Data and Model-centric Approaches for Arabic Flat and Nested NER.</strong> ArabicNLP 2023.</p><p>[2] Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic named entity corpus and recognition using BERT. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636, Marseille, France. European Language Resources Association.</p><p>[3] Xiaochen Wang and Yue Wang. 2022. Sentence-level resampling for named entity recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2151–2165, Seattle, United States. Association for Computational Linguistics.</p><p>[4] Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yueting Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 947–961, Dublin, Ireland. Association for Computational Linguistics.</p><p>[5] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. DiffusionNER: Boundary diffusion for named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3875–3890, Toronto, Canada. Association for Computational Linguistics.</p><p>[6] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.</p><p>[7] Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2de66c1fd583" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/flat-and-nested-arabic-named-entity-recognition-2de66c1fd583">Flat and Nested Arabic Named Entity Recognition</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Architecture: Bridging the Gap Between IT and Data Teams]]></title>
            <link>https://insights.elyadata.com/data-architecture-bridging-the-gap-between-it-and-data-teams-d49d93d021f?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/d49d93d021f</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Wed, 22 Nov 2023 10:53:35 GMT</pubDate>
            <atom:updated>2023-11-22T10:53:35.374Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Part 0: Introduction to Data architect job</strong></h4><p><strong>Disclaimer:</strong> This article is the first in a series of articles centered on data architecture, encompassing both theoretical concepts and practical case studies, inspired by <a href="http://www.elyadata.com/"><strong>Elyadata</strong></a> data architecture experience. Our insights are results of hands-on experience assisting numerous large and medium-sized organizations in constructing and industrializing data products, and facilitating synergy between IT and data experts.</p><p><strong>General Introduction</strong></p><p>The current market is observing a massive demand for data architecture expertise, underscoring the ever-growing importance of the data architect’s role. Over the past few years, organizations have focused their efforts on constructing data products, specially AI-based models and applications for business usage. However, despite these efforts, many of these projects have failed to reach the production phase. The primary reason for this situation seems to be a lack of understanding and effective communication between IT professionals and data experts within the organizations, this is not the only reason.</p><p>Typically, IT departments have well-defined processes for tasks like application development and deployment, data access, and environment creation. Frequently, data experts were not fully aware about these procedures, cultural norms, and work methodologies, leading to conflicts, impasses, and wasted effort. Such misunderstandings have only amplified the need for data architecture expertise.</p><p>In this article, we will explore the role of data architects and highlight the essential skills they require.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDzZx2jR2cFeWu0GJfoaaw.png" /></figure><h3><strong>The Role of a Data Architect</strong></h3><p>In recent times, we’ve noticed that organizations are starting to realize the significance of data architects in their ongoing data projects. However, it’s interesting to note that they are often brought in at later stages rather than during the initial phases of data initiatives. Interestingly, this is not surprising. In fact, during the early phases of the data hype, many organizations held the optimistic belief that building a highly skilled team of data scientists would be sufficient to ideate, develop, and operationalize data products. However, they tend to overlook that data products, fundamentally, are software solutions with data at their core. The creation and management of a software product require a broader spectrum of skills and more comprehensive processes than those found in data science alone.</p><p>The role of the data architect has evolved as an essential one, especially for bridging the gap between:</p><ul><li>Data science teams keen on pushing their work to production</li><li>IT departments dedicated to preserving their processes</li></ul><p>The data architect can be seen as the superhero, balancing both ends.</p><p>To enable seamless integration of data within the organization, the data architect must work collaboratively with all stakeholders involved in data and software processes. These include Ops, Enterprise/Solution Architects, Application Architects, Security, DPOs, Data Scientists, Data Engineers, Managers, Product Owners, and Business Sponsors. Besides numerous coffee breaks, the data architect’s role involves:</p><ul><li><strong>Analyzing Current IT Infrastructure:</strong> This entails a comprehensive study of the current IT methodologies, including the various systems, software applications, and hardware used, and the relationships between them. A data architect must have a clear understanding of these components to create a better data management strategy.</li><li><strong>Comprehensive Documentation of Data Projects:</strong> The data architect’s responsibility (from RACI matrix) involves creating complete records of each data project. This includes the project’s rationale (its purpose and intended outcomes), the key stakeholders (both from the business and technical sides), the data sources and their types (e.g., relational databases, NoSQL databases, APIs, etc.), data flows (how data moves and is transformed across systems), and software components involved (like ETL tools, data modeling tools, and data processing frameworks).</li><li><strong>Identifying Synergies and Opportunities for Reuse:</strong> The data architect should conduct a thorough analysis of existing IT solutions against the requirements of the new data project before development begins. This involves identifying shared functionalities and opportunities for reusability to prevent duplication of efforts. For example, if an existing database or data pipeline can fulfill the same function as proposed in the new project, it should be utilized rather than creating a new one.</li><li><strong>Challenging Existing Architectural Norms:</strong> A data architect must not hesitate to challenge the status quo if it does not align with the best architectural practices or meet the project’s needs. This might mean advocating for a shift from monolithic to microservices architecture, or from on-premises to cloud-based infrastructure. The data architect should be ready to counter resistance to change, typically encountered in the form of sentiments like “We’ve always done it this way!” or “It worked fine before you came.”</li><li><strong>Identifying Gaps and Proposing Improvements:</strong> The data architect is tasked with identifying deficiencies in the existing data architecture. This could involve detecting security vulnerabilities, scalability issues, or areas of inefficiency. Once these gaps are identified, the architect needs to formulate a clear, step-by-step roadmap to address these issues. This might involve suggesting new technologies, architectural changes, or process improvements to enhance the current set-up.</li></ul><p>In conclusion, the integration of a data architect into the team should enhance transparency around data products and bring a clear vision about the next moves.</p><h3><strong>The Skills of a Data Architect</strong></h3><p>If you’re a leader contemplating the necessary skills your data architect should possess, or if you’re a data/software professional considering transitioning to a data architect role, this section should provide definitive answers:</p><ul><li><strong>Strong Communication and Soft Skills</strong>: A data architect should be receptive to feedback without personalizing it, and must possess an open mindset for continuous learning. They need to acknowledge the vast domain of unknowns that they are yet to explore.</li><li><strong>Decision-Making Capability</strong>: They must have the ability to make and defend decisions, even when faced with ambiguous requirements.</li><li><strong>Experience in Production Environments</strong>: A data architect should have substantial experience in production environments. Only having proof-of-concept experience may not suffice for effective data architecture. This experience could stem from either IT or data realms — ideally, a combination of both.</li><li><strong>Knowledge of Software Architectural Patterns and their Limitations</strong>: A thorough understanding of various software architectural patterns and their respective constraints is essential.</li><li><strong>In-depth Understanding of Data-related Concepts</strong>: This includes knowledge of data ingestion, storage, transport, processing, quality, privacy, and governance, among others.</li><li><strong>Awareness of Data Platform Market and Cloud Solutions</strong>: Good knowledge about the market for data platforms and cloud solutions is key.</li><li><strong>Familiarity with Machine Learning, NLP, Computer Vision Algorithms</strong>: An understanding of various machine learning, natural language processing, and computer vision algorithms is crucial.</li><li><strong>Knowledge of Deployment Infrastructures</strong>: Familiarity with systems like Kubernetes is essential.</li><li><strong>Solid Software Development Background</strong>: They should understand and be able to explain the main design patterns and their limitations.</li><li><strong>Effective Presentation Skills</strong>: Being a good slide maker is a bonus, as it helps in clearly communicating complex ideas and designs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBBQH-H9XgzjRie2-HPBpQ.gif" /></figure><h3><strong>Conclusion</strong></h3><p>In a nutshell, the pivotal role of a data architect is to harmonize the operations of IT and data teams, thereby streamlining the execution of data-centric projects. Data architects draw from a diverse skill set, embracing technical proficiency, a keen understanding of business needs, and exceptional soft skills.</p><p>This article, and the ones to follow, are an encapsulation of the insights and experiences that Elyadata has accrued over the years. We urge you to visit our website to gain a broader understanding of our work. We hope this series inspires and informs your own data architecture journey, and we look forward to continuing this exploration in future articles. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d49d93d021f" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/data-architecture-bridging-the-gap-between-it-and-data-teams-d49d93d021f">Data Architecture: Bridging the Gap Between IT and Data Teams</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time Data Synchronization]]></title>
            <link>https://insights.elyadata.com/real-time-data-synchronization-2938b8115125?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/2938b8115125</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Thu, 31 Aug 2023 13:31:17 GMT</pubDate>
            <atom:updated>2023-08-31T13:31:17.115Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Streaming Database Updates with Debezium, Kafka Streams, and Kafka Connect</strong></p><h3>Introduction</h3><p>In today’s microservices architecture, event-driven communication has transitioned from being a luxury to a necessity. The traditional approach of using point-to-point communication between microservices is no longer sufficient to meet the evolving demands of modern applications. Event-driven architecture offers numerous advantages, making it a fundamental aspect of building scalable, decoupled, and responsive systems.</p><p>Designing an event driven architecture needs a messaging queue (such as Kafka, RabbitMQ …) to be integrated in the architecture to act as transporter who stream events between microservices.</p><p>Traditionally, to build event driven architecture, Kafka producer and consumer API have been used to achieve event-driven data synchronization between microservices. While this approach serves its purpose, it comes with drawbacks, such as complex code, manual integration, high coupling, and potential data inconsistencies.</p><h4><strong>Data inconsistency:</strong></h4><p>Consider a scenario where multiple microservices (Profile Management, Email Service, Analytics), need to consume the same event produced by User Management Microservice. Each microservice performs a specific action based on the event, such as updating its own database or triggering additional processes. With the traditional approach of using independent Kafka consumers, there is a risk of data inconsistency if one of the microservices encounters processing delays or errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*0wFJBzw9SOi-YJsGNwzksw.gif" /></figure><h4><strong>Complex code:</strong></h4><p>Consider a scenario where a microservice needs to update its database by consuming events from Topic 1 and Topic 2. The microservice might have to implement logic to concatenate or merge the data from these topics based on certain criteria or business rules.</p><p>As the number of topics or the complexity of the merging logic increases, the codebase becomes more intricate and harder to maintain.</p><p>Coordinating the consumption, merging, and updating of data across multiple microservices becomes a non-trivial task, making the system more prone to errors and inconsistencies. This complexity can have various impacts, including increased development effort, longer testing cycles, and higher chances of introducing bugs. It can also limit the agility and scalability of the system, as modifications to the merging logic or adding new topics require extensive code changes and thorough testing.</p><p>To overcome the limitations of the traditional approach, we introduce Debezium, Kafka Connect, and Kafka Streams as robust tools that enhance real-time data synchronization in microservices architecture. Debezium, an open-source Change Data Capture (CDC) platform, captures and streams database changes in real-time, ensuring low-latency data synchronization and reliable change tracking. Kafka Connect, on the other hand, provides a framework for seamless integration of data sources and sinks with Kafka, enabling easy consumption and production of data. Finally, Kafka Streams, a powerful stream processing library, facilitates real-time data processing and transformation, ensuring data is processed as events flow through the system.</p><p>By leveraging the capabilities of Debezium, Kafka Streams, and Kafka Connect, we can streamline the process of real-time data synchronization. Debezium seamlessly integrates with Kafka Connect to capture database changes and produce change events to Kafka topics.</p><p>These change events can then be processed and transformed in real-time using Kafka Streams, allowing us to perform calculations, enrich data, or apply any required transformations.</p><p>Finally, Kafka Connect consumes the processed events from Kafka topics and writes them to the destination database, ensuring automated and accurate data synchronization between microservices.</p><h3><strong>Real Case Scenario: Synchronizing Prices between Microservices</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcB2xjBj0TLk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcB2xjBj0TLk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcB2xjBj0TLk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a39cb8e62a24ac9104a408f06f5909b/href">https://medium.com/media/7a39cb8e62a24ac9104a408f06f5909b/href</a></iframe><p>To delve deeper into the practical implementation of real-time data synchronization, let’s explore a concrete example. Imagine a scenario where we have two microservices: Microservice A, responsible for managing products, and Microservice B, handling order fulfilment.</p><p>In Microservice <strong>A</strong>, we have a Products table that stores essential information about various products, including their unique IDs, names, descriptions, prices, categories and available quantities. On the other hand, Microservice <strong>B</strong> maintains an Orders table, containing order details such as order IDs, corresponding product IDs, customer IDs, quantities and total prices.</p><p>Now, the challenge arises when we want to ensure that the prices in Microservice <strong>B</strong>’s Orders table remain up to date with the latest prices from Microservice <strong>A</strong>’s Products table. Any changes in prices should be reflected immediately to guarantee accurate calculations for the total price of orders.</p><p>To achieve this synchronization, we turn to the powerful combination of Debezium, Kafka Streams, and Kafka Connect.</p><h4><strong>Step 1: Capture Data Changes</strong></h4><p>The journey begins with Debezium, which integrates with the database systems used by both microservices. Debezium leverages the transaction logs stored by the databases to capture and track data changes. As each operation is logged in the transaction log, Debezium detects the insertions, updates, and deletions performed on the tables within the databases.</p><p>In our case, when an admin creates a new product through the Microservice A API, a call is launched to store the new product in the database. Debezium, by reading the transaction log, identifies this change, captures the updated price information, and propagates it to a Kafka topic.</p><h4><strong>Step 2: Data transformations with Kafka Streams</strong></h4><p>Before the message produced by Debezium can be consumed by the second database (Microservice B’s Orders table), it often requires some adjustments or transformations. For example, if we are streaming changes from a NoSQL database to a relational database, specific transformations need to be applied to match the schema and data format of the target database.</p><p>Moreover, in our scenario, let’s say the price in Microservice A’s database is denominated in US dollars, but we need to persist the price in a different currency in Microservice B’s database. This requires applying currency conversion or other modifications to the price value before streaming it to the second database.</p><p>This is where Kafka Streams comes into play. Kafka Streams is a powerful data processing and transformation library within the Kafka ecosystem. It consumes messages from the first Kafka topic, applies the necessary modifications or transformations to the message, and produces the updated message to a second Kafka topic.</p><p>In our case, Kafka Streams consumes the price change events from the topic produced by Debezium, performs currency conversion or any required modifications, and produces the adjusted message to another Kafka topic.</p><h4><strong>Step 3: Data synchronization</strong></h4><p>To inform the second database (Microservice B’s Orders table) about the updated prices, Kafka Connect and a Kafka Sink Connector come in. Kafka Connect is a powerful framework for integrating Kafka with external systems, and a Sink Connector is a specific type of connector used for writing data from Kafka topics to external databases.</p><p>We can configure a Kafka Sink Connector to read the adjusted price messages from the Kafka topic produced by Kafka Streams and apply them directly to the second database, ensuring that the prices in Microservice B’s Orders table are synchronized with the latest updates from Microservice A’s Products table.</p><p>The Sink Connector uses the appropriate database-specific APIs or drivers to write the data to the target database. This involves executing the necessary database operations, such as INSERT, UPDATE, or DELETE statements, based on the incoming messages.</p><p>By leveraging the combined power of Debezium, Kafka Streams, and Kafka Connect, we have successfully synchronized the prices between the two microservices’ databases in real-time. Any changes in product prices are instantly captured, transformed, and propagated to the Orders microservice, enabling accurate calculations for order.</p><h3><strong>Conclusion</strong></h3><p>In this article, we have gained a high-level understanding of the pipeline involved in streaming and synchronizing data between two databases using Debezium, Kafka Streams, and Kafka Connect. While we have covered the basic concepts and benefits, there is still much more to explore in each component.</p><p>In the upcoming articles, we will take a deeper dive into the core components of this pipeline. In the second article, we will focus on Capture Data Change (CDC) using Kafka Connect and Debezium. We will explore how Debezium captures and tracks data changes using transaction logs and how Kafka Connect enables the seamless integration between databases and Kafka topics.</p><p>In the third article, our focus will shift to Kafka Streams. We will delve into the inner workings of Kafka Streams and understand how it facilitates data processing, transformations, and event-driven computations within the Kafka ecosystem. We will explore its powerful capabilities for real-time data manipulation and how it enhances the streaming pipeline for synchronization between microservices’ databases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2938b8115125" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/real-time-data-synchronization-2938b8115125">Real-time Data Synchronization</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Hiring and Onboarding Journey at Elyadata]]></title>
            <link>https://insights.elyadata.com/navigating-the-hiring-and-onboarding-journey-at-elyadata-bf8b16dfd9db?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf8b16dfd9db</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[onboarding-process]]></category>
            <category><![CDATA[hiring-process]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Tue, 18 Jul 2023 11:29:34 GMT</pubDate>
            <atom:updated>2023-07-18T11:29:34.581Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="http://www.elyadata.com/"><strong>Elyadata</strong></a>, we believe in hiring the best talent and ensuring they are well-equipped to contribute to our projects. Our hiring process is designed to assess a candidate’s technical skills, problem-solving abilities, and cultural fit. We also have a robust onboarding program to ensure new hires are well-integrated into our team and understand our systems and processes. Here’s a detailed look at our hiring and onboarding process, which is deeply intertwined with our tech landscape as seen on our <a href="https://radar.elyadata.com/"><strong>Technology Radar</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ed0d2_Ho0trtDTOCIoSGlQ.png" /></figure><h4>Step 1: Online Technical Assessment</h4><p>The first step in our hiring process is an online technical assessment. This test is designed to assess your knowledge and skills in a wide range of technologies and fields. Depending on your profile, you may be tested on programming languages such as Python and/or Java, Kafka, CI/CD concepts, Kubernetes, Angular, React, and more. These technologies are part of our ‘Adopt’ quadrant in our Technology Radar, indicating their importance in our tech landscape.</p><h4>Step 2: Coding Exercise</h4><p>Candidates who perform well on the online technical assessment will be contacted for the next step: a coding exercise. This is a take-home assignment that you can complete at your own pace. Once you’ve completed and shared your work with us, we’ll schedule a face-to-face interview. This exercise gives us a deeper understanding of your coding abilities and how you approach problem-solving.</p><h4>Step 3: Face-to-Face Interview</h4><p>During the face-to-face interview, we’ll delve deeper into your knowledge of software engineering concepts. You can expect to be questioned on topics such as design patterns, object-oriented programming (OOP), Git, Docker, and more. We’ll also discuss the coding exercise you completed, giving you an opportunity to explain your thought process and approach. This step helps us assess your technical knowledge, communication skills, and cultural fit with our team.</p><h4>Step 4: HR Interview</h4><p>The next step is an HR interview to assess your qualifications for the job. This interview helps us determine whether you are a good fit for our company culture. We believe that a harmonious cultural fit is crucial for both the candidate’s job satisfaction and the overall productivity of our team.</p><h4>Step 5: Communication of Decision</h4><p>After the HR interview, we will communicate our decision to you in a short time. We understand that job hunting can be a stressful process, and we aim to provide you with our decision as quickly as possible.</p><h4>Step 6: Preparing for Arrival</h4><p>If you are selected, we will begin preparing for your arrival. This includes preparing contracts and arranging for the necessary equipment you will need to perform your job effectively.</p><h4>Step 7: Onboarding Phase</h4><p>Once you join us, you will enter the onboarding phase. This phase is divided into two parts:</p><ul><li>Orientation and Training: You will join at least one chapter and learn the basics of CI/CD, Docker, Git, and microservice architecture. These are key technologies and concepts in our tech landscape and understanding them is crucial for your work with us.</li><li>Integration and Support: We believe in open communication and will have regular check-ins to address any questions, concerns, or challenges you may face. We want to ensure that you feel supported and are able to integrate smoothly into our team.</li></ul><h4>Step 8: Joining Our Projects</h4><p>After you’ve completed the onboarding phase, you’ll join our ongoing projects. By this point, you’ll have a solid understanding of our technologies, processes, and culture, and you’ll be ready to start contributing to our work.</p><p>We believe that this comprehensive hiring and onboarding process ensures that we bring on board the best talent and equip them with the knowledge and skills they need to succeed. Our process is deeply rooted in our tech landscape, ensuring that our team members are well-versed in the technologies and methodologies we use. We look forward to welcoming you to our team and seeing the great things we can accomplish together.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf8b16dfd9db" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/navigating-the-hiring-and-onboarding-journey-at-elyadata-bf8b16dfd9db">Navigating the Hiring and Onboarding Journey at Elyadata</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elyadata: Pioneering the Future of Software engineering and Data expertise]]></title>
            <link>https://insights.elyadata.com/elyadata-pioneering-the-future-of-software-engineering-and-data-expertise-14ae5ead6d95?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/14ae5ead6d95</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Wed, 12 Jul 2023 10:56:31 GMT</pubDate>
            <atom:updated>2023-07-12T10:56:31.593Z</atom:updated>
            <content:encoded><![CDATA[<p>As technology keeps advancing quickly, data has become an invaluable asset. Businesses everywhere need to manage and use their data effectively. Recognizing this significant shift, we at <a href="http://www.elyadata.com/"><strong>Elyadata</strong></a> understand this big change. We’re not just adapting to it, we’re <strong>leading</strong> the way. We are improving the role of Software Engineers and Data experts in our environment to improve our <strong>fitness for purpose</strong>. We strongly believe that <strong>combining</strong> <strong>software engineering with data expertise</strong> is the best way to help our clients. This combination will also prepare us for the future, where data will drive many innovations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6hX1ak9ByW3O-FHx_OGXjA.png" /></figure><h4>Blending Software Engineering and Data Expertise</h4><p>Why are we leaning into this change? Our strategic move is fueled by a strong conviction that the combination of software engineering and data expertise is the golden key to unlocking our clients’ aspirations and preparing for the looming wave of <strong>data-driven innovation</strong>. To actualize this, we’re investing significantly in developing our engineers’ abilities. This means sharpening their software development skills based on agile principles while also nurturing their expertise across a spectrum of <strong>data capacities</strong>, from<strong> data engineering</strong> to the subtleties of <strong>data science</strong>.</p><h4>Incorporating DevOps</h4><p>We see <strong>DevOps</strong> knowledge as a cornerstone of <strong>software data engineering</strong>. By integrating DevOps practices, our engineers ensure the continuity of all stages of the software development life cycle — from idea inception to production deployment. Our focus is not just about making connections; it’s about making them efficiently and sustainably through automation, continuous integration, and continuous deployment.</p><p>Our paradigm shift mirrors the evolution of data expertise. Where once it was considered a niche skill, data knowledge is now a part of the everyday toolkit for software engineers. In the same vein, we’re betting that the future of digital transformation will be steered by the harmonious blend of software engineering and data proficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gkauv5pr3vbgjhIfxiuD1g.png" /></figure><h4>Forging the Future: Elyadata’s vision</h4><p>By embracing this <strong>forward-thinking approach</strong>, we’re poised to architect advanced data solutions that align perfectly with our clients’ unique needs. At <strong>Elyadata</strong>, we don’t just want to keep up with the ever-evolving data landscape; we aim to define it. Our Software Data Engineers aren’t just team members; they’re the <strong>torchbearers</strong> leading us into the future of digital transformation.</p><p>If you found this piece insightful, please follow us in <a href="https://www.linkedin.com/company/elyadata"><strong>LinkedIn</strong></a>. Stay tuned for more updates from us in our quest to redefine the data world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14ae5ead6d95" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/elyadata-pioneering-the-future-of-software-engineering-and-data-expertise-14ae5ead6d95">Elyadata: Pioneering the Future of Software engineering and Data expertise</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps pipeline using Kedro & Mlflow]]></title>
            <link>https://insights.elyadata.com/mlops-pipeline-using-kedro-mlflow-c157591a335b?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/c157591a335b</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 14:45:33 GMT</pubDate>
            <atom:updated>2023-06-22T14:45:33.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*NuGkGsrFMBjFihCuNyXEjQ.png" /></figure><h3><strong>Kedro</strong></h3><h4><strong>What is kedro?</strong></h4><p>Kedro is an open-source Python framework for creating reproducible, sustainable, and modular data science code. It borrows concepts from software engineering best practices and applies them to machine learning code; the concepts applied include modularity, separation of interests, and version control.</p><ul><li><strong>Open source</strong> Python framework: Kedro is free and maintained by its community, making it easy to use.</li><li><strong>Reproducible code</strong>: with Kedro, you can create data pipelines and use them for different data sources.</li><li><strong>Sustainable code</strong>: Kedro’s framework makes it easy to maintain code and for a team to work collaboratively on the same pipeline.</li><li><strong>Modular code</strong>: functions can be used in different parts of the pipeline and different pipelines.</li></ul><p><strong>Installation guideline</strong></p><p>The installation of <strong>Kedro</strong> is done using the PyPI package :</p><pre>pip install kedro</pre><p>To verify the installation, run this command :</p><pre>kedro info</pre><p>If all goes well, you will see the following image, followed by the installed version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*0mINk3NJUiaQZCCrLdAmxg.png" /></figure><h4><strong>Creating a project with kedro interactively</strong></h4><p>To create a Kedro project, run the following command:</p><pre>kedro new</pre><p>Next, you will have to answer three questions:</p><ol><li><em>project_name</em>: name of the project (use a readable name separated by underscores);</li><li><em>repo_name</em>: project folder name (you can leave it blank, and kedro will use the project name)</li><li><em>python_package</em>: name of the python package (can also be left blank).</li></ol><h4><strong>Creating a new project from a configuration file</strong></h4><p>You can create a new project from a configuration file if you prefer. The file must contain:</p><pre>output_dir: ~/code<br>project_name: Get Started<br>repo_name: get-started<br>python_package: get_started</pre><p>To create the new project:</p><pre>kedro new - config config.yml</pre><h4><strong>Creating a pipeline</strong></h4><p>To generate a new pipeline template, run:</p><pre>kedro pipeline create data_processing</pre><h3><strong>MLflow</strong></h3><h4><strong>What is MLflow?</strong></h4><p>MLflow is a platform to streamline machine learning development, including tracking experiments, packaging code into reproducible runs, and sharing and deploying models. MLflow offers a set of lightweight APIs that can be used with any existing machine learning application or library (TensorFlow, PyTorch, XGBoost, etc), wherever you currently run ML code (e.g. in notebooks, standalone applications, or the cloud).</p><p>MLflow’s current components are:</p><ul><li><a href="https://mlflow.org/docs/latest/tracking.html">MLflow Tracking</a>: An API to log parameters, code, and results in machine learning experiments and compare them using an interactive UI.</li><li><a href="https://mlflow.org/docs/latest/projects.html">MLflow Projects</a>: A code packaging format for reproducible runs using Conda and Docker, so you can share your ML code with others.</li><li><a href="https://mlflow.org/docs/latest/models.html">MLflow Models</a>: A model packaging format and tools that let you easily deploy the same model (from any ML library) to batch and real-time scoring on platforms such as Docker, Apache Spark, Azure ML, and AWS SageMaker.</li><li><a href="https://mlflow.org/docs/latest/model-registry.html">MLflow Model Registry</a>: A centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of MLflow Models.</li></ul><h4><strong>Installation</strong></h4><pre>pip install mlflow</pre><h4><strong>Setting a tracking URI</strong></h4><p>MLflow runs can be recorded to local files, to an SQLAlchemy compatible database, or remotely to a tracking server.</p><p>By default, the MLflow Python API logs runs locally to files in an mlruns directory wherever you ran your program.</p><pre>mlflow.set_tracking_uri()</pre><pre>- Local file path (specified as file:/my/local/dir)<br>- Database encoded as &lt;dialect&gt;+&lt;driver&gt;://&lt;username&gt;:&lt;password&gt;@&lt;host&gt;:&lt;port&gt;/&lt;database&gt;<br>- HTTP server (specified as https://my-server:5000) which is a server hosting an MLflow tracking server.</pre><h4><strong>Creating an experiment</strong></h4><pre>mlflow.create_experiment()</pre><p>creates a new experiment and returns its ID.</p><p>Runs can be launched under the experiment by passing the experiment ID to</p><pre>mlflow.start_run()</pre><h4><strong>Creating &amp; ending a run</strong></h4><pre>mlflow.start_run()<br>mlflow.end_run()</pre><p>If no active run exists, a new one is started.</p><h4><strong>Logging parameters</strong></h4><p>With Mlflow, you can log a single key-value param/ multiple params at once in the currently active run. The key and value are both strings.</p><pre>mlflow.log_param()<br>mlflow.log_params()</pre><h4><strong>Logging metrics</strong></h4><p>Logging key-value metrics is possible &amp; MLflow remembers the history of values for each metric.</p><p>The metric value must always be a number.</p><pre>mlflow.metric()<br>mlflow.metrics()</pre><h4><strong>Logging artifacts</strong></h4><pre>mlflow.log_artifact()<br>mlflow.log_artifacts()</pre><p>logs a local file or directory as an artifact, optionally taking an artifact_path to place it within the run’s artifact URI.</p><h3><strong>Kedro-Mlflow pipeline</strong></h3><h4><strong>Kedro-Mlflow plugin</strong></h4><p><strong>kedro-mlflow</strong> is a kedro-plugin for lightweight and portable integration of mlflow capabilities inside kedro projects. It enforces Kedro principles to make mlflow usage as production ready as possible.</p><h4><strong>Kedro-mlflow installation</strong></h4><p>Kedro-mlflow is only compatible with <em>kedro&gt;=0.16.0</em> and <em>mlflow&gt;=1.0.0.</em></p><p>If you have a project created with an older version of Kedro, see this <a href="https://github.com/kedro-org/kedro/blob/main/RELEASE.md">migration guide</a>.</p><h4><strong>Install from PyPI</strong></h4><p>kedro-mlflow is available on PyPI, so you can install it with pip:</p><pre>pip install kedro-mlflow</pre><h4><strong>Install from sources</strong></h4><p>you can install the package from Github:</p><pre>pip install git+https://github.com/Galileo-Galilei/kedro-mlflow.git</pre><h4><strong>Check the installation</strong></h4><p>Type kedro info in a terminal to check the installation. If it has succeeded, you should see the following ascii art:</p><pre> _            _ <br>| | _____  __| |_ __ ___ <br>| |/ / _ \/ _` | &#39;__/ _ \ <br>|   &lt;  __/ (_| | | | (_) | <br>|_|\_\___|\__,_|_|  \___/ <br>v0.16.&lt;x&gt; <br> <br>kedro allows teams to create analytics <br>projects. It is developed as part of <br>the Kedro initiative at QuantumBlack. <br> <br>Installed plugins: <br>kedro_mlflow: 0.11.3 (hooks:global,project) </pre><h4><strong>Using the plugin</strong></h4><p>After the installation, the plugin can be activated using:</p><pre>kedro mlflow init</pre><p>If successful, you’ll see the following message: <em>‘conf/local/mlflow.yml’ </em>successfully updated.</p><p>The conf/local folder is updated and you can see the <em>mlflow.yml</em> file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*4lETnDqSfYNPDx79MhmAWQ.png" /></figure><p><em>Optional: If you have configured your own mlflow server, you can specify the tracking uri in the </em>mlflow.yml (replace the highlighted line below):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*pvFVlmbAZMVqGIZgDMQCtQ.png" /></figure><h3><strong>Project example with Kedro &amp; Mlflow</strong></h3><h4><strong>Project Components</strong></h4><p>In this section, we present an example of kedro pipeline which contains 3 main pipelines:</p><ul><li><strong>Data Preparation:</strong> the goal of this pipeline is to extract annotated images (text of annotation) from label studio and find the corresponding images.</li><li><strong>Data Science: </strong>this pipeline is used to train Trocr approach using the data extracted in the Data Preparation pipeline, and then evaluate the model using the evaluation dataset.</li><li><strong>Data Versioning: </strong>in order to<strong> </strong>capture the versions of our models in Git commits, we used Data Version Control (<a href="https://dvc.org/doc">DVC</a>) to store models in each experiment in Azure Blob Storage Container.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*yH6ypBCRAPI78-wI3Kgymg.png" /></figure><h4><strong>Data Preparation Pipeline</strong></h4><ul><li>The <em>data_preparation/nodes.py</em> contains extract_data_from_label_studio node and other functions. This node extracts the annotated data (filename and annotated text) from Label Studio and saves them in a file.</li><li>For <em>data_preparation/pipeline.py</em>, we create data preparation pipeline by assembling the node in the pipeline and specifying the name of the function (node), the input and the output of this node and the name of the node.</li><li>In <em>base/parameters/data_prepara.yml</em> file, the data preparation pipeline settings are defined. We specified the connection parameters of Label Studio (Token and URL) and data extraction parameters (like number of takes to be extracted and the path of the file in which the data will be stored).</li></ul><h4><strong>Data Science Pipeline</strong></h4><ul><li>The <em>data_science/nodes.py</em> file contains the training &amp; evaluation code of Trocr model. We also use mlflow to log the parameters, metrics &amp; artifacts of the run.</li><li>For <em>data_science/pipeline.py</em>, we specify the input and output of each node (functions defined in <em>nodes.py</em>)</li><li>The parameters of the data science pipeline can be found in <em>parameters/data_science.yml</em></li></ul><h4><strong>Data Versioning Pipeline</strong></h4><p>Similarly to the previous pipeline, the nodes of the data versioning pipeline are defined in <em>data_versioning/nodes.py</em>. They define the remote configurer interface, the DVC Azure remote configuration, and the DVC and Git clients. <em>pytorch_pipeline/pipeline_registry.py</em> is used to register the created pipelines</p><h4><strong>Running the project</strong></h4><pre>kedro run</pre><p>You can also run a specific pipeline:</p><pre>kedro run - pipeline my_pipeline</pre><p>If you specify kedro run without the — pipeline option, it runs the default pipeline from the dictionary returned by <em>register_pipelines()</em>.</p><p>Launching Mlflow UI:</p><pre>kedro mlflow ui</pre><h4>References</h4><p><a href="https://mlflow.org/docs/">https://mlflow.org/docs/</a></p><ul><li><a href="https://kedro-mlflow.readthedocs.io/">Welcome to kedro-mlflow&#39;s documentation! - kedro-mlflow 0.11.8 documentation</a></li><li><a href="https://docs.kedro.org/">Welcome to Kedro&#39;s documentation! - kedro 0.18.10 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c157591a335b" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/mlops-pipeline-using-kedro-mlflow-c157591a335b">MLOps pipeline using Kedro &amp; Mlflow</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Qwik- Introduction to Resumability]]></title>
            <link>https://insights.elyadata.com/qwik-introduction-to-resumability-e5df5990de50?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5df5990de50</guid>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[qwik]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 14:19:20 GMT</pubDate>
            <atom:updated>2023-06-22T14:27:59.466Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*Sa-D75TUjcMlXyf-4ToTqw.png" /></figure><h3>Introduction</h3><p>Functional programming is a programming paradigm that emphasizes the organization of changing-state and changeable data.</p><p>While it has remained closely associated with this context for a long time, imperative languages (first procedural, then object-oriented) have grown and will one day dominate the programming world.</p><p>With that noted, it’s frequently characterized in terms of related notions such as pure functions, lazy evaluation, pattern matching, and so on.</p><p>Regardless of the complexity and the size of the app, the goal is to have instant-on-running web applications, operational upon launch, where interactive behavior is added to its pages.</p><p>That is, interactive components which help put the glow on static unremarkable page and transforms it into a captivating one that holds the reader’s attention.</p><p>Interactive elements are not only great for bringing real tangibel functionality to a site, but they can also be more engaging and incredibly valuable for significant customization.</p><p>More structure to the code has been added as the demands on that interaction rose. Imperative sprinkles evolved into declarative frameworks we know and use today, such as React, Angular, and Vue.</p><p>The whole representation of the UI now lives in JavaScript.</p><p>To restore the ability to server render, these frameworks also create HTML on the server. When we launch our app in the browser, they re-run the same code, adding event handlers and ensuring the app is in the proper state.</p><p>Before diving deeper into further concepts, we’ll start by introducing Qwik:</p><blockquote>Qwik is an open-source DOM-centric, resumable web-app framework designed for best possible time-to-interactive.</blockquote><p>There is a lot to unpack, and a lot more than we could possibly cover in this part. So, we will break this up into a series of concepts.</p><p>Still, the question is : <em>How to crush time-to-interactive to a blink of an eye on the slowest mobile device ?</em></p><p>We believe there is a better way to build sites :</p><ul><li>A way that does not involve massive amounts of JavaScript that must be downloaded eagerly on startup before the site becomes interactive.</li><li>A way that allows the developer to think about adding features rather than how to break up the huge code-base into smaller chunks.</li><li>A way to have instant-on sites for a better user experience. And all of that, in a way that is independent of the size of the application code-base.</li></ul><p>The unique aspect about it , it’s not what it does but how it achieves it. The strategy stands as follows :</p><ul><li>1ST PART Delaying the execution and download of JavaScript: It is considered as a significant departure from traditional practices.</li><li>2ND PART Serializing the execution state of the application and the framework on the server: Qwik can save the state of the application at any given point and then resume it later.</li><li>3RD PART Resume the application on the client: The client doesn’t have to start from scratch each time the application is loaded.</li></ul><h3><strong>Traditional Approach</strong></h3><p>The primary distinction between <strong>Client-Side Rendering [CSR]</strong> and <strong>Server-Side Rendering [SSR]</strong> is the location of the page’s rendering. <strong>[SSR]</strong> renders the page on the server, whereas <strong>[CSR]</strong> renders it on the client. In fact, the client-side controls the routing dynamically, without refreshing the page each time the client requests a different route.</p><h4><strong>Client-Side Rendering [CSR]</strong></h4><p>When a user wants to access a website, the browser sends requests to the server and eventually receives HTML, CSS, and JS files from the server.</p><p>The browser executes those files and renders a dynamic website where users can see and interact with the content. And this is how we reach INTERACTIVITY .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yyy9I5KQ7osyQBi2kRK1Vg.png" /></figure><p>Henceforth,</p><p>because of how CSR works, it may take time to make websites visible to the users, especially in case of poor Internet connection or complicated ones with large sizes. In such cases, users usually see a blank page.</p><p>Furthermore, CSR websites cannot support a good [ — ] SEO because there is no information about the website provided in the web’s source code.</p><h4><strong>Server-Side Rendering [SSR]</strong></h4><p>In server-side rendering, when a user wants to access a website, the browser sends requests for that page to the server and eventually receives a full-content HTML page. Instead of spending some time executing JS files like CSR, the browser just needs to display the page using provided HTML files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYqphK0Da6FqMGoG3uwA7A.png" /></figure><p>Users can soon see the content of the website instead of having a blank screen in front of them.</p><p>As discussed above, SSR seems to be a good way to go when coding websites with certain benefits like improving SEO and decreasing the initial load time.</p><p>However, the page that the browser shows us initially is not immediately interactable for a few seconds.</p><p>Another thing to be added is that SEO has become an inevitable option for a website wanting to have a better rank in Google search results. Therefore, many developers choose to code SSR websites.</p><p><strong>EXPLANATION</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGscBhnG8EAS6W4LZZNGNg.png" /></figure><p>At first, this blank page is shown while JS loading, although the user can see some content on the SSR website, they cannot do anything to the page because the page is just a plain static HTML file with no event handlers attached.</p><p>This is when Hydration comes to play.</p><p>Hydration is a method in where interactions are added to the static HTML files sent from the server to the browser. And <strong>“hydrate” them means attaching some event handlers to them.</strong></p><p><strong>EXAMPLE</strong></p><p>Suppose we have a static HTML file sent from the server with 2 button components. There is nothing the user can do with them. Therefore, we have to “hydrate” them by attaching some event handlers to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*54WYl60yPUGsGMR2X_lIVQ.png" /></figure><p>To do this, besides receiving HTML files from the server, the client also has to fetch all the JS files (just like <strong>CSR</strong>) and re-execute them to find what components to bind to corresponding Elements.</p><h4><strong>Difference between both renderings</strong></h4><p>The SSR architecture requires the server to execute JS codes and render a static HTML to the client. However, the client cannot just use it right away, it has to fetch JS codes and do the exact same thing as the server does — executing them again to find the exact event handler to bind to the correct elements.</p><h4>Why not Hydration ?</h4><p>This hydration process wastes quite a lot of time and resources. It can take up to 10s for a mobile phone to load a fully interactive SSR website if that website is heavy.</p><h3><strong>Qwik’s Core Principle</strong></h3><p>Qwik applications start up fast because there is a minimal amount of JavaScript code to execute, which makes its core principle is to prioritize speed and efficency. By delaying the application download, Qwik can provide near-instant startup performance that surpasses that of many other frameworks.</p><p>Its speed comes from not doing things (such as <strong>hydration</strong>) that other frameworks have to do.</p><p>Hydration or re-hydration refers to the process in which client-side JavaScript converts a static HTML web page into a dynamic web page by attaching event handlers to the HTML elements. This can be resource-intensive.</p><h4><strong>Hydration is Pure Overhead</strong></h4><p>In web development, client-side JavaScript breathes life into a static HTML webpage by appending event handlers to HTML elements, thereby morphing it into a dynamic web page.</p><p>However, the act of attaching this latter to the DOM is not the challenging or taxing part of the hydration, and so it misses the point of why anyone would call hydration an overhead which is a work that can be avoided and still leads to the same end result. ( the task could be by-passed)</p><p>As its core, hydration involves re-executing code, meaning it’s repeating jobs which were previously performed by the server. This redundancy is a key reason why hydration should be an overhead.</p><p>Moreover, the event handler is a closure that encapsulates the behavior of the event handler. It is what should happen if a user triggers this event.</p><p><strong>The complexity increases when we consider that the WHAT is a closure : </strong>it encompasses and closes over APP_STATE and FRAMEWORK_STATE :</p><ul><li>APP_STATE : corresponds to the state of the application. Without it, the application has nothing dynamic to present to the user.</li><li>FRAMEWORK_STATE : refers to the internal state of the framework. Without it, the framework does not know which DOM nodes to update and when to perform them.</li></ul><p><strong>And for WHERE</strong> , it’s the location of the DOM element where the WHAT needs to be attached to.</p><p>The question then arises: How do we recover the <strong>WHAT</strong> (APP_STATE + FRAMEWORK_STATE) and the <strong>WHERE</strong>?</p><p>Put simply, hydration is overhead because it duplicates work.</p><p>The server constructs the WHERE and WHAT (APP_STATE + FRAMEWORK_STATE), but the information for the client is discarded instead of being serialized for the client.</p><p>The client then receives HTML that lacks sufficient information to rebuild the application.</p><p>The information deficit compels and forces the client to eagerly download the application and execute it to recover the <strong>WHERE</strong> and <strong>WHAT</strong>. And that is expensive.</p><p>In simpler words, hydration is a hack (workaround) to recover the APP_STATE + FRAMEWORK_STATE by eagerly executing the application code in the browser and involves the following :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O4YQ9pyHDhzr7aWGJYyqlw.png" /></figure><p>Still, there are issues. The current approach has the following:</p><ul><li>Requires the template code to be downloaded in advance.</li><li>Requires template code to be executed in advance.</li><li>Requires the event handler code to be downloaded in advance to be attached.</li></ul><p>The above approach does not scale, as the application becomes more complicated. The amount of code needed to download beforehand and execute, grows proportionally with the size of the application.</p><p>This negatively impacts the application startup performance and hence the user experience.</p><h4><strong>What’s wrong with JavaScript ?</strong></h4><p>JavaScript, while being an essential part of modern web development, can also be a double-edged sword. As the complexity of a website increases, so does the amount of JavaScript required to make it interactive. This entire code negatively affects the speed at which the site loads.</p><p>Now imagine a scenario where a user is trying to access a website on a slow network or a device with limited processing powers. The large amount of Javascript to download can cause the site to load slowly, leading to poor user experience. So in other words, re-rendering the entire application on page load can be costly as pages get larger, especially on slower networks and devices.</p><p>But the challenges don’t stop there. One more thing to add, too much JavaScript manifests itself in two problems:</p><ul><li>A large amount of code is shipped to the client, which may take a long time on slow networks.</li><li>The execution of large amounts of JavaScript code can delay the time it takes (as part of hydration) in order to get the site interactive, once on the client.</li></ul><p>In summary, too much of JavaScript can lead to slower load times and delayed interactivity.</p><h4><strong>Qwik comes to the rescue</strong></h4><p>The following showcases the improvements of insights regarding the performance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*WyNPmaRUQUfXlFalz1oUxQ.png" /></figure><p>Imagine the current speed is of 25.3s and we decided to optimize image rendering → the time speed remains the same and no noticeable improvement is going to be shown.</p><p>Next, you choose to lighten the load and tweak the CSS by removing useless items, but again the speed doesn’t change. However, if we reduce the JavaScript, we’re able to boost the speed score by 60 points and escalates to a whooping 96% → it will speed up by 23s. That’s a massive-improvement.</p><p>This jump is just mind-blowing and worth consideration. It’s a game-changing approach.</p><p><strong>This where Qwik takes the idea of delay loading of JavaScript to the extreme.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4gRz64JK9z0L_aRManM8w.png" /></figure><p>Qwik is a revolutionary framework that fundamentally rethinks how we structure and deliver web applications. It’s not about writing less JavaScript, but rather managing intelligently when and how the JavaScript is delivered. Qwik is about not having to ship all of that hefty JavaScript package to the client at once, on application startup.</p><p>This innovative framework is the brainchild of the same minds that brought up AngularJS, Ionic and Stencil. Its execution should partially be done on the server, paused, and resumed at the client . They have introduced what constitutes the outstanding RESUMABILITY process.</p><p>In essence, no extra work is done, and the web application will become leaner, faster, smoother and more responsive.</p><h4><strong>Resumability Overview</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e67msmcXoz_5Lg2KSvxKzA.png" /></figure><p>When an application starts in the browser, it typically needs to initialize various components, set up event handlers and perform other tasks to get ready for user interaction. This can be time-consuming and not particularly powerful.</p><p>Resumability addresses this issue by allowing the browser to pick up exactly where the server left off. The server does the heavy lifting of executing the application, and then serializes the state and sends it to the browser who will deserialize and resumes execution without redoing what’s done.</p><p>EXAMPLE</p><p>Imagine a computer that goes into hibernation mode. When it’s awake, it resumes exactly where you left it, with all your applications and files open just where you left them. Now take this up and apply it to the server/browser network boundary. This is considered the essence of <strong>RESUMABILITY.</strong></p><p>In a resumable setup, the system as entailed before, initially attaches some global event handlers. However it doesn’t run the entire code at once. It waits for a certain user interaction and then executes only the necessary code in response to that specific interaction.</p><p>Think of it as a well-orchestrated play where actors only come on stage when it’s their turn to perform, rather than crowding from the beginning.</p><p>In this way, RESUMABILITY is tailored to each visitor’s needs and every action is purposeful.</p><blockquote><strong>No effort is wasted.</strong></blockquote><h4><strong>Comparison between Resumability and Hydration</strong></h4><p>In a resumable system, the process of creating an event handler is deferred (<strong>lazy</strong>) until the event itself is actually triggered. Therefore, the creation is strictly on an “<strong>as-needed</strong>” basis. So, Qwik here, creates the event handler by deserializing it, thereby the client does not redo any task that was already performed.</p><p>In a hydration process, the system preemptively creates the event handler before the event itself gets triggered and this is how the approach is considered to be therefore <strong>eager</strong>. It may then lead to superfluous unneeded operations.</p><p>Contrastingly, that’s why in resumable frameworks, we are not really hydrating. Instead, we can still postpone loading code until we absolutely need it. This way is akin to keeping our tools in the toolbox until we need to use them, rather than laying them all out.</p><p>This can significantly streamline the process and reduce the bundle towards achieving the ideal of “<strong>0Kb JavaScript by default</strong>”.</p><h4><strong>Resumability: A No-Overhead Alternative to Hydration</strong></h4><p>To eliminate unnecessary overhead, the framework needs to do more than just to prevent redundancy, it must break away and step from the above: attaching the <strong>WHAT</strong> to the <strong>WHERE</strong>.</p><p>Indeed, it focuses on transferring all of the information data from the server to the client. It’s only when a user initiates an interaction that the client is compelled to download the necessary code t manage that certain particular interaction.</p><p>So in the context of Qwik, the client is not performing any redundant, duplicated work that the server has already executed it; therefore, there is no overhead meaning not unnecessary load will be processed.</p><p>The approach not only removes overhead but also makes the whole process more efficient and user-friendly. It’s not only about performance but it makes the application more robust by ensuring that the server and the browser are always in sync.</p><blockquote><strong>If you wanna be quick, think Qwik.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5df5990de50" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/qwik-introduction-to-resumability-e5df5990de50">Qwik- Introduction to Resumability</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the Tech Landscape with Elyadata’s Tech Radar]]></title>
            <link>https://insights.elyadata.com/navigating-the-tech-landscape-with-elyadatas-tech-radar-b4bb415641f7?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4bb415641f7</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[techradar]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 14:14:17 GMT</pubDate>
            <atom:updated>2023-06-22T14:14:17.832Z</atom:updated>
            <content:encoded><![CDATA[<h3>Elyadata Technology Radar</h3><h4>Navigating our Technology Landscape</h4><h3>Navigating the Technology Landscape</h3><p>In the dynamic world of technology, staying updated with the latest trends and tools is crucial. Inspired by the excellent work from <a href="https://www.thoughtworks.com/radar"><em>ThoughtWorks</em></a>, Elyadata has developed a comprehensive <a href="https://radar.elyadata.com/"><strong>Technology Radar</strong></a> to guide businesses, engineers, and individuals through the frontiers of technology used by our different teams. This tool is designed to provide insights into our current engineering landscape and the technologies we are adopting, trialing, assessing, and holding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*5KA_oTmLFMBnE7_tmYbUQw.png" /><figcaption>A screenshot of Elyadata tehcnology Radar <a href="https://radar.elyadata.com/index.html">website</a></figcaption></figure><h3>Four Quadrants of the Technology Radar</h3><p>The Elyadata Tech Radar is divided into four quadrants, each focusing on a specific area of technology:</p><p>1. <strong>Software Engineering</strong>: This quadrant includes development languages and low-level development frameworks. These tools are essential for implementing custom software of all kinds.</p><p>2. <strong>Tools &amp; Platforms</strong>: This quadrant focuses on data-related content. It includes tools and platforms that facilitate data handling and processing.</p><p>3. <strong>Data Engineering</strong>: This quadrant clusters architectures around delivered services. It includes technologies that are integral to data engineering and management.</p><p>4. <strong>Methods &amp; Patterns</strong>: This quadrant includes different methodologies used in projects. It includes practices that guide the development process and patterns that provide solutions to common problems.</p><p>Each quadrant is further divided into four stages: Adopt, Trial, Assess, and Hold. This classification helps users understand the maturity and applicability of each technology.</p><h3>Understanding the Stages: Adopt, Trial, Assess, and Hold</h3><p>1. <strong>Adopt</strong>: Technologies in the ‘<strong>Adopt</strong>’ stage are those that we have high confidence in and recommend for use. These technologies have been proven to add significant value and are recommended for use in relevant scenarios.</p><p>2. <strong>Trial</strong>: Technologies in the ‘<strong>Trial</strong>’ stage are those that we believe have potential and are worth exploring in a controlled setting. These technologies are promising but need to be tested in a safe-to-fail environment.</p><p>3. <strong>Assess</strong>: Technologies in the ‘<strong>Assess</strong>’ stage are those that we are considering for future use. These technologies are on our radar, and we are exploring their potential benefits and impact.</p><p>4. <strong>Hold</strong>: Technologies in the ‘<strong>Hold</strong>’ stage are those that we currently do not recommend for new projects. These technologies might be deprecated, have better alternatives, or do not align with our current strategy.</p><h4>A Tool for Customers, Leads, Engineers, and Candidates</h4><p>The Elyadata Tech Radar is a valuable resource for our customers, leads, engineers, and candidates. It provides a clear picture of our current engineering landscape and the technologies we are focusing on. By understanding the technologies we use and our approach to adopting new ones, our stakeholders can align their strategies and expectations with ours.</p><h4>Regular Updates and Additions</h4><p>The Elyadata Technology Radar is regularly updated to reflect the latest trends in the tech industry. This ensures that our stakeholders are always informed about our current focus and the direction we are heading.</p><h3>The Elyadata Promise</h3><p>Elyadata is committed to help business leaders transform their organizations and ensure continued success by using advanced technology and best practices. The Technology Radar is a testament to this commitment, providing a comprehensive and up-to-date guide to the latest technologies.</p><p>Stay tuned to the <a href="https://radar.elyadata.com/"><strong>Elyadata Technology Radar</strong></a> for the latest updates and insights into the world of technology. Follow us to stay ahead of the curve and leverage the power of technology to drive your business forward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4bb415641f7" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/navigating-the-tech-landscape-with-elyadatas-tech-radar-b4bb415641f7">Navigating the Tech Landscape with Elyadata’s Tech Radar</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Bloom Filters: A Probabilistic Method for Efficient Data Querying]]></title>
            <link>https://insights.elyadata.com/demystifying-bloom-filters-a-probabilistic-method-for-efficient-data-querying-cc186b80d08b?source=rss-48d50203581------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc186b80d08b</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[problem-solving]]></category>
            <category><![CDATA[data-structures]]></category>
            <dc:creator><![CDATA[Elyadata]]></dc:creator>
            <pubDate>Thu, 22 Jun 2023 14:07:33 GMT</pubDate>
            <atom:updated>2023-06-22T14:10:34.128Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>This article aims to explain how Bloom filters work in details, provide illustrative examples, and discuss their real-world applications</p><h4><strong>The Data Dilemma: Efficiently Identifying Set Membership</strong></h4><p>In the realm of data management, we often encounter complex issues. One such problem pertains to quickly and efficiently determining if an element belongs to a set. Imagine developing an email service that requires the implementation of a spam filter. This filter must be capable of identifying and flagging emails from known spam senders. The question arises: how can your service rapidly, and on a large scale, discern if an incoming email originates from a spam sender?</p><p>The immediate resolution might be to compile all spam senders in a list or set, and then execute a lookup each time an email is received. However, this method possesses two inherent difficulties:</p><p>1. <strong>Space inefficiency</strong>: Hosting millions of senders would demand substantial memory resources.</p><p>2. <strong>Time inefficiency</strong>: Sifting through a list or set can prove tedious, particularly when handling vast data sets.</p><p>An innovative solution lies in the utilization of a probabilistic data structure, introduced by Burton Howard Bloom in 1970, known as the Bloom Filter.</p><h3><strong>Solution to the Problem</strong></h3><h4><strong>Harnessing the Power of Bloom Filters</strong></h4><p>Bloom filters serve as potent and compelling data structures utilized in an array of applications, particularly in areas where memory utilization needs to be optimized. The function is based on probabilistic principles, accepting a minimum probability of error in return for a significant reduction in memory usage. Bloom filters excel in verifying whether an element is part of a set, whether to determine the availability of a username, to validate if a file has been previously uploaded, or to address our initial problem of filtering spam emails.</p><h3><strong>Bloom Filters: A Deep Dive</strong></h3><h4><strong>What Exactly is a Bloom Filter?</strong></h4><p>In essence, a Bloom filter is a data structure that swiftly and memory-efficiently ascertains whether an element is part of a set. It is probabilistic, which implies that it provides two potential responses: the element definitely does not belong to the set, or it might be part of the set. This results in a “<strong>firm no</strong>” or a “<strong>probable yes</strong>”. Thus, while false negatives (the Bloom filter claiming an absent element is present) are non-existent, false positives (the Bloom filter claiming a present element is absent) are possible.</p><h3><strong>The Compromise: Trade-offs in Bloom Filters</strong></h3><h4><strong>The Essence of Probabilistic Data Structures: Trade-offs</strong></h4><p>Bloom filters exemplify a significant trade-off in software engineering. They accept the occasional incorrect false positive response in return for significant memory savings compared to other data structures like hash tables, which provide accurate answers but are more memory-intensive. Hence, Bloom filters are particularly suited to scenarios that can tolerate some false positives but cannot accept any false negatives. Note that once an element is added to a Bloom filter, it cannot be removed.</p><h3><strong>Mechanics of Bloom Filters</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FDWMLwhzvJIw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DDWMLwhzvJIw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FDWMLwhzvJIw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c708ce76b2dada76c90e34ac7dc771e6/href">https://medium.com/media/c708ce76b2dada76c90e34ac7dc771e6/href</a></iframe><h4><strong>Inside a Bloom Filter: How Does it Work?</strong></h4><p>A Bloom filter starts as an array of ‘m’ bits, all set to 0. It also has ‘k’ different hash functions, each capable of hashing a given input into one of the ‘m’ array positions. When an element is added to the Bloom filter, it is hashed by the ‘k’ hash functions, setting the bits at the resultant positions to 1. To verify if an element is part of the set, the Bloom filter hashes it with the same functions, checking if the bits at all output positions are 1. If they are, the Bloom filter states that the element might be part of the set. If any bit isn’t 1, the element definitely is not in the set.</p><pre>import mmh3<br>from bitarray import bitarray<br><br>class BloomFilter:<br>    def __init__(self, size, error_rate):<br>        &quot;&quot;&quot;<br>        Creates a Bloom filter with the given size and error rate.<br><br>        Args:<br>            size: The size of the Bloom filter in bits.<br>            error_rate: The desired false positive rate.<br>        &quot;&quot;&quot;<br>        self.size = size<br>        self.error_rate = error_rate<br>        self.hash_num = int(-(-size * math.log(error_rate)) / math.log(2))<br>        self.bit_array = bitarray(size)<br>        self.bit_array.setall(0)<br><br>    def add(self, item):<br>        &quot;&quot;&quot;<br>        Adds the given item to the Bloom filter.<br><br>        Args:<br>            item: The item to add.<br>        &quot;&quot;&quot;<br>        for i in range(self.hash_num):<br>            index = mmh3.hash(item, i) % self.size<br>            self.bit_array[index] = 1<br><br>    def __contains__(self, item):<br>        &quot;&quot;&quot;<br>        Checks if the given item is in the Bloom filter.<br><br>        Args:<br>            item: The item to check.<br><br>        Returns:<br>            True if the item is in the Bloom filter, False otherwise.<br>        &quot;&quot;&quot;<br>        for i in range(self.hash_num):<br>            index = mmh3.hash(item, i) % self.size<br>            if self.bit_array[index] == 0:<br>                return False<br>        return True<br></pre><h3><strong>Real-world Implementations of Bloom Filters</strong></h3><h4><strong>From Theory to Practice: Real-Life Implementations of Bloom Filters</strong></h4><p>Bloom filters are implemented in diverse real-world situations. For instance, many NoSQL databases use Bloom filters to minimize disk reads for non-existent keys. Content delivery networks like Akamai utilize Bloom filters to avoid caching ‘one-hit-wonders’ or web pages requested only once. Previously, web browsers like Chrome employed Bloom filters for identifying malicious URLs. In cybersecurity, password validators use a Bloom filter to deter users from selecting weak passwords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1E-0EmYOKksIt3jAAcvPiw.png" /></figure><h3><strong>In Summary</strong></h3><h4><strong>Bloom Filters: A Double-Edged Sword in Software Engineering</strong></h4><p>Bloom filters emerge as an influential tool in software engineering, offering a memory-efficient strategy to track elements in a set. Although they have limitations, such as potential false positives and inability to remove items, these drawbacks are often overshadowed by their benefits in many applications. As with many aspects of software engineering, understanding and effectively employing Bloom filters demands a careful consideration of trade-offs.</p><h3>Pushing Boundaries</h3><h4>An Exploration of Advanced Probabilistic Data Structures</h4><p>For new methodologies, we suggest to explore this research paper:</p><p>Zhang, H., Lim, H., Leis, V., Andersen, D.G., Kaminsky, M., Keeton, K. and Pavlo, A., 2018, May. Surf: Practical range query filtering with fast succinct tries. In <em>Proceedings of the 2018 International Conference on Management of Data</em> (pp. 323–336). <a href="https://www.cs.cmu.edu/~pavlo/papers/mod601-zhangA-hm.pdf">https://www.cs.cmu.edu/~pavlo/papers/mod601-zhangA-hm.pdf</a>. It</p><p>In this paper, the authors introduce a new data structure known as the Succinct Range Filter (SuRF), which supports both single-key lookups and range queries. It also offers tunable false positive rates, making it adaptable to different application requirements. The paper assesses the application of SuRF in RocksDB as a replacement for its Bloom filters, reducing I/O by filtering requests before accessing on-disk data structures.</p><h3><strong>Further Reading</strong></h3><p>1. Broder, A. and Mitzenmacher, M., 2004. Network applications of bloom filters: A survey. <em>Internet mathematics</em>, <em>1</em>(4), pp.485–509.</p><p>2. Wu, Y., He, J., Yan, S., Wu, J., Yang, T., Ruas, O., Zhang, G. and Cui, B., 2021. Elastic bloom filter: deletable and expandable filter using elastic fingerprints. <em>IEEE Transactions on Computers</em>, <em>71</em>(4), pp.984–991.</p><p>3. Patgiri, R., Nayak, S. and Muppalaneni, N.B., 2021, January. Is bloom filter a bad choice for security and privacy?. In <em>2021 International Conference on Information Networking (ICOIN)</em>(pp. 648–653). IEEE.</p><p>4. R. Xie, et al., “Hash Adaptive Bloom Filter,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece, 2021 pp. 636–647.</p><p>5. Nayak, S. and Patgiri, R., 2021, October. countBF: A general-purpose high accuracy and space efficient counting bloom filter. In <em>2021 17th International Conference on Network and Service Management (CNSM)</em> (pp. 355–359). IEEE.</p><p>6. Nayak, S. and Patgiri, R., 2021. robustBF: A high accuracy and memory efficient 2d bloom filter. <em>arXiv preprint arXiv:2106.04365</em>.</p><p>7. Vaidya, K., Knorr, E., Kraska, T. and Mitzenmacher, M., 2020. Partitioned learned bloom filter. <em>arXiv preprint arXiv:2006.03176</em>.</p><p>8. Yang, C., Tao, X., Zhao, F. and Wang, Y., 2020. Secure data transfer and deletion from counting bloom filter in cloud computing. <em>Chinese Journal of Electronics</em>, <em>29</em>(2), pp.273–280.</p><p>9. Luo, L., Guo, D., Ma, R.T., Rottenstreich, O. and Luo, X., 2018. Optimizing bloom filter: Challenges, solutions, and comparisons. <em>IEEE Communications Surveys &amp; Tutorials</em>, <em>21</em>(2), pp.1912–1949.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc186b80d08b" width="1" height="1" alt=""><hr><p><a href="https://insights.elyadata.com/demystifying-bloom-filters-a-probabilistic-method-for-efficient-data-querying-cc186b80d08b">Demystifying Bloom Filters: A Probabilistic Method for Efficient Data Querying</a> was originally published in <a href="https://insights.elyadata.com/">Elyadata</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>