<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bruno Messias on Medium]]></title>
        <description><![CDATA[Stories by Bruno Messias on Medium]]></description>
        <link>https://medium.com/@devmessias?source=rss-c18988aae45d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CJhQSKP2w5_GyCtnMSDUmw.jpeg</url>
            <title>Stories by Bruno Messias on Medium</title>
            <link>https://medium.com/@devmessias?source=rss-c18988aae45d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 02:18:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@devmessias/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reducing LangChain MMR Latency by 90% with Better Engineering]]></title>
            <link>https://medium.com/@devmessias/reducing-langchain-mmr-latency-by-90-with-better-engineering-865eef6fe767?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/865eef6fe767</guid>
            <category><![CDATA[numba]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[vector-search]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Wed, 20 Nov 2024 16:34:30 GMT</pubDate>
            <atom:updated>2024-11-20T18:06:02.126Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>With the surge of interest in AI, many new professionals are exploring Python and libraries like LangChain, often without fully understanding the trade-offs and costs associated with early-stage abstractions. Minor inefficiencies in these abstractions can significantly impact latency.In this post, I share some optimizations responsible for a 90% improvement in the latency.</blockquote><blockquote>This case serves as a study example, especially for newcomers, emphasizing the importance of analyzing library implementations before abandoning a method.</blockquote><p>When working at CGU with lawsuits and legislation, it is common for the database to contain many documents with highly similar chunks. This results in simple cosine similarity searches returning redundant chunks, increasing costs without adding value.</p><p>To make searches more relevant in contexts where the knowledge base has significant redundancy, a possible approach is to use <em>Maximal Marginal Relevance</em> (MMR). This ranking technique balances relevance and diversity in responses, selecting documents that are not only similar to the query but also show a degree of difference from each other. If you are not familiar with MMR, I recommend consulting specialized references, for example here:</p><p><a href="https://www.cs.bilkent.edu.tr/~canf/CS533/hwSpring14/eightMinPresentations/handoutMMR.pdf">https://www.cs.bilkent.edu.tr/~canf/CS533/hwSpring14/eightMinPresentations/handoutMMR.pdf</a></p><p>Lan Chu post: <a href="https://www.linkedin.com/posts/lanchuhuong_using-semantic-similarity-methods-like-cosine-activity-7243924873322860544-QxcH?utm_source=share&amp;utm_medium=member_desktop">https://www.linkedin.com/posts/lanchuhuong_using-semantic-similarity-methods-like-cosine-activity-7243924873322860544-QxcH?utm_source=share&amp;utm_medium=member_desktop</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NUsZm7in8MGei-tFupfrgQ.png" /><figcaption><a href="https://www.linkedin.com/posts/lanchuhuong_using-semantic-similarity-methods-like-cosine-activity-7243924873322860544-QxcH?utm_source=share&amp;utm_medium=member_desktop">https://www.linkedin.com/posts/lanchuhuong_using-semantic-similarity-methods-like-cosine-activity-7243924873322860544-QxcH?utm_source=share&amp;utm_medium=member_desktop</a></figcaption></figure><h3><strong>Challenge with LangChain Implementation</strong></h3><p>When using the standard LangChain implementation, I noticed that performance was extremely slow for my context. Analyzing the source code revealed certain issues that, after a reimplementation, resulted in cases of more than 90% reduction in response time — from 0.3 seconds to 0.001 seconds. This optimization made the use of MMR viable again.</p><p>Below, is the MMR langchain implementation</p><pre># langchain mmr implementation for all vector stores<br>def maximal_marginal_relevance(<br>    query_embedding: np.ndarray,<br>    embedding_list: list,<br>    lambda_mult: float = 0.5,<br>    k: int = 4,<br>) -&gt; List[int]:<br>    &quot;&quot;&quot;Calculate maximal marginal relevance.&quot;&quot;&quot;<br>    if min(k, len(embedding_list)) &lt;= 0:<br>        return []<br>    if query_embedding.ndim == 1:<br>        query_embedding = np.expand_dims(query_embedding, axis=0)<br>    similarity_to_query = cosine_similarity(query_embedding, embedding_list)[0]<br>    most_similar = int(np.argmax(similarity_to_query))<br>    idxs = [most_similar]<br>    selected = np.array([embedding_list[most_similar]])<br>    while len(idxs) &lt; min(k, len(embedding_list)):<br>        best_score = -np.inf<br>        idx_to_add = -1<br>        similarity_to_selected = cosine_similarity(embedding_list, selected)<br>        for i, query_score in enumerate(similarity_to_query):<br>            if i in idxs:<br>                continue<br>            redundant_score = max(similarity_to_selected[i])<br>            equation_score = (<br>                lambda_mult * query_score - (1 - lambda_mult) * redundant_score<br>            )<br>            if equation_score &gt; best_score:<br>                best_score = equation_score<br>                idx_to_add = i<br>        idxs.append(idx_to_add)<br>        selected = np.append(selected, [embedding_list[idx_to_add]], axis=0)<br>    return idxs</pre><p>In this post, I will focus on the optimization process, the decisions made, and the lessons learned. Python’s slowness is a well-known factor, and in <em>Retrieval-Augmented Generation</em> (RAG) applications or any system involving information retrieval, applying good engineering practices is just as essential as studying new techniques with LLMs. Therefore, being able to identify and apply these optimizations is crucial if you are starting in the field.</p><h3><strong>Exploring Trade-offs: Space vs. Time</strong></h3><p>When I start optimizing code, one of the first points I analyze is the possibility of trading execution time for memory. However, constraints should be analyzed. For example, a solution that requires terabytes of memory would not be feasible.</p><p>Inspecting LangChain’s code, I noticed that cosine similarity between all chunks and already selected chunks is recalculated in each <em>while</em> loop iteration. As the list of selected chunks grows, computational cost increases. For large values of <em>k</em> value this made the MRR impracticable for my scenario</p><pre># langchain implementation<br>while len(idxs) &lt; min(k, len(embedding_list)):<br>        #...<br>    similarity_to_selected = cosine_similarity(embedding_list, selected)</pre><p>MMR is typically applied after an initial filtering so the number of vectors is usually not large (generally less than 100). The <em>k</em> value can range from 20 to 60, and in some cases, it can be even larger, depending on the additional reranking process following MMR. Given this analysis, I concluded there is no good reason not to precompute all similarities, resulting in a maximum 100x100 matrix in the worst case. part of the performance gain came from this precomputation</p><p>But we can do better and look for other easy optimizations.</p><p><strong>Chunk Selection Verification and Native Python Functions Optimization</strong></p><p>One step of the MRR algorithm involves checking whether a chunk has been selected in previous iterations. The original code performs this verification by checking the existence of an integer in the index list, which grows with each iteration and has a maximum size of k. This has O(N) complexity, and note that this comparison is inside nested iterators. This is bad!</p><pre>for i, query_score in enumerate(similarity_to_query):<br>    if i in idxs:<br>        continue</pre><p>A straightforward optimization is to use a boolean array, where arr[i]=True indicates that the i-th chunk has already been selected. indicates that the i-th chunk has already been selected. This approach reduces the comparison complexity to O(1). We will incorporate this improvement into the Numba implementation.</p><p>Another alternative is to use idxs not to break the iteration but rather to filter the elements in the array to be processed. This will become clearer when you review my implementation.</p><p><strong>Avoid Native Python Functions</strong></p><p>Avoiding native Python functions like max can improve performance, as these functions are significantly slower compared to their NumPy counterparts. Moreover, broadcasting in NumPy allows for implicit parallelization of operations, which can yield additional performance gains — <em>provided</em> your CPU has a sufficient number of threads to support this parallel execution.</p><h4>NumPy implementation</h4><p>Putting everything together, we have a fast implementation of the same algorithm used by LangChain that is even simpler. If you’re planning to have a long career using Python, take some time to study NumPy broadcasting and learn how to work with indexing and slicing effectively.</p><p><a href="https://numpy.org/doc/stable/user/basics.broadcasting.html">https://numpy.org/doc/stable/user/basics.broadcasting.html</a></p><pre>def fast_max_marginal_relevance(<br>    query_embedding: np.ndarray,<br>    embedding_array: np.ndarray,<br>    lambda_mult: float = 0.5,<br>    k: int = 30,<br>)-&gt; list[int]:<br>    &quot;&quot;&quot;Calculate maximal marginal relevance.&quot;&quot;&quot;<br>    size_list = min(k, embedding_array.shape[0])<br><br>    if min(k, size_list) &lt;= 0:<br>        return []<br><br>    # Ensure query_embedding is 2D<br>    query_embedding = query_embedding.reshape(1, -1)<br><br>    similarity_to_query = cosine_similarity(query_embedding, embedding_array)[0]<br><br>    most_similar = np.argmax(similarity_to_query)<br>    idxs = [most_similar]<br><br>    # Precompute cosine similarities between all embeddings<br>    similarity_to_all = cosine_similarity(embedding_array)<br><br>    for _ in range(1, size_list):<br>        # Compute similarity to selected embeddings<br>        similarity_to_selected = similarity_to_all[:, idxs]<br><br>        redundant_scores = np.max(similarity_to_selected, axis=1)<br>        scores = lambda_mult * similarity_to_query - (1 - lambda_mult) * redundant_scores<br><br>        # Mask already selected indices<br>        scores[idxs] = -np.inf<br><br>        # Select the index with the best score<br>        idx_to_add = np.argmax(scores)<br>        idxs.append(idx_to_add)<br><br>    return idxs</pre><p>To evaluate the performance of this implementation, I created a grid of parameters, including kand the number of chunks (size). For each point on the grid, I conducted 100 random samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*90-ZzM7Bt8m72YUY3jGT3g.png" /><figcaption>The langchain implementation is very slow. For my case, 90% slower.</figcaption></figure><h4>Just-in-time compilation and numba to rescue</h4><p>The fast_max_marginal_relevance is fast due NumPy implementations. However, if your code is running inside a small pod or the algorithm cannot have an efficient parallelized implementation, relying solely on NumPy may not save you from Python’s inherently slow iterators. One way to overcome this bottleneck is using Numba.</p><p>If you’re new to Python, it’s worth taking some time to learn about Just-in-Time (JIT) compilation and Numba. Numba allows you to create functions that rival the performance of C implementations without sacrificing much of Python’s simplicity. Below is my current Numba implementation:</p><p><em>Note: While you can use NumPy calls inside a Numba-decorated function, the compatibility isn’t always 1:1. In some cases, some NumPy features may not be fully supported by Numba’s JIT compilation.</em></p><pre>@njit()<br>def numba_max_marginal_relevance(<br>    query_similarities: np.ndarray,<br>    precomputed_similarities: np.ndarray,<br>    lambda_mult: float,<br>    k: int<br>)-&gt; np.ndarray:<br>    num_docs = query_similarities.shape[0]<br>    # Pre-allocate arrays with specific types<br>    selected_indices = np.zeros(k, dtype=np.int64)<br>    # Initialize with first best match<br>    selected_indices[0] = np.argmax(query_similarities)<br><br>    # Track selected documents using boolean array for faster lookup<br>    is_selected = np.zeros(num_docs, dtype=np.bool_)<br>    is_selected[selected_indices[0]] = True<br><br>    for i in range(1, k):<br>        max_score = -np.inf<br>        best_index = -1<br>        for idx in range(num_docs):<br>            if is_selected[idx]:<br>                continue<br><br>            diversity_term = np.max(precomputed_similarities[idx, selected_indices[:i]])<br>            mmr_score = lambda_mult * query_similarities[idx] - (1 - lambda_mult) * diversity_term<br><br>            if mmr_score &gt; max_score:<br>                max_score = mmr_score<br>                best_index = idx<br><br>        selected_indices[i] = best_index<br>        is_selected[best_index] = True<br><br>    return selected_indices</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*8lFE21-l4A8cM2RM9znnAw.png" /></figure><p>The Numba implementation performs slightly faster than Python+NumPy for large values of k, and for smaller values, you can still gain a few percentage points in performance. Keep in mind that these metrics depend on the computational resources your retrieval system is running on.</p><p>I hope this post demonstrates the importance of not relying solely on pre-built implementations. More importantly, spend time analyzing your code before deciding to abandon a technique just because of high latency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=865eef6fe767" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Anna Karenina Principle in Code Quality: Addressing PySpark Challenges with PyASTrX]]></title>
            <link>https://medium.com/@devmessias/the-ana-karenina-principle-in-code-quality-addressing-pyspark-challenges-with-pyastrx-db402aa95efa?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/db402aa95efa</guid>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[flake8]]></category>
            <category><![CDATA[pysaprk]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 11:01:42 GMT</pubDate>
            <atom:updated>2024-09-30T12:52:53.967Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Lwln994XrrqG7EZtpSOpA.png" /><figcaption>How pyastrsx solve withColumn bad practice</figcaption></figure><p><a href="https://www.linkedin.com/feed/update/urn:li:activity:7243601258987606016/">In a recent post, Sem Sinchenko</a> explained that using pyspark withColumn call within a loop can lead to a huge drop in performance. This motivated me to write this post, not to address this specific bad practice, but the challenge of constant identifying and blocking bad practices using the current tools for code quality that can make you and your team an unhappy group. <strong>A bad pattern in your code could consume so many computational resources that your superb machine learning model might not save as much money as expected. Your ROI is reduced by bad practices that aren’t (and shouldn’t be) covered by flake8.</strong></p><p>In every project and company, I’ve worked with each project was unique. Due to the libraries used or even the culture of the engineers or data scientists involved. As a result, the definition of “unhappy” code varies significantly. However, <strong>linters — the tools designed to enforce code quality — are often too rigid to easily adapt to the specific needs of each project.</strong></p><p>Let me explain as an example:</p><ul><li>First, you identify a pattern of code that shouldn’t be considered good in your context. In that case</li><li>If you don’t have a PyPi account, create one. Maybe you need to use a internal repository at your company</li><li>If don’t know about the standard library ast in python, is time to learn it.</li><li>Write a walker class to navigate into the AST that should reach the unhappy pattern of code (Don’t try to understand this phrase, is just to show you how complicated the stuff could be)</li><li>Create a library based on Flake8 specifications.</li><li>Publish the lib</li></ul><p>In the end, you and your team are dealing with an additional package with hundreds of lines just to check one specific patter of bad practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*obdJ-BTMOlrBmTb3-varPQ.png" /><figcaption>I’m not advocating to abandon flake8, that doesn’t make any sense.<br>Consider giving a star for Sinchenko’s flake8 extension and pyastrx repo.<a href="https://github.com/SemyonSinchenko/flake8-pyspark-with-column">https://github.com/SemyonSinchenko/flake8-pyspark-with-column</a></figcaption></figure><p><strong>Ok, now if you’re works like me that constants shift between projects from software engineering, data science and AI imagine doing that for each bad practice. This scales well? Obviously not</strong>. This is the reason that I developed a tool assumes that each project will be happy in its own way, and this new bad pattern of pysaprk can be found and blocked in a project just adding the following lines in a .pyastrx.yaml file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YKynFbkvwgJQcjEl" /><figcaption>pyastrx spec to detect withColumn bad practice. If inside of the scope <strong>(body)</strong> of a loop<strong> (For)</strong> there are any function <strong>(func) </strong>calls <strong>(Call)</strong> which the attribute that renames a column <strong>(@attr=’withColumn’ or @attr=’withColumnRenamed’) </strong>we should raise a <strong>warning</strong></figcaption></figure><p>And it’s done! Everything is handled with just a few lines, including the metadata of this pattern.</p><p>You might argue that fewer lines of code don’t necessarily mean less cognitive load. But if you think that, let me show how using XML representations of Python’s AST (which is the way that pyastrx works) has significantly lower cognitive load, let’s put in a phrase the problem identified by Sinchenko .</p><blockquote>If inside of the scope <strong>(body)</strong> of a loop<strong> (For)</strong> there are any function <strong>(func) </strong>calls <strong>(Call)</strong> which the attribute renames a column <strong>(@attr=’withColumn’ or @attr=’withColumnRenamed’) </strong>we should raise a <strong>warning</strong></blockquote><p>Can you get it? For me, to have a less cognitive load is to have an easier way to translate my problem statement into a language specification and XPATHs is universal way to navigate into a tree, in that case the AST tree!</p><p>PyASTrX is a tool which I developed for my use, can do much more stuff see more here: <a href="https://pyastrx.readthedocs.io/en/latest/">https://pyastrx.readthedocs.io/en/latest/.</a></p><p>I’m not going to end the post just yet — after all, like most people who works with Opens Source Software we make this just for fun, not for profit and for this reason let’s talk broader: philosophy, literature and software engineering.</p><h3>The Anna Karenina principle and Software Engineering</h3><p>The famous opening line of Tolstoy’s Anna Karenina book is “Happy families are all alike; every unhappy family is unhappy in its own way”, can be put inside the context of software engineering: “All simples projects are alike, but every project is complex in its own way”. For example, a project that deals with data pipelines with pyspark code, sklearn models and tests to detect data drifts and everything served using a fastapi application is incredibly complex. Remember that adding a withColumn in a piece of code could consume so much computational resource that somehow the amount of money that your superb machine learning model is saving, it’s reduced by just one bad practice that It’s not (and shouldn’t covered) by flake-8. This led us to the Anna Karenina Priciniple (AKP) in code quality and today complex software systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*WoK56F7zhvaJ8MNc" /><figcaption>Tolstoy’s Anna Karenina Principle</figcaption></figure><h4>The Anna Karenina Principle in Philosophy and Complexity</h4><p>The AKP finds parallels in various fields, from philosophy to statistics and catastrophe theory. It embodies the idea that success (whether in life, in statistics or in a complex system design) depends on multiple factors aligning harmoniously. However, just one failure of a misaligned element, like a withColumn in the wrong place, disrupts the entire system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*Z4Yq22i1UH8VLSra" /><figcaption>AKP in statistics</figcaption></figure><p>We, as developers, data scientists, AI engineers, etc. When working in a software system we must balance performance, readability, scalability, code quality, and cognitive load. For that reason, tools and systems in general that resonates with AKP principle can make our projects better.</p><p>Consider how this principle applies to bad coding practices. A tool like PyASTrX (as introduced earlier) embodies a solution to the problem of generality in code quality enforcement by allowing project-specific patterns to be blocked or highlighted. The simplicity of adding a few lines in a `.pyastrx.yaml` file to define unwanted patterns demonstrates how AKP-thinking can help teams focus on project-specific pain points. Unlike traditional linters that enforce rigid, one-size-fits-all rules, PyASTrX provides flexibility, recognizing that what is considered “bad code” varies widely from one project to another.</p><h3>The Future of Code Quality: Embracing the Uniqueness of Each Project</h3><p>The key lesson from the Anna Karenina Principle in software is that there is no universal solution to maintaining high code quality. Each project, with its unique mix of business requirements, libraries, and team dynamics, demands tailored solutions. In this sense, tools that allow customization and flexibility, like PyASTrX, align with the principle by addressing the specific factors that lead to project success or failure.</p><h4>Final notes:</h4><ul><li>I’m not advocating to abandon flake8, that doesn’t make any sense.</li><li>Consider giving a star for Sinchenko’s flake8 extension and pyastrx repo.</li><li><a href="https://github.com/SemyonSinchenko/flake8-pyspark-with-column">https://github.com/SemyonSinchenko/flake8-pyspark-with-column</a></li><li><a href="https://github.com/pyastrx/pyastrx">https://github.com/pyastrx/pyastrx</a></li><li>I’m open to discussions or talks. As I said, this is a tool that fits very well for me. If this way doesn’t fit with your way of working, that’s perfectly fine.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/0*wU6bylSDSfE52EM9.png" /><figcaption>looking for possible mistakes, bad practices or code smells can be a pain, so PyASTrX provides a human-friendly output as default.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4bq4kFMpQfVgoSXD.png" /><figcaption>pyastrx friendly interface</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db402aa95efa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System design e IA: quando, qual e como utilizar um rerank? Parte I]]></title>
            <link>https://medium.com/@devmessias/system-design-e-ia-quando-qual-e-como-utilizar-um-rerank-parte-i-f706e9e100e9?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f706e9e100e9</guid>
            <category><![CDATA[rerank]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[cohere]]></category>
            <category><![CDATA[llmops]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Fri, 27 Sep 2024 14:08:58 GMT</pubDate>
            <atom:updated>2024-09-27T14:15:17.972Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72jpPgCgHjRWC8aSfWtaLg.png" /></figure><p>Uma coisa comum que percebo entre novos profissionais que trabalham com sistemas que integram IA (sejam engenheiros ou cientistas de dados) é não considerar as decisões sobre modelos e seus impactos em termos de engenharia e produto. <strong>Decisões técnicas (seja um if ou um novo gpt) tomadas sem uma análise de dados adequada, resultam em problemas no design do sistema. </strong>Atualmente trabalho com IAs no contexto jurídico, especificamente para CGU, e decisões ruins tem um impacto alto pois até um pouco de latência significa reduzir o desempenho de funcionários altamente capacitados e com uma demanda muito alta de trabalho.</p><p>Neste post, vou discutir como analisar a adoção de um rerank em um sistema de Recuperação de Respostas (RAG) sob a perspectiva de design de sistemas. Isso pode ser útil tanto para entrevistas quanto para otimizar o desempenho de sistemas que você já utiliza. As principais perguntas que abordarei são:</p><ul><li>As interações dos meus usuários justificam a <strong>complexidade adicional do rerank?</strong></li><li><strong>Qual modelo</strong> de rerank devo utilizar?</li><li><strong>Como a mudança</strong> de modelo<strong> impactará o sistema? Os thresholds e fluxos permanecerão?</strong></li><li>O rerank está gerando <strong>mais custos, mas </strong>melhorando a experiência do usuário (UX) ?</li><li>Está reduzindo custos e melhorando a UX ao mesmo tempo?</li></ul><p>Levantar essas perguntas desde o início é crucial, porque tecnologias populares podem consumir recursos sem gerar o retorno esperado — ou até piorar a experiência dos usuários. Decisões de engenharia são cruciais, às vezes mais importantes do que ter acesso ao modelo mais recente com boas pontuações, e para fazer essas decisões você precisa metrificar.</p><h4>O Problema do Contexto em LLMs</h4><p>Em sistemas que utilizam LLMs, o contexto é um recurso caro. O aumento da quantidade de contexto processado não apenas piora a latência, mas também eleva a chance de o modelo gerar respostas incorretas ou “alucinações” (ao adotar um novo LLM busque sempre as métricas de <strong>needle-in-a-haystack</strong>). Embora a abordagem mais comum — buscar os vetores mais similares ao input do usuário com base na similaridade semântica dos embeddings — resolva boa parte dos casos, nem sempre é suficiente. Isto é, a similaridade semântica entre a pergunta de um usuário e alguns chunks não garante que esses chunks sejam úteis para responder à pergunta.</p><p>A imagem a seguir ilustra isso de forma mais clara:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*if3njZF6vzNdVuwWYr3jEA.png" /><figcaption>Imagem extraída de <a href="https://arxiv.org/pdf/2405.19893">https://arxiv.org/pdf/2405.19893</a></figcaption></figure><p>Agora mesmo que a similaridade semântica seja suficiente para seus usuários, você pode encontrar casos em que<strong> múltiplos chunks possuem pontuações muito próximas. Isso torna difícil escolher um corte adequado no contexto, especialmente se estiver utilizando um threshold simples. Thresholds e fluxos são decisões de negócio que impactam todo o produto.</strong> Por exemplo, você pode configurar seu sistema para redirecionar a pergunta a um humano quando nenhum chunk atinge uma similaridade superior a 0.5. Resumindo os principais problemas que surgem quando o “R” do seu RAG não está suficientemente otimizado:</p><ul><li><strong>Custo</strong>: O processamento de contexto excessivo aumenta o consumo de tokens e os custos de infraestrutura.</li><li><strong>Latência</strong>: Contextos maiores geram tempos de resposta mais longos, o que prejudica a experiência do usuário.</li><li><strong>Confiabilidade</strong>: O sistema pode se tornar menos eficaz, tentando encontrar a “agulha no palheiro” sem sucesso.</li></ul><p><strong>O Trade-off de Acurácia vs. Performance</strong></p><p>Em tutoriais e mesmo consultorias você pode se deparar com a seguinte solução para um sistema para reduzir todas essas dores que discutimos acima ao usar somente similaridade semântica</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7_jvynUceWmK_Xgp34M9Q.png" /></figure><p>Um dos primeiros passos para projetar um sistema eficiente é entender problema que você está tentando resolver, considerando principalmente:</p><ul><li><strong>Quem são</strong> seus usuários e como eles interagem com o sistema;</li><li><strong>A escala</strong> do sistema;</li><li>O <strong>desempenho</strong> esperado;</li><li>Os <strong>custos</strong> envolvidos.</li></ul><p>Se você sair implementando tecnologias ou bibliotecas você pode acabar empilhando problemas.</p><p><strong>Observabilidade e Métricas</strong></p><p>Para monitorar o desempenho do seu sistema RAG, é essencial registrar as similaridades dos chunks utilizados no contexto, assim como dos chunks ideais que deveriam ter sido selecionados. Essa última observação só é possível se você tiver um dataset curado.<strong> Se você ainda não mantém registros dessas métricas de similaridade, o mais adequado é investir tempo na implementação desses logs, em vez de tentar aplicar um rerank sem dados para embasar suas decisões.</strong></p><p>Supondo que você tenha um dataset curado, com perguntas e a marcação dos chunks que deveriam conter a resposta, é possível analisar os logs da aplicação para identificar os diferentes cenários que ocorrem. Dependendo do sistema, você pode observar uma mistura de casos, como os que serão apresentados a seguir.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*N6qyMoIdfwjdgzyj9cet_g.png" /><figcaption>(<em>Os chunks com a informação necessária para responder à questão estão marcados em negrito e sublinhado.</em>)</figcaption></figure><p>Vamos analisar cada caso</p><p><strong>Case 1 (ótimo):</strong></p><p><em>Retrieval:</em></p><ul><li>Similaridades de topo altas</li><li>Similaridades de fundo baixas</li><li>Chunks corretos estão no topo</li></ul><p><em>Corte por threshold:</em></p><ul><li>O threshold é efetivo para cortar os chunks irrelevantes</li></ul><p><em>Usuário:</em></p><ul><li>Recebe a resposta correta</li><li>Baixa latência (contexto pequeno)</li></ul><p><em>Custo:</em></p><ul><li>Econômico: você economizou tokens</li><li>O usuário está satisfeito</li></ul><p><strong>Case 2 (bom):</strong></p><p><strong>Retrieval:</strong></p><ul><li>Similaridades de topo altas</li><li>Similaridades de fundo também altas</li><li>Chunks corretos estão no topo</li></ul><p><em>Corte por threshold:</em></p><ul><li>O threshold é inefetivo</li></ul><p><em>Usuário:</em></p><ul><li>Provavelmente recebe a resposta correta</li><li>O contexto grande pode causar o problema da “agulha no palheiro”</li><li>Latência alta devido ao tamanho do contexto</li></ul><p><em>Custo:</em></p><ul><li>Tokens desnecessários geraram aumento de custos</li><li>O usuário pode estar satisfeito, mas houve ineficiência</li></ul><p><strong>Case 3 (ruim):</strong></p><p><em>Retrieval:</em></p><ul><li>Similaridades de topo baixas</li><li>Similaridades de fundo baixas</li></ul><p><em>Corte por threshold:</em></p><ul><li>O threshold foi efetivo para cortar os chunks irrelevantes</li></ul><p><em>Usuário:</em></p><ul><li>Não recebe a resposta correta</li><li>Baixa latência (contexto pequeno)</li></ul><p><em>Custo:</em></p><ul><li>Econômico: você economizou tokens</li><li>No entanto, o usuário não está satisfeito, pois a resposta correta não foi fornecida, mesmo que estivesse disponível</li></ul><p><strong>Case 4 (terrível):</strong></p><p><em>Retrieval:</em></p><ul><li>Similaridades de topo altas</li><li>Similaridades de fundo baixas</li><li>O chunk correto está mais abaixo</li></ul><p><em>Corte por threshold:</em></p><ul><li>O threshold foi efetivo, mas cortou o chunk correto</li></ul><p><em>Usuário:</em></p><ul><li>Recebe a resposta errada</li><li>Baixa latência (contexto pequeno)</li></ul><p><em>Custo:</em></p><ul><li>Desperdício de tokens, pois foram utilizados para retornar uma resposta incorreta</li><li>O usuário pode assumir que a resposta está correta, o que é perigoso</li><li>É necessário implementar mecanismos para evitar o envio de respostas erradas</li></ul><p>A predominância de cada caso vai depender de diversas variáveis:</p><ul><li><strong>Modelo de embedding usado, idioma e contexto da sua aplicação</strong>: Por exemplo, em contextos jurídicos, embeddings podem ter um desempenho inferior, gerando vetores muito próximos uns dos outros. Isso dificulta, ou até inviabiliza, o projeto de um sistema com uma regra de corte simples, como um score mínimo de similaridade.</li><li><strong>Tipo de usuário majoritário e base de conhecimento</strong>: Se você está indexando um FAQ e uniu corretamente tanto a pergunta quanto a resposta antes de realizar o embedding para cada item, uma busca por similaridade pode ser suficiente para atender às necessidades do usuário.</li><li><strong>Tipo de técnica para quebra dos textos</strong>: A maneira como você divide os textos pode impactar diretamente a precisão do sistema de retrieval, influenciando a quantidade de tokens processados.</li><li><strong>Complexidade da fase de retrieval</strong>: Se o seu sistema envolve etapas mais complexas, como filtragem por metadados ou técnicas como text2query, o contexto utilizado na fase de inferência do modelo pode ser pequeno o suficiente para que os ganhos de uma melhor ordenação dos chunks não justifiquem o aumento de latência e complexidade.</li></ul><h4>Quando e qual Rerank vale a pena? Thresholds duros e suaves</h4><p>Agora que o problema já está claro, e você acredita que o rerank pode ser útil, surgem outras perguntas:</p><p>- Como posso inserir isso na arquitetura impactando o mínimo possível a latência?</p><p>- Como convencer os PMs a investirem mais dinheiro?</p><p>Para responder a essas perguntas, vou trazer um caso real que analisei. Eu tinha aproximadamente 100 questões de usuários com os chunks com a informação necessária marcados. Para cada questão, salvei o texto de cada chunk e a similaridade cosseno. Nessas questões, a quantidade de tokens no contexto estava, em média, em torno de 8000 tokens, e os chunks não possuíam tamanho uniforme.</p><p>A primeira coisa que podemos avaliar é o corte do contexto usando diferentes modelos de rerank. Nesse caso, utilizei três modelos: Cohere Multilingual, InRanker Base e InRanker Small. Os dois últimos são abertos e foram criados pela UNICAMP.</p><p>Para cada pergunta e seu respectivo conjunto de chunks, avaliei os scores obtidos por cada rerank. O que vamos analisar é: dado um threshold para o score do rerank, quantos porcento dos tokens que iriam para o contexto podem ser eliminados? A seguir, temos o gráfico que ilustra esses resultados.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*jh7MujlbnebpJ-ECg5rhCQ.png" /></figure><p>Com tudo isso em mãos, podemos pensar de forma mais criteriosa nas decisões de engenharia e negócios.</p><p>O Cohere Multilingual é um modelo proprietário e, dependendo do seu caso de uso, pode ser mais caro do que utilizar um modelo InRanker hospedado em uma VM. No entanto, surgem os trade-offs.</p><p>Nesse caso, o Cohere apresentou um corte de contexto mais significativo. O InRanker Base, mesmo com um threshold próximo de 0.9, mal atinge a porcentagem de corte mínimo obtida com o Cohere. Como nosso dataset é curado, eu pude avaliar que com um threshold próximo de 75% temos uma boa margem de segurança para manter os chunks com a informação necessária dentro do contexto. Isso tem algumas<strong> implicações no design do sistema:</strong></p><ul><li>Embora o Cohere possa ser mais caro por chamada e introduza maior latência, ele proporciona um ganho substancial no corte do contexto, com thresholds mais duros.</li><li>Com thresholds mais duros e um contexto menor, o dinheiro extra que você pode estar gastando com o reranker da Cohere pode ser compensado por:</li><li>Economia de tokens gastos na inferência do seu modelo.</li><li>Redução da latência na inferência do seu modelo de linguagem.</li></ul><p><strong>O gráfico acima pode ser útil para orientar um time técnico, mas como ter uma visão financeira?</strong> Uma possibilidade seria traçar um mapa da porcentagem de tokens economizados com base no tamanho médio do contexto sem a regra de corte, e colorir por economia financeira, levando em consideração uma regra específica. Para simplificar, utilizei números fictícios de consumo diário de tokens e os custos de tokens por milhão do GPT-4o e do custo por chamada reranker Cohere Multilingual.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRIZFZjEG6A2Wi1HUg6oSA.png" /></figure><p>Novos insights:</p><ul><li>Quanto menor o contexto enviado aos LLMs, maior deve ser a porcentagem de tokens economizada por meio de algum critério de corte que você aplique no rerank.</li><li>Mesmo que o gráfico pareça promissor, lembre-se do “Case 1”. Mesmo que o rerank gere um custo pequeno em cenários pessimistas, o aumento de latência ainda pode resultar em uma experiência desagradável para o usuário, afetando outras métricas importantes do seu negócio.</li></ul><p>Agora você pode reavaliar seu sistema. Por exemplo, em casos onde a pergunta gera um contexto muito pequeno devido a um threshold baixo de similaridade ou à ausência de filtragem por metadados, não há motivos para aplicar um rerank. Uma alternativa seria redesenhar seu sistema nesses cenários.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qej3f2wMAM1SxOlkmT8W6w.png" /></figure><p>Esta foi a primeira parte sobre design de sistemas envolvendo rerank e como ponderar algumas decisões. Na próxima, discutiremos métricas de qualidade para o ordenamento dos chunks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f706e9e100e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic-first: Ontology and Taxonomy as Foundations in dbt models Development]]></title>
            <link>https://medium.com/@devmessias/semantic-first-ontology-and-taxonomy-as-foundations-in-dbt-models-development-d14aa85d2bc1?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d14aa85d2bc1</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 19:40:32 GMT</pubDate>
            <atom:updated>2023-11-27T19:54:23.973Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is an English translation of an </em><a href="https://medium.com/r?url=https%3A%2F%2Fengineering.idwall.co%2Fsemantic-first-ontologia-e-taxonomia-como-pilares-no-desenvolvimento-de-pipelines-de-dados-3bbf40a68750"><em>article I originally wrote in Portuguese on October 11 for IdWall</em></a><em>, the company I work for. I have made several modifications to enhance the content and adapted the analogies to better suit an English-speaking audience</em></p><blockquote>Semantic-first is a methodology that was successfully implemented in the creation of one of our largest data pipelines. Thanks to the Semantic-first approach, we were able to accelerate testing and optimizations, minimize communication failures, and ensure an efficient evolution of the system. This approach also allowed new team members to become familiar with and contribute quickly to the pipeline.</blockquote><p>Many refer to data as the “new oil.” And in many respects, the comparison is valid. Just as oil is transported and transformed through pipelines, data is also moved and shaped by digital pipelines. However, this analogy has its limits.</p><p>While oil, once refined, has a limited set of derivatives, data has an endless array of interpretations, applications, and restrictions. And just as a poorly designed pipeline can result in disastrous leaks, a poorly constructed data pipeline can lead to misinterpretations, misguided decisions, and costs for an organization. In the context of LGPD (Brazil’s General Data Protection Law), a mistake in data management can result in lawsuits, damage to reputation, and loss of customer trust.</p><p>The complexity of data demands a careful approach, where semantics and consequently ontology and taxonomy play crucial roles to ensure that data is transported, understood, and utilized correctly, as per data protection regulations.</p><p>This semantic-first approach, besides allowing quick data integration, ensures its correct interpretation. Utilizing metaprogramming, libraries can translate the knowledge represented about the data and data pipelines directly into dbt models with metadata, tests, and optimizations being generated automatically.</p><p>Furthermore, this approach easily allows coupling new features like automatic updates in Confluence, language models, and any system that wants to take advantage of these representations. In summary, semantic-first optimizes maintainability, reduces cognitive load, and ensures fluidity in integrating new tables and information sources.</p><h3>Communication Failures Due to Implicit Knowledge and Their Consequences</h3><p><strong><em>What does it mean to know something?</em></strong></p><p><strong><em>What makes a thing what it is?</em></strong></p><p>At first glance, these inquiries might seem more suited for a philosophical debate, not a discussion on engineering and data analysis. However, upon careful reflection, you may realize their importance when building pipelines.</p><p>To illustrate, imagine a situation involving two characters, John and his son Tim. In a text conversation, Tim informs John that he found a bat in his room. However, instead of thinking about the animal, John imagines Tim is talking about a baseball bat and advises him to just put it in the closet. Tim, who is an obedient child, takes the advice literally and cautiously places the live bat in John’s wardrobe. Later that evening, John experiences a flying surprise when he opens his wardrobe to pick out a shirt for dinner. The encounter leaves him in stitches of laughter upon realizing the miscommunication, and a funny family story to tell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*STh5xyJ6TcNCez8S" /><figcaption><em>Image by author generated by Dalle-3 using ChatGPT</em></figcaption></figure><p>This humorous misunderstanding, related to the exchange of information, illustrates how implicit knowledge can create potentially hazardous situations. Both Tim and John exchanged information with each other, communicating the state of a thing, names, and actions, but each had a different mental map about the things they were talking about.</p><blockquote>“The limits of my language mean the limits of my world” — Ludwig Wittgenstein</blockquote><p>If confusions can arise in a simple conversation between people who have known each other for a lifetime, imagine, in an organization, where data is constantly shared among hundreds of tables and thousands of columns. Each column carries a meaning that can be interpreted in different ways by different people and systems.</p><p>In this context of communication failures, there exists an assertion called 80/20. This assertion states that data scientists spend only 20% of their time analyzing and creating models and 80% of the time searching, cleaning, and organizing data. So, at best, disorganized data wastes workforce. But even in this best case, we have other serious side effects:</p><ul><li><strong>How do we ensure that, during this cleaning and organization, analysts and data scientists adopted a consistent set of practices, taxonomy, and methods?</strong></li><li><strong>How do we ensure that all data consumers and producers are referring to the same “bat”?</strong></li></ul><h3>The Pitfalls of Code-First Approaches in Data Pipelines</h3><p>When building data pipelines, an approach that many teams might adopt is to pick a technology like dbt and start implementing transformations and models while simultaneously defining the taxonomy or simply ignoring it as new cards in JIRA are pulled. However, this practice of prioritizing code can lead to a series of challenges.</p><p>The technology/code-oriented methodology can lead to significant inconsistencies across different projects. Each project may adopt distinct practices and taxonomies, making it not only challenging to optimize and catalog data but also to maintain consistency and interoperability among different systems. This results in a fragmented data environment, where the lack of a common language prevents true integration and optimized utilization of data.</p><p>Moreover, the absence of a data meaning structuring from the beginning can result in additional maintenance and refactoring costs, as needs evolve and new requirements emerge, and people leave the company taking with them the implicit taxonomy stored only in their brains.</p><p>The lack of a standardized approach to knowledge representation prevents organizations from achieving the full potential of their data assets, as they are not accessible. According to<strong> [1],</strong> accessible data are:</p><p><strong>Accessible data = bits + semantics</strong></p><p>Remember the example of the bat and the conversation between Tim and John. All the headache would have been avoided if, before John proposed a solution, they had checked if they were aligned regarding the meaning of things. Of course, bureaucracy of this level for day-to-day conversations would make life very sad.</p><p>But in our data pipelines, this bureaucracy reduces communication failures and all the problems they generate. Therefore, it’s interesting that taxonomy and ontology are the pillars of processes involving data.</p><h3>Taxonomy</h3><p>In the data world, taxonomy organizes information into hierarchical structures, grouping concepts or entities by common characteristics. In a system suffering from the same misunderstanding as John and his son, we could have a rule that every string identifying an entity that is an animal should follow the class of this animal, in this case following the pattern thing+mammal, which would help in avoiding situations like the risk of a rabies contamination in the house leading to a catastrophe. With LGPD (Brazil’s General Data Protection Law), having a centralized and updated taxonomy is crucial, ensuring the correct classification of data and preventing legal violations such as the incorrect classification of data as PI/PII (Personal Information/Personally Identifiable Information). Moreover, it reduces rework and cognitive load, standardizing nomenclatures and descriptions for documentation, allowing for reuse even in API documentation using Swagger.</p><h3>Ontology</h3><p>Although ontology is an ancient field, it plays a central role in current technology. In the context of IT, ontology defines and models concepts, categories, and relations in a specific domain of knowledge, using specifications like RDF/OWL. This enables computers to “communicate” with each other while minimizing ambiguities. By establishing taxonomy and ontology as the foundations of your data pipelines, you enhance the precision with which the knowledge/information these pipelines aim to deliver is absorbed and shared.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*7sgOl77l0Xi_PhYv" /><figcaption>Image by author</figcaption></figure><p>One of the key features of an ontology is enabling disambiguation and automatic reasoning. As an illustrative example, consider the disambiguation of a object from a work of art. This demonstrates how ontology can distinguish between different interpretations or meanings of the same term or object, facilitating clearer understanding and automated logical processing.</p><h3>Semantic-first in Practice</h3><p>Structuring a data pipeline project with ontology and taxonomy as foundational pillars might seem daunting at first, especially when there is no central repository containing knowledge about the organization’s data, even if it’s just a few web pages or spreadsheets. However, consider the following excerpt from a old book:</p><blockquote><em>“The notion of representation of knowledge is at heart an easy one to understand. It simply has to do with writing down, in some language or communicative medium, descriptions or pictures that correspond in some salient way to the world or the state of the world. In Artificial intelligence, we are concerned with writing down descriptions of the world in such a way that an intelligent machine can come to new conclusions about its environment by formally manipulating these descriptions.” — Readings in Knowledge Representation, 1985 [4]</em></blockquote><p>Surprisingly, this excerpt is from a book that’s nearly four decades old! And although you might have come across a post on LinkedIn or heard a consultant claim that a graph database is essential for a robust and scalable knowledge repository, that is not always the case.<strong> Depending on the situation, a series of YAML files describing your data, combined with solid DevOps and/or MLOps processes, may be more than adequate. After all, graphs are about relationships. And if it makes no difference whether you represent these relationships through text, why worry?</strong> Thus<strong>, the main task is to define processes that allow the extraction of knowledge and meaning, which are often implicit — whether in human minds, Confluence pages, database systems, etc</strong>. The good news is that there are frameworks that help guide this journey in search of the meaning of things within an organization.</p><h3>What do we have similar to CRISP-DM for ontology?</h3><p>Just as CRISP-DM provides a guide for data projects, systematic approaches for the creation of knowledge representations have been developed. It is convenient to find and adapt the one that best fits your purpose and the set of tools and techniques you and/or your team are skilled in.</p><p>This can range from less formal methods with conceptual maps drawn on a whiteboard, widely used in teaching, to the development of knowledge graphs constantly feeding a database like RDF4J. A simplified flow in which knowledge (or ontology) can be built by teams is represented in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KJS2Ec1F9yTT2KhUrtXM6w.png" /><figcaption>Image by author</figcaption></figure><p>There is a discussion on how to systematically incorporate ontology development into teams that work with agile methodology. For more information, you can read:</p><p><a href="https://inass.org/2019/2019043017.pdf"><em>An Agile Methodology for Ontology Development</em></a></p><p><a href="https://www.w3.org/community/owled/files/2016/11/OWLED-ORE-2016_paper_6.pdf"><em>A Simplified Agile Methodology for Ontology Development</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*OxKh8k1Ynp0HCMed.png" /><figcaption>Diagram illustrating an agile approach for ontology development in an organization from [5]</figcaption></figure><p>It would not be feasible to describe all these systematic approaches to knowledge representation here, so I recommend the following books:</p><ul><li><strong>Designing and building enterprise knowledge graphs. </strong>This is my favorite book on KGs applied in the industry.</li><li><strong>Exploiting Linked Data and Knowledge Graphs in Large Organizations. </strong>It includes many real-world business cases and discussions about the adoption process.</li></ul><p>After the initial interactions in the process of knowledge extraction and representation, it is possible to start building semantic-first data pipelines.</p><h3>How to Build Your Knowledge Representations?</h3><p>If your organization does not yet have a systematized repository to represent knowledge about data, it’s time to create one!</p><p>As previously mentioned in this post, don’t get seduced by the latest trending technology. Conduct a Proof of Concept (PoC) with your team to assess the technical feasibility of using a graph database technology. If the PoC indicates that there is not enough maturity yet, simply discard this option.</p><p>A straightforward approach is to represent knowledge about the organization’s data in a series of text files. Considering that YAMLs are frequently used by engineers, analytic engineers, and data analysts, why not opt for them?</p><p>In the following visual representation, I present a system where knowledge representation is produced and applied to various tasks. Some components of this system are optional, giving you total freedom in the choice of technologies and how to implement them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYKm34qkmi4EJLdqPIRM8g.png" /><figcaption>Image by author</figcaption></figure><p>The initial step is always to represent the knowledge, the meaning, the semantics of our data that will be conveyed in the pipelines. For example, we can create a YAML file describing the columns that we will use in all the dbt models present in our pipeline:</p><pre># columns.yaml<br> cpf:<br>    name: CPF<br>    description: &quot;...&quot;<br>    type: &quot;STRING&quot;<br> relationship_type:<br>    name: &quot;Relationship Type&quot;<br>    description: &quot;....&quot;<br>     type: &quot;...&quot;<br>     accepted_values:<br>         - mother<br>         - ...</pre><p>and another describing the categorization of these columns:</p><pre># categories.yaml<br> contact:<br>   - phone_number<br>   - email<br> personal_financial:<br>   - income<br>   - related_companies<br>   - ....</pre><p>Using the RDF/OWL specification, the knowledge from the two aforementioned YAML files could be represented as follows:</p><pre>@prefix rdf: &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt; .<br>@prefix rdfs: &lt;http://www.w3.org/2000/01/rdf-schema#&gt; .<br>@prefix owl: &lt;http://www.w3.org/2002/07/owl#&gt; .<br>@prefix xsd: &lt;http://www.w3.org/2001/XMLSchema#&gt; .<br>@prefix ex: &lt;http://example.org/ontology#&gt; .<br><br><br># Definition of classes based on the categories.yaml file<br>ex:Contact a owl:Class ;<br>    rdfs:label &quot;Contact&quot; .<br>ex:PersonalFinancial a owl:Class ;<br>    rdfs:label &quot;Personal Financial&quot; .<br># Definition of properties based on the columns.yaml file<br>ex:cpf a owl:DatatypeProperty ;<br>    rdfs:label &quot;CPF&quot; ;<br>    rdfs:comment &quot;...&quot; ;<br>    rdfs:domain ex:Contact ;<br>    rdfs:range xsd:string .<br>ex:relationshipType a owl:ObjectProperty ;<br>    rdfs:label &quot;Relationship Type&quot; ; <br>    rdfs:comment &quot;....&quot; ;<br>    rdfs:domain ex:PersonalFinancial ;<br>    rdfs:range rdfs:Resource .<br># Accepted instances for the relationship type<br>ex:Mother a owl:NamedIndividual ;<br>    rdfs:label &quot;mother&quot; .<br># Add other instances as needed.<br><br># Subclasses for Contact and PersonalFinancial<br>ex:PhoneNumber a owl:Class ;<br>    rdfs:subClassOf ex:Contact ;<br>    rdfs:label &quot;Phone Number&quot; .<br>ex:Email a owl:Class ;<br>    rdfs:subClassOf ex:Contact ;<br>    rdfs:label &quot;Email&quot; .<br>ex:Income a owl:Class ;<br>    rdfs:subClassOf ex:PersonalFinancial ;<br>    rdfs:label &quot;Income&quot; .<br>ex:RelatedCompanies a owl:Class ;<br>    rdfs:subClassOf ex:PersonalFinancial ;<br>    rdfs:label &quot;Related Companies&quot; .</pre><p>The advantage of the RDF/OWL specification is that we can store our knowledge representation in a graph database such as RDF4J. And RDF/OWL graph databases have a bunch of cool features that are not found in regular databases, such as automatic reasoning. However, we will not discuss these advantages here, but I invite you to explore more about them.</p><h3>Representing Your Pipelines</h3><p>The next step involves representing the knowledge about our pipeline, that is, what we want to achieve with our pipeline.<em> ‘From which source should the data come? Where should it go? How will each column be mapped/transformed?’</em> By answering these questions and validating them with your team, you could, for example, represent a pipeline that transforms a source_table_name table into a std_my_new_table table, changing only the name of the CPF column and grouping name columns in the following way:</p><pre># kr/std/models.yaml<br>std_my_new_table:<br>  folder: folder_name # where the generated dbt files are saved<br>  table_src: source_table_name # name of the table from which data will be extracted<br>  description: &quot;A useful description of this table&quot;<br>  columns: # here we list the source columns and the mapped field<br>    cpf: CPF<br>    cpf_status: status_na_receita_federal # the source tables/sources may not be following a taxonomy<br>    full_name:<br>      - firstname<br>      - middlename<br>      - lastname</pre><h3>Ensuring Consistency in the Semantics of Pipelines</h3><p>If the knowledge representation of the data is formalized, we can create processes to check whether the pipelines respect the agreed-upon semantics or if such semantics are available in our knowledge repository. Here at Idwall, these tests can be easily executed using a target in the Makefile. To start, imagine you have a directory representing your pipelines as follows:</p><pre>kr/<br>│<br>├── LAYER1/<br>│   └── models.yaml<br>│<br>├── LAYER2/<br>│   └── models.yaml<br>│<br>└── ...</pre><p>A pseudo-code that could compose a CI/CD job to validate if all columns in the data pipeline could be:</p><pre>Function LoadYAML(file_path):<br>  Open file_path as file:<br>    Return the content of the file parsed as YAML<br><br>Function ValidateColumns(model, defined_columns):<br>  For each column in model[&#39;columns&#39;]:<br>    If column is not in defined_columns:<br>      Report error &quot;Undefined column&quot;<br><br>Function ValidateModels(model_path, defined_columns):<br>  model = LoadYAML(model_path)<br>  For each item in model[&#39;models&#39;]:<br>    ValidateColumns(item, defined_columns)<br><br>Function Main():<br>  defined_columns = LoadYAML(&quot;path/to/columns.yaml&quot;)<br>  For each LAYER in ListOfLayers:<br>    model_path = &quot;kr/&quot; + LAYER + &quot;/models.yaml&quot;<br>    ValidateModels(model_path, defined_columns)<br><br>Execute Main()</pre><p>However, if your organization has adopted a more formal way of knowledge representation, for example, using a KG that supports SPARQL, you could use the following query to validate the existence of the full_name column in your KG:</p><pre>ASK WHERE {<br>  ?instance a ex:Column;<br>            ex:hasName &quot;full_name&quot; .<br>}</pre><h3>Automatic Generation of dbt Schemas and Tests</h3><p>From these representations, we can create scripts that read these representations and automatically generate the schema.yml files for the dbt models, including information such as descriptions, data types, and tests. This is what we did here at Idwall!</p><p>To make it clearer, here is a basic pseudo-code that you can use to guide your project:</p><pre>Function LoadYAML(file_path):<br>    Open file_path as file:<br>        Return the file content parsed as YAML<br><br>Function GenerateDBTSchema(columns_def, categories_def):<br>    Initialize model_schema as a dictionary with keys &quot;name&quot; and &quot;columns&quot;<br>    <br>    For each column, column_info in columns_def:<br>        Initialize column_schema as a dictionary with keys &quot;name&quot;, &quot;description&quot;, and &quot;tests&quot;<br>        Add column name and description information for each column<br>        <br>        If type of column_info is &quot;string&quot;:<br>            Add test &quot;is_string&quot; to column_schema[&quot;tests&quot;]<br>        Else if type of column_info is &quot;bool&quot;:<br>            Add test &quot;is_boolean&quot; to column_schema[&quot;tests&quot;]<br>        # Add other type tests as needed<br>        <br>        If &quot;accepted_values&quot; is present in column_info:<br>            Add test &quot;accepted_values&quot; and the accepted values to column_schema[&quot;tests&quot;]<br>        <br>        Add column_schema to model_schema[&quot;columns&quot;]<br>    <br>    # Additional logic can be added to handle categories_def<br>    <br>    Add model_schema to dbt_schema[&quot;models&quot;]<br>    Return dbt_schema<br><br>Function SaveYAML(data, file_path):<br>    Open file_path as file:<br>        Write data as YAML in the file<br><br>Function Main():<br>    columns_def = LoadYAML(&quot;path/to/columns.yaml&quot;)<br>    categories_def = LoadYAML(&quot;path/to/categories.yaml&quot;)<br>    <br>    dbt_schema = GenerateDBTSchema(columns_def, categories_def)<br>    <br>    SaveYAML(dbt_schema, &quot;dbt_my_org/models/my_model/schema.yaml&quot;)<br><br>Execute Main()</pre><p>This approach has proven to be quite powerful for generating metadata and tests. <strong>For example, we managed to go from a pipeline with zero tests to over 500 in just one day! </strong>Moreover, we have also been able to generate complex and useful metadata today that would have taken dozens of hours or days to create and maintain manually</p><h3>Unlocking Metaprogramming: Automated Creation of dbt Models</h3><p>When classifying “animals”, we can observe that the “mammals” class is defined by properties like “have fur” and “nurse their young”. However, a rich classification goes beyond these intrinsic characteristics. For example, <strong>a lion is not just defined by its physical features but also by the ecological relationships it maintains, being a predator of many animals and prey of few.</strong></p><p>Now think about the case of data pipelines and cataloging. Why group tables only by the data they contain? Some tables are constructed by standardizing and grouping columns from other tables, while others arise from stacking different tables together.</p><p>Similar to the natural world, where it would be odd to always use the same collar on a wolf and a domestic dog, when organizing our pipelines, grouping tables considering only the characteristics of the data they carry is not always a good idea.</p><p>Imagine a pipeline that has only two layers, organized as follows:</p><ul><li>LAYER1 (e.g., std): focuses on standardization and aggregation of columns, ensuring that data are clean and in a consistent format. It has only a single table as a data source.</li><li>LAYER2 (e.g., stack): dedicates itself to stacking rows from different std tables</li></ul><p>Given that each table in the stack class “stacks” rows from other std layer tables, we are creating classes in our data ontology based on transformations, not content, although we could create sub-layers that divide by content. This proves extremely useful, for example, when programming in PySpark, we are always thinking about transformations.</p><p>Therefore, we can create a code that generates the code for these transformations, allowing us to metaprogram our pipeline. Suppose we want to create a table that stacks all the rows from all the tables that describe fruits, this could be represented by a file like this:</p><pre># kr/stacks/models.yaml<br>stack_fruits:<br>  regex_files: &quot;^std_fruit_&quot;<br>  description: &quot;Stacks all fruit tables&quot;</pre><p>dbt models can be written in both SQL and Python. The logic for a metaprogramming approach for this simple stacking example would be something like this, expressed in pseudo-code:</p><pre>Function CheckStdTableCond(model_name, regex_file):<br>    If model_name does not match regex_file:<br>        Return False<br>    Return True<br><br>Function GetStdTables(std_tables, regex_file):<br>    Initialize std_tables_filtered as an empty dictionary<br>    For each model_name, model in std_tables:<br>        If CheckStdTableCond(model_name, regex_file) is True:<br>            Add model_name and model to std_tables_filtered<br>    Return std_tables_filtered<br><br>Function InsertModelDef(tree, std_tables):<br>    Initialize std_tables_names with the keys of std_tables<br>    Create an AST function node to define a &#39;model&#39; function with empty parameters and body<br>    For each std_name in std_tables_names:<br>        Add nodes to the function body to create dbt references and rename columns<br>    Add a node to the function body to join all tables by name and assign the result to &#39;df&#39;<br>    Add a node to the function body to return &#39;df&#39;<br>    Add the function node to the body of the AST tree<br>    Return tree<br><br>Function GenCode(std_tables):<br>    Create an empty AST tree<br>    Insert dbt model definition into the AST tree using InsertModelDef<br>    Convert the AST tree into source code and return<br><br>Function Main():<br>    Load all std tables from a configuration<br>    For each stack_name, stack in configuration stacks:<br>        Get regex_files from stack<br>        Get corresponding std tables using GetStdTables<br>        Generate code using GenCode<br>        Save the generated code (sql or python) in the dbt models folder<br><br>Execute Main()</pre><p>If tomorrow, an analytics engineer who has never heard of dbt and just joined your organization wants to use a table that stacks tables they would simply need to declare a new section with three lines in the kr/stacks/models.yaml file.</p><p><strong>This approach of building code that translates simple specifications into optimized dbt models has allowed new team members to easily contribute to the pipeline at IdWall, integrating new data, updating metadata and transformations at the same time, ensuring very few violations of taxonomy standards.</strong></p><h3>The Future is a Dialogue with Your Pipelines: Integrating LLMs and OWL/RDF Pipelines</h3><p>Imagine a scenario — okay, maybe you don’t even need to imagine! — where your company operates with dozens of pipelines, some of them managing/creating hundreds of tables and totaling tens of thousands of columns. In this context, tricky questions can arise from PMs or new team members about the data: <em>“In which table can I find information X? What does column Y mean? What is the function of this MongoDB collection?” In even more complex situations, we might face questions like: “How can I track the origin of financial data imported from Credit Bureau A?”</em></p><p>To answer these crucial questions, many teams find themselves diving into the depths of a data catalog, navigating through a search box, checkboxes, colors, and popups in the hope of finding precise answers — assuming, of course, that such an organized and always up-to-date data catalog is available and easily accessible to team members, which is not always the case. Is it for you? If not, I’m sorry :(.</p><p>Now, think with me: if you adopted a semantic-first approach and formalized the knowledge about both the data and the transformations , and if you did this using a Knowledge Graph (KG), now you have the capability to dialogue with your pipelines. Utilizing the KG as a source of facts for an LLM, like ChatGPT, you can navigate and interact with your pipelines in an incredibly intuitive and informative way. Fascinating, isn’t it?</p><p>However, the text has already extended quite a bit, and delving deeper into this topic will be a journey for another time.</p><h3>References</h3><ul><li>[1]J. Sequeda e O. Lassila, <strong>Designing and building enterprise knowledge graphs</strong>. 2021. This is my favorite book on KGs applied in the industry.</li><li>[2]<strong> (português) Dados abertos conectados</strong>. Novatec Editora, 2022.<strong> Livro disponível gratuitamente </strong><a href="https://ceweb.br/livros/dados-abertos-conectados//prefacio/"><strong>aqui</strong></a><strong>.</strong></li><li>[3] J. M. Gomez-Perez, J. Z. Pan, G. Vetere, e H. Wu, Orgs., <strong>Exploiting Linked Data and Knowledge Graphs in Large Organisations</strong>, 1st ed. 2017. Cham: Springer International Publishing/ : Imprint: Springer, 2017.</li><li>[4] R. J. Brachman e H. J. Levesque, Orgs., <strong>Readings in knowledge representation</strong>. Los Altos, Calif: M. Kaufmann Publishers, 1985.</li><li>[5] Cairo University, A. Abdelghany, N. Darwish, Cairo University, H. Hefni, e Cairo University,  <strong>An Agile Methodology for Ontology Development</strong> , IJIES, vol. 12, no 2, p. 170 181, abr. 2019, doi: 10.22266/ijies2019.0430.17.</li><li>[6] Linked Data Engineering (Semantic Web) OpenHPI by Prof. Dr. Harald Sack <a href="https://www.youtube.com/playlist?list=PLoOmvuyo5UAfY6jb46jCpMoqb-dbVewxg">https://www.youtube.com/playlist?list=PLoOmvuyo5UAfY6jb46jCpMoqb-dbVewxg</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d14aa85d2bc1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic-first: ontologia e taxonomia como pilares no desenvolvimento de pipelines de dados]]></title>
            <link>https://medium.com/engineering-idwall/semantic-first-ontologia-e-taxonomia-como-pilares-no-desenvolvimento-de-pipelines-de-dados-3bbf40a68750?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bbf40a68750</guid>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[taxonomia]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[ontologia]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Wed, 11 Oct 2023 18:24:49 GMT</pubDate>
            <atom:updated>2023-10-12T23:13:34.981Z</atom:updated>
            <content:encoded><![CDATA[<p><em>No #dedevparadev do mês de Outubro, o Bruno Messias, Cientista de Dados na idwall, compartilha um pouco sobre a abordagem Semantic-First, a implementação da metodologia e como ela pode ajudar no desenvolvimento de pipelines de dados.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yB560KPsyUIQ6SSPeQ0AQ.png" /></figure><blockquote>Semantic-first é uma abordagem que estamos desenvolvendo na Idwall para criação de pipelines de dados. Esta metodologia foi implementada com sucesso na criação de uma de nossas maiores pipelines. Graças à abordagem Semantic-first, fomos capazes de acelerar a criação de testes e otimizações, minimizar falhas de comunicação e garantir uma evolução eficiente do sistema. Esta abordagem também permitiu que novos membros da equipe se familiarizassem e contribuíssem rapidamente para a pipeline.</blockquote><p>Muitos se referem aos dados como o “novo petróleo”. E, em muitos aspectos, a comparação é válida. Assim como o petróleo é transportado e transformado por meio de pipelines, os dados também são movidos e moldados por pipelines digitais. No entanto, essa analogia tem seus limites.</p><p>Enquanto o petróleo, uma vez refinado, tem um conjunto limitado de derivados, os dados possuem uma infinidade de interpretações, aplicações e restrições. E assim como um gasoduto mal projetado pode resultar em vazamentos desastrosos, um pipeline de dados mal construído pode levar a interpretações errôneas, decisões equivocadas e custos para uma organização. No contexto da LGPD, um erro na gestão de dados pode resultar em processos, danos à reputação e perda de confiança dos clientes.</p><p>A complexidade dos dados exige uma abordagem cuidadosa, onde a semântica e consequentemente ontologia e taxonomia desempenham papéis cruciais para garantir que os dados sejam transportados, compreendidos e utilizados de forma correta, conforme as regulamentações de proteção de dados.</p><p>Na Idwall, reconhecemos a importância desses conceitos e estamos desenvolvendo uma abordagem <em>semantic-first</em> para criação de pipelines. Esta estratégia, além de permitir uma integração rápida dos dados, assegura sua correta interpretação. Utilizando metaprogramação, nossas bibliotecas traduzem o conhecimento representado sobre os dados diretamente em modelos dbt com metadados, testes e otimizações sendo gerados de forma automática.</p><p>Além disso, essa abordagem permite facilmente acoplar novas features como atualização automática no confluence, modelos de linguagem e qualquer sistema que queria se aproveitar dessas representações. Em resumo, a <em>semantic-first</em> otimiza a manutenibilidade, reduz a carga cognitiva e garante uma fluidez na integração de novas tabelas e fontes de informação.</p><h3>Falhas de comunicação devido a conhecimento implícito e suas consequências</h3><ul><li><strong>O que significa conhecer alguma coisa?</strong></li><li><strong>O que faz uma coisa ser o que ela é?</strong></li></ul><p>A primeira vista, essas indagações podem parecer mais adequadas a um debate filosófico, não há uma discussão sobre engenharia e análise de dados. No entanto, ao refletir cuidadosamente sobre elas, você poderá perceber a importância delas ao construir pipelines.</p><p>Para ilustrar, imagine uma situação fictícia envolvendo dois personagens, José e sua filha Maria. Em uma conversa de WhatsApp, Maria avisa a José que a manga dela está estragada. No entanto, ao invés de pensar no uniforme que Maria estava usando, José imagina que Maria está falando de uma manga fruta e fala para Maria apenas cortar a parte estragada. Maria, que é uma criança obediente, pega a tesoura e corta a manga da camisa. Maria, que<strong> violou o contrato </strong>de vestimenta da escola, <strong>recebe uma advertência</strong> por usar um estilo punk.</p><p>Esse equívoco, relacionado à troca de informação, ilustra como o conhecimento implícito gera desentendimentos e pode até resultar em uma criança de castigo. Tanto Maria quanto José trocaram informações entre si, comunicando o estado de uma coisa, nomes e ações, mas cada um tinha um mapa mental diferente sobre as coisas que estavam falando.</p><blockquote>“The limits of my language mean the limits of my world” — Ludwig Wittgenstein</blockquote><p>Se confusões podem surgir em uma conversa simples entre pessoas que se conhecem por uma vida, imagine, em uma organização, onde dados são constantemente compartilhados entre centenas de tabelas e milhares de colunas. Cada coluna carrega um significado que pode ser interpretado de diferentes maneiras por diferentes pessoas e sistemas.</p><p>Nesse contexto de falhas de comunicação, existe uma asserção chamada de 80/20. Essa afirmação diz que cientistas de dados gastam apenas 20% do tempo analisando e criando modelos e 80% do tempo buscando, limpando e organizando dados. Então, na melhor das hipóteses, dados desorganizados desperdiçam força de trabalho. Mas até neste melhor dos casos temos outros graves efeitos colaterais:</p><ul><li><strong>Como garantimos que, durante essa limpeza e organização, os analistas e cientistas de dados adotaram um conjunto consistente de práticas, taxonomia e métodos?</strong></li><li><strong>Como asseguramos que todos os consumidores e produtores de dados estejam se referindo a mesma “manga”?</strong></li></ul><h3>Priorizar código pode ser uma má ideia.</h3><p>Ao construir pipelines de dados, uma abordagem que muitas equipes podem adotar é escolher uma tecnologia como o dbt e começar a implementar transformações e modelos simultaneamente definindo a taxonomia ou simplesmente ignorando enquanto novos cards no JIRA são puxados. No entanto, essa prática de priorizar o código pode conduzir a uma série de desafios.</p><p>A metodologia orientada à tecnologia/código pode levar a incongruências significativas entre diferentes projetos. Cada projeto pode adotar práticas e taxonomias distintas, dificultando não apenas otimizar e catalogar os dados, mas também manter a consistência e interoperabilidade entre diferentes sistemas. Isso resulta em um ambiente de dados fragmentado, onde a falta de uma linguagem comum impede a verdadeira integração e utilização otimizada dos dados.</p><p>Além disso, a ausência de uma estruturação de significado de dados desde o início pode resultar em custos adicionais de manutenção e refatoração, à medida que as necessidades evoluem e novos requisitos emergem e pessoas deixam a empresa levando com elas a taxonomia implícita armazenada apenas nos seus cérebros.</p><p>A falta de uma abordagem padronizada e estruturada para a representação do conhecimento impede que organizações alcancem o pleno potencial dos seus ativos de dados, pois eles não são acessíveis:</p><p><strong>Dados acessíveis = bits + semântica alinhada [1]</strong></p><p>Lembre do exemplo da manga e da conversa entre Maria e João. Toda a dor de cabeça teria sido evitada se antes de João propor uma solução eles tivessem checado se estavam alinhados quanto ao significado das coisas. É claro que uma burocracia desse nível para conversas do dia-a-dia tornaria a vida muito triste.</p><p>Mas em nossos pipelines de dados essa burocracia reduz falhas de comunicação e todos os problemas que elas geram. Por isso é interessante que taxonomia e ontologia sejam os pilares de processos envolvendo dados</p><h3>O que são Ontologia e Taxonomia?</h3><h4>Taxonomia</h4><p>A taxonomia, familiar a qualquer um que já jogou <strong><em>“Quem sou eu?”</em></strong> ou frequentou aulas de biologia. No mundo dos dados, ela organiza informações em estruturas hierárquicas, agrupando conceitos ou entidades por características comuns. Uma taxonomia bem estruturada facilita a recuperação e análise de dados.</p><p>Em um sistema que sofresse do mesmo problema que João e sua filha, poderíamos ter uma regra de que toda menção de fruta deveria seguir o padrão <strong>nome_da_fruta+prefixo <em>_fruit </em></strong>e isso evitaria o problema de ter danificado o uniforme da escola.</p><p>Com a LGPD, uma taxonomia centralizada e atualizada é vital, garantindo a classificação correta de dados e evitando violações legais como por exemplo a classificação errada se um dado é PI/PII. Além disso, reduz retrabalho e carga cognitiva, padronizando nomenclaturas e descrições para documentações, permitindo a reutilização até mesmo em documentações em APIs usando o Swagger.</p><h4>Ontologia</h4><p>Embora a ontologia seja um campo antigo, ela desempenha um papel central na tecnologia atual. No contexto da TI, ela define e modela conceitos, categorias e relações em um domínio específico de conhecimento, usando especificações como RDF/OWL. Isso permite uma que computadores “conversem” entre-si sem inconsistências.</p><p>Ao definir que taxonomia e ontologia serão as bases das suas pipelines você aumenta a taxa de acerto com que o conhecimento / informação que tais pipelines objetivam entregar é absorvido e compartilhado.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ltgouq7so3YgmpFz" /><figcaption>Uma das grandes features de uma ontologia é permitir desambiguações e “reasoning” automático. Na imagem acima vemos a desambiguação de um objeto cachimbo de uma obra de arte.</figcaption></figure><h3>Semantic-first na prática</h3><p>Estruturar um projeto de data pipeline com ontologia e taxonomia como pilares pode, à primeira vista, parecer uma tarefa árdua, sobretudo quando não há um repositório central contendo o conhecimento acerca dos dados da organização, mesmo que seja apenas algumas páginas Web ou planilhas. No entanto, leia atentamente e DUAS VEZES o seguinte trecho:</p><blockquote>“The notion of representation of knowledge is at heart an easy one to understand. It simply has to do with writing down, in some language or communicative medium, descriptions or pictures that correspond in some salient way to the world or the state of the world. In Artificial intelligence, we are concerned with writing down descriptions of the world in such a way that an intelligent machine can come to new conclusions about its environment by formally manipulating these descriptions.” — Readings in Knowledge Representation, 1985 [4]</blockquote><p>Surpreendentemente, esse trecho foi extraído de um livro com quase quatro décadas! <strong>E, embora você possa ter se deparado com um post no LinkedIn ou ouvido um consultor afirmar que é imprescindível um banco de grafos para um repositório de conhecimento robusto e escalável, essa não é a realidade</strong>.</p><p>Dependendo do caso, uma série de arquivos YAML descrevendo seus dados, aliados a sólidos processos de DevOps e/ou MLOps, pode ser mais do que adequada. Afinal, grafos dizem respeito a relações. E se não faz diferença se você representar essas relações por meio de texto, porque se preocupar?</p><p>Desta maneira, <strong>a principal tarefa é definir processos que permitam extrair conhecimento e significado, que muitas vezes estão implícitos — seja em mentes humanas, páginas no Confluence, sistemas de banco de dados etc.</strong> A boa notícia é que existem frameworks que ajudam a orientar essa jornada em busca do significado das coisas em uma organização.</p><h4>O que temos de parecido com CRISP-DM para ontologia?</h4><p>Assim como o CRISP-DM oferece um guia para projetos de dados, abordagens sistemáticas para a criação de representações de conhecimento foram desenvolvidas. É conveniente encontrar e adaptar aquela que melhor se adequa ao seu propósito e o conjunto de ferramentas e técnicas que você e/ou seu time domina.</p><p>Isso pode variar desde métodos menos formais com mapas conceituais desenhados em um quadro branco amplamente empregados no ensino, até a elaboração de grafos de conhecimento alimentando constantemente um banco como RDF4J.</p><p>Um fluxo simplificado em que um conhecimento (ou ontologia) pode ser construída por times é representado no diagrama abaixo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IzWZEL-v8aJUPaTbzH1bQ.png" /></figure><p>Como e como o quê você e seu vai executar esse fluxo vai depender de vocês e o que for mais confortável. Talvez uma planilha já seja o ótimo.</p><p>Há uma discussão de como incorporar mais sistematicamente a confecção de uma ontologia em times que trabalham com a metodologia ágil. Para mais informações você pode ler o trabalho<a href="https://inass.org/2019/2019043017.pdf"> <em>An Agile Methodology for Ontology Development</em></a><em> </em>ou<a href="https://www.w3.org/community/owled/files/2016/11/OWLED-ORE-2016_paper_6.pdf"> <em>A Simplified Agile Methodology for Ontology Development</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeGp0M0a5MH3XfNKZRbRLg.png" /><figcaption>Diagrama representado uma proposta ágil para desenvolvimento de uma ontologia em uma organização [5]</figcaption></figure><p>Para oferecer um abordagem orientada a requisitos para extração e formalização de conhecimento,<a href="https://tishchungoora.medium.com/?source=post_page-----964fab49ec88--------------------------------"> Tish Chungoora</a> criou um diagrama que pode ser visto<a href="https://miro.medium.com/v2/resize:fit:720/format:webp/1*7Xia-gVs9TxtPBnCHj59tA.png"> aqui</a>. Outra abordagem, amplamente discutida em livros e casos de sucesso, envolve a criação de<a href="https://tishchungoora.medium.com/ontology-competency-questions-3d213eb08d33"> questões de competência</a>.</p><p>Nesse método, em um processo de entrevista com consumidores de dados, levantamos questionamentos que nossa representação de conhecimento precisa responder.</p><p>Não seria viável descrever aqui todas essas abordagens sistemáticas de representação de conhecimetno, então recomendo os seguintes livros:</p><ul><li><strong>Designing and building enterprise knowledge graphs. </strong>Este é meu livro preferido sobre KGs aplicados em indústria.</li><li><strong>Dados abertos conectados. </strong>Um livro em português disponível gratuitamente<a href="https://ceweb.br/livros/dados-abertos-conectados//prefacio/"> aqui</a>.</li><li><strong>Exploiting Linked Data and Knowledge Graphs in Large Organizations. </strong>Muitos cases de empresas reais e discussão sobre o processo de adoção.</li></ul><p>Após as primeiras interações no processo de extração e representação de conhecimento, já é possível começar a construir pipelines de dados semantic-first.</p><p>Vale ressaltar que esse processo de representação de conhecimento é contínuo, uma vez que, com o passar do tempo, novos conceitos e significados emergem na organização em que você atua.</p><h4>Como construir minhas representações de conhecimento?</h4><p>Se sua organização ainda não possui um repositório sistematizado para representar o conhecimento sobre os dados, está na hora de criar um!</p><p>Como dito anteriormente nesse post, não se deixe seduzir pela última tecnologia em voga. Realize uma PoC com sua equipe para avaliar a viabilidade técnica de usar uma tecnologia de banco de grafos. Se a PoC indicar que ainda não há maturidade suficiente, simplesmente descarte essa opção.</p><p>Uma abordagem simples é representar o conhecimento sobre os dados da organização em uma série de arquivos de texto. Considerando que os YAMLs são frequentemente utilizados por engenheiros, engenheiros analíticos e analistas de dados, por que não optar por eles?</p><p>Na imagem abaixo, apresento uma representação pitoresca de um sistema em que representação de conhecimento é produzida e aplicada a diversas tarefas. Alguns componentes desse sistema são opcionais, dando-lhe total liberdade na escolha de tecnologias e no como.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JpnTgkRqTRaoyO_pbWU94Q.png" /></figure><p>O passo inicial é sempre representar o conhecimento, o significado, a semântica dos nossos dados que serão trafegados nas pipelines. Por exemplo, podemos criar um arquivo YAML descrevendo as colunas que usaremos na nossa primeira pipeline:</p><pre># columns.yaml<br> cpf:<br>    name: CPF<br>    description: &quot;...&quot;<br>    type: &quot;STRING&quot;<br> relationship_type:<br>    name: &quot;Tipo da   relação&quot;<br>    description: &quot;....&quot;<br>     type: &quot;...&quot;<br>     accepted_values:<br>         - mother<br>         - ...</pre><p>e outro descrevendo a categorização dessas colunas:</p><pre># categories.yaml<br> contact:<br>   - phone_number<br>   - email<br> personal_financial:<br>   - income<br>   - related_companies<br>   - ....</pre><p>Usando a especificação RDF/OWL o conhecimento dos dois YAMLs acima poderia ser representado da seguinte forma:</p><pre>@prefix rdf: &lt;http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt; .<br>@prefix rdfs: &lt;http://www.w3.org/2000/01/rdf-schema#&gt; .<br>@prefix owl: &lt;http://www.w3.org/2002/07/owl#&gt; .<br>@prefix xsd: &lt;http://www.w3.org/2001/XMLSchema#&gt; .<br>@prefix ex: &lt;http://example.org/ontology#&gt; .<br><br># Definição das classes com base no arquivo categories.yaml<br>ex:Contact a owl:Class ;<br>    rdfs:label &quot;Contact&quot; .<br><br>ex:PersonalFinancial a owl:Class ;<br>    rdfs:label &quot;Personal Financial&quot; .<br><br># Definição das propriedades com base no arquivo columns.yaml<br>ex:cpf a owl:DatatypeProperty ;<br>    rdfs:label &quot;CPF&quot; ;<br>    rdfs:comment &quot;...&quot; ;<br>    rdfs:domain ex:Contact ;<br>    rdfs:range xsd:string .<br><br>ex:relationshipType a owl:ObjectProperty ;<br>    rdfs:label &quot;Tipo da relação&quot; ;<br>    rdfs:comment &quot;....&quot; ;<br>    rdfs:domain ex:PersonalFinancial ;<br>    rdfs:range rdfs:Resource .<br><br># Instâncias aceitas para o tipo de relação<br>ex:Mother a owl:NamedIndividual ;<br>    rdfs:label &quot;mother&quot; .<br><br># Adicione outras instâncias conforme necessário.<br><br><br># Subclasses para Contact e PersonalFinancial<br>ex:PhoneNumber a owl:Class ;<br>    rdfs:subClassOf ex:Contact ;<br>    rdfs:label &quot;Phone Number&quot; .<br><br>ex:Email a owl:Class ;<br>    rdfs:subClassOf ex:Contact ;<br>    rdfs:label &quot;Email&quot; .<br><br>ex:Income a owl:Class ;<br>    rdfs:subClassOf ex:PersonalFinancial ;<br>    rdfs:label &quot;Income&quot; .<br><br>ex:RelatedCompanies a owl:Class ;<br>    rdfs:subClassOf ex:PersonalFinancial ;<br>    rdfs:label &quot;Related Companies&quot; .</pre><p>A vantagem da especificação RDF/OWL é que podemos armazenar a representação do nosso conhecimento em um banco de grafos tal como RDF4J. E bancos de grafos RDF/OWL possuem um monte de features legais que não podem ser encontradas em bancos comuns tais como reasoning automatico. Contudo, não discutiremos essas vantagens aqui mas convido você a procurar mais sobre.</p><h4>Representando suas pipelines</h4><p>O passo seguinte consiste em representar o conhecimento sobre nossa pipeline, ou seja, o que queremos com nossa pipeline.<em> “Os dados deverão sair de qual fonte? Para onde eles deverão ir? Como cada coluna sera mapeada/transformada?”</em> Ao responder essas perguntas e validar elas com seu time, você poderia, por exemplo, representar uma pipeline que transforma uma tabela <strong>source_table_name</strong> para uma tabela <strong>std_my_new_table</strong>,alterando apenas o nome das coluna <strong>CPF</strong> e agrupando colunas de nomes da seguinte maneira:</p><pre># kr/std/models.yaml<br>std_my_new_table:<br>  folder: folder_name # onde os arquivos dbt gerados são salvos<br>  table_src: source_table_name # nome da tabela que será extraída os dados<br>  description: &quot;Uma descrição útil dessa tabela&quot;<br>  columns: # aqui listamos as colunas de origem e o campo mapeado<br>    cpf: CPF<br>    cpf_status: status_na_receita_federal # as tabelas/fontes de origem podem não estar seguindo uma taxonomia<br>    full_name:<br>      - firstname<br>      - middlename<br>      - lastname</pre><h4>Garantido consistência na semântica das pipelines</h4><p>Se a representação de conhecimento dos dados está formalizada, podemos criar processos que verifiquem se as pipelines respeitam a semântica alinhada entre todos ou<em> se há essa semântica disponível</em> no nosso repositório de conhecimento. Aqui na Idwall, esses testes podem ser executados de maneira fácil usando um target no Makefile. Para começar, imagine que você tenha um diretório representando suas pipelines da seguinte forma:</p><pre>kr/<br>│<br>├── LAYER1/<br>│   └── models.yaml<br>│<br>├── LAYER2/<br>│   └── models.yaml<br>│<br>└── ...</pre><p>Um pseudo-código que poderia compor um job CI/CD que validaria se todas as colunas na pipeline de dados poderia ser:</p><pre>Função LoadYAML(caminho_arquivo):<br>    Abra caminho_arquivo como arquivo:<br>        Retorne o conteúdo do arquivo parseado como YAML<br><br>Função ValidateColumns(modelo, colunas_definidas):<br>    Para cada coluna em modelo[&#39;columns&#39;]:<br>        Se coluna não está em colunas_definidas:<br>            Reporte erro &quot;Coluna não definida&quot;<br><br>Função ValidateModels(caminho_modelo, colunas_definidas):<br>    modelo = LoadYAML(caminho_modelo)<br>    Para cada item em modelo[&#39;models&#39;]:<br>        ValidateColumns(item, colunas_definidas)<br><br>Função Main():<br>    colunas_definidas = LoadYAML(&quot;caminho/para/columns.yaml&quot;)<br>    <br>    Para cada LAYER em ListaDeLayers:<br>        caminho_modelo = &quot;kr/&quot; + LAYER + &quot;/models.yaml&quot;<br>        ValidateModels(caminho_modelo, colunas_definidas)<br><br>Execute Main()</pre><p>Contudo, se sua organização adotou uma maneira de representação de conhecimento mais formal por exemplo, usando um KG que suporta SPARQL, você poderia utilizar a seguinte query para validar a existência da coluna <strong>full_name</strong> no seu KG:</p><pre>ASK WHERE {<br>  ?instance a ex:Column;<br>            ex:hasName &quot;full_name&quot; .<br>}</pre><h4>Geração automática de schemas dbt e testes</h4><p>A partir dessas representações, podemos criar scripts que leiam essas representações e gerem automaticamente os arquivos <strong>schema.yml</strong> para os modelos dbt, incluindo informações como descrições, tipos de dados, e testes. Que é o que fizemos aqui na Idwall!</p><p>Para deixar mais claro, aqui está um pseudo-código básico que você pode usar para guiar seu projeto:</p><pre>Função LoadYAML(caminho_arquivo):<br>    Abra caminho_arquivo como arquivo:<br>        Retorne o conteúdo do arquivo parseado como YAML<br><br>Função GenerateDBTSchema(columns_def, categories_def):<br>    Inicialize model_schema como um dicionário com chaves &quot;name&quot; e &quot;columns&quot;<br>    <br>    Para cada coluna, column_info em columns_def:<br>        Inicialize column_schema como um dicionário com chaves &quot;name&quot;, &quot;description&quot;, e &quot;tests&quot;<br>        Adicione informações de column name e description para cada coluna<br>        <br>        Se type de column_info é &quot;string&quot;:<br>            Adicione teste &quot;is_string&quot; a column_schema[&quot;tests&quot;]<br>        Senão se type de column_info é &quot;bool&quot;:<br>            Adicione teste &quot;is_boolean&quot; a column_schema[&quot;tests&quot;]<br>        # Adicione outros testes de tipo conforme necessário<br>        <br>        Se &quot;accepted_values&quot; está presente em column_info:<br>            Adicione teste &quot;accepted_values&quot; e os valores aceitos a column_schema[&quot;tests&quot;]<br>        <br>        Adicione column_schema a model_schema[&quot;columns&quot;]<br>    <br>    # Lógica adicional pode ser adicionada para lidar com categories_def<br>    <br>    Adicione model_schema a dbt_schema[&quot;models&quot;]<br>    Retorne dbt_schema<br><br>Função SaveYAML(data, caminho_arquivo):<br>    Abra caminho_arquivo como arquivo:<br>        Escreva data como YAML no arquivo<br><br>Função Main():<br>    columns_def = LoadYAML(&quot;caminho/para/columns.yaml&quot;)<br>    categories_def = LoadYAML(&quot;caminho/para/categories.yaml&quot;)<br>    <br>    dbt_schema = GenerateDBTSchema(columns_def, categories_def)<br>    <br>    SaveYAML(dbt_schema, &quot;dbt_minha_org/models/meu_modelo/schema.yaml&quot;)<br><br>Execute Main()</pre><p>Essa abordagem se mostrou bastante poderosa para geração de metadados e testes. <strong>Conseguimos, por exemplo, sair de uma pipeline com 0 testes para mais de 500 em apenas um dia</strong>! Além do que, conseguimos também gerar metadados complexos e úteis hoje que gastariam dezenas de horas para serem feitos e mantidos de forma manual.</p><h4>Destravando metaprogramação: criação automática e simplificada de modelos dbt</h4><p>Ao classificar “animais”, podemos observar que a classe “Mamíferos” é definida por propriedades como “tem pelos” e “amamenta seus filhotes”. Contudo, uma classificação rica vai além dessas características intrínsecas. Por exemplo, <strong>um leão não é apenas definido por suas características físicas, mas também pelas relações ecológicas que mantém, sendo predador de muitos animais e presa de poucos.</strong></p><p>Agora pense no caso de pipelines e catalogação de dados. Por qual motivo agrupar tabelas apenas pelos dados que ela contem? Algumas tabelas são construídas pela padronização e agrupamento de colunas de outras tabelas, enquanto outras surgem do empilhamento entre diferentes tabelas.</p><p>Semelhante ao mundo natural, onde seria estranho sempre usar a mesma coleira em um lobo e um cachorro doméstico, quando organizamos nossas pipelines, agrupar tabelas considerando apenas as características dos dados que elas carregam nem sempre é uma boa ideia,</p><p>Imagine uma pipeline que possua apenas duas camadas, organizadas da seguinte forma:</p><ul><li>LAYER1 (ex.: std): foca na padronização e agregação de colunas, assegurando que os dados estejam limpos e em um formato consistente. Tem apenas uma única tabela como fonte de dados.</li><li>LAYER2 (ex.: stack): dedica-se a empilhar linhas de diferentes tabelas da camada std, criando visões mais complexas e agregadas dos dados.</li></ul><p>Dado que cada tabela da classe stack “empilha” linhas de outras tabelas da camada std, estamos criando classes em nossa ontologia de dados baseadas nas transformações e não no conteúdo, embora pudéssemos criar subcamadas que se dividem pelo conteúdo. Isso se revela extremamente útil, pois, por exemplo, quando programamos em pyspark estamos sempre pensando em transformções.</p><p>Portanto, podemos criar um código que gera o código dessas transformações, permitindo-nos metaprogramar nosso pipeline. Suponha que desejamos criar uma tabela que empilhe todas as linhas de todas as tabelas que descrevem frutas, isso poderia ser representado por um arquivo assim:</p><pre># kr/stacks/models.yaml<br>stack_fruits:<br>  regex_files: &quot;^std_fruit_&quot;<br>  description: &quot;Empilha todas as tabelas de fruta&quot;</pre><p>Modelos dbt podem ser escritos tanto em SQL quanto em Python. A lógica de uma metaprogramação para esse exemplo simples de empilhamento seria algo do tipo, expresso em pseudo-código:</p><pre>Função CheckStdTableCond(model_name, regex_file):<br>    Se model_name não corresponde a regex_file:<br>        Retorne Falso<br>    Retorne Verdadeiro<br><br>Função GetStdTables(std_tables, regex_file):<br>    Inicialize std_tables_filtred como um dicionário vazio<br>    Para cada model_name, model em std_tables:<br>        Se CheckStdTableCond(model_name, regex_file) é Verdadeiro:<br>            Adicione model_name e model a std_tables_filtred<br>    Retorne std_tables_filtred<br><br>Função InsertModelDef(tree, std_tables):<br>    Inicialize std_tables_names com as chaves de std_tables<br>    Crie um nó de função AST para definir uma função &#39;model&#39; com parâmetros e corpo vazio<br>    Para cada std_name em std_tables_names:<br>        Adicione nós ao corpo da função para criar referências dbt e renomear colunas<br>    Adicione um nó ao corpo da função para unir todas as tabelas por nome e atribuir o resultado a &#39;df&#39;<br>    Adicione um nó ao corpo da função para retornar &#39;df&#39;<br>    Adicione o nó da função ao corpo da árvore AST<br>    Retorne tree<br><br>Função GenCode(std_tables):<br>    Crie uma árvore AST vazia<br>    Insira a definição do modelo dbt na árvore AST usando InsertModelDef<br>    Converta a árvore AST em código fonte e retorne<br><br>Função Main():<br>    Carregue todas as tabelas std de uma configuração<br>    Para cada stack_name, stack em stacks da configuração:<br>        Obtenha regex_files de stack<br>        Obtenha tabelas std correspondentes usando GetStdTables<br>        Gere código usando G enCode<br>        Salve o código gerado (sql ou python) na pasta de modelos dbt<br><br>Execute Main()</pre><p>Se amanhã, um analista que nunca ouviu falar em dbt e acabou de entrar na sua organização quiser usar uma tabela que empilhe as tabelas de camisas com manga, bastaria declarar uma nova seção com três linhas no arquivo<strong> kr/stacks/models.yaml</strong>.</p><p><strong>Esta abordagem de construir código que traduz especificações simples em modelos de dbt otimizados permitiu que novos membros da equipe contribuíssem facilmente para a pipeline, integrando novos dados, atualizando metadados e transformações ao mesmo tempo, garantindo com pouquíssimas violações de padrões de taxonomia.</strong></p><h4>O futuro é um diálogo com suas pipelines. Integrando LLMs e KGs</h4><p>Imagine um cenário — ok, talvez você nem precise imaginar! — em que sua empresa opera com dezenas de pipelines, algumas delas gerenciando/criando uma centena de tabelas e totalizando dezenas de milhares de colunas.</p><p>Nesse contexto, perguntas capciosas podem surgir de PMs ou novos colaboradores acerca dos dados: <em>“Em qual tabela encontro a informação X? O que significa a coluna Y? Qual é a função desta collection do MongoDB?”</em> Em situações ainda mais complexas, podemos nos deparar com questionamentos como: <em>“Como posso rastrear a origem dos dados financeiros importados do Bureau de Crédito A?”</em></p><p>Para responder a essas perguntas cruciais, muitas equipes se veem mergulhando nas profundezas de um catálogo de dados, navegando por uma caixa de busca, checkboxes, cores e popups na esperança de encontrar respostas precisas — isso, claro, pressupondo que uma que há essa organização e um catálogo de dados sempre atualizado estejam disponíveis e sejam de fácil acesso para os colaboradores, o que nem sempre é o caso. É o seu? Se não, sinto muito : ( .</p><p>Agora, reflita comigo: se você adotou uma abordagem <em>semantic-first </em>e formalizou o conhecimento tanto sobre os dados quanto sobre as transformações (pipelines), e se fez isso utilizando um Knowledge Graph (KG), agora você tem a capacidade de dialogar com seus pipelines. Utilizando o KG como uma fonte de fatos para um LLM, como o ChatGPT, você pode navegar e interagir com seus pipelines de uma maneira incrivelmente intuitiva e informativa. Fascinante, não é?</p><p>No entanto, o texto já se estendeu bastante e aprofundar-nos nesse tópico será uma jornada para outro momento.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Y68OM0oH_UChEWk1-73HA.png" /></figure><h3>Referências</h3><ul><li>[1]J. Sequeda e O. Lassila, <strong>Designing and building enterprise knowledge graphs</strong>. 2021. <strong>Meu predileto!</strong></li><li>[2]<strong> Dados abertos conectados</strong>. Novatec Editora, 2022.<strong> Livro disponível gratuitamente </strong><a href="https://ceweb.br/livros/dados-abertos-conectados//prefacio/"><strong>aqui</strong></a><strong>.</strong></li><li>[3] J. M. Gomez-Perez, J. Z. Pan, G. Vetere, e H. Wu, Orgs., <strong>Exploiting Linked Data and Knowledge Graphs in Large Organisations</strong>, 1st ed. 2017. Cham: Springer International Publishing/ : Imprint: Springer, 2017.</li><li>[4] R. J. Brachman e H. J. Levesque, Orgs., <strong>Readings in knowledge representation</strong>. Los Altos, Calif: M. Kaufmann Publishers, 1985.</li><li>[5] Cairo University, A. Abdelghany, N. Darwish, Cairo University, H. Hefni, e Cairo University,  <strong>An Agile Methodology for Ontology Development</strong> , IJIES, vol. 12, no 2, p. 170 181, abr. 2019, doi: 10.22266/ijies2019.0430.17.</li><li>[6] Linked Data Engineering (Semantic Web) OpenHPI by Prof. Dr. Harald Sack <a href="https://www.youtube.com/playlist?list=PLoOmvuyo5UAfY6jb46jCpMoqb-dbVewxg">https://www.youtube.com/playlist?list=PLoOmvuyo5UAfY6jb46jCpMoqb-dbVewxg</a> <strong>Um excelente curso em inglês sobre dados conectados</strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bbf40a68750" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineering-idwall/semantic-first-ontologia-e-taxonomia-como-pilares-no-desenvolvimento-de-pipelines-de-dados-3bbf40a68750">Semantic-first: ontologia e taxonomia como pilares no desenvolvimento de pipelines de dados</a> was originally published in <a href="https://medium.com/engineering-idwall">engineering.idwall</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Correlation matrix with the nested stochastic block model: the graph structure of the stock market.]]></title>
            <link>https://medium.com/@devmessias/correlation-matrix-with-the-nested-stochastic-block-model-the-graph-structure-of-the-stock-market-4b531b84ee2d?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b531b84ee2d</guid>
            <category><![CDATA[correlation-matrices]]></category>
            <category><![CDATA[community-detection]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[dataviz]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Mon, 12 Dec 2022 13:21:04 GMT</pubDate>
            <atom:updated>2022-12-14T17:08:51.830Z</atom:updated>
            <content:encoded><![CDATA[<p>A Portuguese version of this post is available at <a href="https://opencodecom.net/post/2022-05-16-grafos-e-modelo-de-blocos-aninhados-para-matrizes-de-correlacao-clusterizacao-do-mercado-de-acoes/">https://opencodecom.net/</a></p><p>In this post, I’ll show you how the nested stochastic block model (nSBM) can extract insights from correlation matrices using a combination with a filtering edge graph technique called disparity filter. I’ll use as a case study a correlation matrix derived from the stock market data.</p><p>Don’t be afraid, you will reproduce the beautiful plot presented below and understand what it means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*eKOgDs7AfrruNSYLv7jpvg.png" /></figure><p>If you want to reproduce, use a conda environment because of the graph-tool dependency. Besides that, check if you have the following packages installed:</p><pre>matplotlib, pandas, yfinance</pre><p>Probably you will need to install the igraph package</p><pre>$ pip install python-igraph</pre><p>Now, install the graph-tool made by <a href="https://twitter.com/tiagopeixoto">Tiago Peixoto</a>.</p><pre>$ conda install -c conda-forge graph-tool</pre><p>No less important, we will need to remove weakling edges from the graph. To do so, we will use my package, edgeseraser. The repository of the project is available here: <a href="https://github.com/devmessias/edgeseraser">devmessias/edgeseraser</a>. To install the edgeseraser run the following command:</p><pre>$ pip install edgeseraser</pre><h3>Introduction</h3><p>Exploratory data analysis allows us to gain insights or improve our choices for the next steps in the data science process. Sometimes, we can explore the data assuming that all the entities are independent of each other. What happens when it’s not the case?</p><h3>Graphs</h3><p>A graph is a data structure used to store objects (vertexes) with binary relationships. Those relationships are called edges. Each edge can have generic data associated with it. Like a numeric value. Maybe you can think that this data structure is rare in the world of data. But if you look carefully at the data spread on the internet, graphs are everywhere: social networks, economic transactions, correlation matrices and many more.</p><h3>Correlation matrices</h3><p>A correlation matrix is a storing mechanism that quantifies relationships between pairs of random variables. Thus, a correlation matrix is at least a weighted graph. The main point of this post is to show how we can use tools of network science to analyze the correlation matrix. Ok, but maybe you’re asking: Why do we need to do this?</p><p>Correlation matrices are commonly used to discover and segment the universe of possible random variables. With this segmentation, we can analyze the dataset in a granular way. Therefore, improving our efficiency by deleting unnecessary information.</p><p>A widely spread way to analyze correlation matrices using graph analytics is through the use of minimum spanning trees. I won’t go into details about this method (check the following posts <a href="https://hudsonthames.org/networks-with-mlfinlab-minimum-spanning-tree-mst/#:~:text=A%20Minimum%20Spanning%20Tree%20%28MST%29%20is%20a%20graph%20consisting%20of,a%20stock%20crisis%20%5B2%5D.">Networks with MlFinLab: Minimum Spanning </a>and <a href="https://www.bcb.gov.br/pec/wps/ingl/wps351.pdf">Dynamic spanning trees in stock market networks: The case of Asia-Pacific</a>) because I want to show how we can use a different technique that at the same time is more flexible and powerful when we want to understand the hidden group structure of a correlation matrix.</p><p>This technique is called the nested stochastic block model (nSBM) and it’s a probabilistic model that allows us to infer the hidden group structure of a graph. The nSBM is a generalization of the stochastic block model (SBM). The SBM is a generative model that allows us to generate graphs with several communities. This generative model also creates those communities using parameters that describe the probability of a connection between vertices living in the same or different communities. Although the SBM is a powerful tool to analyze the structure of a graph, it’s not enough to infer the small communities (small groups of stocks) in our graph. To do so, we need to use the nSBM.</p><p>Ok, let’s go to the actual stuff and code.</p><h3>Downloading and preprocessing the data</h3><h3>Closing prices of the S&amp;P 500</h3><p>Let’s import what we need</p><pre><strong>import</strong> yfinance <strong>as</strong> yf<br><strong>import</strong> pandas <strong>as</strong> pd<br><strong>import</strong> numpy <strong>as</strong> np<br><strong>import</strong> matplotlib.pyplot <strong>as</strong> plt<br><strong>import</strong> matplotlib <strong>as</strong> mpl<br><strong>import</strong> igraph <strong>as</strong> ig<br><strong>from</strong> edgeseraser.disparity <strong>import</strong> filter_ig_graph</pre><p>To keep things simple, I choose to use the following data available <a href="https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv">here</a>. This data encompasses the S&amp;P500 from a specific time with symbols and sectors like this:</p><p>SymbolNameSectorMMM3MIndustrialsAOSA. O. SmithIndustrialsABTAbbott LaboratoriesHealth CareABBVAbbVieHealth Care</p><pre>!wget <a href="https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv">https://raw.githubusercontent.com/datasets/s-</a><a href="https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv"><strong>and</strong></a><a href="https://raw.githubusercontent.com/datasets/s-and-p-500-companies/master/data/constituents.csv">-p-500-companies/master/data/constituents.csv</a></pre><p>After downloading the data we just need to load it into a pandas dataframe.</p><pre>df = pd.read_csv(&quot;constituents.csv&quot;)<br>all_symbols = df[&#39;Symbol&#39;].values<br>all_sectors = df[&#39;Sector&#39;].values<br>all_names = df[&#39;Name&#39;].values</pre><p>The following dictionaries will map the symbols to sectors and names later on</p><pre>symbol2sector = dict(zip(all_symbols, all_sectors))<br>symbol2name = dict(zip(all_symbols, all_names))</pre><p>Time to download the information about each symbol. Let’s request one semester of data from yahoo finance API</p><pre>start_date = &#39;2018-01-01&#39;<br>end_date = &#39;2018-06-01&#39;<br><strong>try</strong>:<br>    prices = pd.read_csv(<br>        f&quot;sp500_prices_{start_date}_{end_date}.csv&quot;, index_col=&quot;Date&quot;)<br>    tickers_available = prices.columns.values<br><strong>except</strong> FileNotFoundError:</pre><pre>    df = yf.download(<br>        list(all_symbols),<br>        start=start_date,<br>        end=end_date,<br>        interval=&quot;1d&quot;,<br>        group_by=&#39;ticker&#39;,<br>        progress=True<br>    )<br>    tickers_available = list(<br>        set([ticket <strong>for</strong> ticket, _ <strong>in</strong> df.columns.T.to_numpy()]))<br>    prices = pd.DataFrame.from_dict(<br>        {<br>            ticker: df[ticker][&quot;Adj Close&quot;].to_numpy()<br>            <strong>for</strong> ticker <strong>in</strong> tickers_available<br>        }<br>    )<br>    prices.index = df.index<br>    prices = prices.iloc[:-1]<br>    <strong>del</strong> df<br>    prices.to_csv(<br>        f&quot;sp500_prices_{start_date}_{end_date}.csv&quot;)</pre><h3>Returns and correlation matrix</h3><p>We first start by computing the percentage of price changes for each stock, the return of each stock.</p><pre>returns_all = prices.pct_change()<br><em># the isn&#39;t no return in the first day<br></em>returns_all = returns_all.iloc[1:, :]<br>returns_all.dropna(axis=1, thresh=len(returns_all.index)//2., inplace=True)<br>returns_all.dropna(axis=0, inplace=True)<br>symbols = returns_all.columns.values</pre><p>We then compute the correlation matrix using these returns</p><pre><em># plot the correlation matrix with ticks at each item</em><br>correlation_matrix = returns_all.corr()<br>plt.title(f&quot;Correlation matrix from {start_date} to {end_date}&quot;)<br>plt.imshow(correlation_matrix)<br>plt.colorbar()<br>plt.savefig(&quot;correlation.png&quot;, dpi=150)<br>plt.clf()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*vYpZs6FvyoNdLz33EQVv9w.png" /><figcaption>Correlation matrix for S&amp;P500 off the first 2018 semester. Yes, it’s a mess!</figcaption></figure><p>Ok, look into the image above. If you try to find groups of stocks that are less correlated just using visual inspection, you will have a hard time. This works just for a small set of stocks. Obviously, that is another approach like using linked cluster algorithms, but they are still hard to analyze when the number of stocks increases.</p><h3>Removing irrelevant correlations with the edgeseraser library</h3><p>Our goal here is to understand the structure of the stock market communities, so we will keep just the matrix elements with positive values.</p><pre>pos_correlation = correlation_matrix.copy()<br>pos_correlation[pos_correlation &lt; 0.] = 0<br># we must remove auto-edges<br>np.fill_diagonal(pos_correlation.values, 0)</pre><p>Now, we create a weighted graph where each vertex is a stock and each positive correlation is a weight associated to the edge between two stocks. Notice that we must consider just the graph giant component.</p><pre>g = ig.Graph.Weighted_Adjacency(pos_correlation.values, mode=&#39;undirected&#39;)<br>g.vs[&quot;symbol&quot;] = returns_all.columns<br>cl = g.clusters()<br>g = cl.giant()<br>n_edges_before = g.ecount()</pre><p>The graph above has too much information, which is problematic for graph analysis. So, we must first identify and keep only the relevant edges. To do so, we use the disparity filter available in my edgeseraser package.</p><pre>_ = filter_ig_graph(g, .25, cond=&quot;both&quot;, field=&quot;weight&quot;)<br>cl = g.clusters()<br>g = cl.giant()<br>n_edges_after = g.ecount()</pre><pre>print(f&quot;Percentage of edges removed: {(n_edges_before - n_edges_after)/n_edges_before*100:.2f}%&quot;)<br>print(f&quot;Number of remained stocks: {len(symbols)}&quot;)<br>...<br>Percentage of edges removed: 95.76%<br>Number of remained stocks: 492</pre><p>We have deleted most of the edges, and only a backbone remained in our graph. This backbone structure can now extract insights from the stock market.</p><h3>nSBM: finding the hierarchical structure</h3><h3>Converting from igraph to graph-tool</h3><p>Here, we will use the graph-tool lib to infer the hierarchical structure of the graph. Thus, we must first convert the graph from igraph to graph-tool. We can accomplish that using this code:</p><pre><strong>import</strong> graph_tool.all <strong>as</strong> gt</pre><pre>gnsbm = gt.Graph(directed=False)<br><strong>for</strong> v <strong>in</strong> g.vs:<br>    gnsbm.add_vertex()<br><em># e as arestas</em><br><strong>for</strong> e <strong>in</strong> g.es:<br>    gnsbm.add_edge(e.source, e.target)</pre><h3>How to use the graph-tool nSBM?</h3><p>With the sparsified correlation graph that encompasses the stock relationships, we can ask ourselves how those relationships ties stocks together. Here, we will use the nSBM to answer this.</p><p>The <a href="https://dx.doi.org/10.1103/PhysRevX.4.011047">nSBM</a>. is an inference method that optimizes a set of parameters that defines a generative graph model for our data. In our case, to find this model for the sparsified s&amp;p500 correlation graph, we will cal the method minimize_nested_blockmodel_dl</p><pre>state = gt.minimize_nested_blockmodel_dl(gnsbm)</pre><p>The code below is used just to define the colors for our plot.</p><pre>symbols = g.vs[&quot;symbol&quot;]<br>sectors = [symbol2sector[symbol] <strong>for</strong> symbol <strong>in</strong> symbols]<br>u_sectors = np.sort(np.unique(sectors))<br>u_colors = [plt.cm.tab10(i/len(u_sectors))<br>          <strong>for</strong> i <strong>in</strong> range(len(u_sectors))]</pre><pre>u_colors[0] = [0, 1, 0, 1]<br>sector2color = {sector: color <strong>for</strong> sector, color <strong>in</strong> zip(u_sectors, u_colors)}</pre><pre>rgba = gnsbm.new_vertex_property(&quot;vector&lt;double&gt;&quot;)<br>gnsbm.vertex_properties[&#39;rgba&#39;] = rgba<br><strong>for</strong> i, symbol <strong>in</strong> enumerate(symbols):<br>    c = sector2color[symbol2sector[symbol]]<br>    rgba[i] = [c[0], c[1], c[2], .5]</pre><p>Calling the draw method, we create a plot with the nSBM result. You can play a little, changing the `β∈(0,1) parameter to see how the things change. This parameter controls the strength of the <strong>edge-bundling</strong> algorithm.</p><pre>options = {<br>    &#39;output&#39;: f&#39;nsbm_{start_date}_{end_date}.png&#39;,<br>    &#39;beta&#39;: .9,<br>    &#39;bg_color&#39;: &#39;w&#39;,<br>   <em>#&#39;output_size&#39;: (1500, 1500),</em><br>    &#39;vertex_color&#39;: gnsbm.vertex_properties[&#39;rgba&#39;],<br>    &#39;vertex_fill_color&#39;: gnsbm.vertex_properties[&#39;rgba&#39;],<br>    &#39;hedge_pen_width&#39;: 2,<br>    &#39;hvertex_fill_color&#39;: np.array([0., 0., 0., .5]),<br>    &#39;hedge_color&#39;: np.array([0., 0., 0., .5]),<br>    &#39;hedge_marker_size&#39;: 20,<br>    &#39;hvertex_size&#39;:20<br>}</pre><pre>state.draw(**options)</pre><p>Finally, we can see the hierarchical structure got from our filtered graph using the nSBM algorithm.</p><pre>plt.figure(dpi=150)<br>plt.title(f&quot;Sectors of the S&amp;P 500 from {start_date} to {end_date}&quot;)<br>legend = plt.legend(<br>    [plt.Line2D([0], [0], color=c, lw=10)<br>        <strong>for</strong> c <strong>in</strong> list(sector2color.values())],<br>    list(sector2color.keys()),<br>    bbox_to_anchor=(1.05, 1),<br>    loc=2,<br>    borderaxespad=0.)</pre><pre>plt.imshow(plt.imread(f&#39;nsbm_{start_date}_{end_date}.png&#39;))<br>plt.xticks([])<br>plt.yticks([])<br>plt.axis(&#39;off&#39;)</pre><pre>plt.savefig(f&#39;nsbm_final_{start_date}_{end_date}.png&#39;, bbox_inches=&#39;tight&#39;,<br>            dpi=150, bbox_extra_artists=(legend,), facecolor=&#39;w&#39;, edgecolor=&#39;w&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*eKOgDs7AfrruNSYLv7jpvg.png" /><figcaption>nSBM result for the correlation matrix of S&amp;P500 returns.</figcaption></figure><p>Wow, beautiful, isn’t it? But maybe you can only see a beautiful picture and not be able to understand what is happening. So, let’s try to understand what this plot is telling us.</p><h3>How to analyze?</h3><ul><li>Each circle (leaf) represents a stock. A vertex in the original graph.</li><li>Each group of leafs which resembles a broom is a community detected by the nSBM. Navigating backwards through the arrows, we can see the hierarchical community structure. The image above shows three communities that belong to the same grandfather community.</li></ul><p>We can see interesting stuff in the community structure. For example, most stocks related to <strong>consumer staples</strong>, <strong>real state</strong>, and <strong>utilities</strong> belongs to the same community.</p><p>What is the meaning of the edges?</p><ul><li>A larger number of edges between <strong>Financials</strong>, <strong>Industrials</strong> and <strong>Information technology</strong> survive the filtering technique, which shows a strong relationship between these sectors.</li></ul><p>The visual inspection of the edges relationships is meaningful only because of the use of edge bundling algorithm. Let’s see how a weak bundling turns our job into a hard one, setting the beta=0,5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*Fb4-KovEtdEcXm0c_0txvA.png" /><figcaption>Can you understand what is happening here?</figcaption></figure><p>A visual inspection is sometimes not enough, and maybe you need to create an automatic analysis, which is easy to do with the graph-tool. For example, to get a summary of the hierarchical community structure got by the nSBM algorithm we can call the print_summary method.</p><pre>state.print_summary()</pre><pre>l: 0, N: 483, B: 25<br>l: 1, N: 25, B: 6<br>l: 2, N: 6, B: 2<br>l: 3, N: 2, B: 1<br>l: 4, N: 1, B: 1</pre><p>At the first level, we have <strong>21</strong> communities for all the <strong>11</strong> sectors.</p><p>If we want to know which communities the <strong>TSLA</strong> stock belongs to, we go through the hierarchy in the reverse order,</p><pre>symbol = &quot;TSLA&quot;<br>index_tesla = symbols.index(symbol)<br>print(symbol, symbol2sector[symbol], symbol2name[symbol])<br>r0 = state.levels[0].get_blocks()[index_tesla]<br>r1 = state.levels[1].get_blocks()[r0]<br>r2 = state.levels[2].get_blocks()[r1]<br>r3 = state.levels[3].get_blocks()[r2]<br>(r1, r2, r3)</pre><pre>(&#39;TSLA&#39;, &#39;Consumer Discretionary&#39;, &#39;Tesla&#39;)<br>(19, 0, 0)</pre><h3>Conclusion</h3><p>The aim of this post was to give you a first impression of a method that combines the use of nSBM and edge filtering techniques to enhance our understanding of the correlation structure of the data, especially with financial data. However, we can also apply this method in different scenarios as <a href="https://www.science.org/doi/10.1126/sciadv.aaq1360">topic modeling in NLP</a> and <a href="https://arxiv.org/abs/2110.01421">survey analysis</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b531b84ee2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dissecting process and failures in Linux with lsof and strace: cases for MlOps/DevOps]]></title>
            <link>https://medium.com/@devmessias/dissecting-process-and-failures-in-linux-with-lsof-and-strace-cases-for-mlops-d7755b2ce6ca?source=rss-c18988aae45d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7755b2ce6ca</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Bruno Messias]]></dc:creator>
            <pubDate>Fri, 18 Feb 2022 13:52:03 GMT</pubDate>
            <atom:updated>2022-02-24T13:39:59.418Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*2ftcNTTBO12MDJm8_8Fdmw.png" /></figure><p>In DevOps or MlOps discovering what a process is doing now can save your system from a catastrophe. But sometimes we are already in a failure. When those failures happen, the following questions appear:</p><ul><li><em>The process it’s hanging and I don’t know why!</em></li><li><em>What is the cause of the problem?</em></li><li><em>Is it a network issue?</em></li><li>…and so on and so forth.</li></ul><p>Being able to answer these questions faster and with a precise answer can save you or give you a promotion. I’ll show you some simple examples of how those questions can be answered.</p><h3>Table of Contents</h3><p>· <a href="#50e2">Network issues</a><br> ∘ <a href="#061a">Is my server alive?</a><br> ∘<a href="#b1b6"> What is causing the process hanging?</a><br>· <a href="#8b45">Problems with regular files</a><br> ∘ <a href="#310a">Which processes is this file attached to?</a><br> ∘ <a href="#2e7b">I made a mistake! Deleted an important file! How can I recover it?</a><br> ∘ <a href="#9e65">Silent errors associated with files and permissions</a><br> ∘ <a href="#16a4">Is this process using a cache? Where can I find this cache?</a><br>· <a href="#a142">Dissecting your database system</a><br> ∘ <a href="#03bf">Is this process well behaved? How many connections does it have?</a><br> ∘ <a href="#099c">Everything is working properly?</a><br>· <a href="#3fd7">Extras related to files (/proc/) and strace</a><br> ∘ <a href="#6fb8">Creating a SYS CALL summary: what does my program do?</a><br> ∘ <a href="#61ee">Did this process start with the correct environment variables?</a><br> ∘ <a href="#4a53">I forgot to redirect the outputs! What can I do now?</a><br> ∘ <a href="#9e7e">How this program has been called? What is the working dir of the process?</a><br>· <a href="#3498">Conclusion &amp; Suggestions</a></p><h3>Concepts</h3><p>The first step to understanding a process is to analyze the output. But sometimes the output doesn’t give us enough information to use. Here I’ll talk about how we can extract useful information using the strace and lsof commands. To use these commands it is good to know two concepts: the <em>“Everything is a file ”</em> mantra and the <strong>system call</strong> mechanism.</p><h4>“Everything is a file.” — The UNIX mantra.</h4><p>What comes to your mind when someone talks about files? Maybe a jpeg or a CSV if you work as a data scientist. But in UNIX approach to do stuff everything can be a file, even network connections. When a thing is not a file it has at least a file descriptor associated with it. Maybe you’re thinking that I’m wandering from the post. <em>“How can this stuff help to improve our comprehension about a process or failure?”</em> The answer is straightforward: if everything is a file, we can use the same set of tools to list, read and interact (API) with files to analyze a generic process. Here is where lsof appears.</p><p>LSOF</p><p>lsof is an acronym for <strong>l</strong>ist of <strong>o</strong>pen <strong>f</strong>iles. In simple terms, Lsof is a command-line tool that can list open file descriptors in your machine. Besides that, Lsof allows using a set of different filters to give you a filtered list of opened files. Thus, we can list the open file descriptors of a user or a process.</p><p>To put all the open files in your machine use the following command:</p><pre>username:/$ lsof &gt; lsof_everything.txt</pre><p>The file lsof_everything.txt is huge and will look similar to this</p><pre>COMMAND     PID   TID TASKCMD               USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME<br>systemd       1                             root  cwd   unknown                                         /proc/1/cwd (readlink: Permission denied)<br>systemd       1                             root  rtd   unknown                                         /proc/1/root (readlink: Permission denied)<br>systemd       1                             root  txt   unknown                                         /proc/1/exe (readlink: Permission denied)</pre><p>Take some time to analyze the output. The output relates to the first lines to the root user and you don’t have permission to gain information about these files, which is good. Let’s remove this wasteful information filtering the SYS CALLs related to just your user.</p><pre>username:/$ lsof -u username &gt; lsof_my.txt</pre><p>The txt file is still big. Try to look into this file to see if you can find anything interesting, like a webpage address.</p><p>We have a lot of different columns in the file. But don’t be worried, I’ll show you the columns that I believe are the most important ones.</p><ul><li><strong>COMMAND: </strong>The command name used to initiate the process</li><li><strong>PID: </strong>This is an integer number that identifies a process, <strong>P</strong>rocess <strong>ID</strong>entification number.</li><li><strong>USER:</strong> The user to whom the process belongs.</li><li><strong>TYPE:</strong> This may be one of the most important columns. It has more than 60 possible values. Such column says the type of the node associated with the file. If the file is related with connections and sockets you will see things like that: <strong>IPV4, IPV6,</strong> <strong>unix</strong>, <strong>INET</strong>, etc. If it’s an regular file (csv, jpeg, txt, etc.) you will see the <strong>REG</strong> value.</li><li><strong>NODE:</strong> This helps us to identify the node associated with the file descriptor. It can be a number, a string, etc. In the case of internet protocols this column will have values like <strong>TCP, UDP</strong>.</li><li><strong>NAME:</strong> The values here will change a lot. For example, sometimes it can be a web server address or just a cryptic string. Regardless of the difficulty of interpreting the values in this column you should look carrefully here.</li></ul><p>If you want a more deep understanding of the columns just uses man lsof .</p><h4>System Calls and strace</h4><p>The SYSTEM_CALL is a mechanism that allows a program to ask the kernel for some resources, like the access of data stored in the disk. Therefore, if we have a tool to intercept those calls, we can have a deep comprehension of what a program is doing or what it want’s to do in your system. A celebrated tool to intercept the system calls is the strace.</p><p>If the strace command is not available in your system, install it. In apt-based distros just calling the following command should be enough.</p><pre>$ apt install strace</pre><p>You can use strace in two different ways. The first one is to call strace followed by the strace arguments and the command to be intercepted:</p><pre>$ strace ARGS command</pre><p>In the second way we will replace the command by the -p argument followed by the process identification number (<strong>PID</strong>) of the process to be intercepted:</p><pre>$ strace ARGS -p PID</pre><p>To discover the PID of a process, you can use the command ps aux | grep -i &#39;[p]rocess_name.</p><p>Let’s see an example. We will ask strace to intercept any system call performed by the ls command and we want to record the time that the system call was performed using the -t argument.</p><pre>$ strace -t ls</pre><p>The output will be something similar to this:</p><pre>18:02:23 execve(&quot;/usr/bin/ls&quot;, [&quot;ls&quot;], 0x7fffa727a418 /* 54 vars */) = 0<br>18:02:23 brk(NULL)                      = 0x55ebef60c000<br>18:02:23 access(&quot;/etc/ld.so.preload&quot;, R_OK) = -1 ENOENT (No such file or directory)<br>18:02:23 openat(AT_FDCWD, &quot;/etc/ld.so.cache&quot;, O_RDONLY|O_CLOEXEC) = 3<br>...</pre><pre>18:02:23 execve(&quot;/usr/bin/ls&quot;, [&quot;ls&quot;], 0x7fffa727a418 /* 54 vars */) = 0<br>18:02:23 brk(NULL)                      = 0x55ebef60c000<br>18:02:23 access(&quot;/etc/ld.so.preload&quot;, R_OK) = -1 ENOENT (No such file or directory)<br>18:02:23 openat(AT_FDCWD, &quot;/etc/ld.so.cache&quot;, O_RDONLY|O_CLOEXEC) = 3<br>...</pre><p>Each line of the strace output represents a single SYSTEM CALL. It’s easy to see the following pattern:</p><p><strong>Name of the SYS CALL(Arguments to be used in the system call) = Result of the SYS CALL</strong></p><p>Yes, it’s hard to understand the output or impossible without using a manual. For each SYS CALL you can use the man command to get more information about what each line represents. For example, to see what openat does and the meaning of each argument, use the following command:</p><pre>$ man 2 openat</pre><p>openat is a sys call responsible for requesting a file to be opened. The result of the last line in the output of ls commands, 3, means a successfully SYS CALL.</p><h3>Network issues</h3><p>First, install the following packages:</p><pre>$ python -m pip install requests flask</pre><p>Create a script server_mlops.py which we will use to simulate a server with network issues</p><pre># server_mlops.py<br>import time<br>import flask<br>​<br>app = flask.Flask(__name__)<br>​<br>@app.route(&#39;/&#39;)<br>def hello_world():<br>    sleep_time = flask.request.args.get(&#39;sleep&#39;, default=10, type=int)<br>    print(&#39;sleep_time:&#39;, sleep_time)<br>    time.sleep(sleep_time)<br>    return &#39;Hello World!&#39;<br>​<br>if __name__ == &#39;__main__&#39;:<br>    app.run()</pre><p>Starts the server:</p><pre>$ python server_mlops.py</pre><p>GET The PID of the process:</p><pre>$ ps aux | grep -i &#39;[s]erver_mlops.py&#39;</pre><p>You will see a output similar to this:</p><pre>devmess+ 19321 18.0  0.3  29716 24792 pts/5    S+   14:27   0:00 python server_mlops.py</pre><p>The number in front of the username (19321) is the PID of the process.</p><h4>Is my server alive?</h4><p>We have to use a unique set of filters to answer this question using the lsof. Therefore, we need to use the argument -a , which represents an AND operator. The -i argument asks to show just the files associated with connections and finally the argument -p INT makes the losf list just the files opened by the process with the PID INT.</p><pre>$ lsof -a -i -p 19321</pre><p>You will have a output similar to this</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/025c6a312230c519f06ef4da1470666e/href">https://medium.com/media/025c6a312230c519f06ef4da1470666e/href</a></iframe><p>The output shows that at least our server is listening in the 5000 port. Try to remove the -a argument and see what happens.</p><h4>Is my process stuck waiting for someone? What is causing the process hanging?</h4><p>This can happen in a myriad of cases. For example, in the dependency management systems like pip/conda. Thus, it is superb to know if you can answer fast if you have a problem on your side or not. Let’s create a simple simulation of this issue. To do so, create the client_mlops.py using the following code:</p><pre>#!/usr/bin/env python<br>#client_mlops.py<br>​<br>import requests<br>import argparse<br>​<br>parser = argparse.ArgumentParser()<br>parser.add_argument(<br>    &#39;--sleep&#39;, type=int, help=&#39;time to sleep&#39;, default=0)<br>args = parser.parse_args()<br>​<br>print(&#39;Ask for localhost:5000 to sleep for {} seconds&#39;.format(args.sleep))<br>r = requests.get(&#39;http://localhost:5000&#39;, params={&#39;sleep&#39;: int(args.sleep)})<br>print(r.text)</pre><p>In the above code, we have the sleep argument. This argument will ask server_mlops to wait for a couple of seconds before sending the answer. Let’s see how these situations appear to us at the system call level.</p><p>Start the client_mlops.py with the strace:</p><pre>$ strace -e poll,select,connect,recvfrom,sendto python client_mlops.py --sleep=20</pre><p>The -e argument followed by poll,select,connect,recvfrom,sendto asks to filter just the sys calls related with connections issues. The output of this fake failure will be something like this</p><pre>connect(4, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr(&quot;127.0.0.1&quot;)}, 16) = 0<br>connect(4, {sa_family=AF_INET6, sin6_port=htons(5000), inet_pton(AF_INET6, &quot;::1&quot;, &amp;sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0<br>connect(4, {sa_family=AF_INET6, sin6_port=htons(5000), inet_pton(AF_INET6, &quot;::1&quot;, &amp;sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ECONNREFUSED (Connection refused)<br>connect(4, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr(&quot;127.0.0.1&quot;)}, 16) = 0<br>sendto(4, &quot;GET /?sleep=10 HTTP/1.1\r\nHost: l&quot;..., 154, 0, NULL, 0) = 154<br>recvfrom(4,</pre><p>In the last line we see an unfinished recvfrom <strong>SYS_CALL</strong> . Do you want to know more about recvfrom? Execute man 2 recvfrom in a terminal session. But what matters here? The point is that: strace is telling us there is no problem with our client program, something is problematic on the server side.</p><p>You can also use the lsof to investigate this problem. Let’s simulate this scenario.</p><p>Starts the client again:</p><pre>$ python client_mlops.py --sleep=100</pre><p>Now, get the PID using ps aux | grep -i &#39;[c]lient_mlops.py&#39; and execute the lsof with the obtained PID</p><pre>lsof -a -i -p 19321</pre><p>the output will be something like this</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20612e90f9d532954b835f085a0cfb9a/href">https://medium.com/media/20612e90f9d532954b835f085a0cfb9a/href</a></iframe><p>What lsof is telling us is that: ‘’<em>You client seems fine. At least it is connected with the server</em>“”. What does this mean? This means the answer was not of great help.</p><h3>Problems with regular files</h3><p>Sometimes you can have a problem with regular files and you don’t know. Nothing was printed in the output. For example, a wrong cache being used, a program that tries to access a file that doesn’t have permission, a malicious or a bad writing process accessing/creating files in your system, etc.</p><p>Here we will simulate simple examples related to regular files. To do so, first, copy any file to the tmp folder or just call the following command:</p><pre>$ man strace &gt; /tmp/dummy_file.txt</pre><h4>Which processes is this file attached to?</h4><p>Being able to answer this question can be quite useful. For example, suppose that there is a huge file in your disk being created, almost filling up your system and you want to discover which process is doing this.</p><p>First, create the following script.</p><pre>#!/usr/bin/env python<br># file_open.py<br>import time<br>​<br>f = open(&#39;/tmp/dummy_file.txt&#39;, &#39;r&#39;)<br>input(&#39;Press Enter to continue...&#39;)</pre><p>Now open two different terminal sessions and perform the following command python file_open.py. in each one.</p><p>To find all the processes which are attached to dummy_file.txt you just need to call lsof like this:</p><pre>$ lsof /tmp/dummy_file.txt</pre><p>The output should be something similar to this:</p><pre>COMMAND   PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME<br>python  15411 devmessias    3r   REG    8,2        0 2911031 /tmp/dummy_file.txt<br>python  20777 devmessias    3r   REG    8,2        0 2911031 /tmp/dummy_file.txt</pre><p>We have two distinct processes, two different PIDs, using the same file.</p><h4>I made a mistake! Deleted an important file! How can I recover it?</h4><p>Here the simulation will be more than one more process attached to a file and you accidentally perform some action that deletes the file.</p><p>Create a file accident.txt. Open a terminal session and perform the following command. <strong>Don’t close the session!</strong></p><pre>$ python -c &#39;f=open(&quot;accident.txt&quot;, &quot;r&quot;);input(&quot;...&quot;)&#39;</pre><p>In another session, perform the following commands:</p><pre>$ rm accident.txt<br>$ ls accident.txt</pre><p>And it’s gone :(</p><pre>ls: cannot access &#39;acidente.txt&#39;: No such file or directory</pre><p>Don’t worry! Linux has a lot of cool aspects and one of them will help us to recover our file.</p><p>In Linux any process has a directory associated with it, this directory will be inside of the /proc folder. And what do these directories store? A lot of things! I assure you that you can be surprised. For example, these folders also store the file descriptors associated with any process using accident.txt. Let’s see if there are any processes using this file.</p><pre>$ lsof -u your_username | grep &#39;accident.txt&#39;</pre><p>In my case, I’ve obtained the following output:</p><pre>python    22465 devmessias    3r      REG                8,2     37599   14288174 path/accident.txt (deleted)</pre><p>This is great news! We have a process PID 22465 that still has a file descriptor associated with accident.txt. Now we can use a simple cp command to recover the data. To do so, we need to use the file descriptor number. In my case is 3 which is the number in front of the r character in the output above.</p><pre>$ cp /proc/22465/fd/3 recovered.txt</pre><p>Now just call nano recovered.txt and testify the result . It’s not magic! It’s just how the <strong>process pseudo-filesystem</strong> works!</p><h4>Silent errors associated with files and permissions</h4><p>Let’s create a simple example of an undue access to a file using the following script</p><pre>#!/usr/bin/env python<br># file_404.py<br>import time<br>​<br>try:<br>    f = open(&#39;/tmp/file_that_dosent_exist.csv&#39;, &#39;r&#39;)<br>except FileNotFoundError:<br>    pass<br>​<br>try:<br>    # create a file with sudo and then change the permission using chmod 700<br>    f = open(&#39;/tmp/file_wrong_permission.csv&#39;, &#39;r&#39;)<br>except PermissionError:<br>    pass<br>​<br>input(&#39;Press Enter to continue...&#39;)</pre><p>Call python file_404.py, nothing will appears.</p><p>To trace any SYS CALL related to regular files made by python file_404.py you just need to use the -e trace=file arg, like this:</p><pre>$ strace -f -e trace=file python file_404.py</pre><p>The argument -f says to strace to monitor any children&#39;s process. You probably will use -f every time when you’re dealing with strace in real case scenarios.</p><p>The output of the previous command should be like this</p><pre>lstat(&quot;something/file_404.py&quot;, {st_mode=S_IFREG|0644, st_size=242, ...}) = 0<br>openat(AT_FDCWD, &quot;file_404.py&quot;, O_RDONLY) = 3<br>openat(AT_FDCWD, &quot;/tmp/file_that_dosent_exist.csv&quot;, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, &quot;/tmp/file_wrong_permission.csv&quot;, O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)</pre><p>Cool! With the strace we are able to identify errors even when the programmer used a dangerous practice in the code.</p><p>Ok, but let’s improve our output. We can filter the strace output redirecting into it the awk (or grep) and performing a conditional check that each line should start with the open string and have the pattern = -1 in the line. The -1 means that the openat SYS CALL had returned an error.</p><pre>$ strace -f -e trace=file python file_404.py 2&gt;&amp;1 | awk &#39;/^open/ &amp;&amp; /= -1/ {print}&#39;</pre><p>The output now will be simpler and easier to analyze:</p><pre>openat(AT_FDCWD, &quot;/home/devmessias/anaconda3/pyvenv.cfg&quot;, O_RDONLY) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, &quot;/tmp/arquivo_404.csv&quot;, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>openat(AT_FDCWD, &quot;/tmp/arquivo_permission.csv&quot;, O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)</pre><p>If you are using the last versions of strace (5.2&gt;=) you can use a more simple command like this:</p><pre>$ strace -f -e trace=openat -e status=failed python file_404.py</pre><h4>Is this process using a cache? Where can I find this cache? Which configs files does this process use?</h4><p>It is a tedious task to search for cache or configs files used by a process and sometimes we need to delete these cache files. Another task that appears very often is to discover which files a process is using or if it is doing anything strange in your system. Maybe you want to discover if your python script is using the correct libs and files. For all these situations, you can use the following command:</p><pre>$ strace -f -e trace=file command</pre><p>If you want to get just the SYS CALLs that was performed successfully do the next:</p><pre>$ strace -f -e trace=file comando 2&gt;&amp;1 | awk &#39;/^open/ &amp;&amp; !/= -1/ {print}&#39;</pre><p>In the above command the ! is as a negation parameter for the awk search.</p><p>Again, the new versions of strace allows the status flag:</p><pre>$ strace -f -e trace=openat -e status=successful python file_404.py</pre><h3>Dissecting your database system</h3><p>The strace and lsof are powerful tools to discover and solve bugs. You don’t need to believe me. Just check the number of bugs in MySQL that strace was used to tackle <a href="http://mysqlentomologist.blogspot.com/2017/12/using-strace-for-mysql-troubleshooting.html">“using strace for mysql troubleshooting</a>. Thus, it’s not a surprise that we can use strace in our daily life dealing with databases.</p><h4>Is this process well behaved? How many connections does it have?</h4><p>In MlOps or DevOps we always need to deal with database connections. Sometimes we must check if a process is closing these connections or is creating more than necessary. If these connections are made using the <strong>TCP</strong> protocol you can list all established connections using the following command:</p><pre>$ lsof -iTCP -sTCP:ESTABLISHED</pre><p>As expected, we get a lot of unwanted information like website addresses and other application communications. If we want to list just the <strong>TCP</strong> connection in a process, we must pass the <strong>PID</strong> like this:</p><pre>$ lsof -iTCP -sTCP:ESTABLISHED -p 22157<br>​<br>COMMAND   PID       USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME<br>python  22157 devmessias    4u  IPv4 9474517      0t0  TCP localhost:35932-&gt;localhost:mysql (ESTABLISHED)<br>python  22157 devmessias    5u  IPv4 9474518      0t0  TCP localhost:35934-&gt;localhost:mysql (ESTABLISHED)<br>python  22157 devmessias    6u  IPv4 9475529      0t0  TCP localhost:37048-&gt;localhost:5000 (ESTABLISHED)</pre><p>As can you see we also have some connections in our server that are not related to the mysql. If we want to investigate just the mysql connections between all the processes in our system just do:</p><pre>$ lsof -iTCP:mysql -sTCP:ESTABLISHED <br>python  22157 devmessias    4u  IPv4 9474517      0t0  TCP localhost:35932-&gt;localhost:mysql (ESTABLISHED)<br>python  22157 devmessias    5u  IPv4 9474518      0t0  TCP localhost:35934-&gt;localhost:mysql (ESTABLISHED)</pre><p>The process identified by <em>22157</em> PID has two connections with our mysql server.</p><p>Notice we have used a pattern in the -i argument. This pattern follows this structure:</p><pre>lsof -i[protocol][@hostname|hostaddr][:service|port]</pre><h4>Everything is working properly?</h4><p>Let’s give you a taste of what we can extract from the mysql service using strace in order to get a comprehension about the processes.</p><p>Get the PID of mysqld</p><pre>$ps aux | grep -i &#39;[m]ysqld&#39;<br>mysql    14001     1  0 13:41 ?        00:00:18 /usr/sbin/mysqld</pre><p>We will ask the strace to increase the size of the strings up to 50 characters (-s 50) and we will omit any SYS CALL of the type io_getevents , nanosleep and futex.</p><pre>$ sudo strace -s 50 -f -e trace=!io_getevents,nanosleep,futex -p 10767</pre><p>Chose one of your databases and tables to do the following examples. Here, I’ve performed this SQL query:</p><pre>SELECT * FROM  product WHERE product_id = 1;</pre><p>My output prompted some stuff like this</p><pre>[pid 14334] recvfrom(52, &quot;,\0\0\0&quot;, 4, MSG_DONTWAIT, NULL, NULL) = 4<br>[pid 14334] recvfrom(52, &quot;\3SELECT * FROM  product WHERE product_id = 1&quot;, 44, MSG_DONTWAIT, NULL, NULL) = 44<br>[pid 14334] sendto(52, &quot;\1\0\0\1\5F\0\0\2\3def\16farmers_market\7product\7product\nprodu&quot;..., 477, MSG_DONTWAIT, NULL, 0 &lt;unfinished ...&gt;<br>[pid 14207] sched_yield( &lt;unfinished ...&gt;<br>[pid 14334] &lt;... sendto resumed&gt;)       = 477<br>[pid 14207] &lt;... sched_yield resumed&gt;)  = 0<br>...</pre><p>We can see the SQL query above. This also shows how strace can help us to gain a deep understanding about our system. We can see how the sql queries are communicating using de recvfrom and sendfrom calls. The man 2 recvfrom says the first number, 52, represents the file descriptor associated with a UNIX socket.</p><p>We can use this approach to investigate IO problems (<a href="https://newbiedba.wordpress.com/2017/01/04/using-strace-in-linux-to-troubleshoot-database-performance-issues/">using-strace-in-linux-to-troubleshoot-database-performance-issues</a>) as well many others. But let’s simulate a lock condition and see what happens.</p><p>Start a session and initiate any transaction. Don’t finish with a <strong>COMMIT;</strong> command!</p><pre># first session<br>MariaDB [you_db]&gt; BEGIN;<br>Query OK, 0 rows affected (0.001 sec)<br>MariaDB [your_db]&gt; UPDATE customer SET customer_first_name = &#39;something&#39; WHERE customer_id=1;<br>Query OK, 0 rows affected (0.001 sec)<br>Rows matched: 1  Changed: 0  Warnings: 0<br>MariaDB [you_db]&gt;</pre><p>Looks in the strace output, something like this should appear</p><pre>[pid 14334] recvfrom(52, &quot;I\0\0\0&quot;, 4, MSG_DONTWAIT, NULL, NULL) = 4<br>[pid 14334] recvfrom(52, &quot;\3UPDATE customer SET customer_first_name = &#39;Brun&quot;..., 73, MSG_DONTWAIT, NULL, NULL) = 73<br>[pid 14334] sendto(52, &quot;0\0\0\1\0\0\0\3\0\0\0(Rows matched: 1  Changed: 0  Warnings:&quot;..., 52, MSG_DONTWAIT, NULL, 0) = 52<br>[pid 14334] recvfrom(52, 0x7fd354007348, 4, MSG_DONTWAIT, NULL, NULL) = -1 EAGAIN <br>[pid 14334] poll([{fd=52, events=POLLIN|POLLPRI}], 1, 28800000<br>​</pre><p>The poll is a system call that will wait for any change in the file descriptor 52. The process is waiting for the COMMIT; clause in our first mysql session. Note the absence of the enclosing ) in the last line.</p><p>Open another mysql session, try to execute the same SQL query (whitouth the BEGIN)</p><pre># second session<br>MariaDB [your_db]&gt; UPDATE customer SET customer_first_name = &#39;something&#39; WHERE customer_id=1;</pre><p>The db row it’s in a lock state. If we look in the strace output just some new lines will be printed, like this</p><pre>&lt;unfinished ...&gt;<br>[pid 29884] recvfrom(83, &quot;I\0\0\0&quot;, 4, MSG_DONTWAIT, NULL, NULL) = 4<br>[pid 29884] recvfrom(83, &quot;\3UPDATE customer   SET customer_first_name = &#39;someth&quot;..., 73, MSG_DONTWAIT, NULL, NULL) = 73</pre><p>Our second transaction is waiting for the first to be committed in our db. If you perform a commit in the first session this will happen in the strace output:</p><pre>[pid 14334] &lt;... poll resumed&gt;)         = 1 ([{fd=52, revents=POLLIN}])<br>[pid 14334] recvfrom(52, &quot;\7\0\0\0&quot;, 4, MSG_DONTWAIT, NULL, NULL) = 4<br>[pid 14334] recvfrom(52, &quot;\3COMMIT&quot;, 7, MSG_DONTWAIT, NULL, NULL) = 7<br>...</pre><p>Your first transaction was performed and the lock was released allowing the second transaction to be executed.</p><h3>Extras related to files (/proc/) and strace</h3><p>I gave you some examples about connection issues, regular files accidents and database management. Here I’ll give more examples that I believe are not so useful although they are very interesting.</p><h4>Creating a SYS CALL summary: what does my program do?</h4><p>An overview of what your program does can help you to perform some optimizations or discover something strange. We can use the strace -c to get an overview of the system calls. For example, the following command gave me a summary of system calls of a make sync-env that I’m using in one of my projects:</p><pre>$ strace -c -e trace=!\wait4 make sync-env</pre><p>The exclamation mark, -e trace=!\wait4, in the above command tells strace to ignore any wait4 system call.</p><p>What I’ve obtained was this</p><pre>% time     seconds  usecs/call     calls    errors syscall<br>------ ----------- ----------- --------- --------- ----------------<br> 14,54    0,000209           6        33        13 openat<br> 13,01    0,000187          17        11           vfork<br> 12,32    0,000177           7        25           mmap<br>  8,49    0,000122           3        31           close<br>  8,42    0,000121           5        21           rt_sigprocmask<br>  8,14    0,000117           6        17           read<br>  6,89    0,000099           5        19        11 stat<br>  5,85    0,000084           3        23           fstat<br>  2,85    0,000041           8         5           mprotect<br>  2,64    0,000038           9         4           write<br>  2,51    0,000036           2        16           fcntl<br>  2,02    0,000029           3         9           rt_sigaction<br>  1,95    0,000028          14         2           readlink<br>  1,95    0,000028          14         2           getdents64<br>  1,25    0,000018           4         4           brk<br>  1,25    0,000018          18         1         1 access<br>  1,25    0,000018           3         5           pipe<br>  1,11    0,000016           4         4           ioctl<br>  0,84    0,000012           6         2           getcwd<br>  0,70    0,000010          10         1           munmap<br>  0,49    0,000007           7         1           lstat<br>  0,49    0,000007           7         1           execve<br>  0,49    0,000007           3         2           prlimit64<br>  0,35    0,000005           5         1           chdir<br>  0,21    0,000003           3         1           arch_prctl<br>------ ----------- ----------- --------- --------- ----------------<br>100.00    0,001437                   241        25 total</pre><p>What can we extract from the above output? A lot of things. For example make sync-env spent 14% of the time doing system calls of the type opennat and 20 of these 33 openat calls had some problem.</p><h4>Did this process start with the correct environment variables?</h4><p>We have several reasons to use environment variables. These variables are easy to configure, improve security, and prevent errors. So they are used everywhere to store a secret, point to a lib and much more. However, sometimes we are not so sure whether a process is using the correct environment variables or not.</p><p>Let’s try something simple</p><pre>$ ANSWER=42 python script.py</pre><p>As I said previously the /proc is responsible for storing the state of any process running in your machine. So, it’s not a surprise that we can extract the environment variables from that.</p><p>To print the environment variables of the process with a PID 4301 just call this cat /proc/4031/environ and get an ugly output. To improve the output we can use the tr and replace the null characters \0 by break lines \n. Like this:</p><pre>$ tr &#39;\0&#39; &#39;\n&#39; &lt; /proc/4031/environ</pre><p>You will have a output similar to this</p><pre>ANSWER=42<br>SHELL=/bin/bash<br>LANGUAGE=en_US<br>JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/bin/java<br>...more stuff</pre><p>If you want to look just at the environment variables with a given string pattern you can use awk , grep, or anything that you feel more comfortable. For example, doing this</p><pre>$ tr &#39;\0&#39; &#39;\n&#39; &lt; /proc/4031/environ 2&gt;&amp;1 | awk &#39;/^CONDA/ {print}&#39;</pre><p>I’ve obtained this</p><pre>CONDA_EXE=/home/devmessias/anaconda3/bin/conda<br>CONDA_PREFIX=/home/devmessias/anaconda3<br>CONDA_PROMPT_MODIFIER=(base) <br>CONDA_SHLVL=1<br>CONDA_PYTHON_EXE=/home/devmessias/anaconda3/bin/python<br>CONDA_DEFAULT_ENV=base</pre><h4>I forgot to redirect the outputs! What can I do now?</h4><p>Suppose you started a process without redirecting the output to a file. Maybe you forgot or you are too optimistic about the problems and now you want to persist the errors. If restart the process it’s out of question you can use the strace to solve your headache. Let’s see how we can solve that.</p><p>The system calls responsible to request the kernel to write in the <strong>stdin, stdout</strong> and <strong>stderr</strong> is the write . If you want to know more you should read the manual</p><pre>$ man 2 write</pre><p>But the most important part of the write manual is on the top and is this:</p><pre>NAME<br>       write - write to a file descriptor<br>SYNOPSIS<br>       #include &lt;unistd.h&gt;<br>       ssize_t write(int fd, const void *buf, size_t count);</pre><p>As you can see the first argument is an integer that represents the file descriptor. If <strong>fd=1</strong> this means a writing in the <strong>stdout</strong> and if <strong>fd=2</strong> the writing will be in the <strong>stderr</strong>. So, it’s an easy piece here. We just need to monitor any sys call write with the fd equals to 1 or 2 and save the values in a file.</p><p>When I need to do this (two or three times in my life time) I use the following pattern</p><pre>$ strace -f -t -etrace=write -s 1000 -p 4320 2&gt;&amp;1 | grep --line-buffered -e &#39;write(2, &#39; -e &#39;write(1, &#39; &gt; out.txt</pre><p>I’m asking strace to monitor any SYS CALL write from the process with the PID 4320 or children created by them (-f) . And saving the output in the out.txt file.</p><p>The following code changes the server_mlops.py to help you to explore more this scenario.</p><pre># server_mlops.py<br>import time<br>import flask<br>import sys<br>​<br>app = flask.Flask(__name__)<br>​<br>@app.route(&#39;/&#39;)<br>def hello_world():<br>    sleep_time = flask.request.args.get(&#39;sleep&#39;, default=10, type=int)<br>    print(&#39;sleep_time:&#39;, sleep_time)<br>    for i in range(sleep_time):<br>        print(f&#39;INFO: {i} of sleep_time \n asdf \t &#39;)<br>        print(f&#39;ERROR: Example msg {i}&#39;, file=sys.stderr) <br>        time.sleep(1)<br>    return &#39;Hello World!&#39;<br>​<br>if __name__ == &#39;__main__&#39;:<br>    app.run()</pre><h4>How this program has been called? What is the working dir of the process?</h4><p>Ok, you can answer these questions using htop. However we can get the same information without installing anything, just looking in a file inside of the /proc/ folder, like this</p><pre>$ tr &#39;\0&#39; &#39;\t&#39; &lt; /proc/A_PID_NUMBER/cmdline</pre><p>In my case I’ve obtined this</p><pre>python client.py --sleep 1000</pre><p>To discover the working directory do this</p><pre>$ readlink /proc/A_PID_NUMBER/cwd</pre><h3>Conclusion &amp; Suggestions</h3><p>I hope that after reading this post you can be more prepared to face problems in your daily tasks. But if you have any suggestion you can send me an email <a href="https://medium.com/devmessias@gmail.com">devmessias@gmail.com</a>. Thanks!</p><p>If you want to know more about strace and linux in general I strongly recommend spent some hours navigating and reading Julia Evans’ blog <a href="https://jvns.ca/">https://jvns.ca/</a>.</p><h3>Acknowledgements</h3><blockquote><a href="https://www.linkedin.com/in/reynaldoallanf">Reynaldo Allan Fulin</a> and <a href="https://www.linkedin.com/in/elisarma">Elisa Ribeiro</a> .</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7755b2ce6ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>