<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Juan Vieira on Medium]]></title>
        <description><![CDATA[Stories by Juan Vieira on Medium]]></description>
        <link>https://medium.com/@ju4nv1e1r4?source=rss-4c9aed6dca41------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lL_Bcye9Fc8SAVmVN58KgQ@2x.jpeg</url>
            <title>Stories by Juan Vieira on Medium</title>
            <link>https://medium.com/@ju4nv1e1r4?source=rss-4c9aed6dca41------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:53:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ju4nv1e1r4/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Como reduzi alucinações de LLMs usando Injeção de Contexto e Redis (sem fine-tuning)]]></title>
            <link>https://medium.com/@ju4nv1e1r4/como-reduzi-alucina%C3%A7%C3%B5es-de-llms-usando-inje%C3%A7%C3%A3o-de-contexto-e-redis-sem-fine-tuning-48c316b8069a?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/48c316b8069a</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[context-engineer]]></category>
            <category><![CDATA[caching]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 13:01:07 GMT</pubDate>
            <atom:updated>2025-12-16T13:01:07.705Z</atom:updated>
            <content:encoded><![CDATA[<p>Ás vezes, a solução para uma IA mais inteligente não é um modelo maior, mas uma arquitetura melhor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJrp_2DPiD-MXl__AqoASQ.png" /><figcaption>Image credits: Towards AI</figcaption></figure><p>Em engenharia de IA, existe uma tentação constante de resolver problemas de performance com “força bruta”: usar o modelo mais caro, fazer <em>fine-tuning</em> com milhares de exemplos ou criar cadeias de RAG complexas. Outra forma que a maioria dos profissionais encontram, seria o prompt engineering. Mas muitas vezes, seja pelo excesso (ruído semântico) ou pela falta de informações e/ou instruções claras.</p><p>Recentemente, enfrentei um desafio comum, mas crítico, em um módulo: o LLM sofria de “amnésia” e alucinações constantes. A solução, no entanto, não veio de um novo modelo, mas de um princípio clássico da engenharia: <strong>Gerenciamento de Estado.</strong></p><p>Neste post, vou compartilhar como utilizei <strong>Redis</strong> e um padrão de <strong>Injeção Dinâmica de Contexto</strong> para limpar a janela de atenção da IA, economizar tokens e, principalmente, fazer o LLM parar de inventar informações que não existiam em seu gigante contexto.</p><h3>A Poluição do Contexto</h3><p>O cenário era um assistente de agendamento. O fluxo parecia simples: o usuário pede horários disponíveis, o programa consulta a API e devolve as opções.</p><p>Porém, na prática, enfrentávamos dois problemas graves:</p><ul><li><strong>A Janela de Atenção Saturada:</strong> Ao retornar listas extensas de horários diretamente no histórico da conversa, poluíamos o contexto. O modelo precisava “ler” centenas de tokens de horários passados para responder a uma nova pergunta simples. O resultado? O modelo se confundia, misturava tudo, e pior, alucinava horários que não existiam na lista original.</li><li><strong>A Quebra de Fluxo:</strong> Se o usuário escolhesse um horário, mas o sistema precisasse interromper para pedir um dados, a “intenção” de agendamento se perdia no meio da conversa. O usuário fornecia os seus dados e o LLM respondia com “Obrigado. Em que posso ajudar?”, esquecendo totalmente que estávamos no meio de uma reserva.</li></ul><p>Precisávamos desacoplar a “memória de curto prazo” (o chat) dos “dados brutos” (a agenda).</p><h3>A Solução: Dynamic Context Injection</h3><p>A arquitetura que desenhei baseia-se no conceito de que a LLM deve ter acesso apenas à “verdade” do momento atual, sem carregar o peso de interações passadas irrelevantes. Para isso, utilizei o Redis como um gerenciador de estado.</p><p>A solução se apoia em quatro pilares:</p><h4>1. Gerenciamento de Estado usando Redis (com TTL)</h4><p>Em vez de jogar os horários disponíveis no texto do chat, passei a armazenar o estado da agenda no Redis. Criamos chaves estruturadas como ctx:agenda:hoje ou ctx:user:{id}:intent. Defini um <strong>TTL (Time-to-Live)</strong> curto, de cerca de 10 minutos. Isso garante que os dados sejam efêmeros: se o usuário sair e voltar horas depois, não queremos que a IA tente agendar um horário que já expirou.</p><h4>2. Higiene de Contexto</h4><p>Um dos maiores causadores de alucinação é a informação contraditória no histórico. Se o usuário perguntou sobre X e depois mudou para Y, os horários de X antigos no histórico atrapalham a inferência. Implementei uma lógica de limpeza: quando a intenção muda, o código apaga as chaves conflitantes no Redis. Isso garante um contexto “higienizado”.</p><h4>3. Pipeline de Injeção no System Prompt</h4><p>Esta é a parte mais importante da Engenharia de IA. Desenvolvi um método interceptador, _get_dynamic_context, que roda <em>antes</em> da chamada ao LLM. Ele verifica se existe algo &quot;quente&quot; no Redis. Se houver, ele injeta esses dados diretamente nas instruções do sistema (<em>System Prompt</em>), e não no histórico de mensagens.</p><p><strong>O fluxo funciona assim:</strong></p><ol><li>Usuário solicita algo.</li><li>Backend busca na API e salva no Redis (não manda tudo pro chat).</li><li>O System Prompt recebe: <em>&quot;O usuário está vendo a seguinte lista: [JSON do Redis]. Use isso como verdade absoluta.&quot;</em></li><li>O histórico da conversa permanece limpo e leve.</li></ol><h4>4. Resiliência de Fluxo (State Recovery)</h4><p>Para resolver o problema da interrupção, implementei flags de estado, como was_pending = True. Se o sistema precisa parar para pedir um dado, ele salva o estado anterior. Assim que o usuário responde com o dado faltante, o sistema consulta essa flag, &quot;lembra&quot; o que estava fazendo e retoma a busca de aulas automaticamente, sem que o usuário precise repetir sua solicitação.</p><h3>Implementação Simplificada</h3><p>Para ilustrar, aqui está um exemplo em Python (simplificado) de como o interceptador funciona:</p><pre>import redis<br><br>r = redis.Redis(host=&#39;localhost&#39;, port=6379, db=0)<br><br>def build_system_prompt(user_id, base_prompt):<br>    &quot;&quot;&quot;<br>    Constrói o prompt do sistema injetando contexto dinâmico<br>    apenas se ele existir e for relevante.<br>    &quot;&quot;&quot;<br>    context_keys = [f&quot;ctx:appointments:{user_id}&quot;, f&quot;ctx:booking_status:{user_id}&quot;]<br>    dynamic_context = []<br><br>    for key in context_keys:<br>        data = r.get(key)<br>        if data:<br>            dynamic_context.append(data.decode(&#39;utf-8&#39;))<br>    <br>    if dynamic_context:<br>        injected_instruction = &quot;\n### DADOS ATUAIS DO SISTEMA ###\n&quot;<br>        injected_instruction += &quot;\n&quot;.join(dynamic_context)<br>        injected_instruction += &quot;\nUtilize APENAS os dados acima para responder.&quot;<br>        <br>        return base_prompt + injected_instruction<br>    <br>    return base_prompt</pre><h3>Resultados e Impacto</h3><p>Essa mudança de arquitetura trouxe ganhos que foram além da simples correção de bugs:</p><ul><li><strong>Redução Drástica de Alucinações:</strong> A segregação entre instruções e histórico aumentou a precisão. O modelo parou de inventar horários porque a “janela de contexto” dele agora contém apenas os dados válidos injetados no momento da inferência.</li><li><strong>Otimização de Tokens (Custo):</strong> Como não repetimos listas enormes de JSON no histórico da conversa (que é reenviado a cada interação), economizamos tokens de input significativamente.</li><li><strong>UX Fluida:</strong> O usuário sente que está conversando com alguém que tem memória. A reconexão automática do fluxo após fornecer dados cadastrais eliminou uma das maiores fricções do produto.</li><li><strong>Observabilidade:</strong> Com logs estruturados indicando quando o contexto foi injetado (ou falhou), conseguimos debugar interações complexas com muito mais facilidade.</li></ul><h3>Conclusão</h3><p>Como Engenheiros de IA, muitas vezes focamos muito no modelo e pouco no sistema ao redor dele. Este caso provou que tratar o contexto da LLM como um recurso gerenciável é mais eficiente do que esperar que o modelo “se vire” com um prompt gigantesco.</p><p>A chave para reduzir alucinações nem sempre é mais inteligência artificial. Às vezes, é apenas uma boa e velha arquitetura de software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48c316b8069a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Separation of Concerns]]></title>
            <link>https://medium.com/@ju4nv1e1r4/separation-of-concerns-87bfc8ee939f?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/87bfc8ee939f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[separation-of-concerns]]></category>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 16:29:03 GMT</pubDate>
            <atom:updated>2025-10-31T16:29:03.770Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*HwFH-DlYnQlRW8NfXxg8cA.png" /></figure><p>If you follow the world of AI/ML, you’ve probably noticed that the buzzword is “agent.” The idea is that, with complex function calls and prompts, we can transform a generalist LLM (like Gemini 1.5 Pro or GPT-4o) into a central brain that can reason, make decisions, route tasks, and control tools.</p><p>Here I want to talk about going against the market trend.</p><p>As an AI/ML engineer, I’ve noticed that handing over the entire decision flow to a single generalist model is neither efficient nor safe. In practice, it’s expensive and opens up a huge space for often inexplicable bugs, failures, loops, and erratic behavior.</p><p>The reason is simple: LLMs are machine learning models. Their core function is to generate text. Making a logical decision is a deterministic task, and LLMs are, by definition, probabilistic systems.</p><h3>Probabilistic Reasoning vs. Deterministic Logic</h3><p>This is the crucial distinction.</p><p>When you force a generalist LLM to act as an “agent,” you are asking it to be your application server. You are asking a trained model to predict the next token most likely to execute business logic.</p><p>You are introducing multiple probabilistic failure points into a system that should be deterministic.</p><p>An LLM agent needs to decide:</p><ol><li>Whether to call a tool;</li><li>Which tool to call (from several options);</li><li>Which parameters to extract from the text to populate the tool;</li><li>How to format the output (usually a JSON).</li></ol><p>At each of these steps, the model can fail. It might hallucinate a parameter that doesn’t exist, choose the wrong tool due to a nuance in the user’s language, or, as many of us have seen, fail to generate a valid JSON.</p><p>Don’t get me wrong, the ability to function call is impressive. But it’s a probabilistic suggestion, not a logical execution. And in AI/ML engineering, we don’t work with suggestions.</p><h3>Returning Logic to the Experts</h3><p>In recent months, I have been conducting Proof of Concept (PoC) projects with a different architecture, based on the classic principle of Software Engineering: <strong>Separation of Concerns.</strong></p><p>Instead of a monolithic LLM-agent, I build a pipeline of models:</p><p><strong>The Router:</strong> A small, fast, expert model (it could be a zero-shot classifier, a fine-tuned BERT-based classifier, or even a Gemma 2B) takes the first line. Its only task is deterministic: to analyze user intent.</p><p><strong>The Extractor:</strong> If necessary, another expert model (such as a <strong>NER — Named Entity Recognition model</strong>) extracts the entities in a structured way.</p><p><strong>The Generator: </strong>Only after the logic has been handled, the intent classified, and the data extracted, do I call the generalist LLM.</p><p>In an example from my current work: an LLM needed to decide whether or not to continue a conversation. Instead of polluting the main model’s prompt with this logic, I developed a separate service that makes this decision.</p><p>The generalist LLM no longer needs to decide. It only needs to perform its specialty: generating text, given a clear context and an unambiguous instruction.</p><p><strong>Advantages (and Disadvantages)</strong></p><p>The advantages of this approach are obvious to any engineer:</p><p>Reliability: My expert classifier model has testable and consistent accuracy (e.g., 99.8%). A generalist LLM, depending on the prompt, can perform at 80% to 95% on the same task, in a non-deterministic way. I can unit test my decision model.</p><p>Efficiency (Cost and Latency): An inference in a Gemma 2B or a bert-based model is orders of magnitude faster and cheaper than an API call to a frontier model (like Gemini 1.5 Pro).</p><p>Maintainability: If the routing logic changes, I update and deploy my expert microservice. I don’t need to mess with a giant system prompt and pray that the change hasn’t broken another 20 “emergent” features.</p><p>Of course, there are disadvantages. The complexity of the architecture increases. Instead of an API endpoint, I now have a microservices pipeline. Fine-tuning these expert models requires data and some effort.</p><p>But that’s the difference between an impressive demo and a robust product.</p><h3>The Dazzle Forgot the Basics</h3><p>AI/ML engineering has advanced significantly, creating generalist models with incredible reasoning capabilities. But in this dazzling frenzy, it seems we’ve forgotten the good old classic NLP models (and the smaller, more focused LLMs).</p><p>A text classifier, a NER model, or a fine-tuned summarization model are still more than capable — and often superior — for specific tasks.</p><p>The future of robust AI, in my opinion, is not a single “super-brain” agent. It’s a collaborative ecosystem: expert and deterministic models handling the logic, delivering a clean and validated task so that the generalist LLM can focus on what it does best: dealing with the nuance, context, and creativity of human language.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87bfc8ee939f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pausa, Paternidade e Código]]></title>
            <link>https://medium.com/@ju4nv1e1r4/pausa-paternidade-e-c%C3%B3digo-e3db7abc1664?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3db7abc1664</guid>
            <category><![CDATA[pause]]></category>
            <category><![CDATA[fatherhood]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Tue, 16 Sep 2025 09:50:59 GMT</pubDate>
            <atom:updated>2025-09-16T09:50:59.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4CdOa1CoWAGzmtC15BSlw@2x.jpeg" /></figure><p>Para quem acompanha meu trabalho por aqui, um silêncio de três meses pode parecer uma eternidade. E, de certa forma, foi. Nesse período, estive imerso na experiência mais intensa e transformadora da minha vida, um turbilhão de acontecimentos que redefiniram minhas prioridades, minha resiliência e até mesmo minha relação com a tecnologia.</p><p>Eu me tornei pai. E, quase simultaneamente, comecei em um novo desafio como Engenheiro de IA.</p><p>No dia 22 de maio, com apenas duas semanas na nova empresa, o Lucas chegou ao mundo. Ele decidiu vir antes da hora, com oito meses de gestação, e sua chegada nos trouxe uma série de batalhas que não esperávamos enfrentar. Não vou entrar nos detalhes das complicações, mas posso dizer que o caminho foi íngreme.</p><p>Tivemos dias de incerteza, noites de preocupação e um desafio que dói no coração de qualquer pai: aos dois meses de vida precisou passar por uma cirurgia. Ver alguém tão pequeno e frágil passar por um procedimento assim testa os seus limites. E foi nesse momento de vulnerabilidade extrema que eu pude contar com um apoio (além da família) que fez toda a diferença: o da minha empresa.</p><p>Em um cenário onde a pressão por resultados imediatos é a norma, especialmente para um recém-chegado, encontrei algo raro: humanidade e paciência. A empresa não apenas me concedeu uma licença paternidade estendida, como demonstrou uma compreensão imensa quando o Lucas precisou de sua cirurgia ou durante as emergências médicas que enfrentamos. Tive o tempo e a tranquilidade necessários para estar ao lado da minha família, para ser o pai que meu filho precisava, sem que isso gerasse uma sombra de preocupação sobre meu novo cargo. Devo muito a eles por esse apoio. Hoje, com três meses, Lucas está forte, saudável e enchendo nossos dias de alegria. E eu, com a saúde mental estável, me sinto pronto para o que vier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*d0aT30fF0w3j8cGFQUW3FQ@2x.jpeg" /></figure><p>Profissionalmente, essa jornada também foi um divisor de águas. Quem me conhece sabe da minha transição de Cientista de Dados e do meu interesse crescente por Engenharia de ML/IA, design de sistemas e arquitetura de software. Sempre tive um apreço maior pelo código, pela estrutura que sustenta a inteligência, do que pela fase de experimentação.</p><p>Esses três meses de trabalho, entrelaçados com a experiência da paternidade, me ensinaram mais do que um ano inteiro. A necessidade de ser resiliente em casa transbordou para o trabalho. A capacidade de manter a calma sob pressão, de focar na solução e entrega de resultados e de entender que sistemas complexos — sejam eles humanos ou computacionais — exigem paciência e cuidado, tornou-se uma das minhas principais soft skills. A resiliência.</p><p>Tive a sorte de encontrar um time incrível. Conheci pessoas brilhantes, que não pensam apenas em linhas de código, mas no sistema como um todo, em nível de arquitetura. Fui inserido em um projeto de IA generativa que foge do comum. Aqui, não estamos falando do mais do mesmo que vemos por ai sendo divulgado por pessoas que se dizem especialistas em IA, mas de uma solução verdadeiramente escalável, com um nível de complexidade altíssimo, projetada desde o início para ser manutenível e robusta.</p><p>Essa combinação de um desafio técnico estimulante com uma cultura que valoriza as pessoas é o que me faz acordar todos os dias, sentar na cadeira e ligar o notebook com uma vontade genuína de construir algo relevante.</p><p>A pausa foi necessária, mas o aprendizado foi contínuo. Agora, com as coisas se ajeitando, sinto a necessidade de voltar a fazer algo que me faz muito bem: escrever e compartilhar conhecimento.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*RHPGtFOauNcRoCR0J8T2tw@2x.jpeg" /></figure><p><em>Vou fazer uma promessa aqui haha.</em></p><p>Prometo retornar com conteúdos sobre engenharia de IA/ML, trazendo um pouco da bagagem e das perspectivas que adquiri e estou adquirindo nessa jornada. O Juan de antes voltou, mas agora acompanhado de uma nova e poderosa versão de si mesmo.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3db7abc1664" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating ML Models Deployment with GitHub Actions + Cloud Run]]></title>
            <link>https://medium.com/google-cloud/automating-ml-models-deployment-with-github-actions-cloud-run-17e6a6e95e30?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/17e6a6e95e30</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Thu, 08 May 2025 16:36:46 GMT</pubDate>
            <atom:updated>2025-05-08T16:36:46.093Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-0nBxjRXClLzPTos8iagQ.png" /></figure><p>A few weeks ago, I decided to do an experiment: instead of focusing only on building machine learning models with good performance, I decided to practice what often gets left behind for those who are starting out: putting everything into production with good MLOps and DevOps practices.</p><p>I talked <a href="https://medium.com/@ju4nv1e1r4/supervised-or-unsupervised-for-anomaly-detection-94543a2b0e28">about the models</a> and some stages of the pipeline of <a href="https://github.com/ju4nv1e1r4/ml-cidds">this project</a> here on <a href="https://medium.com/@ju4nv1e1r4">Medium</a> and I’ve been talking about it for a few weeks on <a href="https://www.linkedin.com/in/juanvieira85/">my LinkedIn profile</a>.</p><p>My goal was to build and design an anomaly detection system that actually works end-to-end. And when I say that, I mean much more than just running locally or well-commented notebooks.</p><h4>The main focus was:</h4><ul><li><strong>Automation</strong> (the highest level of MLOps maturity we can achieve)</li><li><strong>Reproducibility</strong> (about the ability of the model to be reused and retrained over and over again)</li><li><strong>Maintainability</strong> (the code and architecture should be clean enough to make the code easy to maintain)</li><li><strong>Observability</strong> (our eyes should be seeing the full picture of what is happening, whether with the data, the system or the ML models)</li></ul><h3>Why does CI/CD matter in Machine Learning projects?</h3><p>While CI/CD is standard in traditional software development, in the context of ML it is still often overlooked. The truth is that model deployment should not be a manual, insecure, or error-prone process.</p><h4>Without CI/CD:</h4><ul><li>You may forget to update environment variables.</li><li>Deployment may be different from machine to machine.</li><li>Small manual errors can crash the API.</li></ul><h4>With CI/CD:</h4><ul><li>Every push to the production branch triggers a clean and documented build.</li><li>Artifacts are versioned and auditable.</li><li>Deployment is consistent, safe, and fast.</li></ul><p>I talk more about this importance in a post about <a href="https://medium.com/@ju4nv1e1r4/maturidade-em-mlops-um-caso-pr%C3%A1tico-no-contexto-de-classifica%C3%A7%C3%A3o-de-cr%C3%A9dito-f4e7fd5fca98">MLOps Maturity that I wrote here on Medium</a>.</p><h3>Building Automation: GitHub Actions + Cloud Run</h3><p>It consists of two main components:</p><ol><li>An API with FastAPI that serves two anomaly detection models.</li><li>A subscriber, which receives new data via Pub/Sub and triggers inferences.</li></ol><h4>Github Actions</h4><p>First of all, let’s talk about Github Actions to contextualize it.</p><p>GitHub Actions is a continuous integration (CI) and continuous delivery (CD) platform that integrates directly with GitHub. It allows you to automate, customize, and execute software development workflows in your repository.</p><p>With GitHub Actions, you can create workflows defined in YAML files that are triggered by events in your repository, such as pushes, pull requests, or the creation of issues. These workflows consist of jobs, which in turn are composed of steps. Each step executes commands or uses “actions,” which are reusable automated tasks.</p><p>This tool makes it easy to automate tasks such as compiling code, running tests, deploying applications, and any other repetitive task in the development lifecycle, all without leaving GitHub.</p><h4>The Main Components</h4><p>Now that you know what Github Actions is all about, we can move on to the main files that make up our automation pipeline.</p><h4>1. cloudbuild.yaml — Build with Cloud Build</h4><p>This file defines the build of two Docker images, one for the API and one for the subscriber. Both images are pushed to Google Cloud Container Registry after the build.</p><pre>substitutions:<br>  _IMAGE_TAG_API: gcr.io/$PROJECT_ID/anomaly-detection-api<br>  _IMAGE_TAG_SUB: gcr.io/$PROJECT_ID/anomaly-detection-sub<br><br>steps:<br>  - name: &#39;gcr.io/cloud-builders/docker&#39;<br>    args: [&#39;build&#39;, &#39;-f&#39;, &#39;Dockerfile.api&#39;, &#39;-t&#39;, &#39;${_IMAGE_TAG_API}&#39;, &#39;.&#39;]<br><br>  - name: &#39;gcr.io/cloud-builders/docker&#39;<br>    args: [&#39;build&#39;, &#39;-f&#39;, &#39;Dockerfile.sub&#39;, &#39;-t&#39;, &#39;${_IMAGE_TAG_SUB}&#39;, &#39;.&#39;]<br><br>  - name: &#39;gcr.io/cloud-builders/docker&#39;<br>    args: [&#39;push&#39;, &#39;${_IMAGE_TAG_API}&#39;]<br><br>  - name: &#39;gcr.io/cloud-builders/docker&#39;<br>    args: [&#39;push&#39;, &#39;${_IMAGE_TAG_SUB}&#39;]<br><br>images:<br>  - &#39;${_IMAGE_TAG_API}&#39;<br>  - &#39;${_IMAGE_TAG_SUB}&#39;</pre><p><strong><em>Tip:</em></strong><em> Use substitutions to ensure that image names are flexible and reusable across different environments.</em></p><h4>2. .github/workflows/cloud-run.yaml – CI/CD com GitHub Actions</h4><p>The file starts with this section that already defines a super powerful trigger: every push to the prod branch starts the automatic deployment:</p><pre>on:<br>  push:<br>    branches:<br>      - prod</pre><p>Here I set some environment variables:</p><pre>env:<br>  PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}<br>  RUN_REGION: us-central1<br>  IMAGE_TAG_API: gcr.io/${{ secrets.GCP_PROJECT_ID }}/anomaly-detection-api:${{ github.sha }}<br>  IMAGE_TAG_SUB: gcr.io/${{ secrets.GCP_PROJECT_ID }}/anomaly-detection-sub:${{ github.sha }}</pre><p>Here we give this pipeline a name:</p><pre>jobs:<br>  setup-build-deploy:<br>    name: Setup, Build, and Deploy<br>    runs-on: ubuntu-latest</pre><p>If we are going to use cloud run, we must include Google Cloud configuration and authentication steps in the pipeline, to allow Github Actions to operate your account (to be more specific, a project on your Google Cloud):</p><pre>steps:<br>    - name: Checkout code<br>      uses: actions/checkout@v3<br><br>    - name: Authenticate with Google Cloud<br>      uses: google-github-actions/auth@v2<br>      with:<br>        credentials_json: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}<br><br>    - name: Set up gcloud<br>      uses: google-github-actions/setup-gcloud@v2<br>      with:<br>        project_id: ${{ env.PROJECT_ID }}<br>        install_components: &#39;beta&#39;<br><br>    - name: Verify gcloud configuration<br>      run: |<br>        gcloud config list<br>        gcloud info</pre><p>Here I like to include a debug step to see if the files are actually inside the containers, check if the environment variables are actually being set and then start the build:</p><pre>    - name: List files in the current directory<br>      run: |<br>        ls -la<br><br>    - name: Print Environment Variables<br>      run: |<br>        echo &quot;PROJECT_ID=${{ secrets.GCP_PROJECT_ID }}&quot;<br>        echo &quot;SERVICE_NAME=${{ secrets.API_SERVICE_NAME }}&quot;<br>        echo &quot;SERVICE_NAME=${{ secrets.SUB_SERVICE_NAME }}&quot;<br>        echo &quot;GITHUB_SHA=${GITHUB_SHA}&quot;<br><br>    - name: Submit build to Cloud Build<br>      run: |<br>        gcloud builds submit \<br>          --config cloudbuild.yaml \<br>          --substitutions=_IMAGE_TAG_API=$IMAGE_TAG_API,_IMAGE_TAG_SUB=$IMAGE_TAG_SUB</pre><p>I do a final check after building the containers and starting the deployments:</p><pre>    - name: List files in the directory (verify if .env exists - DEBUG)<br>      run: |<br>        ls -la<br><br>    - name: Deploy API to Cloud Run<br>      run: |-<br>        gcloud run deploy ${{ secrets.API_SERVICE_NAME }} \<br>          --image ${{ env.IMAGE_TAG_API }} \<br>          --platform managed \<br>          --region &quot;$RUN_REGION&quot; \<br>          --allow-unauthenticated \<br>          --timeout=300s \<br>          --set-env-vars &quot;GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }},GCP_PUBSUB_TOPIC=${{ secrets.GCP_PUBSUB_TOPIC }},GCP_PUBSUB_SUBSCRIPTION=${{ secrets.GCP_PUBSUB_SUBSCRIPTION }}&quot;<br><br>    - name: Deploy Subscriber to Cloud Run<br>      run: |-<br>        gcloud run deploy ${{ secrets.SUB_SERVICE_NAME }} \<br>          --image ${{ env.IMAGE_TAG_SUB }} \<br>          --platform managed \<br>          --region &quot;$RUN_REGION&quot; \<br>          --allow-unauthenticated \<br>          --timeout=300s \<br>          --set-env-vars &quot;GCP_PROJECT_ID=${{ secrets.GCP_PROJECT_ID }},GCP_PUBSUB_TOPIC=${{ secrets.GCP_PUBSUB_TOPIC }},GCP_PUBSUB_SUBSCRIPTION=${{ secrets.GCP_PUBSUB_SUBSCRIPTION }}&quot;</pre><p><strong><em>Tip:</em></strong><em> Take good care of your secrets. They are essential for security and organization in the pipeline. And yes, GitHub Actions handles them very well via secrets.GCP_PROJECT_ID, for example.</em></p><h3>Real Benefits of Automation</h3><p>After full automation, the project became much more robust and easier to maintain. Some immediate benefits:</p><ul><li><strong>Zero manual deployment</strong>: push to the branch and that’s it.</li><li><strong>Simple scalability</strong>: Cloud Run scales on demand.</li><li><strong>Consistency</strong>: clean builds ensure environment reproducibility.</li></ul><h3>Final Word</h3><p>This project was an excellent way to consolidate knowledge that is often overlooked by those who are focused only on models. Machine Learning in production is not just about accuracy: <strong>it is about reliability, automation, versioning, and maintenance.</strong></p><blockquote>My biggest lesson was: CI/CD is the bridge between the code that runs on your machine and the system that runs in the real world.</blockquote><p>If you are just starting out, don’t wait until you have “mastered everything” to start practicing this. Start simple, automate the basics, and evolve over time.</p><p>If you want to talk about ML automation, MLOps practices or have questions about how to implement something similar, I’m available to exchange ideas!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17e6a6e95e30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/automating-ml-models-deployment-with-github-actions-cloud-run-17e6a6e95e30">Automating ML Models Deployment with GitHub Actions + Cloud Run</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pub/Sub Messaging in Machine Learning Systems: more than communication, it’s architecture — Part 1]]></title>
            <link>https://medium.com/google-cloud/pub-sub-messaging-in-machine-learning-systems-more-than-communication-its-architecture-part-1-4e9d92688391?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e9d92688391</guid>
            <category><![CDATA[pub-sub]]></category>
            <category><![CDATA[gcp-app-dev]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Wed, 30 Apr 2025 11:21:59 GMT</pubDate>
            <atom:updated>2025-04-30T11:21:59.445Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pub/Sub Messaging in Machine Learning Systems: more than communication, it’s architecture — Part 1</h3><h4>How decoupling critical steps in your pipeline can reduce latency, increase resilience, and facilitate retraining automations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4a7sz2jyIOM7bCyHCz6uzQ.png" /></figure><p>When working with Machine Learning systems in production, the biggest headache is often not the models themselves, but how data travels between pipeline components. More than just an infrastructure detail, this decision directly impacts performance, scalability, and resilience.</p><p>In this post, I share one of the best choices I made in a personal project: adopting messaging with Google Cloud Pub/Sub to structure an event-driven architecture in my ML systems. This choice, in addition to being educational, showed me how decoupling steps can reduce latency and pave the way for more robust automations, such as retraining and even other parts of development.</p><h3>The Project</h3><p>The project is an anomaly detection system in which I developed:</p><ul><li>A supervised model and an unsupervised model;</li><li>A FastAPI API that serves both models via a unified endpoint;</li><li>Automated deployment on Cloud Run via GitHub Actions;</li><li>A client that consumes the API with a cronjob (30 requests/hour from 9am to 6pm);</li><li>A monitoring dashboard with data drift metrics (KS and JS), distributions and API performance;</li><li>And of course, the focus of this post, a messaging system with Google Cloud Pub/Sub.</li></ul><p><strong><em>You can see the complete source code in this repository -&gt; </em></strong><a href="https://github.com/ju4nv1e1r4/ml-cidds"><strong><em>https://github.com/ju4nv1e1r4/ml-cidds</em></strong></a></p><p><em>The project is under development.</em></p><h3>Why Pub/Sub?</h3><p>Imagine that each API inference needs to be saved to feed future steps such as monitoring, auditing, reprocessing and retraining.</p><p>The most direct approach would be:</p><p><strong>Model runs → saves to the database → responds to the user</strong></p><p>But this has a classic problem: <strong>coupling</strong>. The model gets “stuck” waiting for steps that are not part of the inference itself, which increases latency and weakens the system.</p><p><strong>So what to do?</strong></p><h3>Event-Driven Architecture with Pub/Sub</h3><p>In my project, I replaced the traditional flow with an event-driven architecture, with Pub/Sub.</p><h4>What is Pub/Sub?</h4><p>Pub/Sub is an <strong>asynchronous</strong> messaging system where:</p><ul><li><strong>Publishers</strong> send messages to a <strong>topic</strong>;</li><li><strong>Subscribers</strong> consume these messages from the <strong>topic</strong>.</li></ul><p>This model ensures that the components are <strong>decoupled</strong>, allowing them to evolve and scale independently.</p><h3>How did my flow look with Pub/Sub?</h3><p><strong>1. Publisher:</strong><br>My FastAPI serves the models and, after performing the inference, publishes a message to the <strong>Pub/Sub topic</strong> with:</p><ul><li>The input data;</li><li>The inference results;</li><li>Timestamp and useful metadata.</li></ul><p><strong>2. Topic:</strong><br>This is the intermediate channel. It stores the message <strong>temporarily until someone consumes it</strong>.</p><p><strong>3. Subscriber:</strong><br>A separate microservice (also deployed on Cloud Run), listens to the topic and processes the received data. This microservice:</p><ul><li>Persists the data in historical datasets;</li><li>Updates the monitoring data;</li><li>Prepares the data for future retraining.</li></ul><h3>Latency Reduction</h3><p>This decoupling brought a direct benefit: <strong>reduced latency</strong> in the API.</p><h4>Before:</h4><p>The API took ~11s to respond (model + data persistence).</p><h4>After:</h4><p>The API responds in ~6s.</p><p><strong>Persistence and post-processing</strong> are done <strong>asynchronously</strong>.</p><p>With this, the model “does its part” and moves on, while another part of the system takes care of the rest, without blocking the response to the user.</p><h3>Event-based retraining? Yes!</h3><p>The next step will be to set up an automated retraining system, triggered by data drift detection (already monitored in the dashboard).</p><h4>How will this work?</h4><ol><li>The monitoring detects that there was a relevant drift;</li><li>A new event is published in another Pub/Sub topic;</li><li>A specific subscriber starts the retraining pipeline with the saved data (when there is enough data for retraining).</li></ol><p>In other words, Pub/Sub will be an essential part of this ML system, from inference to retraining. <strong>But that will be for the second part of this post.</strong></p><h3>Benefits of this approach</h3><ul><li><strong>Complete decoupling</strong> between API, persistence, monitoring and retraining;</li><li><strong>Reduced latency</strong> in model inference;</li><li><strong>Ease of maintenance</strong> and system evolution;</li><li>Resilient and <strong>scalable architecture</strong>, with Cloud Run taking care of orchestration;</li><li>Automation ready for <strong>real MLOps</strong>, even in a personal project.</li></ul><p>Pub/Sub messaging isn’t just an architectural technique, it’s an enabler for modern machine learning systems. Instead of building a rigid, coupled framework, you can create dynamic, modular, and highly responsive workflows. If you’re working in ML and you’re not already using messaging in your system, it might be time to rethink.</p><h4>Did you like it?</h4><p>If this type of content interests you, follow me here. I usually write about MLOps, ML engineering and hands-on projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e9d92688391" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/pub-sub-messaging-in-machine-learning-systems-more-than-communication-its-architecture-part-1-4e9d92688391">Pub/Sub Messaging in Machine Learning Systems: more than communication, it’s architecture — Part 1</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Home Server: building my own server with an old laptop]]></title>
            <link>https://medium.com/@ju4nv1e1r4/my-home-server-building-my-own-server-with-an-old-laptop-b085624da008?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/b085624da008</guid>
            <category><![CDATA[experimental]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[home-server]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Mon, 28 Apr 2025 21:01:30 GMT</pubDate>
            <atom:updated>2025-04-28T21:07:30.035Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*EC5YfrRFuIvK1KeFl2Ugow.png" /><figcaption>Tired of this, huh?!</figcaption></figure><p>In December 2024, I started studying different ways of deploying, monitoring and infrastructure. I admit: all paths eventually lead to the cloud. But my uncle, an IT professional with over 20 years of experience in the hardware area, always told me stories about home servers, which always made me curious. That’s when I decided: I’m going to create my own server at home!</p><h3>The Gear</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9o2VDoJcfIV4oOzS-PjLLA.jpeg" /></figure><p>Here at home, I had an old laptop, a Compaq Presario (I don’t remember the exact model), with 4 GB of RAM and 100 GB of disk space. I was thinking about selling it, but I ended up turning it into my home server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GydwpuQTsu-uepGLBXjksw.jpeg" /></figure><p>The first step was to choose the operating system. After some research, the choice was obvious: Ubuntu Server.</p><h3>Getting Started</h3><p>The process started by downloading the latest stable version of Ubuntu Server from the official website: <a href="https://ubuntu.com/download/server">https://ubuntu.com/download/server</a>.</p><p>Then, I created a bootable USB drive using Rufus: <a href="https://rufus.ie/">https://rufus.ie/</a>.</p><p>(At this stage, it is important to have a backup of your data! In my case, the laptop was already empty of anything important.)</p><h3>Installing Ubuntu Server</h3><p>I won’t go into the details of the installation here, but if you need help, I recommend these videos:</p><ul><li><a href="https://www.youtube.com/watch?v=zs2zdVPwZ7E">Eng</a>lish</li><li>P<a href="https://www.youtube.com/watch?v=3XkHSi0qCkc">ortuguese (Brazil)</a></li></ul><p>In general, the installation is simple: choose the language, keyboard layout, set a fixed IP (or not), proxy (if necessary), disk partition and, in the “profile setup”, define a name and password. Then, in the resource selection, you can add snaps such as Docker, gcloud CLI, among others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*5gCpiQD6nAm86kFaQSHFSw.jpeg" /></figure><p><strong>Big Spoiler:</strong> for most of the process, you just have to press Enter — but read everything carefully!</p><h3>Deploying Something</h3><p>To test my server, I decided to make a simple deployment. I took an old Machine Learning model (which predicts house prices) and uploaded it without using any cloud services. The model was already in an API made with Flask, hosted on my GitHub. I cloned the project on the server and created a Procfile with the following content:</p><pre>web: gunicorn --bind 0.0.0.0:5001 --workers 3 app:app</pre><p>Then I installed Gunicorn:</p><pre>pip install gunicorn</pre><p>Here’s what happens:</p><ul><li>The Gunicorn server will listen on port 5001</li><li>Three processes will be created ( — workers 3)</li><li>It looks for the app object in app.py</li></ul><p>Before starting, you need to release the port in the firewall:</p><pre>sudo ufw allow 5001</pre><p>And then run it:</p><pre>gunicorn --bind 0.0.0.0:5001 --workers 3 app:app</pre><p>Done! Now, just access the machine’s IP on the local network, like this: <a href="http://192.168.x.x:5001/">http://192.168.x.x:5001</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*SbMesqJFNSvHXegr7DCwWA.jpeg" /></figure><p>(Unfortunately, my provider doesn’t allow me to expose my network publicly, otherwise I would have my own personal “cloud” haha.)</p><h3>Scheduling a Cronjob</h3><p>I am developing an anomaly detection model in parallel and decided to use a cronjob to consume my models. To make everything more automatic, I created a client that consumes my API, simulating a production environment.</p><p>(I talk more about this in my article <a href="https://medium.com/@ju4nv1e1r4/supervised-or-unsupervised-for-anomaly-detection-94543a2b0e28">Supervised or Unsupervised for Anomaly Detection</a>.)</p><p>Since this is an educational/experimental project, no one actually uses the model, so I created scripts that simulate real access, running at scheduled times with cron.</p><p>To schedule this, I edited the crontab:</p><pre>crontab -e</pre><p><strong>(In the editor, I chose option 1 — Nano.)</strong></p><p>I added these lines to the end of the file:</p><pre>0 8 * * * source /home/my-user/dev/ml-cidds/env/bin/activate &amp;&amp; /usr/bin/python3 /home/my-user/dev/ml-cidds/scripts/client.py &amp;<br>0 19 * * * pkill -f /home/my-user/dev/ml-cidds/scripts/client.py</pre><p>To explain:</p><ul><li>At 8am, the virtual environment is activated and client.py starts running, consuming the API.</li><li>At 7pm, the script is terminated with pkill.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B2q5h6bj4oSswtDp9ehq1Q.jpeg" /></figure><p>The client script is available in the repository: <a href="https://github.com/ju4nv1e1r4/ml-cidds">https://github.com/ju4nv1e1r4/ml-cidds</a>.</p><h3>Extra tip:</h3><p>To leave the home server on 24/7 and close the laptop, follow this :</p><p>Look for this line (it may be commented out with #):</p><pre>#HandleLidSwitch=suspend</pre><p>Now change it to:</p><pre>HandleLidSwitch=ignore</pre><p>You can also add these other two lines to reinforce the behavior:</p><pre>HandleLidSwitchDocked=ignore<br>HandleLidSwitchExternalPower=ignore</pre><p><strong>In nano</strong>, press:</p><ul><li><em>Ctrl + O (to save);</em></li><li><em>Enter (to confirm);</em></li><li><em>Ctrl + X (to exit).</em></li></ul><h4><strong>And it’s done!</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FjASm-hkUXLgZo4EqEcz-w.jpeg" /></figure><h3>Closing</h3><p>Today, I use my home server to deploy proof of concepts before moving to a cloud service. This helps me avoid the famous “it works on my machine” scenario, testing environments, automation, cronjobs, and monitoring locally.</p><p>If you don’t have a home server, you can use Docker to simulate production environments, but I’ll talk about that in another post!</p><p>Thanks for following along, and see you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b085624da008" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Supervised or Unsupervised for Anomaly Detection?]]></title>
            <link>https://medium.com/@ju4nv1e1r4/supervised-or-unsupervised-for-anomaly-detection-94543a2b0e28?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/94543a2b0e28</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 20:18:54 GMT</pubDate>
            <atom:updated>2025-04-25T20:20:58.918Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C30GMPS7e4ErHt3J9tC5vw.png" /></figure><p>My last project, I faced a question that most data scientists will eventually need to answer throughout: <em>Should I go with a supervised or an unsupervised approach for anomaly detection?</em></p><p>After talking to a few peers and reading some technical content, I came to a common conclusion: <strong><em>You test both</em></strong>.</p><h3>Dataset: CIDDS (Coburg Intrusion Detection Data Sets)</h3><p>The project was based on CIDDS-001, a network traffic dataset designed for intrusion detection research. It contains flow-based traffic records with labels for normal and anomalous activities (brute force, port scans, DoS).<br>CIDDS is a great dataset to experiment with both supervised and unsupervised anomaly detection since it has labeled traffic (but not perfectly balanced) and a good level of noise, just like you’d expect from real-world data.</p><h3>Supervised Model: Optimization and Training</h3><p>For the supervised approach, I worked with classic classifiers. The goal wasn’t just to get a decent model. It was to maximize generalization, especially for positive detections (the anomalies).</p><p>I built an Optimize class to streamline the optimization pipeline, starting with RandomizedSearchCV to explore broadly, followed by BayesSearchCV to fine-tune around the most promising regions.</p><pre>class Optimize:<br>    def __init__(<br>        self,<br>        model,<br>        param_grid,<br>        X_train,<br>        X_test,<br>        y_train,<br>        y_test,<br>        cv,<br>        verbose=1,<br>        n_iter=25,<br>        scoring=&quot;recall&quot;,<br>        thresholds=None,<br>    ):<br>        self.model = model<br>        self.param_grid = param_grid<br>        self.X_train = X_train<br>        self.y_train = y_train<br>        self.X_test = X_test<br>        self.y_test = y_test<br>        self.cv = cv<br>        self.verbose = verbose<br>        self.n_iter = n_iter<br>        self.scoring = scoring<br>        self.thresholds = thresholds or {<br>            &quot;recall&quot;: 0.8,<br>            &quot;precision&quot;: 0.8,<br>            &quot;f1_score&quot;: 0.8,<br>            &quot;accuracy&quot;: 0.8,<br>            &quot;roc_auc&quot;: 0.8,<br>        }<br><br>    def refine_param_grid(self, best_params, spread=0.2):<br>        new_grid = {}<br><br>        for param, best_val in best_params.items():<br>            if isinstance(best_val, (int, float)):<br>                min_val = best_val * (1 - spread)<br>                max_val = best_val * (1 + spread)<br><br>                if isinstance(best_val, int):<br>                    new_grid[param] = list(<br>                        range(max(int(min_val), 1), int(max_val) + 1)<br>                    )<br>                else:<br>                    new_grid[param] = (<br>                        min_val,<br>                        max_val,<br>                    )<br>            else:<br>                new_grid[param] = [best_val]<br><br>        return new_grid<br><br>    def with_random_search(self) -&gt; dict:<br>        rs = RandomizedSearchCV(<br>            self.model,<br>            self.param_grid,<br>            cv=self.cv,<br>            n_jobs=2,<br>            verbose=self.verbose,<br>            n_iter=self.n_iter,<br>            scoring=self.scoring,<br>        )<br><br>        rs.fit(self.X_train, self.y_train)<br><br>        best_model = rs.best_estimator_<br>        predictions = best_model.predict(self.X_test)<br>        proba = rs.predict_proba(self.X_test)[:, 1]<br><br>        accuracy = accuracy_score(self.y_test, predictions)<br>        recall = recall_score(self.y_test, predictions)<br>        cf = classification_report(self.y_test, predictions)<br>        rocauc_score = roc_auc_score(predictions, proba)<br>        precision = precision_score(self.y_test, predictions)<br>        cm = confusion_matrix(self.y_test, predictions)<br><br>        print(f&quot;Best params: {rs.best_params_}\n&quot;)<br>        print(&quot;\nClassification Report\n&quot;, cf)<br>        print(f&quot;Accuracy Score: {accuracy}\n&quot;)<br>        print(f&quot;Recall Score: {recall}\n&quot;)<br>        print(f&quot;Precision Score: {precision}\n&quot;)<br>        print(f&quot;\nROC AUC Score: {rocauc_score}\n&quot;)<br>        print(f&quot;Confusion Matrix: {cm}&quot;)<br><br>        best_params = rs.best_params_<br>        self.param_grid = self.refine_param_grid(best_params)<br><br>        return best_params<br><br>    def with_bayesian_search(self) -&gt; dict:<br>        bs = BayesSearchCV(<br>            self.model,<br>            self.param_grid,<br>            cv=5,<br>            n_jobs=2,<br>            verbose=self.verbose,<br>            n_iter=self.n_iter,<br>            scoring=self.scoring,<br>        )<br><br>        bs.fit(self.X_train, self.y_train)<br><br>        best_model = bs.best_estimator_<br>        predictions = best_model.predict(self.X_test)<br>        proba = bs.predict_proba(self.X_test)[:, 1]<br><br>        accuracy = accuracy_score(self.y_test, predictions)<br>        recall = recall_score(self.y_test, predictions)<br>        cf = classification_report(self.y_test, predictions)<br>        f1 = f1_score(self.y_test, predictions)<br>        rocauc_score = roc_auc_score(predictions, proba)<br>        precision = precision_score(predictions, self.y_test)<br>        cm = confusion_matrix(predictions, self.y_test)<br><br>        print(f&quot;Best params: {bs.best_params_}\n&quot;)<br>        print(&quot;\nClassification Report\n&quot;, cf)<br>        print(f&quot;Accuracy Score: {accuracy}\n&quot;)<br>        print(f&quot;Recall Score: {recall}\n&quot;)<br>        print(f&quot;Precision Score: {precision}\n&quot;)<br>        print(f&quot;F1 Score: {f1}\n&quot;)<br>        print(f&quot;\nROC AUC Score: {rocauc_score}\n&quot;)<br>        print(f&quot;Confusion Matrix: {cm}&quot;)<br><br>        if (<br>            recall &gt;= self.thresholds.get(&quot;recall&quot;, 0)<br>            and precision &gt;= self.thresholds.get(&quot;precision&quot;, 0)<br>            and accuracy &gt;= self.thresholds.get(&quot;accuracy&quot;, 0)<br>            and f1 &gt;= self.thresholds.get(&quot;f1_score&quot;, 0)<br>            and rocauc_score &gt;= self.thresholds.get(&quot;roc_auc&quot;, 0)<br>        ):<br>            return bs.best_params_<br>        else:<br>            print(<br>                &quot;&quot;&quot;<br>                Metrics are not good enough.<br>                Check your steps before this optimization.<br>                &quot;&quot;&quot;<br>            )<br>            return {}</pre><p>And here’s the optimization and training that followed:</p><pre>import logging<br><br>from colorama import Fore, init<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import (<br>    accuracy_score,<br>    classification_report,<br>    confusion_matrix,<br>    f1_score,<br>    precision_score,<br>    recall_score,<br>    roc_auc_score,<br>)<br>from sklearn.model_selection import StratifiedKFold<br><br>from src.ml.optmize import Optimize<br>from src.models.load import Load<br>from utils.constants import Config<br>from utils.gcp import CloudStorageOps<br><br>init(autoreset=True)<br>logging.basicConfig(<br>    level=logging.INFO, format=&quot;%(asctime)s - %(levelname)s - %(message)s&quot;<br>)<br>c = Config()<br>logger = c.logger<br>gcs = CloudStorageOps(&quot;ml-anomaly-detection&quot;)<br><br>PATH_FILE = &quot;preprocessed/CIDDS-001.csv&quot;<br><br>data = Load(file_path=PATH_FILE)<br><br>df = data.load_dataset()<br>X_train, X_test, y_train, y_test = data.split_dataset(df)<br><br>skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)<br><br>param_grid = {<br>    &quot;n_estimators&quot;: [50, 100, 200],<br>    &quot;max_depth&quot;: [5, 10, 20, None],<br>    &quot;min_samples_split&quot;: [2, 5, 10],<br>}<br><br>my_thresholds = {<br>    &quot;recall&quot;: 0.90,<br>    &quot;precision&quot;: 0.60,<br>    &quot;roc_auc&quot;: 0.40,<br>    &quot;f1_score&quot;: 0.75,<br>    &quot;accuracy&quot;: 0.5,<br>}<br><br>opt = Optimize(<br>    RandomForestClassifier(),<br>    param_grid,<br>    X_train,<br>    X_test,<br>    y_train,<br>    y_test,<br>    cv=skf,<br>    verbose=0,<br>    n_iter=1,<br>    scoring=&quot;recall&quot;,<br>    thresholds=my_thresholds,<br>)<br><br>logging.info(&quot;===== Randomized Search =====&quot;)<br>best_params_random = opt.with_random_search()<br><br>logging.info(&quot;\n===== Bayesian Search =====&quot;)<br>best_params_bayes = opt.with_bayesian_search()<br><br>print(&quot;---\n&quot;)<br>logging.info(Fore.YELLOW + &quot;Starting training with RandomForestClassifier...&quot;)<br><br>try:<br>    model = RandomForestClassifier(<br>        n_estimators=210,<br>        min_samples_split=10,<br>        min_samples_leaf=3,<br>        max_depth=20,<br>        max_features=&quot;sqrt&quot;,<br>        criterion=&quot;gini&quot;,<br>        bootstrap=False,<br>    )<br>    model.fit(X_train, y_train)<br>    logging.info(Fore.GREEN + &quot;Model training is complete.&quot;)<br>except Exception as fit_error:<br>    logging.error(Fore.RED + f&quot;Error: {fit_error}&quot;)<br><br>try:<br>    y_pred = model.predict(X_test)<br>    y_proba = model.predict_proba(X_test)[:, 1]<br><br>    acc = accuracy_score(y_test, y_pred)<br>    precision = precision_score(y_test, y_pred)<br>    recall = recall_score(y_test, y_pred)<br>    f1 = f1_score(y_test, y_pred)<br>    rocauc = roc_auc_score(y_test, y_proba)<br>    cm = confusion_matrix(y_test, y_pred)<br>    cr = classification_report(y_test, y_pred)<br>    logging.info(Fore.GREEN + &quot;Predictions is done. Check this metrics:\n&quot;)<br>except Exception as pred_and_eval_error:<br>    logging.error(Fore.RED + f&quot;Error: {pred_and_eval_error}&quot;)<br><br>print(&quot;1. Classification Report\n&quot;, cr)<br>print(&quot;\n2. Confusion Matrix\n&quot;, cm)<br>print(&quot;\n3. ROC AUC Score: &quot;, rocauc)<br>print(&quot;\n4. Accuracy Score: &quot;, acc)<br>print(&quot;\n5. Precision Score: &quot;, precision)<br>print(&quot;\n6. Recall Score: &quot;, recall)<br>print(&quot;\n7. F1 Score: &quot;, f1)</pre><p>This two-stage optimization helped improve recall and F1-score, the most relevant metrics in this context.</p><p>Obs: To avoid overfitting masked by high accuracy, I defined <strong>minimum metric thresholds. </strong>Any model below these would be discarded regardless of how good it looked on the surface.</p><h3>Unsupervised Model: Isolation Forest</h3><p>When labels can’t be trusted or simply don’t exist, unsupervised methods step in. I chose Isolation Forest due to its simplicity and speed. And even with far less control over the modeling process, it delivered a strong baseline.<br>Before implementing, I reviewed a few great resources to better understand the algorithm:</p><ol><li><a href="https://medium.com/datarisk-io/detec%C3%A7%C3%A3o-de-anomalias-usando-o-isolation-forest-5d29728fb255">This excellent article by Mariana Ramos Queiroz</a></li><li><a href="https://medium.com/@limyenwee_19946/unsupervised-outlier-detection-with-isolation-forest-eab398c593b2">A deeper dive into how Isolation Forest works</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html">The official documentation</a></li></ol><pre>import logging<br><br>import pandas as pd<br>from colorama import Fore, init<br>from google.resumable_media.common import InvalidResponse<br>from sklearn.ensemble import IsolationForest<br>from sklearn.metrics import (<br>    classification_report,<br>    f1_score,<br>    precision_score,<br>    recall_score,<br>)<br>from sklearn.model_selection import train_test_split<br><br>from utils.gcp import CloudStorageOps<br><br>init(autoreset=True)<br>logging.basicConfig(<br>    level=logging.INFO, format=&quot;%(asctime)s - %(levelname)s - %(message)s&quot;<br>)<br><br><br>PATH_FILE = &quot;preprocessed/CIDDS-001.csv&quot;<br>gcs = CloudStorageOps(&quot;ml-anomaly-detection&quot;)<br><br>try:<br>    load = gcs.load_from_bucket(PATH_FILE)<br>    logging.info(Fore.GREEN + f&quot;File loaded successfully: {PATH_FILE}.&quot;)<br>    print(&quot; &quot;)<br>except InvalidResponse as load_error:<br>    logging.error(Fore.RED + f&quot;Error: {load_error}&quot;)<br>    print(&quot; &quot;)<br><br>try:<br>    if load is not None:<br>        df = pd.read_csv(io.BytesIO(load))<br>    else:<br>        logging.error(&quot;Load returned None, cannot read CSV.&quot;)<br>        print(&quot; &quot;)<br>    logging.info(Fore.BLUE + f&quot;File loaded: {PATH_FILE}.&quot;)<br>    print(&quot; &quot;)<br>except FileNotFoundError as file_error:<br>    logging.error(Fore.RED + f&quot;Error: {file_error}&quot;)<br>    print(&quot; &quot;)<br><br>try:<br>    X = df.drop(columns=[&quot;Unnamed: 0&quot;, &quot;class&quot;, &quot;is_attack&quot;])<br>    y = df[&quot;is_attack&quot;]<br>    logging.info(Fore.BLUE + &quot;Features and Target splitted.&quot;)<br>    print(&quot; &quot;)<br>except Exception as cols_error:<br>    logging.error(Fore.RED + f&quot;Error: {cols_error}&quot;)<br>    print(&quot; &quot;)<br><br>try:<br>    X_train, X_test, y_train, y_test = train_test_split(<br>        X,<br>        y,<br>        test_size=0.3,<br>        random_state=42,<br>        stratify=y,<br>    )<br>    logging.info(<br>        Fore.BLUE<br>        + f&quot;Data successfully splitted -&gt; Train: {X_train.shape}; Test: {X_test.shape}&quot;<br>    )<br>    print(&quot; &quot;)<br>except Exception as split_error:<br>    logging.error(Fore.RED + f&quot;Error: {split_error}&quot;)<br>    print(&quot; &quot;)<br><br><br>print(&quot;---\n&quot;)<br>print(Fore.YELLOW + &quot;Starting experiments with IsolationForest...&quot;)<br>print(&quot; &quot;)<br><br>model = IsolationForest(<br>    n_estimators=100,<br>    max_samples=&quot;auto&quot;,<br>    contamination=0.5,<br>    max_features=1.0,<br>    random_state=42,<br>)<br><br>model.fit(X_train)<br><br>y_pred_raw = model.predict(X_test)<br>y_pred = [1 if val == -1 else 0 for val in y_pred_raw]<br><br>print(&quot;\nClassification Report\n&quot;, classification_report(y_test, y_pred))<br>print(f&quot;Recall: {recall_score(y_test, y_pred):.4f}&quot;)<br>print(f&quot;Precision: {precision_score(y_test, y_pred):.4f}&quot;)<br>print(f&quot;F1-score: {f1_score(y_test, y_pred):.4f}&quot;)</pre><p>Using only a few hyperparameter adjustments, the model showed promising recall values and a surprisingly competitive F1-score.</p><h3>Results</h3><p>Despite the distinct natures of the models, both delivered valuable results.</p><p>The <strong>supervised model</strong>, after a full optimization cycle, produced <strong>high recall and strong precision</strong>, achieving a <strong>robust F1-score</strong>. This came at the cost of more complexity: label curation, iterative validation, and tighter control of thresholds.</p><p>The <strong>unsupervised IsolationForest</strong>, on the other hand, provided <strong>good recall</strong> and a <strong>solid baseline</strong> almost out-of-the-box. Its biggest advantage: <strong>speed and simplicity</strong>, especially in environments where labels are unreliable or unavailable.</p><p><em>You can see more in: </em><a href="https://github.com/ju4nv1e1r4/ml-cidds/blob/main/README.md"><em>https://github.com/ju4nv1e1r4/ml-cidds/blob/main/README.md</em></a></p><h3>Conclusion: Why Not Both?</h3><p>Each model type has its tradeoffs.</p><ul><li>Supervised learning offers <strong>fine control and strong metrics </strong>(when labels are reliable).</li><li>Unsupervised learning brings <strong>flexibility and speed</strong>, especially in messy or incomplete datasets.</li></ul><p>In the end, I decided to deploy both models to production. My API serves predictions from both and continuously logs results for ongoing evaluation.</p><p>This gives me the opportunity to observe real-world performance, gather usage data, and decide, over time, which model is better suited to the operational context.</p><p>Because in ML, no system is production-ready overnight.<br>You have to test, observe, and iterate. Always.</p><p><em>If you liked this article, </em><strong><em>follow me here on Medium</em></strong><em> and </em><strong><em>sign up to receive my content via e-mail.</em></strong></p><p><em>Link to the project repository: </em><a href="https://github.com/ju4nv1e1r4/ml-cidds"><em>https://github.com/ju4nv1e1r4/ml-cidds</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94543a2b0e28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluate and Deploy an LLM Agent with ADK on Google Cloud]]></title>
            <link>https://medium.com/google-cloud/evaluate-and-deploy-an-llm-agent-with-adk-on-google-cloud-c61ae95ab129?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/c61ae95ab129</guid>
            <category><![CDATA[agent-development-kit]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[adk]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 08:07:47 GMT</pubDate>
            <atom:updated>2025-04-12T08:07:47.893Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zw-OaeJsyZ1g8D8ys9eBQ.png" /></figure><p>In my previous post, I showed how I built a development agent using Gemini and the Agent Development Kit (ADK), a powerful tool that makes it easy to create autonomous and collaborative agents. This agent, the Development Tutor, is able to interact naturally with developers, answering technical questions in a light, humorous and didactic tone.</p><p>But building the agent was just the beginning of the journey.</p><p>In this new episode, I share the next step of the project: evaluating the LLM’s performance in the context of the agent and the complete deployment in the cloud. I’ll show you how to evaluate the responses and how I set everything up to run reliably on Google Cloud Run.</p><p>If you like the idea of ​​creating custom agents with LLMs and want to see how to take them to production, this is the reading for you.</p><h3>Project Structure</h3><p>First of all, I organized the project by creating two new folders:</p><pre>├── deployment         # deploy scripts<br>├── development_tutor  <br>├── eval               # evaluation directory<br>├── img                # Images<br>├── README.md<br>└── requirements.txt<br><br><br># The eval folder has both the evaluation script and the test data.</pre><p>I created a small JSON file with expected interactions:</p><pre>[<br>    {<br>        &quot;query&quot;: &quot;Olá!&quot;,<br>        &quot;expected_tool_use&quot;: [],<br>        &quot;reference&quot;: &quot;Olá, eu me chamo R2-D2... Pi, pi, pow! Eu não tenho um nome kkkj, mas pode me chamar de Dev. Como posso te ajudar?&quot;<br>    },<br>    {<br>        &quot;query&quot;: &quot;Como anda a comunidade de machine learning em Go?&quot;,<br>        &quot;expected_tool_use&quot;: [],<br>        &quot;reference&quot;: &quot;E aí! Suave? Então, a parada é a seguinte: a comunidade de Machine Learning em Go ainda tá crescendo...&quot;<br>    }<br>]</pre><p>This file serves as a comparison base to verify that the agent responds in a manner close to what we expect.</p><p>Then, created eval script:</p><pre>import os<br>import pathlib<br>import pytest<br>from dotenv import load_dotenv<br>from google.adk.evaluation.agent_evaluator import AgentEvaluator<br><br>@pytest.fixture(scope=&quot;session&quot;, autouse=True)<br>def load_env():<br>    load_dotenv()<br><br>def test_agent():<br>    AgentEvaluator.evaluate(<br>        agent_module=&quot;development_tutor&quot;,<br>        eval_dataset_file_path_or_dir=str(<br>            pathlib.Path(__file__).parent / &quot;eval_data/chat.test.json&quot;,<br>        ),<br>        num_runs=2<br>    )</pre><p>This script runs the agent twice for each question in the dataset and evaluates whether the answers are as expected. I used pytest to make it easier to run like a traditional test.</p><h4>Evaluation in Dev UI</h4><p>Run Web Dev UI:</p><pre>adk web</pre><ol><li>Start a new conversation with the agent.</li><li>Go to the Eval tab.</li><li>Click on the “+” symbol to create a new evaluation.</li><li>Give it a name and confirm.</li><li>Select the current conversation and click on “Add current session to eval”.</li><li>Finally, click on “Run Evaluation”.</li></ol><p>The result will be a simple Passed or Failed, with insights into the similarity between the expected response and the one generated by the agent.</p><p>Check this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9Ox3ES2EE57OXkfvnj10w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6dNNtfnzQ_t8K1JhjZIgbw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsgK3gi12vXA5CeEJTTggg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWZuIfnQlbOl0lP_SjwICA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MD8_t1BcgkUqrjjsnYLsjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAlDColNWtNLmPFdgEeizw.png" /></figure><h3>Deploy to Google Cloud</h3><p>After evaluating the agent and ensuring that it was responding coherently, the next step was to put it live. And here’s the good news: deploying with the ADK is so simple that you can do it with a bash script in just a few lines.</p><h4>I chose Google Cloud (and life became easy)</h4><p>I chose to use Google Cloud Run to host my agent, mainly because it is serverless, scalable and already integrated with the ADK CLI. No need to worry about infrastructure. The idea here was to launch the agent with minimal friction.</p><p>I created a script called deploy.sh inside the deployment/ folder:</p><pre>#!/bin/bash<br><br>set -x<br><br>source development_tutor/.env<br><br>name_vars(){<br>    export AGENT_PATH=&quot;./development_tutor&quot;<br>    export SERVICE_NAME=&quot;dev-tutor&quot;<br>    export APP_NAME=&quot;dev-tutor-app&quot;<br>}<br><br>deploy_service(){<br>    adk deploy cloud_run \<br>    --project=$GOOGLE_CLOUD_PROJECT \<br>    --region=$GOOGLE_CLOUD_LOCATION \<br>    --service_name=$SERVICE_NAME \<br>    --app_name=$APP_NAME \<br>    --with_ui \<br>    $AGENT_PATH<br>}<br><br>main(){<br>    name_vars<br>    deploy_service<br>}<br><br>main<br><br>exit 0</pre><p>The script basically does three things:</p><ol><li>It loads environment variables with source development_tutor/.env, such as GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION.</li><li>It sets the service and app names, as well as the agent path.</li><li>It runs the deploy with the ADK command, pointing to Google Cloud Run and including the web interface with — with_ui.</li></ol><pre>bash deployment/deploy.sh</pre><p>The agent is packaged and published directly to Cloud Run. The CLI takes care of everything: build, upload, deploy, and even link to the agent’s web interface.</p><p>Did you like this post? Recommend it to your friends, coworkers and send me your feedback!</p><p>Repository: <a href="https://github.com/ju4nv1e1r4/agents-with-adk">https://github.com/ju4nv1e1r4/agents-with-adk</a><br>ADK Docs: <a href="https://google.github.io/adk-docs/">https://google.github.io/adk-docs/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c61ae95ab129" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/evaluate-and-deploy-an-llm-agent-with-adk-on-google-cloud-c61ae95ab129">Evaluate and Deploy an LLM Agent with ADK on Google Cloud</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Agent: Getting Started with the Google Agent Development Kit (ADK)]]></title>
            <link>https://medium.com/google-cloud/from-zero-to-agent-getting-started-with-the-google-agent-development-kit-adk-08aefbc82f1e?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/08aefbc82f1e</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[agent-development-kit]]></category>
            <category><![CDATA[gcp-app-dev]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[adk]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 00:34:11 GMT</pubDate>
            <atom:updated>2025-04-12T00:34:11.173Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-EyccfIyyvstrjlh3qSjuw.png" /></figure><p>Google has released yet another new tool. With the launch of the <strong>Agent Development Kit (ADK)</strong>, developers now have a new way to create intelligent agents — not just those that chat, but also those that actually perform tasks, make decisions and interact with systems autonomously.</p><p>ADK is a complete platform for those who want to go beyond basic bots. It allows you to create conversational and non-conversational agents, with the right to orchestrate tasks, execute complex flows, and all this with the support of a well-thought-out, scalable, and integrated structure.</p><p>In this post, I’ll show you the basics of ADK and how to start playing with this powerful new feature. There will be code, practical examples and that step-by-step guide we like. Let’s code together!</p><h3>Installation</h3><p>I recommend using a virtual environment like venv:</p><p>In a conventional way:</p><pre># to create venv<br>python3 -m venv env<br><br><br># to activate your venv<br>source env/bin/activate</pre><p>I’ve been using the uv package manager a lot lately. This dependency manager was written in rust and optimizes the installation time of libs (I will talk about it another time):</p><pre># same thing but with &#39;uv&#39;<br><br>uv venv env<br>source env/bin/activate</pre><p>Then you install with:</p><pre>pip install google-adk<br><br># or<br><br>uv pip install google-adk</pre><h3>About my project</h3><p>To start exploring the power of ADK, I created a simple project with two agents that work together. One of them is the “conversation agent” — it receives the user’s questions and manages the flow of the conversation. But before answering anything, it asks for help from the second agent: the “researcher”, which activates a Google search tool and brings fresh information to compose the final answer.</p><p>This exchange between agents is the essence of the idea: they collaborate to deliver a more useful response, always based on updated information. It is a basic example, but it already shows the potential of ADK to orchestrate more complex tasks between different specialized agents.</p><p>The project structure follows the format suggested by ADK itself, which organizes everything into well-defined directories:</p><pre>.<br>├── development_tutor/<br>│   ├── agent.py              # main agent<br>│   ├── prompt.py             # Prompt base for main agent<br>│   ├── shared_libraries/     # Constants<br>│   ├── sub_agents/           # SubAgents<br>│   │   └── researcher/       # Researcher Agent<br>│   │       ├── agent.py<br>│   │       ├── prompt.py<br>│   └── tools/                # Tools<br>│       └── search.py         # Google Search Tool<br>└── README.md</pre><p>This modular format is one of the coolest things about the ADK — it encourages a clean architecture from the start, with room to scale and add more tools or agents as the project evolves.</p><h3>Creating an Agent with the ADK</h3><p>Creating an agent in ADK is surprisingly straightforward. You define the model, give it a name, write a prompt with clear instructions, and connect the tools it will use. That’s exactly what I did when building the researcher, the agent responsible for searching for information on the web.</p><p>It is a simple but functional agent: it receives a request, uses a Google search tool and prepares a summary with the information found. It then delivers this content to the main agent, which will decide how to use it in the final response.</p><p>Here is the code that defines the researcher agent:</p><pre>from google.adk.agents import Agent<br>from . import prompt<br>from ...shared_libraries import constants<br>from ...tools.search import google_search_grounding<br><br>researcher = Agent(<br>    model=constants.BASE_MODEL,<br>    name=&quot;researcher&quot;,<br>    description=&quot;Search a topic on google.&quot;,<br>    instruction=prompt.RESEARCHER_PROMPT,<br>    tools=[google_search_grounding]<br>)</pre><p>The prompt that guides the agent:</p><pre>RESEARCHER_PROMPT=&quot;&quot;&quot;<br>You are an agent with the autonomy to do research on the web.<br>Search the requested topics on the web.<br><br>Follow the steps:<br>1. Call the `google_search_grounding` tool and search on the topic<br>2. Gather the information in a draft<br>3. Summarize the draft and transfer it to `development_tutor`<br>&quot;&quot;&quot;</pre><p>This prompt is where the magic begins. It guides the agent’s behavior, telling it how to use the search tool, what to do with the results, and how to prepare the response.</p><p>The ADK works really well with structured instructions like this — it interprets the steps as a roadmap for the agent to follow. This makes everything more controlled and predictable, which is great especially when you’re first testing.</p><h3>Connecting Tools</h3><p>Another cool thing about ADK is the concept of tools — basically, any external action that an agent can call during its execution. And the coolest thing? These tools can be functions, APIs… or even other agents!</p><p>In my case, I created a tool called google_search_grounding, which is nothing more than an encapsulated agent, specialized in searching for answers directly on Google. This agent works behind the scenes, responding directly and concisely, based on search results.</p><pre>from google.adk.agents import Agent<br>from google.adk.tools.agent_tool import AgentTool<br>from google.adk.tools.google_search_tool import google_search<br>from ..shared_libraries import constants<br><br>_researcher = Agent(<br>    model=constants.BASE_MODEL,<br>    name=&quot;researcher&quot;,<br>    description=&quot;Pesquisa na web sobre um assunto.&quot;,<br>    instruction=&quot;&quot;&quot;<br>    Answer the user&#39;s question directly using Google&#39;s search engine; provide a brief but concise answer.<br>    Instead of a detailed answer, provide the developer with an immediate action item in a single sentence.<br>    Don&#39;t ask the user to verify or look up information on their own; that&#39;s your job; do your best to be informative.<br>    &quot;&quot;&quot;,<br>    tools=[google_search]<br>)<br><br>google_search_grounding = AgentTool(agent=_researcher)</pre><p>Here, _researcher is a mini-agent with a single goal: to use the google_search tool (already provided by the ADK) and return a simple and useful answer. This mini-agent is then packaged with AgentTool, turning it into a tool usable by other agents — like our main researcher.</p><h4>Why is this useful?</h4><p>This approach has <strong>several advantages</strong>:</p><ol><li><strong>Modularity</strong>: you can reuse this agent/tool ​​in other contexts.</li><li><strong>Intelligent</strong> <strong>delegation</strong>: the researcher does not need to know exactly how the search is done — he only knows that he can count on this tool to obtain the information.</li><li><strong>Separation of responsibilities</strong>: each agent does what it knows best, and you can adjust the behavior of each part independently.</li></ol><h3>The Main Agent</h3><p>With the sub-agents and tools ready, the next step was to create my main agent: the development_tutor. This is the brain of the operation — it is the one that talks directly to the user, guides the reasoning and decides when to call a sub-agent to give that help.</p><pre>from google.adk.agents.llm_agent import Agent<br><br>from .shared_libraries import constants<br>from . import prompt<br>from .sub_agents.researcher.agent import researcher <br><br>root_agent = Agent(<br>    model=constants.TUTOR_MODEL,<br>    name=constants.AGENT_NAME,<br>    description=constants.DESCRIPTION,<br>    instruction=prompt.ROOT_PROMPT,<br>    sub_agents=[<br>        researcher<br>    ]<br>)</pre><p>In practice, he is the instructor of the show. And like any good instructor, he <strong>follows a script</strong>. In my case, I created a prompt with some very specific sections, which shape the agent’s behavior in each phase of the conversation. Take a look:</p><pre>ROOT_PROMPT=&quot;&quot;&quot;<br>You are essentially an instructor. Your focus is on programming languages ​​and development.<br>You will help developers develop their solutions and answer questions, mainly about API documentation, frameworks, packages and the like.<br>Your teaching method consists of an iterative teaching process, where you will help with step-by-step instructions and cooperation between developer and instructor.<br><br>Follow the steps below:<br>1. Please follow the &lt;Greetings&gt; section to greet the user and gather information and continue in the flow.<br>2. Go to the &lt;Search&gt; section to ensure that your answers are up to date, don&#39;t stop. Keep in the flow.<br>3. Go to the &lt;Tone&gt; section and continue in the flow.<br>4. Please follow the &lt;Key Constraints&gt; when trying to answer the user&#39;s query.<br>3. At the end, always ask if the user is satisfied with the answers.<br><br>&lt;Greetings&gt;<br>1. Introduce yourself to the user in a joking manner, saying that your name is that of a character from geek culture, randomly changing the character&#39;s name every time.<br>&lt;Example&gt;<br>Hello, my name is [character-name]... Bazinga! I don&#39;t have a name lol, but you can call me Dev. How can I help you?<br>&lt;/Example&gt;<br>2. Ask how you can help the user. &lt;/Greetings&gt;<br><br>&lt;Search&gt;<br>1. Call `researcher` to help you research the latest information on the topic<br>2. Rephrase your answer first and continue with the flow<br>&lt;/Search&gt;<br><br>&lt;Tone&gt;<br>1. Adjust your tone of voice to a more technical tone<br>2. Use examples to answer the questions<br>3. Be friendly and jovial (you can use slang if you prefer).<br>&lt;\Tone&gt;<br><br>&lt;Key Constraints&gt;<br>- Your task is to provide a problem-solving answer.<br>- Complete all steps<br>- Answer in Brazilian Portuguese<br>&lt;/Key Constraints&gt;<br>&quot;&quot;&quot;</pre><h4>How development_tutor thinks:</h4><p><strong>&lt;Greetings&gt;</strong>: He always starts the conversation with good humor — using names of random geek characters and breaking the ice. It’s a fun presentation, but it starts with a direct question: “How can I help you today?”</p><p><strong>&lt;Search&gt;</strong>: Before answering anything, he calls the researcher sub-agent to search for updated information on Google. This avoids guesswork and ensures more reliable and fresh answers.</p><p><strong>&lt;Tone</strong>&gt;: Here he adjusts the style. The language is technical, but accessible — with examples, didactic explanations and even some slang, if it fits the context. It’s like that nice instructor who knows the ropes, but doesn’t force the issue.</p><p><strong>&lt;Key Constraints&gt;</strong>: Finally, some basic rules that ensure that everything is done in Portuguese, with a focus on solving problems clearly and completely.</p><p>This kind of structure in ADK is what makes the creation process so powerful. We can define exactly what an agent should do, how it should act, and who it can count on along the way.</p><p><strong>Do not forget</strong>: .env in the right place</p><p>If you’re going to use the ADK with Google’s Gemini, either via Google AI Studio or Vertex AI, you’ll need a .env file with your settings. But be careful: this file doesn’t go in the root of the project. It should be inside your agent folder — right next to agent.py.</p><p>This file defines how the ADK will connect to the Google model, and can be configured in two ways, depending on the platform you’re using:</p><pre># If using Gemini via Google AI Studio<br>GOOGLE_GENAI_USE_VERTEXAI=&quot;False&quot;<br>GOOGLE_API_KEY=&quot;paste-your-actual-key-here&quot;<br><br># # If using Gemini via Vertex AI on Google CLoud<br>GOOGLE_CLOUD_PROJECT=&quot;your-project-id&quot;<br>GOOGLE_CLOUD_LOCATION=&quot;your-location&quot; #e.g. us-central1<br>GOOGLE_GENAI_USE_VERTEXAI=&quot;True&quot;</pre><p><strong><em>Quick tip:</em></strong><em> Don’t forget to install python-dotenv if you want to load these variables into your Python environment. But if you’re running the standard ADK, it already takes care of this behind the scenes — just make sure you keep the file in the right place.</em></p><h3>For God’s Sake… do not forget .gitignore!</h3><p>If you’ve made it this far, you’ve probably already figured out that .env stores sensitive information. So don’t even think about versioning this file, okay?</p><p>Use a decent .gitignore. And by “decent”, I mean one that ignores the basics of Python: __pycache__, .env, .pyc files, virtualenv folders and other junk that doesn’t need to go into the repository.</p><p>To make your life easier, use this template from GitHub that’s ready to use: <a href="https://github.com/github/gitignore/blob/main/Python.gitignore">https://github.com/github/gitignore/blob/main/Python.gitignore</a></p><p>Just copy the content to a .gitignore file in the root of your project and you’re done.</p><h3>Running the Agent</h3><p>Once you’ve set everything up, it’s showtime! 😎</p><p>To run the agent directly in the terminal, just run:</p><pre>adk run development_tutor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7pt5Gs9vXht5VZYelb7Rw.png" /></figure><p>Oh my gosh, this is so beautiful to see!</p><p>We ran the development_tutor agent, and it arrived with that charismatic look, introducing itself as “Dev” (or R2D2 if you’re in the mood for nerdy humor 😂), and gave a super didactic answer to the ML engineer who is just starting out with Rust.</p><p>Here you can start to see the magic happen — literally a conversation between the agents happening right in front of you. The development_tutor exchanges ideas with the researcher, who Googles the subject and brings everything together.</p><h4>Want more? There’s an automatic UI too!</h4><p>If you like interfaces, the ADK also gives you a ready-made web UI built with FastAPI under the hood — smooth and modern. Just run it:</p><pre>adk web</pre><p>And there you have it: interface running at <a href="http://localhost:8000/">http://localhost:8000</a>, with everything organized so you can test your agents in a more visual way. You can send questions, follow the answers and see the flow happening in real time.</p><p>See for yourself:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ueNe8BktEiq-hGVjAtNEQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*66FdXx_geK8TMz8awxGJ-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFfxdcNcVtl4zRhxyVEm1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-DVa8BTpv3kj2qUzMWykA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWLi1DRcKpTT1oYW-fqNHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtPdw6_Cl9P0mWFgTF6XZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YrTHxKy7MC-XLJPcZKG93w.png" /></figure><h3>The End</h3><p>Even though it was a simple project, I like to see it in production. My next step will be to deploy it on Google Cloud. Follow me on Medium, sign up to receive my content by email and stay up to date! Here is the docs: <a href="https://google.github.io/adk-docs/">https://google.github.io/adk-docs/</a> and my repo: <a href="https://github.com/ju4nv1e1r4/agents-with-adk">https://github.com/ju4nv1e1r4/agents-with-adk</a>. See you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08aefbc82f1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/from-zero-to-agent-getting-started-with-the-google-agent-development-kit-adk-08aefbc82f1e">From Zero to Agent: Getting Started with the Google Agent Development Kit (ADK)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Qualidade de dados no TensorFlow com tf.data: o que todo modelo precisa]]></title>
            <link>https://medium.com/@ju4nv1e1r4/qualidade-de-dados-no-tensorflow-com-tf-data-o-que-todo-modelo-precisa-5d7c7d513f0e?source=rss-4c9aed6dca41------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d7c7d513f0e</guid>
            <category><![CDATA[machine-learning-engineer]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[tf-data]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Juan Vieira]]></dc:creator>
            <pubDate>Wed, 09 Apr 2025 23:09:42 GMT</pubDate>
            <atom:updated>2025-04-09T23:16:07.832Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5iH9EVyoHYhmYSY6ZXUeg.jpeg" /></figure><p>Em projetos de Machine Learning, na maioria das vezes o foco está na arquitetura do modelo ou na escolha dos hiperparâmetros. Mas a verdade é que <strong>a qualidade dos dados e o modo como eles são entregues ao modelo durante o treinamento influenciam diretamente nos resultados</strong>, tanto em performance quanto em capacidade de generalização.</p><p>Sempre que começo a desenvolver uma rede neural e os resultados são insatisfatórios, <strong>não importa o que eu faça ou qual técnica complexa eu tire do fundo do baú — simplesmente não resolve</strong>. Mas quando eu passo mais tempo nos dados, o jogo vira.</p><p>Felizmente, a TensorFlow oferece a API tf.data, que permite construir pipelines de dados eficientes e reutilizáveis. Neste post, vamos explorar os principais conceitos e métodos do tf.data, com foco em boas práticas para garantir qualidade e desempenho no treinamento de modelos. Já abre o teu jupyter notebook/google colab ai e vamos codar um pouco juntos!</p><h3>O que é tf.data e por que usá-lo?</h3><p>Em aplicações reais de Machine Learning, especialmente com grandes volumes de dados ou arquiteturas de Deep Learning mais robustas, não basta apenas ler os dados de forma ingênua — como carregar tudo na memória com NumPy ou iterar sobre arquivos com loops tradicionais. Isso rapidamente se torna um gargalo, tanto em termos de desempenho quanto de organização do código.</p><p>A API tf.data foi projetada para resolver justamente esse problema. Ela permite a construção de pipelines de dados de forma <strong>modular, feita para atingir uma boa performance e integrada ao TensorFlow</strong>. Com ela, você pode:</p><ul><li>Carregar dados de diferentes fontes (arquivos CSV, imagens, TFRecord, etc.);</li><li>Realizar transformações (como normalização, data augmentation, etc.);</li><li>Embaralhar, agrupar em lotes (batching) e repetir o conjunto;</li><li>Processar dados em paralelo e pré-carregar em memória com prefech(), otimizando o tempo entre batches;</li><li>Criar pipelines reutilizáveis para treinamento, validação e inferência.</li></ul><p>Em outras palavras, ao invés de perder tempo com loops e estruturas manuais, o tf.data te dá uma forma escalável e nativa de preparar seus dados como parte da pipeline de treino. Isso traz vantagens tanto na qualidade dos dados quanto na velocidade de execução do modelo.</p><h3>Principais métodos do tf.data na prática</h3><p>Primeiro, vamos gerar uns dados com NumPy + Tensorflow (ambos já vem instalados no google colab):</p><pre>import tensorflow as tf<br>import numpy as np<br><br>num_samples = 500_000 # vamos criar um conjunto de dados grande<br>num_features = 10<br><br>X = np.random.rand(num_samples, num_features).astype(np.float32) # dados de entrada<br>y = np.random.randint(0, 2, size=(num_samples,)).astype(np.int32) # labels binários<br><br># aqui criamos um dataset com tf.data<br>dataset = tf.data.Dataset.from_tensor_slices((X, y))<br><br>for features, label in dataset.take(3): # take(3) é o equivalente ao .head(3) do pandas<br>    print(&quot;Features:&quot;, features.numpy(), &quot;Label:&quot;, label.numpy())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*PidWc2C6IN93QYcj3I4upA.png" /><figcaption>output: google colab</figcaption></figure><p>A seguir, vamos explorar os métodos mais usados do tf.data, explicando o propósito de cada um e mostrando como eles impactam o pipeline.</p><h4>1 - “.batch(batch_size)”</h4><p>Agrupa os dados em lotes. Isso é essencial para o treinamento de redes neurais, já que os otimizadores geralmente trabalham com mini-batches.</p><pre>batch_size = 32<br>batched_dataset = dataset.batch(batch_size)<br><br>for batch_features, batch_labels in batched_dataset.take(1):<br>    print(&quot;Shape das features:&quot;, batch_features.shape)<br>    print(&quot;Shape dos labels:&quot;, batch_labels.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*VIxcKZHZdruUuQPWJpLc_Q.png" /><figcaption>output: google colab</figcaption></figure><p><em>Importante: batching sempre deve vir </em><strong><em>depois</em></strong><em> do shuffle para garantir que os dados embaralhados não sejam “reorganizados” em batches fixos.</em></p><h4>2 - “.shuffle(buffer_size)”</h4><p>Embaralha os dados para que o modelo não aprenda padrões indesejados da ordem do dataset.</p><pre>shuffled_dataset = dataset.shuffle(buffer_size=1000)<br><br>for features, label in shuffled_dataset.take(1):<br>    print(&quot;Amostra embaralhada:&quot;, features.numpy(), &quot;Label:&quot;, label.numpy())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*cM0T4OO5sOoDZKLmoRw8YQ.png" /><figcaption>output: google colab</figcaption></figure><p><em>O parâmetro buffer_size define o número de elementos que ficam na memória para serem embaralhados. Quanto maior, melhor a aleatoriedade, e também, mais uso de memória.</em></p><h4>3 - “.repeat(count)”</h4><p>Permite que o dataset seja repetido automaticamente, útil para treinar por múltiplas épocas sem redefinir o iterador.</p><pre>repeated_dataset = dataset.repeat(2)  # vai duplicar os dados<br><br># só imprimindo os primeiros exemplos para demonstrar<br>for features, label in repeated_dataset.take(3):<br>    print(&quot;Repetido:&quot;, features.numpy(), &quot;Label:&quot;, label.numpy())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*rzWRDnByJBdPKRxEq4NugA.png" /></figure><p><em>Cuidado: .repeat() sem argumento (dataset.repeat()) repete infinitamente — isso é útil com .model.fit(), mas perigoso ao iterar manualmente.</em></p><h4>4 - “.map(func, num_parallel_calls=tf.data.AUTOTUNE)”</h4><p>Aplica uma função transformação em cada item do dataset. É onde normalmente fazemos pré-processamentos, normalizações e etc.</p><pre># função de exemplo -&gt; multiplicar os dados por 2<br>def preprocess(x, y):<br>    return x * 2.0, y<br><br>mapped_dataset = dataset.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)<br><br>for x, y in mapped_dataset.take(1):<br>    print(&quot;Transformado:&quot;, x.numpy(), &quot;Label:&quot;, y.numpy())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*L-bdHfnMkI85W4wlV1-LVw.png" /></figure><h4>5 - “.prefetch(buffer_size=tf.data.AUTOTUNE)”</h4><p>Carrega o próximo batch enquanto o modelo ainda está treinando com o batch atual. Isso melhora a performance geral.</p><pre>pipeline = (<br>    dataset<br>    .shuffle(1000)<br>    .map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)<br>    .batch(32)<br>    .prefetch(tf.data.AUTOTUNE)<br>)<br><br>for batch_x, batch_y in pipeline.take(1):<br>    print(&quot;Batch final (com prefetch):&quot;, batch_x.shape, batch_y.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*58OtVFTRW0ZWuCXuiwN9TQ.png" /></figure><p><em>Boas práticas: sempre finalize seu pipeline com .prefetch() para aproveitar ao máximo o paralelismo.</em></p><blockquote>Sobre paralelismo e performance elevada, recomendo esses dois links:</blockquote><blockquote><a href="https://www.tensorflow.org/guide/data_performance?hl=pt-br">https://www.tensorflow.org/guide/data_performance?hl=pt-br</a></blockquote><blockquote><a href="https://www.tensorflow.org/guide/distributed_training?hl=pt-br">https://www.tensorflow.org/guide/distributed_training?hl=pt-br</a></blockquote><h3>tf.data e boas práticas em MLOps</h3><p>Eu não poderia deixar de falar em MLOps, não é mesmo? haha</p><p>O uso da API tf.data vai além de facilitar o código: ele se alinha diretamente com princípios de MLOps, como reprodutibilidade, modularidade e performance.</p><p>Aqui vão algumas boas práticas que aplicam tanto ao dia a dia quanto à preparação de pipelines para produção:</p><h4>1. <strong>Pipeline definido fora do loop de treinamento</strong></h4><p>Crie o pipeline de dados antes de chamar model.fit() ou o loop de treino. Isso mantém a separação clara entre preparação de dados e lógica de treino.</p><pre>def create_pipeline(X, y, batch_size=32, shuffle_buffer=1000):<br>    dataset = tf.data.Dataset.from_tensor_slices((X, y))<br>    dataset = dataset.shuffle(shuffle_buffer)<br>    dataset = dataset.map(preprocess, num_parallel_calls=tf.data.AUTOTUNE)<br>    dataset = dataset.batch(batch_size)<br>    dataset = dataset.prefetch(tf.data.AUTOTUNE)<br>    return dataset</pre><h4>2. Evite operações pesadas no .map()</h4><p>Sempre que possível, mova operações pesadas (como transformações com lógica complexa) para antes do pipeline. Isso reduz o custo por amostra dentro do fluxo de treino.</p><h4>3. Use AUTOTUNE sempre que possível</h4><p>A TensorFlow faz um bom trabalho estimando o paralelismo ideal para .map() e .prefetch(). Isso ajuda seu pipeline a se adaptar ao hardware disponível (CPU, GPU, etc).</p><h4>4. <strong>Separação clara entre treino, validação e teste</strong></h4><p>Evite usar o mesmo pipeline para tudo. Dados de validação e teste não devem ter shuffle(), repeat() nem augmentações. Um exemplo:</p><pre># supondo que você já separou seus dados...<br>train_dataset = create_pipeline(X_train, y_train)<br>val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)).batch(32).prefetch(tf.data.AUTOTUNE)</pre><h4>5. <strong>Facilidade de logging e debug</strong></h4><p>Como tf.data é iterável, fica fácil usar <strong><em>for x in dataset.take(n): print(x)</em></strong> em qualquer etapa para inspecionar os dados. Isso é ouro pra debug.</p><h3>Pra fechar e ir treinar no notebook</h3><p>Na prática, um pipeline de dados é tão importante quanto a arquitetura de um modelo. Não adianta usar o <em>state-of-the-art</em> se os dados estão mal alimentados. A API tf.data traz exatamente o que a gente precisa: controle, eficiência e integração nativa com o TensorFlow, sem precisar reinventar a roda.</p><p>Com poucos métodos — .map(), .shuffle(), .batch(), .prefetch() — dá pra montar um pipeline robusto que escala do seu notebook pro ambiente de produção sem mudar tanto.</p><p>No final das contas, modelagem de verdade começa nos dados. Se o seu modelo ainda não está entregando, talvez não seja o otimizador. Talvez seja o pipeline de dados.</p><p>Curtiu o conteúdo? Estou sempre trazendo mais conteúdos como esse, sempre estimulando você a melhorar performance e de forma prática. Me segue aqui no Medium ou conecta comigo lá no <a href="https://www.linkedin.com/in/juanvieira85/">LinkedIn</a> pra acompanhar! Valeu e até breve!</p><p>No final das contas, modelagem de verdade começa nos dados. Se o seu modelo ainda não está entregando, talvez não seja o otimizador. Talvez seja o pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d7c7d513f0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>