<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rodrigo Higashi on Medium]]></title>
        <description><![CDATA[Stories by Rodrigo Higashi on Medium]]></description>
        <link>https://medium.com/@rockigo?source=rss-b42500199fcb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4wxmFYovakdc6tHP</url>
            <title>Stories by Rodrigo Higashi on Medium</title>
            <link>https://medium.com/@rockigo?source=rss-b42500199fcb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:33:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rockigo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Turning Documentation into an AI Assistant with Web Scraping, Embeddings, and RAG]]></title>
            <link>https://ai.plainenglish.io/turning-documentation-into-an-ai-assistant-with-web-scraping-embeddings-and-rag-1d1827ad8e35?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d1827ad8e35</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[faiss]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[groq]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Wed, 08 Apr 2026 00:44:40 GMT</pubDate>
            <atom:updated>2026-04-12T18:40:11.850Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/p/bcbf77018559?postPublishedType=initial">here</a>.</p><p>When we’re learning a new technology, there’s a very common problem: the official documentation isn’t always easy to consult.</p><p>In theory, the documentation is the most reliable source. In practice, it tends to be:</p><ul><li>too long,</li><li>spread across many pages,</li><li>difficult to navigate,</li><li>and not always easy to search when you don’t know exactly which keywords to use.</li></ul><p>With this in mind, I built Docstoteles, a project that transforms any web documentation into an AI-powered chat, allowing you to ask questions in natural language about that content.</p><p>The idea was to combine web scraping, embeddings, vector search and RAG (Retrieval-Augmented Generation) into a simple application, yet one that is very useful for study, exploration and productivity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3mbGKj6Aa8hh_tnkNfLS6A.png" /><figcaption>Image generated by AI</figcaption></figure><h3>The problem: documentation is not always “conversational”</h3><p>Anyone who has tried to learn something new directly from the official documentation knows how this goes.</p><p>You may want to find out, for example:</p><ul><li>how to install a tool</li><li>how to configure a feature</li><li>what the difference is between two approaches</li><li>or which command to use in a given scenario</li></ul><p>But instead of simply asking that question, you usually need to:</p><ul><li>find the right page</li><li>navigate through menus and subtopics</li><li>open multiple links</li><li>and hope you use the exact keyword the internal search engine understands</li></ul><p>In other words, the information is there — but accessing it is not always smooth.</p><p>That was the idea behind this project:</p><p><strong>what if I could talk to documentation as if it were an assistant?</strong></p><h3>The project idea</h3><p>Docstoteles was built to do exactly that.</p><p>The workflow is simple:</p><ul><li>the user provides the URL of a documentation website</li><li>the system maps its pages</li><li>extracts the content through web scraping</li><li>transforms that content into a vector-based knowledge base</li><li>and then allows natural-language questions about it</li></ul><p>In practice, this means that instead of relying only on manual navigation, you can ask questions like:</p><ul><li>“How do I install Oh My Posh on Windows?”</li><li>“How do I initialize this in PowerShell?”</li><li>“What is the recommended configuration method?”</li><li>“What is the difference between these two approaches?”</li></ul><p>And the system will try to answer based on the documentation it previously scraped.</p><h3>The stack used</h3><p>To build the project, I used a simple and highly educational stack:</p><ul><li><strong>Streamlit</strong> for the interface</li><li><strong>Firecrawl</strong> for page scraping</li><li><strong>LangChain</strong> to orchestrate the RAG pipeline</li><li><strong>Hugging Face Embeddings</strong> to transform text into vectors</li><li><strong>FAISS</strong> as a local vector store</li><li><strong>Groq</strong> as the language model provider</li></ul><p>This combination was especially interesting because it made it possible to build a functional project without relying on heavy infrastructure.</p><p>On top of that, using accessible tools makes this project especially valuable for <strong>portfolio building</strong>, <strong>hands-on learning</strong>, and <strong>experimentation</strong>.</p><h3>Practical implementation</h3><p>One interesting aspect of this project is that it was built using accessible tools, including free-tier resources.</p><p>For scraping, for example, I used the <strong>free version of Firecrawl</strong>, which was already enough to validate the idea, extract real documentation content, and build a functional knowledge base for the system.</p><p>This shows that it is already possible to build useful RAG applications without necessarily depending on complex infrastructure from the start.</p><p>At the same time, this type of architecture has plenty of room to grow. In future versions, it would be possible to increase scraping depth, process more pages, work with multiple documentation sources, and make context retrieval even more robust.</p><h3>How the project works internally</h3><p>The most interesting part of Docstoteles is not just “doing scraping” or “calling an AI model”.</p><p>The real value lies in <strong>how these pieces connect</strong>.</p><h3>1) Content collection</h3><p>The first step is to collect the documentation content.</p><p>Starting from an initial URL, the system uses Firecrawl to:</p><ul><li>map the documentation links</li><li>visit the discovered pages</li><li>extract the content into a usable format</li><li>and save it locally as .md files</li></ul><p>This step is important because the language model does not automatically know the content of that specific website.</p><p>That content needs to be brought into the pipeline first.</p><h3>2) Chunking</h3><p>After the documents are saved, a fundamental step in any RAG system comes in: <strong>chunking</strong>.</p><p>Instead of feeding entire pages into the AI, the text is split into smaller pieces using a text splitter.</p><p>This is important because it:</p><ul><li>improves context retrieval</li><li>reduces noise</li><li>and helps the system find more relevant excerpts for each question</li></ul><p>In this project, I used <strong>RecursiveCharacterTextSplitter</strong>, which is a very common approach for this type of pipeline.</p><h3>3) Embeddings: turning text into meaning</h3><p>Once the content is split into smaller parts, each chunk is converted into an <strong>embedding</strong>.</p><p>Embeddings are vector representations of text.</p><p>Instead of looking only at exact words, they help the system capture <strong>semantic similarity</strong>.</p><p>That means the user does not necessarily need to ask a question using the exact same wording as the documentation.</p><p>For example:</p><ul><li>the documentation might talk about <strong>“initialization”</strong></li><li>while the user may ask <strong>“how do I set this up?”</strong></li></ul><p>Even without a perfect literal match, the system can still retrieve useful passages based on similarity in meaning.</p><p>This is one of the most interesting parts of the project, because this is where search stops being just <strong>keyword-based</strong> and starts becoming <strong>semantically smarter</strong>.</p><h3>4) Vector search with FAISS</h3><p>Once the embeddings are ready, they are stored in a <strong>vector store</strong>, using FAISS.</p><p>When the user asks a question, the application:</p><ul><li>transforms the question into an embedding</li><li>compares that vector against the stored vectors</li><li>retrieves the most relevant chunks</li><li>and only then sends them to the language model</li></ul><p>This step is what prevents the system from depending exclusively on the model’s memory.</p><p>Instead of asking the AI to “guess” the answer, the application first retrieves relevant evidence from the documentation itself.</p><h3>5) RAG: response generation with retrieved context</h3><p>This is where the core of the project comes in: <strong>RAG (Retrieval-Augmented Generation)</strong>.</p><p>The logic is simple and powerful:</p><ul><li>first, the system retrieves the most relevant excerpts</li><li>then, the language model generates a response based on that context</li></ul><p>This pattern is extremely useful because it improves response quality in scenarios where there is a specific knowledge base, such as:</p><ul><li>technical documentation</li><li>internal policies</li><li>knowledge bases</li><li>manuals</li><li>FAQs</li><li>or institutional content</li></ul><p>In the case of Docstoteles, this allows the AI to answer questions about <strong>that specific website</strong>, rather than relying only on the model’s general knowledge.</p><h3>What this kind of project solves in practice</h3><p>At first glance, this kind of application may seem simple — but it solves very real problems.</p><h3>For learning</h3><p>It allows you to explore documentation more fluidly, asking questions the same way you would naturally think about them.</p><h3>For productivity</h3><p>Instead of wasting time searching through pages manually, you reduce search friction.</p><h3>For technology exploration</h3><p>It is especially useful when you want a first contact with a new tool without depending entirely on manual navigation.</p><h3>For portfolio value</h3><p>This kind of project demonstrates several important skills at once:</p><ul><li>Python</li><li>text data handling</li><li>API consumption</li><li>web scraping</li><li>embeddings</li><li>vector stores</li><li>LLM integration</li><li>AI application structuring</li></ul><p>In other words, it is a relatively small project in size, but a <strong>high-value project in terms of demonstrated capability</strong>.</p><h3>What I learned while building this project</h3><p>Even though the project may look relatively straightforward, it brought several practical lessons.</p><p>Some examples include:</p><ul><li>the importance of correctly naming environment variables</li><li>how small configuration details can break API integrations</li><li>how models can be deprecated and require updates</li><li>and how the “simple” part of an application is often in the idea — not necessarily in the implementation</li></ul><p>It was also a great exercise in understanding, in practice, that <strong>RAG does not mean “an AI that knows everything.”</strong></p><p>In fact, the logic is almost the opposite: the value lies in making the AI consult the right source before answering.</p><p>That difference changes quite a lot in how we think about LLM-based applications.</p><h3>Possible future improvements</h3><p>As a foundational project, Docstoteles still has plenty of room to evolve. Some interesting future improvements would be:</p><ul><li>allowing PDF uploads</li><li>chatting with multiple collections at the same time</li><li>displaying sources and retrieved excerpts in the answers</li><li>allowing collection deletion through the interface</li><li>improving the visual chat experience</li><li>and eventually publishing a cloud-based version</li></ul><p>In other words, the foundation is already there — and now it can grow in many different directions.</p><h3>Conclusion</h3><p>Building Docstoteles was a practical way to bring together several important concepts in applied AI into a single project:</p><ul><li>web scraping</li><li>embeddings</li><li>vector search</li><li>RAG</li><li>LLMs</li><li>and an interactive interface</li></ul><p>More than just “building a chatbot”, the idea here was to transform static documentation into something more useful, navigable, and conversational.</p><p>And this kind of idea goes far beyond technical documentation.</p><p>The same logic can be applied to:</p><ul><li>knowledge bases</li><li>internal materials</li><li>manuals</li><li>educational content</li><li>and much more</li></ul><p>At the end of the day, this project reinforced an idea that I find increasingly important:</p><p>very often, the value of AI is not in “knowing everything”,<strong> but in being able to access the right information well.</strong></p><h3>Project GitHub</h3><p>If you want to explore the code, adapt the idea, or test it with other documentation sources, I made the project available on GitHub: <a href="https://github.com/rodrigohigashi/docstoteles"><strong>Docstoteles</strong></a>.</p><p>You can also explore a live version of the project here: [Streamlit Cloud App](<a href="https://docstoteles-en.streamlit.app/">https://docstoteles-en.streamlit.app</a>), where Docstoteles is deployed and fully interactive, allowing you to test the RAG pipeline in real time.</p><p>You can also check out my other projects on <a href="https://github.com/rodrigohigashi">GitHub</a> or read more articles in my <a href="https://www.datascienceportfol.io/rockigo">portfolio</a>.</p><p><strong>A message from our Founder</strong></p><p>Hey, <a href="https://linkedin.com/in/sunilsandhu">Sunil</a> here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.</p><p>If you want to show some love, please take a moment to follow me on <a href="https://linkedin.com/in/sunilsandhu">LinkedIn</a>, <a href="https://tiktok.com/@messyfounder">TikTok</a>, <a href="https://instagram.com/sunilsandhu">Instagram</a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/">weekly newsletter</a>. And before you go, don’t forget to clap and follow the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d1827ad8e35" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/turning-documentation-into-an-ai-assistant-with-web-scraping-embeddings-and-rag-1d1827ad8e35">Turning Documentation into an AI Assistant with Web Scraping, Embeddings, and RAG</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformando documentação em um assistente de IA com Web Scraping, Embeddings e RAG]]></title>
            <link>https://medium.com/@rockigo/transformando-documenta%C3%A7%C3%A3o-em-um-assistente-de-ia-com-web-scraping-embeddings-e-rag-bcbf77018559?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/bcbf77018559</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[groq]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[faiss]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Wed, 08 Apr 2026 00:25:19 GMT</pubDate>
            <atom:updated>2026-04-12T18:41:37.601Z</atom:updated>
            <content:encoded><![CDATA[<p>Este artigo também está disponível em inglês <a href="https://medium.com/p/1d1827ad8e35?postPublishedType=initial">aqui</a>.</p><p>Quando estamos estudando uma tecnologia nova, existe um problema muito comum: <strong>a documentação oficial nem sempre é fácil de consultar</strong>.</p><p>Na teoria, a documentação é a fonte mais confiável. Na prática, ela costuma ser:</p><ul><li>longa demais,</li><li>dividida em muitas páginas,</li><li>difícil de navegar,</li><li>e nem sempre fácil de pesquisar quando você <strong>não sabe exatamente quais palavras-chave usar</strong>.</li></ul><p>Foi pensando nisso que construí o <strong>Docstoteles</strong>, um projeto que transforma qualquer documentação da web em um <strong>chat com IA</strong>, permitindo fazer perguntas em linguagem natural sobre aquele conteúdo.</p><p>A ideia foi unir <strong>web scraping</strong>, <strong>embeddings</strong>, <strong>busca vetorial</strong> e <strong>RAG (Retrieval-Augmented Generation)</strong> em uma aplicação simples, mas muito útil para estudo, exploração e produtividade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3mbGKj6Aa8hh_tnkNfLS6A.png" /><figcaption>Imagem gerada por IA</figcaption></figure><h3>O problema: documentação nem sempre é “conversável”</h3><p>Quem já tentou aprender algo novo direto pela documentação sabe como isso funciona:</p><p>Você quer descobrir, por exemplo:</p><ul><li>como instalar uma ferramenta,</li><li>como configurar uma feature,</li><li>qual a diferença entre duas abordagens,</li><li>ou qual comando usar em determinado cenário.</li></ul><p>Mas, em vez de simplesmente perguntar isso, normalmente você precisa:</p><ol><li>procurar a página certa,</li><li>navegar por menus e subtópicos,</li><li>abrir vários links,</li><li>e torcer para usar a palavra exata que o buscador interno entende.</li></ol><p>Ou seja: <strong>a informação existe</strong>, mas o acesso até ela nem sempre é fluido.</p><p>Foi daí que veio a ideia do projeto:<br> <strong>e se eu pudesse conversar com a documentação como se ela fosse um assistente?</strong></p><h3>A proposta do projeto</h3><p>O <strong>Docstoteles</strong> foi criado para fazer exatamente isso.</p><p>O fluxo funciona assim:</p><ol><li>o usuário informa a URL de uma documentação;</li><li>o sistema faz o <strong>mapeamento das páginas</strong>;</li><li>extrai o conteúdo com <strong>web scraping</strong>;</li><li>transforma esse conteúdo em uma <strong>base vetorial</strong>;</li><li>e depois permite fazer perguntas em linguagem natural sobre aquilo.</li></ol><p>Na prática, isso significa que, em vez de depender só de navegação manual, você pode perguntar coisas como:</p><ul><li>“Como instalar o Oh My Posh no Windows?”</li><li>“Como inicializar isso no PowerShell?”</li><li>“Qual é a forma recomendada de configuração?”</li><li>“Qual a diferença entre essas duas abordagens?”</li></ul><p>E o sistema tenta responder com base <strong>na própria documentação que foi raspada</strong>.</p><h3>A stack utilizada</h3><p>Para construir o projeto, usei uma stack simples e bastante didática:</p><ul><li><strong>Streamlit</strong> para a interface;</li><li><strong>Firecrawl</strong> para o scraping das páginas;</li><li><strong>LangChain</strong> para orquestrar o pipeline de RAG;</li><li><strong>Hugging Face Embeddings</strong> para transformar texto em vetores;</li><li><strong>FAISS</strong> como banco vetorial local;</li><li><strong>Groq</strong> como provedor do modelo de linguagem.</li></ul><p>Essa combinação foi interessante porque permitiu montar um projeto funcional sem depender de uma infraestrutura pesada.</p><p>Além disso, o fato de usar ferramentas acessíveis torna o projeto muito bom para <strong>portfólio</strong>, <strong>aprendizado prático</strong> e experimentação.</p><h3>Implementação prática</h3><p>Um ponto interessante desse projeto é que ele foi construído com ferramentas acessíveis, incluindo recursos gratuitos.</p><p>No scraping, por exemplo, utilizei a versão free do Firecrawl, que já foi suficiente para validar a proposta, extrair conteúdo real da documentação e montar uma base funcional para o sistema.</p><p>Isso mostra que já é possível construir aplicações úteis com RAG sem depender, necessariamente, de uma infraestrutura complexa logo no início.</p><p>Ao mesmo tempo, esse tipo de arquitetura também tem bastante espaço para crescer. Em versões futuras, seria possível expandir a profundidade do scraping, aumentar a quantidade de páginas processadas, trabalhar com múltiplas documentações e tornar a recuperação de contexto ainda mais robusta.</p><h3>Como o projeto funciona por dentro</h3><p>A parte mais interessante do Docstoteles não é apenas “fazer scraping” ou “chamar uma IA”.<br> O valor real está em <strong>como essas peças se conectam</strong>.</p><h3>1) Coleta do conteúdo</h3><p>O primeiro passo é coletar o conteúdo da documentação.</p><p>A partir de uma URL inicial, o sistema usa o <strong>Firecrawl</strong> para:</p><ul><li>mapear os links da documentação;</li><li>visitar as páginas encontradas;</li><li>extrair o conteúdo em formato utilizável;</li><li>e salvar isso localmente como arquivos .md.</li></ul><p>Essa etapa é importante porque o modelo de linguagem <strong>não sabe automaticamente</strong> o conteúdo daquele site específico.<br> Ele precisa que esse conteúdo seja <strong>trazido para dentro do pipeline</strong>.</p><h3>2) Quebra em chunks</h3><p>Depois de salvar os documentos, entra uma etapa fundamental em qualquer sistema RAG: o <strong>chunking</strong>.</p><p>Em vez de jogar páginas inteiras para a IA, o texto é quebrado em pedaços menores usando um <strong>text splitter</strong>.</p><p>Isso é importante porque:</p><ul><li>melhora a recuperação de contexto,</li><li>reduz ruído,</li><li>e ajuda o sistema a encontrar trechos mais relevantes para cada pergunta.</li></ul><p>No projeto, usei RecursiveCharacterTextSplitter, que é uma abordagem bastante comum para esse tipo de pipeline.</p><h3>3) Embeddings: transformando texto em significado</h3><p>Depois que o conteúdo é quebrado em partes menores, cada trecho é convertido em um <strong>embedding</strong>.</p><p>Embeddings são representações vetoriais do texto.<br> Em vez de olhar apenas para palavras exatas, eles ajudam o sistema a capturar <strong>semelhança semântica</strong>.</p><p>Isso significa que o usuário não precisa necessariamente perguntar usando as mesmas palavras da documentação.</p><p>Por exemplo:</p><ul><li>a documentação pode falar em “initialization”</li><li>e o usuário pode perguntar “como eu configuro isso?”</li></ul><p>Mesmo sem correspondência literal perfeita, o sistema ainda pode recuperar trechos úteis por <strong>proximidade de significado</strong>.</p><p>Essa é uma das partes mais interessantes do projeto, porque é aqui que a busca deixa de ser só “palavra-chave” e passa a ser mais <strong>inteligente semanticamente</strong>.</p><h3>4) Busca vetorial com FAISS</h3><p>Com os embeddings prontos, eles são armazenados em uma <strong>vector store</strong>, usando <strong>FAISS</strong>.</p><p>Quando o usuário faz uma pergunta, a aplicação:</p><ol><li>transforma a pergunta em embedding;</li><li>compara esse vetor com os vetores da base;</li><li>recupera os trechos mais relevantes;</li><li>e só então envia isso para o modelo de linguagem.</li></ol><p>Esse passo é o que faz o sistema não depender exclusivamente da memória do modelo.</p><p>Em vez de pedir para a IA “adivinhar” a resposta, a aplicação primeiro busca <strong>evidência relevante</strong> dentro da documentação.</p><h3>5) RAG: resposta com contexto recuperado</h3><p>Aqui entra o coração do projeto: o <strong>RAG (Retrieval-Augmented Generation)</strong>.</p><p>A lógica é simples e poderosa:</p><ul><li>primeiro, o sistema <strong>recupera</strong> os trechos mais relevantes;</li><li>depois, o modelo de linguagem <strong>gera</strong> a resposta com base nesse contexto.</li></ul><p>Esse padrão é muito útil porque melhora a qualidade da resposta em cenários onde existe uma base específica de conhecimento, como:</p><ul><li>documentação técnica,</li><li>políticas internas,</li><li>bases de conhecimento,</li><li>manuais,</li><li>FAQs,</li><li>ou conteúdo institucional.</li></ul><p>No caso do Docstoteles, isso permite que a IA responda <strong>sobre aquele site específico</strong>, e não apenas com base no conhecimento geral do modelo.</p><h3>O que esse tipo de projeto resolve na prática</h3><p>Esse tipo de aplicação pode parecer simples à primeira vista, mas resolve problemas bem reais.</p><h3>Para estudo</h3><p>Você consegue aprender uma documentação de forma mais fluida, fazendo perguntas do jeito que pensaria naturalmente.</p><h3>Para produtividade</h3><p>Em vez de perder tempo procurando páginas específicas, você reduz o atrito de busca.</p><h3>Para exploração de tecnologias</h3><p>É útil quando você quer ter um primeiro contato com uma ferramenta nova sem depender exclusivamente de navegação manual.</p><h3>Para portfólio</h3><p>Esse tipo de projeto demonstra várias competências importantes de uma vez:</p><ul><li>Python</li><li>manipulação de dados textuais</li><li>consumo de APIs</li><li>web scraping</li><li>embeddings</li><li>vector stores</li><li>integração com LLMs</li><li>estruturação de aplicações com IA</li></ul><p>Ou seja: é um projeto pequeno no tamanho, mas <strong>grande no valor demonstrado</strong>.</p><h3>O que aprendi construindo esse projeto</h3><p>Apesar de o projeto parecer relativamente direto, ele trouxe vários aprendizados práticos.</p><p>Alguns exemplos:</p><ul><li>a importância de <strong>nomear corretamente variáveis de ambiente</strong>;</li><li>como pequenos detalhes de configuração podem quebrar a integração com APIs;</li><li>como modelos podem ser descontinuados e exigir atualização;</li><li>e como a parte “simples” de uma aplicação muitas vezes está na ideia — não necessariamente na implementação.</li></ul><p>Também foi um ótimo exercício para entender, na prática, que RAG não significa “<strong>uma IA que sabe tudo</strong>”.<br> Na verdade, a lógica é quase o oposto: <strong>o valor está em fazer a IA consultar a fonte certa antes de responder.</strong></p><p>Essa diferença muda bastante a forma como pensamos aplicações com LLMs.</p><h3>Possíveis melhorias futuras</h3><p>Como projeto base, o Docstoteles ainda tem bastante espaço para evoluir.</p><p>Algumas melhorias interessantes seriam:</p><ul><li>permitir upload de PDFs;</li><li>conversar com múltiplas coleções ao mesmo tempo;</li><li>exibir fontes e trechos usados na resposta;</li><li>permitir exclusão de coleções pela interface;</li><li>melhorar a experiência visual do chat;</li><li>e eventualmente publicar uma versão em nuvem.</li></ul><p>Ou seja: a base já está pronta, e agora ele pode crescer em várias direções.</p><h3>Conclusão</h3><p>Construir o Docstoteles foi uma forma prática de juntar vários conceitos importantes de IA aplicada em um único projeto:</p><ul><li><strong>web scraping</strong></li><li><strong>embeddings</strong></li><li><strong>busca vetorial</strong></li><li><strong>RAG</strong></li><li><strong>LLMs</strong></li><li><strong>interface interativa</strong></li></ul><p>Mais do que “fazer um chatbot”, a proposta aqui foi transformar documentação estática em algo mais útil, navegável e conversável.</p><p>E esse tipo de ideia tem aplicação muito além de documentação técnica.<br> A mesma lógica pode ser usada para bases de conhecimento, materiais internos, manuais, conteúdo educacional e muito mais.</p><p>No fim das contas, esse projeto reforçou uma ideia que acho cada vez mais importante: <em>muitas vezes, o valor da IA não está em “saber tudo”, mas em </em><strong><em>acessar bem a informação certa</em></strong><em>.</em></p><h3>GitHub do projeto</h3><p>Se você quiser explorar o código, adaptar a ideia ou testar com outras documentações, deixei o projeto disponível no GitHub: <a href="https://github.com/rodrigohigashi/docstoteles">docstoteles</a>.</p><p>Você também pode acessar a versão ao vivo do projeto aqui: [Streamlit Cloud App](<a href="https://docstoteles-pt.streamlit.app/">https://docstoteles-pt.streamlit.app</a>), onde o Docstoteles está implantado e totalmente interativo, permitindo testar o pipeline de RAG em tempo real.</p><p>Confira meus projetos completos no <a href="https://github.com/rodrigohigashi">GitHub</a> ou veja mais artigos no <a href="https://www.datascienceportfol.io/rockigo">meu portfólio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bcbf77018559" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What If a Neural Network Could Diagnose Heart Disease?]]></title>
            <link>https://ai.plainenglish.io/what-if-a-neural-network-could-diagnose-heart-disease-c5c1637b185b?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5c1637b185b</guid>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Sun, 28 Dec 2025 14:47:55 GMT</pubDate>
            <atom:updated>2026-02-05T07:00:19.694Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/@rockigo/e-se-uma-rede-neural-pudesse-diagnosticar-doen%C3%A7a-card%C3%ADaca-d4bf4b14c073?postPublishedType=repub">here</a>.</p><p>Artificial intelligence has proven to be a powerful ally in the healthcare field, and this project is a practical example of that. In this work, I apply artificial neural networks to predict the presence of heart disease based on clinical variables from multiple patients. The dataset used contains information such as age, sex, blood pressure, cholesterol levels, among other features. The goal is to train a model that, given these data, can identify whether a person is at risk of developing heart disease — a task represented by a binary variable: 1 for presence of the disease and 0 for absence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*h56BE_AazFIlnaUj.jpeg" /><figcaption>AI-generated image illustrating the human cardiovascular system.</figcaption></figure><h3><strong>1 Understanding the Business</strong></h3><p><strong>1.1 Statement of the Problem</strong></p><p>Cardiovascular diseases remain one of the leading causes of death worldwide, according to the World Health Organization. Early diagnosis of these conditions is essential to increase the chances of effective treatment and survival. Cardiology, the branch of medicine dedicated to the study and care of the heart, often relies on clinical exams, medical history, and symptom assessment — processes that can be subjective or time-consuming.</p><p>In this context, the use of artificial intelligence emerges as a way to support medical diagnosis by providing a second opinion based on statistical patterns learned from real patient data. The objective of this project is to develop a predictive model capable of identifying, based on clinical variables, whether a patient has a higher or lower risk of developing heart disease.</p><p>This type of study can have practical applications in several domains, such as the diagnosis of diabetes, kidney disease, liver disease, COVID-19, depression, or other mental health disorders — in addition to use cases outside healthcare, including bank fraud detection or spam filtering.</p><p><strong>1.2 Objective</strong></p><p>The main objective of this study is to build a predictive model capable of automatically classifying whether a patient has heart disease or not, based on their clinical characteristics. The dataset itself, sourced from Kaggle, already includes the target variable, where a value of 1 indicates the presence of the disease, while 0 represents its absence.</p><p>In addition to predicting the presence or absence of heart disease, this analysis also aims to understand which features have the greatest influence on the diagnosis, such as chest pain type, resting blood pressure, cholesterol levels, or elevated glycemic indicators. Although the primary focus is on the application of neural networks, gaining insights into the factors that most influence the model’s decision is also highly relevant for practical applications in the healthcare domain.</p><h3><strong>2 Understanding the Data</strong></h3><p><strong>2.1 Data description and collection</strong></p><p>This study uses a dataset sourced from Kaggle known as the Heart Disease UCI dataset, originally compiled in 1988. The dataset consists of records collected from four different medical centers: Cleveland, Hungary, Switzerland, and Long Beach V. In total, it contains 76 attributes; however, in practice, most experiments and published analyses use a subset of 14 features, including the target variable.</p><p>The target variable represents the presence of heart disease and is defined as a binary field, where:</p><ul><li>0 indicates absence of the disease</li><li>1 indicates presence of the disease</li></ul><p>Below are the variables used in this study, along with their respective descriptions and data types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*Y6bwhBlcW9snmXfNnRA44g.png" /><figcaption>Table 1 — Metadata</figcaption></figure><p><strong>2.2 Exploratory Data Analysis</strong></p><p><strong>2.2.1 Sweetviz Report</strong></p><p>Before training the neural network, I explored the dataset using Sweetviz to better understand which variables are most strongly associated with the diagnosis of heart disease. Features such as age (around 60 years), chest pain type = 4, high resting blood pressure (approximately 180 mmHg), and lower maximum heart rate (around 90 bpm) stood out, showing positivity rates ranging from roughly 40% to 60%. These insights helped guide the subsequent modeling decisions.</p><p><strong>2.3 Training the model</strong></p><p>During the data preparation stage, I separated categorical and numerical features, transforming the categorical ones using OneHotEncoder to generate dummy variables. After preprocessing, all transformed features were combined and normalized using StandardScaler, ensuring the data was properly prepared as input for the artificial neural network.</p><h3>🤖 <strong>3. Modeling with Artificial Neural Networks</strong></h3><p>With the data prepared, the dataset was split into training and testing sets (70/30). Feature scaling was applied <strong>only to the training set</strong>, preventing data leakage and preserving the integrity of the test set. To ensure reproducibility of the results, a fixed random seed was used during neural network training.</p><p>The constructed neural network follows a <strong>Multilayer Perceptron (MLP)</strong> architecture with two hidden layers:</p><ul><li><strong>Hidden layer 1</strong>: 4 neurons, tanh activation, L1 = 0.01 and L2 = 0.01 regularization, input_dim equal to the number of input features, and a dropout rate of 0.2 to mitigate overfitting.</li><li><strong>Hidden layer 2</strong>: 8 neurons, ReLU activation, the same regularization parameters as the previous layer, and a dropout rate of 0.1.</li><li><strong>Output layer</strong>: 1 neuron with sigmoid activation, producing the final probability of a heart disease diagnosis.</li></ul><p>The model was compiled using <strong>binary cross-entropy</strong> as the loss function, the <strong>Adam optimizer</strong>, and <strong>AUC</strong> as the primary evaluation metric — since the main goal is to measure the model’s ability to distinguish between the two classes (presence or absence of heart disease).</p><h4>🧠<strong> 3.1 Neural Network Structure and Initial Evaluation</strong></h4><p>The model was built using Keras’ <strong>Sequential API</strong>, stacking layers in a linear fashion. The resulting architecture contains only <strong>161 trainable parameters</strong>, making it a lightweight network well suited to the size of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*Sq71w8zGl_sjuxsnNA3M0g.png" /><figcaption>Table 2 — Neural Network Structure</figcaption></figure><p>Before any fine-tuning of hyperparameters, the model exhibited <strong>unsatisfactory performance</strong>, with AUC values well below what would be expected from a binary classifier. On the training set, the AUC reached <strong>23.7%</strong>, while on the test set it dropped further to <strong>16.4%</strong>. These results indicate that, in its initial configuration, the neural network failed to capture meaningful patterns in the data, performing worse than a random binary classifier.</p><p>This outcome highlights the importance of <strong>architecture design and hyperparameter tuning</strong> as critical steps to improve the model’s learning capacity and generalization performance.</p><h4>🔁 <strong>3.2 Neural Network Training</strong></h4><p>The neural network was trained for <strong>50 epochs</strong> with a <strong>batch size of 8</strong>, using <strong>10% of the training data for validation</strong>. To prevent overfitting and speed up convergence, <strong>early stopping</strong> was applied while monitoring the <strong>validation AUC</strong>, with the following criteria:</p><ul><li><strong>patience = 20</strong>: training is stopped if the validation AUC does not improve for 20 consecutive epochs.</li><li><strong>min_delta = 0.01</strong>: an improvement of at least 1% in AUC is required to be considered significant.</li><li><strong>restore_best_weights = True</strong>: at the end of training, the model automatically restores the weights from the epoch with the best validation performance.</li></ul><h3>📈 3.3 Final Neural Network Performance</h3><p>With hyperparameter tuning and early stopping in place, the model achieved a <strong>remarkable performance improvement</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*MDmcT49wAQ5xxibSWi1s1g.png" /><figcaption>Table 3 — Baseline assessment and after adjustment of neural networks (AUC metric)</figcaption></figure><p>These results indicate that:</p><ul><li>The combination of <strong>L1/L2 regularization and dropout</strong> helped control overfitting, even with the increased model complexity compared to the initial configuration.</li><li>The use of <strong>early stopping</strong> preserved the best-performing weights before any loss of generalization occurred.</li><li>The model learned <strong>meaningful patterns</strong> from the data rather than simply memorizing the training set, as evidenced by the small gap between training and test metrics (approximately <strong>2 percentage points</strong> after tuning).</li></ul><p>In practical terms, this means that the neural network is able to <strong>distinguish between patients with and without heart disease in more than 91% of cases</strong>, achieving a robust performance comparable to traditional machine learning techniques applied to this dataset.</p><p>The <strong>AUC-per-epoch plot</strong> shows that the model learns in a stable manner and generalizes well:</p><ul><li>The validation curve remains consistently above the training curve, indicating strong generalization capability.</li><li>Early stopping halts the training at the optimal point, effectively preventing overfitting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*HWYa6dKEM8SR2DbBeh_TgQ.png" /><figcaption>Figure 2 — Model performance by season</figcaption></figure><h3>🔄 3.4 Training Function &amp; Hyperparameter Search</h3><p>To optimize the neural network’s performance, we created a function responsible for <strong>defining, compiling, and training the model</strong> based on selected hyperparameters: the <strong>number of neurons in the hidden layers</strong> and the <strong>dropout rate</strong>. The model is evaluated using <strong>AUC</strong> on both the <strong>training</strong> and <strong>test</strong> datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*p96S4Z6GZjCwc4AK9Rbv5w.png" /><figcaption>Figure 3 — ANN Training Function</figcaption></figure><p>Using this function, we performed a <strong>simple grid search</strong> over the hyperparameter space — testing <strong>1 to 4 neurons</strong> per hidden layer and <strong>dropout values between 0 and 0.2</strong> — resulting in the training of <strong>50 different models</strong>, from which the best configuration was selected based on AUC performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*iqzm7FTuPoISknV19F88KQ.png" /><figcaption>Figure 4 — Generating multiple neural network models using different combinations of hyperparameters</figcaption></figure><p>After training all models, the results were consolidated into a <strong>DataFrame</strong>. The <strong>difference</strong> column helps identify models that achieve high test AUC while avoiding overfitting, distinguishing them from models that merely memorized the training data.</p><h3>🔍 3.5 Hyperparameter Analysis</h3><p>To understand the impact of each hyperparameter on the neural network’s performance, we generated plots showing how <strong>AUC (training and test)</strong> behaves as we vary:</p><ul><li>The number of neurons in the <strong>first hidden layer (C1)</strong></li><li>The number of neurons in the <strong>second hidden layer (C2)</strong></li><li>The <strong>dropout rate</strong> in the first hidden layer (Dropout C1)</li></ul><p>In addition, we analyzed the <strong>trade-off between performance and generalization</strong> based on the difference between training and test AUC.</p><p>📈 <strong>3.5.1 Number of Neurons in Hidden Layer 1 (C1)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*yN5gZvsKiIND2mbZwSntGQ.png" /><figcaption>Figure 5 — Number of Neurons in Hidden Layer 1 (C1)</figcaption></figure><p>In the left plot, we observe a <strong>monotonically increasing behavior</strong> of AUC as the number of neurons in C1 increases. Unlike the previous analysis, there are no intermediate oscillations: each increment in the number of neurons leads to a consistent improvement in both training and test performance. Moreover, the curves remain very close to each other throughout the entire range, indicating stable learning behavior.</p><p>In the right plot, the <strong>difference between training and test AUC decreases progressively</strong> as the number of neurons increases. This suggests that slightly more complex architectures are better able to capture meaningful patterns in the data without harming generalization.</p><p>📌 <strong>Interpretation:</strong> increasing the number of neurons in C1 consistently improves model performance while simultaneously reducing the gap between training and test metrics, reinforcing a scenario of good generalization under active regularization.</p><p>📈 <strong>3.5.2 Number of Neurons in Hidden Layer 2 (C2)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*wPDldP-25n93LTQ7GgF8gg.png" /><figcaption>Figure 6 — Number of Neurons in Hidden Layer 2 (C2)</figcaption></figure><p>In the left plot, performance remains relatively stable between 1 and 2 neurons in C2. Beyond this point, a clear upward trend in AUC is observed for both training and test sets as the number of neurons increases, indicating that slightly deeper architectures begin to extract additional useful information from the data.</p><p>The right plot shows that the <strong>difference between training and test AUC remains small</strong> across the entire analyzed range, with minor fluctuations. Importantly, these values stay within a range considered safe, with no significant signs of overfitting.</p><p>📌 <strong>Interpretation:</strong> C2 shows lower sensitivity at very small neuron counts but starts to contribute positively to performance as its capacity increases. The observed variations do not compromise the model’s generalization ability.</p><p>📉<strong> 3.5.3 Dropout Rate in Hidden Layer 1</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*G_DK3g1Jw1pk9dKoP1m3tg.png" /><figcaption>Figure 7 — Dropout Rate in Hidden Layer 1</figcaption></figure><p>In the left plot, we observe that the <strong>training AUC exhibits a slight peak around a dropout value of 0.1</strong>, while the <strong>test performance also reaches its highest value at this point</strong>. When dropout is increased to 0.2, both curves decline, suggesting that <strong>excessive regularization begins to impair the model’s learning capacity</strong>.</p><p>The right plot reinforces this interpretation: the <strong>difference between training and test AUC is smallest for dropout = 0.1</strong>, increasing again both when no regularization is applied (dropout = 0) and when regularization becomes too strong (dropout = 0.2).</p><p>📌 <strong>Interpretation:</strong> a <strong>moderate dropout rate (around 10%)</strong> provides the best balance between learning and regularization, reducing the gap between training and test performance without overly constraining the network’s capacity.</p><p>📊 <strong>3.5.4 Analysis of the Relationship Between Performance and Generalization</strong></p><p>The scatter plot below relates the <strong>test AUC</strong> to the <strong>difference between training and test AUC</strong> for all <strong>48 evaluated models</strong>. The horizontal reference lines at <strong>–0.05, 0, and +0.05</strong> define the range considered acceptable for good generalization.</p><p>Most models are concentrated within this interval, while a few exhibit more extreme differences, indicating potential <strong>overfitting</strong> (large positive difference) or <strong>underfitting</strong> (strong negative difference). This visualization supports the criteria adopted for selecting the most robust models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*9oL99kQsz2Dc0rKicqGvZg.png" /><figcaption>Figure 8 — Scatter Plot of Test AUC vs. Train–Test Difference</figcaption></figure><p>💡 <strong>Note on Result Visualization</strong></p><p>In some plots within this section — particularly in the dropout analysis — we use <strong>two complementary representations</strong> on the axis that shows the difference between training and test AUC.</p><ul><li>The <strong>continuous line</strong> represents the <strong>average train–test difference</strong> for each evaluated hyperparameter value, providing an aggregated view of model behavior.</li><li>The <strong>individual points</strong> correspond to each trained model, linking its test AUC to the respective train–test difference.</li></ul><p>This combined approach allows us to analyze both <strong>global trends</strong> and the <strong>dispersion of individual results</strong>, offering a more comprehensive view of the trade-off between performance and generalization in the neural network.</p><h3>🎯 3.6 Best Model Selection</h3><p>After testing <strong>48 different hyperparameter combinations</strong>, we applied a filter to the DataFrame to retain only the models that exhibited:</p><ul><li><strong>Good generalization</strong>, with a difference between training and test AUC within the range <strong>–0.05 to +0.05</strong>;</li><li><strong>Excellent test performance</strong>, with a <strong>test AUC greater than 0.88</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*CI_MdVjEIoeek1CIE5Izdg.png" /><figcaption>Table 4 — Filtered Dataframe</figcaption></figure><p>Based on these criteria, we were able to identify a <strong>reduced set of promising models</strong>, which can now serve as candidates for <strong>cross-validation</strong>, <strong>final tuning</strong>, or <strong>practical deployment</strong>.</p><h3>🩺 3.7 Estimating Heart Disease Risk for a New Patient</h3><p>With the best model selected, we now move to the <strong>real-world prediction task</strong>: estimating the probability that a new patient will develop heart disease based on their clinical characteristics.</p><p><strong>Process steps:</strong></p><ol><li><strong>Patient creation:</strong><br> We define a DataFrame containing the variables that represent the clinical profile of a new patient (df_patient).</li><li><strong>Feature preparation:</strong><br> We apply the same data transformations used during training:</li></ol><ul><li>Encoding of categorical variables using x_cat_enc.categories_;</li><li>Normalization using the mean, standard deviation, and variance parameters stored in x_scaler.var_.</li></ul><p><strong>3. Patient transformation:</strong><br> We use the function preprocess_x(df_patient) to ensure that the data is properly formatted for model input.</p><p><strong>4. Model selection:</strong><br> We select the best-performing model based on test performance (<strong>AUC = 0.926</strong>), as identified in the previous analysis stage.</p><p><strong>5. Prediction:</strong><br> Finally, we use selected_model.predict(x_patient) to obtain the predicted probability of heart disease risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*Aq4rndlyck7fbLDEdb5KsA.png" /><figcaption>Figure 9 — Probability of risk of heart disease</figcaption></figure><h3>📌 Interpretation</h3><p>The model estimates that this patient has approximately a <strong>49% probability</strong> of belonging to the heart disease risk class, given the provided set of clinical variables.</p><p>This value represents an <strong>intermediate risk</strong>, reinforcing the importance of complementary medical evaluation and clinical follow-up. The model should be understood as a <strong>decision-support tool</strong>, not as a definitive diagnostic instrument.</p><h3>🔚 4. Conclusion</h3><p>During the re-execution of this project for publication purposes, variations in AUC values were observed compared to previous runs. This instability is directly related to the <strong>stochastic nature of neural network training</strong>, especially when random seeds are not explicitly fixed.</p><p>To ensure <strong>reproducibility and consistency across executions</strong> — including between the Portuguese and English versions of this project — the practice of fixing a <strong>global experiment seed</strong> was adopted. As a result, the outcomes became deterministic, enabling reliable comparisons and more rigorous publication of results.</p><p>This highlights the importance of <strong>good machine learning practices</strong>, such as controlling randomness, documenting the environment, and maintaining methodological transparency. More than the final metrics themselves, the primary value of this project lies in the <strong>process</strong>, the <strong>learnings</strong>, and the construction of a <strong>reproducible and interpretable pipeline</strong>. 🚀</p><p>Check out my full projects on <a href="https://github.com/rodrigohigashi/">GitHub</a> or explore more articles in <a href="https://www.datascienceportfol.io/rockigo">my portfolio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5c1637b185b" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/what-if-a-neural-network-could-diagnose-heart-disease-c5c1637b185b">What If a Neural Network Could Diagnose Heart Disease?</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From a Real-World Problem to a Python-Powered Solution: Converting PDFs to Word Easily and for Free]]></title>
            <link>https://blog.venturemagazine.net/from-a-real-world-problem-to-a-python-powered-solution-converting-pdfs-to-word-easily-and-for-free-4d67a8ad1ecb?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d67a8ad1ecb</guid>
            <category><![CDATA[words]]></category>
            <category><![CDATA[pdf-converter]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 14:16:50 GMT</pubDate>
            <atom:updated>2025-07-30T05:40:23.337Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/@rockigo/de-um-problema-real-a-uma-solu%C3%A7%C3%A3o-com-python-convertendo-pdfs-para-word-de-forma-pr%C3%A1tica-e-88c39871fe38">here</a>.</p><p>Recently, I came across a very common situation in hospitals, clinics, and medical research environments: the need to convert PDF files into Word documents to allow easier editing, note-taking, or content reuse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*0wk1_EXIVseCDqXtqBsnjQ.jpeg" /><figcaption>Words contained in a PDF being extracted to Word — image generated with Sora, OpenAI’s AI tool.</figcaption></figure><h3>🧠 1. Introduction: Where the Idea Came From</h3><p>During a visit to an university hospital, while talking to doctors on the team, I realized a real pain point in their daily routine: the difficulty of extracting information from PDF files and transferring it into the hospital’s internal system.</p><p>What seems simple at first — just copying and pasting text — turned out to be time-consuming, error-prone, and frustratingly repetitive. For healthcare professionals, who deal with tight schedules and heavy workloads, this type of obstacle is far more common than it should be.</p><p>It’s also important to highlight that while PDFs are widely used, they were never designed for easy data extraction. In fact, the format was created to preserve layout and formatting across different devices — which makes the file structure inherently complex. This means the text you see on the screen isn’t always straightforward for a system to read or copy. You may run into issues like misplaced line breaks, jumbled reading order, or, in some cases, you can’t even select the text — especially when the file is actually a scanned image.</p><p>That’s when the idea came to life: to automate this process through a simple app — a PDF-to-Word converter accessible via any web browser, with no installation required.</p><h3>🧪 2. Where I Learned the Technical Side</h3><p>I was first introduced to PDF manipulation during a class from Asimov Academy’s Artificial Intelligence course — specifically in the module <em>Exploring the OpenAI API</em>. At the time, I thought that kind of approach could be the ideal way to solve the problem.</p><p>However, I later came across another course on the same platform called <em>Reading and Manipulating PDFs</em>, which taught exactly what I needed: how to extract data from PDF files in a direct and efficient way using Python.</p><p>On top of that, I had already learned how to use Streamlit through another Asimov course. With that knowledge — and a little help from ChatGPT to fine-tune parts of the code — I was able to build the first functional version of the project.</p><p>The main libraries used were:</p><ul><li>pypdf — for reading and extracting text from PDF files;</li><li>streamlit — to create the app’s web interface;</li><li>pathlib and io — for handling file paths and data streams.</li></ul><h3>🛠️ 3. Why Build a Custom Solution?</h3><p>Although there are many websites today that offer PDF-to-Word or PDF-to-text conversion, most of these solutions come with important limitations: some restrict the number of files you can convert before charging; others require user registration upfront; and some insert watermarks in the final file — which can compromise the document’s presentation or its professional use.</p><p>In hospital, academic, or legal contexts, these barriers make practical adoption difficult on a daily basis. Building a custom solution means not having to rely on external platforms, avoiding the risks of exposing sensitive data, and gaining full control over the text extraction process.</p><p>Moreover, the application can be adapted and improved according to the user’s needs — something that’s impossible when you depend on a closed, ready-made tool.</p><h3>❌ 4. Current Limitations</h3><p>Although functional and easy to use, the app has some limitations worth highlighting:</p><ul><li>Does not convert PDFs that are images: if the PDF was scanned or generated as an image (without extractable text), the app cannot perform a proper conversion. In these cases, OCR (Optical Character Recognition) techniques, such as Tesseract, would be required.</li><li>Formatting may not be 100% preserved: since the focus was on quickly converting the textual content, the PDF’s visual structure (such as columns, tables, bold text, etc.) may be lost during conversion.</li><li>Conversion limited to one file at a time: the app does not yet support batch processing of multiple PDFs, which would be an interesting improvement for the future.</li></ul><h3>💡 5. Potential Applications</h3><p>Beyond use by doctors and other healthcare professionals, this app can help:</p><ul><li>Students who receive materials in PDF format and want to take notes;</li><li>Legal professionals who need to adapt documents;</li><li>Anyone who receives scanned files and wants to reuse the textual content;</li><li>Anyone who needs to copy excerpts from PDFs without having to open Word and do it manually.</li><li>Want to try it yourself?<br> 🔗 <a href="https://extractor-text-pdf-en.streamlit.app/">Click here to access the app</a></li><li>🎥 <strong>Quick demo (45s)</strong><br> Here’s a short video showing how the app works:</li><li>[<a href="https://www.loom.com/share/c5e40884aec343bb8c5603fd5966e8d9">Embed do vídeo Loom here</a>]</li></ul><h3>✅ 6. Conclusion</h3><p>Creating this app was more than just a technical exercise. It was a practical response to a real problem observed in the field. Seeing the sparkle in the eyes of those who used it and found it helpful is the kind of feedback that means more than any code metrics.<br> If you also found it useful, share it with someone who might need it. And if you want to offer suggestions or propose improvements (such as scanned PDF conversion or multiple file processing), I’m open to feedback!</p><p>Check out my complete projects on <a href="https://github.com/rodrigohigashi">GitHub</a> or see more articles in <a href="https://www.datascienceportfol.io/rockigo">my portfolio</a>.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏️<strong>️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d67a8ad1ecb" width="1" height="1" alt=""><hr><p><a href="https://blog.venturemagazine.net/from-a-real-world-problem-to-a-python-powered-solution-converting-pdfs-to-word-easily-and-for-free-4d67a8ad1ecb">From a Real-World Problem to a Python-Powered Solution: Converting PDFs to Word Easily and for Free</a> was originally published in <a href="https://blog.venturemagazine.net/">Venture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[De um problema real a uma solução com Python: convertendo PDFs para Word de forma prática e…]]></title>
            <link>https://medium.com/@rockigo/de-um-problema-real-a-uma-solu%C3%A7%C3%A3o-com-python-convertendo-pdfs-para-word-de-forma-pr%C3%A1tica-e-88c39871fe38?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/88c39871fe38</guid>
            <category><![CDATA[pdf-converter]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[words]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 14:15:54 GMT</pubDate>
            <atom:updated>2025-07-29T20:15:05.680Z</atom:updated>
            <content:encoded><![CDATA[<h3>De um problema real a uma solução com Python: convertendo PDFs para Word de forma prática e gratuita</h3><p>Este artigo também está disponível em inglês <a href="https://medium.com/@rockigo/from-a-real-world-problem-to-a-python-powered-solution-converting-pdfs-to-word-easily-and-for-free-4d67a8ad1ecb">aqui</a>.</p><p>Recentemente, me deparei com uma situação bastante comum em hospitais, clínicas e ambientes de pesquisa médica: a necessidade de <strong>converter arquivos PDF para Word</strong> para facilitar edições, anotações ou reaproveitamento de conteúdo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wk1_EXIVseCDqXtqBsnjQ.jpeg" /><figcaption>Palavras contidas num PDF sendo extraídas para Word— imagem gerada com Sora, ferramenta de IA da OpenAI.</figcaption></figure><h3>🧠 1. Introdução: de onde surgiu a ideia</h3><p>Durante uma visita a um hospital universitário, conversando com médicas da equipe, percebi uma dor real no dia a dia delas: a dificuldade de coletar informações contidas em arquivos PDF e transferi-las para o sistema interno do hospital.</p><p>Algo que, à primeira vista, parece simples — copiar e colar o conteúdo — na prática se mostrava moroso, impreciso e recorrente. Para profissionais da saúde, que lidam com tempo escasso e alta demanda, esse tipo de obstáculo acaba sendo um problema mais comum do que parece.</p><p>Além disso, vale lembrar que o formato PDF, por mais comum que seja, não foi feito para facilitar a extração de dados. Ele foi pensado para preservar o layout e a formatação dos documentos em diferentes dispositivos, o que acaba tornando a estrutura interna do arquivo bastante complexa. Ou seja: nem sempre o texto que vemos visualmente no PDF está disponível de forma direta para ser copiado por um sistema. Isso pode gerar problemas como quebra de linha fora de lugar, ordem confusa de leitura ou, em alguns casos, simplesmente não permitir selecionar o texto — especialmente quando o arquivo é, na verdade, uma imagem digitalizada.</p><p>Foi aí que surgiu a ideia de automatizar essa tarefa com um app simples: um conversor de PDF para Word, acessível por qualquer navegador, sem necessidade de instalação.</p><h3>🧪 2. Onde aprendi a parte técnica</h3><p>Inicialmente, tive contato com manipulação de PDFs por meio de uma aula da <strong>Asimov Academy</strong> no curso de Inteligência Artificial, especificamente na parte de “<strong>Explorando a API da OpenAI</strong>”. Na época, pensei que esse tipo de abordagem seria o ideal para resolver o problema.</p><p>No entanto, acabei encontrando outro curso dentro da própria Asimov chamado <strong>“Lendo e Manipulando PDFs”</strong>, que ensinava exatamente o que eu precisava: como extrair dados de arquivos PDF de forma direta e eficiente com Python.</p><p>Além disso, eu já havia aprendido como usar o <strong>Streamlit</strong> em outro curso da <strong>Asimov</strong>. Com esse conhecimento, e com uma forcinha do <strong>ChatGPT</strong> para ajustar partes do código, consegui montar a primeira versão funcional do projeto.</p><p>As principais bibliotecas utilizadas foram:</p><ul><li>pypdf – para leitura e extração de texto do PDF;</li><li>streamlit – para criação da interface web do app;</li><li>pathlib e io – para manipulação de arquivos e streams de dados.</li></ul><h3>🛠️3. Por que criar uma solução própria?</h3><p>Embora existam hoje diversos sites que oferecem a conversão de arquivos PDF para Word ou texto, muitas dessas soluções têm restrições importantes: algumas impõem um limite na quantidade de arquivos convertidos e a partir daí passam a cobrar; outras exigem que o usuário se cadastre antes de usar, e há ainda aquelas que inserem marca d’água no arquivo final — o que pode comprometer a apresentação ou uso profissional do documento.</p><p>No contexto hospitalar, ou mesmo acadêmico e jurídico, essas barreiras dificultam a adoção prática no dia a dia. Criar uma solução própria significa, portanto, não depender de plataformas externas, não correr riscos com privacidade dos dados sensíveis e ter total controle sobre o processo de extração de texto.</p><p>Além disso, a aplicação pode ser adaptada e evoluída conforme a necessidade do usuário, algo impossível de se fazer quando se depende de uma ferramenta pronta e fechada.</p><h3>❌ 4. Limitações atuais</h3><p>Apesar de funcional e simples de usar, o app tem algumas limitações que merecem ser destacadas:</p><ul><li><strong>Não converte PDFs que são imagens</strong>: se o PDF foi escaneado ou gerado como uma imagem (sem texto extraível), o app não consegue fazer a conversão correta. Para esses casos, seria necessário aplicar técnicas de OCR (Reconhecimento Óptico de Caracteres), como o Tesseract.</li><li><strong>Formatação pode não ser preservada 100%</strong>: como o foco foi converter rapidamente o conteúdo textual, a estrutura visual do PDF (como colunas, tabelas, negrito, etc.) pode se perder na conversão.</li><li><strong>Conversão limitada a um arquivo por vez</strong>: o app ainda não aceita múltiplos PDFs em lote, o que seria uma melhoria interessante para o futuro.</li></ul><h3>💡 5. Possíveis aplicações</h3><p>Além do uso por médicas e outros profissionais de saúde, esse app pode ajudar:</p><ul><li>Estudantes que recebem materiais em PDF e querem fazer anotações;</li><li>Profissionais do direito que precisam adaptar documentos;</li><li>Quem recebe arquivos escaneados e quer reaproveitar o conteúdo textual;</li><li>Qualquer pessoa que precise copiar trechos de PDF sem precisar abrir o Word e fazer isso manualmente.</li><li>Quer testar por conta própria?<br> 🔗 <a href="https://extractor-text-pdf-pt.streamlit.app/">Clique aqui para acessar o app</a></li><li>🎥 <strong>Vídeo explicativo (45s)</strong><br> Veja como o app funciona na prática:</li><li>[<a href="https://www.loom.com/share/079471e18ebc4d1fb09221d0a9a44e41">Embed do vídeo Loom aqui</a>]</li></ul><h3>✅ 6. Conclusão</h3><p>Criar esse app foi mais do que um exercício técnico. Foi uma resposta prática a um problema real, observado em campo. Ver o brilho nos olhos de quem usou e achou útil é o tipo de feedback que vale mais que qualquer métrica de código.</p><p>Se você também achou útil, compartilha com alguém que precise. E se quiser dar sugestões ou propor melhorias (como conversão de PDFs escaneados ou múltiplos arquivos), estou aberto a feedbacks!</p><p>Confira meus projetos completos no <a href="https://github.com/rodrigohigashi">GitHub</a> ou veja mais artigos no <a href="https://www.datascienceportfol.io/rockigo">meu portfólio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88c39871fe38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[E Se Uma Rede Neural Pudesse Diagnosticar Doença Cardíaca?]]></title>
            <link>https://medium.com/@rockigo/e-se-uma-rede-neural-pudesse-diagnosticar-doen%C3%A7a-card%C3%ADaca-d4bf4b14c073?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4bf4b14c073</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning-python]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 20:48:07 GMT</pubDate>
            <atom:updated>2025-12-28T14:49:01.916Z</atom:updated>
            <content:encoded><![CDATA[<p>Este artigo também está disponível em inglês <a href="https://medium.com/@rockigo/what-if-a-neural-network-could-diagnose-heart-disease-c5c1637b185b?postPublishedType=initial">aqui</a>.</p><p>A inteligência artificial tem se mostrado uma aliada poderosa na área da saúde, e este projeto é um exemplo prático disso. Aqui, aplico redes neurais artificiais para prever a presença de doenças cardíacas com base em variáveis clínicas de diversos pacientes. A base de dados utilizada contém informações como idade, sexo, pressão arterial, níveis de colesterol, entre outras características. O objetivo é treinar um modelo que, a partir desses dados, seja capaz de identificar se uma pessoa tem ou não risco de apresentar uma doença cardíaca — uma tarefa traduzida em uma variável binária: 1 para presença de doença, 0 para ausência.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*qygCrS_VVd12EphaYe6bkA.jpeg" /><figcaption>Imagem gerada por IA ilustrando o sistema cardiovascular humano.</figcaption></figure><h3>1 Entendimento do Negócio</h3><p><strong>1.1 Declaração do Problema</strong></p><p>As doenças cardiovasculares continuam sendo uma das principais causas de morte no mundo, segundo a Organização Mundial da Saúde. Diagnosticar esses problemas de forma precoce é fundamental para aumentar as chances de tratamento e sobrevivência. A cardiologia, ramo da medicina dedicado ao estudo e cuidado do coração, depende muitas vezes de exames clínicos, histórico médico e avaliação de sintomas — o que pode ser subjetivo ou demorado. Diante desse cenário, surge a proposta de usar inteligência artificial para apoiar o diagnóstico, oferecendo uma segunda opinião baseada em padrões estatísticos aprendidos a partir de dados reais de pacientes. O objetivo deste projeto é desenvolver um modelo preditivo capaz de identificar, com base em variáveis clínicas, se um paciente tem maior ou menor risco de apresentar uma doença cardíaca.</p><p>Esse tipo de estudo pode ter aplicações práticas em diversos contextos como o diagnóstico de diabetes, doenças renais, doenças hepáticas, COVID, depressão ou transtornos mentais — além de cenários fora da saúde, como transações de fraudes bancárias ou detecções de spams.</p><p><strong>1.2 Objetivo</strong></p><p>O principal objetivo deste estudo é construir um modelo preditivo capaz de classificar automaticamente se um paciente apresenta ou não doença cardíaca, com base em suas características. A própria base de dados, extraída do Kaggle, já inclui a variável <em>target</em>, onde o valor 1 indica presença da doença, enquanto o valor 0 representa ausência.</p><p>Além de prever a presença ou ausência da doença, essa análise também busca compreender quais características têm maior influência sobre o diagnóstico, como o tipo de dor no peito, a pressão arterial em repouso, os níveis de colesterol, ou um índice glicêmico elevado. Embora o foco principal esteja na aplicação de redes neurais, entender os fatores que mais pesam na decisão do modelo também é um ponto relevante para aplicações práticas na área da saúde.</p><h3>2 Entendimento dos Dados</h3><p><strong>2.1 Descrição dos dados e coleta</strong></p><p>Este estudo utiliza um conjunto de dados extraído do <a href="https://www.kaggle.com/">Kaggle</a>, conhecido como <strong>Heart Disease UCI</strong>, originalmente compilado em 1988. O dataset é composto por registros provenientes de quatro centros médicos diferentes: Cleveland, Hungria, Suíça e Long Beach V. Ao todo, ele contém <strong>76 atributos</strong>, mas, na prática, os experimentos e análises publicados utilizam um <strong>subconjunto de 14 colunas</strong>, incluindo a variável alvo.</p><p>A variável target representa a presença de doença cardíaca, sendo um campo binário onde:</p><ul><li>0 indica ausência da doença,</li><li>1 indica presença da doença.</li></ul><p>Abaixo, estão descritas as variáveis utilizadas neste estudo, com suas respectivas descrições e tipos:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*ZCJ45Az5XPnh-52Uon8fag.png" /><figcaption>Tabela 1 — Metadados</figcaption></figure><p>2.2 <strong>Análise exploratória</strong></p><p><strong>2.2.1 Relatório Sweetviz</strong></p><p>Antes de treinar a rede neural, explorei a base com o Sweetviz pra entender quais variáveis mais se associam ao diagnóstico de doença cardíaca. Variáveis como <strong>idade (~60 anos)</strong>, <strong>tipo de dor no peito = 4</strong>, <strong>pressão arterial alta (~180 mmHg)</strong> e <strong>frequência cardíaca baixa (~90 bpm)</strong> se destacaram com mais de <strong>40% a 60%</strong> de positividade, ajudando a direcionar a modelagem.</p><p><strong>2.3 Treinando o modelo</strong></p><p>Na etapa de preparação dos dados, separei as variáveis categóricas e numéricas, transformando as categóricas com <strong>OneHotEncoder</strong> para gerar variáveis dummies. Após o tratamento, uni todas as features transformadas e normalizadas com <strong>StandardScaler</strong>, deixando tudo pronto para alimentar a RNA.</p><h3>🤖 3. Modelagem com Redes Neurais Artificiais</h3><p>Com os dados preparados, a base foi dividida entre treino e teste (<strong>70/30</strong>). A padronização dos dados foi feita <strong>apenas no conjunto de treino</strong>, evitando vazamento de informação e mantendo a integridade do teste. Para garantir reprodutibilidade dos resultados, foi utilizada uma seed fixa durante o treinamento da rede neural.</p><p>A rede neural construída segue uma arquitetura do tipo <strong>MLP (Perceptron Multicamadas)</strong> com duas camadas ocultas:</p><ul><li><strong>Camada oculta 1</strong>: 4 neurônios, ativação <strong>tanh</strong>, regularização <strong>L1=0.01 e L2=0.01</strong>, input_dim igual ao número de variáveis de entrada e <strong>dropout de 0.2</strong> para reduzir overfitting.</li><li><strong>Camada oculta 2</strong>: 8 neurônios, ativação <strong>ReLU</strong>, com as mesmas regularizações da camada anterior e <strong>dropout de 0.1</strong>.</li><li><strong>Camada de saída</strong>: 1 neurônio com ativação <strong>sigmoid</strong>, gerando uma probabilidade final de diagnóstico.</li></ul><p>O modelo foi compilado com <strong>função de perda binária (binary crossentropy)</strong>, otimizador <strong>Adam</strong> e <strong>AUC como métrica principal</strong> — já que o foco é medir a capacidade do modelo em distinguir entre os dois grupos (com ou sem doença cardíaca).</p><h4>🧠 3.1 Estrutura e Avaliação Inicial da RNA</h4><p>O modelo foi construído com a API <strong>Sequential</strong> do Keras, empilhando camadas de forma linear. A arquitetura resultante possui apenas <strong>161 parâmetros treináveis</strong>, tornando-se uma rede leve e adequada para o volume de dados.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*3fFv6OIjyTUoRzp_Qd7DEg.png" /><figcaption>Tabela 2 — Estrutura das Redes Neurais</figcaption></figure><p>Antes de qualquer ajuste mais fino nos hiperparâmetros, o modelo apresentou desempenho insatisfatório, com valores de AUC significativamente abaixo do esperado para um classificador binário. No conjunto de treino, a AUC foi de 23,7%, enquanto no conjunto de teste o valor caiu para 16,4%. Esses resultados indicam que, nessa configuração inicial, a rede neural não conseguiu capturar padrões relevantes nos dados, apresentando um comportamento inferior ao de um classificador binário. Esse cenário reforça a importância da etapa de ajuste de arquitetura e hiperparâmetros para melhorar a capacidade de generalização do modelo.</p><h4>🔁3.2 Treinamento da Rede Neural</h4><p>O treinamento da RNA foi realizado com <strong>50 épocas</strong> e <strong>batch size = 8</strong>, utilizando <strong>10% dos dados de treino para validação</strong>. Para evitar overfitting e acelerar o processo, foi aplicado <strong>early stopping</strong> monitorando a métrica <strong>AUC na validação</strong>, com os seguintes critérios:</p><ul><li>patience = 20: se o AUC não melhorar por 20 épocas consecutivas, o treinamento é interrompido.</li><li>min_delta = 0.01: é necessário um ganho mínimo de 1% na AUC para ser considerado melhoria.</li><li>restore_best_weights = True: ao final, o modelo retorna automaticamente aos melhores pesos obtidos durante o treinamento.</li></ul><h3>📈 3.3 Desempenho (final) da Rede Neural</h3><p>Com o ajuste de hiperparâmetros e o <strong>early stopping</strong> em ação, o modelo atingiu um salto de performance notável:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*Y2_mJ2B6PlGCr2otXfJiEw.png" /><figcaption>Tabela 3 — Avaliação Baseline e após ajuste das redes neuraisi (métrica AUC)</figcaption></figure><p>Esses resultados indicam que:</p><ul><li>A combinação de regularização (L1/L2) e dropout contribuiu para controlar o overfitting, mesmo com o aumento da complexidade do modelo em relação à configuração inicial.</li><li>O uso de early stopping permitiu preservar os melhores pesos antes que a perda de generalização se manifestasse.</li><li>O modelo aprendeu padrões relevantes nos dados, e não apenas os exemplos de treino, como evidenciado pela pequena diferença entre as métricas de treino e teste (aproximadamente 2 pontos percentuais após o ajuste).</li></ul><p>Na prática, isso significa que a RNA consegue distinguir pacientes com e sem doença cardíaca em mais de 91% das vezes, um desempenho robusto e comparável a técnicas tradicionais de machine learning aplicadas a esse dataset.</p><p><strong>O gráfico de AUC por época mostra que o modelo aprende de forma estável e generaliza bem:</strong></p><ul><li>A curva de validação se mantém consistentemente acima da curva de treino, indicando boa capacidade de generalização.</li><li>O early stopping interrompe o treinamento no momento ideal, evitando overfitting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*YWnwh0ZFz2bb_G3NVnZGRQ.png" /><figcaption>Figura 2 — Desempenho do modelo por épocas</figcaption></figure><h3>🔄 3.4 Função de Treinamento &amp; Busca de Hiperparâmetros</h3><p>Para otimizar a performance da rede neural, criamos uma função que define, compila e treina o modelo com base em determinados hiperparâmetros: número de neurônios nas camadas ocultas e taxa de dropout. O modelo é avaliado pelo AUC tanto no conjunto de treino quanto no conjunto de teste.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*1mHnC8qE0YS5PCpU3889Gw.png" /><figcaption>Figura 3 — Função de Treinamento de RNA</figcaption></figure><p>Com essa função, executamos uma busca simples em uma grade de hiperparâmetros (número de neurônios entre 1 e 4 e dropout entre 0 e 0.2), treinando 50 modelos diferentes para escolher o melhor com base no AUC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*L0xnw6M8wO9c1ZrLWvBtkA.png" /><figcaption>Figura 4 — Gerando múltiplos modelos de RNA com diferentes combinações de hiperparâmetros</figcaption></figure><p>Com todos os modelos treinados, consolidamos os resultados em um DataFrame. A coluna diferença nos ajuda a distinguir modelos com alta AUC no teste, mas que não ‘decoraram’ o conjunto de treino.</p><h3>🔍 3.5 Análise dos Hiperparâmetros</h3><p>Para entender o impacto de cada hiperparâmetro sobre o desempenho da RNA, geramos gráficos que mostram o comportamento do AUC (em treino e teste) conforme variamos:</p><ul><li>O número de neurônios na primeira camada oculta (C1),</li><li>O número de neurônios na segunda camada oculta (C2),</li><li>A taxa de dropout na primeira camada oculta (Dropout C1).</li></ul><p>Além disso, analisamos o balanço entre desempenho e generalização com base na diferença entre AUC de treino e teste.</p><p>📈<strong> 3.5.1 Número de Neurônios na Camada Oculta 1 (C1)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*--IaDEHIHO7Wf8eCYR9K2w.png" /><figcaption>Figura 5 — Número de Neurônios na Camada Oculta 1 (C1)</figcaption></figure><ul><li>No gráfico da esquerda, observa-se um comportamento <strong>monotonicamente crescente</strong> do AUC à medida que aumentamos o número de neurônios na C1. Diferentemente da análise anterior, não há oscilações intermediárias: cada incremento no número de neurônios resulta em melhora consistente tanto no desempenho em treino quanto em teste. Além disso, as curvas permanecem muito próximas entre si ao longo de todo o intervalo, indicando estabilidade do aprendizado.</li><li>No gráfico da direita, a diferença entre AUC de treino e teste <strong>diminui progressivamente</strong> conforme o número de neurônios aumenta, sugerindo que arquiteturas ligeiramente mais complexas conseguem capturar melhor os padrões dos dados sem comprometer a generalização.</li></ul><p>📌 <strong>Interpretação</strong>: o aumento do número de neurônios na C1 contribui de forma consistente para o desempenho do modelo e, simultaneamente, reduz a discrepância entre treino e teste, reforçando um cenário de boa generalização sob regularização ativa.</p><p>📈<strong> 3.5.2. Número de Neurônios na Camada Oculta 2 (C2)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*lL-5yJV0drdrjrB076N3JQ.png" /><figcaption>Figura 6 — Número de Neurônios na Camada Oculta 2 (C2)</figcaption></figure><ul><li>No gráfico da esquerda, o desempenho se mantém relativamente estável entre 1 e 2 neurônios na C2. A partir desse ponto, observa-se uma <strong>tendência clara de crescimento do AUC</strong> tanto em treino quanto em teste conforme aumentamos o número de neurônios, indicando que arquiteturas um pouco mais profundas passam a extrair melhor informações adicionais dos dados.</li><li>O gráfico da direita mostra que a diferença entre AUC de treino e teste permanece pequena ao longo de toda a faixa analisada, com leves oscilações. Ainda assim, os valores se mantêm dentro de uma zona considerada segura, sem indícios relevantes de overfitting.</li></ul><p>📌 <strong>Interpretação</strong>: a C2 apresenta menor sensibilidade para valores muito baixos de neurônios, mas passa a contribuir positivamente para o desempenho quando sua capacidade é aumentada. As variações observadas não comprometem a generalização do modelo.</p><p>📉 <strong>3.5.3 Taxa de Dropout na Camada Oculta 1</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*E1m6DEu78-tAoCcm5vSVlA.png" /><figcaption>Figura 7 — Taxa de Dropout na Camada Oculta 1</figcaption></figure><ul><li>No gráfico da esquerda, nota-se que o AUC de treino apresenta um <strong>leve pico em torno de dropout = 0.1</strong>, enquanto o desempenho em teste também atinge seu melhor valor nesse ponto. Para dropout = 0.2, observa-se uma queda em ambas as curvas, sugerindo que o excesso de regularização começa a prejudicar a capacidade de aprendizado do modelo.</li><li>O gráfico da direita reforça essa leitura: a diferença entre AUC de treino e teste é menor para dropout = 0.1, aumentando novamente quando não há regularização (dropout = 0) ou quando ela é excessiva (dropout = 0.2).</li></ul><p>📌 <strong>Interpretação</strong>: uma taxa de dropout moderada (em torno de 10%) oferece o melhor equilíbrio entre aprendizado e regularização, reduzindo a discrepância entre treino e teste sem limitar excessivamente a capacidade da rede.</p><p>📊 3.5.4 <strong>Análise da relação entre desempenho e generalização</strong></p><p>O gráfico de dispersão abaixo relaciona o AUC no conjunto de teste com a diferença entre AUC de treino e teste para todos os 48 modelos avaliados. As linhas horizontais em -0.05, 0 e +0.05 delimitam a faixa considerada aceitável de generalização.</p><p>Observa-se que a maioria dos modelos se concentra dentro desse intervalo, enquanto alguns apresentam diferenças mais extremas, indicando possível overfitting (diferença positiva elevada) ou underfitting (diferença negativa acentuada). Essa visualização reforça o critério adotado na seleção dos modelos mais robustos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*67-xlihDByaJ_uwvouwHIg.png" /><figcaption>Figura 8— Gráfico de Dispersão entre Diferença e AUC Test</figcaption></figure><p>💡 <strong>Nota sobre a visualização dos resultados</strong></p><p>Em alguns gráficos desta seção, especialmente na análise do dropout, utilizamos duas representações complementares no eixo da diferença entre AUC de treino e teste.</p><p>A linha contínua representa a <strong>média da diferença treino–teste</strong> para cada valor de hiperparâmetro avaliado, oferecendo uma visão agregada do comportamento do modelo.<br> Já os pontos individuais correspondem a <strong>cada modelo treinado</strong>, relacionando seu AUC de teste com a respectiva diferença entre treino e teste.</p><p>Essa abordagem combinada permite analisar simultaneamente tendências globais e a dispersão dos resultados, fornecendo uma visão mais completa do equilíbrio entre desempenho e generalização da rede neural.</p><h3>🎯3.6 Seleção do Melhor Modelo</h3><p>Após testar 48 combinações diferentes de hiperparâmetros, aplicamos um filtro no dataframe para selecionar apenas os modelos que apresentaram:</p><ul><li><strong>Boa generalização</strong>, com diferença entre AUC de treino e teste entre -0.05 e +0.05;</li><li><strong>Excelente desempenho em teste</strong>, com AUC superior a 0.88.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*yWyK5eJuOtBCOprPnMNshg.png" /><figcaption>Tabela 4 — Dataframe filtrado</figcaption></figure><p>Com base nesses resultados, podemos agora identificar um conjunto reduzido de modelos promissores, servindo como candidatos para validação cruzada, ajustes finais ou aplicações práticas.</p><h3>🩺 3.7 Estimando o Risco de Doença Cardíaca para um Novo Paciente</h3><p>Com o melhor modelo selecionado, partimos agora para a <strong>predição real</strong>: estimar a probabilidade de um novo paciente desenvolver doença cardíaca com base em suas características.</p><h4>Etapas do processo:</h4><ol><li><strong>Criação do paciente:</strong><br> Definimos um dataframe com as variáveis que representam o perfil clínico de um novo paciente (df_paciente).</li><li><strong>Preparação das variáveis:</strong><br> Aplicamos a mesma transformação de dados que foi feita no treino:</li></ol><ul><li>Encoding das variáveis categóricas com x_cat_enc.categories_</li><li>Normalização com os parâmetros de média, desvio padrão e variância de x_scaler.var_</li></ul><p><strong>3. Transformação do paciente:</strong><br> Utilizamos a função prepara_x(df_paciente) para garantir que os dados estejam no formato correto de entrada para o modelo.</p><p><strong>4. Escolha do modelo:</strong><br> Selecionamos o melhor modelo com base no desempenho em teste (AUC = 0.926), encontrado na etapa anterior da análise.</p><p><strong>5. Predição:</strong><br> Finalmente, utilizamos modelo_selecionado.predict(x_paciente) para obter a <strong>probabilidade de risco de doença cardíaca</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*Rxm3vNtxH7OtCJjI4v5PKw.png" /><figcaption>Figura 9— Probabilidade de risco de doença cardíaca</figcaption></figure><p>📌 <strong>Interpretação</strong>:<br>O modelo estima que esse paciente possui aproximadamente <strong>49% de probabilidade</strong> de pertencer à classe de risco para doença cardíaca, considerando o conjunto de variáveis clínicas fornecidas.</p><p>Esse valor indica um <strong>risco intermediário</strong>, o que reforça a importância de avaliação médica complementar e acompanhamento clínico, já que o modelo atua como uma ferramenta de apoio à decisão — e não como diagnóstico definitivo.</p><h3>🔚 4. Conclusão</h3><p>Durante a reexecução do projeto para fins de publicação, observei variações nos valores de AUC em comparação com execuções anteriores. Essa instabilidade está diretamente relacionada ao caráter estocástico do treinamento de redes neurais, especialmente quando seeds aleatórias não são explicitamente fixadas.</p><p>Para garantir reprodutibilidade e consistência entre execuções — inclusive entre as versões em português e inglês deste projeto — foi adotada a prática de <strong>fixar a seed global do experimento</strong>. Com isso, os resultados passaram a ser determinísticos, permitindo comparações confiáveis e publicação dos resultados com maior rigor técnico.</p><p>Esse ponto reforça a importância de boas práticas em machine learning, como controle de aleatoriedade, documentação do ambiente e transparência metodológica. Mais do que os números finais, o principal valor deste projeto está no processo, nos aprendizados e na construção de um pipeline reprodutível e interpretável. 🚀</p><p>Confira meus projetos completos no <a href="https://github.com/rodrigohigashi">GitHub</a> ou veja mais artigos no <a href="https://www.datascienceportfol.io/rockigo">meu portfólio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4bf4b14c073" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Transcribed Voice into Text Using AI: My Project with OpenAI Whisper]]></title>
            <link>https://ai.plainenglish.io/turning-voice-into-text-with-ai-my-transcription-project-with-openai-whisper-ac9718db50d2?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac9718db50d2</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transcription]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 19:13:50 GMT</pubDate>
            <atom:updated>2025-07-08T14:31:50.952Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/@rockigo/transformando-voz-em-texto-com-ia-meu-projeto-de-transcri%C3%A7%C3%A3o-com-openai-whisper-f66758d80bd0">here</a>.</p><p>A simple and powerful solution with Python, .txt, .csv, timestamp and accessibility</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/0*QQihiwGSakJUm0e3.jpeg" /><figcaption>Musical notes turning into text — image generated with Sora, OpenAI’s AI tool.</figcaption></figure><h3><strong>1. introduction: where the idea came from</strong></h3><p>I’ve always enjoyed studying languages. And during one of those study sessions, I realized that some audio files didn’t have transcripts available — which made it hard for me to follow along, review, or even understand certain parts.<br> That’s when the idea came to me: why not create a simple AI tool that automatically transcribes audio with timestamps?</p><h3>2. Where I learned the technical part</h3><p>The foundation for this came from the course <em>“Exploring the OpenAI API”</em> on the <strong>Asimov Academy</strong> platform.<br> There, I learned about OpenAI’s <strong>Whisper</strong> model, which allows you to transcribe audio with just a few Python commands.<br> From there, I explored the documentation, refined the project with the help of <strong>ChatGPT</strong>, and adapted it into two useful formats: .txt for reading and .csv for data analysis.</p><h3><strong>3. Why two files?</strong></h3><p>The transcript is saved in two formats:<br> .txt → ideal for those who simply want to read the audio content, with timestamps.<br> .csv → allows you to import the data into tools like <strong>Excel, Power BI, or Python</strong>, making it easier to perform sentiment analysis, or work with speeches, lectures, or meetings.</p><h3><strong>4. API limitations</strong></h3><p>Since the model used is <strong>Whisper-1</strong>, there is a limit of <strong>10 minutes of audio or 25MB per file</strong>, according to OpenAI.</p><h3><strong>5. Possible applications</strong></h3><p>This type of project can be useful in many areas:</p><ul><li>Language study</li><li>Accessibility (for the hearing impaired)</li><li>Recording meetings</li><li>Analyzing interviews and testimonials</li><li>Transcribing podcasts, lessons or videos for quick reference</li><li>Creating textual databases for NLP analysis</li></ul><h3><strong>6. Demonstrations (explanatory video)</strong></h3><p>I recorded a video showing how the project works:</p><p>📹 English explanation: [<a href="https://www.loom.com/share/ee946f4abe554280b5520223a2c1d64a">Link EN video</a>]</p><h3><strong>7. Conclusion</strong></h3><p>This was my 10th published project, and the second one with a practical use of AI powered by OpenAI.<br> The idea is simple but powerful: turning a personal challenge into a useful solution — with real potential to help others.<br> Feedback is always welcome! 😊</p><p>Check out my complete projects on <a href="https://github.com/rodrigohigashi">GitHub</a> or see more articles in <a href="https://www.datascienceportfol.io/rockigo">my portfolio</a>.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac9718db50d2" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/turning-voice-into-text-with-ai-my-transcription-project-with-openai-whisper-ac9718db50d2">How I Transcribed Voice into Text Using AI: My Project with OpenAI Whisper</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformando Voz em Texto com IA: Meu Projeto de Transcrição com OpenAI Whisper]]></title>
            <link>https://medium.com/@rockigo/transformando-voz-em-texto-com-ia-meu-projeto-de-transcri%C3%A7%C3%A3o-com-openai-whisper-f66758d80bd0?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f66758d80bd0</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[transcription]]></category>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 18:58:16 GMT</pubDate>
            <atom:updated>2025-07-01T19:15:00.965Z</atom:updated>
            <content:encoded><![CDATA[<p>Este artigo também está disponível em inglês <a href="https://medium.com/@rockigo/turning-voice-into-text-with-ai-my-transcription-project-with-openai-whisper-ac9718db50d2">aqui</a>.</p><p>Uma solução simples e poderosa com Python, .txt, .csv, timestamp e acessibilidade</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*vu4SDiHnuNuBeKPS2p3JmQ.jpeg" /><figcaption>Notas musicais se transformando em texto — imagem gerada com Sora, ferramenta de IA da OpenAI.</figcaption></figure><h3><strong>1. Introdução: de onde surgiu a ideia</strong></h3><p>Sempre gostei de estudar idiomas. E numa dessas sessões de estudo, percebi que alguns áudios não tinham transcrição disponível — o que me dificultava acompanhar, revisar ou mesmo compreender certas partes.<br> Foi aí que me veio a ideia: por que não criar uma <strong>ferramenta simples, com IA</strong>, que transcreva o áudio automaticamente, com timestamps?</p><h3><strong>2. Onde aprendi a parte técnica</strong></h3><p>A base para isso veio do curso <strong>“Explorando a API da OpenAI”</strong> da plataforma <strong>Asimov Academy</strong>.<br> Lá conheci o modelo <strong>Whisper</strong> da OpenAI, que permite transcrever áudios com poucos comandos via Python.<br> A partir disso, explorei a documentação, ajustei o projeto com ajuda do <strong>ChatGPT</strong> e adaptei para dois formatos úteis: .txt para leitura e .csv para análise de dados.</p><h3><strong>3. Por que dois arquivos?</strong></h3><p>A transcrição é salva em dois formatos:</p><ul><li><strong>.txt</strong> → ideal para quem quer simplesmente <strong>ler o conteúdo do áudio</strong>, com marcação de tempo.</li><li><strong>.csv</strong> → permite importar os dados para ferramentas como <strong>Excel, Power BI ou Python</strong>, facilitando análise de <strong>sentimentos, discursos, palestras ou reuniões</strong>.</li></ul><h3><strong>4. Limitações da API</strong></h3><p>Como o modelo usado é o <strong>Whisper-1</strong>, há um limite de <strong>10 minutos de áudio ou 25MB por arquivo</strong>, segundo a própria OpenAI.</p><h3><strong>5. Possíveis aplicações</strong></h3><p>Esse tipo de projeto pode ser útil em várias áreas:</p><ul><li>Estudo de idiomas</li><li>Acessibilidade (para pessoas com deficiência auditiva)</li><li>Gravação de reuniões</li><li>Análise de entrevistas e depoimentos</li><li>Transcrição de podcasts, aulas ou vídeos para consulta rápida</li><li>Criação de bancos de dados textuais para análise de NLP</li></ul><h3><strong>6. Demonstrações (vídeo explicativo)</strong></h3><p>Gravei um vídeo mostrando como o projeto funciona:</p><p>📹 Explicação em português: [<a href="https://www.loom.com/share/f36f128595dc4411b87f5a1a4c2c5d4a">Link do vídeo PT</a>]</p><h3><strong>7. Conclusão</strong></h3><p>Esse foi meu <strong>10º projeto publicado</strong>, sendo o <strong>2º com uso prático de IA aplicada com OpenAI</strong>.<br> A ideia é simples, mas poderosa: transformar um desafio pessoal em uma solução útil — com potencial real de ajudar outras pessoas.</p><p>Feedbacks são sempre bem-vindos! 😊</p><p>Confira meus projetos completos no <a href="https://github.com/rodrigohigashi">GitHub</a> ou veja mais artigos no <a href="https://www.datascienceportfol.io/rockigo">meu portfólio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f66758d80bd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can We Predict Movie Success Before Release? A Data Science Approach ]]></title>
            <link>https://blog.cubed.run/this-article-is-also-available-in-portuguese-here-5d970aed4d7d?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d970aed4d7d</guid>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 20:12:06 GMT</pubDate>
            <atom:updated>2026-04-19T12:20:34.507Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/@rockigo/analisando-filmes-com-adaboost-e-gradient-boosting-prevendo-avalia%C3%A7%C3%B5es-no-imdb-62a06a45a4de">here</a>.</p><p>The following text is about a data science project focused on the entertainment domain, specifically the film industry. I used an IMDB dataset with the goal of predicting movie ratings based on different available variables. To do this, I created a binary target variable to simplify the movie rating. The idea is to apply machine learning techniques — with an emphasis on Ensemble Learning methods — in order to build a robust predictive model capable of automatically classifying movies based on their features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*doj2EZ2xe_s08EpD.png" /><figcaption>IMDb, IMDbPro, STARmeter and all related logos are IMDb.com commercial trademarks, Inc. or affiliates.</figcaption></figure><p><strong>1 Business Understanding</strong><br> <strong>1.1 Problem Statement</strong></p><p>This project is set in the context of the entertainment industry, focusing on analyzing movie ratings using public data from IMDB — one of the largest movie databases in the world. The main objective is to predict, based on available movie attributes (such as genre, runtime, release year, among others), whether a movie is likely to be well-rated or not by the audience. To achieve this, I created a binary target variable, where ‘1’ indicates a well-rated movie and ‘0’ a poorly-rated one.</p><p>This type of prediction can have practical applications in streaming platforms, production companies, or content distributors, helping in decision-making about which movies to include in their catalogs, recommend to users, or invest in for marketing. By using Ensemble Learning techniques, the goal is to achieve higher predictive accuracy by leveraging the combination of multiple algorithms to capture patterns that individual models might miss. This predictive model can therefore support data-driven strategies in the competitive environment of movie recommendation and curation.</p><p><strong>1.2 Objective</strong><br> The main objective of this project is to build a predictive model capable of automatically classifying whether a movie is likely to be well-rated or not, based on its features. Using the IMDB dataset, a binary target variable (Binary_Rating) was created, where movies with higher ratings are labeled as “1” (good) and the others as “0” (bad).</p><p><strong>This analysis aims to answer questions such as:</strong></p><ul><li>Which features most influence a movie’s high rating? Do genre, director, cast, duration, or release year play decisive roles?</li><li>Is it possible to predict a movie’s critical success in advance?</li><li>How do Ensemble Learning techniques compare to individual models in the classification task?</li><li>Can this type of prediction be useful in recommendation systems or content curation on streaming platforms?</li></ul><p>By addressing these questions, the project not only tests the effectiveness of machine learning methods but also highlights their strategic value for businesses that handle large volumes of audiovisual content.</p><p><strong>2 Understanding the Data</strong><br> <strong>2.1 Data Description and Collection</strong><br> To carry out the analysis, a dataset containing information on 1,000 IMDB movies was used. This dataset, available in CSV format, includes variables such as title, genre, description, director, cast, release year, duration, user rating, number of votes, revenue, and critic score.</p><p>The selected variables were strategically chosen to represent both the content and audience reception aspects of the films, enabling a solid foundation for predictive analysis. With this dataset, it’s possible to apply machine learning techniques — especially Ensemble Learning methods — to more accurately predict how a movie will be perceived by the audience, even before its release or screening.</p><p>To achieve this, a new variable called binary_rating was created to simplify the evaluation of each movie: if the IMDB score (Rating) is greater than or equal to 7, the movie is considered good (value 1); otherwise, it is considered bad (value 0). This transformation makes it easier to build classification models, turning the problem into a more straightforward task suitable for supervised learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ioRMS91RwI_6iNaO.png" /><figcaption>Figure 1 — Database</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/0*m88jj922XKc7alPV.png" /><figcaption>Figure 2 — Metadata</figcaption></figure><p><strong>2.2 Exploratory Analysis</strong></p><p>We can see that the dataset contains 1,000 movies ranging from 2006 to 2016. A describe() was performed on the table to understand the dispersion measures of these variables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*LWCTHAHLFEUh6OMW.png" /><figcaption>Figure 3 — Database with Describe</figcaption></figure><p>During the initial inspection of the data, missing values were found in two variables: Revenue (Millions) and Metascore. For the revenue variable, the missing values were imputed using the median in order to mitigate the influence of outliers observed in its distribution. As for the Metascore, which has a more concentrated range (between 11 and 100), the mean was used to fill in the gaps, since the distribution does not show significant distortions. This treatment ensures that the analyses and subsequent models are based on a clean and consistent dataset. No duplicate records were found, which reinforces the initial consistency of the data.</p><p>During data preparation, some columns were dropped for not adding direct value to the predictive analysis in this context. Variables like Title, Description, Director, and Actors were excluded because they represent textual or high-cardinality information, which would require additional natural language processing (NLP) techniques that are beyond the scope of this project. This decision aims to keep the focus on numerical and categorical variables that are more structured and directly applicable to Ensemble Learning models.</p><p><strong>2.2.1 Genre Variable Processing</strong></p><p>In the dataset, each movie can have up to three different genres listed together in a single column, separated by commas — for example, “Action, Adventure, Fantasy.” To help the model better understand which genres are associated with each movie, it was necessary to transform this information.</p><p>First, these genres were separated, turning that single column with multiple genres into a list of genres for each movie. Then, we broke down this list so that each genre became isolated, allowing the creation of new columns for each genre type (such as “Action,” “Comedy,” “Drama,” etc.). In these new columns, we placed a value of 1 if the movie had that genre, or 0 otherwise.</p><p>Finally, these genre columns were merged back into the original dataset, so the model can clearly and easily use this information to try to predict whether a movie will be rated as good or bad. After this transformation, the original <em>Genre</em> column was removed from the dataset to avoid redundancy.</p><p><strong>2.2.2 Dispersion by Class</strong></p><p>The variable <em>Year</em> was excluded from the dispersion analysis, as it did not show a significant relationship with the other numerical variables. Additionally, its temporal scale makes it harder to visualize correlation patterns with continuous features such as revenue or metascore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pZL1Mpi1AIWcPMjN.png" /><figcaption>Figure 4 — Dispersion by class</figcaption></figure><p>o investigate whether a movie’s genre influences audience ratings, a heatmap was created showing the correlation between the genres — represented as binary variables (0 or 1) — and the <em>binary_rating</em> variable.</p><p>This approach helps to understand if certain types of movies are more likely to receive good ratings. Since each movie can have multiple genres, it was necessary to transform this information into separate columns (as shown earlier) before performing the analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*t4cjUMvgcmIohERlZNoFQw.png" /><figcaption>Figure 5 — Heatmap bygenre and binary_rating</figcaption></figure><p>The heatmap revealed that the genres with the strongest positive correlation to high ratings are:</p><p>🎭 Drama<br> 📜 Biography<br> 🐭 Animation</p><p>In other words, movies in these genres tend to be better received by audiences.</p><p>On the other hand, the genres with the most pronounced negative correlation — indicating a lower chance of receiving good ratings — were:</p><p>😂 Comedy<br> 💥 Action<br> 👻 Horror (with the worst correlation of all)</p><p>These insights help understand audience preference patterns and can be valuable when predicting the success of new releases based on their characteristics.</p><p><strong>2.2.3 Support with Automated Diagnostic Tools</strong></p><p>In addition to the manual analyses described above, automated tools such as Pandas Profiling and Sweetviz were also used. These tools generate comprehensive reports with descriptive statistics, visualizations, and alerts about potential issues in the dataset, such as outliers, low-variance variables, or collinearity.</p><p>These reports served as support to identify points of attention and guide data treatment, but they were not included in full in this article to maintain focus and reading flow.</p><p><strong>2.3 Training the Model</strong></p><p>To evaluate the predictive performance of the dataset, two ensemble learning algorithms were applied: AdaBoost and Gradient Boosting. Both are decision tree-based methods that combine several weak classifiers to form a robust model.</p><h4><strong>Model 1: AdaBoost</strong></h4><p>AdaBoost (Adaptive Boosting) is an algorithm that combines multiple weak classifiers (usually shallow decision trees) to form a stronger classifier. It works by giving more weight to misclassified samples at each iteration, aiming to improve performance.</p><p>🔧 Hyperparameters used<br> For AdaBoost, the following parameters were used:</p><p>AdaBoostClassifier(n_estimators=100, learning_rate=0.5)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*TGn-J7UgGckh112IMzrTYg.png" /><figcaption>Figure 6 — AdaBoost results</figcaption></figure><p><strong>Model 2: Gradient Boosting</strong><br> Gradient Boosting, on the other hand, builds trees sequentially, correcting the errors of the previous model based on the gradient of the error. It is a more powerful and customizable method than AdaBoost.</p><p>🔧 Hyperparameters used<br> GradientBoostingClassifier( n_estimators=100, learning_rate=0.1,<br>subsample=0.7, min_samples_leaf=30, max_depth=2)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*8bXBNDbJmRidFZmMBEmI_Q.png" /><figcaption>Figure 7 — Gradient Boosting results</figcaption></figure><p>🧠 <strong>Conclusion</strong><br> Both models showed good performance, with quite consistent results between training and testing. Gradient Boosting had a slightly better performance across almost all metrics, although with a bit more variation (mild overfitting). This is common when there is more freedom to model, as was the case with the chosen hyperparameters.</p><p>These settings were tuned with the goal of avoiding overfitting, keeping the model simpler and capable of generalizing well to new data. When analyzing feature importance in the Gradient Boosting model, the following stood out as most relevant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*LBvpn3-Gn7Dht1Z-Pw5mAA.png" /><figcaption>Figure 8 — Top 3 most important features from Gradient Boosting Model</figcaption></figure><ol><li><strong>Metascore</strong></li><li><strong>Votes</strong></li><li><strong>Revenue (Millions)</strong></li></ol><p>Interestingly, two of these variables — revenue and metascore — had missing values in the original dataset. This shows that even with partially missing data, these variables contained valuable information for prediction, and proper treatment of missing values was essential to preserve their predictive power in the model.</p><p>Check out my complete projects on <a href="https://github.com/rodrigohigashi">GitHub</a> or read more articles in my <a href="https://www.datascienceportfol.io/rockigo">portfolio</a>.</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d970aed4d7d" width="1" height="1" alt=""><hr><p><a href="https://blog.cubed.run/this-article-is-also-available-in-portuguese-here-5d970aed4d7d">Can We Predict Movie Success Before Release? A Data Science Approach 🎬</a> was originally published in <a href="https://blog.cubed.run/">Cubed</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Chatbot for Commodity Analysis using GPT-3.5 and Financial Data ]]></title>
            <link>https://medium.com/@rockigo/my-first-ai-project-a-chatbot-for-commodity-analysis-with-gpt-3-5-e17af96a2fa0?source=rss-b42500199fcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e17af96a2fa0</guid>
            <dc:creator><![CDATA[Rodrigo Higashi]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 17:24:48 GMT</pubDate>
            <atom:updated>2026-06-04T11:59:44.554Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is also available in Portuguese <a href="https://medium.com/@rockigo/meu-primeiro-projeto-de-ia-um-chatbot-para-an%C3%A1lise-de-commodities-com-gpt-3-5-58b756ac1319">here</a>.</p><p>I have developed a conversational system that simplifies the analysis of financial data, enabling users to analyse commodities such as oil, gold and coffee using natural language, graphical visualisations and automated interpretation.</p><p>The aim is to reduce the effort required to consult historical data, identify trends and obtain initial market analyses. The proposal demonstrates how language models can act as an interaction layer over structured data, making exploratory analyses more accessible and intuitive.</p><p>Users interact using natural language and can request analyses such as:</p><blockquote>“Show me crude oil prices for the last month.”</blockquote><p>Based on this, the system generates a structured response containing:</p><ul><li>historical data</li><li>statistical metrics</li><li>graphical visualisations</li><li>interpretation in natural language</li></ul><h4>📊<strong> Visual analysis using candlestick charts</strong></h4><p>The system transforms historical data into interactive visualisations that make it easier to identify trends, volatility and price fluctuations.</p><p>Example of output generated by the chatbot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2C-v4-l5TQJsuRHX4M2-7g.jpeg" /><figcaption>Figure 1 — Candlestick chart generated by Chatbot commodities</figcaption></figure><h4>⚙️<strong> How the system works</strong></h4><p>When a question is asked, the chatbot automatically runs an analysis workflow:</p><ul><li>It identifies the commodity mentioned (e.g. gold, oil, coffee)</li><li>Collects historical data via the Yahoo Finance API</li><li>Processes the period requested by the user</li><li>Generates candlestick charts of price movements</li><li>Calculates metrics such as:</li><li>average</li><li>minimum and maximum</li><li>standard deviation</li><li>percentage change</li><li>Assesses simple market conditions (e.g. price below the first quartile)</li><li>Sends the structured data to GPT-3.5</li><li>Returns an interpretation in natural language</li></ul><h4>🎥<strong> System demo</strong></h4><p>You can see how the chatbot works in full in the video below:</p><p>👉 [<a href="https://www.loom.com/share/0e6dd9371bf14e389564a0dec3183144">video link here</a>]</p><h4>🧠<strong> Technology stack</strong></h4><p>The project was developed in Python, incorporating:</p><ul><li>OpenAI API (GPT-3.5-turbo-0125)</li><li>yfinance for financial data</li><li>Plotly for generating candlestick charts</li><li>A modular architecture for organising data flow and interacting with the model</li></ul><h4>🔎 <strong>Technical approach (inspired by CRISP-DM)</strong></h4><p>The project was built using concepts inspired by the CRISP-DM framework, which is widely used in data and analytics projects.</p><h4><strong>1. Business Understanding</strong></h4><p>The aim was to create a simpler and more interactive way of exploring recent trends in financial commodities using natural language.</p><h4><strong>2. Data Understanding</strong></h4><p>Historical commodity data was obtained from Yahoo Finance, including assets such as oil, gold, coffee and maize.</p><h4><strong>3. Data Preparation</strong></h4><p>The collected data was processed to:</p><ul><li>organise it chronologically</li><li>calculate statistical metrics</li><li>prepare the information used by the model</li></ul><h4><strong>4. Modelling</strong></h4><p>The system incorporates:</p><ul><li>GPT-3.5 for text interpretation</li><li>support functions for data collection</li><li>candlestick chart generation</li><li>simple rules for exploratory analysis</li></ul><h4><strong>5. Evaluation</strong></h4><p>The tests involved validating the consistency of the generated responses, interpreting the graphs and checking the consistency of the metrics presented.</p><h4><strong>6. Deployment</strong></h4><p>The current version runs via a terminal, with the possibility of future development into a web interface using Streamlit.</p><h4>🚀 <strong>Conclusion</strong></h4><p>The project demonstrates a practical application of generative AI combined with real-world market data, integrating financial analysis, data visualisation and automated interpretation into a simple and accessible conversational workflow.</p><p>***</p><p>Check out my complete projects on <a href="https://github.com/rodrigohigashi">GitHub</a> or see more articles in <a href="http://www.datascienceportfol.io/rockigo">my portfolio</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e17af96a2fa0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>