<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vitor Beltrão on Medium]]></title>
        <description><![CDATA[Stories by Vitor Beltrão on Medium]]></description>
        <link>https://medium.com/@vitorbeltrao300?source=rss-9319367687b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZIbXWXn273l9rY0c-gH9pQ.jpeg</url>
            <title>Stories by Vitor Beltrão on Medium</title>
            <link>https://medium.com/@vitorbeltrao300?source=rss-9319367687b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:25:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vitorbeltrao300/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What Happens Between Your Prompt and the LLM Response?]]></title>
            <link>https://medium.com/@vitorbeltrao300/what-happens-between-your-prompt-and-the-llm-response-c7bae15d6bc0?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7bae15d6bc0</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 17:40:51 GMT</pubDate>
            <atom:updated>2026-06-24T17:40:51.544Z</atom:updated>
            <content:encoded><![CDATA[<h3>What Happens Between Your Prompt and the LLM Response? Exploring the Fundamental Components of the Transformers Library</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VmteEvuMjBqfjh3BmIgew.png" /><figcaption><a href="https://huggingface.co/docs/transformers/index">https://huggingface.co/docs/transformers/index</a></figcaption></figure><p>As the Transformers documentation says:</p><p><em>Transformers acts as the model-definition framework for state-of-the-art machine learning models in text, computer vision, audio, video, and multimodal models, for both inference and training.</em></p><p>Summarizing that sentence in an image, it would look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ncHUCsqmlyIaB4mJfwI-A.png" /><figcaption>Image generated by AI.</figcaption></figure><h4>Pipeline: the simplest way to use a model.</h4><p><em>The </em><a href="https://huggingface.co/docs/transformers/v5.12.0/en/main_classes/pipelines#transformers.Pipeline"><em>Pipeline</em></a><em> is a simple but powerful inference API that is readily available for a variety of machine learning tasks with any model from the Hugging Face </em><a href="https://hf.co/models"><em>Hub</em></a><em>.</em></p><p>Basically, what’s happening behind the scenes in this class is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3lIzFmflOeHvESzjxjbFA.png" /><figcaption>Image generated by AI.</figcaption></figure><p>To use this class, simply do:</p><pre>from transformers import pipeline<br><br># Creates a pre-configured pipeline for sentiment analysis<br>sentiment_analyzer = pipeline(<br>    task=&quot;sentiment-analysis&quot;, <br>    device=&quot;cuda&quot;<br>  )<br><br>result = sentiment_analyzer(<br>    &quot;I&#39;m super excited to be on the way to LLM mastery!&quot;<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*PRqIsq0TD5p-VQZNr55JKA.png" /><figcaption>Result from the code above.</figcaption></figure><p>Note that we didn’t specify the model we wanted, so it used one by default. If we want a specific model for this task, simply add the argument:</p><pre># Creates a pre-configured pipeline for sentiment analysis<br>sentiment_analyzer = pipeline(<br>    task=&quot;sentiment-analysis&quot;, <br>    model=&quot;nlptown/bert-base-multilingual-uncased-sentiment&quot;,<br>    device=&quot;cuda&quot;<br>  )<br><br>result = sentiment_analyzer(<br>    &quot;I should be more excited to be on the way to LLM mastery!&quot;<br>  )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*j1LrvFvLpm6iNLgfiOxxwg.png" /><figcaption>Result from the code above.</figcaption></figure><p>We used a specific model for sentiment analysis that also generates a more specific output for us. Let’s perform another task.</p><pre># Named Entity Recognition<br>ner = pipeline(<br>  task=&quot;ner&quot;, <br>  device=&quot;cuda&quot;<br>)<br><br>result = ner(<br>  &quot;AI Engineers are learning about the amazing pipelines from HuggingFace&quot;<br>)<br><br>for entity in result:<br>  print(entity)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*UUzE6IAsWdNc5Dl15rZ2ow.png" /><figcaption>Result from the code above.</figcaption></figure><p>Named Entity Recognition (NER) is a Natural Language Processing (NLP) task that involves identifying and classifying named entities in a text, that is, locating passages that represent real-world concepts and labeling them into categories such as:</p><ul><li>Per: Persons.</li><li>ORG: Organizations.</li><li>LOC: Locations.</li><li>DATE: Dates.</li></ul><p>NER is one of the most fundamental tasks of NLP precisely because it transforms raw text into structured information, which opens doors to countless practical applications. Let’s look at another example.</p><pre># Question Answering with Context<br>question=&quot;What are Hugging Face pipelines?&quot;<br>context=&quot;Pipelines are a high level API for inference of LLMs with common tasks&quot;<br><br>question_answerer = pipeline(<br>  &quot;question-answering&quot;, <br>  device=&quot;cuda&quot;<br>)<br><br>result = question_answerer(<br>  question=question, <br>  context=context<br>)<br><br>print(result)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*Bjgesn-mSHiaVu4vkTQJdQ.png" /><figcaption>Result from the code above.</figcaption></figure><p>This code performs context-based question answering: the model doesn’t “know” the answer from memory; it reads a piece of text and extracts the answer directly from it.</p><ul><li><strong>Question</strong>: the question asked by the user.</li><li><strong>Context</strong>: the paragraph where the model will look for the answer. The model can only answer based on this text.</li></ul><p>I think you get the idea. There are several tasks we can perform; to access them, see the <a href="https://huggingface.co/docs/transformers/main_classes/pipelines">documentation</a>.</p><h4>Tokenizer</h4><p><em>A tokenizer is in charge of preparing the inputs for a model. The library contains tokenizers for all the models.</em></p><pre>from transformers import AutoTokenizer<br><br>tokenizer = AutoTokenizer.from_pretrained(<br>  &quot;llava-hf/llava-1.5-7b-hf&quot;, <br>  trust_remote_code=True<br>)<br><br>text = &quot;I am excited to show Tokenizers in action to my LLM engineers&quot;<br>tokens = tokenizer.encode(text)<br>print(tokens)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*SyIXAo9x1KdNww8irmnLuQ.png" /><figcaption>Result from the code above.</figcaption></figure><pre>character_count = len(text)<br>word_count = len(text.split(&#39; &#39;))<br>token_count = len(tokens)<br>print(<br>  f&quot;There are {character_count} characters, {word_count} words and {token_count} tokens&quot;<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*gs0tNZ9BgfaGakE6oPIxmA.png" /><figcaption>Result from the code above.</figcaption></figure><p>What happened here? The code converts a phrase into numeric IDs: the only way a language model can “read” text. Loads the specific tokenizer for the <strong>LLaVA 1.5 7B</strong> model and transforms the phrase into a list of numbers, where each number represents a token (word, part of a word, or punctuation). Of course we can access it for any other existing model.</p><p><strong>Why is this matters?</strong></p><p>It’s the mandatory entry point for any LLM, without tokenization, no model can process text. Furthermore:</p><ul><li>Each model has its own vocabulary, so it’s essential to use the correct tokenizer for each one.</li><li>This directly affects the cost and usage limits (APIs charge per token, not per word).</li><li>It influences the quality of the model, since poor tokenization hinders the learning of certain languages ​​and concepts.</li></ul><p>And we can decode the tokens to read what those tokens represent:</p><pre># Decode tokens<br>tokenizer.decode(tokens)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*BOdu2TLvOy-R1MlAdi8FRA.png" /><figcaption>Result from the code above.</figcaption></figure><p>The token &quot;&lt;s&gt;&quot; at the beginning of the phrase is a special token called a <strong>beginning of sequence</strong>, added automatically by the tokenizer. It is not part of the original text, the tokenizer inserts it to signal to the model where the text begins. Each model may have its own specific special features.</p><p><strong>The input to an LLM is a sequence of Token IDs. The output is the probability distribution of the next Token ID to follow this input. That&#39;s it!</strong></p><p>Now it’s your turn, try other models. For anything else, visit the <a href="https://huggingface.co/docs/transformers/main_classes/tokenizer">documentation</a>.</p><h4>Quantization</h4><p><em>Quantization techniques reduce memory and computational costs by representing weights and activations with lower-precision data types like 8-bit integers (int8). This enables loading larger models you normally wouldn’t be able to fit into memory, and speeding up inference.</em></p><p>Without quantization, running a 7B parameter model would require ~14GB of VRAM. With this configuration, it fits in ~4GB, making it possible to run large LLMs on common hardware. For more information, visit the <a href="https://huggingface.co/docs/transformers/main_classes/quantization#quantization">documentation</a>.</p><pre>from transformers import AutoModelForCausalLM, BitsAndBytesConfig<br><br># Model used<br>PHI = &quot;microsoft/Phi-4-mini-instruct&quot;<br>messages = [<br>  {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Tell a joke for a room of Data Scientists&quot;}<br>]<br><br># Quantization Config<br>quant_config = BitsAndBytesConfig(<br>  load_in_4bit=True, # The model weights are loaded in 4 bits<br>  bnb_4bit_use_double_quant=True, # It applies a second quantization on top of the first<br>  bnb_4bit_compute_dtype=torch.bfloat16, # It specifies that internal calculations (not storage) are done in 16 bits<br>  bnb_4bit_quant_type=&quot;nf4&quot; # Use the NormalFloat4 type<br>)<br><br># The model<br>model = AutoModelForCausalLM.from_pretrained(<br>  PHI, <br>  device_map=&quot;auto&quot;, <br>  quantization_config=quant_config<br>)<br><br>memory = model.get_memory_footprint() / 1e6<br>print(f&quot;Memory footprint: {memory:,.1f} MB&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*7toulgvd8QlVFX8kyIFjBQ.png" /><figcaption>Result from the code above.</figcaption></figure><p>So, in this way, we have quantization.</p><h4>Final Exercise</h4><p>Finally, let’s do an audio transcription task.</p><pre>import torch<br>from transformers import pipeline<br><br>pipe = pipeline(<br>    &quot;automatic-speech-recognition&quot;,<br>    model=&quot;openai/whisper-medium.en&quot;,<br>    dtype=torch.float16,<br>    device=&#39;cuda&#39;,<br>    return_timestamps=True<br>)<br><br>result = pipe(audio_filename)<br>transcription = result[&quot;text&quot;]<br>print(transcription)</pre><p>With the code block above, you can easily obtain a transcript of any audio file.</p><h4>Conclusion</h4><p>Again, quoting the documentation:</p><p><em>Transformers is designed for developers and machine learning engineers and researchers. Its main design principles are:</em></p><ol><li><strong><em>Fast and easy to use</em></strong><em>: Every model is implemented from only three main classes (configuration, model, and preprocessor) and can be quickly used for inference or training with </em><a href="https://huggingface.co/docs/transformers/v5.12.0/en/main_classes/pipelines#transformers.Pipeline"><em>Pipeline</em></a><em> or </em><a href="https://huggingface.co/docs/transformers/v5.12.0/en/main_classes/trainer#transformers.Trainer"><em>Trainer</em></a><em>.</em></li><li><strong><em>Pretrained models</em></strong><em>: Reduce your carbon footprint, compute cost and time by using a pretrained model instead of training an entirely new one. Each pretrained model is reproduced as closely as possible to the original model and offers state-of-the-art performance.</em></li></ol><p>That’s all for today! Visit my <a href="https://vitorbeltrao.github.io/">website</a> and contact me if you have any questions or would like to schedule a one-on-one meeting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7bae15d6bc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Diffusion Models with HuggingFace: Generating Images in a Few Lines of Code]]></title>
            <link>https://medium.com/@vitorbeltrao300/introduction-to-diffusion-models-with-huggingface-generating-images-in-a-few-lines-of-code-2030ce922bd2?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2030ce922bd2</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Mon, 15 Jun 2026 23:31:53 GMT</pubDate>
            <atom:updated>2026-06-15T23:31:53.510Z</atom:updated>
            <content:encoded><![CDATA[<p>Diffusion models are a type of generative AI that create data like images or audio by starting from random noise and gradually refining it into meaningful output. They learn this process by adding noise to real data during training and then reversing it step by step. This allows them to generate realistic and high-quality samples from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*66tLGrwpXuml3zg-ds780w.png" /><figcaption>Diffusion model: <a href="https://www.geeksforgeeks.org/artificial-intelligence/what-are-diffusion-models/">https://www.geeksforgeeks.org/artificial-intelligence/what-are-diffusion-models/</a></figcaption></figure><p>The <a href="https://huggingface.co/docs/diffusers/index">HuggingFace</a> documentation says:</p><p><em>Diffusion models are a relatively recent addition to a group of algorithms known as &quot;generative models&quot;. The goal of generative modeling is to learn to generate data, such as images or audio, given a number of training examples. A good generative model will create a diverse set of outputs that resemble the training data without being exact copies.</em></p><p>The secret to diffusion models success is the iterative nature of the diffusion process. Generation begins with random noise, but this is gradually refined over a number of steps until an output image emerges. At each step, the model estimates how we could go from the current input to a completely denoised version. However, since we only make a small change at every step, any errors in this estimate at the early stages (where predicting the final output is extremely difficult) can be corrected in later updates.</p><p>In summary, what this type of model is doing is:</p><ul><li>During training, noise is gradually added to real data so the model learns how data degrades.</li><li>The model is trained to reverse this process by removing noise step-by-step.</li><li>This helps it learn complex data patterns and generate high-quality outputs.</li><li>At inference, it starts with random noise and gradually denoises it to produce data like images, audio or text.</li></ul><h4>Practical step with huggingface</h4><p>First, if you haven’t already done so, create a free <a href="https://huggingface.co/">HuggingFace</a> account and generate a token for you.</p><pre># Log in to HuggingFace using your token<br>from huggingface_hub import login<br>login(hf_token, add_to_git_credential=True)</pre><p>Then, in a very simple way, using the code below you can quickly create images.</p><pre># using the AutoPipeline class<br>from diffusers import AutoPipelineForText2Image<br>import torch<br><br>pipe = AutoPipelineForText2Image.from_pretrained(<br>    &quot;stabilityai/sdxl-turbo&quot;, <br>    dtype=torch.float16, <br>    variant=&quot;fp16&quot;<br>).to(&quot;cuda&quot;)<br><br>prompt = &quot;A class of students learning AI engineering&quot;<br><br>image = pipe(<br>    prompt=prompt, <br>    num_inference_steps=4, <br>    guidance_scale=0.0).images[0]<br><br>display(image)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*O0XEe8wc2_t9TX_zA8IUIg.png" /><figcaption>Image generated by the model above.</figcaption></figure><p>Alternatively,</p><pre>from diffusers import DiffusionPipeline<br>import torch<br><br>pipe = DiffusionPipeline.from_pretrained(<br>    &quot;stabilityai/stable-diffusion-xl-base-1.0&quot;, <br>    torch_dtype=torch.float16, <br>    use_safetensors=True, <br>    variant=&quot;fp16&quot;<br>  ).to(&quot;cuda&quot;)<br><br>prompt = &quot;A class of students learning AI engineering&quot;<br><br>image = pipe(<br>    prompt=prompt, <br>    num_inference_steps=30).images[0]<br><br>display(image)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdJg8kfWr_MOOMDkj-1oRg.png" /><figcaption>Image generated by the model above.</figcaption></figure><p><em>DiffusionPipeline</em> focuses on the model, automatically loading it based on what it is; while <em>AutoPipeline</em> focuses on the task, configuring the weights specifically for the action you want to perform (such as creating, editing, or restoring images).</p><p>We can create an ensemble to further improve the result:</p><pre>from diffusers import DiffusionPipeline<br>import torch<br><br>base = DiffusionPipeline.from_pretrained(<br>    &quot;stabilityai/stable-diffusion-xl-base-1.0&quot;, <br>    torch_dtype=torch.float16, <br>    variant=&quot;fp16&quot;, <br>    use_safetensors=True).to(&quot;cuda&quot;)<br><br>refiner = DiffusionPipeline.from_pretrained(<br>    &quot;stabilityai/stable-diffusion-xl-refiner-1.0&quot;, <br>    text_encoder_2=base.text_encoder_2, <br>    vae=base.vae, <br>    torch_dtype=torch.float16, <br>    use_safetensors=True, <br>    variant=&quot;fp16&quot;).to(&quot;cuda&quot;)<br><br># define how many steps and what % of steps to be run <br># on each experts (80/20) here<br>n_steps = 40<br>high_noise_frac = 0.8<br><br>prompt = &quot;A class of students learning AI engineering&quot;<br><br># run both experts<br>image = base(<br>    prompt=prompt,<br>    num_inference_steps=n_steps,<br>    denoising_end=high_noise_frac,<br>    output_type=&quot;latent&quot;,<br>).images<br><br>image = refiner(<br>    prompt=prompt,<br>    num_inference_steps=n_steps,<br>    denoising_start=high_noise_frac,<br>    image=image,<br>).images[0]<br><br>display(image)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FoLOwm_iQvqEzlWE3uW9fw.png" /><figcaption>Image generated by the model above.</figcaption></figure><p>The Base model acts as the architect, defining the overall composition and shapes of the image in the first 80% of the process; then, the Refiner model takes on the role of finisher, polishing textures and fine details in the remaining 20%. This division of tasks ensures images with a solid structure and high-fidelity finish, optimizing memory usage by sharing essential components between the two pipelines.</p><h4>Conclusion</h4><p>Diffusion models have democratized AI image generation, and the Diffusers library makes their use extremely simple. Classes like <em>AutoPipelineForText2Image</em> and <em>DiffusionPipeline</em> abstract the complexity of the models, allowing developers to focus more on application than implementation.</p><p>While these models offer impressive results, it’s important to consider the consumption of computational resources and inference time. Still, for those wishing to explore generative AI in their projects, they represent an excellent entry point into one of the most interesting fields of current artificial intelligence.</p><p>That’s all for today! Visit my <a href="https://vitorbeltrao.github.io/">website</a> and contact me if you have any questions or would like to schedule a one-on-one meeting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2030ce922bd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adversarial conversation between Chatbots]]></title>
            <link>https://medium.com/@vitorbeltrao300/adversarial-conversation-between-chatbots-bec0f2b3a74d?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bec0f2b3a74d</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Mon, 04 May 2026 12:11:12 GMT</pubDate>
            <atom:updated>2026-05-04T12:11:12.756Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFibF8bUAuNqafcEM-hzgw.png" /><figcaption>Image created by AI.</figcaption></figure><p>In the <a href="https://medium.com/@vitorbeltrao300/pdf-summarizer-how-to-use-all-frontier-and-open-source-models-using-the-completions-api-8e4a9fdc6788">last article</a>, we learned how to use OpenAI’s API completions to make requests from various LLMs, both from OpenAI itself and from Google and Anthropic, and even open-source models like Ollama.</p><p>Now, let’s create a conversation between chatbots, which is a very fun exercise using frontier or open-source models, and also how to connect to their APIs directly.</p><h4><strong>OpenAI Model</strong></h4><p>We already talked about it in the last article, but we’ll repeat it to make everything clearer. Once again, you must create your credentials, you should access the <a href="https://platform.openai.com/chat">OpenAI Platform</a>. You should have an API Key and add some payment method; unfortunately, frontier models are not free.</p><pre># Import necessary libraries<br>import os<br>from openai import OpenAI<br>from google import genai<br>from anthropic import Anthropic<br>from langchain_openai import ChatOpenAI<br>from litellm import completion<br>from dotenv import load_dotenv<br>from IPython.display import Markdown, display<br><br># Import your credentials<br>load_dotenv(override=True)<br>openai_api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br>gemini_api_key = os.getenv(&quot;GOOGLE_API_KEY&quot;)<br>anth_api_key = os.getenv(&quot;ANTH_API_KEY&quot;)<br>openrouter_api_key = os.getenv(&quot;OPENROUTER_API_KEY&quot;)</pre><pre># Instantiate the open ai class<br>openai = OpenAI(<br>    api_key=openai_api_key<br>)<br><br># Define the context and interaction with the model<br>messages = [<br>    {<br>        &quot;role&quot;: &quot;system&quot;, <br>        &quot;content&quot;: &quot;You are an helpful assistant&quot;<br>    }, # general behavior of the model<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: &quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>    } # user entry<br>]<br><br># Method chat.completions.create to send a message<br>response = openai.chat.completions.create(<br>    model=&quot;gpt-5-mini-2025-08-07&quot;, <br>    messages=messages<br>)<br><br># Get the message from the API response<br>display(Markdown(response.choices[0].message.content))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*0T1feL6XYScUd4pHxhgUaA.png" /><figcaption>Result of code above.</figcaption></figure><p>We already knew that. Easy, right?</p><h4><strong>Google Model</strong></h4><p>To create your Google credentials, follow this <a href="https://aistudio.google.com/">link</a>.</p><pre># Instantiate the google gen ai class<br>gemini = genai.Client(<br>    api_key=gemini_api_key<br>)<br><br># Method models.generate_content to send a message<br>response = gemini.models.generate_content(<br>    model=&quot;gemini-2.5-flash-lite&quot;, <br>    contents=&quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>)<br><br># Get the message from the API response<br>print(response.text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*-GRR0xUzhLzkGYHMZSLpiQ.png" /><figcaption>Result of code above.</figcaption></figure><h4><strong>Anthropic Model</strong></h4><p>To create your Anthropic credentials, follow this <a href="https://console.anthropic.com/">link</a>.</p><pre># Instantiate the anthropic class<br>anth = Anthropic(<br>    api_key=anth_api_key<br>)<br><br># Method messages.create to send a message<br>response = anth.messages.create(<br>    model=&quot;claude-sonnet-4-5-20250929&quot;,<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;user&quot;, <br>            &quot;content&quot;: &quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>        }<br>    ],<br>    max_tokens=100 # to control the output and costs<br>)<br><br># Get the message from the API response<br>print(response.content[0].text)</pre><p>Pretty straightforward, right? Now let’s also see how to use Routers and abstraction layers that can connect to all the models above.</p><h4><strong>Abstraction Frameworks</strong></h4><p>These are tools that create a unified and decoupled interface for interacting with different language models, providers, and execution flows.</p><p><strong>What problem do they solve?</strong></p><p>Without these libraries, you would have:</p><ul><li>Code coupled to a provider (e.g., OpenAI directly).</li><li>Difficulty in changing models.</li><li>Repetition of logic (prompt, retry, parsing, etc.).</li><li>Lack of standardization.</li></ul><p>With them, you gain:</p><ul><li>Model portability.</li><li>Pipeline composition.</li><li>Integration with external tools.</li><li>Observability and control.</li></ul><p>Here we will discuss three available tools: <a href="https://openrouter.ai/">OpenRouter</a>, <a href="https://www.langchain.com/">LangChain</a>, and <a href="https://www.litellm.ai/">LiteLLM</a>. Each has its pros and cons, and it is up to the professional to understand which one is best suited to their context.</p><pre># openrouter<br>openrouter = OpenAI(<br>    base_url=&quot;https://openrouter.ai/api/v1&quot;, <br>    api_key=openrouter_api_key<br>)<br><br>messages = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: &quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>    }<br>]<br><br>response = openrouter.chat.completions.create(<br>    model=&quot;z-ai/glm-4.5&quot;, <br>    messages=messages,<br>    max_tokens=1000<br>)<br><br>display(Markdown(response.choices[0].message.content))</pre><pre># langchain<br>llm = ChatOpenAI(model=&quot;gpt-5-mini&quot;)<br><br>response = llm.invoke(<br>    &quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>)<br><br>display(Markdown(response.content))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*b8lTAvN1VBkNwnsLK90Qag.png" /><figcaption>Result of code above.</figcaption></figure><pre># litellm<br>messages = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: &quot;Tell a joke for a student on the journey to becoming an expert in LLM Engineering&quot;<br>    }<br>]<br><br>response = completion(<br>    model=&quot;openai/gpt-4.1&quot;, <br>    messages=messages<br>)<br><br>reply = response.choices[0].message.content<br><br>display(Markdown(reply))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*QSWCZWSa9vvJkzrktTfPJA.png" /><figcaption>Result of code above.</figcaption></figure><h4><strong>Adversarial Conversation Between Chatbots</strong></h4><p>Now let’s do this fun exercise of a conversation between chatbots. This exercise is very interesting for learning more about these tools.</p><p>Let’s establish a conversation between <em>gpt-4o-mini </em>and <em>gemini-2.5-flash-lite</em>.</p><pre># setting params<br>gpt_model = &quot;gpt-4o-mini&quot;<br>gemini_model = &quot;gemini-2.5-flash-lite&quot;<br><br>gpt_system = &quot;You are a chatbot who is very argumentative; \<br>you disagree with anything in the conversation and you challenge everything, in a snarky way.&quot;<br><br>gemini_system = &quot;You are a very polite, courteous chatbot. You try to agree with \<br>everything the other person says, or find common ground. If the other person is argumentative, \<br>you try to calm them down and keep chatting.&quot;<br><br>openai_client = OpenAI(<br>    api_key=openai_api_key<br>)<br><br>gemini_client = OpenAI(<br>    base_url=&quot;https://generativelanguage.googleapis.com/v1beta/openai/&quot;, <br>    api_key=gemini_api_key<br>)<br><br># auxiliar functions<br>def get_messages_for(speaker, history, system_prompt):<br>    &quot;&quot;&quot;It reverses the roles of the historical account to the speaker&#39;s perspective&quot;&quot;&quot;<br>    messages = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_prompt}]<br>    for turn in history:<br>        role = &quot;assistant&quot; if turn[&quot;speaker&quot;] == speaker else &quot;user&quot;<br>        messages.append({&quot;role&quot;: role, &quot;content&quot;: turn[&quot;content&quot;]})<br>    return messages<br><br>def call_model(client, model, system_prompt, history, speaker):<br>    messages = get_messages_for(speaker, history, system_prompt)<br>    response = client.chat.completions.create(<br>        model=model,<br>        messages=messages,<br>        max_tokens=50<br>    )<br>    return response.choices[0].message.content<br><br># model config<br>models = {<br>    &quot;gpt&quot;:    {&quot;client&quot;: openai_client,  &quot;model&quot;: gpt_model,    &quot;system&quot;: gpt_system},<br>    &quot;gemini&quot;: {&quot;client&quot;: gemini_client,  &quot;model&quot;: gemini_model, &quot;system&quot;: gemini_system},<br>}<br><br># unique history with speaker identification.<br>history = []<br><br># initial message — assigned to Gemini for GPT to respond first.<br>initial_message = &quot;Hi There&quot;<br>history.append({&quot;speaker&quot;: &quot;gemini&quot;, &quot;content&quot;: initial_message})<br><br>turn_order = [&quot;gpt&quot;, &quot;gemini&quot;]<br><br>for i in range(3):<br>    print(f&quot;--- Rodada {i+1} ---&quot;)<br>    for speaker in turn_order:<br>        cfg = models[speaker]<br>        reply = call_model(cfg[&quot;client&quot;], cfg[&quot;model&quot;], cfg[&quot;system&quot;], history, speaker)<br>        history.append({&quot;speaker&quot;: speaker, &quot;content&quot;: reply})<br>        print(f&quot;{speaker.upper()}: {reply}\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nAZylkcCjmhp2ms-ylrdzg.png" /><figcaption>Result of code above.</figcaption></figure><h4>Conclusion</h4><p>In this article, we saw how to use each vendor’s APIs separately, as well as how to use abstraction frameworks.</p><p>We also saw how to use two language models to create a conversation between them. This is a great exercise to learn how these models work and how we can use them.</p><p>That’s all for today! Visit my <a href="https://vitorbeltrao.github.io/">website</a> and contact me if you have any questions or would like to schedule a one-on-one meeting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bec0f2b3a74d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PDF Summarizer: How to use all frontier and open-source models using the completions API]]></title>
            <link>https://medium.com/@vitorbeltrao300/pdf-summarizer-how-to-use-all-frontier-and-open-source-models-using-the-completions-api-8e4a9fdc6788?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e4a9fdc6788</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Fri, 20 Mar 2026 13:14:55 GMT</pubDate>
            <atom:updated>2026-03-20T13:14:55.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKLId6JMjhSwCSmhi_pzGw.png" /><figcaption>Image created by AI.</figcaption></figure><p>Nowadays, we use APIs for connecting with the strongest LLMs available. The companies behind these LLMs, such as OpenAI, Anthropic, Google and DeepSeek, have built web endpoints.</p><p>It would be painful if we needed to build HTTP requests every time we wanted to call an API. To make this simple, the team at OpenAI wrote a python utility known as a “Python Client Library” which wraps the HTTP call. So you write python code and it calls the web.</p><p>And <strong>THAT is what the library <em>openai</em> python client library is:</strong></p><p>- A lightweight python utility.</p><p>- Turns your python requests into an HTTP call.</p><p>- Converts the results coming back from the HTTP call into python objects.</p><p><strong>What it is NOT:</strong></p><p>- It’s not got any code to actually run a Large Language Model! No GPT code! It just makes a web request.</p><p>- There’s no scientific computing code, and nothing particularly specialized for OpenAI.</p><p>Let’s get started!</p><h3><strong>1. Read a PDF File</strong></h3><p>Let’s read any PDF file. I chose my resume, but feel free to choose any one you like.</p><pre># Import necessary libraries<br>import os<br>import requests<br>from dotenv import load_dotenv<br>from IPython.display import Markdown, display<br>from openai import OpenAI<br>from pypdf import PdfReader</pre><pre>reader = PdfReader(&#39;inputs\Vitor Beltrao CV.pdf&#39;) # the path to your pdf file<br>print(f&quot;Total pages: {len(reader.pages)}&quot;)<br><br>full_text = &quot;&quot;<br>for page in reader.pages:<br>    full_text += page.extract_text() + &quot;\n&quot;<br><br>print(&quot;\n--- Full Text ---&quot;)<br>print(full_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*HjYOLP-XbHGQnHxEUXt7zQ.png" /><figcaption>Result of the code above.</figcaption></figure><h3><strong>2. Call a frontier model to do your task</strong></h3><p>Frontier models are the most advanced, large-scale artificial intelligence models that define the current state-of-the-art in AI capabilities, typically excelling in reasoning, coding, and multimodal tasks (text, image, audio).</p><h4><strong>OpenAI Model</strong></h4><p>To make your credentials, you should access the <a href="https://platform.openai.com/chat">OpenAI Platform</a>. You should have an API Key and add some payment method; unfortunately, frontier models are not free.</p><pre># Load environment variables in a file called .env<br>load_dotenv(override=True)<br>api_key = os.getenv(&#39;OPENAI_API_KEY&#39;)<br><br># Check the key<br>if not api_key:<br>    print(&quot;No API key was found - please head over to the troubleshooting notebook in this folder to identify &amp; fix!&quot;)<br>elif not api_key.startswith(&quot;sk-proj-&quot;):<br>    print(&quot;An API key was found, but it doesn&#39;t start sk-proj-; please check you&#39;re using the right key - see troubleshooting notebook&quot;)<br>elif api_key.strip() != api_key:<br>    print(&quot;An API key was found, but it looks like it might have space or tab characters at the start or end - please remove them - see troubleshooting notebook&quot;)<br>else:<br>    print(&quot;API key found and looks good so far!&quot;)</pre><pre># Instantiate the open ai class<br>openai = OpenAI()<br><br># Define the context and interaction with the model<br>messages = [<br>    {<br>        &quot;role&quot;: &quot;system&quot;, <br>        &quot;content&quot;: &quot;You are an HR specialist&quot;<br>    }, # general behavior of the model<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: f&quot;Use this resume below to create a simple and concise cover letter: {full_text}&quot;<br>    } # user entry<br>]<br><br># Method chat.completions.create to send a message<br>response = openai.chat.completions.create(<br>    model=&quot;gpt-5-mini-2025-08-07&quot;, # choose the model that you want<br>    messages=messages<br>)<br><br># Get the message from the API response<br>display(Markdown(response.choices[0].message.content))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4Z3wiVztmY6DMPjZsecIw.png" /><figcaption>Result of the code above. The input for the model was to create a cover letter based on my resume.</figcaption></figure><p>When you make the python call: <em>openai.chat.completions.create()</em></p><p>It simply makes a web request to this url: <em>https://api.openai.com/v1/chat/completions</em></p><p>And it converts the response to python objects.</p><p>The Chat Completions API endpoint will generate a model response from a list of messages comprising a conversation.</p><p>That’s it.</p><p>Here’s the <a href="https://developers.openai.com/">API documentation</a>.</p><p><strong>With that context, how do I use other LLMs?</strong></p><p>All the other major LLMs have API endpoints that are compatible with OpenAI. You can use OpenAI utility for converting python to web requests. They allow you to change the utility from calling <em>https://api.openai/com/v1</em> to calling any web address that you specify. So you can use the OpenAI utility even for calling models that are NOT OpenAI, like this:</p><p><em>not_actually_openai = OpenAI(base_url=&quot;https://somewhere.completely.different/&quot;, api_key=&quot;another_providers_key&quot;)</em></p><p>Here are all the OpenAI-compatible endpoints from the major providers. It even includes using Ollama, locally. Ollama provides an endpoint on your local machine, and they made it OpenAI compatible too — very convenient.</p><ul><li>ANTHROPIC_BASE_URL = <a href="https://api.anthropic.com/v1/">https://api.anthropic.com/v1/</a></li><li>DEEPSEEK_BASE_URL = <a href="https://api.deepseek.com/v1">https://api.deepseek.com/v1</a></li><li>GEMINI_BASE_URL = <a href="https://generativelanguage.googleapis.com/v1beta/openai/">https://generativelanguage.googleapis.com/v1beta/openai/</a></li><li>GROK_BASE_URL = <a href="https://api.x.ai/v1">https://api.x.ai/v1</a></li><li>GROQ_BASE_URL = <a href="https://api.groq.com/openai/v1">https://api.groq.com/openai/v1</a></li><li>OPENROUTER_BASE_URL = <a href="https://openrouter.ai/api/v1">https://openrouter.ai/api/v1</a></li><li>OLLAMA_BASE_URL = <a href="http://localhost:11434/v1">http://localhost:11434/v1</a></li></ul><h4><strong>Google Model</strong></h4><p>Now, let’s use a Google model to make requests using the same library. Of course, large companies also have their own libraries. If instead of using the OpenAI library, you want to use the Google library directly, here is the necessary <a href="https://ai.google.dev/gemini-api/docs">documentation</a>.</p><pre>load_dotenv(override=True)<br>google_api_key = os.getenv(&#39;GOOGLE_API_KEY&#39;)<br><br>gemini = OpenAI(<br>    base_url=&quot;https://generativelanguage.googleapis.com/v1beta/openai/&quot;, <br>    api_key=google_api_key<br>)<br><br>messages = [<br>    {<br>        &quot;role&quot;: &quot;system&quot;, <br>        &quot;content&quot;: &quot;You are an HR specialist&quot;<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: f&quot;Use this resume below to create a simple and concise cover letter: {full_text}&quot;<br>    }<br>]<br><br>response = gemini.chat.completions.create(<br>    model=&quot;gemini-3-flash-preview&quot;, <br>    messages=messages<br>)<br><br>display(Markdown(response.choices[0].message.content))</pre><p>This code will return the cover letter that was requested.</p><h4><strong>Anthropic Model</strong></h4><p>We couldn’t leave out the Anthropic model. Again, if you want to use Anthropic’s own library, just follow the documentations:</p><ul><li><a href="https://platform.claude.com/dashboard">To create an API key</a>.</li><li><a href="https://platform.claude.com/docs/en/home">API documentation</a>.</li></ul><pre>load_dotenv(override=True)<br>anth_api_key = os.getenv(&#39;ANTH_API_KEY&#39;)<br><br>anthropic = OpenAI(<br>    base_url=&quot;https://api.anthropic.com/v1/&quot;, <br>    api_key=anth_api_key<br>)<br><br>messages = [<br>    {<br>        &quot;role&quot;: &quot;system&quot;, <br>        &quot;content&quot;: &quot;You are an HR specialist&quot;<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: f&quot;Use this resume below to create a simple and concise cover letter: {full_text}&quot;<br>    }<br>]<br><br>response = anthropic.chat.completions.create(<br>    model=&quot;claude-haiku-4-5&quot;, <br>    messages=messages<br>)<br><br>display(Markdown(response.choices[0].message.content))</pre><p>Simple, right?</p><h3><strong>3. Call open-source models</strong></h3><p>Finally, we can also use open-source models like Ollama. To learn more about Ollama, read this <a href="https://medium.com/@vitorbeltrao300/how-do-i-run-my-llm-locally-2d6ed0c69447">article</a> which will give you an overview.</p><pre># verify if ollama is running<br>requests.get(&quot;http://localhost:11434&quot;).content</pre><p>If the cell above doesn’t print “Ollama is running” then open your terminal and run: “ollama serve”.</p><pre># download llama 3 from meta<br>!ollama pull llama3:8b</pre><p>Here you can use any model available on Ollama. Be careful with the size of the models you choose; always select a model that can run on your local machine and that is good in terms of quality and speed.</p><p>Then we can use the completions method.</p><pre>OLLAMA_BASE_URL = &quot;http://localhost:11434/v1&quot;<br>ollama = OpenAI(<br>    base_url=OLLAMA_BASE_URL, <br>    api_key=&#39;a&#39; # here you can pass any word<br>)<br><br>messages = [<br>    {<br>        &quot;role&quot;: &quot;system&quot;, <br>        &quot;content&quot;: &quot;You are an HR specialist&quot;<br>    },<br>    {<br>        &quot;role&quot;: &quot;user&quot;, <br>        &quot;content&quot;: f&quot;Use this resume below to create a simple and concise cover letter: {full_text}&quot;<br>    }<br>]<br><br>response = ollama.chat.completions.create(<br>    model=&quot;llama3:8b&quot;, <br>    messages=messages<br>)<br><br>display(Markdown(response.choices[0].message.content))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJBj6qkDaYzioLkLrVAZLg.png" /><figcaption>Result of the code above.</figcaption></figure><h3><strong>Conclusion</strong></h3><p>We’ve seen how to use frontier models and open-source models to solve a very common problem in people’s daily lives: summarizing documents, but you can use them to perform any task that these models can solve. Feel free to use whichever best fits your context; both work very well!</p><p>We used the OpenAI Completions API to perform this task. We can say that the Completions API is basically:</p><p>1. The simplest way to call an LLM</p><p>2. It’s called Chat Completions because it’s saying: “here is a conversation, please predict what should come next”</p><p>3. The Chat Completions API was invented by OpenAI, but it’s so popular that everybody uses it!</p><p>That’s all for today! Visit my <a href="https://vitorbeltrao.github.io/">website</a> and contact me if you have any questions or would like to schedule a one-on-one meeting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e4a9fdc6788" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do I run my LLM locally?]]></title>
            <link>https://medium.com/@vitorbeltrao300/how-do-i-run-my-llm-locally-2d6ed0c69447?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d6ed0c69447</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Sat, 07 Mar 2026 14:32:39 GMT</pubDate>
            <atom:updated>2026-03-07T14:32:39.037Z</atom:updated>
            <content:encoded><![CDATA[<p>Here we will use <a href="https://ollama.com/">Ollama</a>, an open-source tool that allows anyone to download and run powerful models directly from their computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSxdCUvdFyVRVKlsQJFwJw.png" /><figcaption>Image generated by AI.</figcaption></figure><p>To get straight to the point, first download Ollama by clicking the download button on the main page or run this command in your terminal:</p><pre>irm https://ollama.com/install.ps1 | iex</pre><p>After installation, open your terminal and type:</p><pre>ollama run &lt;your chosen model&gt;</pre><p>After the “run” command, type the model you chose. You can see all available models on the Ollama website under the <a href="https://ollama.com/search">Models</a> tab. After this command above, your machine will begin downloading the model, and then you’re ready to start interacting with it, as shown in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*NnJLWngomm8uiNLVvoIf0g.png" /><figcaption>Downloading the model from ollama.</figcaption></figure><p>In this case, we downloaded a Gemma model, which is a family of lightweight models from Google, built with Gemini technology. I chose a lighter model with 270 million parameters. If your machine is more robust, you will probably be able to choose models with more parameters; feel free to use the best model for you.</p><p>Finally, we can interact with the model!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*Fvlxyz8HdhRj8Tm44dC8GQ.png" /><figcaption>Interacting with the chosen model.</figcaption></figure><p>Since this is a simpler model, don’t expect too much from it. The idea here is for you to see how more robust models (with more parameters) are able to interact with you in a more fluid and intelligent way.</p><p>The advantage of using a template locally is that it runs entirely on your machine. When interacting with the template, the text doesn’t travel over the internet to third-party servers, making it much safer if you want to use sensitive data. So overall, you can feel much more secure about it.</p><p>Other advantages of using a model locally, besides <strong>privacy</strong>, are that <strong>it’s</strong> <strong>free</strong>: you don’t pay for tokens or monthly fees; <strong>Offline use</strong>: you can interact with the model without necessarily being connected to the internet.</p><p>The only downside is that you might need a powerful machine to run more robust models, as we mentioned above.</p><p>That’s all for today! Visit my <a href="https://vitorbeltrao.github.io/">website</a> and contact me if you have any questions or would like to schedule a one-on-one meeting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d6ed0c69447" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Variance Inflation Factor]]></title>
            <link>https://medium.com/@vitorbeltrao300/understanding-the-variance-inflation-factor-a0fd379c8dab?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0fd379c8dab</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 00:24:15 GMT</pubDate>
            <atom:updated>2025-07-30T00:24:15.248Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*HdFnl1MYWolyOQgpzqwi9A.png" /><figcaption>AI generated image</figcaption></figure><p>In regression analysis, we know that assumptions must be met. Otherwise, the results may be misleading and/or underperform.</p><p>One of these assumptions is the <strong>non-multicollinearity between the independent variables</strong>. Multicollinearity occurs when two or more independent variables are highly correlated.</p><p>The most common way to quantify this correlation is to use statistical tools such as Pearson and/or Spearman correlation coefficient, but we also have a metric created for this purpose: <strong>the variance inflation factor (VIF)</strong>.</p><p>As the name suggests, VIF quantifies how much the variance is inflated, but which variance?</p><blockquote>The variance related to the standard error of its coefficients. The range of standard errors of its coefficients will be larger and this gives us a misleading result.</blockquote><p>The VIF exists for each explanatory variable in a multiple regression model. For example, the VIF related to the respective estimated regression coefficient θ_i, denoted VIF_i, is the factor by which the variance θ_i is inflated by the existence of correlation between the explanatory variables in the model.</p><p>The VIF of the i-th predictor is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*epWNvWYhHg9ztonsHB0Agw.png" /><figcaption>VIF of the i-th predictor</figcaption></figure><p>Where, R²_i is the R² value obtained by regression of the i-th predictor against the other explanatory variables. According to the <a href="https://www.statsmodels.org/stable/index.html">statsmodels </a>statistical library:</p><blockquote>The VIF is a measure of the increase in variance of the coefficient estimates if an additional variable is introduced into the regression.</blockquote><h4><strong>Interpretation of VIF values</strong></h4><ul><li><strong>VIF = 1</strong> → No multicollinearity.</li><li><strong>1 &lt; VIF ≤ 5</strong> → Moderate multicollinearity.</li><li><strong>5 &lt; VIF ≤ 10</strong> → Multicollinearity may be affecting the model. It might needs investigation.</li><li><strong>VIF &gt; 10</strong> → Severe multicollinearity!</li></ul><h4><strong>Solutions for a high VIF</strong></h4><ol><li>Remove explanatory variables that are highly correlated with other variables and are of less importance to the analysis or the business. <strong>The decision of which variables to remove is up to the analyst and can be scientific or practical.</strong></li><li>Use feature engineering techniques, such as combining them into one (if that makes sense, of course).</li><li>Use regularized models. Methods such as Ridge or Lasso can help address multicollinearity.</li><li>Use dimensionality reduction models such as PCA.</li></ol><h4>Summary</h4><ul><li>The VIF measures how much the variance of a coefficient is inflated because of correlation with other variables.</li><li>It is calculated by the inverse of 1 — R² of the regression obtained on the other variables.</li><li>It is a diagnostic tool to help in decision-making whether or not there is excessive multicollinearity that can affect the interpretation of regression coefficients.</li></ul><h4>Practical example</h4><p>Let’s look at a practical example to understand how to use this metric.</p><p>The dataset refers to data collected during concrete manufacturing. We have all the explanatory variables typically measured during concrete manufacturing, and the target variable is the concrete’s compressive strength.</p><ol><li><strong>Get the dataset</strong></li></ol><p>The dataset was downloaded from Kaggle. For more information about the data, visit the <a href="http://bit.ly/44P5Hsk">link</a>.</p><pre># import the necessary libraries<br>import pandas as pd<br>from statsmodels.stats.outliers_influence import variance_inflation_factor<br>from statsmodels.formula.api import ols</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*1j3c2apR4J-b9uoHcb-q4A.png" /><figcaption>Dataset read as a pandas dataframe</figcaption></figure><pre># analyze VIF<br>independent_variables = df.drop(<br>    columns=[&#39;strength&#39;], axis=1) # we only want the explanatory variables<br><br>vif = pd.DataFrame()<br>vif[&quot;features&quot;] = independent_variables.columns<br>vif[&quot;vif_factor&quot;] = [<br>    variance_inflation_factor(independent_variables.values, i) for i in range(independent_variables.shape[1])]<br>vif = vif.sort_values(&#39;vif_factor&#39;, ascending=False)<br>vif</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*-CdQFUIJDj3BjnsYqTQ__Q.png" /><figcaption>VIF of explanatory variables</figcaption></figure><p>VIF gives us a quick and simple way to quickly understand which variables require special attention, in this case: “<em>coarse_aggregate</em>,” “<em>water</em>,” “<em>fine_aggregate</em>,” and “<em>cement</em>.” These variables are highly correlated with one or more variables in the dataset. Let’s run a regression model on this complete data and analyze the results.</p><p><strong>2. First model fitted with all variables</strong></p><pre># adjusted regression model<br>model = ols(<br>    &#39;strength ~ coarse_aggregate + water + fine_aggregate + cement + superplasticizer + fly_ash + blast_furnace_slag + age&#39;,<br>    data=df)<br>df_result = model.fit()<br>print(df_result.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*2ES7Xon-z20KEFCPe7VL5w.png" /><figcaption>Statistical output of the regression model</figcaption></figure><p>According to the results, we can see that two variables, “<em>coarse_aggregate</em>” and “<em>fine_aggregate</em>,” are not statistically significant and were not considered explanatory for our results. In fact, these two were the variables that had a high VIF. Let’s start by removing the variables with the highest VIF (<em>coarse_aggregate </em>and <em>fine_aggregate</em>) and analyze the results again.</p><pre>independent_variables = df.drop(columns=[<br>    &#39;strength&#39;,<br>    &#39;coarse_aggregate&#39;,<br>    &#39;fine_aggregate&#39;<br>    ], axis=1)<br><br>vif = pd.DataFrame()<br>vif[&quot;features&quot;] = independent_variables.columns<br>vif[&quot;vif_factor&quot;] = [<br>    variance_inflation_factor(independent_variables.values, i) for i in range(independent_variables.shape[1])]<br>vif = vif.sort_values(&#39;vif_factor&#39;, ascending=False)<br>vif</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*TY7CarAWAj4q2jU_tcaorQ.png" /><figcaption>VIF of the remaining variables: see how much the VIF has already decreased</figcaption></figure><p><strong>3. Second adjusted model eliminating the variables “<em>coarse_aggregate</em>” and “<em>fine_aggregate</em>”</strong></p><pre>model2 = ols(<br>    &#39;strength ~ water + cement + superplasticizer + fly_ash + blast_furnace_slag + age&#39;,<br>    data=df)<br>df_result2 = model2.fit()<br>print(df_result2.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*Pf8FXF80ZvtfSXbZFo8O4w.png" /><figcaption>Statistical output of the second regression model</figcaption></figure><p>By removing the two variables mentioned above, you can see that the new VIF calculation for the remaining variables was lower, meaning we now have less multicollinearity, although it still exists. Further investigation is needed to completely remove this multicollinearity, but the idea here was simply to show how to do it and analyze it in practice, so we’ll conclude here.</p><p>Note that even removing the two variables “<em>coarse_aggregate</em>” and “<em>fine_aggregate</em>,” our R2 barely changed, and the standard errors of the remaining variables decreased significantly, including the intercept.</p><p><strong>Finally, remember</strong>: each analysis is different from the other, use the necessary tools to direct your analysis to get best results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0fd379c8dab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Calibration]]></title>
            <link>https://medium.com/@vitorbeltrao300/model-calibration-a92c6f9ea910?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a92c6f9ea910</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 21:15:20 GMT</pubDate>
            <atom:updated>2025-01-16T21:15:20.510Z</atom:updated>
            <content:encoded><![CDATA[<p>Assuming that the model gave a probability of 10% to 40% for some instance of the data set. How do you know how good/reliable these probabilities are?</p><blockquote>This is where calibration comes in to help us! Performing calibration is essential to obtain sensible and interpretable probabilities as outputs.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*gwKUctRyqcPV-o8v.jpg" /><figcaption>credit: <a href="https://gnbsgy.org/">https://gnbsgy.org/</a></figcaption></figure><p>Without any calibration, the model outputs cannot be interpreted as true probabilities.</p><h4>How to interpret the output of a calibrated model?</h4><p>Let’s assume we have a cat/dog image classifier.</p><p>Ideally, if the model outputs a probability whose value is 0.10, what we would like this to represent is the fact that if we took 10 images of this same photo, then in reality 1 in 10 of those photos will be dog photos (or cat) truly.</p><p><strong>This is exactly how we should interpret the outputs of a calibrated model!</strong></p><h4>How to interpret the output of a not calibrated model?</h4><p>In other hand, if the model is not calibrated, we should not expect this probability (0.10) to mean that 1 of the 10 photos will actually be dog (or cat) photos.</p><h4>When not to calibrate models?</h4><p>Calibrating an ML model by adjusting the predicted probability to better reflect the actual probabilities is often useful in contexts where important decisions depend on these probabilities.</p><p>However, there are situations where it makes no sense or is unnecessary to calibrate a model. An example of these situations is when we need to <strong>make decisions based on classes</strong>.</p><p>Imagine you are training a model for binary classification with the goal of categorizing text messages as “spam” or “not spam”. <strong>The ultimate goal of the model is just to predict the correct class and not worry about the associated probability</strong>.</p><h4>When to calibrate models and why can it be critical?</h4><p>When we must make important decisions based on output probabilities, we must calibrate the models.</p><p>Calibration can be critical in the following situations:</p><ul><li><strong>Reliability of predictions</strong></li></ul><p>Uncalibrated models can lead to predictions that do not correspond to reality. For example, models can overestimate the probability of events, leading to misinterpretation of results.</p><ul><li><strong>Risk-based decisions</strong></li></ul><p>In business environments such as finance, healthcare or insurance, decisions based on the predictions of an uncalibrated model can result in large financial losses or health risks. For example, a model that incorrectly predicts the probability of an adverse event can lead to inappropriate decisions. In these cases, calibration is essential to ensure that mitigation strategies and feedback considerations are accurate.</p><ul><li><strong>Model efficiency</strong></li></ul><p>Calibrated models tend to be more effective and useful in their predictions, increasing confidence in decisions made based on these models.</p><p>Calibrating machine learning models is a crucial step to ensure predictions are not only accurate but also reliable. By aligning predicted probabilities with observed reality, we increase confidence in our models, promoting more informed decisions. <strong>Understand when calibration of your models is crucial and when it is not</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a92c6f9ea910" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classificador naive Bayes]]></title>
            <link>https://medium.com/@vitorbeltrao300/classificador-naive-bayes-98e380127e5e?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/98e380127e5e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[matemática]]></category>
            <category><![CDATA[estatistica]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Thu, 02 Jan 2025 12:53:48 GMT</pubDate>
            <atom:updated>2025-01-02T12:53:48.330Z</atom:updated>
            <content:encoded><![CDATA[<ul><li>Se P(y1 | X) &gt; P(y2 | X), então classifique X ∈ y1.</li><li>Se P(y2 | X) &gt; P(y1 | X), então classifique X ∈ y2.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*gyi0VOUS0FJRIvTUNGGSsg.png" /></figure><p>Temos duas classes equiprováveis (y1 e y2) e definimos a região de separação como X0. Qualquer ponto que cair a esquerda de X0 terá maior probabilidade de pertencer a classe y1 e qualquer ponto que cair a direita de X0, terá maior probabilidade de pertencer a classe y2.</p><p>Na decisão bayesiana, a <strong>principal limitação</strong> é que na maioria das vezes não conhecemos a distribuição de probabilidade conjunta. Estimá-la é muito complicado, principalmente se a dimensão dos dados é muito alta.</p><p>É aqui que entra o naive Bayes, <strong>assumimos que as variáveis são independentes</strong>, o que torna tudo mais fácil e é por conta disso que vem o nome “naive”.</p><p>No algoritmo naive Bayes, a classificação é feita utilizando o teorema de bayes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*ZIwiX3buQfvnWN7AOq33yQ.png" /><figcaption>Teorema de Bayes</figcaption></figure><p>A função de máxima verossimilhança é dada por:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*3CabopvcLJJEonhq3urqMg.png" /><figcaption>Máxima verossimilhança</figcaption></figure><p>Se temos diversos atributos X1, X2, …, Xn; para cada um desses atributos vamos estimar uma função de probabilidade e multiplicá-los. Dessa forma vamos obter o valor da máxima verossimilhança.</p><p>Se quiser entender a teoria sobre o conceito de máxima verossimilhança, leia este <a href="https://medium.com/@vitorbeltrao300/estimativa-de-m%C3%A1xima-verossimilhan%C3%A7a-5a5b6c31d770">artigo</a>.</p><p>Finalmente, classificaremos a categoria que obteve maior pontuação:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*MdajawRX4giV0ft_AEatSA.png" /><figcaption>Regra de classificação final</figcaption></figure><p>Note que “yi” acima não é uma probabilidade, pois não dividimos pela evidência P(X).</p><p>Não precisamos da evidência P(X) para fazer a classificação, mas se quisermos calcular as probabilidades precisamos dela. Para calcular a evidência, basta:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*yaYwJuWFjxxT7COEjHPGww.png" /><figcaption>Cálculo da evidência P(X)</figcaption></figure><p><strong>Observação</strong>: a probabilidade a priori pode ser qualquer probabilidade que quisermos, geralmente a obtemos a partir do conjunto de dados de treinamento.</p><p>Por enquanto é isso, em breve farei outros materiais com exemplos práticos!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98e380127e5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Estimativa de máxima verossimilhança]]></title>
            <link>https://medium.com/@vitorbeltrao300/estimativa-de-m%C3%A1xima-verossimilhan%C3%A7a-5a5b6c31d770?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a5b6c31d770</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[matemática]]></category>
            <category><![CDATA[estatistica]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Fri, 27 Dec 2024 14:02:56 GMT</pubDate>
            <atom:updated>2024-12-27T14:02:56.352Z</atom:updated>
            <content:encoded><![CDATA[<p>Em estatística e machine learning, usamos um modelo (uma maneira matemática) para descrever o processo que resulta nos dados que são observados.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*5XkDKE_jvNEoa65y.jpg" /><figcaption><a href="https://br.freepik.com/">https://br.freepik.com/</a></figcaption></figure><blockquote>A máxima verossimilhança é um método matemático usado para descobrir quais são os valores (parâmetros) mais prováveis de terem causado os dados que você observou. É como tentar adivinhar as regras de um jogo olhando apenas para os resultados que aconteceram.</blockquote><p>Em outras palavras, a estimativa de máxima verossimilhança é um método que determina valores para os parâmetros de um modelo. Os valores dos parâmetros são encontrados de modo que <strong>maximizam</strong> a probabilidade de que o processo descrito pelo modelo tenha produzido os dados que foram realmente observados.</p><p>Vamos comentar rapidamente sobre dois exemplos práticos muito úteis para ilustrar o que escrevemos acima.</p><h4>1. Distribuição normal</h4><ul><li><strong>Objetivo</strong>: estimar os parâmetros de uma distribuição normal (μ, σ) que melhor explicam os dados observados.</li><li><strong>Dados</strong>: valores contínuos (altura, peso, temperatura, etc).</li><li><strong>Função de verossimilhança</strong>: calcular a probabilidade total de observar todos os dados, ou seja, a distribuição de probabilidade conjunta de todos os pontos de dados observados (x1, x2, …, xn). Para um único ponto de dados, essa é a função:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*cwX7Nl7XT_KJCaw7v3cP9g.png" /></figure><p>Caso queira calcular para vários pontos (o que sempre é o caso), basta fazer o produtório dos pontos individuais usando a fórmula acima.</p><ul><li><strong>Cálculo</strong>: a máxima verossimilhança ajusta μ e σ para <strong>maximizar</strong> a probabilidade de observar os dados fornecidos. Para fazer isso, utilizamos diferenciação, que nos ajuda a encontrar máximos e mínimos de funções.</li></ul><h4>2. Regressão logística</h4><ul><li><strong>Objetivo</strong>: estimar os coeficientes de um modelo (θ) que prevê a probabilidade de uma observação pertencer a uma classe específica y ∈ {0, 1}, baseado em variáveis explicativas (x1, x2, …, xn).</li><li><strong>Dados</strong>: valores categóricos para a variável alvo (y) e valores contínuos e/ou categóricos para as variaveis explicativas (X).</li><li><strong>Função de verossimilhança</strong>: mede a probabilidade de observar as classes y dado as entradas X. Para uma amostra (Xi, yi):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*yL1CtyYLM3_Az-EG8fu3aA.png" /><figcaption>l(b) é o log da máxima verossimilhança (é utilizado o log para simplificar os cálculos).</figcaption></figure><ul><li><strong>Cálculo</strong>: a máxima verossimilhança ajusta os coeficientes (θ) para maximizar a probabilidade de obsevar as classes reais (y) dos dados, dadas as predições do modelo.</li></ul><p>Diferente da distribuição normal, aqui a máxima verossimilhança não ajusta parâmetros de uma distribuição (como no caso da normal); em vez disso, ajusta coeficientes para criar um modelo que melhor separa as classes com base nas variáveis explicativas.</p><p>Em resumo, a máxima verossimilhança pode ser usada em diferentes contextos:</p><ul><li>Ajustar a forma da distribuição dos dados.</li><li>Ajustar um modelo preditivo.</li></ul><p>Aqui vimos exemplos desses dois contextos, mas podemos utilizar em vários outros.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a5b6c31d770" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applied stats: Geometric probability distribution]]></title>
            <link>https://medium.com/@vitorbeltrao300/applied-stats-geometric-probability-distribution-ac5af115279c?source=rss-9319367687b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac5af115279c</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vitor Beltrão]]></dc:creator>
            <pubDate>Tue, 16 Jul 2024 10:37:37 GMT</pubDate>
            <atom:updated>2024-07-16T10:37:37.148Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*aLQ0hjJei8QtH3C-qQ6QcQ.png" /><figcaption>Graph of a geometric distribution</figcaption></figure><h4>Geometric distribution: theory</h4><p>For a binomial distribution, we are looking at the number of successes in a finite number of attempts. For more details on the binomial distribution, see one of my <a href="https://medium.com/@vitorbeltrao300/applied-stats-probability-distributions-binomial-95276bdcab54">articles</a>.</p><p>For a <strong>geometric random variable</strong>, most of the conditions we put on the binomial random variable still apply:</p><ul><li>each trial must be independent.</li><li>each trial can be called a “success” or “failure”.</li><li>the probability of success on each trial is constant.</li></ul><p>The difference is that for a geometric random variable, we’re looking at how many trials we have to use <strong>UNTIL </strong>we get a certain success.</p><p>To find the probability that a success S occurs on the nth attempt, when a success has a probability of p, and therefore a failure has a probability of 1 − p, we use this formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*0ndIIWHiL-lyQdV4MONmNQ.png" /></figure><p>If we look closely at this formula, we see that we’re really just multiplying the probability of failure over and over again until the trial right before we have a success, and then multiplying by the probability of a success.</p><p>In other words, if we want to find the probability that we get our first success on the 7th trial, then the probability will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*RRDDztc1VhCf4p5EOAlyhA.png" /></figure><p>Notice that the exponents add to 7, since we needed 7 trials to get the first success.</p><p>Let’s study some particularities of this probability distribution:</p><ul><li><strong>Less than</strong></li></ul><p>Sometimes we can be asked to find the probability that it takes less than a specific number of trials in order to get our first success. For instance, continuing with the example we just worked through, we could be asked to find the probability that it takes us less than 4 games to win a prize.</p><p>This is the same as saying that we win a prize on game 1, 2, or 3. If we call a success S, that means we want either S &lt; 4 or S ≤ 3, which mean the same thing in the case of a geometric random variable.</p><p><strong>P(S &lt; 4) = P(S = 1) + P(S = 2) + P(S = 3)</strong></p><ul><li><strong>At most</strong></li></ul><p>This is slightly different than being asked the probability that it takes us less than 4 games to win a prize. In this case, the only difference is that we included P(S = 4), which had not been included above.</p><p><strong>P(S &lt; 5) = P(S = 1) + P(S = 2) + P(S = 3) + P(S = 4)</strong></p><ul><li><strong>More than</strong></li></ul><p>Continuing with the same example, we could be asked to find the<br>probability that it will take more than 2 games to win a prize.</p><p>Remember that all probability distributions add to 1. If we’re looking for the probability that it takes more than 2 trials to win a prize, we can find the probability of winning on the first trial and the probability of winning on the second trial, and then subtract those probabilities from 1, which will give us all the total probability of all outcomes, other than winning on the first or second game.</p><p>So the probability that it takes more than 2 games to win is:</p><p><strong>P(S &gt; 2) = 1 — P(S &lt; 2)</strong></p><ul><li><strong>At least</strong></li></ul><p>If it takes us at least 2 games to win, that means we could win a prize in the second game. We could write this as S &gt; 1 or as S ≥ 2. But either way, we failed once and then succeeded sometimes in the second game or later.</p><p><strong>P(S ≥ 2) = 1 — P(S ≤ 1)</strong></p><p>We can calculate the expected value and standard deviation to make more accurate predictions, in this case:</p><ul><li>E(x) = 1 / p.</li><li>σ² = (1 — p) / p².</li></ul><h4>Geometric distribution: practice</h4><p>Let’s start with some examples where you could apply this probability distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0ENE6Y44Uct3TX4rphqUg.png" /></figure><p><strong>Example 1</strong></p><p>A lamp factory finds 3 out of every 60 lamps defective. What is the probability that the first defective bulb will be found when the 6th is tested?</p><pre>from scipy.stats import geom<br><br>p = 3/60<br><br># probability of the 6th defective lamp<br>print(geom.pmf(k=6, p=p)) # In this case, the 1st to 5th are not defective<br><br># Probability that it will take us LESS THAN 6 tries to find a defective one<br>print(geom.cdf(5, p)) # P(S &lt; 6) = P(S = 1) + P(S = 2) + P(S = 3) + P(S = 4) + P(S = 5)<br><br># probability that it will take MORE THAN 6 attempts to find a defective one<br>print(1 - poisson.cdf(7, p)) # P(S &gt; 6) = P(S = 7) + P(S = 8) + P(S = 9) + ...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Cz2D4oW0WKjuumcq0JrPNA.png" /><figcaption>Output of the above code.</figcaption></figure><p><strong>Example 2</strong></p><p>Probability of fraud in a bank’s transactions.</p><p>I collected a dataset from kaggle and it contains the transactions of a bank and one of the variables is whether that transaction is a fraud or not. Let’s see the frequency distribution of this variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*iEWUEEK_C5yQB4sIo_swrQ.png" /><figcaption>Frequency distribution of the fraud variable. Label 0 is non-fraud and label 1 is fraud.</figcaption></figure><pre>probability_of_non_fraud = (value_counts[0] / len(df_fraud))<br>probability_of_fraud = (value_counts[1] / len(df_fraud))<br><br>print(&#39;The probability of not having a fraud is:&#39;, probability_of_non_fraud)<br>print(&#39;The probability of having a fraud is:&#39;, probability_of_fraud)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*omRAaMUnN1D7lw1NyP34ew.png" /><figcaption>The output of the above code.</figcaption></figure><p>A) What is the probability that the 100th transaction is fraudulent?</p><pre># in this case, from the 1st to the 99th are normal transactions<br>print(geom.pmf(k=100, p=probability_of_fraud))</pre><p>B) What is the probability that it will take us LESS THAN 10 tries to find a fraud?</p><pre># P(S &lt; 10) = P(S = 1) + P(S = 2) + P(S = 3) + ... + P(S = 9)<br>print(geom.cdf(k=9, p=probability_of_fraud))</pre><p>C) What is the probability that it will take LESS THAN 100 attempts to find a fraud?</p><pre># P(S &lt; 100) = P(S = 1) + P(S = 2) + P(S = 3) + ... + P(S = 99)<br>print(geom.cdf(k=99, p=probability_of_fraud))</pre><p>D) Probability of find a fraud until the 100th transaction</p><pre>p = probability_of_fraud<br>k = np.arange(0, 100)<br>pmf = geom.pmf(k, probability_of_fraud)<br><br>fig = go.Figure(data=[go.Bar(x=k, y=pmf)])<br>fig.update_layout(title=f&quot;Geometric Distribuition: probability of find a fraud until the 100th transaction&quot;,<br>                  xaxis_title=&quot;Number of attempts in which the 1st success occurs&quot;,<br>                  yaxis_title=&quot;Probability&quot;,<br>                  font=dict(size=18),<br>                  width=700,<br>                  title_x=0.5,<br>                  height=400,<br>                  template=&quot;simple_white&quot;)<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*B35t1NyGPmeB1oaY2NnKOA.png" /><figcaption>The output of the above code.</figcaption></figure><p>Now it’s up to you. Use what you learned to apply it to your daily life and better understand the phenomena!</p><p><strong>Credits</strong></p><ul><li><a href="https://renatabiaggi.com/e-b-a/">E.B.A - Renata site</a></li><li><a href="https://www.kristakingmath.com/?source=post_page-----1b50f8a2b33d--------------------------------">Krista King Math | Online math help</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac5af115279c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>