<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naomy Duarte Gomes on Medium]]></title>
        <description><![CDATA[Stories by Naomy Duarte Gomes on Medium]]></description>
        <link>https://medium.com/@naomy-gomes?source=rss-6d5e8ccd9890------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Xq9K0zNSND35slp9S78DHA.jpeg</url>
            <title>Stories by Naomy Duarte Gomes on Medium</title>
            <link>https://medium.com/@naomy-gomes?source=rss-6d5e8ccd9890------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:05:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@naomy-gomes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The cosine similarity and its use in recommendation systems]]></title>
            <link>https://naomy-gomes.medium.com/the-cosine-similarity-and-its-use-in-recommendation-systems-cb2ebd811ce1?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb2ebd811ce1</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[cosine-similarity]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 17:07:25 GMT</pubDate>
            <atom:updated>2023-01-31T17:07:25.043Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Cosine similarity is a metric based on the cosine distance between two objects and can be used in recommendation systems such as movie and book recommenders. In this article, we will learn what it is and how it can be used to make recommendations by identifying similar items.</em></p><p><strong>Introduction</strong></p><p>Recommendation systems are part of our everyday life. Just to cite some common examples, we get recommendations when we shop online on Amazon and when we open Youtube or Netflix. The purpose of these systems is to recommend what you may want to buy based on what you have in your bag or what video or movie to watch next based on the ones you already watched.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*4WaLD1ztbWJ1ELVoExLMFw.png" /><figcaption>Fig. 1 — Similarities. Image from Shutterstock.</figcaption></figure><p>A recommendation system can be as complicated as we want, such as the ones that use deep learning [1,2], but they can also be simple and based on the similarity between items. And one way to calculate the similarity between items is with the use of <strong>cosine similarity</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*MGRKAv-Wx38DHncco3Vl1g.png" /><figcaption>Fig. 2— The “frequently bought together” recommendations at Amazon.</figcaption></figure><p><strong>The cosine between two vectors</strong></p><p>To understand this metric of similarity, we first need to understand some concepts. Suppose we have a table of books 1 and 2 containing their genre, according to Fig. 3. To each word in the genre table, we create another column in a second table, where if the word is in the genre, we give it 1, if it&#39;s not, then is 0. Since we have the genres Science Fiction and Fiction, we create another table with these two words. If we draw a graph where the x-axis is the Science axis and the Y-axis is the Fiction axis, we can associate a point to each book. For example, book 1 will be the blue point with a Sience-axis of 1 and a Fiction-axis of 1 (Science Fiction). Book 2 will be the yellow point with a Sience-axis of 0 and a Fiction-axis of 1 (Fiction). We draw a vector from the origin to the points, which we call the book-vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XS20WwxcOzu01mpb1YkuqQ.png" /><figcaption>Fig. 3— Book table containing genres and a graph showing the book-vectors.</figcaption></figure><p>Now, we can see that the book-vectors form an angle θ with each other. The cosine of this angle is our measure of similarity, and it is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*KGMVoFyTkcPaoR1-JyMCMg.png" /></figure><p>where <strong>A</strong> and <strong>B</strong> are the vectors we are considering, ||<strong>A</strong>|| and ||<strong>B</strong>|| are their norm (length). The <strong>A</strong>i and <strong>B</strong>i in the formula are the components of each vector. Book-vector 1 is (1,1), and book-vector 2 is (0,1). Let’s calculate the cosine similarity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*ecQmQhk0m-iSqzY3fiREUQ.png" /></figure><p>which says two things: first, these vectors have some similarity, and second, θ is 45º, something that we already expected and could calculate by using the Pythagorean theorem and calculating the cosine using the sides of the triangle.</p><p>If both books were Science Fiction, we would have the same book-vectors (1,1) and the cosine would be 1, meaning they are the same. But if book 1 were Science Fiction (1,1) and book 2 Terror (0,0), in this case, they would have nothing in common, and the cosine would be 0. Therefore, <em>high similarity means a cosine close to 1, and low similarity, a cosine close to 0</em><strong><em>.</em></strong></p><p><strong>Calculating using Python</strong></p><p>We can cite at least two ways to calculate the cosine similarity between two given vectors. One is using numpy:</p><pre>import numpy as np<br>from numpy.linalg import norm <br><br>A = np.array([1,8])<br>B = np.array([9,2])<br><br>cos_sim = np.dot(A,B)/(norm(A)*norm(B))<br>print(f&quot;The cosine similarity is: {round(cos_sim,2)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*KqUxmE55PfL-H6315pLrJQ.png" /></figure><p>Another way is using sklearn:</p><pre>from sklearn.metrics.pairwise import cosine_similarity<br><br>cos_sim = cosine_similarity(A.reshape(1, -1),B.reshape(1, -1))<br>print(f&quot;The cosine similarity is: {cos_sim}&quot;)</pre><p>Either way, calculating the cosine similarity is straightforward and, if done by hand, only requires a simple mathematical formula.</p><p><strong>Application as a book recommender</strong></p><p>In <a href="https://github.com/naomyduarteg/LIA-FastAPI-SQL">this</a> repository, I utilize the cosine similarity to build a book recommender.</p><p><a href="https://github.com/naomyduarteg/LIA-FastAPI-SQL">GitHub - naomyduarteg/LIA-FastAPI-SQL: LIA is a Library In API created with Python, FastAPI and SQLite to save information about the books you read and get book recommendations</a></p><p>For this project, the main idea is that a user will enter the books she/he read and classify them from 1 to 5, with 5 being the highest grade, thus creating a table that contains the columns: book title, author, description, genre, pages, and classification. Once the table is created, we convert it to pandas dataframe, then normalize using the function <em>normalize(data)</em> and one-hot encode the categorical columns using the function <em>ohe(df, enc_col)</em> (which is basically what we did when we create the other table in Fig. 2). It is important to normalize numerical columns since big numbers can bias the results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c267510e0b9b1c80a7349639d32d440e/href">https://medium.com/media/c267510e0b9b1c80a7349639d32d440e/href</a></iframe><p>The function <em>recommend(book_id, owner_id, df) </em>will calculate the cosine similarity between the book inputted by the user and all the other books in the table, creating a column with these values and then selecting the 10 most similar books to recommend to the user. The cosine similarity calculation itself is pretty straightforward and is defined by the function <em>cosine_sim(v1, v2), s</em>imilar to what we did with numpy, the only difference here is that the vectors have more than 2 dimensions.</p><p>You can run the API and test the recommendations.</p><p><strong>Conclusions</strong></p><p>The cosine similarity gives a useful measure of how similar two objects are. It is a rather simple mathematical concept and easy to implement computationally. It can be used for many purposes: in machine learning as a distance metric, with textual data to compare two documents, and in recommendation systems.</p><p><strong>References</strong></p><p>[1] Da’u, A., Salim, N. Recommendation system based on deep learning methods: a systematic review and new directions. <em>Artif Intell Rev</em> <strong>53</strong>, 2709–2748 (2020). <a href="https://doi.org/10.1007/s10462-019-09744-1">https://doi.org/10.1007/s10462-019-09744-1</a></p><p>[2] Schedl, M. (2019). Deep Learning in Music Recommendation Systems. <em>Frontiers in Applied Mathematics and Statistics</em>, <em>5</em>. <a href="https://doi.org/10.3389/fams.2019.00044">https://doi.org/10.3389/fams.2019.00044</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb2ebd811ce1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CRUD with Python, FastAPI, and MongoDB]]></title>
            <link>https://naomy-gomes.medium.com/crud-with-python-fastapi-and-mongodb-e830c6c538f4?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/e830c6c538f4</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Thu, 27 Oct 2022 20:03:58 GMT</pubDate>
            <atom:updated>2023-02-07T18:03:36.090Z</atom:updated>
            <content:encoded><![CDATA[<p><em>CRUD (Create, Read, Update, Delete) operations are the basic operations a software application must perform. In this tutorial, we will learn how to use the FastAPI framework to construct a REST API using Python as the programming language and the non-relational database MongoDB.</em></p><p><strong>Introduction</strong></p><p>When entering the world of software applications, we often encounter <strong>CRUD</strong> <em>(</em><strong><em>C</em></strong><em>reate, </em><strong><em>R</em></strong><em>ead, </em><strong><em>U</em></strong><em>pdate, </em><strong><em>D</em></strong><em>elete)</em> operations, for they are the basic operations one must learn to start developing software. In such apps, the user must be able to <em>create data</em> that will be stored in a database, retrieve it from the database and <em>read it, update </em>the data when needed and <em>delete it</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZa9Bopf1LULnsvFEilk5g.png" /></figure><p>For example, consider an application to sell books. We need a mediator to perform the transfer of information between the seller and the buyers. An <strong>API</strong> (<strong>A</strong>pplication <strong>P</strong>rogramming <strong>I</strong>nterface) is such a mediator, as it is the interface that contains the code and methods that allows communication between those two parties. The information about the books (ex. title, author, quantity available) must be stored in a database and a user interface allows the user to interact with the app.</p><p>In this project, we will focus on how to build a REST API for selling books using the FastAPI framework [1] with Python, and store the data about the books and the users who will buy them in the non-relational database MongoDB [2], without the interactive interface for the user (front-end).</p><p><strong>Setting up the environment</strong></p><p>Clone the <a href="https://github.com/naomyduarteg/CRUD_python_mongo">repository</a>. Enter the project directory. When in it, create a Python virtual environment to install all dependencies needed without any conflicts with other packages versions you may already have:</p><pre>python3 -m venv .venv</pre><p>Activate the virtual environment. If you are a Windows user:</p><pre>.venv/Scripts/activate</pre><p>If you are a Linux/Apple user:</p><pre>.venv/bin/activate</pre><p>And now install the dependencies:</p><pre>pip install -r requirements.txt</pre><p><strong>Setting up the structure</strong></p><p>It is a good practice to divide our projects into folders (which will be the <strong>packages</strong>) and files (which will be the <strong>modules</strong>) so it is better organized and easy to deal with (see Python’s <a href="https://docs.python.org/3/tutorial/modules.html">modules</a>). The structure of the project is as follows:</p><pre>├── CRUD_python_mongo<br>│   │── .env<br>│   ├── src<br>│   │    ├── __init__.py<br>│   │    ├── rules<br>│   │    │      ├── __init__.py<br>│   │    │      ├── books.py<br>│   │    │      ├── addresses.py<br>│   │    │      └── users.py<br>│   │    │<br>│   │    ├── endpoints<br>│   │    │      ├── __init__.py<br>│   │    │      ├── books.py<br>│   │    │      ├── addresses.py<br>│   │    │      └── users.py<br>│   │    │<br>│   │    ├── models<br>│   │    │      ├── __init__.py<br>│   │    │      ├── books.py<br>│   │    │      ├── addresses.py<br>│   │    │      └── users.py<br>│   │    <br>│   ├── routes<br>│   │    ├── __init__.py<br>│   │    └── api.py<br>│   ├── __init__.py<br>│   ├── main.py<br>│   └── requirements.txt</pre><p>The <em>.env</em> (it appears as <em>.env.example</em> at the GitHub repository. Change it to <em>.env</em>. This happens because the gitignore file ignores this kind of file for safety) contains the variables that will make the connection to the MongoDB Atlas cluster and create the database. You will need to create an account at MongoDB Atlas and get your <a href="https://www.mongodb.com/docs/atlas/tutorial/connect-to-your-cluster/?_ga=2.242666818.130471116.1666027758-870539579.1663515625#connect-to-your-atlas-cluster">URI</a> to connect. Here, the database is called “pymongo_tutorial”, but feel free to choose the name you like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*oZBDkM5_dRRA_HER-3v6Hg.png" /></figure><p>The <em>main.py</em> is the file we run using <a href="https://www.uvicorn.org/">uvicorn</a> (one of the dependencies in the <em>requirements.txt</em> file). It is also the file where we load the variables ATLAS_URI and DB_NAME to connect to the database and where we register the endpoints through the <em>include_router </em>function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*zaBWA2Kk90glaSwOavnoQQ.png" /></figure><p>In the<em> routes</em> folder, there is the <em>api.py</em> file where the routes will be defined for each one of the objects we are working with: books, users, and users’ addresses. Different routes allow us to separate prefixes to the path operations, tags, responses, and dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*bUbNJMyQcFO2aqdOZRufhw.png" /></figure><p>The <em>__init__.py</em> at each folder is used “to make Python treat directories containing the file as packages” [3], which means you can import the modules (the files .py inside each folder). See more about paths <a href="https://docs.python.org/3/library/sys.html#sys.path">here</a>.</p><p><strong>Models</strong></p><p>This subpackage contains the models that define the objects we are working with. The definitions are made by creating a class that inherits from<a href="https://pydantic-docs.helpmanual.io/usage/models/"> pydantic’s BaseModel</a> class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xo4Qeh4aBVYO6irq89zhUA.png" /></figure><p>As we are working with books and users, we must define the structure of the data that represent these objects. Let’s start by considering books. By writing the <em>class Books</em> we define that each book registered must have an id, title, description, price (a value greater than zero), pages (also greater than zero), and genre. If the price or pages is zero or less, an error will be returned. With the use of the decorator @<em>validator </em>we can define that genre must be either Fiction or Non-fiction and that anything different will raise an error (another option is to use <a href="https://pydantic-docs.helpmanual.io/usage/types/#enums-and-choices">Enum</a>, try it yourself). In the <em>class Config, </em>the<em> schema_extra </em>is used to define an example of the data that can be received and will be stored in the MongoDB database.</p><p>We can also write a class to update book information. As updating that information is not mandatory, we define the fields as Optional (this code is also in the models/books.py).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UvqwH0zLlnfKisOBQxcldg.png" /></figure><p>Similar code can be written for users in the models/users.py</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N0g8YhabKdcooT2rvfk5Pg.png" /></figure><p>Users must have an id, name, email, and password. The <em>EmailStr </em>requires the input to be a valid email, and <em>SecretStr </em>is used to store sensitive information such as passwords, formatting them as ‘**********’.</p><p>We can also define a class for users’ addresses, which will be in models/addresses.py:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KbbkM04fn6lwN7J4Zlj6w.png" /></figure><p>Addresses registered by users must have an id, a name for the address (such as House, Work, etc), and the id of the user so the address is correctly associated, street, number, city, state, and code.</p><p><strong>Rules</strong></p><p>This is the subpackage where we define the functions that will be used as rules to create, read, update and delete documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXeSeD7e4kQ5SehtZnEoWg.png" /></figure><p>In the example above, we have the following functions:</p><ul><li><em>get_collection_users: </em>accesses the users&#39; database in MongoDB.</li><li><em>create_user</em>: inserts users’ information using the method <em>insert_one</em> of pymongo and shows it using <em>find_one</em>.</li><li><em>list_users</em>: shows a list of users by using the method <em>find</em>, where the number of documents shown is defined by <em>limit</em>.</li><li><em>find_user</em>: finds the user by its ID and returns it. If the user does not exist, returns an error.</li><li><em>delete_user</em>: deletes user by its ID by using the method <em>delete_one</em> and returns a message of success. If the user is not found, returns an error.</li></ul><p>Similar functions are defined in rules/addresses.py and rules/books.py, with the addition of the update function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oMmfEn2EZ5_M7OIveR7Uvg.png" /></figure><p>This function updates the book’s information by using the <em>update_one </em>method and the MongoDB <em>$set</em> operator. The id of the book to be updated is asked as well as the information to be updated (for example, the book’s price). It will then loop over the information already stored in the database and update the requested information.</p><p>In rules/books.py there’s also the option to list books by authors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOpQh1S4JUc9x6jZlRIQ1A.png" /></figure><p>This function uses the MongoDB <a href="https://www.mongodb.com/docs/manual/aggregation/">aggregation operation</a>, where <em>$match</em> will filter documents by the field author.</p><p>It is also important to remember to import the required modules to the rules package, such as User, UserAddress, UpdateAddress, Books, and UpdateBooks.</p><p><strong>Endpoints</strong></p><p>This is the subpackage where the HTTP methods corresponding to the CRUD operations will be defined. Let’s consider the addresses endpoints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9NM8uFpzhdTD7eyI4bX1g.png" /></figure><p>We first must import the modules we’ll use: <em>UserAddress</em>, <em>UpdateAddress</em>, and <em>addresses</em> from rules/addresses.py. Next, we define the route, which is <em>address</em>.</p><p>The first method used is the POST method, which creates user addresses. The GET method reads information, and it will be used to read a list of addresses and to get the user’s addresses by informing the user id. The PUT method updates the address and the DELETE method deletes an address. The same logic goes for books and users.</p><p><strong>Swagger documentation</strong></p><p>FastAPI provides the Swagger documentation of the API where we can test it. To access the documentation, we must run uvicorn:</p><pre>uvicorn main:app --reload</pre><p>If everything works fine, the message “Project connected to the MongoDB database!” will appear. Go to <a href="http://127.0.0.1:8000/docs">http://127.0.0.1:8000/docs</a> and you’ll see</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRlCY3j9beqSsP5jygH3Rw.png" /></figure><p>It is possible to make some tests. Let’s see some tests with books.</p><ul><li>Creating a book</li></ul><p>The request body already has the model provided by the <em>class Config </em>that can be filled accordingly<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyZKi7nJ0yPUXzfgyKlmeA.png" /></figure><ul><li>Reading books</li></ul><p>We can read a list of books at get/books/, read books by id at get/books/{id}/ and read books by author at get/books/{id}/{author}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*INps1HfNhBrpAiqJ5Zl2Xw.png" /></figure><ul><li>Update books</li></ul><p>We can update books by their id. In the example below, the price is updated to 20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9F4C_28Zfl8iGuA9yUw3sQ.png" /></figure><p>The answer to the update is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmUGR7SDitkGz3TAUiTbag.png" /></figure><ul><li>Delete books</li></ul><p>Finally, there’s the option to delete books by id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7y77d4iOR87RLFCHn_4fog.png" /></figure><p><strong>Visualizing MongoDB database</strong></p><p>It is now time to go to the MongoDB database! This can be done either by signing in to your mongo account at <a href="https://www.mongodb.com/">https://www.mongodb.com/</a> or by downloading and opening MongoDB Compass (and using the URI used at ATLAS_URI to connect to the database). In this last case, once connected, go to Database and search for the project’s corresponding database, which is <em>pymongo_tutorial</em>. Once there, and once documents for each of our objects (users, books, and addresses) were created, we must see the three collections:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NhyrBof7Ipusr48W2e3K6w.png" /></figure><p>Inside each collection are the documents created. At the <em>users</em> collection we have this example of a user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4--UcoW3dzZRLPw-m6fc1Q.png" /></figure><p>Back when we defined our classes at <em>models</em>, each class had an <em>_id. </em>This id is created by MongoDB as we call the GET method and fill each field. Therefore, the ObjectId we see is the id created by MongoDB and it will be used to update, read or delete objects. MongoDB Compass also offers the option to edit and delete documents.</p><h4>Conclusions</h4><p>This project was an example of a simplified CRUD Rest API using FastAPI and MongoDB. The complete code is in <a href="https://github.com/naomyduarteg/CRUD_python_mongo">this GitHub repository</a>. Clone the repository and play around! (Also, take some time to understand the entire code, as it would be too long of a Medium article to explain everything here). Another possibility worth noticing regarding the creation of API is to write the code using <a href="https://docs.python.org/3/library/asyncio.html">async syntax</a>, which can provide better performance to your API.</p><p><strong>References</strong></p><p>[1] <a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com/</a></p><p>[2] <a href="https://www.mongodb.com/">https://www.mongodb.com/</a></p><p>[3] <a href="https://docs.python.org/3/tutorial/modules.html">https://docs.python.org/3/tutorial/modules.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e830c6c538f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object-oriented programming with Python: core principles and examples]]></title>
            <link>https://naomy-gomes.medium.com/object-oriented-programming-with-python-core-principles-and-examples-ca64473735a8?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca64473735a8</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[object-oriented]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Sun, 11 Sep 2022 16:29:24 GMT</pubDate>
            <atom:updated>2022-09-11T16:29:24.977Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Object-oriented programming (OOP) is a very efficient approach to writing software. Its main difference concerning procedural programming is that POO has procedures applied to each object instead of globally, allowing the code to be more independent and therefore reusable.</em></p><h4>Introduction</h4><p>When it comes to programming languages, they are classified according to their features, according to what they allow and do not allow us to do. This classification is called <em>paradigms. </em>There is the <em>procedural paradigm</em>, where the procedure is a sequence of instructions that will execute following the steps informed. The purpose of this paradigm is to dictate sequences of actions that will be executed in an orderly manner and produce the desired result. It is simple, straightforward, and has a code that&#39;s easy to follow, but it’s <em>not reusable</em>. The object-oriented paradigm, on the other hand, defines procedures that are applied to specified objects, such that each part of the code will run according to them, thus allowing reusability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*AVOWlVkC6OrD1YDTjmQf3A.png" /></figure><p>In the OOP we have two important concepts to be defined: <em>class</em> and <em>object</em>. A class is a set of properties that is common to certain types of objects. For example, we can define a class that will cover all the dogs, which are animals with well-developed noses, four legs, a tail, and that bark. The objects of this class are the dogs, which have in common all these characteristics but can have specific characteristics themselves, such as the color of the furr, the size, etc. Objects are the basic unit of OOP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*h3xkF-m18PcTX9tY76swEw.png" /><figcaption>Fig. 1: Creating the Dogs class, its attributes and methods, and the object matilda.</figcaption></figure><p>In fig. 1 we can see the creation of the Dogs class. It has attributes and methods that will be common to all objects. The attributes are the data relevant to the class. Methods are functions that belong to a class and can only be called by objects.</p><p>At the bottom of the code, the object ‘matilda’ is created and contains the name, the age, and the color of the furr that are characteristic of it (characteristic of the dog, in this case). We also call the objects <em>instances of the class.</em></p><p>Now that we know what are classes and objects, we can talk about the core principles of OOP, which are: <em>encapsulation, inheritance, polymorphism, and abstraction</em>.</p><h4>Encapsulation</h4><p>Encapsulation is the process of making a certain part of the code private. It encapsulates variables within a class making them inaccessible outside the class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*5U1q2nCoZR6azYSvCJs-fw.png" /><figcaption>Fig. 2: Creating the BankAccount class and defining its attributes and methods.</figcaption></figure><p>In fig. 2, we create the BankAccount class with a private attribute “number”. If one tries to access this private attribute (print(account1.__number)), it will print an error, as opposed to public attributes such as the name. The retrieve_balance method will return the balance of the account.</p><h4>Inheritance</h4><p>Inheritance is the process of making a new class based on an existing one. When we construct a <em>subclass/child class </em>that has the same attributes as another <em>superclass/parent class </em>(and possibly new ones), we can make the first inherit the latter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*g-mR7RH_hJym2vWeK0nMwA.png" /><figcaption>Fig. 3: Creating the child class CrediCard that inherits the BankAccount attributes and methods, but can also define new ones.</figcaption></figure><p>In fig.3, the child class CreditCard inherits the parent class BankAccount. We define this inheritance by writing Childclass(Parentclass) and the super() function. As we can see, the child class will have all the attributes and methods of the parent class plus the “limit” attribute and the new method get_limit that returns the available limit of the credit card.</p><h4>Polymorphism</h4><p>Polymorphism happens when child classes alter/override the behavior of the methods they inherit, adapting them to their necessities. Therefore, one method can have more than one structure and functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*njtu7rCYKcvTQse1NBnbZA.png" /><figcaption>Fig. 4: The child class overrides the method retrieve_balance to better adapt its necessities.</figcaption></figure><p>In fig. 4, the inherited method retrieve_balance of CreditCard was altered to return the balance, the limit, and the total that can be spent by the client considering the balance of the account + limit. This method is polymorphic as it has different functions in the parent class and the child class.</p><h4>Abstraction</h4><p>Abstraction happens when we create an abstract class that will not have its attributes and methods implemented, meaning that it cannot be instantiated. Therefore, if we want to instantiate its methods and attributes, we will need to create a child class. In Python, we create an abstract class by using the <em>ABC (abstract base class) </em>module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*SEg47NxicFt4IPyYlu6y2w.png" /><figcaption>Fig. 5: Creation of an abstract class Books that cannot be implemented and its child class NovelBooks that can be implemented.</figcaption></figure><p>In fig. 5, we create the abstract class Books which is telling other possible child classes that they will have the attributes title, author, price, and discount, a property that returns the information on the book, and a method to calculate the price considering a discount. Next, we create the child class NovelBooks with the same properties as the abstract class but that can now be instantiated. If we try to instantiate the abstract class, we will receive the message “TypeError: Can’t instantiate abstract class Books with abstract method get_price”.</p><p>You may now be wondering why use abstract classes. From the example above, we can see it is useful when we want to define a structure for other classes to follow, therefore ensuring consistency in how other classes will be implemented. This can be a good practice when working in a team, to ensure a pattern for subclass creation.</p><h4>Conclusion</h4><p>We can see how efficient OOP can be. It allows us to reuse code and define structures that must be followed, ensuring consistency. We can define many classes and subclasses as we think necessary, with the advantages offered by the use of inheritance and polymorphism. It is also possible to assure the security of defined parameters by the use of encapsulation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca64473735a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bias-variance dilemma, overfitting, and underfitting]]></title>
            <link>https://naomy-gomes.medium.com/bias-variance-dilemma-overfitting-and-underfitting-4204f004de5b?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/4204f004de5b</guid>
            <category><![CDATA[bias-variance-tradeoff]]></category>
            <category><![CDATA[statistical-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Tue, 26 Jul 2022 20:05:49 GMT</pubDate>
            <atom:updated>2022-07-26T20:05:49.775Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Bias and variance have a significant role in model performance, potentially leading to overfitting or underfitting. When looking for a good supervised machine learning model, we want to avoid both situations and find the best condition for bias and variance. Therefore, it is substantial to understand bias and variance and how we can best adjust these errors.</em></p><h4>Introduction</h4><p>When writing a supervised machine learning (ML) model, whether it is meant for classification or regression, our goal is that the model predicts the most accurately the target data, especially for new data. For example, consider a model to predict churn. We want it to capture well the relationships between the features and make accurate predictions on clients that it has never seen before, so the bank can take measures in advance to avoid churn.</p><p>Sometimes, our model performs very well on training data and we feel happy and confident with it, but when the test data is provided, the results are disappointing. This is known as <em>overfitting</em>. On the other hand, sometimes it can perform very poorly even on training data. This is known as <em>underfitting</em>. These two situations mean that the error associated with our model’s predictions is high and it won’t be useful with new data. The predictions won’t be reliable and the model is unable to generalize beyond the training set. To understand what is happening to make better models and avoid overfitting or underfitting, we need to comprehend two important concepts: bias and variance.</p><h4>Bias and variance</h4><ul><li>Bias</li></ul><p>Imagine you have a dataset with one feature (x) and one target (y), represented by the blue dots in fig. 1), and you write a model to fit it. You can run it many times with your <em>training dataset</em> and get an average model prediction, due to randomness in the dataset (meaning that the model will predict a range of outcomes and we get the average of these outcomes). <strong><em>The (squared) difference between these average model predictions and the actual values we are trying to predict is bias</em></strong>. It can also be seen as the error introduced in the model when the space of functions to fit is small and thus does not contain the best classifier, biasing the choice.</p><p>So, if your model fits your data highly, like in fig. 1a), then the difference between the average predictions and the true mean will be very small. Thinking about the function space, in this case, it is a large one that allows the algorithm to choose polynomials of a higher order such as the curve in fig. 1a). You have then a low bias model. Your model can capture very well the relationship between feature and target. On the other hand, if your model is a straight line like in fig. 1b), in this case, it isn’t a good fit for your data, doesn&#39;t capture well the relationship between feature and target, and thus you have large differences between the average prediction and the true values. The function space is limited, and therefore your model can’t choose a better function. You have a high bias model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wNw-L7lWuAk_rRZ5lMDQg.png" /><figcaption>Fig. 1 — Different model complexity with a) complicated function with low bias and b) simple function with high bias.</figcaption></figure><ul><li>Variance</li></ul><p>So far we have considered the average predictions and the true values. <strong>If we now look at the predictions for each point and take the squared difference between them and the mean prediction of the model, this is the variance</strong>. Variance is a measure of how much your model’s predictions for each point move around its mean. A high variance model may give good predictions sometimes, but it can also predict absurd values because its predictions vary too much, just like the complex curve in fig. 1a). A low variance model will maintain its predictions consistent, within a small range, like the straight line in fig 1b), but it may not be the right one.</p><p>If we take a look at a graph showing bias and variance as functions of the complexity of the function space, then we’ll see that bias decreases with the increasing complexity of the function space. In this case, our model can choose a high complexity function to fit the data the most accurately possible and is not biased by simple solutions. On the other hand, variance increases with model complexity, because the more complex a model is, the larger the fluctuations in its predictions. The total error in fig. 2 is the sum of the bias and variance errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*5WAkYuDx2LJsOCpTApNJUQ.png" /><figcaption>Fig. 2 — Bias and variance as a function of the complexity of the function space.</figcaption></figure><p>Mathematically speaking about bias and variance:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f64db489e1ce3e862697a4580470085/href">https://medium.com/media/7f64db489e1ce3e862697a4580470085/href</a></iframe><h4>Model complexity</h4><p>As we can notice in fig. 2, bias and variance are competing in model performance. Where bias is low and variance is high, the model is complex and takes into account a lot of parameters. It fits very well the training set but does not generalize well. It is<em> overfitting</em>. Where bias is high and variance is low, the model is simple, but in this case, it does not fit or generalize well. It is<em> underfitting</em>. Bias and variance are then in a tug of war regarding the model’s performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*hbpIk255wIu320R43V0gyw.png" /><figcaption>Fig. 3 — Bias and variance are in a tug of war. High bias or variance can break your model performance. (Courtesy of my dogs)</figcaption></figure><p>But thankfully there is a solution where the total error reaches the minimum. So what can we do to avoid overfitting or underfitting and have our model’s error stay at this minimum? There are a lot of discussions in statistical learning about estimating errors and techniques to improve the model’s performance (see refs. [1,2]). But if you are searching for more “hands-on” solutions, without getting too much into math and statistics, there are a few steps we can take towards finding the tradeoff bias-variance:</p><ol><li>Some a priori knowledge about our dataset can be very useful when choosing which features are the most relevant to consider. Sometimes, having fewer features but good ones is better than considering all the features available. With more features, the model can become very complex and overfit. Therefore, feature selection is an important step.</li><li>Choose an accurate measure of the model’s performance. Depending on the goal of your model, the type of metrics you choose can lead to an erroneous evaluation of the model’s performance. For example, in situations where the costs of false positives and false negatives matter, such as in disease prediction, we need a measure that encodes this question, and accuracy won’t work (see <a href="https://naomy-gomes.medium.com/metrics-to-evaluate-classification-models-b18d645b7fac">my article</a> on metrics to evaluate classification models).</li><li>Testing as many algorithms as possible, making use of cross-validation to avoid looking at only training errors, and tuning model parameters is an opportunity to explore different model complexities and choose the one that finds a good balance between bias and variance.</li></ol><h4>Conclusion</h4><p>Bias and variance are two errors that can jeopardize model performance. High complexity models may seem to be a good idea since they perform so well on training data, but they are, in fact, overfitting due to low bias and high variance. On the other hand, low complexity models may not capture the relationship between features and target and perform very badly, underfitting due to high bias and low variance. Consequently, understanding the tradeoff bias — variance and finding the best model that minimizes both is crucial.</p><h4>References</h4><p>[1] <a href="https://hastie.su.domains/ElemStatLearn/">Hastie, Trevor, et al. <em>The elements of statistical learning: data mining, inference, and prediction</em>. Vol. 2. New York: Springer, 2009.</a></p><p>[2] <a href="https://arxiv.org/abs/0810.4752">Von Luxburg, Ulrike, and Bernhard Schölkopf. “Statistical learning theory: Models, concepts, and results.” <em>Handbook of the History of Logic</em>. Vol. 10. North-Holland, 2011. 651–706.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4204f004de5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metrics to evaluate classification models]]></title>
            <link>https://naomy-gomes.medium.com/metrics-to-evaluate-classification-models-b18d645b7fac?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/b18d645b7fac</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[classification-algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[evaluation-metric]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Fri, 01 Jul 2022 15:58:31 GMT</pubDate>
            <atom:updated>2022-07-01T15:58:31.361Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Machine learning classification models assign each point in a dataset to a class. Once this categorization is performed, how can we evaluate the accuracy of the classification? What are the best metrics for each type of dataset? Let&#39;s see which metrics are available, what each of them means, and how we can better analyze model performance.</em></p><h4>Classification algorithms</h4><p>Supervised machine learning algorithms can be classified into Regression and Classification algorithms. Roughly speaking, the first type predicts numerical values such as house prices and scores, while the latter predicts categorical values such as yes or no (binary); spam or not-spam (binary); mammal, fish, reptile, amphibian, or bird (multiclass). Therefore, when we want to assign categories to our data, we must rely on Classification algorithms. These algorithms learn from a given dataset where the classes are already assigned and then classify new observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*noByo2xBqRVbC5gJl2A6Qg.png" /><figcaption>Fig. 1 — The dataset contains two categories: yellow triangles and green rectangles. A classification algorithm learns from this data and predicts a class for new data.</figcaption></figure><p>Classification algorithms can be divided into two types: <em>lazy learners</em> and <em>eager learners</em>. <em>Lazy learners</em> store the dataset and only make predictions when asked, meaning when test data is provided to be classified. They will, therefore, spend more time classifying than learning. One example is the K-NN algorithm. <em>Eager learners</em>, on the other hand, start learning from the provided dataset and classify data before being supplied with the test data. Therefore, these algorithms spend more time learning than classifying. Examples include decision trees and Naive-Bayes.</p><h4>Evaluation</h4><p>Once a classification algorithm is chosen, and new observations are categorized, we must analyze classification performance; in other words, we must define a way to know how good the algorithm is in predicting the correct categories. And for this, we have what is called <strong>evaluation metrics</strong>. There are many evaluation metrics. Let&#39;s understand the most used ones:</p><ul><li><strong>Accuracy</strong>: is a metric that calculates how many correct predictions the model made. Consider the table in Fig. 2 (also called the <a href="https://en.wikipedia.org/wiki/Confusion_matrix"><em>confusion matrix</em></a>). The correct predictions are in the green and purple blocks. The incorrect predictions are in the blue blocks. The model was able to predict correctly 500 True Positives (TP) + 300 True Negatives (TN) totalizing 800 correct predictions over a total of 500 + 125 + 145 + 300 = 1070 predictions, which means the model was right in 74.76% of its predictions. This is the accuracy of the model. In Fig. 2, the False Positive (FP) and the False Negative (FN) are represented by the blue blocks and are, respectively, the Positive class predicted when the actual class is Negative, and the Negative class predicted when the actual class is Positive.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*ueifhRUo-AiWQRSwnmn2Wg.png" /><figcaption>Fig. 2 — Actual class and predictions made by the model. Green block: True Positive, purple block: True Negative, blue blocks: False Positive and False Negative.</figcaption></figure><p>The mathematical formula for the accuracy, as you may have already imagined, is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*46Hllvlq_WoHSxaK-2KbuQ.png" /></figure><p>which means we sum the TP (green block) with the TN (purple block) predicted by the model and divide by the total number of predictions (TP + TN + FP + FN — green, purple, and blue blocks). The higher the accuracy, the better the model* (* some <em>critical observations </em>about this statement<em> </em>will be discussed in the next section).</p><ul><li><strong>Recall</strong>: is a measure of how many True Positive the model predicts over the total number of actual Positive. The mathematical formula is:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*T3yxhnV8cB7Oac7zd4iSHQ.png" /></figure><p>In our table, Recall is given by the 500 TP over the 500+125 = 625 actual Positive, which returns the value of 0.8. We can notice that the best value of Recall is 1 when the model could predict the Positive class correctly, and there is no FN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*yXSy3IF7kkh3acPSTJNLMA.png" /><figcaption>Fig.4 — Recall measures how many TP out of the total number of Positive.</figcaption></figure><p>We can conclude that Recall is an excellent metric to evaluate the best model when we don&#39;t want our model to predict many False Negatives. For example, we can think of sick patient prediction in a scenario of contagious disease. If a patient is predicted as Negative (doesn&#39;t have the disease) but is an actual Positive (has the disease), then this error can cause the spreading of the disease. Therefore, we want a model to predict the most correctly possible True Positives, which is done by minimizing the number of False Negative predictions, thus increasing the value of Recall.</p><ul><li><strong>Precision</strong>: measures how many True Positive the model predicts over the total number of predicted Positive. The mathematical formula is:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*V8nHLjilp78YyI3Kdjm3Uw.png" /></figure><p>In our table, Precision is given by the 500 TP over the 500+145 = 645 predicted Positive, which returns the value of ~0.78. We can notice that the best value of Precision is 1, which means the model could predict the Positive class correctly and there is no FP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*M8ZlU3Tb4mtQv0BpAAaQhQ.png" /><figcaption>Fig.5 — Precision measures how many TP out of the total number of predicted Positive.</figcaption></figure><p>Similarly to Recall, we can conclude Precision is an excellent metric to evaluate model performance if predicting too many False Positives has a high cost. The example we can think of here is spam detection. If an email is not spam (Negative) but is predicted as spam (False Positive), the receiver may lose essential emails. Therefore, we will seek a model that will better correctly predict spam by minimizing the number of False Positives, thus increasing the value of Precision.</p><ul><li><strong>F1 score</strong>: is a measure defined as a function of Precision and Recall, the mathematical formula given by</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*HVxzkHOsagJTI_veZTdp0g.png" /></figure><p>Therefore, we can say that the F1 score will be of good use when seeking a balance between Precision and Recall. For example, if FN and FP predictions come at equal costs.</p><ul><li><strong>AUC ROC</strong>: a ROC (Receiver Operating Characteristic) is a curve in a graph where the y-axis is the True Positive Rate, a.k.a. Recall, and the x-axis is the False Positive Rate (FPR), which is given by the False Positives over the total actual Negative. The mathematical formula for FPR is :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*30l60uiLK8UfbIf7lZAOJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*gzLD18b35P56nTprQPnCTA.png" /><figcaption>Fig. 6 — False Positive Rate is calculated as the False Positive (FP) over the total actual Negative (FP + TN).</figcaption></figure><p>Several ROC curves are shown below. Each point of the curves corresponds to a different classification threshold, a value that is chosen depending on the costs of predicting false positives. For example, the x=y curve (pink) is the curve for a not-skilled model since it predicts TP and FP with the same rates, which is equivalent to a random prediction. Above that curve, the model becomes skilled, and the best curve is when it predicts all the Positive classes correctly, whereas below that curve, the model gets worse until it reaches a point where all predictions are FP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*DeJ452wXWKm1pF6kZ2ylfg.png" /><figcaption>Fig. 7 — ROC curves.</figcaption></figure><p>The AUC is the Area Under the (ROC) Curve. It&#39;s an integral over the entire range of the TP and FP rates; therefore, it&#39;s an aggregate measurement of the model&#39;s performance for all the thresholds. As we can see from Fig. 8, the highest value for AUC is 1 (area of the square with sides 1) for the best curve when the model makes all Positive predictions correctly. An AOC of 0.5 (area of the triangle with sides 1) happens for the random performance, and the zero value is for the worst curve (no area under that curve).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*S7XpGrMP6Neac3mi7Ai2jA.png" /><figcaption>Fig. 8 — AUC for two different ROC curves.</figcaption></figure><p>AUC is, therefore, an excellent metric to measure the quality of the model&#39;s predictions without worrying about choosing a threshold.</p><h4>Balanced and imbalanced datasets</h4><p>We now come back to the * in the Accuracy description. It was said: &quot;The higher the accuracy, the better the model,&quot; but this is only a valid statement if your dataset is balanced, meaning that the number of observations in the classes is more or less the same, which is not always the case. So if we consider, for example, spam detection and disease prediction, it is very likely we will find a highly imbalanced dataset. For these cases, accuracy is not a good metric. For example, consider this <a href="https://www.kaggle.com/datasets/kamilpytlak/personal-key-indicators-of-heart-disease">Heart Disease dataset</a> where we have 319795 lines representing the adults interviewed. If we plot a pie chart of this dataset, only 8.6% of the people affirmed they have heart disease. Therefore, if a model predicts that 90% don&#39;t have heart disease, the accuracy will be high, but it does not mean you have a good model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*ZqxsY9LN1zw80D2g2y0wEQ.png" /></figure><p>Therefore, when dealing with an imbalanced dataset, we must rely on metrics such as Recall, Precision, F1 score, or AUC ROC. For example, in the case of the heart disease dataset, we want our model to predict TP well and avoid FN, so the patient can get adequate treatment if they have heart disease. We could then choose Recall and ROC AUC to analyze model performance.</p><h4>Conclusions</h4><p>There are many metrics available to analyze model performance. Each of them can be more or less suitable depending on whether the dataset is balanced or imbalanced and the costs of each prediction. Understanding each of the metrics available and knowing the best one to use is crucial to obtaining better models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b18d645b7fac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K-means clustering explained with Python]]></title>
            <link>https://naomy-gomes.medium.com/k-means-clustering-explained-with-python-c7c69177b932?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7c69177b932</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Fri, 10 Jun 2022 14:37:05 GMT</pubDate>
            <atom:updated>2022-06-10T14:37:05.343Z</atom:updated>
            <content:encoded><![CDATA[<p><em>K-means clustering is an unsupervised machine learning algorithm to analyze the formation of clusters in the data, which means finding groups of datapoints. How does it work? What is the logic behind this algorithm? What are the advantages of clustering? Let’s explore the particularities of this clustering technique.</em></p><h4>Clustering</h4><p>Sometimes, when analyzing data, some natural similarities between the datapoints arise and we observe the formation of patterns. The ways that these collections of patterns organize are the clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*hebrVjmNet4DSohvReGktg.png" /><figcaption>Fig. 1 — Datapoints on the left naturally form clusters (green, blue, and yellow).</figcaption></figure><p>When clustering happens in a dataset, it allows us to explore relationships among the datapoints. This can be useful as a way of drawing direct conclusions about the dataset or can be used to create new features that will make it simpler for a model to learn the data.</p><p>Take as an example a <a href="https://www.kaggle.com/datasets/kandij/mall-customers">dataset</a> of mall customers where the features are: Genre, Age, Annual Income, and Spending Score. The Annual Income is measured in thousands of dollars and the Spending Score is a value between 1 and 100 assigned to each customer based on how much they spend in the mall. The higher the value, the more they spend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*W6AVjo0glL6iW_3ISv5GEg.png" /><figcaption>Fig. 2 — First five columns of the mall customers dataset.</figcaption></figure><p>A mall would be interested in analyzing how much a customer spends and how much they make in a year, to better recommend products, discounts, etc. The datapoints we would like to analyze here are, therefore, from the features Annual Income and Spending Score. Let’s take a look at a scatter plot of these two columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*8x_8Cju6bx3IlX0Mh96-dg.png" /><figcaption>Fig. 3 — Scatter plot of the features Annual Income and Spending Score.</figcaption></figure><p>We can notice some clustering formations arising naturally in this dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*9Yl_oGD6TddKHoIkgLz5Ig.png" /><figcaption>Fig. 4 — Inferring cluster formation by eye with the features Annual Income and Spending Score.</figcaption></figure><p>We could also be interested in whether the datapoints of the Age and Spending Score features present clusters. Let’s do a scatter plot and see what is the datapoint distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*UEKX_-zkwyaIxjf2hw7yzQ.png" /><figcaption>Fig. 5 — Scatter plot of the features Age and Spending Score.</figcaption></figure><p>There is no naturally arising cluster formation, at least not so obvious to the eye as in the previous case. A closer inspection could, however, reveal some potential clustering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*xZiLxU2yPsQRsn1UIYOfQw.png" /><figcaption>Fig. 6 — Inferring cluster formation by eye with features Age and Spending Score.</figcaption></figure><p>We will analyze these two cases and write the k-means algorithm for both of them.</p><h4>Algorithm and application with Python</h4><p>K-means clustering is an unsupervised machine learning algorithm that works based on the squared error criterion, which is derived from the Euclidean distance. This distance is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*yWzKxMDZpk41NwWo30rtDw.png" /><figcaption>Fig. 7 — Euclidean distance between two points in two dimensions.</figcaption></figure><p>What the algorithm will do is use this squared error to analyze how well the datapoints are assigned to a certain cluster. It will search to minimize the squared error, meaning that the distance from the datapoints to their assigned cluster center will be minimal.</p><p>The algorithm behind the k-means clustering is as follows:</p><ol><li>Choose the number k of clusters.</li><li>Select, randomly, k points where the k centroids will be initialized.</li><li>For each point in the dataset, assign it to the nearest centroid. The cluster formation begins.</li><li>Once the clusters are formed, calculate the center of mass of each of them and move their respective centroids to this center of mass.</li><li>Reassign each data point to the nearest centroid again. If that step doesn’t reassign the datapoints to new clusters, then convergence is reached.</li></ol><p><a href="https://halderadhika91.medium.com/k-means-clustering-understanding-algorithm-with-animation-and-code-6e644993afab">Here</a> are some animations of the clustering process.</p><p>You may now be wondering: how do we know how many clusters to choose? And to answer that question, there is the <em>Elbow method</em>. It consists of a plot of the WCSS ( Within-Cluster Sum of Square ) as a function of the number k of clusters. The WCSS is the sum of the squared distance between each point and the centroids of the clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*TJmw6J8SxzHnFow6lI3lFQ.png" /></figure><p>When the curve shows an elbow, that’s the number of clusters from which the variation in the squared error won’t change much, and it’s this number we choose to be k.</p><p>Now that we know how to choose k, let’s implement the k-means on the mall customers dataset.</p><p>Importing libraries, reading the table, and looking at its shape:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*Q7v60gl1RS2aGqLEP8c3Uw.png" /></figure><p>As we saw in Fig. 3, the scatter plot of columns Annual Income and Spending Score suggests clustering. We will then select these columns and apply the k-means algorithm to generate the elbow plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*0lt-YgkHP1RRZoo0LdYP_g.png" /></figure><p>which indicates the number of clusters to be 5. We can finally apply k-means to the data, which will give us the clusters (five clusters: 0, 1, 2, 3, 4) that each line of the columns we choose belongs to, here represented by the new column Cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*xik6KH3Ii8e4P_DolkbfmQ.png" /></figure><p>The next step is to graphically see the clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9Rr2lMRXC0_aD7ayJc4qQ.png" /></figure><p>If we compare the cluster formation with k-means to the cluster formation we could see by eye in Fig. 4, there’s an agreement. Now, let’s see what kind of cluster formation, if any, there is for the features Age and Spending Score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*dTx_VgHf23K4k25xG4uHmQ.png" /></figure><p>The elbow method suggests k=4, one more cluster that we could infer by eye in Fig. 6. Applying k-means and adding the column Cluster2, we have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*7f5vocELpoy0JUOX_tQzoQ.png" /></figure><p>We now can see the scatter plot with the cluster formation and compare it to Fig. 6:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyGDbTjwM2mtZdZXN2fCyQ.png" /></figure><p>The k-means algorithm identified 4 clusters and, contrary to what I drew in Fig. 6, it separated the upper left points into two clusters (pink and green), considered the points on the bottom as belonging to the same cluster (yellow), and the points on the middle right as another cluster (blue).</p><h4>Silhouette analysis</h4><p>Silhouette analysis can be used to tell how well the datapoints fit in a cluster. In this method, cluster sizes can be analyzed and we can also obtain average values of silhouette coefficients that measure how far or close the datapoints of one cluster are to other clusters. This coefficient is in the range [-1,1], where negative values show the intersection of clusters, 0 means that clusters are very near, and close to 1 values indicate clusters are far from each other. More on the silhouette analysis on k-means clustering can be found <a href="https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html">here</a>. In this text, we will restrict ourselves to calculating the mean Silhouette Coefficient for each of the analyzed cases.</p><p>For the features Annual Income and Spending Score, we have a silhouette score of ~0.55, which is a good value and confirms what we saw, that is, clusters are, on average, far apart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*MbjQdHs2g9D9YZini2bWOg.png" /></figure><p>For the features Age and Spending Score, we have a silhouette score of ~0.50, which is lower than the first score, but it is also good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*mFr8Sz8hlpY9bG6YX6f58g.png" /></figure><h4>Observations</h4><p><strong>Important observation 1</strong>: the k-means algorithm relies on features being equally scaled to work properly, otherwise it can result in wrong clustering. Therefore, it is important to standardize features (in the present case, features were equally scaled and there was no need to standardize). <strong>Important observation 2</strong>: the groups should have more or less the same number of observations. <strong>Important observation 3</strong>: k-means is sensitive concerning the initial conditions of the centroids. That’s why we chose ‘k-means++’, but there are many discussions and several other techniques to deal with this problem (see <a href="https://hal.inria.fr/inria-00321515/file/verbeek01tr.pdf">here</a>), which will not be discussed.</p><p>With the conditions above satisfied, and if the dataset has clustering formation, we can obtain valuable information about our data with k-means clustering. Another use of it is to improve model performance. As an example, in the Titanic Spaceship Kaggle code that can be found <a href="https://www.kaggle.com/code/naomyduartegomes/spaceship-titanic-lgbm-clustering">here</a>, clustering of the columns Room Service, Food Court. Shopping Mall, Spa, VR, and Deck improved model performance, making it easier for the model to find the relationships between those columns.</p><h4>Conclusions</h4><p>The k-means algorithm is a widely used algorithm to analyze data and the interrelationships between features. It can be used in different situations such as separating clients based on their spending habits, their musical taste or TV preferences, etc. We can also use it to improve model performance. The elbow method allows us to choose the number of clusters if it&#39;s not visually obvious, and the algorithm is very straightforward to apply in the dataset, making k-means a very advantageous method to be used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c69177b932" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An overview of Principal Component Analysis]]></title>
            <link>https://naomy-gomes.medium.com/an-overview-of-principal-component-analysis-510e0935b406?source=rss-6d5e8ccd9890------2</link>
            <guid isPermaLink="false">https://medium.com/p/510e0935b406</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Naomy Duarte Gomes]]></dc:creator>
            <pubDate>Mon, 16 May 2022 13:15:25 GMT</pubDate>
            <atom:updated>2023-05-12T19:16:53.723Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Principal component analysis (PCA) is an unsupervised machine learning algorithm that calculates the principal components of a given dataset. But what are principal components and why do they matter? What can be achieved with PCA and why should we learn it? Let’s explore the math and the meaning behind this algorithm.</em></p><h3><strong>The basic linear algebra</strong></h3><p>In math language, PCA is a linear transformation from a set of variables <strong>A</strong> to another set <strong>T</strong> where the new variables of <strong>T</strong> are orthogonal, meaning they are uncorrelated. To go from <strong>A</strong> to <strong>T</strong>, we need some linear algebra.</p><p>Let’s start by assuming that the set of variables <strong>A</strong> is a <em>n x m </em>matrix<em>, </em>where <em>n</em> is the number of rows (which, in terms of datasets, are the measurements) and <em>m</em> is the number of variables (features). For the sake of demonstration, the matrix <strong>A </strong>will be a 4 x 3 matrix, representing a dataset with 4 measurements for each column and 3 features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*lBegIGc3Y3lcVZ7e56FvRg.png" /></figure><p>We can see that matrix <strong>A </strong>has different scales in its columns. The next step is therefore to standardize matrix <strong>A</strong> to form a new matrix <strong>B</strong> with zero mean. We standardize by subtracting from each column element the mean value of its column and dividing by the standard deviation of the column:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*oSpFrMd_xNbnkgLUZVrtyA.png" /></figure><p>Transforming data to the same scale is very important to PCA since linear transformations are very sensitive to scales, which means if a variable has a larger deviation from its mean, it will dominate the results (a discussion on whether we must standardize or normalize data to obtain better computational results can be found <a href="https://towardsdatascience.com/normalization-vs-standardization-quantitative-analysis-a91e8a79cebf">here</a>).</p><p>Now we calculate the covariance matrix <strong>C </strong>of<strong> B </strong>defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*pFvZI2LUNAh4kRChsInVtQ.png" /></figure><p>Covariance is a measure of how a pair of variables change conjointly. If both of them increase or decrease, their covariance is positive and we say they are correlated. If one decreases and the other increases, their covariance is negative and they are inversely correlated. The diagonal elements of <strong>C</strong> are the variances (covariance of an element with itself):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*CpPz_ky52OTkVMG6ElaUTQ.png" /></figure><p>We must now diagonalize matrix <strong>C</strong>, which corresponds to finding its eigenvalues and eigenvectors that give us the linear transformation we seek for. Since <strong>C</strong> is the covariance matrix, its eigenvalues are the directions in which there is the most variance in the data and the eigenvalues are the scale of the variances in each direction. The eigenvalue-eigenvector equation is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*SFe7HTyDQlZOdhMMAkwzzA.png" /></figure><p><strong><em>Eigenvectors of distinct eigenvalues are orthogonal for symmetric matrices </em></strong>(which is the case of matrix<strong><em> </em>C</strong>).<strong> </strong>Solving the eigenvalue-eigenvector equation we have then a matrix V where each column is an eigenvector and the eigenvalues are given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*y0FN0qmxuE5JyKJmQEFwZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*5-8h_cHLBma1f2pvNxi_PA.png" /></figure><p>The linear combinations <em>λ</em><strong><em>V </em></strong><em>are what are called the principal components [1](although some authors refer to the eigenvectors as the principal components).</em><strong><em> </em></strong>To each eigenvector (column of the eigenvector matrix <strong>V</strong>), there is a corresponding eigenvalue. If we sort the eigenvalues in order of decreasing values (in this case, λ2 &gt; λ3 &gt; λ1) and then sort the eigenvectors according to their respective eigenvalues (V2, V3, V1), we can write an ordered eigenvector matrix <strong>W</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*80fKTI9TceJdTWFUYjZ1cg.png" /></figure><p>Finally, the transformed matrix <strong>T</strong> we were looking for is given by the projection of matrix <strong>B</strong> (standardized version of matrix <strong>A</strong>) onto matrix <strong>W</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/118/1*nteChX4dQLbGrLGJVKqYrQ.png" /></figure><p>Since each eigenvalue is proportional to the explained variance, which is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/103/1*DziQDa-odEORtETPR3o0ZA.png" /></figure><p>(meaning that each eigenvalue is roughly its explained variance) we can actually lower the dimension of matrix <strong>T</strong> by choosing the first <em>i</em> eigenvalues and respective eigenvectors, writing the matrix <strong>W</strong> with only these chosen eigenvectors and project <strong>B</strong> on it. The choice of the number of eigenvalues depends on what is the variance explained we are looking for (90%, 80%, and so on. Usually, more than 80% of the variance explained is already good, but this is a factor to analyze depending on the dataset and your goals.)</p><p>By choosing only the first<em> i </em>eigenvectors, we still have a lot of information about our original dataset. This is the beauty of PCA! Therefore, besides the dimensionality reduction of <strong>T</strong>,<strong> </strong>it<strong> </strong>will also have variables that are orthogonal to each other (remember this is one thing we were seeking at the beginning: uncorrelated variables). We have then made a dimensionality reduction of the matrix (dataset)<strong> A</strong> and generated uncorrelated variables. Thus, if dealing with datasets that contain many features, we can reduce this number with PCA, obtaining a smaller dataset, easier to deal with, containing the most information about the original dataset and whose variables are independent.</p><h3>Application on a dataset with Python</h3><p>We will consider the <a href="https://www.kaggle.com/datasets/fernandolima23/classification-in-asteroseismology?q=asteroseismology">Classification in Asteroseismology</a> Kaggle dataset. In this dataset, a red giant branch of stars is analyzed and the objective is to predict whether a red giant is becoming a <a href="https://doi.org/10.1093/mnras/stx1174">helium-burning star</a>. We only have three columns of features (<em>Dnu, numax </em>and <em>epsilon</em>) and the target column POP is 1 for helium-burning and 0 for red giant, therefore we will be able to reduce this dataset to two PCs and see the clustering.</p><p>We use first import the dataset and divide it into features (X) and target (y — remember we worked with a matrix of features in the previous sections, therefore if your dataset has a target column, it must be separated):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*eVXTmcMUMndqpHWsBYpwxw.png" /><figcaption>Fig. 1 — Complete stars dataset divided into X and y datasets.</figcaption></figure><p>The next step is to standardize the features, as we can see in Fig. 1 that the features have different scales and this can jeopardize the results of PCA. We use the function <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html?highlight=standard%20scaler#sklearn.preprocessing.StandardScaler">StandardScaler()</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*Nkux_BJ36x9_26DKWbIOEw.png" /><figcaption>Fig. 2 — Standardizing the data.</figcaption></figure><p>Now, we must apply PCA to the standardized dataset by calling <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html">PCA()</a>. Let us look at the cumulative variance explained and the variance explained as a function of the number of PCs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27tXsGmt5rpDRJxxuWtbEQ.png" /><figcaption>Fig. 3 — The elbow method: percentage of variance explained as a function of the number of PCs.</figcaption></figure><p>Fig. 3 shows the number of PCs and the percentage of the variance explained. The blue line is the cumulative explained variance and it reaches 100% of the variance explained, while the orange line is the variance explained, which starts with the first component having the greater value. We see there&#39;s an elbow in each curve that indicates the number of components we must pick. This is the <a href="https://en.wikipedia.org/wiki/Elbow_method_(clustering)#:~:text=In%20cluster%20analysis%2C%20the%20elbow,number%20of%20clusters%20to%20use.">elbow method</a>. In this case, 2 is the number of PCs we will use. For the next step, we apply PCA again, choosing the parameter <em>n_components = 2 </em>and construct a new dataset with the PCs as columns and add the target column<em>:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*urCOnK8-85cb-lw3EN086w.png" /><figcaption>Fig. 4 — PCA with 2 components and new dataset generated with the PCs as columns and the target column.</figcaption></figure><p>Plotting the PCs with the target column as hue to see the clustering:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*ndANiPbroipG6e6ZuJKJ-Q.png" /><figcaption>Fig. 5 — Plotting of PCs with POP as hue.</figcaption></figure><p>From Fig. 5, we see the red giants are predominantly distributed in a diagonal line, while the helium-burning is in the left upper corner. Some helium-burning stars can be found close to the red giant branch, and I believe this is because they have some similarities (after all, a helium-burning is one phase of the red giant). What is most important here is that PCA was able to reduce the dimension of the dataset with 100% of the variance explained and we could see the clustering. For larger datasets, we may not be able to choose 2, or 3 components, but we still can use PCA to diminish the dataset size.</p><h3>Key steps and some limitations</h3><p>Let’s now summarize the key steps to perform the PCA:</p><ul><li>Divide the dataset into features and target;</li><li>Standardize the features;</li><li>Apply PCA and use the elbow method to see how many components to choose;</li><li>Choose the number of components and apply PCA to obtain the final reduced dataset.</li></ul><p>There is one important limitation in applying PCA: the new variables will be less interpretable, as we could see with the dataset generated previously. If we need to keep more than four dimensions, it will not be possible to visualize the PCs in one graph and see clustering.</p><h3>Conclusions</h3><p>PCA is a great technique to reduce the dimension of the dataset while keeping as much information as possible. If the dataset is not so big and the number of PCs is 2 or 3, we can even see clustering by plotting the PCs. It can also be used to reduce the dimension of images.</p><p>I hope this article was helpful! Please leave comments and suggestions :)</p><h3>References</h3><p>[1] Jolliffe Ian T. and Cadima Jorge 2016 Principal component analysis: a review and recent developments. <em>Phil. Trans. R. Soc. A.</em>3742015020220150202 <a href="https://doi.org/10.1098/rsta.2015.0202">https://doi.org/10.1098/rsta.2015.0202</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=510e0935b406" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>