<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arif Enes Aydın on Medium]]></title>
        <description><![CDATA[Stories by Arif Enes Aydın on Medium]]></description>
        <link>https://medium.com/@aydnarifenes?source=rss-6e6fe90d9ac5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xJAxPgT6rTfI92tq</url>
            <title>Stories by Arif Enes Aydın on Medium</title>
            <link>https://medium.com/@aydnarifenes?source=rss-6e6fe90d9ac5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:15:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aydnarifenes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What are vector databases?]]></title>
            <link>https://medium.com/@aydnarifenes/what-are-vector-databases-435d8a8742bd?source=rss-6e6fe90d9ac5------2</link>
            <guid isPermaLink="false">https://medium.com/p/435d8a8742bd</guid>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Arif Enes Aydın]]></dc:creator>
            <pubDate>Sat, 18 May 2024 15:14:09 GMT</pubDate>
            <atom:updated>2024-05-18T15:14:09.035Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post, we will explore the world of vector databases, a specialized type of database designed to handle complex data types efficiently. Traditional databases, which store data in rows and columns, are well-suited for structured data like financial records or customer information. However, they struggle with more complex data types such as images, audio, and text.</p><p>Vector databases address this challenge by converting complex data into numerical representations known as vectors. This allows for efficient indexing, storing, and retrieving of data based on similarity. We will explore what vectors are, how vector databases work, and why they are important in today’s data-driven world.</p><h4>Vectors</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*yFWZCs80Tri-JWRs.jpg" /><figcaption>Vector Space (Source: <a href="https://weaviate.io/blog/what-is-a-vector-database">https://weaviate.io/blog/what-is-a-vector-database</a>)</figcaption></figure><p>First, we need to understand what vectors are in terms of “vector databases”. Vectors are arrays of numerical values that can describe complex objects. These objects could be words, sentences, images, audio, video, or even other types of data. Because vectors represent objects and their meanings, they enable the search for similar objects. So, how are these vectors obtained?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKDmJqAX_fr8HzMIpYRhxw.jpeg" /><figcaption>Vectors. (Source: <a href="https://qdrant.tech/articles/what-are-embeddings/">https://qdrant.tech/articles/what-are-embeddings/</a>)</figcaption></figure><p>Vectors can be generated from the raw data using machine learning methods such as feature extraction algorithms, word embeddings, or other deep learning networks.</p><p>For text data, word embedding techniques can be used. Word embeddings are obtained by training neural network model on large text corpora, where words are represented as dense vectors in a vector space. These embeddings are typically learned as weights in the hidden layers of the neural network. Techniques like Word2Vec, GloVe, and BERT learn semantic relationships by using context words or leveraging global co-occurrence statistics, thus helping generate dense vector representations (embeddings) of words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*59wL_yjKSElZXBBv24vJOQ.jpeg" /><figcaption>Using context words to generate embeddings. (Source: <a href="https://arxiv.org/pdf/1309.4168v1">https://arxiv.org/pdf/1309.4168v1</a>)</figcaption></figure><p>As for image data, the extraction of the embeddings is similar. Image vectors (embeddings) are extracted from the activations of intermediate layers in pretrained deep learning models such as VGG, ResNet, or other related models. The images undergo preprocessing steps to ensure consistency and compatibility with the chosen model. Following this, the preprocessed images are fed into the deep learning model, where features are extracted from a designated layer, often a fully connected layer. After extraction, these embeddings can be used in vector databases, for example to obtain similar images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*l9c8f0FRlJQ06IvPNLeXJw.png" /><figcaption>Embeddings extracted from fully connected layer. (Source: <a href="https://www.upgrad.com/blog/basic-cnn-architecture/">https://www.upgrad.com/blog/basic-cnn-architecture/</a>)</figcaption></figure><blockquote><strong>Large Language Models (LLMs)</strong> are advanced models trained on vast amounts of text data. They learn to understand and generate human-like text by capturing complex patterns and relationships in language. Examples include OpenAI’s GPT and Meta’s Llama series.</blockquote><p>As LLMs and RAG gain popularity, there’s a growing focus in vector databases. To handle these advanced models well, vector databases are using more sophisticated techniques to represent text better. Models like <a href="https://openai.com/index/new-and-improved-embedding-model"><em>ada-002</em></a>, <a href="https://cohere.com/blog/introducing-embed-v3"><em>Cohere Embed V3</em></a>, and <a href="https://github.com/microsoft/unilm"><em>E5</em></a> are examples of this, and more can be found on <a href="https://huggingface.co/spaces/mteb/leaderboard"><em>MTEB Leaderboard</em></a>. They’re designed to understand language more deeply, which is crucial for tasks like natural language processing. This progress in embedding methods is helping improve how we search for and understand information.</p><blockquote><strong>Retrieval-augmented generation (RAG)</strong> enhances language models by integrating facts from external sources, such as a vector database. It overcomes a limitation in LLMs, which are proficient at responding to general prompts but may lack depth on specific topics.</blockquote><h3>Vector Databases</h3><p>Now that we’ve discussed vectors and how to extract them, let’s shift our focus to vector databases and understand how they operate.</p><p>A vector database indexes, stores, and grants access to structured or unstructured data, such as text or images, along with their vector embeddings. This enables users to swiftly locate and retrieve similar objects at scale in real-world applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x9pa4TrYEoO2Ct7i.png" /><figcaption>Workflow of a vector database. (Source: <a href="https://www.pinecone.io/learn/vector-database/">https://www.pinecone.io/learn/vector-database/</a>)</figcaption></figure><p>Initially, the vector database indexes structured or unstructured data (such as text documents) provided by the user, alongside their corresponding vector embeddings (extracted with an embedding model). Indexing is conducted using <a href="https://weaviate.io/blog/what-is-a-vector-database#vector-indexing-for-approximate-nearest-neighbor-approach">ANN (Approximate Nearest Neighbors)</a> algorithms, such as HNSW, PQ, or LSH. This operation organizes the data to make retrieval efficient using similarity metrics.</p><p>One of the most popular techniques used for indexing/searching is <a href="https://www.pinecone.io/learn/series/faiss/hnsw/"><strong>Hierarchical Navigable Small Worlds (HNSW)</strong></a><strong>. </strong>It basically organizes data in a hierarchical structure for fast and efficient nearest neighbor search. Two basic components that made this algorithm are the <em>probability skip list</em> and <em>navigable small world</em>.</p><ul><li><a href="https://en.wikipedia.org/wiki/Skip_list">Skip lists</a> works as a stacked linked list. The bottom layer is an ordinary ordered linked list. From bottom to top layers, the number of skipped node increases. A search for a given element starts at the top layer, and proceed horizontally until the current element is greater or equal to given element. If current element greater than the target element, the search continues on next lower layer after returning to previous element.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*WoUIRhJ-ULOIBlKQ" /><figcaption>Skip lists. (Source: <a href="https://commons.wikimedia.org/wiki/File:Skip_list.svg">https://commons.wikimedia.org/wiki/File:Skip_list.svg</a>)</figcaption></figure><ul><li><a href="https://www.pinecone.io/learn/series/faiss/hnsw/">Navigable small world (NSW)</a> is a type of graph. Every node of a NSW is connected to several other nodes, which called friends. Each node is reachable through other nodes. During a search operation, the algorithm starts at a predefined entry point and traverses to nearby vertices, greedily moving to the nearest neighboring vertices in each friend list. This process continues until no closer vertices are found.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BiK-j9XJYs-zn9G1.png" /><figcaption>Navigable small world. (Source: <a href="https://www.pinecone.io/learn/series/faiss/hnsw/">https://www.pinecone.io/learn/series/faiss/hnsw/</a>)</figcaption></figure><ul><li>HNSW takes the main principle of skip list, and integrates it on navigable small world structure. The search starts at the top layer, and proceeds until the closest node to query node is found. After that, the search continues on the next lower layer in the same way, until the best result is found on bottom layer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bzVGiDjEkENMcDNF.png" /><figcaption>Search operation in HNSW. (Source: <a href="https://www.pinecone.io/learn/series/faiss/hnsw/">https://www.pinecone.io/learn/series/faiss/hnsw/</a>)</figcaption></figure><ul><li>The main takeaway of HNSW is quick navigation across dataset, allowing fast and efficient search. However, this structure can be memory-intensive, especially if the average number of connections per node is large.</li></ul><p>To briefly summarize other indexing algorithms:</p><ul><li><a href="https://www.pinecone.io/learn/vector-database/"><strong>Product Quantization (PQ)</strong></a>, is a technique in vector quantization and compression. It partitions high-dimensional vector spaces into smaller subspaces and quantizes each subspace separately.</li><li><a href="https://www.pinecone.io/learn/vector-database/"><strong>Locality-Sensitive Hashing (LSH)</strong></a>, is a technique used to efficiently approximate nearest neighbors in high-dimensional spaces. It works by hashing input data points in such a way that similar points are likely to hash to the same or nearby hash buckets with high probability.</li></ul><p>After indexing is completed, the database will be ready for querying. Here’s a breakdown of how a vector database handles queries:</p><ol><li>Query: The user enters a text query into the system.</li><li>Embedding Extraction: The system extracts embeddings from the input text using a pre-trained embedding model.</li><li>Comparison: The system compares the embeddings of the query text with the embeddings of indexed data points in the vector database. This comparison is typically done using similarity metrics. For this operation, similarity measures such as cosine similarity, Euclidean distance, Manhattan distance, dot product, and Hamming distance can be used.</li><li>Retrieval: Based on the similarity scores calculated in the comparison step, the system retrieves the most similar data points from the database.</li><li>Result Presentation: The retrieved data points are presented to the user, typically in order of similarity to the query text.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hjqXYsJ8jlPloK7r.png" /><figcaption>Detailed Workflow of a vector database. (Source: <a href="https://thedataquarry.com/posts/vector-db-2/">https://thedataquarry.com/posts/vector-db-2/</a>)</figcaption></figure><h4>Advantages / Disadvantages</h4><h4><strong>Advantages</strong></h4><ul><li><strong>Efficient Retrieval: </strong>Vector databases excel in swiftly retrieving similar objects, even in large datasets, thanks to their indexing techniques like HNSW, PQ, or LSH. This efficiency is crucial for real-time applications where speed is paramount.</li><li><strong>Scalability:</strong> These databases are designed to handle large-scale data effectively. As datasets grow in size, vector databases can still maintain high performance without significant degradation.</li><li><strong>Versatility:</strong> Vector databases aren’t limited to specific data types. Whether it’s text, images, audio, or other forms of data, they can efficiently index and retrieve information, making them versatile for various use cases.</li><li><strong>Semantic Understanding:</strong> With advancements in embedding methods, vector databases can better capture the semantic relationships between objects. This semantic understanding enhances the accuracy of search results, providing more relevant matches to user queries.</li></ul><h4><strong>Disadvantages</strong></h4><ul><li><strong>Resource Intensiveness:</strong> Some indexing algorithms, like HNSW, can be memory-intensive, particularly when dealing with large datasets or high-dimensional spaces. This resource requirement might pose challenges for deployment, especially in resource-constrained environments.</li><li><strong>Complexity of Implementation:</strong> Implementing and managing a vector database, especially at scale, can be complex. It requires expertise in both database management and machine learning techniques for embedding generation and indexing.</li><li><strong>Accuracy Trade-offs:</strong> While the retrieval speed is impressive, the search results may lack sufficient accuracy for some applications, especially the approximate nearest neighbor search.</li><li><strong>Dependency on Embedding models:</strong> The effectiveness of a vector database is highly dependent on the embedding models used to generate the vectors. If these models are not well-trained or do not fairly represent the data, the search results will be of poor quality. More specifically, Database Administrators will have to ensure that the metadata and features of the data are constantly fed into the vectorization process constraints. Thus, continued training and updating of the models are indispensable.</li></ul><h4>Applications / Use Cases</h4><p><strong>Content Recommendation:</strong> Vector databases power content recommendation systems across multiple platforms such as streaming services, e-commerce, and social networks. Analyzing user behavior and relationships between items, these databases aid in providing personalized recommendations. This has a positive impact on user engagement, content consumption, and overall user satisfaction.</p><p><strong>Image and Video Retrieval:</strong> In image and video databases, vector representations support the retrieval of visually similar content. Its applications include reverse image search, video recommendations, and content moderation. These are all areas where one must quickly and accurately identify the desired multimedia content.</p><p><strong>Natural Language Processing (NLP):</strong> Such vector databases are core machinery for tasks in NLP, like semantic search, question answering, and sentiment analysis. They allow fast and accurate retrieval of relevant documents or passages by indexing text embeddings, empowering users to extract insights from vast text corpora efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*h5Xat8db3fzPxlWV.png" /><figcaption>Retrieval Augmented Generation (RAG) (<a href="https://towardsdatascience.com/retrieval-augmented-generation-rag-from-theory-to-langchain-implementation-4e9bd5f6a4f2">https://towardsdatascience.com/retrieval-augmented-generation-rag-from-theory-to-langchain-implementation-4e9bd5f6a4f2</a>)</figcaption></figure><p><strong>Anomaly Detection:</strong> In cybersecurity and anomaly detection applications, vector databases aid in identifying abnormal patterns or behaviors within large datasets. By comparing the embeddings of incoming data points with those of normal behavior, these databases can flag anomalies in real-time, helping organizations mitigate potential threats effectively.</p><p><strong>Product Search and Recommendation:</strong> E-commerce platforms leverage vector databases to enhance product search and recommendation systems. By indexing product embeddings on a variety of attributes like descriptions, images, and user preferences, these databases offer personalized search results and recommendations that could achieve better conversion rates and customer satisfaction.</p><p><strong>Medical Diagnosis and Research:</strong> In healthcare, vector databases help clinicians diagnose and researchers perform studies by indexing patient data, medical images, and articles. These databases, through comparing the features of the patients, medical records, and diagnostic images, help clinicians in making well informed decisions and scientists in discovering the hidden trends for disease diagnosis and treatment.</p><p><strong>Genomic Analysis:</strong> Vector databases find applications in genomic analysis by indexing genetic sequences and molecular data. Researchers will have fast and easy access to similar sequences or to recognize gene variations related to diseases that will hasten advancement in personalized medicine and genetic research.</p><p><strong>Supply Chain Optimization:</strong> Vector database technology facilitates routing optimization, inventory management, and demand forecasting in logistics and supply chain management. A comprehensive approach to spatial data analysis that involves indexing spatial data and historical trends, these databases permit enterprises to select optimal routes, apply resources effectively, and prevent supply chain problems which contribute to improving operational efficiency and reducing the cost.</p><h4>Popular Tools</h4><ul><li><a href="https://www.pinecone.io/"><strong>Pinecone</strong></a><strong>: </strong>Pinecone is a massively scalable vector database offering real time solutions without any delays. The support for different algorithms of indexing such as HNSW and PQ of Pinecone lets it do the search for similarity in a very efficient manner over large datasets. Its well-known integration feature and flexible options of deployment for content recommendation, image retrieval, and NLP (Natural Language Processing) tasks make it an indispensable tool for many developers.</li><li><a href="https://github.com/weaviate/weaviate"><strong>Weaviate</strong></a><strong>:</strong> Weaviate is an open source vector database tool leveraging advanced machine learning to transform data such as text and images into searchable vectors. It performs searches on millions of objects in milliseconds, ensuring high speed. Weaviate offers flexibility by supporting data vectorization during import or using pre-vectorized data, and it integrates with many popular services. Built for production, it includes features for scaling and security and provides additional capabilities like recommendations and summarization.</li><li><a href="https://github.com/facebookresearch/faiss"><strong>Faiss</strong></a><strong>:</strong> Developed by <em>Facebook AI Research (FAIR)</em>, Faiss is an open source library for fast similarity search and factoring in dense vectors. It provides the implementation of well-known methods such as HNSW, PQ, and LSH which makes it very useful in the large-scale applications in the field of image recognition, natural language understanding, and recommendation systems. The use of Faiss and its data structures optimizations, as well as the support for rapid processing with the help of GPUs, make it a perfect tool for high-volume vector search purposes.</li><li><a href="https://milvus.io/"><strong>Milvus</strong></a><strong>: </strong>Milvus is an open platform based on the vector database and can be used for vector similarity search with more scale. It supports many kinds of data including HNSW, PQ and LSH indexes as well as image or gene related searches making it suitable for various types of applications such as multimedia retrieval, bioinformatics and geospatial analysis. The plug-and-play Milvus’s modular structure, along with a RESTful API, both are essential in cloud computing, making it a commonly used platform for both research and production setups.</li><li><a href="https://github.com/tensorflow/similarity"><strong>TensorFlow Similarity</strong></a><strong>:</strong> The TensorFlow Similarity library helps to construct vector similarity search systems using <a href="https://www.tensorflow.org/">TensorFlow</a>. This toolkit gives you the means to create nearest neighbor indexes, query vectors, and evaluate the overall search performance. TensorFlow Similarity integrates with machine learning ecosystem built with TensorFlow and its tools and models, which means that users can build entire pipelines for similarity search for various fields such as image retrieval, recommendation systems, and natural language processing.</li><li><a href="https://www.trychroma.com/"><strong>Chroma</strong></a><strong>: </strong>Chroma is a versatile vector database designed for multimedia content indexing and retrieval. It offers efficient indexing and querying of image, audio, and video embeddings, enabling applications such as reverse image search, music recommendation, and video content discovery. With support for diverse indexing techniques and integration with popular machine learning frameworks, Chroma provides a robust solution for multimedia similarity search tasks.</li></ul><h4>Summary</h4><p>In summary, vector databases offer a powerful solution for indexing, storing, and retrieving complex data types efficiently. By leveraging advanced indexing algorithms and embedding techniques, these databases enable fast and accurate similarity search across large datasets, making them invaluable for a wide range of applications. From content recommendation and natural language processing to image retrieval and anomaly detection, vector databases play a crucial role in modern data-driven systems. However, challenges such as resource intensiveness and dependency on embedding models highlight the need for careful consideration and ongoing optimization. Despite these challenges, the versatility and scalability of vector databases continue to drive innovation in diverse fields, promising exciting opportunities for future developments and applications.</p><p>Thank you for reading.</p><p><strong>Authors:</strong></p><ul><li><a href="https://medium.com/u/6e6fe90d9ac5">Arif Enes Aydın</a></li><li><a href="https://medium.com/u/5b04438401f3">Alihan Sağöz</a></li></ul><h4>References</h4><ol><li><a href="https://www.ibm.com/topics/vector-database">https://www.ibm.com/topics/vector-database</a></li><li><a href="https://weaviate.io/blog/what-is-a-vector-database">https://weaviate.io/blog/what-is-a-vector-database</a></li><li><a href="https://www.cloudflare.com/learning/ai/what-is-vector-database/">https://www.cloudflare.com/learning/ai/what-is-vector-database/</a></li><li><a href="https://www.cloudflare.com/learning/ai/what-are-embeddings/">https://www.cloudflare.com/learning/ai/what-are-embeddings/</a></li><li><a href="https://en.wikipedia.org/wiki/Vector_database">https://en.wikipedia.org/wiki/Vector_database</a></li><li><a href="https://learn.microsoft.com/en-us/semantic-kernel/memories/embeddings">https://learn.microsoft.com/en-us/semantic-kernel/memories/embeddings</a></li><li><a href="https://www.nvidia.com/en-us/glossary/vector-database/">https://www.nvidia.com/en-us/glossary/vector-database/</a></li><li><a href="https://learn.microsoft.com/en-us/semantic-kernel/memories/vector-db">https://learn.microsoft.com/en-us/semantic-kernel/memories/vector-db</a></li><li><a href="https://www.pinecone.io/learn/vector-database/">https://www.pinecone.io/learn/vector-database/</a></li><li><a href="https://www.pinecone.io/learn/vector-embeddings/">https://www.pinecone.io/learn/vector-embeddings/</a></li><li><a href="https://aws.amazon.com/what-is/vector-databases/">https://aws.amazon.com/what-is/vector-databases/</a></li><li><a href="https://www.pinecone.io/learn/vector-embeddings-for-developers/">https://www.pinecone.io/learn/vector-embeddings-for-developers/</a></li><li><a href="https://www.elastic.co/what-is/vector-database">https://www.elastic.co/what-is/vector-database</a></li><li><a href="https://www.elastic.co/what-is/vector-embedding">https://www.elastic.co/what-is/vector-embedding</a></li><li><a href="https://www.pinecone.io/learn/series/rag/embedding-models-rundown/">https://www.pinecone.io/learn/series/rag/embedding-models-rundown/</a></li><li><a href="https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/">https://www.deeplearning.ai/short-courses/vector-databases-embeddings-applications/</a></li><li><a href="https://www.deeplearning.ai/short-courses/building-applications-vector-databases/">https://www.deeplearning.ai/short-courses/building-applications-vector-databases/</a></li><li><a href="https://www.pinecone.io/learn/series/faiss/hnsw/">https://www.pinecone.io/learn/series/faiss/hnsw/</a></li><li><a href="https://www.datastax.com/guides/hierarchical-navigable-small-worlds#navigable-small-words-nsw">https://www.datastax.com/guides/hierarchical-navigable-small-worlds#navigable-small-words-nsw</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=435d8a8742bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Final Week — MOOC Recommendation]]></title>
            <link>https://medium.com/ain311-fall-2022-projects/final-week-mooc-recommendation-2e0cc740de0f?source=rss-6e6fe90d9ac5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e0cc740de0f</guid>
            <category><![CDATA[collaborative-filtering]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Arif Enes Aydın]]></dc:creator>
            <pubDate>Sun, 01 Jan 2023 17:58:36 GMT</pubDate>
            <atom:updated>2023-01-02T09:44:09.784Z</atom:updated>
            <content:encoded><![CDATA[<h3>Final Week — MOOC Recommendation</h3><p>Hello everyone. Welcome to the last blog about our course project. Last week, we preprocessed the data for comment analysis to make ratings out of them. This week, We finalized everything we have done so far and combined collaborative filtering and comment analysis.</p><p>To decide our final model for collaborative filtering, we tested different algorithms to see which was the best performing. These methods are the collection of techniques that is available on Surprise Scikit. Some of them are memory-based and some of them are model-based.</p><p>Here are the root-mean-squared-error values of the different methods we tested:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c5IVmRTK0w_kSF1A_PhbmA.png" /><figcaption>Performances</figcaption></figure><p>The best-performing ones are BaselineOnly, KNNWithBaseline, and SVD algorithms. We tried to fine-tune these three to see if we can improve their performance further. After parameter searching with GridSearchCV, the results showed us that we could not improve performances with different parameters. So we chose SVD as our final model.</p><p>The previous week, we preprocessed the data for the modeling. There are a few steps before the data is ready. Let’s deep dive in.</p><p>We decided to use sentiment analysis scores to predict the rating. So we used the <em>vaderSentiment </em>library to analyze the comments. <em>SentimentIntensityAnalyzer </em>function analyses the comments and returns a dict that contains scores such as:</p><ul><li><em>Negative(neg):</em> Negativity value of the comment</li><li><em>Neutral(neu):</em> Neutrality value of the comment</li><li><em>Positive(pos):</em> Positivity value of the comment</li><li><em>Compound(compound):</em> Compound score of the comment</li></ul><p>After the process, the data looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*Wyy4H1wwibmwAltK.png" /><figcaption>First 5 comments with sentiment scores and rates</figcaption></figure><p>Since all of the features turned into numeric, our solution must be a regression model. At first, we decided to use 4 different algorithms and select the best one by considering their RMSE scores. Also, we tried some variations such as normalization and standardization of the data, and evaluate the effects of these operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*D_M04ak29I4FRSr4.png" /><figcaption>Performances for each algorithm and scaling</figcaption></figure><p>As you may guess, we chose the SVR with no scaling. We initialized some values for three hyperparameters of SVR in a list. By using Grid-Search CV, optimal parameters are determined. After that, we fitted the new SVR model with tuned hyperparameters and the result was not impressive. RMSE score decreased by only 0,001.</p><p>After the disappointment with the result, we thought some things are wrong with the data. So we considered the sub-intervals separately and evaluate how the model predicts each sub-interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*LckNVEDMAT6-pa4u.png" /><figcaption>Average ratings of each sub-interval for actual and predicted values</figcaption></figure><ul><li>In 0–1 interval, the predictions are higher than the actual</li><li>In 4–5 interval, the predictions are lower than the actual</li><li>For other intervals, there are not so many differences</li></ul><p>Why our model is unsuccessful while predicting the 0–1 and 4–5 cases? We think users can be generous or ungenerous. In their comments, their behaviors imply that they are neutral but they can give a high or low rating. This situation affects the results significantly. So let’s detect these inconsistencies.</p><ul><li>Comments rated less than 3 with higher positive scores than negative scores must be considered an inconsistency</li><li>Comments rated higher than 3 with higher negative scores than positive scores must be considered as inconsistency as well</li></ul><p>After eliminating the inconsistent comments, we have seen that approximately 26% of the comments are inconsistent. 94% of the inconsistencies came from positive comments rated very low. RMSE score without inconsistent comments is <strong>0.853. </strong>This means further analysis decreased the RMSE score to 0.235. This improvement is quite high if we compare it with 0.001.</p><p>On our dataset, we applied comment analysis methods to get ratings different than the true ratings given by the user. With these ratings, we fitted our model to see if the results improved or worsened.</p><ul><li>With actual ratings, the RMSE score was ~0.70</li><li>With predicted ratings, we obtained the RMSE as ~0.95</li></ul><p>With the predicted ratings, the obtained results were worse than the true ratings. But it is still capable of making recommendations.</p><p>Finally, we built two demos to show example recommendations and comment analyses.</p><ul><li><a href="https://huggingface.co/spaces/ruvinya/ain311project">https://huggingface.co/spaces/ruvinya/ain311project</a></li><li><a href="https://huggingface.co/spaces/MAli7319/Comment_Analysis">https://huggingface.co/spaces/MAli7319/Comment_Analysis</a></li></ul><p>With this, we have come to an end for both our blog posts and our course project. We have learned many different things during this period. Thank you for your interest in our project journey and we wish you pleasant days.</p><p><strong>Contributors</strong>:</p><ul><li>Arif Enes Aydın</li><li><a href="https://medium.com/u/64a8f601f769">Muhammet Ali Şentürk</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e0cc740de0f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ain311-fall-2022-projects/final-week-mooc-recommendation-2e0cc740de0f">Final Week — MOOC Recommendation</a> was originally published in <a href="https://medium.com/ain311-fall-2022-projects">AIN311 Fall 2022 Projects</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Week 3 — MOOC Recommendation]]></title>
            <link>https://medium.com/ain311-fall-2022-projects/week-3-mooc-recommendation-24fc22cfbef6?source=rss-6e6fe90d9ac5------2</link>
            <guid isPermaLink="false">https://medium.com/p/24fc22cfbef6</guid>
            <category><![CDATA[collaborative-filtering]]></category>
            <dc:creator><![CDATA[Arif Enes Aydın]]></dc:creator>
            <pubDate>Sun, 11 Dec 2022 17:11:27 GMT</pubDate>
            <atom:updated>2022-12-12T06:31:29.669Z</atom:updated>
            <content:encoded><![CDATA[<h3>Week 3 — MOOC Recommendation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UIXMRQv0H0u-qf66.png" /></figure><p>Hello everyone. Last week, we talked about the difficulties we had with the data set and our new data set. This week we will talk about how we process our data set.</p><p>Since there is no user ID in the dataset, we thought of grouping appropriate names to give them identity. Here are the unique names in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*e2wB-CkT_hE_8v7O84ZRcQ.png" /></figure><p>There were too many entries with the same name. To overcome this problem, we thought intuitively: There should be almost nobody ever has purchased more than 10–15 courses and reviewed them. With this idea, we discarded all entries that include the same display name of more than 15.</p><p>We also found that the vast majority of courses had very few reviews. We also removed the entries that included these courses otherwise, the size of the utility matrix that we need to use became enormously larger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0mJCE6zU8CZGhXMBG5Z4Q.png" /></figure><p>With these alterations, we gave the users that we are left an id number. As a result, we have approximately 100k+ reviews from 10k+ users for 3k+ courses. These statistics are subject to change due to needs in the recommendation system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*N4QiosR_4kLy4yNXAGL1zg.png" /><figcaption>fraction of a utility matrix</figcaption></figure><p>Since our progress is a little delayed, we barely started implementing the recommendation system. This week, we explored the needs and capabilities of the system. Firstly, we normalized the matrix and then calculated the cosine similarity. The results is can be used for finding which courses most similar to a given course. Here are the similar courses to the course with id <em>15639</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*E4oQDRmm6sickSuqKUFFFQ.png" /></figure><p>That was all for today. Next week we will consider other options to include our system by doing a literature review and will prepare the progress report for our project. It is hard to decide which way to go since there are not too many sources compared to other machine learning methods.</p><p>Thank you for reading. Hope to see you next week.</p><p><strong><em>Authors</em></strong></p><ul><li>Arif Enes Aydın</li><li>Muhammet Ali Şentürk (<a href="https://medium.com/u/64a8f601f769">Muhammet Ali Şentürk</a>)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24fc22cfbef6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ain311-fall-2022-projects/week-3-mooc-recommendation-24fc22cfbef6">Week 3 — MOOC Recommendation</a> was originally published in <a href="https://medium.com/ain311-fall-2022-projects">AIN311 Fall 2022 Projects</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>