<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prajwal Shreyas on Medium]]></title>
        <description><![CDATA[Stories by Prajwal Shreyas on Medium]]></description>
        <link>https://medium.com/@prajwalshreyas?source=rss-daf1da2a65fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*icUgBe3HTwAJt--svKBHfQ.jpeg</url>
            <title>Stories by Prajwal Shreyas on Medium</title>
            <link>https://medium.com/@prajwalshreyas?source=rss-daf1da2a65fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:40:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prajwalshreyas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optimising Model Inference: A Practical Guide]]></title>
            <link>https://medium.com/analytics-vidhya/optimising-model-inference-a-practical-guide-cbf616f433a4?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf616f433a4</guid>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[optimisation]]></category>
            <category><![CDATA[triton-inference-server]]></category>
            <category><![CDATA[inference]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Mon, 25 Nov 2024 13:17:13 GMT</pubDate>
            <atom:updated>2024-11-25T13:17:13.436Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULI6UpFjg-ZUO5ZEQLVfHQ.jpeg" /></figure><p>Deploying and optimising machine learning models is a key skill for any ML engineer. Efficient inference helps reduce costs, improve latency, and ultimately provide a better user experience. In recent years, LLM (Large Language Model) inference optimisation has become a major focus area, and it’s been exciting to see the rapid advancements that have taken place. Techniques like flash attention v1/v2, which makes the attention mechanism more efficient, and KV (Key-Value) cache optimisation, which helps reduce redundant computations, have been game-changers. These innovations have resulted in significant reductions in inference costs, especially over the last two years.</p><p>In this blog series, I’m going to start small. We’ll begin with a simple example — training a toy deep learning model with the MNIST dataset. From there, we’ll gradually explore some effective ways to optimise inference, taking steps that even beginners can follow. You can find the underlying code in the repo <a href="https://gitlab.com/praj88/llm-optimisation/-/tree/main/inf_opt_1">here</a>. My goal is to make these concepts approachable and easy to understand so you can apply them in your own projects. Let’s dive in!</p><h3>1. Training and Saving a Simple Model</h3><p>We’ll start by training a simple TensorFlow model on the MNIST dataset. MNIST, a dataset of handwritten digits, is a great starting point to demonstrate how to move from model training to optimisation.</p><p>First, we train the model using TensorFlow and save it in the Protobuf format (.pb file). The .pb (Protocol Buffer) format is a versatile representation for TensorFlow models, encapsulating the model graph, weights, and metadata in a highly efficient, portable manner. It allows the model to be easily loaded for inference in various environments, whether on a server, an edge device, or within another TensorFlow session. This format is particularly useful because it preserves the model’s computational graph, enabling optimized inference execution. The .pb file serves as the starting point for our inference journey.</p><pre>import tensorflow as tf<br>from tensorflow.keras.datasets import mnist<br><br># Load and preprocess data<br>(x_train, y_train), (x_test, y_test) = mnist.load_data()<br>x_train, x_test = x_train / 255.0, x_test / 255.0<br># Define the model<br>model = tf.keras.models.Sequential([<br>    tf.keras.layers.Flatten(input_shape=(28, 28)),<br>    tf.keras.layers.Dense(128, activation=&#39;relu&#39;),<br>    tf.keras.layers.Dropout(0.2),<br>    tf.keras.layers.Dense(10, activation=&#39;softmax&#39;)<br>])<br># Compile and train the model<br>model.compile(optimizer=&#39;adam&#39;, loss=&#39;sparse_categorical_crossentropy&#39;, metrics=[&#39;accuracy&#39;])<br>model.fit(x_train, y_train, epochs=5)<br># Save the model<br>model.save(&#39;saved_model/mnist_model.pb&#39;)</pre><p>After training, we save the model as a .pb file, which we’ll use in the next step to evaluate inference performance.</p><h3>2. Benchmarking Inference with the .pb Model</h3><p>The next step is to evaluate the model’s inference performance by benchmarking latency and throughput. We use the provided script benchmarks/base_inf_bm.py for this purpose. The script loads the saved .pb model and measures the time taken to make predictions on the test dataset.</p><p>Here are the benchmarking results for the .pb model:</p><pre>Inference Benchmarking Results:<br>p25 Latency: 10.293906571289062 ms<br>p50 Latency: 10.822534561157227 ms<br>p75 Latency: 11.699853527606201 ms<br>p99 Latency: 14.364442823517505 ms<br>Mean Latency: 11.323435306549072 ms<br>Throughput: 5651.995023364038 samples/sec</pre><p>In these results, we measure latency at different percentiles (p25, p50, p75, p99), as well as the mean latency. Latency represents the time taken to make a prediction, and throughput measures how many predictions can be processed per second. These metrics are critical for evaluating model performance, especially when optimizing for either response time or the ability to handle high request loads.</p><p>Other useful metrics that can be considered include:</p><ul><li><strong>Memory Usage</strong>: How much memory the model requires during inference. Optimizing memory usage can be important for deployment on resource-constrained environments.</li><li><strong>CPU/GPU Utilization</strong>: Understanding how efficiently the model utilizes the available hardware can help in further optimization.</li></ul><h3>3. Converting to ONNX and furthur benchmarking</h3><p>The next step is to evaluate the model’s inference performance by benchmarking latency and throughput. We use the provided script benchmarks/base_inf_bm.py for this purpose. The script loads the saved .pb model and measures the time taken to make predictions on the test dataset.</p><p>After establishing this baseline, we convert the model to the ONNX format for further optimizations. The ONNX format has become a popular choice for improving inference performance, as it supports several optimizations that significantly reduce latency and improve throughput. We use the ONNX package to convert our TensorFlow model to ONNX.</p><p>The ONNX conversion process involves saving the model in the .pb format, using tf2onnx to convert it to ONNX, and then optionally converting it to FP16 for optimized inference. The use of ONNX introduces several graph-level optimizations, operator improvements, and reduced precision capabilities, all of which contribute to faster inference.</p><p>Here’s the custom script used for the conversion:</p><pre>import tensorflow as tf<br>import os<br>import onnx<br>from onnxmltools.utils.float16_converter import convert_float_to_float16<br>from tensorflow.python.compiler.tensorrt import trt_convert as trt<br><br># Define paths<br>saved_model_dir = &#39;inf_opt_1/models/mnist_model&#39;<br>onnx_model_path = &#39;inf_opt_1/models/mnist_model.onnx&#39;<br>onnx_fp16_model_path = &#39;inf_opt_1/models/mnist_fp16_model.onnx&#39;<br>################## Convert TensorFlow SavedModel to ONNX<br>os.system(f&#39;python -m tf2onnx.convert --saved-model {saved_model_dir} --output {onnx_model_path} --opset 13&#39;)<br># Load the FP32 ONNX model<br>onnx_model = onnx.load(onnx_model_path)<br># Convert the model to FP16<br>fp16_model = convert_float_to_float16(onnx_model)<br># Save the FP16 model<br>onnx.save(fp16_model, onnx_fp16_model_path)</pre><p>In this script, we use TensorFlow to load the saved model and onnxmltools to convert the ONNX model from FP32 to FP16, enabling faster inference. The tf2onnx module is invoked through a system command to perform the initial conversion, specifying an opset version (in this case, opset 13) for compatibility.</p><p>Once converted, we benchmark the model using benchmarks/onnx_inf_bm.py to observe the performance improvements.</p><p>To get a better understanding of how well our model performs, we’ll measure:</p><ul><li><strong>Latency</strong>: The time taken to make a single prediction.</li><li><strong>Throughput</strong>: The number of predictions processed per second.</li></ul><pre>python benchmarks/base_inf_bm.py --model_path saved_model/mnist_model.pb</pre><p>This will give us a baseline performance of our TensorFlow model before any optimizations.</p><h3>Benchmarking Results and Analysis</h3><p>Here are the benchmarking results we obtained for the FP32 ONNX model:</p><pre>Inference Benchmarking Results:<br>p25 Latency: 2.682209014892578 ms<br>p50 Latency: 2.740025520324707 ms<br>p75 Latency: 2.7539730072021484 ms<br>p99 Latency: 2.960498332977504 ms<br>Mean Latency: 2.8399025207824707 ms<br>Throughput: 22535.841947564913 samples/sec</pre><p>We also benchmarked the FP16 ONNX model, and here are the results:</p><pre>Performing FP16 model benchmarking<br>Inference Benchmarking Results:<br>p25 Latency: 2.973616123199463 ms<br>p50 Latency: 3.0012130737304688 ms<br>p75 Latency: 3.067314624786377 ms<br>p99 Latency: 3.2110691070556934 ms<br>Mean Latency: 3.077104091644287 ms<br>Throughput: 20798.77641246801 samples/sec</pre><p>Interestingly, we see that the FP16 model shows slightly lower performance compared to the FP32 model when running on CPUs. The mean latency increased from approximately 2.84 ms to 3.08 ms, and the throughput decreased from around 22536 samples per second to 20799 samples per second.</p><p>The reason for this behaviour is tied to how CPUs handle lower-precision data types. While FP16 is designed to reduce computation times by using fewer bits, CPUs are generally optimized for FP32 arithmetic and often lack specialized hardware support for FP16 operations. As a result, executing FP16 operations on CPUs can introduce overhead instead of improving speed. FP16 precision is primarily beneficial on GPUs that have dedicated hardware (such as tensor cores) capable of efficiently handling reduced-precision calculations, leading to substantial gains in performance. Therefore, using FP16 on CPU hardware can sometimes result in lower performance compared to FP32.</p><p>Comparing these results with the original TensorFlow model’s performance, we can see a significant improvement with ONNX. The mean latency decreased from approximately <strong>11.32 ms to 2.84 ms</strong> for the FP32 ONNX model, which is nearly a <strong>4x reduction</strong>. Additionally, the throughput increased from around 5652 samples per second to 22536 samples per second, demonstrating the efficiency of the ONNX Runtime in optimizing inference workloads.</p><p>The uplift in performance using ONNX can be attributed to several factors:</p><ul><li><strong>Graph Optimizations</strong>: ONNX Runtime applies various graph optimizations, such as node fusion, constant folding, and redundant node elimination, to reduce the number of computations required during inference.</li><li><strong>Operator Optimizations</strong>: ONNX leverages optimized operator implementations that make use of highly efficient backend libraries like Intel MKL or NVIDIA TensorRT, which are tailored for fast execution.</li><li><strong>Kernel Fusion</strong>: ONNX Runtime combines multiple operations into a single kernel where possible, minimizing the overhead associated with multiple kernel launches and leading to faster inference.</li><li><strong>Precision Management</strong>: ONNX supports reduced precision (e.g., FP16, INT8) to speed up inference on compatible hardware, although this benefit is more pronounced on GPUs rather than CPUs.</li></ul><p>These optimizations explain why the ONNX version of the model outperforms the original TensorFlow version so significantly when run on the same hardware.</p><h3>4. Deploying with Triton Inference Server and Batch Size Tuning</h3><p>To scale model inference in production, using a specialized inference server like NVIDIA’s Triton Inference Server can be highly beneficial. Triton is designed to handle multiple models and provide optimizations for both latency and throughput.</p><p>We’ll use Docker to pull the Triton Inference Server image and deploy our ONNX model:</p><pre>docker pull nvcr.io/nvidia/tritonserver:latest<br>docker run --rm -p8000:8000 -p8001:8001 -p8002:8002 -v $(pwd)/model_repository:/models nvcr.io/nvidia/tritonserver:latest tritonserver --model-repository=/models</pre><p>The /model_repo directory should contain our ONNX model in a specific structure that Triton expects. Each model should be in its own directory, with a configuration file (config.pbtxt) that defines various parameters such as platform, input/output information, and batch size. The structure should look like this:</p><pre>/model_repo<br>    /mnist_model<br>        /1<br>            model.onnx<br>        config.pbtxt</pre><ul><li><strong>Model Directory</strong>: Each model has its own directory (e.g., mnist_model), which contains a subdirectory (/1) for versioning. The model file (model.onnx) is stored inside the version folder.</li><li><strong>Config File</strong>: The config.pbtxt file defines the model&#39;s configuration, such as the type of model, input shapes, output shapes, batch sizes, and optimization settings. This configuration helps Triton manage the model efficiently and make decisions related to batching and hardware optimization. Once the server is up and running, we can use benchmarking tools such as perf_analyzer to measure the inference performance.</li></ul><p>We benchmarked the ONNX model deployed on Triton Inference Server with two different batch sizes — 8 and 64 — to observe the effect of batch size on performance.</p><p>To further analyze the performance, we also collected trace logs from Triton Inference Server. These logs provided detailed breakdowns of request handling times, including receive, send, compute, and infer times for each batch size.</p><h4>batch size 8</h4><p>Here are the benchmarking results and trace analysis for :</p><pre>Inference Benchmarking Results:<br>p25 Latency: 2.001523971557617 ms<br>p50 Latency: 2.6334524154663086 ms<br>p75 Latency: 3.416776657104492 ms<br>p99 Latency: 14.781000614166631 ms<br>Mean Latency: 3.7192511558532715 ms<br>Throughput: 2150.9706295069063 samples/sec</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*n9nyApcMp7uIyB8IB40VKA.png" /><figcaption>tracer log for batch 8 triton inference server</figcaption></figure><p>In the trace log for batch size 8, we observe that the majority of time is spent in the handler’s compute phase, particularly during inference (1580.40 micro seconds on average). The HTTP overhead and request handling are also non-negligible, contributing to the overall latency.</p><h4>batch size 64:</h4><pre>Inference Benchmarking Results:<br>p25 Latency: 6.688952445983887 ms<br>p50 Latency: 7.341504096984863 ms<br>p75 Latency: 9.792625904083252 ms<br>p99 Latency: 20.493354797363622 ms<br>Mean Latency: 9.151227474212646 ms<br>Throughput: 6993.597326735279 samples/sec</pre><p>From the trace log for batch size 64:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*IN7V0xazuH-U0OEhpvjDpw.png" /><figcaption>tracer log for batch 64 triton inference server</figcaption></figure><p>For batch size 64, the average latency per inference request increased significantly compared to batch size 8, lets explore the trade-offs in latency and through put for the two batch sizes.</p><h3>Analysis of Overheads and Inference Times</h3><p>The trace logs reveal that as the batch size increases, the inference time per request decreases, leading to improved throughput. This is because the larger batch allows for more efficient utilization of compute resources, amortizing the overhead across more samples. For batch size 8, the average latency time (2605us) i.e. 325us per inference, which is significantly higher compared to batch size 64 (6566us) i.e. 102us per inference. This is a <strong>3x</strong> improvement in performance.</p><p>From the benchmarking results, we see that batch size 8 has a lower mean latency (3.72 ms) and lower throughput (2150.97 samples/sec), while batch size 64 shows higher latency (9.15 ms) but significantly better throughput (6993.60 samples/sec). This highlights the trade-off between latency and throughput: batch size 64 improves throughput by processing more data simultaneously, but it requires more time for each batch, resulting in <strong>higher latency per batched request</strong>.</p><p>The underlying hardware utilization also plays a role here. With larger batch sizes, the hardware (such as CPU or GPU) is utilized more efficiently, as it processes <strong>multiple inputs in parallel</strong>, thus maximizing resource usage. However, this efficiency comes at the cost of increased processing time per batch, which directly impacts latency.</p><p>Ultimately, these results illustrate the trade-off between throughput and latency when tuning batch sizes. Batch size 64 provides significantly better throughput at the cost of increased latency, while batch size 8 offers lower latency but reduced throughput. The choice of batch size should be based on the specific requirements of the application — whether it prioritizes response time or the ability to process more data per unit time.</p><h4>Comparing these results of triton inference server vs raw CPU performance:</h4><p>We observe that both latency and throughput are lower when using Triton Inference Server in a Docker container compared to running the FP16 ONNX model directly on the CPU. This is primarily due to the additional overhead introduced by the container environment and Triton Server’s features, such as logging, model management, and HTTP request handling, which add delays to the inference process.</p><p>Another reason for the lower performance is the memory limitation imposed by the Docker container. By default, the container’s memory limit is set to 16GB, whereas running the model directly on the CPU allows access to the full system memory i.e. 32GB for the mac where this test was conducted, which can result in better performance. The increased memory availability leads to more efficient data handling and reduced swapping, positively impacting both latency and throughput.</p><p>Despite the reduced throughput and increased latency, Triton Server provides significant advantages in production, such as dynamic batching, model versioning, and easy scalability, which often outweigh the performance losses introduced by the Docker environment.</p><p>Experimenting with different batch sizes allows you to find the optimal balance between latency and throughput, depending on your application’s requirements.</p><h3>Conclusion</h3><p>In this blog, we’ve explored how to optimize the inference of a simple TensorFlow model using multiple tools and techniques:</p><ul><li>Training and saving a model as a .pb file.</li><li>Benchmarking base performance.</li><li>Converting to ONNX for better inference performance.</li><li>Deploying the model using Triton Inference Server.</li><li>Tuning batch sizes to find the best trade-off between latency and throughput.</li></ul><p>These steps demonstrate how you can take a straightforward model and, with a series of optimizations, make it production-ready and highly efficient.</p><p>I hope this guide helps you better understand model optimization techniques! In the next blogs, we will delve deeper into kernel operations and hardware-specific accelerations that enable faster inference. If you have questions or want to share your own experience with model inference optimization, feel free to leave a comment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf616f433a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/optimising-model-inference-a-practical-guide-cbf616f433a4">Optimising Model Inference: A Practical Guide</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BERT vs GPT3: What’s the differece?]]></title>
            <link>https://medium.com/analytics-vidhya/bert-vs-gpt3-whats-the-differece-6304ea9f531c?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6304ea9f531c</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Sat, 03 Aug 2024 19:16:54 GMT</pubDate>
            <atom:updated>2024-08-03T19:18:38.609Z</atom:updated>
            <content:encoded><![CDATA[<p>The key differences between BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) models lie in their architectures, training objectives, and the types of problems they are designed to solve. Here’s a detailed comparison:</p><h3>Architecture</h3><p><strong>BERT</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*623YgCYLAWh4wklb.png" /><figcaption>Source: <a href="https://huggingface.co/blog/bert-101">https://huggingface.co/blog/bert-101</a></figcaption></figure><ul><li><strong>Architecture</strong>: BERT uses a transformer encoder architecture.</li><li><strong>Bidirectional</strong>: BERT is designed to read text in both directions (left-to-right and right-to-left) simultaneously, capturing context from both sides.</li><li><strong>Layers</strong>: Typically includes multiple transformer encoder layers (e.g., BERT-base has 12 layers, BERT-large has 24 layers).</li></ul><p><strong>GPT3</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*fwisc7ZWnsHVKbRy.png" /><figcaption>Source: <a href="https://en.m.wikipedia.org/wiki/File:Full_GPT_architecture.png">https://en.m.wikipedia.org/wiki/File:Full_GPT_architecture.png</a></figcaption></figure><ul><li><strong>Architecture</strong>: GPT uses a transformer decoder architecture.</li><li><strong>Unidirectional</strong>: GPT reads text unidirectionally (left-to-right), which aligns with how humans generally read and write.</li><li><strong>Layers</strong>: Consists of multiple transformer decoder layers (e.g., GPT-2 has 12 layers, GPT-3 has up to 96 layers).</li></ul><h3>Training Objectives</h3><p><strong>BERT</strong>:</p><ul><li><strong>Masked Language Modeling (MLM)</strong>: BERT is trained by randomly masking some of the tokens in the input and predicting them based on the context provided by the other, non-masked tokens. This allows BERT to learn bidirectional representations.</li><li><strong>Next Sentence Prediction (NSP)</strong>: BERT is also trained to predict whether a given pair of sentences follows consecutively in the original text, which helps in understanding sentence relationships.</li></ul><p><strong>GPT</strong>:</p><ul><li><strong>Auto-regressive Language Modeling</strong>: GPT is trained to predict the next word in a sentence based on the preceding words. This unidirectional approach makes GPT particularly strong at generating coherent and contextually relevant text.</li><li><strong>No Sentence-level Objective</strong>: GPT does not have a next sentence prediction task, focusing instead on the generation of the next token.</li></ul><h3>Types of Problems Solved</h3><p><strong>BERT</strong>:</p><p><strong>Natural Language Understanding (NLU)</strong>: BERT excels in tasks that require understanding the context of text, such as:</p><ul><li>Sentiment analysis</li><li>Named entity recognition (NER)</li><li>Question answering (e.g., SQuAD)</li><li>Text classification</li><li>Sentence similarity</li></ul><p><strong>Bidirectional Context</strong>: The bidirectional nature of BERT makes it particularly effective for tasks where understanding the full context of a sentence or passage is crucial.</p><p><strong>GPT</strong>:</p><p><strong>Natural Language Generation (NLG)</strong>: GPT is designed to generate coherent and contextually appropriate text, making it well-suited for tasks such as:</p><ul><li>Text completion</li><li>Story and article generation</li><li>Dialogue generation</li><li>Machine translation (to some extent)</li><li>Summarization (as it can generate summaries based on the preceding context)</li></ul><p><strong>Unidirectional Context</strong>: The unidirectional context makes GPT more natural for generation tasks where predicting the next word based on prior context is key.</p><h3>Summary</h3><p><strong>BERT</strong> is primarily a tool for understanding and interpreting text (NLU). Its bidirectional training allows it to capture context from both sides, making it ideal for comprehension tasks.</p><p><strong>GPT</strong> is geared towards generating text (NLG). Its unidirectional, auto-regressive training enables it to produce coherent and contextually relevant text, making it suitable for generative tasks.</p><p>Both models have their strengths and are designed to tackle different types of language tasks effectively. Understanding these differences helps in selecting the appropriate model based on the specific requirements of the problem at hand.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6304ea9f531c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/bert-vs-gpt3-whats-the-differece-6304ea9f531c">BERT vs GPT3: What’s the differece?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local ai companions — solutions for privacy and personalisation ?]]></title>
            <link>https://medium.com/analytics-vidhya/local-ai-companions-solutions-for-privacy-and-personalisation-7f239299fe56?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f239299fe56</guid>
            <category><![CDATA[llama-3]]></category>
            <category><![CDATA[ai-assistant]]></category>
            <category><![CDATA[privacy-by-design]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 16:23:00 GMT</pubDate>
            <atom:updated>2024-06-03T17:32:27.911Z</atom:updated>
            <content:encoded><![CDATA[<h3>Local ai companions: solutions for privacy and personalisation ?</h3><h3>Introduction</h3><p>The advent of large language models (LLMs) has revolutionized the way we interact with technology. These AI models, capable of generating human-like text, are typically run on central servers by companies like OpenAI (chatgpt) and Google ( gemini-1.5). The debate between centrally run large language models (LLMs) and locally run smaller LLMs is a critical discussion in the AI community, focusing on the balance between performance, privacy, and accessibility. Centrally run LLMs leverage massive computational resources to deliver high-performance and highly accurate responses. These models benefit from continuous updates, access to vast datasets, and the ability to handle complex queries efficiently. However, they come with significant privacy concerns, as user data must be transmitted to central servers, raising potential risks of data breaches and misuse. Additionally, dependence on internet connectivity and centralized control can lead to service disruptions and lack of user autonomy.</p><p>On the other hand, locally run smaller LLMs offer enhanced privacy and data security, as all computations are performed on the user’s device. This approach gives users full control over their data and ensures uninterrupted service without reliance on external servers. While local LLMs may not match the sheer scale and raw performance of their centralized counterparts, advancements in model optimization techniques, like distillation and quantization, have made it possible to achieve impressive results with more modest hardware. However, local models often face challenges in staying up-to-date with the latest information and may require more manual updates and maintenance.</p><p>In this context described above I am exploring what we can achieve utilising some of the open-source models and tooling such as langchain, ollama to deploy a LLM assistant that can run on you local machine and act as an ai-assistant(<em>Link to the </em><a href="https://gitlab.com/praj88/local-ai-companion"><em>repo</em></a>).</p><h3>Overview of Local AI Companion</h3><p>The first release of the local AI Companion is designed to be a basic personal assistant that leverages locally run language models and integrates with external tools such as google search to provide accurate and timely information.</p><p><strong>Local LLMs</strong></p><p>Local AI Companion, process data on your device, ensuring that your interactions remain private and secure. To do this we use <a href="https://www.ollama.com/">Ollama</a>, which enables the deployment and execution of large language models (LLMs) directly on local devices, enhancing data privacy and control. It optimizes model performance for local hardware and simplifies the setup process, making advanced AI capabilities accessible without extensive computational resources. Ollama also provides a number of pre-trained LLMs available for use, see the list of available models <a href="https://www.ollama.com/library">here</a>.</p><p><strong>Keeping LLMs Up-to-Date</strong></p><p>One of the significant challenges with LLMs is that they are often trained on static datasets and do not have up-to-date knowledge. This limitation necessitates a mechanism to update model responses with the latest news and information. The project integrates external APIs, like Google Search, to fetch real-time data, ensuring that the responses are current and relevant.</p><p>In the repo I have provided a google search api that can be used. However some users may prefer to use their own search api in those cases please follow this <a href="https://joeyism.medium.com/custom-google-search-api-fbbafe4711eb">blog</a>. Then replace the parameters in the config file in the repo with your own keys for <em>gsearch_api_key</em> and <em>search_engine_id</em>.</p><p><strong>Web interface</strong></p><p>I have used Streamlit as a UI for Local AI assitant. This offers a highly interactive and user-friendly interface, allowing for seamless input and output visualisation. It simplifies the development process with minimal code, enabling rapid prototyping and deployment. Additionally, Streamlit’s integration with Python ensures compatibility with the local AI models and tools, enhancing overall functionality and user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eElrQxRbFGQV4oPeYwq4ig.png" /></figure><p><strong>Features</strong></p><ul><li><strong>Local LLM — Llama3</strong>: Utilizes Llama3, a state-of-the-art language model, running locally on your device to ensure privacy and quick response times.</li><li><strong>Agent Tools</strong>: Integrates agent tools such as Google Search via API to fetch real-time data, keeping the model’s responses current.</li><li><strong>Customization with Hashtags</strong>: Allows users to customize their queries using specific hashtags:</li></ul><p><em>#gs: For Google Search queries.</em></p><p><em>#lcl: For questions directed to the local LLM only (Privacy centric).</em></p><p><em>#ag: For agent-based queries.</em></p><h3>Trade-offs Between Chains and Agents for Tasks</h3><p>During the development of the tool I experimented with utilising just the agent to carry all the required tasks. However during the testing it was noticed the the agents sometimes provides sub-optimal answers and are prone to getting stuck in loops. Hence to avoid this situation there is a chain that is enable to handle most of user queries.</p><p>In the context of AI, chains and agents represent different approaches to handling tasks.</p><ul><li><strong>Chains</strong>: Involve a sequence of operations or models that process data step-by-step. They are highly reliable for structured tasks but may lack flexibility.</li><li><strong>Agents</strong>: Are more autonomous and can make decisions based on a set of rules or goals. They offer greater flexibility and can adapt to a wider range of scenarios but may be less predictable than chains.</li></ul><p><strong>Trade-offs</strong></p><p>Choosing between chains and agents depends on the specific task requirements:</p><ul><li>For tasks requiring high reliability and structure, chains are preferable.</li><li>For tasks that require adaptability and real-time decision-making, agents are more suitable.</li></ul><h3>Installation Steps</h3><p>Setting up the Local AI Companion is straightforward. Follow these steps to get started:</p><p><strong>Clone the Repository</strong>:</p><pre>git clone https://gitlab.com/praj88/local-ai-companion.git cd local-ai-companion</pre><p><strong>Set up the Virtual Environment</strong>: Run the following Python script to create and manage the virtual environment, install necessary packages, and pull the specified model (Replace /path/to/your/virtualenvs with the desired directory for storing your virtual environments.):</p><pre>python setup_and_manage_virtual_envs.py /path/to/your/virtualenvs</pre><p><strong>Run the Application</strong>: Use the provided run_app.sh script to activate the virtual environment and start the Streamlit application:</p><pre>bash ./run.sh</pre><h3>Conclusion</h3><p>The Local AI Companion offers a robust solution for those who are keen to prioritize privacy and the latest information in their AI interactions. By running LLMs locally and integrating with real-time data sources, this tool provides a powerful and secure personal assistant.</p><p>Get started today by following the installation steps and provide feedback on any new features that you would like to request for the ai-assistant.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f239299fe56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/local-ai-companions-solutions-for-privacy-and-personalisation-7f239299fe56">Local ai companions — solutions for privacy and personalisation ?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customising LLMs for domain data using RAG]]></title>
            <link>https://medium.com/analytics-vidhya/customising-llms-for-domain-data-using-rag-d0793dee17ec?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0793dee17ec</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Sat, 20 Apr 2024 18:18:15 GMT</pubDate>
            <atom:updated>2024-04-20T18:18:15.735Z</atom:updated>
            <content:encoded><![CDATA[<p>In the realm of natural language processing, the pursuit of refining and enhancing the capabilities of large language models (LLMs) continues to drive innovation. Two prominent methodologies, fine-tuning and Retrieval-Augmented Generation (RAG), stand out as powerful techniques for optimising LLM performance and adaptability.</p><p>Fine-tuning, while effective, demands substantial data resources and computational power, raising concerns regarding overfitting and catastrophic forgetting. Conversely, RAG offers a promising alternative, leveraging external data sources to enhance LLM responses without the need for extensive customisation. We will delve into pro’s and cons of each of these methods and explore more into ways of enhancing accuracy of RAG based solutions.</p><p><strong>Fine-tuning an LLM: </strong>This approach has a few challenges as listed below.</p><ol><li>Data requirements: Fine-tuning typically requires a large amount of labeled data for the specific task or domain. Obtaining and labeling such data can be time-consuming and costly, especially for niche or specialised domains.</li><li>Computational resources: Fine-tuning LLMs often demands significant computational resources, including high-performance GPUs or TPUs, and can be computationally intensive and time-consuming.</li><li>Overfitting: Fine-tuning on a limited dataset can lead to overfitting, where the model becomes too specialized to the training data and performs poorly on unseen data. Regularization techniques and careful dataset selection are necessary to mitigate this risk.</li><li>Catastrophic forgetting: Fine-tuning can lead to catastrophic forgetting, where the model forgets previously learned knowledge while adapting to new data. Techniques such as knowledge distillation or continual learning can help mitigate this issue.</li></ol><p>Despite these challenges, fine-tuning remains a powerful technique for adapting LLMs to specific tasks or domains and can significantly improve model performance when executed carefully and responsibly. Fine-tuning is more appropriate when organisations want to modify the LLM’s behaviour or train it to understand a different language or domain-specific terminology.</p><p><strong>Retrieval Augmented Generation (RAG): </strong>RAG is an innovative approach that enhances the performance of large language models (LLMs) by incorporating custom data. RAG is suitable as a starting point and may be entirely sufficient for some use cases. It provides a straightforward way to enhance LLM responses with external data without the need for model customization. The benefits of RAG are significant, making it a valuable approach for enhancing the performance of large language models (LLMs):</p><ol><li>Up-to-date and accurate responses: RAG ensures that LLM responses are not based solely on static training data but also incorporate up-to-date external sources. This helps to provide more accurate and relevant answers.</li><li>Reduction of inaccurate responses or hallucinations: By grounding the LLM’s output on relevant external knowledge, RAG helps mitigate the risk of providing incorrect or fabricated information. This reduces the occurrence of hallucinations and can include citations of original sources for human verification.</li><li>Domain-specific and relevant responses: RAG enables LLMs to provide contextually relevant responses tailored to an organization’s proprietary or domain-specific data. This customization enhances the usefulness and relevance of the model’s output.</li><li>Efficiency and cost-effectiveness: Compared to other approaches such as fine-tuning, RAG is simpler and more cost-effective. It allows organizations to leverage existing LLM models without the need for extensive customization, making it a practical solution, especially when models require frequent updates with new data.</li></ol><p>Due to the above advantages RAG has emerged as an industry standard, allowing organisations to deploy any LLM model and customize it to their specific needs, ultimately improving efficiency and effectiveness in various applications</p><h3>How to set-up a RAG infrastructure</h3><p>In order for organisation’s to leverage Retrieval Augmented Generation (RAG) they can utilise the infrastructure provided by the public cloud providers such as AWS. In the below section I will cover high level steps that can we followed for setting up a RAG infrastructure in AWS. Detailed code execution can be obtained in the links attached within the refrence section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/0*1C7XH-11cxOEY7Uv.png" /><figcaption><a href="https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/">https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/</a></figcaption></figure><p>The typical RAG approach involves three main steps:</p><h4><strong>Offline Batch Processing:</strong></h4><p>Initially, documents from a knowledge base are processed in batches. These documents are split into smaller chunks, and each chunk is embedded to represent its semantics using a pre-trained embedding model like Amazon Titan. These embeddings are then used to create a semantic search index.</p><blockquote><em>What is an Embedding ?</em></blockquote><blockquote><em>Embedding are vector representation of any object. In this case we use embedding techniques to transform text into numerical representations in a vector space. This step enables comparisons between user queries and documents within the knowledge library.</em></blockquote><p>In the above diagram the documents are chunked and embeddings extracted using a pre-trainied embedding model Titan. The embeddings are stored in open-search database and will be used for real-time processing in the subsequent step.</p><h4><strong>Real-Time Question Processing</strong></h4><p>When a new question is posed in real-time, the question is converted into an embedding by making a call to the embedding model. This embedding is used to search for and extract the most similar chunks of documents from the knowledge base using a similarity metric and an approximate nearest neighbors algorithm. Metadata filtering i.e. using keywords may also be applied to improve search precision.</p><h4><strong>Prompt Construction</strong></h4><p>A prompt is constructed by concatenating a system message with relevant chunks of documents extracted in the previous step, along with the input question itself. This prompt is then presented to a large language model (LLM) to generate the final answer based on the provided context.</p><h3>Evaluation &amp; enhancing accuracy:</h3><p>Evaluation of LLM responses is possible by establishing a ground truth dataset. Hence this is a critical step in ensuring their accuracy, reliability, and effectiveness. This process involves curating a comprehensive and representative set of data that serves as the benchmark.</p><ul><li>Define evaluation criteria and objectives for the ground truth dataset, considering the specific tasks or domains the LLM is designed to address.</li><li>Identify and collect relevant data sources, ensuring diversity in content, context, and complexity to adequately represent real-world scenarios.</li><li>Annotate the dataset with accurate labels or annotations, providing the ground truth against which the LLM’s outputs will be compared.</li><li>Ensure consistency and quality in annotations through thorough validation and quality control measures.</li><li>Establishing a robust ground truth dataset is crucial for improving the performance of LLMs, as it serves as the benchmark for evaluating and refining model outputs.</li></ul><p>Based on the results from the evaluation you can decide if additional accuracy enhancement is needed. In most real life cases this will be the case. A suitable embedding model and an efficient semantic search module, RAG solutions can effectively retrieve and generate answers to questions that require accessing existing information from a database of documents. However there may be a <strong>certain limits</strong> on the <strong>accuracy</strong> of the responses provided by the LLM. I have suggested a few ways you can further enhance this accuracy:</p><h4>Data Preparation:</h4><p>In an organisation there may be a need to restructure the domain data in order to optimise it for retrieval purposes. This process can be potentially automated using a LLM via construction of task specific prompts.</p><h4>Retrieval Enhancements:</h4><p>As part of the evaluation you will need to review the retrieved documents and rank of the retrieved document. In cases where you identify the issue is from retrieval i.e. wrong documents are being retrieved, you can explore the below mentioned options:</p><p>Entity Extraction using LLMs: To effectively answer questions, relevant metadata and entities from the document’s knowledge base must be extracted into an accessible structured data format. You can experiment with different ways of creating the embedding using the keywords extracted and storing this information in open search databases.</p><p>Re-ranking models: RAG model architectures compare the embeddings of user queries with those of documents in the knowledge library. Similar documents are identified based on the proximity of their embeddings. Howevre this rankign based similarity of embedding’s alone may not be sufficient for gettting the best documents. Hence in Retrieval-Augmented Generation (RAG), the reranker is essential for refining the results obtained from the initial retrieval process.</p><p>After the retriever generates potential candidates from a document collection, the reranker evaluates these candidates more thoroughly.The main function of the reranker is to re-rank candidates based on relevance, coherence, or other criteria to ensure the most suitable options are presented for generation.</p><ul><li>Common reranking techniques include Learning to Rank (LTR), Relevance Feedback, Semantic Similarity, Diversification, Query Expansion, and Contextual Reranking.</li><li>LTR involves training machine learning models to predict document relevance based on various features.</li><li>Relevance Feedback uses feedback from users or other sources to iteratively refine document rankings.</li><li>Contextual Reranking considers contextual information, such as user context or dialogue history, to improve document relevance.</li><li>Hybrid approaches combining multiple reranking techniques or integrating reranking with other retrieval stages can lead to improved performance.</li></ul><h3>Conclusion</h3><p>In the dynamic landscape of LLMs, RAG emerges as a versatile and cost-effective solution, harnessing external data sources to adopt chat-bots powered by LLMs with up-to-date and domain-specific knowledge.</p><p>Through strategic infrastructure setup and utilisation of advanced reranking techniques, organisations can unlock the full potential of RAG, driving efficiency and accuracy in various applications of LLM technology.</p><h4>Additional reference:</h4><ul><li><a href="https://aws.amazon.com/blogs/machine-learning/boosting-rag-based-intelligent-document-assistants-using-entity-extraction-sql-querying-and-agents-with-amazon-bedrock/">Boosting RAG-based intelligent document assistants using entity extraction, SQL querying, and agents with Amazon Bedrock | Amazon Web Services</a></li><li><a href="https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/">Build scalable and serverless RAG workflows with a vector engine for Amazon OpenSearch Serverless and Amazon Bedrock Claude models | Amazon Web Services</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0793dee17ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/customising-llms-for-domain-data-using-rag-d0793dee17ec">Customising LLMs for domain data using RAG</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convex and Non-Convex Optimisation]]></title>
            <link>https://medium.com/data-science/convex-and-non-convex-optimisation-899174802b60?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/899174802b60</guid>
            <category><![CDATA[optimisation]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[price-optimization]]></category>
            <category><![CDATA[convex-optimization]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 16:34:49 GMT</pubDate>
            <atom:updated>2021-01-31T13:52:45.487Z</atom:updated>
            <content:encoded><![CDATA[<h3>Price Optimisation with convex and non-convex loss functions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*_dcpNOWN2bqPmPrM.png" /><figcaption>Typical Loss Surface for a ML Problem (<a href="https://www.cs.umd.edu/~tomg/projects/landscapes/%20and%20arXiv:1712.09913">source</a>)</figcaption></figure><p>Optimisation is used for a number of day-to-day activities from finding the quickest route to our destination using google maps to ordering food through an online app. In this post, we will go through price optimisation for two main types of loss functions i.e. convex and non-convex. We will also cover the best ways to solve these problems using a python library called scipy. The details of the example data and code used in this blog can be found in this <a href="https://gitlab.com/praj88/optimisation">notebook</a>.</p><h3>Convex Optimisation</h3><p>A <em>convex optimisation problem</em> is a problem where all of the constraints are convex functions, and the objective is a convex function if minimising, or a concave function if maximising. A convex function can be described as a smooth surface with a single global minimum. Example of a convex function is as below:</p><pre><em>F(x,y) = x2 + xy + y2.</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/0*qssQvWJrygkVo8tw.png" /><figcaption>Convex Function (<a href="https://en.wikipedia.org/wiki/Convex_function#/media/File:Grafico_3d_x2+xy+y2.png">source</a>)</figcaption></figure><h3>1. Data Generation</h3><p>In order to look at how optimisation works, I am generating a toy data set, which consists of sale’s price, volume and cost. Let’s set the objective of the data set is to find a price that would maximise the total profit.</p><p><strong>Sale’s Price (price): </strong>This is a random number generated between 1000 and 7000. The data is generated using numpy random function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*QVl7VafthtbdYkKDc7m4OQ.png" /><figcaption>Histogram of Price</figcaption></figure><p><strong>Volume: </strong>volume is generated as a straight line function of the sale’s price with a declining slope as shown below:</p><pre><em>f(volume) = (-price/2)+4000</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*M3E34I7o_H-RVdgyvGpnTg.png" /><figcaption>Histogram of volume</figcaption></figure><p>The relationship between sale’s price and volume is as shown below. As expected we see an increase in volume of products sold as the price decreases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*9-IjERL8jW1sCP-s-VUVxQ.png" /></figure><p><strong>Cost: </strong>The cost variable represent the cost of manufacturing of the product. This takes into account some fixed costs (i.e. 300) and a variable costs which is a function of volume (i.e. 0.25).</p><pre><em> f(cost) = 300+volume*0.25</em></pre><p>Now that we have all the variables that we need, we can use the simple formula below to calculate profit:</p><pre><em>f(profit) = (price — cost)*volume</em></pre><p>A 3-D plot of the sale’s price, volume and profit is as shown below. We can see that it is a bell shaped curve with a peak profit at a particular volume and cost. A objective for a business may be to maximise profit and in the following section’s I will show how we can achieve this using scipy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*mYIz-DiZPwmPCBI29wD5lA.png" /><figcaption>3d plot: Price (x), Volume (y) and profit(z)</figcaption></figure><h4><strong>2.Estimate cost and volume from observational data</strong></h4><p>In the above section we have generated a sample data set. However in the real world we may have access to a small set of observations i.e. price, volume and costs for a product. Additionally we will not know the underlying function which governs the relationship between the dependent and independent variable.</p><p>In order to simulate observational data let’s take a cut of the generated data i.e. all points where the price is “&lt;3000”. With this observational data we need to find the relationship between price vs cost and volume vs cost. We can do that by performing a simple linear regression on the observation data set. However for real world problem this may involve building complex non-linear models with a large number of independent variables.</p><p><strong>Linear Regression:</strong></p><ol><li><strong>Price vs Volume</strong></li></ol><p>We take price as an independent variable ‘X’ and volume as a dependent variable ‘y’. Using the stats model library in python, we can determine a linear regression model to capture the relationship between volume and price.</p><p>As seen below from the observational data we are achieving a R-squared of 1 (i.e. 100%), which means the change in volume is perfectly explained by the change in price for the product. However this perfect relationship will rarely be observable in real-life data sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Ix0ChPKdxY-egtNs_w3TxQ.png" /></figure><p>Additionally we can see the bias of -3,999.67 and the coefficient of -0.5 for prices approximates the function used to generate the volume data.</p><p><strong>2. Volume vs Cost</strong></p><p>In a similar manner a linear regression with cost as the dependent variable ‘y’ and volume as the independent variable ‘X’ is performed below. We can see that the bias and co-efficient for the cost model approximates the function used to generate the cost data. Hence we can now use these trained models to determine cost and volume for the convex optimisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*kk5EnZjPeUYg1XGmkqWNsw.png" /><figcaption>Results of Linear regression model —Cost vs Volume .</figcaption></figure><h4>Optimisation using scipy</h4><p>We will use the scipy optimise library for the optimisation. The library has a minimise function, which takes in the below parameters, additional details on the scipy minimise function can be found <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize">here</a>:</p><pre>result = optimize.minimize(fun=objective, <br>                           x0=initial,<br>                           method=’SLSQP’,<br>                           constraints=cons,<br>                           options={‘maxiter’:100, ‘disp’: True})</pre><ol><li><strong>fun:</strong>The objective function to be minimised. In our case we want to maximise the profit. Hence that would mean minimisation of ‘-profit’, where profit is calculate as: <em>profit = (price — cost)*volume</em></li></ol><pre>def cal_profit(price):<br> volume = mod_vol.predict([1, price]) <br> cost = mod_cost.predict([1, volume])<br> profit = (price — cost)*volume<br> return profit</pre><pre>###### The main objective function for the minimisation.<br>def objective(price):<br> return -cal_profit(price)</pre><p><strong>2. x0: </strong>Initial guess i.e in this case we are setting an initial price of 2,000.</p><pre>initial = [2000]</pre><p><strong>3. bounds: </strong>Bounds are the lower and upper limit intervals to be used in the optimisation. It can be set only for for certain optimisation methods which supports bounded inputs such as L-BFGS-B, TNC, SLSQP, Powell, and trust-constr methods.</p><pre>#################### bounds ####################<br>bounds=([1000,10000])</pre><p><strong>4. constraints: </strong>Here we are specifying constraints to the optimisation problem. The constraints in this case is the volume of products that can be produced (i.e. setting this to 3,000 units). It is an inequality constraint i.e. the volume has to be less than the specified number of units. Other types of constrains such as equality constraints are also available for use.</p><pre># Left-sided inequality from the first constraint<br>def constraint1(price):<br>    return 3000-mod_vol.predict([1, price])</pre><pre># Construct dictionaries<br>con1 = {&#39;type&#39;:&#39;ineq&#39;,&#39;fun&#39;:constraint1}</pre><pre># Put those dictionaries into a tuple<br>cons = (con1)</pre><p><strong>5</strong>. <strong>methods</strong>: There a number of methods available for performing the minimisation operation e.g. (see <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize">reference</a>):</p><ul><li>‘Nelder-Mead’ <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html#optimize-minimize-neldermead">(see here)</a></li><li>‘BFGS’ <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html#optimize-minimize-bfgs">(see here)</a></li><li>‘Newton-CG’ <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-newtoncg.html#optimize-minimize-newtoncg">(see here)</a></li><li>‘L-BFGS-B’ <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html#optimize-minimize-lbfgsb">(see here)</a></li><li>‘SLSQP’ <a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html#optimize-minimize-slsqp">(see here)</a></li><li>‘trust-constr’<a href="https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html#optimize-minimize-trustconstr">(see here)</a></li></ul><p>We will use the SLSQP method for optimisation. <strong>Sequential quadratic programming</strong> (<strong>SQP or SLSQP</strong>) is an iterative method for constrained nonlinear optimisation.</p><p>Not all of the above methods support the use of both bounds and constraints. Hence we are using SLSQP, which supports the use of both the bounds and constraints for optimisation.</p><h4>Results of convex optimisation</h4><p>The optimisation was run for a total of 3 iterations. Based on this the objective function has returned a value of -6.6 mil (i.e. -6,587,215.16). This results in a price of 4,577 and volume of 1,710. We are getting a negative value because we are maximising for profit by minimising for negative profit, which is a convex function.</p><pre>Optimization terminated successfully.    (Exit mode 0)<br>            Current function value: -6587215.163007045<br>            Iterations: 3<br>            Function evaluations: 9<br>            Gradient evaluations: 3</pre><p>The results of the optimisation closely matches to the maximum profit as seen in the generated data set. Hence we know that the optimisation has worked as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WN24U7cXDpObsESPAICqlQ.png" /><figcaption>Max profit values: price (x), volume (y) and profit(z)</figcaption></figure><h3>Non Convex Optimisation</h3><p>In most of the machine learning problems we come across loss surfaces which are non-convex in nature. Hence they will have multiple local minimum. Hence while solving for such non-convex problems it may be simpler and less computationally intensive if we “convexify” a problem<em> (make it convex optimisation friendly)</em>, then to use non-convex optimisation. I will demonstrate how we can achieve this through the below example.</p><h4>Non convex data set genration:</h4><p>Here we have created another product with a slightly different price, volume and profit profile. Let’s say that the company can produce only one of these products and wants to use optimisation to find out which of the products it needs to focus on and what price it needs to sell it at.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*MfIHSa_Fsr3a3jB7VcseBA.png" /><figcaption>3d plot: Non-convex data set with product 1 and product 2</figcaption></figure><h4>Use of Convex minimisation for non-convex data</h4><p>For the above data if we use the same convex optimisation as above, the solution we get will be a local minimum as seen below. We get a max profit of <strong>6.86 mil </strong>for a optimised price of <strong>5, 018</strong> and product type<strong> 0</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*06jt_EbEfY0S6E1V_hiSFw.png" /><figcaption>Local minimum from convex optimisation</figcaption></figure><h4>Basinhopping algorithm</h4><p>Basin-hopping is an algorithm that combines a global stepping algorithm along with a local minimisation at each step. Hence this can be used to seek the best of all the local minimum options available for the non-convex loss surface. The below code demonstrates the implementation of this algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*zArgXlhc2mKYhquuZBgSTA.png" /><figcaption>Global minimum from basin-hopping algorithm</figcaption></figure><p>As seen above we have use the same method i.e. SLSQP and the same bounds and constraints as the previous example. However, we now get a slightly different result. The maximum profit is <strong>9.8 mil</strong> with an optimised price of <strong>5,957</strong> and product type of <strong>1</strong>.</p><h4>Conclusion:</h4><p>In summary, we have reviewed the use of optimisation libraries in scipy for a simple convex optimisation data-set which has a single global minimum and also reviewed the use of basin-hopping algorithm in cases where the loss surface has multiple local minimum and a global minimum. Hope you find the above post useful and the framework provided to solve optimisation problems in the real world.</p><h4>References:</h4><ul><li><a href="https://en.wikipedia.org/wiki/Sequential_quadratic_programming">Sequential quadratic programming</a></li><li><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.basinhopping.html">scipy.optimize.basinhopping - SciPy v1.5.2 Reference Guide</a></li><li><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize">scipy.optimize.minimize - SciPy v1.5.2 Reference Guide</a></li><li><a href="https://stats.stackexchange.com/questions/324981/why-study-convex-optimization-for-theoretical-machine-learning">Why study convex optimization for theoretical machine learning?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=899174802b60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/convex-and-non-convex-optimisation-899174802b60">Convex and Non-Convex Optimisation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tweet analytics using NLP]]></title>
            <link>https://medium.com/analytics-vidhya/tweet-analytics-using-nlp-f83b9f7f7349?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/f83b9f7f7349</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[twitter-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Tue, 23 Jul 2019 05:18:30 GMT</pubDate>
            <atom:updated>2020-06-06T16:59:22.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNxlI4puLTObXNbizjQrnQ.jpeg" /></figure><p><strong>Introduction</strong></p><p>In recent days with the explosion of Big Data there is a large demand for organisations and data scientists to perform information extraction using non-traditional sources of data. Research has shown that nearly 80% of data exists as unstructured text data, hence text analytics is fundamental in order to analyse the wealth of information available on chat transcripts, social media posts, reviews, news feeds etc.</p><p><strong>What is Text Analytics?</strong></p><p>Text analytics is the processes of synthesising unstructured data to help discover patterns and enable decision making. Until recent years text analytics had to be performed the old fashioned way i.e. eyeballing and manual categorisation of text, which is inefficient and time consuming. Also this is not a practice solution when dealing with millions of documents such as twitter data.</p><p>Twitter data (also know as tweets) is a rich source of information on a large set of topics. This data can be used to find trends related to a specific keyword, measure brand sentiment or gather feedback about new products and services. This post will provide a step by step guide for text analytics on twitter data.</p><p><strong>How to perform text analytics on twitter data?</strong></p><p>The steps involved in text analytics are:</p><p>Step 1: Collect tweets</p><p>Step 2: Pre-process tweets</p><p>Step 3: Apply sentiment analysis</p><p>Step 4: Apply named entity recognition</p><p>Step 5: Cluster tweets</p><p>Step 6: Visualise analysis</p><p>I have attached a jupyter notebook which performs the above steps on a sample of twitter data <a href="https://gitlab.com/praj88/twitter-analytics/blob/master/scripts/twitter-analytics.ipynb">here</a>.</p><p><strong>Step 1: Collect tweets</strong></p><p>The tweet data can be programmatically accessed in two ways i.e. Twitter Rest API or Streaming API.</p><p>The Rest API enables you to collect the list of all tweets or followers for a particular user. You can also query a user’s account and even modify them provided you have the required permissions. Where as the Streaming API allows you to collect tweets on a real-time basis based on search terms, user ids or locations.</p><p>In order to access the API, you will need 4 pieces of information from Twitter i.e. API key, API secret, Access token and Access token secret. The detailed steps for setting up an account and connecting to the twitter API services using python packages are covered in the blog <a href="http://adilmoujahid.com/posts/2014/07/twitter-analytics/">here</a>.</p><p>For our analysis I have used the twitter streaming API service and collected tweets for twenty FTSE 100 companies over a week starting from 1st October 2016. This has resulted in a total of 37,313 tweets. The output of the streaming API needs to be parsed into a suitable format before any analysis. In the next section we cover the steps involved in pre-processing of tweets.</p><p><strong>Step 2: Pre-Process tweets</strong></p><p>Here we parse the response from the twitter API into a structured table. The response from twitter streaming API’s is in the below format:</p><p>The detailed code for parsing the output from the twitter API is below. The output is structured into key fields such as “doc_id”, “username”, “text” etc. The parsed response can be stored in a database or a JSON file.</p><pre>import simplejson as json<br><br>def get_tweet(doc):<br>  tweetItem = {}<br><br>#Available tweet data<br>  tweetItem[&#39;text&#39;] = doc[&#39;text&#39;]<br> <br>#hashtags are identified and provided as a field in the tweet<br>  tweetItem[&#39;hashtags&#39;] = map(lambda x: x[&#39;text&#39;],doc[&#39;entities&#39;][&#39;hashtags&#39;])<br><br>  # user_mentiones are identified and provided as a field <br>  tweetItem[&#39;user_mentions&#39;] = map(lambda x: x[&#39;screen_name&#39;],doc[&#39;entities&#39;]<br>  [&#39;user_mentions&#39;])</pre><pre>  # symbols e.g. $APPL are identified and provided as a field <br>  tweetItem[&#39;symbols&#39;] = map(lambda x: x[&#39;text&#39;],doc[&#39;entities&#39;][&#39;symbols&#39;])<br>  tweetItem[&#39;coordinates&#39;] = doc[&#39;coordinates&#39;]<br>  tweetItem[&#39;user_id&#39;] = doc[&#39;user&#39;][&#39;id&#39;]<br>  tweetItem[&#39;user_name&#39;] = doc[&#39;user&#39;][&#39;name&#39;]</pre><pre>  try:</pre><pre>      tweetItem[&#39;retweet_id&#39;] = doc[&#39;retweeted_status&#39;][&#39;id&#39;]<br> <br>  except KeyError as e:<br>      tweetItem[&#39;retweet_id&#39;] = 0<br>      pass<br><br>return tweetItem</pre><p><strong>Step 3: Sentiment Analysis</strong></p><p>People express their opinions via tweets and these usually have sentiment associated with it i.e. positive, negative or neutral. Analysis of this sentiment may lead to some useful insight on the topic or company being discussed. Hence we suggest use of sentiment analysis algorithms to perform this analysis. I have created a separate blog with details on performing sentiment analytics using the deep learning algorithm <a href="https://towardsdatascience.com/sentiment-analysis-for-text-with-deep-learning-2f0a0c6472b5">here</a>.</p><p>The sentiment is a useful indicator, however tweets contain additional information such as names of people, places and organisations. The data when extracted may lead to very useful insights on your product or company. So in the next step we cover the usage of named entity recognition algorithms, which is designed to extract this information.</p><p><strong>Step 4: Named Entity Recognition</strong></p><p>The Named Entity Recognition algorithm will identify key tags such as persons, locations or organisations contained within the tweets. For example, if there’s a mention of “London” in your tweet the algorithm would tag that as “Location”. However as tweets generally do not follow the standard english syntax, the accuracy of the tags is around 60% to 70%.</p><p>You can use the Stanford NER tagger to obtain the entity recognition for the tweets. In the example below I have used a tweet by Donald Trump.</p><pre># -*- coding: utf-8 -*-<br><br>from nltk.tag import StanfordNERTagger<br>from nltk.tokenize import word_tokenize<br><br>st = StanfordNERTagger(&#39;/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz&#39;,<br>					   &#39;/usr/share/stanford-ner/stanford-ner.jar&#39;,<br>					   encoding=&#39;utf-8&#39;)</pre><pre>#a tweet by Donald Trump<br>text = &#39;Just had a very good call with <a href="https://twitter.com/SwedishPM">@SwedishPM</a> Stefan Löfven who assured me that American citizen A$AP Rocky will be treated fairly. Likewise, I assured him that A$AP was not a flight risk and offered to personally vouch for his bail, or an alternative....&#39;<br><br>tokenized_text = word_tokenize(text)<br>classified_text = st.tag(tokenized_text)<br><br>print(classified_text)</pre><p><strong>Step 5: Clustering of similar tweets</strong></p><p>A number of tweets are very similar to each other, this may be due to people tweeting about the same topic this causes a lot of noise in the data. Hence in order to reduce the noise it would be useful to cluster the tweets into groups based on their similarity. This can be done using the clustering algorithm.</p><p>The inner workings of the algorithms is based on a machine learning concept know as tf-idf (term frequency — Inverse document frequency) matrix. Here the each tweet is treated as a document (d) and the words within the tweet as term (t).</p><p><strong>Term Frequency-Inverse Document Frequency</strong> is a numerical statistic that demonstrates how important a word is to a corpus.</p><p>Term Frequency is just ratio number of current word to the number of all words in document/string/etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/0*ee9dsN0Ikq_H0_K3.png" /><figcaption>Term Frequency Formula</figcaption></figure><p>Frequency of term t_i, where n_t — the number of t_i in current document/string, the sum of n_k is the number of all terms in current document/string.</p><p>Inverse Document Frequency is a log of the ratio of the number of all documents/string in the corpus to the number of documents with term t_i.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/0*jPYKW4pd0GBnnwzw.png" /><figcaption>Inverse Document Frequency Formula</figcaption></figure><p>tf-idf(t, d, D) is the product tf(t, d) to idf(t, D). Additional details on tf-idf is available in the blog <a href="https://medium.com/@MSalnikov/text-clustering-with-k-means-and-tf-idf-f099bcf95183">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/0*MBR9xXJf4mgehHGV.png" /><figcaption>TF-IDF Formula</figcaption></figure><p>In the code below in order to obtain the clustering, apart from the text input, the only additional parameter that needs to be specified in the input is the similarity threshold. You can set the threshold in the function “<em>get_clusters(processedTweets, similarity_threshold)”</em>. Hence all tweets, which are 80% similar will be clustered together based on similarity measure known as cosine similarity.</p><pre>def cluster_text(dist_text,processedTweets,dist_threshold_txt):<br>    cluster_id = 1<br>    for i in range(dist_text.shape[0]):<br>        try:<br>            doc_dist_array_txt = dist_text[i,]<br>            # identify document ids where the tweets are similar<br>            similarDocs = np.where(doc_dist_array_txt &lt;= dist_threshold_txt)[0]<br>            processedTweets.ix[processedTweets.index.isin(similarDocs) &amp; processedTweets[&#39;cluster_ref&#39;].isin([None]), &#39;cluster_ref&#39;] = cluster_id<br>            cluster_id = cluster_id + 1<br>        except ValueError:<br>            continue<br><br>    return processedTweets<br><br><br><br># %%%%%%%%% Calculate Cosine distance of documents %%%%%%%%%%%%%%<br><br>def tokenize_only(text):<br>    # first tokenize by sentence, then by word to ensure that punctuation is caught as it&#39;s own token<br>    tokens = [word.lower() for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent)]<br>    filtered_tokens = []<br>    # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)<br>    for token in tokens:<br>        if re.search(&#39;[a-zA-Z]&#39;, token):<br>            filtered_tokens.append(token)<br>    return filtered_tokens<br><br>def get_clusters(processedTweets, similarity_threshold):<br>       tweets = processedTweets[&#39;text&#39;].tolist()<br><br>       #Define count vectorizer parameters<br>       vectorizer =  CountVectorizer(max_df=1.0, min_df=1, stop_words=stopwords, tokenizer=tokenize_only) <br><br>       # Get document term frequency matix<br>       dtf_matrix = vectorizer.fit_transform(tweets) #fit the vectorizer to tweets which does not contain&#39;htttp&#39; and &#39;RT&#39;<br>       dist_text = np.around(abs(1 - cosine_similarity(dtf_matrix)),2)<br><br># --------1D clustering based on distance measure<br>       # Pre clustering setup<br>       processedTweets[&#39;cluster_ref&#39;] = None # Tweets that are 1-similarithy% similar, as defined by dist_threshold<br>       # Clustering of tweets based on text<br>       processedTweets = cluster_text(dist_text,processedTweets,dist_threshold_txt = (1-similarity_threshold))<br><br>       return processedTweets</pre><p>The results will contain an additional column “cluster_ref”. The cluster_ref will group all the similar documents into a single group for example if document 1 and document 2 are similar, hence both have cluster_ref of 1.</p><p><strong>Step 6: Visualisation of results</strong></p><p>The visualisation of the above results would help drive insights into the consumer opinion about the product or company. In the below section I will be using Tableau for visualisation purposes. Please see the link to the dashboard <a href="https://public.tableau.com/profile/ample.ai#!/vizhome/TwitterAnalytics_4/WordCloud">here</a>.</p><p>The first graph shown below is the positive, negative and average sentiment (Blue line) of each company. The bars represent the average sentiment for the company and is not weighted for the number of tweets. This <strong>unweighted</strong> sentiment does not provide an accurate comparison between companies as it does not take into account the number of tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qwk-li6XkWzmbSVDASiHGA.png" /><figcaption>Sentiment Unweighted</figcaption></figure><p>In order to over come the limitations seen above, we have <strong>weighted</strong> the sentiment by the number of tweets to calculate a weighted sentiment. This provides a more accurate measure of sentiment. For instance we can see in the first graph that Tesco has almost the same average positive and negative sentiment. However when scaled for the number of tweets we can see that the positive sentiment is much greater than the negative sentiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ChEvungwHyqqeTUb3dLiQ.png" /><figcaption>Sentiment Weighted</figcaption></figure><p>It is useful to see the proportion of positive, negative and neutral tweets for each company as shown below. This can lead to some interesting insights about the brand health of a company. For example we can see that for Schroders despite having a low weighted sentiment it has a very high proportion of positive tweets. This analysis when conducted over time can serve as an indicator of brand health.</p><p>The named entities which we have extracted using our named entity recognition algorithm can be visualised using a word cloud. In the below section I have created a word cloud in order to help with the analysis. The words are coloured by the companies and sized by the number of times it is repeated in the analysis. As default the word cloud is representing the entities which have been repeated at least 10 times. However this can be changed by moving the slider control at the bottom of the dashboard. The word cloud provide an easy way of understanding the most prominent topics being discussed in the tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*QxaMT1CfFRyswA3O813N-Q.png" /></figure><p>We can use the combined dashboard seen below to explore the topics driving the sentiment. We can see from instance from the word cloud that the Phrase “AsdaSainsburyBritish” is highlighted as the most prominent phrase. When we click on it we can see that it is due to tweets about a superbug found in the meat sold at these retailers. This can also be used to see what is driving the large negative positive sentiment in specific companies. For example by clicking on the red bar on the first chart for AstraZeneca and selecting the phrase from the word cloud we can see that the driving factor for negative sentiment here is a failed drug trial for Brilinta.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ek1EYp2H8fUx25kjwbbO5g.png" /></figure><p><strong>Conclusion:</strong></p><p>In conclusion the above techniques can be used to analyse huge volumes of twitter data. Additionally the algorithms can be applied to any kind of text data such as news articles or customer service chat data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f83b9f7f7349" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/tweet-analytics-using-nlp-f83b9f7f7349">Tweet analytics using NLP</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy ML models at Scale]]></title>
            <link>https://medium.com/data-science/deploy-ml-models-at-scale-151204549f41?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/151204549f41</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Mon, 27 May 2019 07:26:29 GMT</pubDate>
            <atom:updated>2019-05-27T12:46:59.820Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploy ML models at scale</h3><h4>Part 1: API service for ML models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gYiavISjtd-ihsAF.png" /><figcaption>ML Model Deployment (<a href="https://www.toptal.com/python/python-machine-learning-flask-example">Source</a>)</figcaption></figure><p>Let’s assume that you have built a ML model and that you are happy with its performance. Then the next step is to deploy the model into production. In this blog series I will cover how you can deploy your model for large scale consumption with in a scalable Infrastructure using AWS using docker container service.</p><p>In this blog I will start with the first step of building an API framework for the ML model and running it in you local machine. For the purpose of this blog, let’s consider the Sentiment classification model built <a href="https://towardsdatascience.com/sentiment-analysis-for-text-with-deep-learning-2f0a0c6472b5">here</a>. In order to deploy this model we will follow the below steps:</p><ol><li>Convert the model into <strong><em>.hdf5 </em></strong>file or <strong><em>.pkl</em></strong> file</li><li>Implement a Flask API</li><li>Run the API</li></ol><h4>Convert the model into “<strong><em>.hdf5” </em></strong>file or ‘.pkl’ file</h4><p>In case the model is a built on sklearn, it would be best to save it as a ‘.pkl’ file. Alternatively if it is a deep learning model then it is recommended to save the model as a ‘HDF’ file. The main difference between ‘.pkl’ and ‘.hdf’ is, pickle requires a large amount of memory to save a data structure to disk, where as HDF is designed to efficiently store large data sets.</p><p><strong>Save model in Pickle(.pkl):</strong></p><pre>from sklearn.externals import joblib  <em># Save to file</em></pre><pre># Fit the model (example of a model in sklearn)<br>model = LogisticRegression()<br>model.fit(X_train, Y_train)</pre><pre>#<em>working directory<br></em>joblib_file = &quot;best_model.pkl&quot;   <br>joblib.dump(model, joblib_file)</pre><p><strong>Save model in HDF(.hdf5):</strong></p><p>Once you have trained your deep learning model in keras or tensorflow you can save the model architecture and its weights using a .hdf5 file system.</p><pre># save the best model and early stopping</pre><pre>saveBestModel = keras.callbacks.ModelCheckpoint(&#39;/best_model.hdf5&#39;, monitor=&#39;val_acc&#39;, verbose=0, save_best_only=True, save_weights_only=False, mode=&#39;auto&#39;, period=1)</pre><h4>Implement a Flask API</h4><p><strong>Step 1: </strong>Load the saved model (as per previous section) by one of the following methods depending on the type of file i.e. ‘hdf5’ or ‘pkl’.</p><p><strong>HDF5:</strong></p><pre>from keras.models import load_model<br>import h5py<br>prd_model = load_model(&#39;best_model.hdf5&#39;)</pre><p><strong>PKL:</strong></p><pre>from sklearn.externals import joblib<br>loaded_model = joblib.load(&#39;best_model.pkl&#39;)</pre><p><strong>Step 2:</strong> Import flask and create a flask application object as shown below:</p><pre>from flask import Flask, url_for, request</pre><pre>app = Flask(__name__)</pre><p><strong>Step 3: </strong>The next step is to build a test API function which returns the “<em>API working” string. </em>This can be used to ensure the health of the API when it is deployed in production. Here we use ‘@app.route’, which is a python decorator ( <em>a decorator is a function that takes another function and extends the behaviour of the latter function without explicitly modifying it)</em></p><pre><a href="http://twitter.com/app">@app</a>.route(‘/apitest&#39;)<br>def apitest():<br>    return ‘API working’</pre><p><strong>Step 4: </strong>The next step is to build a “POST” request api for processing requests to our sentiment model. We are using the path name “/sentiment”. The function reads the json input and converts it into pandas dataframe. It extracts the relevant fields from the json and calls the “get_sentiment_DL” function for processing. “get_sentiment_DL” function contains the trained model which has been loaded via ‘hdf5’ file. It finally will return back the results of the model in the form of json result.</p><pre># main API code<br><a href="http://twitter.com/app">@app</a>.route(‘/sentiment’, methods=[‘POST’])<br>def sentiment():<br>   if request.method == ‘POST’:<br>   text_data = pd.DataFrame(request.json)<br>   text_out = get_sentiment_DL(prd_model, text_data, word_idx)<br> <br>   text_out = text_out[[‘ref’,’Sentiment_Score’]]</pre><pre>   #Convert df to dict and then to Json<br>   text_out_dict = text_out.to_dict(orient=’records’)<br>   text_out_json = json.dumps(text_out_dict, ensure_ascii=False)</pre><pre>   return text_out_json</pre><p><strong>Step 5</strong>: The detailed model processing steps will be performed by “get_sentiment_DL” function. In the case of our deep learning sentiment model we are passing:</p><ol><li>best_model: the loaded model</li></ol><p>2. text_data: Input text for sentiment classification</p><p>3. word_idx: Word index from the GloVe file (details of the model <a href="https://towardsdatascience.com/sentiment-analysis-for-text-with-deep-learning-2f0a0c6472b5">here</a>).</p><pre>def get_sentiment_DL(best_model, text_data, word_idx):</pre><pre>&#39;&#39;&#39;Modle Processing&#39;&#39;&#39;</pre><pre> return sentiment_score</pre><p><strong>Step 6: </strong>Add the below section to run the app. Here <strong>Host</strong> is set as “0.0.0.0” as we are hosting in our local server. However you can configure it to your network settings. <strong>Debug</strong> can be set to True at the time of building the API functionality. The <strong>Port</strong> is set to 5005, however this can be configured as per your requirement.</p><pre>if __name__ == “__main__”:<br>  app.run(host=”0.0.0.0&quot;, debug=False, port=5005)</pre><h4>Run the API</h4><p>To run the API, open a command line window and go to the directory where the code is stored. Run the python script by running the below command <em>(“sentiment.py” is the name of the file with the above API implementation).</em></p><pre>python sentiment.py</pre><p>On running the above command you will be able to see the below result in your command line window:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*HDRS4dLpC0gUU2imN5cm2g.png" /></figure><p>Once the API is running you can test the API by going to your browser and typing “0.0.0.0:5005/apitest”. You will get the below result in you browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*3RV4pbnY9LNeMcBVwxXLvQ.png" /><figcaption>Browser — API Test</figcaption></figure><p>You can now pass any data to the API using python as shown below. The address in our case is “<a href="https://www.seelabs.app/face_rec_train&#39;">http://0.0.0.0:5005/sentiment</a>”. The results of the model will be returned and stored in “response” field.</p><pre>import requests<br>import json<br>import urllib.parse</pre><pre>data_json = &#39;&#39;&#39;{&quot;ref&quot;:[1], &quot;text&quot;:[&quot;I am well&quot;]}&#39;&#39;&#39;</pre><pre>head= {“Content-type”: “application/json”}</pre><pre>response = requests.post(‘<a href="https://www.seelabs.app/face_rec_train&#39;">http://0.0.0.0:5005/sentiment&#39;</a>, json = data_json, headers = head)</pre><pre>result = response.content.decode(‘utf8’)</pre><h4>Conclusion</h4><p>In conclusion, we have covered steps to deploy the model into an api service in your local computer.</p><p>The next step is to deploy this in a cloud server as a micro service. In following blog’s I will cover the use of container service such as docker and deploy it in AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=151204549f41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deploy-ml-models-at-scale-151204549f41">Deploy ML models at Scale</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Embedding’s for Categorical variables (Cat2Vec)]]></title>
            <link>https://medium.com/data-science/deep-embeddings-for-categorical-variables-cat2vec-b05c8ab63ac0?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/b05c8ab63ac0</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cat2vec]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Tue, 16 Apr 2019 21:28:45 GMT</pubDate>
            <atom:updated>2019-09-22T05:13:15.185Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep embedding’s for categorical variables (Cat2Vec)</h3><h4>Use of deep learning on tabular data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*swQ2gwFGri4u4c6pdN6jDA.png" /></figure><p>In this blog I am going to take you through the steps involved in creating a embedding for categorical variables using a deep learning network on top of keras. The concept was originally introduced by Jeremy Howard in his fastai course. Please see the link for more <a href="https://www.fast.ai/2018/04/29/categorical-embeddings/">details</a>.</p><h4>Traditional Embedding</h4><p>Across most of the data sources that we work with we will come across mainly two types of variables:</p><ol><li><strong>Continuous variables: </strong>These are usually integer or decimal numbers and have infinite number of possible values e.g. Computer memory units i.e 1GB, 2GB etc..</li><li><strong>Categorical variables:</strong> These are discrete variables which is used to split the data based on certain characteristics. e.g. Types of computer memory i.e. RAM memory, internal hard disk, external hard disk etc.</li></ol><p>When we build a ML model more often then not it is required for us to transform the categorical variable before we can use it in the algorithm. The transformation applied has a big impact on the performance of the model, especially if the data has a large number of categorical features with high cardinality. Example’s of some of the usual transformation’s applied include:</p><p><strong>One-Hot encoding:</strong> Here we convert each category value into a new column and assign and assign a 1 or 0(True/False) value to the column.</p><p><strong>Binary encoding</strong>: This creates fewer features than one-hot, while preserving some uniqueness of values in the the column. It can work well with higher dimensional ordinal data.</p><p>These usual transformation’s however do not capture the relationship between the categorical variables. Please see the below link for more information on different type of <a href="https://www.datacamp.com/community/tutorials/categorical-data">encoding methods</a>.</p><h3>Data</h3><p>To demonstrate the application of deep embedding’s let’s take an example of the <a href="https://www.kaggle.com/contactprad/bike-share-daily-data">bicycle sharing</a> data from Kaggle. Also link to the git repo is <a href="https://gitlab.com/praj88/deepembeddings">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*bWw2YKMezrrMj3iyoiuxOg.png" /><figcaption>Data — Bike Sharing</figcaption></figure><p>As we can see there are a number of columns in the data set. In order to demonstrate this concept we shall use just the <strong>date_dt, cnt </strong>and<strong> mnth </strong>columns from the<strong> </strong>data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/1*__AiZezZ2WUSQdMzGvwTrg.png" /><figcaption>Selected Columns</figcaption></figure><p>Traditional <strong>one-hot encoding </strong>would result in 12 columns, one of each month. However in this type of embedding equal importance is given to each day of the week and there is no relationship between each of the months.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*NgtEUGgyXeKbR4XBG_-d7w.png" /><figcaption>One hot encoding of the mnth column</figcaption></figure><p>We can see a seasonal pattern’s of each of the months in the below graph. As we can see months 4 to 9 are the peak months. Months 0, 1, 10,11 are months of low demand for bike hire.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*Iv6J51_P8uLOqf6PoxmP8A.png" /><figcaption>Monthly Seasonality</figcaption></figure><p>Additionally when we plot the daily usage for each month, represented by a different colour, we cam see some weekly patterns within each month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*u2phW5bA9vG4h0ht8Qn4pA.png" /><figcaption>Daily usage trends</figcaption></figure><p>Ideally we would expect such relationships to be captured by use of embeddings. In the next section we will examine the generation of these embeddings using a deep network built on top of keras.</p><h4><strong>Deep Encoding</strong></h4><p>The code is as shown below. We will build a perceptron network with dense layer network and a ‘relu’ activation function.</p><p>The input for the network i.e. ‘<strong>x</strong>’ variable is the month number. This is a numeric representation of each of the months in the year and ranges from 0 to 11. Hence the input_dim is set to 12.</p><p>The output for the network i.e. ‘<strong>y</strong>’ is a scaled column of ‘cnt’. However ‘y’ can be increased to include other continuous variables. Here as we are using a single continuous variable we will set the last number of the output dense layer to 1. We will train the model for 50 iterations or epochs.</p><pre>embedding_size = 3</pre><pre>model = models.Sequential()<br>model.add(Embedding(input_dim = 12, output_dim = embedding_size, input_length = 1, name=&quot;embedding&quot;))<br>model.add(Flatten())<br>model.add(Dense(50, activation=&quot;relu&quot;))<br>model.add(Dense(15, activation=&quot;relu&quot;))<br>model.add(Dense(1))<br>model.compile(loss = &quot;mse&quot;, optimizer = &quot;adam&quot;, metrics=[&quot;accuracy&quot;])<br>model.fit(x = data_small_df[&#39;mnth&#39;].as_matrix(), y=data_small_df[&#39;cnt_Scaled&#39;].as_matrix() , epochs = 50, batch_size = 4)</pre><h4>Network Parameters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*png48G-xkG5UZPcgArUVyA.png" /><figcaption>Model Summary</figcaption></figure><p><strong>Embedding Layer:</strong> Here we specify the embedding size for our categorical variable. I have used <strong>3 </strong>in this case<strong>, </strong>if we were to increase this it will capture more details on the relationship between the categorical variables. Jeremy Howard suggests the following solution for choosing embedding sizes:</p><pre># m is the no of categories per feature</pre><pre>embedding_size = min(50, m+1/ 2)</pre><p>We are using an “adam” optimiser with a mean-square error loss function. Adam is preferred to sgd (stochastic gradient descent) as it is much faster optimiser due to its adaptive learning rate. You can find more details on the different types of optimisers <a href="https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/">here</a>.</p><h4><strong>Results</strong></h4><p>The final resulting embedding for each of the months are as follows. Here ‘0’ is for January and ‘11’ for December.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*n8pWCtwF87nco7joKKk-0w.png" /><figcaption>Embedding Table</figcaption></figure><p>When we visualise this using a 3D plot, we can see a clear relationship between the months. The months with similar ‘cnt’ are grouped closer together e.g. months 4 to 9 are very similar to each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*iclS8GD-jKHbr2BRQ2fxmA.png" /><figcaption>Embeddings — Months of the year</figcaption></figure><h4><strong>Conclusion</strong></h4><p>In conclusion we have seen that by using Cat2Vec (categorical variable to vectors) we can represent high cardinality categorical variable using low dimension embedding while preserving the relationship between each of the categories.</p><p>In the next few blogs we will explore on how we can use these embeddings to build supervised and unsupervised machine learning models with better performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b05c8ab63ac0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deep-embeddings-for-categorical-variables-cat2vec-b05c8ab63ac0">Deep Embedding’s for Categorical variables (Cat2Vec)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment analysis for text with Deep Learning]]></title>
            <link>https://medium.com/analytics-vidhya/sentiment-analysis-for-text-with-deep-learning-2f0a0c6472b5?source=rss-daf1da2a65fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0a0c6472b5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Prajwal Shreyas]]></dc:creator>
            <pubDate>Tue, 02 Apr 2019 05:50:59 GMT</pubDate>
            <atom:updated>2021-04-28T17:20:14.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*33q1bTIm6jOzZXf_.png" /></figure><p>I started working on a NLP related project with twitter data and one of the project goals included sentiment classification for each tweet. However when I explored the available resources such as NLTK sentiment classifier and other resource available in python, I was disappointed by the performance of these models. At most I would get about 60% to 70% accuracy on binary classification (i.e only positive or negative class) tasks.</p><p>Hence I started researching about ways to increase my model performance. One of the obvious choices was to build a deep learning based sentiment classification model.</p><p>I am writing this blog post to share about my experience about steps to building a deep learning model for sentiment classification and I hope you find it useful. The link to the code repository can be found <a href="https://gitlab.com/praj88/deepsentiment">here</a>.</p><p>I have designed the model to provide a sentiment score between 0 to 1 with 0 being very negative and 1 being very positive. This was done by building a multi-class classification model i.e 10 class, one class for each decile.</p><p>There are <strong>5 major steps</strong> involved in the building a deep learning model for sentiment classification:</p><p><strong>Step1:</strong> Get data.</p><p><strong>Step 2: </strong>Generate embeddings</p><p><strong>Step 3</strong>: Model architecture</p><p><strong>Step 4</strong>: Model Parameters</p><p><strong>Step 5:</strong> Train and test the model</p><p><strong>Step 6:</strong> Run the model</p><p>I am going to cover each of the above steps in detail below.</p><h4>Step1: Get data</h4><p>Sourcing the labelled data for training a deep learning model is one of the most difficult parts of building a model. Fortunately we can use the <a href="https://nlp.stanford.edu/sentiment/code.html">Stanford sentiment treebank data</a> for our purpose.</p><p>The data set “dictionary.txt” consists of 239,233 lines of sentences with an index for each line. The index is used to match each of the sentences to a sentiment score in the file “labels.txt”. The score ranges from 0 to 1, 0 being very negative and 1 being very positive.</p><p>The below code reads the dictionary.txt and labels.txt files, combines the score to each sentences . This code is found within <strong><em>train/utility_function.py</em></strong></p><pre>def read_data(path):</pre><pre># read dictionary into df</pre><pre>df_data_sentence = pd.read_table(path + ‘dictionary.txt’)</pre><pre>df_data_sentence_processed = df_data_sentence[‘Phrase|Index’].str.split(‘|’, expand=True)</pre><pre>df_data_sentence_processed = df_data_sentence_processed.rename(columns={0: ‘Phrase’, 1: ‘phrase_ids’})</pre><pre># read sentiment labels into df</pre><pre>df_data_sentiment = pd.read_table(path + ‘sentiment_labels.txt’)</pre><pre>df_data_sentiment_processed = df_data_sentiment[‘phrase ids|sentiment values’].str.split(‘|’, expand=True)</pre><pre>df_data_sentiment_processed = df_data_sentiment_processed.rename(columns={0: ‘phrase_ids’, 1: ‘sentiment_values’})</pre><pre>#combine data frames containing sentence and sentiment</pre><pre>df_processed_all = df_data_sentence_processed.merge(df_data_sentiment_processed, how=’inner’, on=’phrase_ids’</pre><pre>return df_processed_all</pre><p>The data is split into 3 parts:</p><ul><li>train.csv : This is the main data which is used to train the model. This is 50% of the overall data.</li><li>val.csv : This is a validation data set to be used to ensure the model does not overfit. This is 25% of the overall data.</li><li>test.csv : This is used to test the accuracy of the model post training. This is 25% of the overall data.</li></ul><h4>Step 2: Generate embeddings</h4><p>Prior to training this model we are going to convert each of the words into a word embedding. You can think of word embeddings as numerical representation of words to enable our model to learn. For more details on word embeddings please read this <a href="https://www.datascience.com/resources/notebooks/word-embeddings-in-python">blog</a>.</p><p><em>What are word embeddings?</em></p><p>They are vector representations that capture the context of the underlying words in relation to other words in the sentence. This transformation results in words having similar meaning being clustered closer together in the hyperplane and distinct words positioned further away in the hyperplane.</p><p><em>How are we going to convert each word into a word embeddings?</em></p><p>We are going to use a pre-trained word embedding model know as <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a>. For our model we are going to represent each word using a 100 dimension embedding. The detailed code for converting the data into word embedding is in within <strong><em>train/utility_function.py. </em></strong>This function basically replace each of the words by its respective embedding by performing a lookup from the GloVe pre-trained vectors. An illustration of the process is shown below, where each word is converted into an embedding and fed into a neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lNsgjlmv5FFVUxm7.png" /><figcaption>Converting Sentences to Embedding for a Neural Network (<a href="https://www.datascience.com/resources/notebooks/word-embeddings-in-python">Source</a>)</figcaption></figure><p>The below code is used to split the data into train, val and test sets. Also the corresponding embeddings for the data is stored in the weight_matrix variable.</p><pre>def load_data_all(data_dir, all_data_path,pred_path, gloveFile, first_run, load_all):</pre><pre>   numClasses = 10</pre><pre># Load embeddings for the filtered glove list<br>    if load_all == True:<br>        weight_matrix, word_idx = uf.load_embeddings(gloveFile)<br>    else:<br>        weight_matrix, word_idx = uf.load_embeddings(filtered_glove_path)</pre><pre>    # create test, validation and trainng data<br>    all_data = uf.read_data(all_data_path)<br>    train_data, test_data, dev_data = uf.training_data_split(all_data, 0.8, data_dir)</pre><pre>train_data = train_data.reset_index()<br>    dev_data = dev_data.reset_index()<br>    test_data = test_data.reset_index()</pre><pre>    maxSeqLength, avg_words, sequence_length = uf.maxSeqLen(all_data)<br>  </pre><pre># load Training data matrix<br>    train_x = uf.tf_data_pipeline_nltk(train_data, word_idx, weight_matrix, maxSeqLength)<br>    test_x = uf.tf_data_pipeline_nltk(test_data, word_idx, weight_matrix, maxSeqLength)<br>    val_x = uf.tf_data_pipeline_nltk(dev_data, word_idx, weight_matrix, maxSeqLength)</pre><pre>    # load labels data matrix<br>    train_y = uf.labels_matrix(train_data)<br>    val_y = uf.labels_matrix(dev_data)<br>    test_y = uf.labels_matrix(test_data)</pre><pre>return train_x, train_y, test_x, test_y, val_x, val_y, weight_matrix</pre><h4>Step3: Model architecture</h4><p>In order to train the model we are going to use a type of Recurrent Neural Network, know as LSTM (Long Short Term Memory). The main advantage of this network is that it is able to remember the sequence of past data i.e. words in our case in order to make a decision on the sentiment of the word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hkpZ3Ezkn7KINHKP.png" /><figcaption>A RNN Network (<a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Source</a>)</figcaption></figure><p>As seen in the above picture it is basically a sequence of copies of the cells, where output of each cell is forwarded as input to the next. LSTM network are essentially the same but each cell architecture is a bit more complex. This complexity as seen below allows the each cells to decide which of the past information to remember and the ones to forget, if you want more information on the inner working of a LSTM please go to this amazing <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">blog</a> <em>(The illustrations are sourced from this blog).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5_DsbLOQt9pYuqTy.png" /><figcaption>A LSTM Cell (<a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Source</a>)</figcaption></figure><p>We are going to create the network using Keras. Keras is built on tensorflow and can be used to build most types of deep learning models. We are going to specify the layers of the model as below. In order to estimate the parameters such as dropout, no of cells etc I have performed a grid search with different parameter values and chose the parameters with best performance.</p><p><strong>Layers:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*iu04m9kXfzAoBNJRmy_xnw.png" /><figcaption>Model Architecture</figcaption></figure><p><strong>Layer 1: </strong>An embedding layer of a vector size of 100 and a max length of each sentence is set to 56.</p><p><strong>Layer 2: </strong>128 cell bi-directional LSTM layers, where the embedding data is fed to the network. We add a dropout of 0.2 this is used to prevent overfitting.</p><p><strong>Layer 3: </strong>A 512 layer dense network which takes in the input from the LSTM layer. A Dropout of 0.5 is added here.</p><p><strong>Layer 4: </strong>A 10 layer dense network with softmax activation, each class is used to represent a sentiment category, with class 1 representing sentiment score between 0.0 to 0.1 and class 10 representing a sentiment score between 0.9 to 1.</p><p>Code to create an LSTM model in Keras:</p><pre>import os<br>import numpy as np<br>import keras<br>from keras.models import Sequential<br>from keras.layers import Dense<br>from keras.layers import Flatten<br>from keras.layers import LSTM<br>from keras.layers.embeddings import Embedding<br>from keras.layers import Bidirectional<br>from keras.preprocessing import sequence<br>from keras.layers import Dropout<br>from keras.models import model_from_json<br>from keras.models import load_model</pre><pre><br>def create_model_rnn(weight_matrix, max_words, EMBEDDING_DIM):</pre><pre># create the model</pre><pre>model = Sequential()<br>model.add(Embedding(len(weight_matrix), EMBEDDING_DIM, weights=[weight_matrix], input_length=max_words, trainable=False))<br>model.add(Bidirectional(LSTM(128, dropout=0.2, recurrent_dropout=0.2)))<br>model.add(Dense(512, activation=&#39;relu&#39;))<br>model.add(Dropout(0.50))<br>model.add(Dense(10, activation=&#39;softmax&#39;))</pre><pre># Adam Optimiser<br>model.compile(loss=&#39;categorical_crossentropy&#39;,optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])</pre><pre>return model</pre><h4><strong>Step 4: Model Parameters:</strong></h4><p><strong>Activation Function:</strong> I have used ReLU as the activation function. ReLU is a non-linear activation function, which helps complex relationships in the data to be captured by the model.</p><p><strong>Optimiser:</strong> We use adam optimiser, which is an adaptive learning rate optimiser.</p><p><strong>Loss function:</strong> We will train a network to output a probability over the 10 classes using <strong>Cross-Entropy loss</strong>, also called Softmax Loss. It is very useful for multi-class classification.</p><h4>Step 5: Train and test the model</h4><p>We start the training of the model by passing the train, validation and test data set into the function below:</p><pre>def train_model(model,train_x, train_y, test_x, test_y, val_x, val_y, batch_size):</pre><pre># save the best model and early stopping<br>saveBestModel = keras.callbacks.ModelCheckpoint(&#39;../best_weight_glove_bi_100d.hdf5&#39;, monitor=&#39;val_acc&#39;, verbose=0, save_best_only=True, save_weights_only=False, mode=&#39;auto&#39;, period=1)</pre><pre>earlyStopping = keras.callbacks.EarlyStopping(monitor=&#39;val_loss&#39;, min_delta=0, patience=3, verbose=0, mode=&#39;auto&#39;)</pre><pre># Fit the model<br>model.fit(train_x, train_y, batch_size=batch_size, epochs=25,validation_data=(val_x, val_y), callbacks=[saveBestModel, earlyStopping])</pre><pre># Final evaluation of the model<br>score, acc = model.evaluate(test_x, test_y, batch_size=batch_size)<br>return model</pre><p>I have run the training on a <strong>batch size of</strong> <strong>500</strong> items at a time. As you increase the batch size the time for training would reduce but it will require additional computational capacity. Hence it is a trade-off between computation capacity and time for training.</p><p>The training is set to run for 25 epochs. One epoch would mean that the network has seen the entire training data once. As we increase the number of epochs there is a risk that the model will overfit to the training data. Hence to prevent the model from overfitting I have enabled early stopping.</p><p>Early stopping is a method that allows us to specify an arbitrary large number of training epochs and stop training once the model performance stops improving on a hold out/validation dataset.</p><p>The model on the test set of 10 class sentiment classification provides a result of <strong>48.6% </strong>accuracy. The accuracy will be much more higher on a 2 class binary (positive or negative) data set.</p><h4>Step 6: Run the model</h4><p>Once the model is trained you can save the model in keras using the below code.</p><pre>model.save_weights(&quot;/model/best_model.h5&quot;)</pre><p>The next step is to use the trained model in real time to run predictions on new data. In order to do this you will need to transform the input data to embeddings, similar to the way we treated our training data. The function <em>live_test</em> below performs the required pre-processing of the data and returns the result of the trained model.</p><p>Here in order to ensure robustness of the results of the model I am taking the average top 3 sentiments bands from the model. This provides a better calibration for the model results.</p><pre>def live_test(trained_model, data, word_idx):</pre><pre>live_list = []<br>live_list_np = np.zeros((56,1))</pre><pre># split the sentence into its words and remove any punctuations.</pre><pre>tokenizer = RegexpTokenizer(r&#39;\w+&#39;)<br>data_sample_list = tokenizer.tokenize(data)<br>labels = np.array([&#39;1&#39;,&#39;2&#39;,&#39;3&#39;,&#39;4&#39;,&#39;5&#39;,&#39;6&#39;,&#39;7&#39;,&#39;8&#39;,&#39;9&#39;,&#39;10&#39;], dtype = &quot;int&quot;)</pre><pre># get index for the live stage<br>data_index = np.array([word_idx[word.lower()] if word.lower() in word_idx else 0 for word in data_sample_list])<br>data_index_np = np.array(data_index)</pre><pre># padded with zeros of length 56 i.e maximum length<br>padded_array = np.zeros(56)<br>padded_array[:data_index_np.shape[0]] = data_index_np<br>data_index_np_pad = padded_array.astype(int)<br>live_list.append(data_index_np_pad)<br>live_list_np = np.asarray(live_list)</pre><pre># get score from the model<br>score = trained_model.predict(live_list_np, batch_size=1, verbose=0)<br>single_score = np.round(np.argmax(score)/10, decimals=2) # maximum of the array i.e single band</pre><pre># weighted score of top 3 bands<br>top_3_index = np.argsort(score)[0][-3:]<br>top_3_scores = score[0][top_3_index]<br>top_3_weights = top_3_scores/np.sum(top_3_scores)<br>single_score_dot = np.round(np.dot(top_3_index, top_3_weights)/10, decimals = 2)</pre><pre>return single_score_dot, single_score</pre><p>As seen in the code below, you can specify the model path, sample data and the corresponding embeddings to the live_test function. It will return the sentiment of the sample data.</p><pre># Load the best model that is saved in previous step<br>weight_path = &#39;/model/best_model.hdf5&#39;<br>loaded_model = load_model(weight_path)</pre><pre># sample sentence<br>data_sample = &quot;This blog is really interesting.&quot;<br>result = live_test(loaded_model,data_sample, word_idx)</pre><h4>Results</h4><p>Let us compare the results of our deep learning model to the NLTK model by taking a sample.</p><p>LSTM Model: This sentence <em>“Great!! it is raining today!!”</em> contains negative context and our model is able to predict this as seen below. it gives it a score of 0.34.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*9KZJVTt5-Hl2QR4Hib5DmA.png" /><figcaption>LSTM Model</figcaption></figure><p>NLTK Model: The same sentence when analysed by the bi-gram NLTK model, scores it as being positive with a score of 0.74.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*kh7NUpqJpnerkgXZen05Ew.png" /><figcaption>NLTK Model</figcaption></figure><p>Hurray !! This comes to the end of the tutorial of creating a deep learning sentiment classification model for text data.</p><p>You can download the source code from <a href="https://gitlab.com/praj88/deepsentiment">gitlab</a> and play around to train the network on your own data. I will cover on how to deploy this model on scale using dockers and api service in a separate blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0a0c6472b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/sentiment-analysis-for-text-with-deep-learning-2f0a0c6472b5">Sentiment analysis for text with Deep Learning</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>