<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Patrick Leung on Medium]]></title>
        <description><![CDATA[Stories by Patrick Leung on Medium]]></description>
        <link>https://medium.com/@patrickhk?source=rss-535b63fbab82------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lPGNlrSz1R6BOdWc0Ak8VA.jpeg</url>
            <title>Stories by Patrick Leung on Medium</title>
            <link>https://medium.com/@patrickhk?source=rss-535b63fbab82------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:42:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@patrickhk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Dashboard is not data analytics]]></title>
            <link>https://medium.com/@patrickhk/dashboard-is-not-data-analytics-88bd03aa7e15?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/88bd03aa7e15</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[pharmaceutical]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Mon, 22 May 2023 09:26:47 GMT</pubDate>
            <atom:updated>2024-11-22T10:54:56.503Z</atom:updated>
            <content:encoded><![CDATA[<p>Dashboard != data analytics. In this blog we will share the difference between a high value analytic dashboard and a low value data reporting dashboard</p><p>We are increasingly aware of the importance of data. Whenever companies mention about innovation and technology, we often come across buzzwords like “big data”, “data analytics”, etc. (in 2023, these would be “ChatGPT”, “Gen AI”)</p><p>While data is important, it is easy for us to <strong>mistakenly believe that building dashboards equates to data analytics and on the right direction, it is a misconception.</strong> A dashboard can be analytic but not all dashboards contain data analytics elements. Let’s break it down into several sentences for better understanding:</p><h3>1. Data reporting is not analytics.</h3><p><strong>Showing users with raw data will not provide insights</strong>, otherwise sending everyone the raw Excel file would be the most efficient and cost-effective option.</p><h4><strong>2. An analytic dashboard allows users to extract actionable insights by quickly looking at the data, enabling them to make decisions.</strong></h4><p>If the dashboard doesn’t offer this capability, it is only a data reporting dashboard.</p><p>It is common to see that many organizational dashboards are solely for data reporting, which often provide limited value. For example, in the context of digital marketing, consider the following two dashboards if a company wants to track email open rates and develop strategies</p><h3><strong>Low-value data reporting dashboard</strong></h3><p>This dashboard displays a <strong>full list of email opening records</strong>. Can you quickly advise a strategy by looking at this table for 5 seconds? Quite difficult.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*V5_hlyTumwcNnI98NPliTQ.jpeg" /><figcaption>Example of low value data reporting dashboard, this table is generated with fake data for demonstration purpose only</figcaption></figure><h3><strong>High-value analytic dashboard</strong></h3><p>This dashboard <strong>represents the email opening time by customers through color intensity</strong>. Can you quickly develop a strategy by looking at this table for 5 seconds? <strong>Yes! We can see that most customers open the promotional email at morning 6–10 am</strong>, especially on Thursday and Friday.</p><p>If I am the sales &amp; marketing team, we should instruct the IT team to send the promotional email at morning time. This is <strong>an actionable insight</strong> extracted from a useful dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iO_P6N5-K_rsr3jraW1HA.jpeg" /><figcaption>Example of high value data reporting dashboard, this table is generated with fake data for demonstration purpose</figcaption></figure><p>Both dashboards utilize the same data, but their outcomes and value differ significantly. It doesn’t require much additional resources to deliver an analytical dashboard. However, it is common, especially in non-tech corporate environments, to see dashboards that only offer basic data reporting.</p><p>Below are some signs and indicators that the dashboard development <strong>may not be in the right direction</strong>:</p><h3>1. Constantly developing new dashboards to replace similar ones</h3><p>Enhancement is different from replacement. We need to enhance our dashboards in response to dynamic market situations, but frequent replacements indicate that the <strong>dashboard development did not thoroughly consider and address the users’ business needs.</strong></p><h3>2. Providing excessive dashboards to users</h3><p>What truly brings value to users? The insights behind data, not the number of dashboards. Data analytics is like funneling, we may extract 1% insights from a huge dataset. One useful dashboard is worth more than 10 related but unhelpful dashboards. Excessive dashboard may indicate that the development is focusing on quantity instead of quality.</p><h3>3. Excessive KPIs and metrics without justifying their business importance</h3><p>This indicates <strong>a lack of understanding of the users’ true business needs and a loss of focus</strong>. As a former frontline sales representative, I can say most KPIs were rarely used and not very helpful to me. The excessive number of KPIs confused me rather than assisting me in conducting my business.</p><p>Data reporting dashboards are important as they can provide comprehensive information to users, but users must process the raw information themselves to identify and extract insights. To provide the greatest value to users, we need to <strong>offer instant and actionable feedback/insights. </strong>Currently, a data analytic dashboard can partially fulfill this objective.</p><p>With the latest advancements in language models, I believe <strong>in future we no longer need dashboards</strong>. Or, there could be a <strong>hybrid approach involving a command prompt and dashboard to provide basic data analytics to users</strong>. If I have time in the future, I will work on building a proof of concept for this hybrid solution.</p><p>(<strong>updated in 20th Nov 2024</strong>: The Q&amp;A function in PowerBI already partially fulfills this need)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FEMyKbk2joJ0-AUmJ9UXIQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88bd03aa7e15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customized Voice to Voice ChatGPT chatbot]]></title>
            <link>https://medium.com/@patrickhk/customized-voice-to-voice-chatgpt-chatbot-32da6e4d7e6f?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/32da6e4d7e6f</guid>
            <category><![CDATA[pharmaceutical]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[voice-to-voice]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 17:17:12 GMT</pubDate>
            <atom:updated>2024-11-22T10:56:44.343Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How to lower the barrier to modern technology?</strong></h3><p>ChatGPT is no longer “new” to everyone, I bet most of us have tried it. To use ChatGPT, we would need to register an account, use a web browser, install some other extra apps, or even a VPN (as it is blocked in Hong Kong). These maybe easy for us, but how about for the elderly? A vast majority of patients are elderly. Would it be easy to motivate a 70-year-old senior to go through all the trouble to use ChatGPT?</p><p>Take my mom as an example, she only texts me with emojis or voice messages because she finds it inconvenient to type. How can we motivate people to step out of their comfort zone and try new technology?</p><p><strong>Features:</strong></p><p>In order to lower the barrier and increase the accessibility of technology for everyone, I optimized my previous ChatGPT chatbot into this version with the following features:</p><ol><li>Supports voice-to-voice conversation. Not everyone likes typing.</li><li>Supports voice customization to increase attraction or provide sentimental value to users. For example, we use the voice of Chris Hemsworth and Elizabeth Olsen in the demo video.</li><li>Integrates with existing instant messaging apps, with no extra app required.</li><li>Free (we offer the voice-to-chat version for free to the public)</li><li>Regulation mechanism and full documentation for future audit purpose (this is to mimic corporate level regulatory requirement.</li></ol><p>Please watch the below YouTube video for a demonstration. Please note that we have trimmed away the voice generation waiting time for a better experience. For a full evaluation, please refer to the uncut version at the end of the article.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMIN4BBdgQwg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMIN4BBdgQwg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMIN4BBdgQwg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/606ce6152dfd0ef9d1b5985223aca387/href">https://medium.com/media/606ce6152dfd0ef9d1b5985223aca387/href</a></iframe><h3>Serverless Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FdbNeM1fQaeoUCXnZ7WJWQ.jpeg" /></figure><h3><strong>Customized Voice</strong></h3><p>We use the voices of celebrities Chris Hemsworth and Elizabeth Olsen in the demo to catch your attention. However, the true significance of using a customized voice is to <strong>maximize the influence and persuasive power on patients</strong>. It is common to see elderly individuals refuse to follow the instructions of healthcare professionals, but they will listen to their children or grandchildren. Imagine if the medical bot used the grandkid’s voice to remind grandpa to take his medicine on time or encourage him to quit smoking. Or, what if we used the voice of a deceased family member to <strong>help depression patients get rid of their grief and restore their life?</strong></p><p>There are too many <strong>ethical and moral</strong> concerns and debates about using a customized voice that closely resembles a real person. We hope to brainstorm and inspire discussions instead of making conclusions and judgments here.</p><h3><strong>Voice generation time and how to improve</strong></h3><p>If you watch the uncut original video (10 minutes) below, you will find out that the total waiting time (2 minutes) to generate the voice response can affect user experience. We have summarized the reasons and potential optimizations to solve the issue:</p><ol><li>The reply from ChatGPT is usually unnecessarily long, and it takes longer to convert larger text into speech. To solve this issue, we can set a smaller maximum token in ChatGPT, but this may affect the comprehensiveness and quality of the reply answer. Alternatively, we can stream the text-to-speech reply instead of capturing the complete speech reply. Or we can split the long single speech reply into several smaller replies.</li><li>Under our current workflow, we upload every detail into S3 and then send out the reply to users. This measure is for the highest compliance and regulation standard, as this can prevent any undocumented message from being sent. However, the drawback is that it will delay the voice message reply time. We can rewrite the code to send the voice message first then upload the file into S3 so users don’t have to wait.</li><li>As this is our non-money making side project, we use some free-tier or slower hostings; therefore, it may take longer to generate the voice response. The generation time will get faster if we choose more premium services.</li></ol><h3><strong>To do / for others who are interested in this topic</strong></h3><ol><li>We can achieve voice to voice message communication between users and ChatGPT, if we can make progress in reducing voice generation time. This could allow us to experiment with real-time phone conversations.</li><li>Currently, we are largely dependent on the ChatGPT model, unless we have developed a stable partnership with OpenAI. Otherwise, we should develop and fine-tune our organization’s own model.</li><li>We should also explore the regulations, compliance, privacy, and ethical concerns of using customized voice technology</li></ol><h3>Updated in 20th Nov 2024:</h3><ul><li>The QR code to connect the linebot for public access is disable</li><li>Real time voice to voice conversation is already enable in the latest GPT model</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32da6e4d7e6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop our ChatGPT-like Large Language Model]]></title>
            <link>https://medium.com/@patrickhk/develop-our-chatgpt-like-large-language-model-9a473f2e79c8?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a473f2e79c8</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pharmaceutical]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Mon, 10 Apr 2023 17:09:16 GMT</pubDate>
            <atom:updated>2023-04-10T17:09:16.722Z</atom:updated>
            <content:encoded><![CDATA[<p>Every day on the Internet, we see new applications or integrations of the ChatGPT. We are well aware of the potential benefits and positive change this technology may bring, but the related<strong> data protection and privacy concerns are key fundamental issues to be resolved or regulated</strong> if pharmaceutical companies want to apply ChatGPT or other large language model (LLM) in their business.</p><p>ChatGPT was <a href="https://www.bbc.com/news/technology-65139406">banned in Italy</a> over privacy concerns in March 2023, sensitive information in <a href="https://www.foxbusiness.com/technology/samsung-employees-reportedly-leaked-sensitive-info-chatgpt-accident">Samsung was reportedly leaked</a> to ChatGPT by accident in March 2023. Why did this happen? It is stated clearly on the <a href="https://help.openai.com/en/articles/6783457-what-is-chatgpt">OpenAI website</a> that <strong>they can review our conversations and use them to train their model</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*F8QyFDALANh-jFXVt4dDsQ.jpeg" /><figcaption><em>Q&amp;A of chatgpt @</em><a href="https://help.openai.com/en/articles/6783457-what-is-chatgpt">https://help.openai.com/en/articles/6783457-what-is-chatgpt</a></figcaption></figure><p>Therefore, <strong>company employees should not provide any proprietary data or information to the ChatGPT</strong>. It is true that API access or partnership with ChatGPT should be relatively safe and exclude company data from the review of OpenAI.</p><p>However, in 2023, where billion $USD company like FTX or SVB (Silicon Valley Bridge Bank) can go bankrupt within a few days, we <strong>should not overly trust any third party </strong>unless we can inspect the source code to confirm there is no possible leak of information to OpenAI or other third parties.</p><p>This may seem overly worried, but pharmaceutical industry has the most stringent regulations and ethical standards. Pharmaceutical companies <strong>should not risk leaking patient data or any proprietary information</strong>.</p><p>In the long run, <strong>the permanent and safest solution is to develop a ChatGPT-like model of our own</strong></p><p>In past it was infeasible to develop or fine-tune our own Large language model (LLM) without corporate-level resources.</p><p>However, with the recent “open source” publication of the <strong>Alpaca model</strong> from the Stanford Lab and fine-tuning method low-rank adaptation (<strong>LoRA</strong>), from March 2023, anyone, including you and me, <strong>can run our own LLM</strong> at home or use free online services such as Colab offered by Google.</p><p>PS: I also ask myself a question:</p><p><strong><em>Since we send emails through Outlook and store company data in OneDrive or AWS, should we worry about our proprietary data being leaked to Microsoft or Amazon?</em></strong></p><p>My own answer is that their pockets are deep enough to compensate pharmaceutical companies or patients with a lawsuit if the worst situation occurs. I think it would be better for the legal and compliance team of pharmaceutical companies to answer this.</p><h3><strong>Project Introduction</strong></h3><p>We experimented <strong>4 models</strong> in our project: the <strong>original Stanford Alpaca 7B model, the fine-tuned Alpaca-Traditional Chinese 7B model, the fine-tuned Alpaca-France 7B model and the fine-tuned Alpaca-Simplified Chinese 7B model</strong>.</p><p>For quick experiment purpose we only applied baseline fine-tuning parameters and trained each model for 3 epochs.</p><p>We compared the performance of the models by asking the same two questions to each of them:</p><p>1. <strong>What is global warming</strong></p><p>2. <strong>What is paracetamol</strong></p><p>The result of our project is recorded in the video in the results section.</p><h3><strong>About the LLaMA model</strong></h3><p>The LLaMA model (Large Language Model Meta AI) is developed by Meta. It is a collection of language models ranging from 7B to 65B parameters trained with trillions of tokens. It has demonstrated the possibility to<strong> train state-of-the-art models</strong> <strong>using publicly available datasets exclusively</strong>, without resorting to proprietary and inaccessible datasets. The LLaMA-13B outperforms GPT-3 (175B) on most benchmarks.</p><p>Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E. and Lample, G. (n.d.). <em>LLaMA: Open and Efficient Foundation Language Models</em>. [online] Available at: <a href="https://arxiv.org/pdf/2302.13971.pdf">https://arxiv.org/pdf/2302.13971.pdf</a> <em>(Feb 2023)</em></p><h3><strong>About the Alpaca model</strong></h3><p>Alpaca is developed and fine-tuned from Meta’s LLaMA 7B model. It was trained with <strong>52K instruction-following demonstration</strong>s generated in the style of self-instruct using text-davinci-003. On the self-instruct evaluation set, Alpaca shows many behaviors similar to OpenAI’s text-davinci-003 but is also surprisingly small and easy/cheap to reproduce.‌</p><p><a href="http://crfm.stanford.edu/">crfm.stanford.edu</a>. <em>Stanford CRFM</em>. [online] Available at: <a href="https://crfm.stanford.edu/2023/03/13/alpaca.html">https://crfm.stanford.edu/2023/03/13/alpaca.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knaiJau3G0xbCNLl7Qxfag.jpeg" /><figcaption><a href="https://crfm.stanford.edu/2023/03/13/alpaca.html">https://crfm.stanford.edu/2023/03/13/alpaca.html</a></figcaption></figure><h3><strong>Low-Rank Adaption (LoRA)</strong></h3><p>LoRA<strong> freezes the pretrained model weights</strong> and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the <strong>GPU memory requirement by 3 times‌.</strong></p><p>Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L. and Chen, W. (n.d.). <em>LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS</em>. [online] Available at: <a href="https://arxiv.org/pdf/2106.09685.pdf">https://arxiv.org/pdf/2106.09685.pdf</a> <em>(Oct 2021)</em></p><h3><strong>Fine tune the LLaMA Alpaca 7B model</strong></h3><p>As the Stanford team only used English content to train the Alpaca 7B model, it will not perform well for other languages such as Chinese or French. In order to fine-tune the Alpaca model, we need to prepare our own dataset.</p><p>We followed their approach to create our own dataset but in the traditional Chinese version. With the limited amount of resources, we only prepared 8k conversations as our dataset. We did it by <strong>randomly selecting 8k conversations out of 52k from the Alpaca dataset and then used GPT to translate them into Traditional Chinese.</strong></p><p>Obviously, this method has limitations because each language has its own system and grammar, and the intrinsic meaning may not stay the same after direct translation from English into Traditional Chinese. However, it is the <strong>most cost-effective method</strong> to generate our own dataset without any third-party support.</p><p>A better way would be to have a team to <strong>prepare, correct, and validate the Traditional Chinese conversations, as well as to crawl the internet to include more Chinese content in the dataset</strong>. We also used other available datasets prepared by others to train the model in French and Simplified Chinese.</p><h3><strong>Result with original Stanford-Alpaca model</strong></h3><p>The result seems satisfying. (Video start — 00:39s)</p><h3><strong>Result with our own prepared translated 8k Traditional Chinese dataset</strong></h3><p>Honestly the result is not good. From the video, we can see <strong>multiple repeated tokens</strong> in both the replies. The model also answered something wrong for Q2. (It took around 2hours to train in the Colab GPU) (Video 00:40–01:38)</p><h3><strong>Result with 52k translated French dataset provided by others</strong></h3><p>The result seems satisfying. We don’t understand French, so we copy and paste the answer in Google Translate to validate. The replies seem logical for both questions. (It took around 6 hours to train in the Colab GPU) (Video 01:39–02:53)</p><h3><strong>Result with 52k translated Simplified Chinese dataset provided by others</strong></h3><p>The result seems average. The reply for the first question is good, but the model failed to understand and answer something wrong for Q2, which is the same as our 8k traditional Chinese model. (It took around 6 hours to train in the Colab GPU)</p><p>(Video 02: 54 — End)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxasGupaIcfc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxasGupaIcfc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxasGupaIcfc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa9c94e5f0b2320069743fb9d7a855e2/href">https://medium.com/media/fa9c94e5f0b2320069743fb9d7a855e2/href</a></iframe><h3><strong>Feasibility to develop our own LLM</strong></h3><p>The significance of our project is to demonstrate that pharmaceutical companies can develop or fine-tune their own LLM. We can fine-tune the LLaMA Alpaca 7B model by providing it with different training data. It means that<strong> if we provide adequate medical information, research papers, and drug information, pharmaceutical companies can develop their own medical LLM for their own use without worrying about any leaks of information to any third parties.</strong></p><p>In fact, many companies are already doing this to develop their own models, such as Bloomberg, which released their BloombergGPT model for financial data.</p><p>Wu, S., Ozanirsoy, Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D. and Mann, G. (n.d.). <em>BloombergGPT: A Large Language Model for Finance</em>. [online] Available at: <a href="https://arxiv.org/pdf/2303.17564.pdf">https://arxiv.org/pdf/2303.17564.pdf</a> <em>(Mar 2023)</em></p><h3><strong>Importance of the training dataset</strong></h3><p>From the experimental results, we can see that the fine-tuned Simplified Chinese model performed better than the traditional Chinese model, even though both are Chinese languages. It is believed that the reason for the performance difference is due to the <strong>dataset quality</strong>.</p><p>The <strong>volume and quality of the training data will largely determine the robustness of the machine learning model</strong>. We only partially extracted and translated 8K of the data for the traditional Chinese model, while we used the complete 52k translated dataset for the Simplified Chinese model.</p><p>If pharmaceutical companies want to develop or fine tune their own LLM, they <strong>should spend efforts to create a good quality training dataset.</strong></p><h3><strong>Run the model locally</strong></h3><p>In the result demonstration for the original Alpaca model, we disconnected the internet connection to show the audience that it can run locally without any worries of transferring/leaking data to any third party.</p><p>For the other models, we resumed the internet connection because we wanted to use Google Translate to show the audience the results of the models in Chinese and French. Also, we trained the models through Colab GPU.</p><p><strong>To infer the result, we can run the model locally without internet access.</strong> (Please notice that even though the model can communicate with the Internet, it doesn’t mean it is leaking information).</p><h3><strong>The result is less optimal compared to ChatGPT?</strong></h3><p>It is unrealistic to think that we can train a model for a few hours with less than $10USD, and it can beat the $29 billion USD (valuation) ChatGPT. It took the OpenAI company a few years of hard work and countless money, time, and resources to build the ChatGPT model.</p><p>It would be a <strong>significant boost</strong> to your company if corporate can create their own LLM with ~70% accuracy as ChatGPT <strong>with a fraction of the R&amp;D cost</strong>. Below is the benchmark of some available LLMs, for reference only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*OUw9qOlfo0MDqwZKXpC94w.png" /><figcaption><em>Benchmark of some LLMs @ </em><a href="https://www.aidemos.info/vicuna-open-source-chatbot-trained-with-300/">https://www.aidemos.info/vicuna-open-source-chatbot-trained-with-300/</a></figcaption></figure><h3><strong>Is the result comparison fair?</strong></h3><p>Honestly, the comparison is not comprehensive enough. For example, it is not enough to evaluate the model’s performance by asking only two questions.</p><p>This is <strong>not a research paper or paid project</strong>, so we don’t bother to benchmark and evaluate the models with many criteria. <strong>The goal of this project is to show people we can fine-tune our own LLM locally, to bring value to corporate without worrying about the leak of proprietary data.</strong></p><h3><strong>Remark</strong></h3><p>The development of LLM is very <strong>rapid</strong>, and there will be some <strong>breakthrough or improvement every few days</strong>. When I was experimenting with the Alpaca model at the end of March 2023, 7B was the standard, while now, 13B or even 30B are the entry-level LLMs. Also, we have more varieties of LLMs, such as <strong>GPT4All, Vicuna 13B, and Dolly</strong>.</p><p>The utilization of <strong>llama.cpp</strong> is astonishing, and now we can even run the model on a Raspberry Pi, without the need for a proper PC. Really Looking Forward to the Future!</p><h3><strong>Source Code</strong></h3><p>You can find all the<strong> fine tuning, inference, dataset preparation source code</strong> and different datasets in the following Github. Thanks to their contribution everyone can experiment with their own LLM now.</p><p><a href="https://github.com/tatsu-lab/stanford_alpaca">https://github.com/tatsu-lab/stanford_alpaca</a></p><p><a href="https://github.com/tloen/alpaca-lora">GitHub - tloen/alpaca-lora: Instruct-tune LLaMA on consumer hardware</a></p><h3><strong>To do / future</strong></h3><p>As we are not authorized by any pharmaceutical company to explore and develop LLM on behalf of them, we don’t have permission to prepare the medical dataset by using their clinical paper or drug information.</p><p>Im thinking to use some off-patent old medicine information to build a dataset to create the local LLM but it’s just a thought at the moment.</p><p><a href="https://peli5.com/develop-our-chatgpt-like-large-language-model">https://peli5.com/develop-our-chatgpt-like-large-language-model</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a473f2e79c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apply ChatGPT in the pharmaceutical industry?]]></title>
            <link>https://medium.com/@patrickhk/apply-chatgpt-in-the-pharmaceutical-industry-f2b9aa6f32bb?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2b9aa6f32bb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pharmaceutical]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Sun, 29 Jan 2023 07:28:16 GMT</pubDate>
            <atom:updated>2023-03-15T02:23:28.955Z</atom:updated>
            <content:encoded><![CDATA[<p>My immediate thought is to build an AI chatbot to handle medical enquiry from doctors, nurses and the general public with the objectives to:</p><p>1. Enhance Health Care Professionals (HCPs) engagement by providing an efficient way to access drugs’ information and clinical study.<br>2. As a strong motivation for HCPs to engage and stay connected with pharma company digitally.<br>3. Optimize the work of medical / sales staff by quickly access or verify the required medical information<br>4. Automate customer services and improve satisfaction from the general public<br>5. More granular screening and segmentation. Redirect potential patients for HCPs; redirect potential HCPs to medical or sales reps.</p><p>To experiment it I quickly parse the GPT 3.0 model into a LINE chatbot and the result is satisfying. The video below is a demo for some simple conversations.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FY8qQljTOWzU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DY8qQljTOWzU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FY8qQljTOWzU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/119405429c6cea809e66e6cc855374f7/href">https://medium.com/media/119405429c6cea809e66e6cc855374f7/href</a></iframe><p>However I do notice sometimes the model will answer incorrect information (confidently 😂 ) if I ask the detail of clinical study. I try asking the same in the latest Chatgpt (3.5) and the issue seems no long exist. But it rings a bell to me that “can I really trust this answer🤔?” But to be fair, we also can have the same doubt even if we ask a human as he/she can also be wrong.</p><p>To really actualize the use of chatgpt model in medical AI chatbot I think we need to fine-tune the model with lots of medical approved information.</p><p>The greatest challenge I expect is content regulation for all machine learning projects. Scientific contents to external audience require prior medical approval because medical information can be life-dependent. However in ML we don’t “control” the output as it is the best value generated by the model base on optimal probability. I think there is a lack of precedence to approve or regulate AI content in pharmaceutical industry.</p><p>In terms of regulation and compliance, a traditional if-else statement content flow is definitely more safer. However, is it the right thing to remain “safe” instead of trying to explore and leverage ML with the potential to bring greater good?</p><p>Any thought 😁? How can we make use of chatgpt in pharma industry? How to evaluate or regulate the AI/ML generated content? or any wild but juicy idea 😋 ??</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2b9aa6f32bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selenium, beautiful soup and regex to download lecture videos from University of London]]></title>
            <link>https://medium.com/@patrickhk/selenium-beautiful-soup-and-regex-to-download-lecture-videos-from-university-of-london-42e7f4011f56?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/42e7f4011f56</guid>
            <category><![CDATA[web-scrape]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[university-of-london]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 16:01:46 GMT</pubDate>
            <atom:updated>2020-07-14T16:01:46.378Z</atom:updated>
            <content:encoded><![CDATA[<p>Currently I am studying the Msc of Data Science from University of London and we can download the lecture videos through student portal for offline learning. This article will share how to use python to automatically download all the learning videos from UoL. Full code will be provided at the end.</p><h3>1. Use Selenium to login the student portal</h3><p>To scrape information from website requiring login, we can use requests.session().postor Selenium.</p><p>In this sharing I will use Selenium to help me login the student portal because it is easy and convenient.</p><p>Go to <a href="https://learn.london.ac.uk/login/index.php">https://learn.london.ac.uk/login/index.php</a> and we want Selenium to input the Username and Password field and click the Log in button for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdwb3NiJw1rghHIzEfEkbg.png" /><figcaption>some information are erased due to privacy</figcaption></figure><p>First check the elements of the webpage and we can see the element_by_id are username, password and loginbtn respectively. Therefore we can use the follow code to get the job done.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6a9e2634e429aaf08e5c6cc4c75651f/href">https://medium.com/media/a6a9e2634e429aaf08e5c6cc4c75651f/href</a></iframe><p>By running the above code in jupyter notebook, a chrome browser will be created and login the student portal automatically.</p><h3>2. Use beautiful soup to scrape and regex to get the video link</h3><p>We have different courses in one semester and each course contains several topics, each topic have several mini lectures. Therefore my goal is to scrape the link of each mini lecture and extract the link for the lecture video then download it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fkt2I-CnHBxEWTELnG0ZcQ.png" /><figcaption>The structure is like that, one course contains several topics, each topic contains several mini lectures</figcaption></figure><p>First we get the html content from selenium and process through beautiful soup. Play around with the soup object you will discover the link of each mini lecture is under the progressEventInfo class, to be accurate, in the a[‘href’], therefore we create a list scraping all the link for the material using the below code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95e8bf51036e3feffaf60cbeb012eb73/href">https://medium.com/media/95e8bf51036e3feffaf60cbeb012eb73/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*Mlg_HlXi_N-SCbij1FM8Lg.png" /></figure><p>So now we have a list called subcontent_linkfor all the course materials. We set a condition to only scrape the link involving lecture video, which is if the title containing keyword lecture summary and webinar</p><pre>for k,v in zip (subcontent_title,subcontent_link):<br>    if &#39;lecture&#39; in k or &#39;summary&#39; in k or &#39;webinar&#39; in k:</pre><p>Then we feed these link and get their content through Selenium then process in soup object. However the link for the lecture video is hidden through the Javascript video player and cannot be scraped directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cgnqYbakaOntx3on_PNgLA.png" /><figcaption>each mini lecture comes with a lecture video, embedded in javascript player and no directly link</figcaption></figure><p>If we manually click the download play button in the Javascript video player, we can discover the link is always with this pattern.</p><pre>http://dr3vr6XXXXXXX.cloudfront.net/videostore/dsm020/mp4/dsm020_topic02_intro01_720.mp4</pre><p>While in the webpage we can get similar link in this form</p><pre>dr3vr6XXXXXXX.cloudfront.net/videostore/dsm020/rss/dsm020_topic02_intro01.rss</pre><p><em>(I masked the first part dr3vr6XXXXXXX as I am not sure if this is some sort of authentication and unique to user)</em></p><p>Great! We can apply regex to get the semi-link and edit the link through replace rss with mp4 and add _1080p then we will obtain the final video link.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2af5c3f6de5ac2e3db9bb8ecad4f2f2/href">https://medium.com/media/e2af5c3f6de5ac2e3db9bb8ecad4f2f2/href</a></iframe><p>Then we just use request to download the video and rename it by using the title of the lecture.</p><h3>Fullcode</h3><p>Full code are availabe in jupyter notebook format and html for easier reading</p><p><a href="https://github.com/fiyero/Selenium_download_lecture_video_from_Univeristy_of_London">https://github.com/fiyero/Selenium_download_lecture_video_from_Univeristy_of_London</a></p><h3>Why not use request.POST?</h3><p>Honestly I did try to use request.session().POST instead of Selenium. I spend hours and read several articles but failed. I can successfully login the student portal but encounter redirects error and cannot scrape the webpage content. I trying adjusting the session.max_redirects or allow_redirects=False but still failed to to the job. So if anyone know how to do it, welcome to share!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42e7f4011f56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Keras to build a English to French translator with various RNN model architecture]]></title>
            <link>https://medium.com/@patrickhk/use-keras-to-build-a-english-to-french-translator-with-various-rnn-model-architecture-a37439005ae8?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/a37439005ae8</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rnn]]></category>
            <category><![CDATA[seq2seq]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 17:57:34 GMT</pubDate>
            <atom:updated>2019-07-11T18:09:59.258Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article is written for summary purpose for my own mini project. Code is provided at the end for your reference. My main purposes are to demonstrate the results and briefly summarize the concept flow to reinforce my learning. I won’t go into so much details about EACH technique I used because there are too MANY well documented tutorials. I don’t want to create another guide by rephrasing and summarizing.</em></p><p>This is the second project of my udacity NLP nanodegree and we are required to use Keras as the framework. I have done similar seq2seq model with pytorch before. I find this project quite interesting and is a good chance for me to get familiarize with Keras.</p><h3>Objective</h3><p>We want to build a RNN model that can input English sentence and output French sentence, so it is actually a translator.</p><h3>Dataset</h3><p>Udacity provided us with two txt files, one containing English sentences and the other containing the corresponding French sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*jChPBcR0VfTDinD53z7L7A.png" /></figure><h3>Data pre processing</h3><p>Our model cannot read raw text in string directly, we have to first convert them into integer. We have to carry out tokenization to split each sentence into tokens and each token will be encoded with a unique token idx.</p><p>We can write a function to apply Tokenizer from Keras to get the tokens</p><pre>from keras.preprocessing.text import Tokenizer</pre><pre>def tokenize(x):<br>x_tk = Tokenizer()<br>    x_tk.fit_on_texts(x)<br>    return x_tk.texts_to_sequences(x), x_tk</pre><p>For example, if our corpus contain just three sentences</p><pre>text_sentences = [<br>    &#39;The quick brown fox jumps over the lazy dog .&#39;,<br>    &#39;By Jove , my quick study of lexicography won a prize .&#39;,<br>    &#39;This is a short sentence .&#39;]</pre><p>After apply Tokenizer from Keras, each sentence will be encoded into:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*1OvkbXYEt3jxtVMqrdBhOg.png" /><figcaption>input is raw English sentence. Output is token idx</figcaption></figure><p>The length of sentence in our corpus are varying, some are longer while some shorter. We want to feed our data by batch and within each batch they should have same length. Therefore we will carry out padding to pad or truncate them into equal length.</p><p>We can write a function to get the max length of sentence and pad all sentences to this length with the help of Keras pad_sequences</p><pre>from keras.preprocessing.sequence import pad_sequences<br>def pad(x, length=None):<br>    if length is None:<br>        length = max([len(sentence) for sentence in x])<br>        <br>    return pad_sequences(x, maxlen=length, padding=&#39;post&#39;)</pre><p>padding = ‘post’ the 0 padding will be added at the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*jkfvKRWbUhoRzw-sY3EOAw.png" /><figcaption>input is before padding, output is after padding</figcaption></figure><p>Apply the above preprocessing steps in our corpus then we will have English and French sentence in token idx form which are our data set!</p><p>x_train is English token idx and y_train is French token idx. Before moving on to build the RNN model we need to know the Eng and French vocab_size, both can be obtained from len(tokenizer.word_index)</p><pre>english_vocab_size = len(english_tokenizer.word_index)<br>french_vocab_size = len(french_tokenizer.word_index)</pre><h3>Write function to map logits back to token label</h3><p>The output of our model is just probability distribution of class token idx. We have to map the token idx back to its token label. For example if output is 3, we should be able to map it back to token “hello”. Don’t forget to add the pad token in the mapping as well.</p><pre>def logits_to_text(logits, tokenizer):</pre><pre>idx_to_words = {id: word for word, id in tokenizer.word_index.items()}<br>    idx_to_words[0] = &#39;&lt;PAD&gt;&#39;</pre><pre>return &#39; &#39;.join([idx_to_words[prediction] for prediction in np.argmax(logits, 1)])</pre><h3>Build the models and Hyperparameters</h3><p>I use CuDNNGRU or CuDNNLSTM instead of normal GRU and LSTM because the CuDNN form can train 3–5 times faster on my GPU. However there is no way to add dropout with CuDNNGRU or CuDNNLSTM therefore you should consider whether sacrificing training time for better generalization. For me this project is not a business product to clients therefore I prefer faster training time.</p><p>For the optimizer I follow Udaicty’s suggestion to use SGD, usually I prefer Adam. The learning rate I used was between 0.01 to 0.001, I didnt spend much time on the hyperparameter.</p><p>For the batch_size I use 512, if GPU is out of memory I switch into 256. For high demanding project like in Kaggle, the number of batch_size is important as it will affect the convergence of the model. But in here I just want it to be trained ASAP so I choose the max batch_size allowed for my GTX 1080ti.</p><h3>Model 1: simple RNN model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*Yt0U1hNdZIMVcsIjjdz4hw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*ZTcoxahBbm2NXlm1jyry7g.png" /><figcaption>result</figcaption></figure><p>validation accuracy is 0.6708 after 5 epochs. (actually I should have trained it 10 epochs to match with other models, so I can have better baseline comparison.)</p><h3>Model 2: Simple RNN with embedding layer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*ZLK5X0OZP22Gb4UsTMreGw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPXswk6wzvs75c923AuNtw.png" /></figure><p>validation accuracy is 0.9357 after 10 epochs.</p><h3>Model 3: Bidirectional RNN</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*H_WfPaTPN6pRvpnmSohATQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*PBic3ulXgZCautxfyCFZJQ.png" /></figure><p>validation accuracy is 0.7422 after 10 epochs.</p><h3>Model 4: Encoder-decoder RNN</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*bDT0ohT-5-WqGT130zip0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*SQh7ayZhHXqFCQCGpfHBTQ.png" /></figure><p>validation accuracy is 0.7021 after 10 epochs</p><h3>Model 5 embedding and bidirectional</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*VltKGpUmTHyV1zW2nDtglg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*E1Nq_rCGjcyYRIOH4-4KvA.png" /></figure><p>validation accuracy is 0.9809 after 20 epochs. (I trained it twice)</p><h3>Testing the model with custom sentence</h3><p>By just looking at the validation accuracy, it seems the model 5 Bidirectional CuDNNGRU with embedding layer perform the best. So lets try our custom sentence into model 5.</p><p>The custom sentence is <strong>“i visit paris in may it was beautiful and relaxing”</strong></p><p>(I really went to Paris for 2 weeks in May and I love it ;) )</p><p>Remember we cannot pass raw text string into our model, we should pre-process it then feed it into our model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*oZK9ZeetksSUiiLrbvfrJA.png" /></figure><p>The output result is <strong>je est paris au mois de mai mais il est relaxant en</strong></p><p>Lets put it into Google translate and see if it can get back into original English sentence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FS9Fv8jz2aSpyKTn-9HC-g.png" /></figure><p>I can get the keyword “May” and “relaxing” but the rest seem..quite bad 😐</p><p>Let’s see if we<strong> switch paris into capital letter Paris </strong>and try again</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsHFl5d6ZTgj6bd2DCZ_rA.png" /></figure><p>Oh it seems much better now 🙂</p><p>Original is <strong>“i visit paris in may it was beautiful and relaxing”</strong></p><p>the translated is “<strong>I am Paris in May but it is relaxing in”</strong></p><h3>Discussion</h3><ol><li>Overfitting for sure. I use CuDNNGRU to sacrifice generalization for the fast training time, I cannot add dropout in CuDNNGRU. However I do can add dropout in between the dense layer, which should help. Overfitting explains why validation accuracy can reach 0.9809 but translated custom sentence doesn’t seems so accurate</li><li>Pitiful dataset. Yes the corpus is too small and definitely not enough to train a proper English-French translator. You can see the training log it only took me around 10s to train for one epoch. I guess a “real” dataset may cause me a few hours for one epoch. To solve this problem I should find another dataset</li><li>The more complicated Encoder-decoder RNN surprisingly performed just slightly better than the simple RNN. I guess for a small dataset, the more complicated architecture tend to underfit more compared to simple architecture. So if our dataset is large, the performance of Encoder-decoder RNN should far better than the simple RNN. Also, usually seq2seq is used in combination with embedding layer for better performance.</li><li>Embedding layer is the hero. Model adapting embedding layer performed much better.</li><li>Why use google translate French &gt;English to check if the output French translation is accurate or not? Because I cant read French, I have no other way to verify the accuracy.</li></ol><h3>Full code</h3><p>Here is the <a href="https://github.com/fiyero/Keras_translation_rnn">full code</a>, enjoy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a37439005ae8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a LDA model for classification with Gensim]]></title>
            <link>https://medium.com/@patrickhk/build-a-lda-model-for-classification-with-gensim-80ca6343c4b9?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/80ca6343c4b9</guid>
            <category><![CDATA[lda]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 06:47:09 GMT</pubDate>
            <atom:updated>2019-07-11T06:47:09.997Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article is written for summary purpose for my own mini project. Code is provided at the end for your reference. My main purposes are to demonstrate the results and briefly summarize the concept flow to reinforce my learning. I won’t go into so much details about EACH technique I used because there are too MANY well documented tutorials. I don’t want to create another guide by rephrasing and summarizing.</em></p><h3>Objective</h3><p>Our goal is to build a LDA model to classify news into different category/(topic).</p><h3>Dataset</h3><p>We will use the abcnews-date-text.csv provided by udaicty. It contains over 1 million entries of news headline over 15 years. The dataset have two columns, the publish date and headline.</p><h3>Data preprocessing to extract data</h3><p>We use pandas to read the csv and select the first 300000 entries as our dataset instead of using all the 1 million entries. Our model will likely be more accurate if using all entries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*zOvgE6_OptV6d5F9wgWBuA.png" /></figure><p>Then we carry out usual data cleansing, including removing stop words, stemming, lemmatization, turning into lower case..etc after tokenization.</p><pre>def lemmatize_stemming(text):<br>    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos=&#39;v&#39;))</pre><pre>def preprocess(text):<br>    result=[]<br>    for token in gensim.utils.simple_preprocess(text) :<br>        <br>        if token not in stopwords.words(&#39;english&#39;) and len(token) &gt; 3:<br>            <br>            result.append(lemmatize_stemming(token))<br>    return result</pre><p>So our processed corpus will be in this form, each document is a list of token, instead of a raw text string.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*tQfdFssvNou2yebtY5cEGg.png" /></figure><h3>Get the Bag of word dict</h3><p>To build LDA model with Gensim, we need to feed corpus in form of Bag of word dict or tf-idf dict.</p><pre>dictionary = gensim.corpora.Dictionary(processed_docs)</pre><p>We filter our dict to remove key : value pairs with less than 15 occurrence or more than 10% of total number of sample</p><pre>dictionary.filter_extremes(no_below=15, no_above=0.1)</pre><h3>Convert into bag of word format for doc2bow</h3><pre>bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*JoZHtg2m8VydhkSSgUtv7g.png" /></figure><h3>(Optional) Get the td-idf corpus form</h3><pre>tfidf = gensim.models.TfidfModel(bow_corpus)<br>corpus_tfidf = tfidf[bow_corpus]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*mbL4l25V21iGmvAFZy6Qfw.png" /></figure><h3>Run LDA model using BOW with gensim</h3><pre>lda_model = gensim.models.LdaMulticore(bow_corpus, <br>                                       num_topics=10, <br>                                       id2word = dictionary, <br>                                       passes = 2, <br>                                       workers=2)</pre><p>Here I choose num_topics=10, we can write a function to determine the optimal number of the paramter, which will be discussed later.</p><h3>Interpret the result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wBRq5MA0AL0_XUTKA7ODA.png" /></figure><p>I only show part of the result in here. Since we set num_topic=10, the LDA model will classify our data into 10 difference topics.</p><p>Each topic is a combination of keywords and each keyword contributes a certain weight to the topic. For example 0.04*”warn” mean token “warn” contribute to the topic with weight =0.04</p><p>The result will only tell you the integer label of the topic, we have to infer the identity by ourselves. For example topic 1 have keywords “gov, plan, council, water, fund…etc” so it makes sense to guess topic 1 is related to politics.</p><p>We can also run the LDA model with our td-idf corpus, can refer to my github at the end.</p><h3>Evaluation of LDA model</h3><p>One common way is to calculate the topic coherence with c_v</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*RUewNf8M2nf3fxom4kU9ZQ.png" /></figure><h3>Get the optimal num_topic for LDA model</h3><p>write a function to calculate the coherence score with varying num_topics parameter then plot graph with matplotlib</p><pre>def compute_coherence_values(dictionary, corpus, texts, limit, start=2, step=3):</pre><pre>coherence_values = []<br>    model_list = []<br>    for num_topics in range(start, limit, step):<br>        model=gensim.models.LdaMulticore(corpus=corpus, id2word=dictionary, num_topics=num_topics)<br>        model_list.append(model)<br>        coherencemodel = CoherenceModel(model=model, texts=texts, dictionary=dictionary, coherence=&#39;c_v&#39;)<br>        coherence_values.append(coherencemodel.get_coherence())</pre><pre>return model_list, coherence_values</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*eQNTOt8XVbRCzplTVwjv3Q.png" /></figure><p>From the graph we can tell the optimal num_topics maybe around 6 or 7</p><h3>Try the LDA model with unseen data</h3><p>Lets say our testing news have headline “My name is Patrick”, pass the headline to the SAME data processing step and convert it into BOW input then feed into the model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*0goRi9M9BtXTUGe10aFR7A.png" /></figure><p>It seems our LDA model classify our “My name is Patrick” news into the topic of “politics”.</p><p>Why? It is possible many political news headline contain People’ name or title as keyword.</p><p>Anyways this is just a toy LDA model, we can see some keywords in the LDA result are actually fragment instead of complete vocab. For example we can see “charg” and “chang”, which should be “charge” and “change”. This is due to imperfect data processing step.</p><h3>Full code</h3><p>Here is the <a href="https://github.com/fiyero/LDA_gensim/blob/master/LDA%20with%20Gensim_git.ipynb">code</a>, enjoy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80ca6343c4b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part of Speech Tagging with Hidden Markov Models (HMM]]></title>
            <link>https://medium.com/@patrickhk/part-of-speech-tagging-with-hidden-markov-models-hmm-4224cb72e9b9?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/4224cb72e9b9</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[hidden-markov-models]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 14:20:09 GMT</pubDate>
            <atom:updated>2019-07-10T14:20:09.989Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part of Speech Tagging with Hidden Markov Models (HMM)</h3><p>This is the first project of my udacity NLP nanodegree. Honestly I think Hidden Markov Models is no longer important in NLP as now we have recurrent neural network. However for the task of part of speech tagging Hidden Markov Models is still practical and easy to train.</p><p>Udacity provide us with several helper functions so is much easier to implement. Lets see how to build most frequency class(MFC) tagger and Hidden Markov Models tagger(HMM).</p><h3>Dataset</h3><p>We will use the Brown Corpus from the NLTK library</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*R30IWDsz15yQAZgA7smVoA.png" /></figure><p>Each sentence starts with a unique identifier on the first line, then followed by one tab-separated word/tag pair on each following line. (b100–5507 is the unique identifier.)</p><p>With the helper function from udacity we can easily build the dataset from the raw txt files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*dPh0lVvwlRzKXiam8A-gxA.png" /></figure><p>Just some stat about our corpus</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*MyITHwNQCqQuzghudrA5tw.png" /></figure><h3>Extract the pos tag and word token data</h3><pre>tags = [tag for word, tag in data.stream()]<br>words = [word for word, tag in data.stream()]</pre><h3>Build the most frequent class tagger (MFC)</h3><p>First define the model class</p><pre>from collections import namedtuple</pre><pre>FakeState = namedtuple(&quot;FakeState&quot;, &quot;name&quot;)</pre><pre>class MFCTagger:<br>    missing = FakeState(name=&quot;&lt;MISSING&gt;&quot;)<br>    <br>    def __init__(self, table):<br>        self.table = defaultdict(lambda: MFCTagger.missing)<br>        self.table.update({word: FakeState(name=tag) for word, tag in table.items()})<br>        <br>    def viterbi(self, seq):<br>        &quot;&quot;&quot;This method simplifies predictions by matching the Pomegranate viterbi() interface&quot;&quot;&quot;<br>        return 0., list(enumerate([&quot;&lt;start&gt;&quot;] + [self.table[w] for w in seq] + [&quot;&lt;end&gt;&quot;]))</pre><pre>mfc_table = {}</pre><p>Write a function to get the frequency count of pos in form of {token:{pos:frequency}}</p><pre>def pair_counts(sequences_A, sequences_B):</pre><pre>dcount = defaultdict(dict) <br>    <br>    d = dict(Counter(list(zip(sequences_A, sequences_B))))<br>    <br>    for key,value in d.items():<br>        dcount[key[0]][key[1]] = value<br>    return dcount</pre><pre>word_counts = pair_counts(words, tags)</pre><p>Get the most frequency count table and build the model</p><pre>for word in data.training_set.vocab:<br>    <br>    mfc_table[word] = max(word_counts[word], key=word_counts[word].get)</pre><pre>mfc_model = MFCTagger(mfc_table)</pre><h3>Apply the model and decode the sequence</h3><pre>for key in data.testing_set.keys[:3]:<br>    print(&quot;Sentence Key: {}\n&quot;.format(key))<br>    print(&quot;Predicted labels:\n-----------------&quot;)<br>    print(simplify_decoding(data.sentences[key].words, mfc_model))<br>    <br>    print()<br>    print(&quot;Actual labels:\n--------------&quot;)<br>    print(data.sentences[key].tags)<br>    print(&quot;\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*SRrky4U_0JWbfCyW3HE9pg.png" /></figure><h3>Evaluate the accuracy of the MFC tagger</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*_vz9sZh-L9n8L5JE87BvJA.png" /></figure><h3>Build the Hidden Markov Models tagger(HMM)</h3><p>this time we will use bi-gram instead of the most frequent count to build the model.</p><pre>import itertools<br>def pairwise(iterable):<br>    t, t_1 = itertools.tee(iterable)<br>    next(t_1, &#39;end&#39;)<br>    return zip(t, t_1)</pre><pre>def bigram_counts(sequences):<br>    return(dict(Counter(pairwise(sequences))))</pre><pre>tag_bigrams = bigram_counts(tags)</pre><p>Count the frequency of each pos as the start sequence:</p><pre>def starting_counts(sequences):</pre><pre>return(dict(Counter([i[0] for i in sequences])))</pre><pre>tags_label= data.Y<br>tag_starts = starting_counts(tags_label)</pre><p>Count the frequency of each pos as the end sequence:</p><pre>def ending_counts(sequences):<br>    return(dict(Counter([i[-1] for i in sequences])))</pre><pre>tag_ends = ending_counts(tags_label)</pre><p>Build the HMM model by using the Pomegranate library</p><pre>from pomegranate import State, HiddenMarkovModel, DiscreteDistribution</pre><pre>basic_model = HiddenMarkovModel(name=&quot;base-hmm-tagger&quot;)</pre><pre>tags = [tag for _, tag in data.stream()]<br>words = [word for word, _ in data.stream()]<br>emission_counts = pair_counts(tags, words)<br>states = {}</pre><pre>for pos_tag in data.tagset:<br>    emission_probabilities = dict()<br>    <br>    for word, occurance in emission_counts[pos_tag].items(): <br>        emission_probabilities[word] = occurance / tag_unigrams[pos_tag]</pre><pre>tag_distribution = DiscreteDistribution(emission_probabilities) <br>    state = State(tag_distribution, name=pos_tag)<br>    states[pos_tag] = state<br>    basic_model.add_state(state)</pre><pre>for pos_tag in data.tagset:<br>    <br>    state = states[pos_tag]<br>    <br>    tags_label= data.Y<br>    tag_starts = starting_counts(tags_label)<br>    tag_ends = ending_counts(tags_label)<br>    start_probability = tag_starts[pos_tag] / sum(tag_starts.values())<br>    basic_model.add_transition(basic_model.start, state, start_probability)<br>    end_probability = tag_ends[pos_tag] / sum(tag_ends.values())<br>    <br>    basic_model.add_transition(state, basic_model.end, end_probability)</pre><pre>for tag_1 in data.tagset:<br>    <br>    state_1 = states[tag_1]<br>    <br>    sum_of_probabilities = 0<br>    <br>    for tag_2 in data.tagset:<br>        state_2 = states[tag_2]<br>        bigram = (tag_1, tag_2)<br>        transition_probability = tag_bigrams[bigram] / tag_unigrams[tag_1]<br>        sum_of_probabilities += transition_probability<br>        basic_model.add_transition(state_1, state_2, transition_probability)    <br>    <br>basic_model.bake()</pre><h3>Apply the model and decode the sequence</h3><pre>for key in data.testing_set.keys[:3]:<br>    print(&quot;Sentence Key: {}\n&quot;.format(key))<br>    print(&quot;Predicted labels:\n-----------------&quot;)<br>    print(simplify_decoding(data.sentences[key].words, basic_model))<br>    print()<br>    print(&quot;Actual labels:\n--------------&quot;)<br>    print(data.sentences[key].tags)<br>    print(&quot;\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*QEiU7nMU2Urc-EPGTaqqmw.png" /></figure><h3>Evaluate the accuracy of the HMM tagger</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*aske8GY2_wlBCJTv1C1-MA.png" /></figure><p>The accuracy of our HMM model is slightly better than our MFC.</p><p>Honestly I feel horrible when I was doing this project because I myself prefer using a language model to do the same task.</p><h3>Full code</h3><p>Here is the <a href="https://github.com/fiyero/HMM_part_of_speech_tag/blob/master/Part%20of%20Speech%20Tagging%20with%20Hidden%20Markov%C2%A0Models_git.ipynb">full code</a>, enjoy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4224cb72e9b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Naive bayes to create a simple Email Spam detector]]></title>
            <link>https://medium.com/@patrickhk/use-naive-bayes-to-create-a-simple-email-spam-detector-72031ec2866?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/72031ec2866</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spam-detector]]></category>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 07:56:21 GMT</pubDate>
            <atom:updated>2019-07-10T07:56:21.737Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This article is written for summary purpose for my own mini project. Code is provided at the end for your reference. My main purposes are to demonstrate the results and briefly summarize the concept flow to reinforce my learning. I won’t go into so much details about EACH technique I used because there are too MANY well documented tutorials. I don’t want to create a guide by rephrasing and summarizing their masterpieces without real added values.</em></p><h3>Dataset</h3><p>We use the SMS SpamCollection dataset from the UCI Machine Learning repository</p><p>There are 2 column, one is label(either ham or spam) and the other is the sms content.</p><pre>import pandas as pd<br>df = pd.read_csv(&#39;SMSSpamCollection&#39;,sep=&#39;\t&#39;, header=None,names=[&#39;label&#39;, &#39;sms_message&#39;])<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*s2ad_v5KmPFmkbJAmx_BKg.png" /></figure><h3>Data Preprocessing</h3><p>we should turn the label into integer, SMS content into Bag of word representation so our model can read</p><pre>df[&#39;label&#39;] = df.label.map({&#39;ham&#39;:0, &#39;spam&#39;:1})<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*9vnh0hWmnLhHIIOwW9qJ0g.png" /></figure><p>We will make use of the sklearn Counter Vectorizer</p><pre>from sklearn.model_selection import train_test_split</pre><pre>X_train, X_test, y_train, y_test = train_test_split(df[&#39;sms_message&#39;],df[&#39;label&#39;])<br>from sklearn.feature_extraction.text import CountVectorizer<br>count_vector = CountVectorizer()</pre><pre>training_data = count_vector.fit_transform(X_train)<br><br>testing_data = count_vector.transform(X_test)</pre><h3>Build the Naive Bayes classifier</h3><p>We use multinomial Naive Bayes from sklearn to fit our transformed data</p><pre>from sklearn.naive_bayes import MultinomialNB<br>naive_bayes = MultinomialNB()<br>naive_bayes.fit(training_data, y_train)</pre><h3>Prediction and model evaluation</h3><pre>predictions = naive_bayes.predict(testing_data)</pre><pre>from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score</pre><pre>print(&#39;Accuracy score: &#39;, format(accuracy_score(y_test, predictions)))print(&#39;Precision score: &#39;, f<br>ormat(precision_score(y_test, predictions)))<br>print(&#39;Recall score: &#39;, format(recall_score(y_test, predictions)))<br>print(&#39;F1 score: &#39;, format(f1_score(y_test, predictions)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*O9Lreo9ArrFqD9mddnLzKw.png" /></figure><h3>Use our trained model to test our custom sentence</h3><p>We will feed the custom sentence “you win this big prize!” to try our model. Remember we have to first convert our input sentence into array instead of pass string directly.</p><pre>import numpy as np<br>test_string=np.array([&#39;you win this big prize!&#39;])</pre><pre>test_string_data = count_vector.transform(test_string)</pre><pre>naive_bayes.predict(test_string_data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*6RgOmJBgYlHYc39AUmVJrQ.png" /></figure><p>We can output as label 1. Back to the pre-processing step, we turn all spam label into 1 while ham into 0, so the custom sentence “you win this big prize!” is classified as spam from our model.</p><h3>Full code</h3><p>Here is the <a href="https://github.com/fiyero/My_NLP_MachineLearning_AI_Trading_repo/blob/master/Spam_classifier_bayes_git.ipynb">full code</a>, enjoy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72031ec2866" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph SLAM for landmark detection and tracking]]></title>
            <link>https://medium.com/@patrickhk/graph-slam-for-landmark-detection-and-tracking-49f270e47c2a?source=rss-535b63fbab82------2</link>
            <guid isPermaLink="false">https://medium.com/p/49f270e47c2a</guid>
            <category><![CDATA[aml]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Leung]]></dc:creator>
            <pubDate>Fri, 21 Jun 2019 23:07:21 GMT</pubDate>
            <atom:updated>2019-06-21T23:08:18.493Z</atom:updated>
            <content:encoded><![CDATA[<p>I have enrolled the udacity computer vision nanodegree and the last project is to build SLAM for tracking and localization.</p><h3>Define a robot class</h3><p>First we have to define a robot class and visualize the robot in a 2D world. Since movement and measurement both have uncertainty, we add motion noise and measurement noise in the motion and sense function.</p><p>The initial pose for the robot was (5,5) in the x,y coordinate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*luJCbUMoOUNXrfVLot4Xqg.png" /><figcaption>initial pose</figcaption></figure><p>After movement of (1,1), the new pose is (5.81287, 5.91232) instead of (6,6) due to uncertainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*u4aSK2QpSpYxjFwyRupovg.png" /></figure><h3>Implement constraints Omega and Xi</h3><p>This is a 2D world so each pose and landmark have x and y component</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*OMZfaWkInj42JII8CN50IA.png" /></figure><p>In the project I use 20 timesteps and 5 landmark locations so I will have 50 constraints and initialize the Omega as a (50,50) matrix with all 0 values.</p><p>Xi will be a row vector of (50,)</p><p>I set the world size as 100 and we assume the robot’s starting position is in the middle of the world with 0 uncertainty. Therefore starting pose is (world_size/2,world_size/2) = (50,50)</p><h3>Implement Graph SLAM</h3><p>We have to update the omega and Xi with motion and measurements data over timesteps. We can introduce variable called alpha and beta to represent the noise in motion and measurement. Remember this is 2D and we have to update in both x and y directions. This is the most challenging part of the project and it will be easier to consider in a simpler constraint example.</p><p>The whole point is to get the best estimate of poses and landmark positions from Omega_inverse * Xi</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*uksj0RDUdRcG4Hmdh1kgQg.png" /></figure><h3>Visualize</h3><p>As a result with updating the measurement and motion data, we can estimate where the robot is after considering the uncertainty factor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*eQP8Bu8JpR672DRjzJyecw.png" /><figcaption>red dot is robot current pose, cross is landmark</figcaption></figure><h3>Full code</h3><p>Is a bit hard to explain the flow as most of the time is spent on the updating the omega and Xi. Will update the step by step concept flow for how to updating the Omega and Xi later</p><p>Here is the <a href="https://github.com/fiyero/SLAM_a">full code</a>, enjoy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49f270e47c2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>