<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jong Shin on Medium]]></title>
        <description><![CDATA[Stories by Jong Shin on Medium]]></description>
        <link>https://medium.com/@jshinm?source=rss-41eee1f80629------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NG0c9r-5zKyp4BE01Ak-qA.jpeg</url>
            <title>Stories by Jong Shin on Medium</title>
            <link>https://medium.com/@jshinm?source=rss-41eee1f80629------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:44:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jshinm/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GPT-4 Turbo vs Gemini vs Open Source Community]]></title>
            <link>https://medium.com/ai-ml-then-more/gpt-4-turbo-vs-gemini-vs-open-source-community-9af21450453d?source=rss-41eee1f80629------2</link>
            <guid isPermaLink="false">https://medium.com/p/9af21450453d</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chatgpt4]]></category>
            <dc:creator><![CDATA[Jong Shin]]></dc:creator>
            <pubDate>Thu, 07 Dec 2023 16:02:03 GMT</pubDate>
            <atom:updated>2023-12-07T16:02:03.149Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ow6mew8LAqe6OG96SIa5rg.png" /><figcaption>Google’s publication on their Gemini model performance. It outperforms current GPT-4 (not GPT-4 Turbo which is not out yet) AND human experts. [1]</figcaption></figure><p>It seems like this quarter’s AI sprint is getting close to the end! A few days ago, we heard from Microsoft about <strong>GPT-4 Turbo</strong>, <strong>DALL-E 3</strong>, multi-modality with <strong>Search Grounding</strong> and more. Yesterday Google announced Gemini, and it is using the current GPT-4 specs for its multi-modal performance benchmarking, which seems to outperform on all benchmarking datasets (except for HellaSwag for general reasoning). Which is indeed exciting!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2M1JkikNWtZmHaAK.gif" /><figcaption>Benchmarking for text. Except for one benchmark dataset, Gemini outperforms GPT-4 overall. [2]</figcaption></figure><p>Now, I wonder how much better <strong>GPT-4 Turbo </strong>is going to be in comparison to the current GPT-4 that Gemini seems to surpass. It would not only depend on how well these models perform on these benchmarking datasets, but things like costs, context window, speed, availability, accessibility, and so on. Either way, it seems like it would not take that long for us to get a Jarvis of our own on our phone, or a pin, or maybe glasses, or even inside of your brain as an implant!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*vpxJI8hNlkgNmNlC.png" /><figcaption>NeurIPS 2023 happening in New Orleans, LA between 9th and 16th</figcaption></figure><p>The tech report [3] for Gemini looks also quite exciting. It would be a great read on my way to NeurIPS 2023 in New Orleans this Saturday (if anyone else is going!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*kSsPWOvDdYN5eAN8AJrgVQ.png" /><figcaption>Figure 2 from the tech report on Gemini. Common ingestion pipeline tokenizes multi-modal data into a single input [3]</figcaption></figure><p>In particular, this figure 2 sticks out to me. I would love to see how they ETL the data from different sources into a single funnel. It seems like they somehow become concatenated into a long sequences of tokens. And, I would love to learn more about that process!</p><p>This is a <a href="https://www.youtube.com/watch?v=jV1vkHv4zq8">brief video</a> [4] from Google on introducing Gemini and some of its functionalities</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*ZDMuF5-Db_ysLpD7T-E5nw.png" /><figcaption>7B Open Source LLM apparently outperforming Grok a 70B model [5]</figcaption></figure><p>Recently, I had some free time deploy OpenChat model locally on my laptop and ran it on CPU. It took about 3–4 mins for each 400–500 response tokens, but if I have the time to wait for it to spit out all the responses in sequence, the quality is worth the wait especially for a 7B model. If you are curious, the reason why I am running the model on CPU is that the model is 24GB, and my GTX3070 has only 8GB. If you have a several GPUs, you can actually parallelize the process by adding this flag --tensor-parallel-size N to your command line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*4w0mfZZkC6JemO1zgl8o6w.png" /><figcaption>Raw form of the response. I just wanted to read something about variance that day, so I asked it to write a bed time story for me instead of putting some effort to do an actual search for it</figcaption></figure><p>My thoughts on GPT-4 Turbo, Gemini and all other upcoming SOTA models is that Open Source community will soon catch on and make something equivalent if not better. For that, I think this open research movement has a non-trivial value not just in pushing boundary of AI research and democratizing AI capabilities but also to enforce ethics and guidance around proper AI practice.</p><p>With all these advancements in AI, I am hoping to get myself a couple brand new GPUs and start playing around different models myself. I could certainly just buy a subscription and use the API, but my ego as a ML engineer/scientist don’t make that a simple decision (plus the charge by token and cloud compute cost can add up).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*bjS-0_fI3vaJrT8S.jpg" /><figcaption>Historical timeline around AI booms and winters. [5] (Credit: Actuaries Digital)</figcaption></figure><p>Anyways, I hope everyone is as hype up as I am about this unprecedented advance in machine intelligence. Not too long ago, people were skeptical about very idea of the neural network, and consequently we all went through the AI winter(s). Now it’s a Spring again, and not just any Spring, but Miami Spring where all four seasons are warm and hot! I am carefully optimistic about where we are heading, and hopeful that we will all live better life at the end.</p><p>Please feel free to leave any comments to share your thoughts! Thank you for reading, and as always, enjoy your AI journey!</p><p><strong>Reference</strong></p><p>[1] <a href="https://deepmind.google/technologies/gemini/#capabilities">https://deepmind.google/technologies/gemini/#capabilities</a></p><p>[2] <a href="https://blog.google/technology/ai/google-gemini-ai/?utm_source=gdm&amp;utm_medium=referral#capabilities">https://blog.google/technology/ai/google-gemini-ai/?utm_source=gdm&amp;utm_medium=referral#capabilities</a></p><p>[3] <a href="https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf">Gemini: A Family of Highly Capable Multimodal Models</a></p><p>[4] <a href="https://www.youtube.com/watch?v=jV1vkHv4zq8">https://www.youtube.com/watch?v=jV1vkHv4zq8</a></p><p>[5] <a href="https://www.actuaries.digital/2018/09/05/history-of-ai-winters/">https://www.actuaries.digital/2018/09/05/history-of-ai-winters/</a></p><p>If you want to read more about AI &amp; ML subjects and more, please come check out my Medium articles!</p><p><a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a></p><p>or visit me in different communities!</p><pre>def visit_me_elsewhere():<br>  Website = &#39;https://modrev.org/jshinm&#39; #personal website<br>  Linkedin = &#39;https://www.linkedin.com/in/jshinm/&#39; #Linkedin<br>  Twitter = &#39;https://twitter.com/JongMShin&#39; #aka X</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9af21450453d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-then-more/gpt-4-turbo-vs-gemini-vs-open-source-community-9af21450453d">GPT-4 Turbo vs Gemini vs Open Source Community</a> was originally published in <a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating in-house audio transcription pipeline that could save millions]]></title>
            <link>https://medium.com/ai-ml-then-more/creating-in-house-audio-transcription-pipeline-that-could-save-millions-eea221442427?source=rss-41eee1f80629------2</link>
            <guid isPermaLink="false">https://medium.com/p/eea221442427</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[whisper]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jong Shin]]></dc:creator>
            <pubDate>Mon, 04 Dec 2023 04:11:40 GMT</pubDate>
            <atom:updated>2024-06-13T02:12:29.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zMmvGpYuz6q2BZRt.png" /><figcaption>In this article, this figure pretty much wraps up what I did for MLOps deployment at high level (Credit: <a href="https://www.c-sharpcorner.com/blogs/mlops">https://www.c-sharpcorner.com/blogs/mlops</a>)</figcaption></figure><p>Last few years, I worked on a number of different AI/ML projects for work and also for my personal inquisitiveness. I am sharing one of those my experiences here that started as rather experimental at first, but later became a customer-facing NLP project, more specifically, transcribing audio calls.</p><p>There are 3rd party vendors that provide transcripts and charge millions of dollars annually. Despite the hefty price tag, some of the quality is subpar.</p><p>So as a ML professional, I got started on a journey to create my own transcription but for a better quality. From a brief research, I realized that there was an open source model called Whisper [1], which was not very well-known back then.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D_po1DkvNGTtHvql-lbAOA.jpeg" /><figcaption>Simple diagram of data workflow. API call is made to the internal API, then on-prem send parquet files to ADLS storage. Databricks is already connected to ADLS through Unity Catalog, so Workflow pipeline unions the data.</figcaption></figure><p>The only problem was that using these models were computationally expensive let alone processing the entire 40,000 calls each day. The company I built this for used <strong>Azure Databricks</strong> as a main platform, and building a cloud solution out of DB GPU clusters was not an economical solution. Even the reserved instances in Azure was too much of the cost for this “experimental” project. I had limited options.</p><p>So as I always do, I started coffee chats with my wonderful colleagues who has been with the company for 5–10 years. We talked about my current conundrum, my favorite place for dining, my friends’ romantic relationship, their preference between dog and cat. Then, I found out that the company had a retired on-premise server that no one has been using for 4 years since the last computer vision project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUDfWL2uw5UA2nV5PMOI-Q.png" /><figcaption>Much more cost effective than using external compute source. Much more so if you have a dedicated server room</figcaption></figure><p>It did not take me too long before I could ssh into this on-premise server that is equipped with 8 GTX 1080 TIs. A little outdated, but definitely workable! After going through arduous process of setting up the environment properly, I was finally able to find all my 8 CUDAs (the server was not in a useable state, so I had to rebuild the system). And then, I wrote a basic pipeline with Streamlit interface to transcribe a set of given calls.</p><p>And it worked! At large-v2, the transcription was done quite magically. In fact, the pipeline also had a subsequent topic classifier running on the server, so showcasing whole pipeline to a business unit was a pretty exciting moment for me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e36PPPXOCjynoyJeWeeRhg.jpeg" /><figcaption>Asset flow between two platforms. Compute extensive work is done solely on on-premise server. The output data is consumed in Databricks where our Stakeholders have selective access through Unity Catalog</figcaption></figure><p>Inspired by this proof of concept, our team decided to take this project into production, which turned out to be the beast of its own, technically speaking. First, it was computationally expensive. Second, I had to automated the process through CI/CD that connects our source and landing targets to our on-premise server. Lastly, it had to be reliable. At that point, I was just glad that I enjoyed challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qBbMf-FqjEZ_YYemKVUCBg.jpeg" /><figcaption>What doesn’t kill you makes you stronger. (Credit: Photo by Анна Рыжкова)</figcaption></figure><p>First thing first, I had to parallelize the process across all 8 GPUs in order to have any shot at processing 40k calls a day. I looked into model parallelism [2] first, but my verdict at the time was that it was not worth the time. Mainly because each Whisper model could fit in a single GPU (2 for medium, 4 for any models smaller). Model parallelism apparently boosts 30% in speed [3], and consequently, unless this method provides more than 100%+ performance by sharing a model across multiple GPUs (asynchronous or not), it is theoretically not possible to outperform running all 8 models at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*rakPlR8FjQVErnvyODXYKg.png" /><figcaption>Data vs Model parallelism. Notice that both agents are split into different GPUs as opposed to my method where I am basically creating a new process for each GPU allocation. (Credit: <a href="https://arxiv.org/pdf/2101.03961.pdf">Fedus et al., 2021</a>)</figcaption></figure><p>So, what I decided to do instead was to feed the data across the GPUs. And, I am not talking about Data parallelism [4] or any other sophisticated methods [5]. I am simply parallelizing by multiprocessing [6] (apparently there is a method to do this in <strong>pytorch</strong>, but I haven’t looked into that yet), which worked out nicely. This setup is nice for scaling up as well because you can simply dockerize the pipeline and run it on another machine (by the way, this project took off, and we are now getting another machine!). I could even pass an argument and specify which GPUs to use, so that I can allocate other GPUs for other projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*u3SMI3g8IMo58eGe2DS49g.png" /><figcaption>Priority queue by heap and also represented by array. (Credit: <a href="https://www.geeksforgeeks.org/priority-queue-using-binary-heap">https://www.geeksforgeeks.org/priority-queue-using-binary-heap</a>)</figcaption></figure><p>The critical part about this way of parallelization is making sure that your data are fed equally. On that note, I built a load balancer for data distribution. At first, I simply sorted the file by size, and then distributed in order. Downside of this is that, a longer file doesn’t necessary take X more times than the file that is 1/X th the size. So to get around that, I used a <strong>priority queue</strong> implemented using heap (ordered binary tree). Then I used this to split a list of files in equal parts by their size. This, in fact, optimized the process quite a lot especially more so at large scale when the distribution of the file size is practically the same in each bin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8ueVpaoZozovgrJY.png" /><figcaption>Diagram for WhisperX pipeline. Notice that followed by VAE, we have custom cut and merge action that optimizes time by ignoring silence. [8]</figcaption></figure><p>Unfortunately, however, with the plain Whisper model by OpenAI was not as performant as I wished it to be. As a rough estimate, I could hit less than 50% load with medium. Thankfully, there were new implementations of Whisper models, namely, <strong>faster-whisper </strong>[7]<strong> </strong>and <strong>WhisperX</strong> [8]. Faster-whisper was indeed much faster, and my team and I could not find any variance from the transcription output. But still, this was not enough. I could hit 80% with the medium model, but I just could not get more than that.</p><p>This was where Whisper-X came in. Surprisingly enough, this implementation allowed me to hit over 100% with the medium (an operational threshold), and hit about 80% with large-v2. Not only that this could accommodate my <strong>custom diarization</strong> module for speaker identification (not through model inference, but by simply splitting audio channels) because it can ignore the silent portion of the audio. I was thrilled!</p><p>This model output was then pushed to an ADLS storage as parquet files after which a Workflow on Databricks is ran on schedule to merge them to a table in Unity Catalog. The last time I checked the pipeline, it had been ongoing for 1.5 months without failing once!</p><p>There are other components that I had to build to get this pipeline to work such as <strong>data ingestion, data trafficking, data cleaning</strong>, which I will discuss in near future. The part that attributed to the success of this project is in large part due to the modularization of each component. Primarily, this design was to parallelize different modules by utilizing CPUs while processing files on GPU. In addition, this allows for not only a better monitoring/troubleshooting workflow but also more efficient development process in my opinion. Also, in the future, if other people wants to collaborate on this project, the whole project codebase is more readable if I break the codebase down into its functional part.</p><p>Obviously, there are much things I can improve on especially with a new whisper release by OpenAI (<strong>Whisper-v3</strong>) [1]. I also heard good things about <strong>insanely-fast-whisper</strong> [9], so I should definitely try that. I should also look into <strong>DeepSpeed </strong>implementation into the pipeline. And as I mentioned above, I should also look into the <strong>pytorch version of multiprocessing</strong>.</p><p>Enjoy your AI journey, and please let me know if you have any questions!</p><p><strong>References</strong></p><p>[1] <a href="https://github.com/openai/whisper">https://github.com/openai/whisper</a></p><p>[2] <a href="https://huggingface.co/transformers/v4.9.2/parallelism.html">https://huggingface.co/transformers/v4.9.2/parallelism.html</a></p><p>[3] <a href="https://arxiv.org/pdf/1802.09941.pdf">https://arxiv.org/pdf/1802.09941.pdf</a></p><p>[4] <a href="https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/data-parallelism-task-parallel-library">https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/data-parallelism-task-parallel-library</a></p><p>[5] <a href="https://arxiv.org/pdf/1802.09941.pdf">https://arxiv.org/pdf/1802.09941.pdf</a></p><p>[6] <a href="https://docs.python.org/3/library/multiprocessing.html">https://docs.python.org/3/library/multiprocessing.html</a></p><p>[7] <a href="https://github.com/SYSTRAN/faster-whisper">https://github.com/SYSTRAN/faster-whisper</a></p><p>[8] <a href="https://github.com/m-bain/whisperX">https://github.com/m-bain/whisperX</a></p><p>[9] <a href="https://github.com/Vaibhavs10/insanely-fast-whisper">https://github.com/Vaibhavs10/insanely-fast-whisper</a></p><p>If you want to read more about AI &amp; ML subjects and more, please come check out my Medium articles!</p><p><a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a></p><p>or visit me in different communities!</p><pre>def visit_me_elsewhere():<br>  Website = &#39;https://modrev.org/jshinm&#39; #personal website<br>  Linkedin = &#39;https://www.linkedin.com/in/jshinm/&#39; #Linkedin<br>  Twitter = &#39;https://twitter.com/JongMShin&#39; #aka X</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eea221442427" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-then-more/creating-in-house-audio-transcription-pipeline-that-could-save-millions-eea221442427">Creating in-house audio transcription pipeline that could save millions</a> was originally published in <a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I am building an internal chatGPT interface for a company]]></title>
            <link>https://medium.com/ai-ml-then-more/how-i-am-building-an-internal-chatgpt-interface-for-my-company-0280171d8919?source=rss-41eee1f80629------2</link>
            <guid isPermaLink="false">https://medium.com/p/0280171d8919</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Jong Shin]]></dc:creator>
            <pubDate>Thu, 30 Nov 2023 09:14:32 GMT</pubDate>
            <atom:updated>2024-06-13T01:40:12.276Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XpJi7co1keQfVstipjyxJg.jpeg" /><figcaption>I can’t show you the actual development for security purpose, but the idea is similar. Credit: Photo by Airam Dato-on: <a href="https://www.pexels.com/photo/a-laptop-sitting-on-a-granite-countertop-15940010/">https://www.pexels.com/photo/a-laptop-sitting-on-a-granite-countertop-15940010/</a></figcaption></figure><p>Have you guys tried out the <strong>inference table</strong> in #databricks? I have recently come across this new functionality in #databricks.</p><p>Essentially it can log input and output being passed through #mlflow Serving Endpoint as a table in #unitycatalog</p><p>This is a great solution for me because I am currently developing an internal #chatgpt interface for a company. With this internal interface, we are able to censor all user prompts by any rules we set, log I/O that are only stored in local cache to be compliant with #infosec, and introduce extra functionalities using model endpoint (i.e., multi-modality, extra security measure, data integration, prompt flow, etc).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*Gat4-NTWAH9LNVeQRN78Xg.png" /><figcaption>Inference table connectivity with MLflow Serving Endpoint [1]</figcaption></figure><p>My current setup for this #mlops project is to use VNet controlled private #azure #openai model endpoint from the #frontend infrastructure built in #reactjs. Then a few of the regex-based censorship rules are triggered from the frontend client side. Filtered input is then passed through the custom #python function served by #mlflow Serving Endpoint in #databricks, where this custom function not only logs I/O through this inference table, but also allows for extra functionalities such as pdf parsing via #langchain. [2]</p><p>If you are interested in learning more about this new functionality that is now in public preview, here is the reference from #databricks</p><p>Happy Hunting my fellow ML Engineers!</p><p>[1] <a href="https://docs.databricks.com/en/machine-learning/model-serving/inference-tables.html.">https://docs.databricks.com/en/machine-learning/model-serving/inference-tables.html</a></p><p>[2] <a href="https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf">https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf</a></p><p>If you want to read more about AI &amp; ML subjects and more, please come check out my Medium articles!</p><p><a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a></p><p>or visit me in different communities!</p><pre>def visit_me_elsewhere():<br>  #visit me in Linkedin community<br>  Linkedin = &#39;https://www.linkedin.com/in/jshinm/&#39;<br><br>  #visit me in my personal website<br>  Website = &#39;https://modrev.org/jshinm&#39;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0280171d8919" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-then-more/how-i-am-building-an-internal-chatgpt-interface-for-my-company-0280171d8919">How I am building an internal chatGPT interface for a company</a> was originally published in <a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where did bias term go in these small GPT models?]]></title>
            <link>https://medium.com/ai-ml-then-more/where-did-bias-term-go-in-these-small-gpt-models-fdb061f93639?source=rss-41eee1f80629------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdb061f93639</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[layer-normalization]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[batch-normalization]]></category>
            <dc:creator><![CDATA[Jong Shin]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 06:47:17 GMT</pubDate>
            <atom:updated>2023-11-28T06:47:17.476Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/904/0*M9UuIJn_YRZlHO9h.png" /><figcaption>BN vs LN. Credit: <a href="https://paperswithcode.com/method/layer-normalization">https://paperswithcode.com/method/layer-normalization</a></figcaption></figure><p>Did you guys know that some of the small to medium #gpt models (e.g., nanoGPT) don’t use bias term?</p><p>The bias term is important because it provides additional degree of freedom to the model and thus better fit the model to the data.</p><p>For instance, imagine a simple linear regression with the simplest affine function which passes through at origin. If there is no bias term, it will always pass through the origin leading to a poor fit.</p><p>Then why are these #gpt models able to perform well without these bias terms? These terms certainly adds computational complexity, but for the right value especially for #nlp tasks where inputs are variable.</p><p>Turns out, Layer Normalization (LN) compensates for the absence of a bias term. It’s the concept first introduced by #geoffreyhinton and his team in this paper in 2016 [1]. As per the paper, LN was designed to overcome the drawbacks of Batch Normalization (BN) where the expectation is calculated by passing through the entire training dataset generally making it computationally impractical, thus this relies on parameter estimation through mini-batch. Then this puts constrains on the size of a mini-batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*8sZrqB9VGijOrZQ70KpUxg.png" /><figcaption>Eq. 2 of the paper. Equation for BN. Notice it’s for each i-th hidden unit over whole distribution</figcaption></figure><p>LN instead sums over all the hidden units in the same layer. This is because one layer’s output is its subsequent layer’s input, hence this causes highly correlated changes. Therefore, each hidden layer shares the same mean and variance instead of per batch as you would see in BN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*F1dlOANmUm8-F_40AlP1Qg.png" /><figcaption>Eq. 3 of the paper. Equation for LN. Notice that mean and variance is shared for each hidden unit per layer</figcaption></figure><p>More importantly, this is important in the case where the test sequence is longer than the training because BN depends on separate statistics per time step whereas LN no longer has such problem as it only depends on the summed inputs to a layer at the current time-step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*FXBunG_V6kVewbul.png" /><figcaption>More types of normalization. Credit: <a href="https://theaisummer.com/normalization/">https://theaisummer.com/normalization/</a></figcaption></figure><p>This doesn’t mean LN should completely replace BN. For instance, BN is known to work well with CNN while LN is particularly useful with sequence-based tasks (e.g., NLP). Having these architectural understanding behind #neuralnetworks is helpful to your AI/ML/DS practice!</p><p>[1] <a href="https://arxiv.org/pdf/1607.06450v1.pdf">https://arxiv.org/pdf/1607.06450v1.pdf</a></p><p>[2] <a href="https://paperswithcode.com/method/layer-normalization">https://paperswithcode.com/method/layer-normalization</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdb061f93639" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-then-more/where-did-bias-term-go-in-these-small-gpt-models-fdb061f93639">Where did bias term go in these small GPT models?</a> was originally published in <a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Q* the beginning of doomsday?]]></title>
            <link>https://medium.com/ai-ml-then-more/q-the-beginning-of-doomsday-42c2e2e4ba73?source=rss-41eee1f80629------2</link>
            <guid isPermaLink="false">https://medium.com/p/42c2e2e4ba73</guid>
            <category><![CDATA[sam-altman]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Jong Shin]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 06:21:54 GMT</pubDate>
            <atom:updated>2023-11-27T06:21:54.256Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="depiction of doomsday AI" src="https://cdn-images-1.medium.com/max/1024/1*UhflPbcCvE-MRQFBsUrwOA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@possessedphotography?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Possessed Photography</a> on <a href="https://unsplash.com/photos/white-and-brown-human-robot-illustration-YKW0JjP7rlU?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Have you heard of <a href="https://www.linkedin.com/feed/hashtag/?keywords=openai&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#openai</a>’s Q* (Q star)?</p><p>It’s not like A* like your search algorithm, but it’s apparently a <a href="https://www.linkedin.com/feed/hashtag/?keywords=openai&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#openai</a>’s AGI project that may have attributed to recent drama involving <a href="https://www.linkedin.com/feed/hashtag/?keywords=samaltman&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#samaltman</a>.</p><p>I was quite surprised how <a href="https://www.linkedin.com/feed/hashtag/?keywords=sam&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#sam</a> was transparent about a partnership with <a href="https://www.linkedin.com/feed/hashtag/?keywords=microsoft&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#microsoft</a> on AGI development (18:49 in this vid, <a href="https://lnkd.in/eECuZAhF">https://lnkd.in/eECuZAhF</a>)</p><p>Also apparently some from <a href="https://www.linkedin.com/feed/hashtag/?keywords=openai&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#openai</a> believe that this is a groundbreaking progress towards AGI and it will also “surpass humans in most economically valuable tasks”.</p><p>Should we be worried? Maybe. Am I worried? Not really. Because even if it’s able to outperform some of these “economically valuable tasks,”[1] it’s still us humans who defines what those tasks are. And I’m sure if we are able to automate some of those tasks, we will come up with other tasks that have not been possible without the AI.</p><p>For now, I’m super thrilled to live in the dawn of AGI era, whether it’s smart or not (let alone being perfect or super-intelligent). I just can’t wait to see what future holds for us all 😃!</p><p>Btw, did they use <a href="https://www.linkedin.com/feed/hashtag/?keywords=python&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7134778415739453441">#python</a> for this?</p><p>[1] <a href="https://lnkd.in/eDeShdrf">https://lnkd.in/eDeShdrf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42c2e2e4ba73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-then-more/q-the-beginning-of-doomsday-42c2e2e4ba73">Q* the beginning of doomsday?</a> was originally published in <a href="https://medium.com/ai-ml-then-more">AI &amp; ML then More</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>