<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arjun on Medium]]></title>
        <description><![CDATA[Stories by Arjun on Medium]]></description>
        <link>https://medium.com/@viswanathan.arjun?source=rss-2c22a3e01083------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mc3s80eK6xlykk-Ca56tsA.jpeg</url>
            <title>Stories by Arjun on Medium</title>
            <link>https://medium.com/@viswanathan.arjun?source=rss-2c22a3e01083------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:26:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@viswanathan.arjun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Voice Agents in 2025 — Part I]]></title>
            <link>https://levelup.gitconnected.com/building-voice-agents-in-2025-part-i-8ceb316fe6e7?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ceb316fe6e7</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Wed, 13 Aug 2025 02:12:02 GMT</pubDate>
            <atom:updated>2025-08-13T02:12:02.440Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Voice AI Agents in 2025 — Part I</h3><h4>Your Voice AI Agent Shouldn’t Sound Like a Fax Machine.</h4><p>Ever leaned out your car window, yelling “NO, I SAID <em>DIET</em> COKE!” at a drive-thru speaker?<br>Or been trapped in IVR hell — “Press 3 for billing… no, *3*… PLEASE WAIT WHILE I TRANSFER–”?</p><p>Yeah. We’ve been building “smart” voice agents since forever with:</p><ul><li>Rule-based scripts dumber than a toaster</li><li>“NLP” that heard “no pickles” as “add lice”</li><li>Robotic loops that make callers want to snap their phone in half</li></ul><p><em>But here’s the kicker: </em><strong><em>Humans don’t process inputs.</em></strong><br>We <em>listen</em>, <em>think</em>, <em>act</em> — in real time. No 5-second lag. No clueless “I DIDN’T CATCH THAT.”</p><p>2025’s generative AI? It’s the antidote to garbage bots.<br>Not an upgrade — a demolition. The pipes (like Twilio, holding strong since my ’15 IVR) are ready.<br>Now we weaponize LLMs — open-source or enterprise — to build agents that <em>actually</em> converse.</p><p>No more rage. No more mangled orders. Time to build agents that don’t suck.</p><p>As humans, we listen, we look for information, we think and gather response, we speak. All of this realtime. so its natural we anticipate this from the AI agents that does this.</p><p><em>Sounds simple, right? Then why does building a robust, real-world agent feel like such a monumental challenge? </em>Research shows human expects response on average within 520ms. So its indeed a tough nut to crack.</p><ol><li>🗣️ Ears (ASR / STT + VAD + Utterance Detection): Turning messy, real-world speech into accurate text, knowing <em>when</em> someone starts and stops talking (especially over background noise!).</li><li>🧠 Brain (LLM / LLM Agent): Understanding intent, reasoning, accessing tools/APIs (booking systems, menus, CRM), managing context, and generating a coherent, task-oriented response.</li><li>👄 Voice (TTS / Speech Synthesis): Converting that text response back into natural, human-sounding speech, ideally with appropriate prosody and pacing.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*jJvYy7hPn0bGsqUAI05lxw.png" /></figure><h3>How to Know Your Voice Agent Doesn’t Suck</h3><p><em>(Because “seems fine” isn’t a metric.)</em></p><p>Before shipping, ruthlessly measure:</p><ol><li>🗣️ <strong><em>Speech Understanding:</em></strong><br><em>Does it get “extra pickles” through wind noise?</em><br><em>Can it handle “rosuvastatin” from a healthcare provider without blinking?</em><br>→ Test with: Background noise, accents, train and test with industry jargons.</li><li>👂 <strong><em>Interruption Handling:</em></strong><br><em>Does “WAIT, NO!” derail it?</em><br>→ Pass/Fail: Graceful recovery = good. Ignoring = bad.</li><li>🧠<strong><em> Tool Mastery:</em></strong><br><em>Can it reliably pick &amp; use functions?</em><br>→ Key: Filling complex params (dates, IDs, preferences) without hallucinating.</li><li>⏱️ <strong><em>Latency:</em></strong><br><em>Is it faster than awkward silence?</em><br>→ Target: Under 2 seconds per turn. &gt;5 seconds = user rage.</li><li>💬 <strong><em>Natural Speech (TTS):</em></strong><br><em>Does it sound human-ish?</em><br>→ Avoid: Robotic vibes. (Stephen Hawking’s GPS = fail).</li><li>💰 <strong><em>Cost-Effectiveness:</em></strong><br><em>Can you run it without selling a kidney?</em><br>→ Track: Cost per call. Scale matters.</li></ol><h4>Forget (for now):</h4><ul><li>Voice cloning (“Meh”) — Some industry leading products offer this, its not something I would prefer using.</li><li>50 languages support? (“Nah, English first”)</li></ul><p>Pass this test? Congrats. Your agent might survive contact with humans.</p><h3><em>Let’s explore how to actually build it</em></h3><h4>Option 1: Full DIY (maximum control, maximum pain)</h4><p><em>(“I enjoy pain and Kubernetes yaml nightmares”)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/235/0*RW47H6waU31WjAbb.jpg" /></figure><ul><li><strong><em>How:</em></strong> Pick your own STT (Deepgram), LLM (GPT-4o), TTS (ElevenLabs) or just any model of your choice. Wire them together.</li><li><strong><em>Pros:</em></strong>Maximum control, Cherry-pick “best” models (in theory).</li><li><strong><em>Cons:</em></strong> Latency death spiral (STT → LLM → TTS = 3+ seconds of awkward silence), Engineering hell (streaming, state, error handling = 6+ months of pain).</li><li><strong><em>Verdict: Only if you hate shipping before 2026.</em></strong></li></ul><h4>Option 2: DIY + Glue Framework (sweet spot for enterprises)</h4><p><em>(“I want control without the migraines”)</em></p><p><strong><em>How:</em></strong></p><ul><li>Choose your own stack of models</li><li>Use tools like Dialogflow , Vapi, or Deepgram Voice Agent.</li><li>Plug in your models. They handle real-time glue.</li></ul><p><strong><em>Pros:</em></strong></p><ul><li>Keep model flexibility (swap STT/LLM/TTS)</li><li>Slash dev time (weeks, not years).</li><li>Enterprise-ready (auth, scaling, compliance baked in).</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Framework limits (can’t tweak <em>every</em> nanosecond).</li><li>Costs add up (but still cheaper than 10 engineers).</li></ul><p><strong><em>Verdict:</em></strong> Best for balance — control + sanity.</p><h4>Option 3: Speech-to-Speech APIs (fast but proprietary)</h4><p><em>(“Just </em>work, I’ve got a demo tomorrow”)</p><p><strong><em>How:</em></strong> One API call. Speech in → Speech out. Done.</p><p><strong><em>Who’s killing it:</em></strong></p><ul><li>OpenAI Realtime API: GPT-4o does it all (~500ms latency).</li><li>Amazon Nova Sonic: Emotion-aware, RAG-friendly, scary human.</li><li>Gemini Live API: Multimodal beast (saw that insane bike tour demo?? <a href="https://www.youtube.com/shorts/4Q-YZla8YT0">https://www.youtube.com/shorts/4Q-YZla8YT0</a> 🤯).</li></ul><p><strong><em>Pros:</em></strong></p><ul><li>Latency? Solved. (sub-second responses).</li><li>Launch in days, not months.</li></ul><p><strong><em>Cons:</em></strong></p><ul><li>Black box (can’t swap the LLM’s “brain”).</li><li>Vendor lock-in (pray they don’t jack up prices, with the competitive market, they just can’t!).</li></ul><p><strong><em>Verdict:</em></strong> MVP rocket fuel. Or if you DGAF about customization.</p><h3>Why I’m Betting on Speech-to-Speech APIs</h3><p><em>(Spoiler: It’s Not About the Fancy Architecture Slides)</em></p><p>Yeah, yeah — the <em>official</em> reasons S2S models (like Nova Sonic or OpenAI) win:</p><ul><li>⚡️ Event-Driven? Sure.</li><li>🔄 Bidirectional Streaming? Obviously.</li><li>🧠 Session Memory? Check.</li><li>🚀 Rapid Dev Cycles? Absolutely.</li></ul><p>But let’s cut through the VC-bait buzzwords.</p><ul><li>Windsurf gets aquired by google while they are at their peak(only next to cursor, lol!)</li><li>Thinking Labs best resources were about to be snatched by Mark!</li></ul><p>The <em>real</em> reason I chain myself to S2S giants?<br>I refuse to mourn my stack because a founder chased a golden parachute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*skV6y70BTfD3IBWJ.gif" /></figure><ul><li>Will they jack up prices? Maybe.</li><li>Are they black boxes? Definitely.</li><li>But they’ll still exist in 2026.</li></ul><p>So yeah — I’ll trade “granular control” for not rebuilding my damn voice agent every 18 months. 🔥</p><p>Finally How does the pricing stand??</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*fRNDObPhHez5KRw6.jpg" /></figure><ol><li>Nova Sonic</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyy6vzfuZP0UZu5J1RVeGg.png" /></figure><p>2. Azure OpenAI Realtime Speech</p><p>4o mini is the cheapest option. pricing per 1M token. divide by 1000 to compare with Nova Sonic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jm5t9etYVIJOwiKQNwd9RQ.png" /></figure><p>3. Gemini Live API</p><p>Gemini 2.0 Flash is cheaper, but 2.5 is miles ahead in quality! hence we prefer 2.5 here for this comparison!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IpzU3cdXgtti-_xgVDK7w.png" /></figure><p>Balancing the model capability vs pocket friendliness ? After going through some benchmarks and some testing, My preference is <strong><em>Amazon Nova Sonic.</em></strong></p><p>PS: I never thought I would vote for AWS again after costing an accidental 10K$ bill 8 years back! :-p</p><figure><img alt="My all time favorite AWS Meme" src="https://cdn-images-1.medium.com/max/618/0*EFgNiey4zf0KuThK" /></figure><p>Thats it for part I , lets dive into developing a speech to speech agent using Amazon Nova Sonic in Part II of this article.</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ceb316fe6e7" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/building-voice-agents-in-2025-part-i-8ceb316fe6e7">Building Voice Agents in 2025 — Part I</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fall 2024 with LLM Agents MOOC]]></title>
            <link>https://medium.com/@viswanathan.arjun/fall-2024-with-llm-agents-mooc-e7fed74e3517?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7fed74e3517</guid>
            <category><![CDATA[uc-berkeley]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[autogen]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Wed, 11 Dec 2024 22:15:58 GMT</pubDate>
            <atom:updated>2024-12-11T22:15:58.105Z</atom:updated>
            <content:encoded><![CDATA[<h4>From AutoGen to GR00T — A Reflection from my learning Journey</h4><p>This fall, amidst the vibrant New England foliage and my new meta quest 3s asking to spend time, I embarked on a journey into the world of LLM agents through UC Berkeley’s LLM Agents MOOC. It was an intellectual whirlwind, packed with cutting-edge research and hands-on experience that left me both exhilarated and humbled by the rapid advancements in this field.</p><p>Led by the Professor <a href="https://www.linkedin.com/in/dawn-song-51586033/">Dawn Song</a> and her team of professors at the <a href="https://rdi.berkeley.edu/">Berkeley Center for Responsible Decentralized Intelligence</a> (RDI) team, Each lecture in the course, was a goldmine of information. From foundational concepts like LLM reasoning and the history of agents to the intricacies of agentic AI frameworks like AutoGen and DSPy, the curriculum provided a comprehensive overview of this burgeoning domain. The course wasn’t just theoretical; it delved into the practical applications of AI agents in enterprise workflows, software development, and even robotics, offering a glimpse into the transformative potential of this technology.</p><p><strong>Hands-on Labs</strong></p><p>The lab work was where the magic truly happened. Building multi-agent collaborative systems using AutoGen was a thrilling experience, allowing me to witness firsthand the power of these autonomous entities working in concert. But perhaps the most impactful aspect of the labs was the focus on AI safety and security. We learned to craft both attack and defense prompts, gaining a deep understanding of the vulnerabilities and safeguards necessary in this era of generative AI. As someone working in the life sciences and healthcare sector, the emphasis on protecting PII and PHI data resonated deeply. In a world increasingly reliant on AI, understanding these safety mechanisms is no longer optional; it’s a fundamental requirement.</p><h4>Autogen &amp; Agent Conversation Patterns</h4><p>Lecture 3, from Chi Wang and Jerry Liu explores AutoGen framework as it stands out as vital for developing LLM Agents, unifying diverse entities such as humans and tools, and facilitating flexible multi-agent orchestrations. The importance of multimodal knowledge assistants was emphasized, with robust retrieval pipelines needed to manage text, images, and more.</p><p>Transitioning agentic AI from prototype to production requires careful infrastructure planning, including modeling workflows as service APIs and integrating human feedback systems. Wang posits that agentic AI represents the future of AI, acting as intermediaries in navigating the digital world. This evolution pivots on the integration of multimodal data processing and innovative frameworks, paving the way for intelligent solutions that enhance human-computer collaboration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxpr4lYveuMUKmZbWlEYjQ.png" /><figcaption>Types of Conversational Agent — an image extracted from Lecture 3</figcaption></figure><h4>Project GR00T</h4><p>One of the most captivating topics explored in the course was NVIDIA’s Project GR00T — their ambitious endeavor to build a generalist robot. Jim Fan’s lecture on this project was particularly inspiring. He argued convincingly that the future of robotics lies not in specialized machines but in versatile, adaptable robots capable of learning and performing a wide array of tasks. This resonates with the trajectory of AI development itself, where generalized models like ChatGPT have revolutionized the field.</p><p>GR00T’s core principles — the data pyramid, the matrix principle, and the foundation agent principle — provide a compelling roadmap for achieving this vision. By leveraging diverse data sources, extensive simulations, and advanced neural network architectures, NVIDIA aims to create robots that can seamlessly navigate and interact with the real world. Technologies like RoboCasa, MimicGen, and HOVER showcase the innovative approaches being employed to overcome the challenges like training of embodied AI. The idea that we’re on the cusp of creating “active kittens in silicon” is both exciting and slightly daunting, highlighting the immense potential and responsibility that comes with developing such advanced technologies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aS-aVTYRohLqhEKKtIAAvQ.png" /><figcaption>10 Humanoids with NVIDIA CEO Jensen in the middle— an image extracted from Lecture 9</figcaption></figure><p>Finally the last lecture from Professor Dawn song, perhaps the longest lecture in the series highlights the complex and interconnected challenges of building safe and trustworthy AI agents. It emphasizes the urgent need for collaborative efforts across disciplines and sectors to navigate this critical juncture in the development and deployment of AI technologies. It was extremely interesting to see developments in both the sides of the world —Adversarial attack and defense in AI Safety.</p><p><strong>Final Thoughts</strong></p><p>I’ve found that each lecture in this course is truly worth watching at least twice. With a year of experience in developing LLM Agents, I can confidently say this course strikes a perfect balance between covering advanced concepts and being accessible for beginners. The lineup of speakers is exceptional, and their passion and expertise are evident throughout their presentations, making it all the more inspiring. I highly recommend this course to anyone involved in the LLM and Generative AI fields. Here is the link to get started — <a href="https://llmagents-learning.org/f24">LLM Agents MOOC</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7fed74e3517" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Host your Python App in Droplets — A Simple/No Docker Approach]]></title>
            <link>https://levelup.gitconnected.com/host-your-python-app-in-droplets-a-simple-no-docker-approach-96e6e0c95f01?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/96e6e0c95f01</guid>
            <category><![CDATA[web-server]]></category>
            <category><![CDATA[droplet]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hosting]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Mon, 22 Jul 2024 00:38:04 GMT</pubDate>
            <atom:updated>2024-07-22T00:38:04.472Z</atom:updated>
            <content:encoded><![CDATA[<h3>Host your Python App in Droplets — A Simple/No Docker Approach</h3><p>So, you’ve poured your heart and soul into building that awesome new Python project. It’s been humming along perfectly on your local machine for weeks. Now comes the big moment: sharing it with the world!</p><p><em>You’re looking for a hosting solution that’s straightforward, quick to set up, and won’t have you tangled in complex configurations. Trust me, you’re not alone in wanting a hassle-free hosting experience. Many developers have been right where you are, and I’m excited to share what I’ve learned along the way!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*t_M-er_i2L_BECYVL6a-ng.png" /></figure><p>My go-to platform lately has been Digital Ocean. For solo developers and small teams, it’s both budget-friendly and dependable. While Digital Ocean offers serverless options, I’ve found their Droplets (Ubuntu mini-servers) to be incredibly easy to use. For a mere $6 a month, I’m able to host two applications that effortlessly handle several hundred daily users.</p><p>I’ve built applications using frameworks like FastAPI and Streamlit, but the approach I’m about to outline should work for most Python-based projects.</p><p>While Docker is a fantastic way to package and deploy applications (and my go-to for production systems at work), I’m aiming for a Docker-less solution here. Why? To keep things as straightforward as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*0-jzRNFJBHpuBX61.png" /></figure><p>I’ll be honest, deploying web apps used to make me cringe. As someone who spends most of their time building prototypes and products, I relied heavily on those magical one-click deployment pipelines. Even with all the advancements that have made deployment smoother, it always felt like a tedious chore</p><p>So what are we gonna do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*H9JInU3CUdr39s5e8fKd3A.png" /></figure><blockquote>There is indeed more convenient and reliable approaches than the one suggested in this article ( use github to pull code instead of copying code from local to remote). This article aims to be make the hosting part simpler/faster for developers.</blockquote><ol><li>Login to Remote environment</li></ol><p>This article assumes you already have a droplet created. If not, head straight to <a href="https://docs.digitalocean.com/products/droplets/how-to/create/">here</a> and get one created. Make sure to set up either <a href="https://docs.digitalocean.com/products/droplets/how-to/add-ssh-keys/">SSH keys</a> or root account password. Once the droplet is created, we get an ip address</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsRCloGminGyI-r0BU-A6Q.png" /></figure><p>copy it and ssh into terminal</p><pre>ssh root@&lt;ip_address&gt;</pre><p>It might either prompt to password or is SSH keys are set it will log you in directly to the droplet.</p><p>2. Now the tested code has to be moved to the droplet. Copy the code base from local to the droplet</p><pre>scp -d -r /path/to/local/folder/ root@&lt;droplet_ip&gt;:/path/to/remote</pre><p>3. Activate the venv just like how we do it in local. <a href="https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-ubuntu-20-04-quickstart">here</a> is an helper article.</p><p>4. Once the venv is activated, Install the prerequisites pip install -r requirements.txt assuming all the packages are in the file. poetry users, you already know what to do :-D. Run and test it in droplet before we can move it to next step.</p><p>5. Once we know our app/api is running successfully when we run it manually, then its important to make it availabel to access from web.</p><p>6. Lets install bunch of web servers. why do we do it? the explanation is towards the end.</p><pre>pip install gunicorn</pre><p>gunicorn is a python package. since we are dealing with fastapi app here, it comes with uvicorn. so it need not be installed explicitly.</p><p>7. Run the app using the below command</p><pre>gunicorn -w 4 -k uvicorn.workers.UvicornWorker auth_management_api:app</pre><p>Here my app is developed on the file auth_management_api.py. replace it with the filename where your app is developed.</p><p>we are adding 4 worker nodes of the uvivorn-fastapi app load balanced by gunicorn. you can verify it by running the command netstat -tlnp | grep 8000 change the port as needed. The app should work just fine if the steps are followed diligently.</p><p>8. Enabling HTTPS. this involves 2 steps. generating a self signed certificate and using that certificate in web server.</p><pre>openssl req -x509 -newkey rsa:4096 -days 365 -nodes -out cert.pem -keyout key.pem</pre><p>This will ask you to enter few details and finally will generate 2 files cert.pem and key.pem. store the files under /root/cert/.</p><p>5. Setup Nginx Web serves</p><pre>sudo apt update<br>sudo apt install nginx</pre><p>once nginx is installed, we need to setup <strong><em>sites-available</em></strong> and <strong><em>sites-enabled. </em></strong><em>This will make the nginx listen to a port and redirect to your app.</em></p><p>start with adding sites-enabled vi /etc/nginx/sites-enabled/your-app-name use any editor of your choice :-) i’m using vim here.</p><pre>server{<br>       listen 443 ssl;<br>       server_name &lt;ip_address&gt;;<br>       ssl_certificate /root/cert/cert.pem;<br>       ssl_certificate_key /root/cert/key.pem;<br>       location / {<br>           include proxy_params;<br>           proxy_pass http://127.0.0.1:8000;<br>       }<br>}</pre><p>closely looking at above config, we understand few things</p><ul><li><strong>HTTPS Setup:</strong> We’re instructing Nginx to listen on port 443, which is the standard port for HTTPS traffic. By providing the ssl_certificate and ssl_certificate_key paths, we ensure that HTTPS certificates are correctly loaded — making our customer’s browsing experience safe and secure.</li><li><strong>Traffic Routing:</strong> Finally, we tell Nginx to direct all incoming HTTPS traffic to our Python app running on port 8000. This seamless connection ensures our application is accessible to the world.</li></ul><p>move this config to <strong><em>sites-enabled</em></strong></p><pre>sudo ln -s /etc/nginx/sites-available/your-app-name /etc/nginx/sites-enabled/</pre><p>finally start the nginx</p><pre>sudo systemctl restart nginx</pre><p>Now our app can be served through web over HTTPS.</p><h4><strong>Now, let’s clarify the roles of our key players: Nginx, Gunicorn, and Uvicorn.</strong></h4><p>Think of it this way:</p><ul><li><strong>Nginx:</strong> Nginx is the friendly doorman of our web app. It greets all incoming requests and directs them to the appropriate place.</li><li><strong>Gunicorn:</strong> Gunicorn is the efficient manager, overseeing a team of workers responsible for handling those requests.</li><li><strong>Uvicorn:</strong> Uvicorn is our high-performing worker, specifically designed to run FastAPI applications. It excels at handling asynchronous requests, making it a great fit.</li></ul><p><strong>Why this combination?</strong> You might wonder why Gunicorn acts as the manager while Uvicorn handles the actual work. Although Gunicorn <em>can</em> be a worker itself, this setup, with Gunicorn as a load balancer and Uvicorn as the worker, is widely regarded as a highly optimized approach by web experts.</p><p><strong>Keep in mind:</strong> The web serving and hosting landscape is constantly evolving, and what’s considered optimal today can change over time. However, this setup provides a solid foundation for most FastAPI deployments.</p><p>overall it looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ORaosI_xIEMWfgwjKOkyyg.png" /></figure><p>So far, we’ve looked at how to host an API that handles incoming requests from the internet. But what about those essential background tasks? Many applications, rely on scheduled jobs for things like nightly cleanups, data processing, or sending out reports. These background tasks are often a critical part of keeping a system running smoothly. Let’s see ,how we can do it, again with an intent to keep it extremely simpler.</p><p>A good choice of scheduler is cron jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kngAqOyv1DGKUFTa.png" /></figure><ol><li>Install cron</li></ol><pre>sudo apt install cron</pre><p>2. Enable it to run in the background</p><pre>sudo systemctl enable cron</pre><p>3. Prepare the command to run. this is the command to be executed every hour.</p><pre>#!/bin/bash<br># Set environment variables<br>export SHELL=/bin/bash<br>export PATH=/root/venv2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin<br>export VIRTUAL_ENV=/root/venv2<br># Change to the correct directory<br>cd /root/aj_bot<br># Activate virtual environment and run the script<br>source /root/venv2/bin/activate<br>python3 trade_bot.py &gt;&gt; /root/aj_bot/trade_bot.log 2&gt;&amp;1</pre><p>here we enable the venv, cd to the folder and run the python file.</p><p>3. Fixing a schedule. The pattern of the cronjob scheduler looks like this</p><pre>minute hour day_of_month month day_of_week command_to_run</pre><p>In my case, i have developed a trading bot. Hence i want to run my program on working days between 9:15 to 3:15 on market hours. UTC is 5:30 hours behind IST. to setup cron job crontab -e and add a new line as follows and save the cron file.</p><pre>46 3-9 * * 1-5 /root/aj_bot/run_trade_bot.sh</pre><ul><li>running on 46th minute(in UTC)</li><li>running between 3 AM — 9 AM every hour in UTC</li><li>* for all days in month</li><li>* for all months in a year</li><li>1–5 indicates Monday to Friday</li></ul><blockquote>Make sure to adjust the cron expression timing based on timezone of the Droplet, Generally VMs run on GMT time.</blockquote><p>Look at how we are enabling venv2(Lets just assume i had a brain fade when I named my virtual environment as venv2) first and then running the program. this is because all the libraries are installed only on venv2.</p><p>Restart the cron to activate the changes sudo systemctl restart cron.</p><p>Check if the cron is running as per our schedule from the cron logs. Cron logs by default is available on sys logs. but we can change it to write it on its own file as well. tail -f /var/log/syslog</p><p>Thats it we are done!</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96e6e0c95f01" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/host-your-python-app-in-droplets-a-simple-no-docker-approach-96e6e0c95f01">Host your Python App in Droplets — A Simple/No Docker Approach</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM for Market Analysis — Part VI — LLM Vs Algorithmic  stock screener]]></title>
            <link>https://python.plainenglish.io/llm-for-market-analysis-part-vi-llm-vs-algorithmic-stock-screener-eefc9879bf28?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/eefc9879bf28</guid>
            <category><![CDATA[cohere]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Mon, 06 May 2024 03:12:53 GMT</pubDate>
            <atom:updated>2024-05-07T00:26:46.990Z</atom:updated>
            <content:encoded><![CDATA[<h3>LLM for Market Analysis — Part VI — LLM Vs Algorithmic stock screener</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*GEFN8HEZKuTwiOGXlMeQRQ.png" /></figure><p>As we’re aware, large language models excel in uncovering insights when prompted. However, their true potential emerges when real-world data is supplied for analysis. The greatest advantage lies in their ability to generate outcomes grounded in rationale rather than sentiment (though biases may still persist!).</p><p>As a retail investor, one of the most significant challenges in my investment and trading journey is grappling with attachment or aversion to certain stocks due to biases from past experiences. These biases often hinder my ability to think rationally.</p><p>In this article, our approach diverges from simply prompting the Large Language Model (LLM). Instead, we present real-time data and task the LLM with conducting a rational stock screening exercise. This shift aims to leverage the LLM’s capacity for unbiased analysis, helping investors overcome emotional biases and make more data driven decisions.</p><p>So what data are we going to give to LLMs?</p><ul><li>PE Ratio</li><li>Debt/Equity</li><li>Return over 1 week, 1 month, 3 months, 6 months, 1 year</li><li>Market Cap</li><li>profit growth</li><li>revenue growth</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ph6us6m1QsgSHiV5JXQn8w.png" /></figure><p>While this is only a sample set, some investors may prefer additional factors such as Return on Equity (ROE), Return on Capital Employed (ROCE), and others. Nevertheless, LLMs trained on finance corpora are adept at handling diverse datasets, ensuring comprehensive analysis.</p><p>In this example, we’ll be utilizing Cohere LLM. However, in reality, after conducting assessments of multiple LLMs, I opted for a llama3 model hosted on an AWS compute instance for this task — primarily to manage expenses efficiently.</p><p>For all my LLM assessments, I rely on <a href="https://mlflow.org/docs/latest/llms/llm-evaluate/index.html">mlflow</a> and <a href="https://docs.ragas.io/en/stable/">Ragas</a> These tools streamline the evaluation process and provide invaluable insights into model performance. Additionally, I plan to delve into a separate series dedicated to LLM evaluation in the future.</p><blockquote>What’s your preferred evaluation framework? Share your thoughts in the comments!</blockquote><h4>So, why Cohere?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/0*G6bLmqFiy19PQA6r.png" /></figure><p>Cohere stands out as an enterprise AI platform offering a range of Large Language Models (LLMs), including Command, Rerank, and Embed. What sets Cohere apart is its commitment to user-friendly APIs, designed to simplify consumption. This emphasis on accessibility aligns with the platform’s overarching goal of making AI more approachable.</p><p>Having personally explored various options, I found Cohere’s ecosystem remarkably easy to navigate — a refreshing departure from the complexities often associated with platforms like AWS, GCP, and Azure. Moreover, Cohere’s seamless support for fine-tuning and developing Retrieval-Augmented Generative (RAG) models further solidified my admiration.</p><p>While I’ve utilized self-hosted LLMs on AWS Sagemaker and experimented with different market offerings, Cohere’s intuitive interface and robust features make it an ideal choice for this experiment. Through sharing my experiences, I hope to provide readers with valuable insights into the diverse landscape of LLM platforms.</p><p>Ok time to dive into the implementation!</p><h4>Part I — historical data</h4><ul><li>If you’ve been following this series, you’re likely familiar with the various sources such as Yahoo Finance, Kite API, and others used to obtain historical data. Numerous articles have delved into this topic extensively.</li><li>For this experiment, I’ve compiled two years’ worth of daily data for all Nifty500 and Nasdaq stocks.</li><li>I’ll assume that you, the reader, are capable of downloading historical data for the stocks you wish to analyze and have organized them neatly within a single folder. This foundational step is crucial for streamlining the stock screening process.</li></ul><h4>Part II — Generate Dataframe for LLM</h4><p>Lets define a screener class that takes a folder as input. This folder is where all the stock historical data is downloaded and stored.</p><ol><li>Create a class with some variables. self.df_all is where the final data frame is stored before passing to LLM for screening</li></ol><pre>#screener.py<br>class Screener:<br>    def __init__(self, folder):<br>        self.index = folder<br>        self.folder = f&#39;./data/historical/analysis/{folder}/&#39;<br>        self.files = os.listdir(self.folder)<br>        self.df_all = pd.DataFrame()</pre><p>2. lets create a method run to generate df</p><pre>#screener.py<br>def run(self):<br>    for index, filename in enumerate(self.files):<br>        if (filename.endswith(&#39;xlsx&#39;) or filename.endswith(&#39;csv&#39;)) and not filename.startswith(&#39;~$&#39;):<br>            df = pd.read_csv(self.folder+filename)<br><br>            df[&#39;TIMESTAMP&#39;] = pd.to_datetime(df[&#39;TIMESTAMP&#39;],utc=True)<br>            df[&#39;TIMESTAMP&#39;] = df[&#39;TIMESTAMP&#39;].dt.date<br>            current_date = df.iloc[-1][&#39;TIMESTAMP&#39;]<br>            <br>            symbol = df.iloc[-1][&#39;SYMBOL&#39;]<br>            <br>            info = yf.Ticker(symbol)<br>            earnings_date = info.info.get(&#39;mostRecentQuarter&#39;,None)<br>            earnings_date = datetime.utcfromtimestamp(earnings_date) if earnings_date else None<br>            earnings_growth = round(info.info.get(&#39;earningsQuarterlyGrowth&#39;,0)*100,2)<br>            close = df.iloc[-1][&#39;CLOSE&#39;]<br>            pe  = info.info.get(&#39;trailingPE&#39;,None)<br>            <br>            revenue_growth = round(info.info.get(&#39;revenueGrowth&#39;,0)*100,2)<br>            debtToEquity = round(info.info.get(&#39;debtToEquity&#39;,0)/100.0,2)<br>            market_cap = str(round(info.info.get(&#39;marketCap&#39;,0)/10000000.0,2)) + &#39;cr&#39;<br>            one_week_ago = self.find_nearest_available_date(current_date - pd.DateOffset(days=7),df)<br>            close_one_week_ago = df.loc[df[&#39;TIMESTAMP&#39;] == one_week_ago,&#39;CLOSE&#39;].iloc[0]<br>            <br>            # sometimes a data before a year may not be there, we simply fall back to the available value <br>            year_ago = self.find_nearest_available_date(current_date - pd.DateOffset(years=1),df)<br>            matching_rows = df.loc[df[&#39;TIMESTAMP&#39;] == year_ago]<br>            if not matching_rows.empty:<br>                close_year_ago = matching_rows.iloc[0][&#39;CLOSE&#39;]<br>            else:<br>                 close_year_ago = close_six_month_ago</pre><p>Dont worry! its simple.</p><ul><li>Iterate through all files and Read the excel file and make timestamp column a datetime column</li><li>Extract the latest date from the sheet</li><li>use yf.Ticker(symbol) to extract some fundamental data and get some information like PE ratio, earnings debt to equity, market cap etc.</li><li>Look at how we have used find_nearest_available_date when we try to extract close price a week ago or month ago, it may end up in a holiday, so here we will try to find a nearest date when market is open and pull the closing price. below is the actual logic</li></ul><pre>#screener.py<br>def find_nearest_available_date(self, date,df):<br>        ctr=0<br>        date = date.date()<br>        while not (df[&#39;TIMESTAMP&#39;]==date).any():<br>            date -= timedelta(days=1)<br>            ctr+=1<br>            if ctr&gt;5:<br>                break<br><br>        return date</pre><p>3. with close price of today with close price of 1 month, 1 year, 3 month, we can calculate the returns using below method.</p><pre>#screener.py<br>def get_percent(self, current,old):<br>    change = current-old<br>    percent = change/old*100<br>    return round(percent,2)</pre><p>4. Finally generate the data frame.</p><pre>#screener.py - under run() method under the for loop (check above for reference)<br>data = {&#39;date&#39;: [current_date],<br>        &#39;symbol&#39;: [symbol],<br>        &#39;pe&#39; : [pe],<br>        &#39;debtToEquity&#39;:[debtToEquity],<br>        &#39;market_cap&#39;:[market_cap],<br>        &#39;close&#39;: [close],<br>        &#39;price_change_one_week&#39;: [self.get_percent(close,close_one_week_ago)],<br>        &#39;price_change_one_month&#39;: [self.get_percent(close,close_one_month_ago)],<br>        &#39;price_change_three_month&#39;:[self.get_percent(close,close_three_month_ago)],<br>        &#39;price_change_six_month&#39;:[self.get_percent(close,close_six_month_ago)],<br>        &#39;price_change_one_year&#39;:[self.get_percent(close,close_year_ago)],<br>        &#39;earnings_date&#39;:earnings_date,<br>        &#39;earnings_growth&#39;:earnings_growth,<br>        &#39;revenue_growth&#39;:revenue_growth<br>        }<br>df_change = pd.DataFrame(data)<br>self.df_all = pd.concat([self.df_all,df_change], axis=0)</pre><p>5. we need to store the data in a CSV</p><pre>#screener.py<br>def store_in_csv(self):<br>        folder = f&#39;./data/historical/screener/&#39;<br>        filename = f&#39;{self.index}_{str(date.today())}.csv&#39;<br>        path = os.path.join(folder,filename)<br>        self.df_all.to_csv(path)</pre><p>6. Now we have all the data required for the main step. heading to the most exciting part — Screening</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*heTtIBCih_792MvY" /></figure><blockquote>During my experimentation, I employed algorithmic screening techniques and leveraged multiple Large Language Models (LLMs) as well for the screening process. It was fascinating to observe the diverse performances and rationale provided by different LLMs when presented with the same dataset side by side.</blockquote><h4>Algorithmic Screener</h4><ul><li>Our rule for algorithmic screener is simple</li><li>Companies with recent results(we have captured result quarter in our df), low debt low P/E good revenue and earnings, yet less 3 months profit for under valuation. we can use any criteria here.</li></ul><pre>def algorithmic_screening(self):<br>        df = self.df_all<br>        # Calculate today&#39;s date<br>        today = datetime.now()<br><br>        # Calculate the date three months from today<br>        three_months_ago = today - timedelta(days=90)<br><br>        # Apply filter conditions<br>        df = df[(df[&#39;pe&#39;] &lt; 25) &amp; <br>                        (df[&#39;debtToEquity&#39;] &lt; 0.5) &amp; <br>                        (df[&#39;earnings_growth&#39;] &gt; 0) &amp; <br>                        (df[&#39;revenue_growth&#39;] &gt; 0) &amp; <br>                        (df[&#39;earnings_date&#39;] &gt; three_months_ago)]<br>        # Sort filtered DataFrame by &#39;one_month&#39; ascending<br>        df = df.sort_values(by=&#39;one_month&#39;, ascending=True)<br>        print(df.head(5))<br>        return </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*izXun_bes-h6r8BHHBf7FA.png" /></figure><p>We get the above results by running the code. How do i confirm this? Lets take BAJAJHLDNG for example, it has 100% earnings growth and 1058% revenue growth, yet in last month it is -3.68% down. Lets check my all time favorite site <a href="https://www.screener.in/company/BAJAJHLDNG/consolidated/">screener.in</a> to cross validate this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9_YXoWDMh16Y_vSmVb98g.png" /></figure><p>Well, thats algorithmic screening based on a specific rule set. As retail investors, we each have our own distinct criteria for stock selection, shaped by our individual investment goals and risk appetites. This personalized approach sets us apart, allowing us to tailor our strategies to meet our specific needs.</p><p>Interestingly, this personalized approach mirrors the functionality of many screening websites’ user interfaces (UI)</p><h4>LLM Screener</h4><p>Install the cohere python client pip install cohere.</p><p>Ok, lets first build a cohere interface as cohere_interface.py. I just love their documentation — <a href="https://docs.cohere.com/reference/chat">https://docs.cohere.com/reference/chat</a></p><pre>#cohere_interface.py<br>import cohere<br>from dotenv import load_dotenv<br>import os<br>import json<br>import re<br>load_dotenv()<br><br>class Cohere:<br>    def __init__(self):<br>        self.co = cohere.Client(os.getenv(&quot;CO&quot;))<br><br>    def chat_cohere(self,message,preamble):<br>        response = self.co.chat(<br>            message=message,<br>            preamble=preamble<br>        )<br>        pattern = r&#39;\{[^{}]*\}&#39;<br>        # Find all JSON matches in the input string<br>        json_matches = re.findall(pattern, response.text)<br>        json_out = [json.loads(match) for match in json_matches]<br>        return json_out<br><br>co = Cohere()</pre><ul><li>Signup in cohere and generate API key and store it in .env</li><li>import all the libraries. create a client under __init__</li><li>implement a simple method chat_cohere that takes 2 input message and preamble.</li><li>preamble is basically the context we want to set the LLM for a specific task and message is the actual input for the LLM.</li><li>They also provide attributes like documents where we can pass the RAG context etc. but we dont need them here.</li><li>simply call their chat method and parse the text response.</li><li>I’m using regex functions to make sure i get the Json response. OpenAI, Gemini are already having JSON response built-in to their models. but I couldnt find that option in Cohere, so i built a simple regex parsing</li><li>We also need to make sure our prompt allows Cohere to generate JSON output.</li><li>next we will be consuming the above in our screener.py that we built above.</li><li>make sure to import the object co from cohere_interface.py in screener.py and call the method we developed in previous step chat_cohere.</li><li>ai_screening method where we will generate a prompt. note how we are passing the the data frame in message parameter and how we are setting context to LLM in preamble.</li></ul><pre>#screener.py<br>def ai_screening(self):<br>    message = f&quot;&quot;&quot;You are given the data of some stocks, Analyse the data and Screen the stock list and select \<br>    2 stocks that is undervalued and is likely to generate better returns in future. Below is the data {self.df_all}&quot;&quot;&quot;<br>    preamble = &quot;&quot;&quot;You are an excellent stock market analyst who can understand the provided data well and take \<br>    careful decisions. Output should strictly be in the form of json with follwing structure<br>    [<br>        {<br>            &quot;stock&quot;: &quot;Name of the stock&quot;<br>            &quot;justification&quot;: &quot;Reason why the stock is screened or selected&quot;<br>        },<br>        {<br>            &quot;stock&quot;: &quot;Name of the stock&quot;<br>            &quot;justification&quot;: &quot;Reason why the stock is screened or selected&quot;<br>        }<br>    ]<br><br>    There should not be additional output other than json<br>        &quot;&quot;&quot;<br>    result = co.chat_cohere(message=message,preamble=preamble)<br>    print(result)</pre><p>Lets look at the JSON results of LLM from the terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cilBBYfxpncTn_IdHwfCJQ.png" /></figure><p>As usual, the complete code will be available in my <a href="https://github.com/vishyarjun">github</a>. feel free to pull it.</p><p>Thats it!! its that easy to use Cohere LLM through their APIs!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*mdy3kULmKe1nn5vI.gif" /></figure><p>If you have decided to use a self hosted LLM, I recommend checkout this article to how to host an LLM in AWS — <a href="https://medium.com/@viswanathan.arjun/deploy-and-host-a-financial-bert-model-as-api-in-aws-sagemaker-7b381dfbcd6a">https://medium.com/@viswanathan.arjun/deploy-and-host-a-financial-bert-model-as-api-in-aws-sagemaker-7b381dfbcd6a</a></p><h4>Nah!! Using LLM to make decision is risky!</h4><p>This sentiment is often echoed, but I hold a somewhat unpopular opinion. I firmly believe that LLMs have the potential to empower retail investors to explore new dimensions in their market decisions. How? It’s quite simple — LLMs possess an analytical prowess that surpasses human capabilities in many respects.</p><p><strong><em>So, Is this fool proof ?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*UIfbo0z3lgp2-Cc5.jpeg" /></figure><p>While leveraging LLMs for decision-making represents a powerful tool in the arsenal of retail investors, it’s essential to recognize that no approach is without its limitations. What I’ve presented here is just one example of how effectively we can harness the capabilities of LLMs.</p><p>In my own investment journey, I go even further by employing RAG techniques. This advanced methodology involves incorporating quarterly statements spanning several years, earnings release data, and order book analyses. By layering these elements, I’m trying to construct a comprehensive framework for evaluating a company’s growth trajectory. But thats a big topic requires a separate series, definitely some other day!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*HWtroWrbwUt8EhPj.jpg" /></figure><blockquote><strong>Some final disclaimer:</strong> These are definitely not investment advices and should be never be considered as one, these are just some interesting ways to use LLMs.</blockquote><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedback, please feel free to comment, It would mean a lot to me!</p><p>In case of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X (formerly Twitter)</a>.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eefc9879bf28" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/llm-for-market-analysis-part-vi-llm-vs-algorithmic-stock-screener-eefc9879bf28">LLM for Market Analysis — Part VI — LLM Vs Algorithmic  stock screener</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy and Host a Financial BERT model as API in AWS SageMaker]]></title>
            <link>https://blog.devops.dev/deploy-and-host-a-financial-bert-model-as-api-in-aws-sagemaker-7b381dfbcd6a?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b381dfbcd6a</guid>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[stock-trading]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Sun, 04 Feb 2024 20:06:02 GMT</pubDate>
            <atom:updated>2024-05-08T16:26:56.580Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*fcHL74SUevKsMxudkty7Ig.png" /></figure><p>In our recent articles, we delved into the realm of Stock Market Analysis using Python and LLM (Large Language Models). This series explores the application of LLM in various facets of market analysis, including technical pattern analysis, fundamental analysis, and news-based trend analysis. If you’re a retail investor keen on leveraging the power of AI and technology for your investments, I highly recommend diving into the comprehensive insights provided in this Python and LLM for Market Analysis series. Whether you’re a seasoned investor or just starting, these articles, from <a href="https://medium.com/@viswanathan.arjun/technical-analysis-with-python-libraries-part-i-88409432c672">Part I</a> to <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-stock-analysis-part-v-streamlit-css-for-a-tailor-made-trading-platform-aa515486ba09">Part V</a>, offer valuable perspectives and strategies to enhance your understanding and decision-making in the dynamic world of finance</p><p>Up until now, our exploration into news sentiment analysis has been powered by cloud-based Google Bard Models. Yet, in my personal projects, I’ve opted for a finely-tuned BloombergGPT model, pre-trained on an extensive dataset of financial information.</p><p>In this article, we’re taking a leap into deploying our bespoke financial model using AWS SageMaker. Our model of choice? FinBERT. This robust choice promises to bring a new dimension to our analysis, and we’ll delve into the intricacies of setting it up and harnessing its potential for more accurate and nuanced financial sentiment evaluation</p><blockquote>In this article, we’re spotlighting the FinBERT model. Worth noting, the insights shared apply not only to FinBERT but also to any Large Language Model (LLM) on Hugging Face. Short and sweet, the principles here can guide you through various models, offering flexibility for your specific language model preferences.</blockquote><ol><li><strong>Why is FinBERT my choice of financial model?</strong></li></ol><p>For a detailed understanding, check out the <a href="https://medium.com/prosus-ai-tech-blog/finbert-financial-sentiment-analysis-with-bert-b277a3607101">FinBERT</a> article by Prosus AI. In essence, Financial Sentiment Analysis differs from regular sentiment analysis.</p><ul><li>In the finance world, a company making a loss may still be seen positively if it’s better than analysts’ expectations.</li><li>Conversely, a profitable company falling short of market expectations can generate negative sentiment.</li><li>Sometimes, seemingly good news triggers large institutional sell-offs, causing prices to drop.</li></ul><p>Relying solely on news sentiment may not suffice; factors like bulk trades and additional data are crucial for a comprehensive analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*0ptEUvJW7Ul-98k1" /></figure><p>Yes, take a deep breathe…</p><p>Unless a model is trained to such nuances, it’s very hard to understand what really went on. FinBERT is one such. More details about model <a href="https://huggingface.co/ProsusAI/finbert">here</a>.</p><h4>2. What is BERT?</h4><p>BERT, which stands for Bidirectional Encoder Representations from Transformers has achieved state-of-the-art performance in various benchmark datasets and has become a foundational model in the field of natural language processing. What sets BERT apart is its bidirectional approach to language understanding. Unlike traditional language models that read text in one direction (either left-to-right or right-to-left), BERT considers both directions simultaneously. This bidirectional context allows BERT to capture complex relationships and nuances in language, making it particularly effective in tasks such as text classification, sentiment analysis, question answering, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/0*9ff2qwKP3dlh_5il.png" /><figcaption>Different from the Sesame Street BERT though!</figcaption></figure><h4>3. How are we going to use AWS SageMaker to host our own LLM?</h4><p>What we are going to build will look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUyIZydAIaKp4l7BifbGVQ.png" /></figure><ul><li>In Short we use <strong>SageMaker</strong> to host our Language Model. we need a dedicated machine for this</li><li><strong>Lambda function</strong>( a small piece of python code triggered by event like API call ) to conveniently interact with language model. We do not need dedicated server, this is serverless, the code just executes and produce results on-demand.</li><li><strong>API Gateway — </strong>this is how external systems including local machine(anything outside of AWS ecosystem) can interact with the model, This will be enabled through traditional REST APIs.</li><li>For more details — <a href="https://aws.amazon.com/api-gateway/">API Gatway,</a> <a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">Lambda Functions</a> and <a href="https://aws.amazon.com/pm/sagemaker/?gclid=CjwKCAiAiP2tBhBXEiwACslfnnYxWgnkjlp445HGtYWvhtURE-9im2QjClUT3_3uuaI_GE7Sli0-mxoCmWQQAvD_BwE&amp;trk=b6c2fafb-22b1-4a97-a2f7-7e4ab2c7aa28&amp;sc_channel=ps&amp;ef_id=CjwKCAiAiP2tBhBXEiwACslfnnYxWgnkjlp445HGtYWvhtURE-9im2QjClUT3_3uuaI_GE7Sli0-mxoCmWQQAvD_BwE:G:s&amp;s_kwcid=AL!4422!3!651751060692!e!!g!!aws%20sagemaker!19852662230!145019225977">SageMaker</a>.</li></ul><h4>4. Concerned about the cost of hosting your own Language Models?</h4><p>Let’s break it down. Using a pay-per-token model like GPT-4 in Azure Open AI, cost will look something like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A0XAukjWa1_qW452K7v0uQ.png" /></figure><p>For a decent accuracy, we will use GPT-4</p><ul><li>Cost of Completion Tokens &gt; Cost of Prompt Tokens. Fortunately for our use case, there is more prompt token and less completion token(10:1 ratio)</li><li>Approximately 1 <strong>token</strong> ~= 4 chars in English</li><li>We will easily have 250 words(~400 tokens) per request. It costing 0.03$ for every 1000 tokens.</li><li>In a day we will easily perform 1000 requests costing us 400K tokens ~ 12$</li><li>Since response is 1/10th and cost is doubled, it’s going to be 2.4$ making it a <strong>total cost of ~14.5$/day.</strong></li></ul><p>When hosting our own model in AWS SageMaker, without fine-tuning, the cost breakdown is as follows:</p><h4>1. AWS Instance for Notebook (ml.t3.medium):</h4><ul><li>Notebook is needed to deploy the language model. we run code that deploys the model.</li><li>1 instance at $0.05 per hour</li><li>Created for 1 hour per day and later deleted</li></ul><h4>2. AWS Instance for Language Model (ml.m5.xlarge):</h4><ul><li>1 instance at $0.23 per hour</li><li>Operational for 8 hours per day and later deleted</li></ul><h4>3. AWS Lambda Serverless Function:</h4><ul><li>Runs custom Python program invoking the LLM.</li><li>AWS Lambda free tier includes 1M free requests and 400K GB-seconds of compute time per month</li></ul><h4>4. AWS API Gateway:</h4><ul><li>Allows API consumption from local machines or any location</li><li>Free tier includes one million API calls received for REST APIs, one million for HTTP APIs, and one million messages with 750,000 connection minutes for WebSocket APIs per month for up to 12 months</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*9H9uyUX9WMBB5CcjgaIvRQ.png" /></figure><p>Even if there are 1M API requests per month, It’s going to be 3.5$ for the entire month. Roughly 0.12$/day(only after free tier ends).</p><p>In a smaller use case, costs for Lambda and API Gateway may be minimal, but for a production-like system, additional charges may apply if exceeding the free tier limits.</p><p><strong>Total Cost: ($0.05 X 1 hour) + ($0.23 X 8 hours) + 0.12$ = 2.01$ per day</strong></p><blockquote>- Above would mean we will clean up all instances once the work is completed.</blockquote><blockquote>- My current setup: So everyday morning I have an automated deployment of these services in-place that takes about ~10/15 mins. and by the end of the day(after 8 hours)my services will be deleted.</blockquote><blockquote>- This calculation is only applicable for minimal use case and may not be applicable to all. but this will give a fair enough idea of what to expect.</blockquote><h3>2$ vs 14$ — Not bad!!</h3><p>Let’s dive into the most important part of it — the implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*LR32Wb_dRVhj1TGx.jpg" /></figure><ol><li>It all starts by us visiting the <a href="https://huggingface.co/ProsusAI/finbert">https://huggingface.co/ProsusAI/finbert</a>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fIlGhnud889KEYoFJrpO7A.png" /></figure><p>2. Click on Deploy and Choose the Amazon SageMaker option and we will be presented with the code. Copy the code.</p><p>3. Create an AWS Account, Initially free tier may be available. Login to the AWS account and navigate to SageMaker.</p><p>4. To deploy the model, create a Notebook instance. Under the ‘Notebook’ section, initiate a new instance, providing it with a name. The process might resemble the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yB0ghpvOK15ymWvUH8M-ug.png" /></figure><p>5. Once the instance is up, we can open the notebook and paste the code copied from huggingface.</p><pre>import sagemaker<br>import boto3<br>from sagemaker.huggingface import HuggingFaceModel<br><br>try:<br> role = sagemaker.get_execution_role()<br>except ValueError:<br> iam = boto3.client(&#39;iam&#39;)<br> role = iam.get_role(RoleName=&#39;sagemaker_execution_role&#39;)[&#39;Role&#39;][&#39;Arn&#39;]<br><br># Hub Model configuration. https://huggingface.co/models<br>hub = {<br> &#39;HF_MODEL_ID&#39;:&#39;ProsusAI/finbert&#39;,<br> &#39;HF_TASK&#39;:&#39;text-classification&#39;<br>}<br><br># create Hugging Face Model Class<br>huggingface_model = HuggingFaceModel(<br> transformers_version=&#39;4.26.0&#39;,<br> pytorch_version=&#39;1.13.1&#39;,<br> py_version=&#39;py39&#39;,<br> env=hub,<br> role=role, <br>)<br><br># deploy model to SageMaker Inference<br>predictor = huggingface_model.deploy(<br> initial_instance_count=1, # number of instances<br> instance_type=&#39;ml.m5.xlarge&#39; # ec2 instance type<br>)<br><br>predictor.predict({<br> &quot;inputs&quot;: &quot;&lt;your input text for sentiment classification goes here&gt;&quot;,<br>})</pre><p>Lets try to understand this code a bit</p><ol><li>Import boto, sagemaker and hugging face model</li><li>initialize role, hub, model</li></ol><ul><li>note how we use ProsusAI/finbert for the model and text-classification for the task.</li></ul><blockquote>You might notice that this is only a classification model. This doesn’t provide a numeric value (-1.0 to 1.0) like Google Bard, refer our previous article <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d">here</a>. thats just fine for now, in the upcoming article we will use another model and fine tune it to predict the intensity of the market sentiment.</blockquote><p>3. Look at this line carefully</p><pre>predictor = huggingface_model.deploy(<br>    initial_instance_count=1, # number of instances<br>    instance_type=&#39;ml.m5.xlarge&#39; # ec2 instance type<br>)</pre><ul><li>here an ml.m5.xlarge instance is spinned up to run the model. If needed, we can change this to another instance type that suits the model.</li><li>executing this line takes upto ~10 mins for the model to be ready.</li><li>Once the model is ready, we can run the predict</li></ul><p>4. The notebook should look something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yNHyDMRdJqaJ3u_Mx9iZvg.png" /></figure><ul><li>the line predictor.predict generates a score and label as output.</li></ul><p>5. Now, deploy the model as a service by creating a Lambda function. Give it a name and paste the below provided code under the Code tab.</p><pre>import json<br>import boto3<br><br>def lambda_handler(event, context):<br>    client = boto3.client(&#39;runtime.sagemaker&#39;)<br>    payload = json.dumps(event)<br>    model = &quot;huggingface-pytorch-inference-xxxx-xx-xx-xx-xx-xx-xxx&quot;<br>    response = client.invoke_endpoint(EndpointName=model, ContentType=&#39;application/json&#39;, Body=bytes(payload,&#39;utf-8&#39;))<br>    print(response)<br>    result = json.loads(response[&#39;Body&#39;].read().decode())<br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: result<br>    }</pre><p>6. As you might see, we are just invoking the model using invoke_endpoint and just parsing the reponse from the model.</p><p>7. Replace the model value with your model. You can find it under the SageMaker Inference tab. Copy the Name not the ARN.</p><p>Legends says put the model variable under Environment configurations and refer it appropriately in the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*7i-GhlVGqvq-n5BZ.jpg" /></figure><p>7. The above code just invokes the SageMaker model and pass the response along with statusCode.</p><p>event and context comes by default. we shouldn’t attempt to change this. event is where the JSON input from API request comes.</p><blockquote>This is just a plain lamda_handler. A production ready instance of lambda_handler will handle more scenarios than this.</blockquote><p>8. Click on Deploy under the code tab each time a change is made to the code.</p><p>9. Make sure to update the policies by following this <a href="https://medium.com/analytics-vidhya/invoke-an-amazon-sagemaker-endpoint-using-aws-lambda-83ff1a9f5443">article</a>. this ensure there is no permission denial between lambda and sagemaker inference model.</p><p>9. Go to test tab and Enter the input as below and hit Test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8s2c_-at910HXlOW_CL3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*AhitaDZfDw49rKZ6yjRTZw.png" /></figure><p>A successful output will look something like this.</p><p>10. Next expose the lambda function over API. Go to AWS open API Gateway Service, create a new API Gateway and choose REST API and click build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*wVLEeHqVoGOBRrDJxvN4mA.png" /></figure><p>11. Give it a name and Create a method. Make it a post and choose Lambda function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*zRHEj4IpTPugHhkO7QE9_g.png" /></figure><p>12. In the Lambda configuration, a dropdown menu will appear where you can choose the appropriate Lambda function you’ve created. Select the correct one.</p><p>13. After creating the method, deploy the API by clicking the button in the top right corner. Choose or create a stage; think of a stage as an environment in this context.</p><p>14. Make sure to enable throttling to avoid a burst of API calls giving a huge dent in the wallet. Yes, that happens to everyone playing around with LLM these days!</p><p>15. For security, its advisable to generate Authorizers as well. or simply create an API Key under API Gateway(newly created resource).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wVdRJ-1qzeoEd3QGshhlRA.png" /></figure><p>Thats it!! we have deployed and hosted our own Language Model now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*Fk13WDD40COq8GRW" /></figure><p>We can test the API using Postman or using a simple python program as well.</p><pre>import requests<br>import json<br><br>API_KEY = &#39;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&#39;<br>URL = &#39;https://xxxxxxxxxxxx.execute-api.uk-east-9.amazonaws.com/dev&#39;<br><br>body = {<br>    &quot;inputs&quot;: &quot;&quot;&quot;In light of recent discussions surrounding inquiries by the Enforcement Directorate (ED), Paytm Payments Bank clarified that no probe against Bank or One97 Communications Ltd by the probe agency. &quot;One 97 Communications Ltd and Paytm Payments Bank operate with the highest ethical standards. We can confirm that neither we nor OCL&#39;s founder-CEO have been the subject matter of investigation by the Enforcement Directorate regarding money laundering,&quot; a spokesperson for Paytm Payments Bank said, news agency ANI reported. <br>    If there are any fresh charges of money laundering against Paytm by RBI, those will be investigated by Directorate of Enforcement as per the law of the land,&quot; Reuters quoted Revenue Secretary Sanjay Malhotra.<br>    The clarification comes amidst reports suggesting otherwise. &quot;Occasionally, some merchants on our platforms have been the subject of inquiries, and we cooperate fully with authorities in such instances. We categorically deny any involvement in money laundering activities and believe fair and responsible journalism is crucial for accurate information dissemination,&quot; the spokesperson added.<br>&quot;&quot;&quot;<br>}<br><br>data = json.dumps(body)<br><br>response = requests.post(URL, data=data,headers={&#39;Content-Type&#39;:&#39;application/json&#39;,&#39;x-api-key&#39;:API_KEY})<br><br>print(response.text)</pre><p>Make sure to replace the API_KEY and URL and run it!!</p><blockquote>Please make sure to delete the model and all the other services associated with it once done. Especially the instances ml.m5.xlarge and ml.t3.medium. Remember its charged / hour.</blockquote><p>I’m intrigued by how effortlessly AWS and HuggingFace have simplified this for us. Curious, I inquired with ChatGPT, and here’s what it had to say.</p><p>Deploying an AI model with AWS and HuggingFace feels like waving a wand and making magic happen. Thanks to the seamless integration and user-friendly interfaces, what used to be a complex process has become as enchanting as a spell. Now, turning your machine learning dreams into reality is as easy as saying Abracadabra with the help of AWS and the magical touch of HuggingFace!</p><p>In the upcoming article, we will discuss about fine tuning a language model trained on large finance corpus and see how the fine tuned model is able to perform better on the market conditions.</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Incase of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b381dfbcd6a" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/deploy-and-host-a-financial-bert-model-as-api-in-aws-sagemaker-7b381dfbcd6a">Deploy and Host a Financial BERT model as API in AWS SageMaker</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python and LLM for Market Analysis — Part V — Streamlit & CSS for news aggregation portal]]></title>
            <link>https://medium.com/@viswanathan.arjun/python-and-llm-for-stock-analysis-part-v-streamlit-css-for-a-tailor-made-trading-platform-aa515486ba09?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa515486ba09</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[stock-trading]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Tue, 23 Jan 2024 04:59:14 GMT</pubDate>
            <atom:updated>2024-01-23T05:57:08.423Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python and LLM for Market Analysis Part V — Streamlit &amp; CSS for news aggregation portal</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wy6-e6nPlLbl_xQ6.png" /></figure><p>In our previous articles, we delved into the world of news aggregation APIs and harnessed the power of Language Models (LLMs) to decipher sentiment. Combining these insights with technical data, we crafted personalized recommendation lists. Now, in this installment, we embark on a thrilling journey to create something truly invaluable — a dashboard dedicated to finance and stock-related news, adorned with insightful technical charts.</p><p>In my pursuit of staying on top of the latest in finance, I’ve invested in multiple premium subscriptions over the years. However, there’s always been a lingering desire for a one-stop solution — a comprehensive news aggregation platform seamlessly intertwined with technical analysis and AI-driven sentiment insights. While existing platforms serve their purpose well, our mission here is to roll up our sleeves and craft our own. Let’s dive into the realm of development and bring this vision to life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Fg1Xg0M__pceOy-r.png" /></figure><p>In my early development days, my focus leaned heavily towards crafting frontends using HTML, CSS, and JS. Backends and servers were not initially on my radar. As I transitioned into a Python Developer role, I found myself dedicating less time to frontend development. Curiosity arose about a framework that would allow me to seamlessly integrate Python’s coding style with frontend tasks.</p><p>I initially experimented with WTForms and Jinja templating, but the quest for dynamic charts and compelling visuals led me to explore further. That’s when I stumbled upon Streamlit — an absolute revelation. Streamlit’s lightweight nature allowed me to stick to the Python coding style I was comfortable with, making project development remarkably straightforward.</p><p>For all my subsequent projects and demos, Streamlit became my tool of choice due to its simplicity. Although I ended up learning ReactJS later for my work, being able to write a frontend in python is a ❤.</p><h3>Story Over….</h3><blockquote>Note: Please read the previous articles <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d">Part-III</a> and <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-stock-market-analysis-part-iv-elasticsearch-for-stock-symbol-ticker-accuracy-1acda81eeec3">Part-IV</a> before reading further for smooth continuity.</blockquote><h3>Whats this UI portal all about?</h3><ol><li><strong>Date Field:</strong> A straightforward date picker allowing users to choose a date, revealing the day’s news.</li><li><strong>Three-Columnar Card Layouts:</strong> Displaying news articles, images, links, AI sentiment, and recommendations in an organized format.</li><li><strong>Download Button:</strong> Initiates the download of data from the news aggregator, coupled with sentiment analysis using the previously developed LLM.</li><li><strong>Progress Bar:</strong> Offers an updated, real-time status of ongoing processes.</li></ol><p>We’re set to develop two essential files:</p><ul><li>Home.py: A Streamlit file dedicated to the frontend.</li><li>styles.css: A simple CSS file for frontend styling and enhancing card layouts.</li></ul><p>Additionally, we’ll enhance our existing implementation in news_tech_trader.py to provide real-time status updates on the UI. Let&#39;s dive into the development process!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*E19DEyRm3fp0tNR3.jpg" /></figure><pre>#Home.py<br>import streamlit as st<br>from datetime import datetime, date<br>import pandas as pd<br>import os<br>from news_tech_trader import nt<br>import time<br><br>st.set_page_config(layout=&quot;wide&quot;)<br>bgs = [<br>    &quot;#8B0000&quot;,&quot;#B22222&quot;,&quot;#DC143C&quot;,&quot;#FF4500&quot;, &quot;#FF6347&quot;,&quot;#FF7F50&quot;,&quot;#FFA07A&quot;,&quot;#FFD700&quot;, &quot;#ADFF2F&quot;,&quot;#008000&quot;<br>]</pre><p>Home.py is a Streamlit server file dedicated to serving content for the UI. In this file, the recommendation class from the previous article (located in news_tech_trader.py) is imported using the line from news_tech_trader import nt. The page layout is configured to be wide, ensuring a full-page app experience.</p><p>Additionally, a palette of 10 colors, ranging from green to amber to red, has been selected and stored in a list (bgs). These colors will be utilized for visually representing AI sentiment in the displayed cards.</p><pre>#Home.py - continues<br>def local_css(file_name):<br>    with open(file_name) as f:<br>        st.markdown(f&#39;&lt;style&gt;{f.read()}&lt;/style&gt;&#39;, unsafe_allow_html=True)<br><br><br>def get_data_frame(file_name):<br>    df = pd.read_csv(file_name)<br>    if &#39;ImageURL&#39; not in df.columns:<br>        df[&#39;ImageURL&#39;] = None<br>    <br>    if &#39;recommendation&#39; not in df.columns:<br>        df[&#39;recommendation&#39;] = None<br>    <br>    if &#39;PRICE_AT_TIME&#39; not in df.columns:<br>        df[&#39;PRICE_AT_TIME&#39;] = None<br>    else:<br>        df = df[df[&#39;PRICE_AT_TIME&#39;].notna()]<br>    df = df.sort_values(by=&#39;impact&#39;, ascending = False)<br>    return df</pre><p>In this segment of Home.py, two methods are defined. The local_css function loads the CSS file using st.markdown, allowing for custom styling. The get_data_frame function is responsible for obtaining the necessary data frame from a CSV file. The method includes data cleanup steps such as checking for null values, ensuring column existence, and sorting the data by the &#39;impact&#39; column in descending order for better presentation.</p><p>Next comes the biggest chunk of this implementation. So gear up..</p><pre>#Home.py - continues<br>local_css(&#39;./styles.css&#39;)<br>def display_news_cards(selected_date):<br>  col1, col2, col3 = st.columns(3)<br>  cols = [col1, col2, col3]<br>  file_name = f&#39;./data/news/{selected_date}.csv&#39;<br>  if os.path.exists(file_name):<br>        df = get_data_frame(file_name)<br>        chunk_size = 3<br>        for i in range(0, len(df), chunk_size):<br>            chunk = df.iloc[i:i+chunk_size].reset_index(drop=True)<br>            for idx, row in chunk.iterrows():<br>                with cols[idx]:<br>                    new_value = int((row[&#39;impact&#39;] + 1) * 50)<br>                    disp = new_value - (new_value%10)<br>                    bg = bgs[disp//10]<br>                    container = st.container()<br>                    container.markdown(<br>                        f&quot;&quot;&quot;<br>                            &lt;div class=&quot;card&quot;&gt;<br>                                &lt;img src={row[&#39;ImageURL&#39;]} alt=&quot;News Image&quot;&gt;<br>                                &lt;div class=&quot;content&quot;&gt;<br>                                    &lt;h2&gt;&lt;a href=&quot;{row[&#39;URL&#39;]}&quot; target=&quot;_blank&quot;&gt;{row[&#39;Title&#39;]}&lt;/a&gt;&lt;/h2&gt;<br>                                    &lt;p class=&quot;organization&quot;&gt;{row[&#39;name&#39;]} - {row[&#39;symbol&#39;]}&lt;/p&gt;<br>                                    &lt;p class=&quot;sentiment&quot;&gt;AI Sentiment(ranges from -1 to 1): {row[&#39;impact&#39;]}&lt;/p&gt;<br>                                    &lt;p class=&quot;sentiment&quot;&gt;AI Recommendation: {row[&#39;recommendation&#39;]}&lt;/p&gt;<br>                                &lt;/div&gt;<br>                                &lt;div class=&quot;sentiment-container&quot;&gt;<br>                                    &lt;!-- Sentiment Fill --&gt;<br>                                    &lt;div class=&quot;sentiment-slider&quot; style=&quot;width: {disp}%; background-color:{bg};&quot;&gt;&lt;/div&gt;  <br>                                &lt;/div&gt;<br>                            &lt;/div&gt;&quot;&quot;&quot;, unsafe_allow_html=True)<br>                <br>    else:<br>        st.text_area(label=&quot;Empty data display&quot;,value=&quot;No Data Available.&quot;,label_visibility=&quot;collapsed&quot;)</pre><blockquote>Note: It might look like a lott of code at first, but please give it a look and a little patience, its really simple.</blockquote><p>Here is the explanation of the code</p><ul><li>This code segment begins by invoking the method responsible for loading the CSS file. Within the display_news_cards function, the initial step involves attempting to read a file based on the date entered by the user. As a reminder from our previous article, we adopted a naming convention for CSV files—&lt;date&gt;.csv—where the date corresponds to the publication date of news articles, AI sentiment, and technical data.</li><li>If the file exists, its contents are presented on the UI as cards. In the absence of the file, a text box is displayed with a message indicating ‘No data available.’ To accommodate the layout, we utilize Streamlit’s st.columns function to define three columns.</li><li>When the file exists, the dataframe is grouped into multiple sets, each containing three items. This grouping ensures that each item is displayed within its designated column. The chunking mechanism, facilitated by setting chunk_size to 3, contributes to an organized presentation of news articles on the UI.</li></ul><blockquote>Note the line <strong>chunk = df.iloc[i:i+chunk_size].reset_index(drop=True) </strong>very carefully. we are resetting the index so that each chunk will always have index 0,1,2. This helps us group the article(row in df) in its corresponding column in our 3 column layout.</blockquote><ul><li>We implement a rounding-off mechanism for the predicted sentiment values, originally ranging from -1.0 to +1.0, to fit within the range of 0 to 100 and rounded to the nearest 10. This rounded value becomes a crucial factor in determining the color representation of the sentiment.</li></ul><pre>new_value = int((row[&#39;impact&#39;] + 1) * 50)<br>disp = new_value - (new_value%10)<br>bg = bgs[disp//10]</pre><ul><li>An exciting aspect of utilizing Streamlit is the ability to seamlessly combine Python code with HTML and CSS to create visually appealing UIs. In this implementation, a container is crafted, representing a card layout for displaying news articles dynamically.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HdHugW1dr2jRrhs0" /></figure><pre>&lt;div class=&quot;card&quot;&gt;<br>  &lt;img src={row[&#39;ImageURL&#39;]} alt=&quot;News Image&quot;&gt;<br>  &lt;div class=&quot;content&quot;&gt;<br>    &lt;h2&gt;&lt;a href=&quot;{row[&#39;URL&#39;]}&quot; target=&quot;_blank&quot;&gt;{row[&#39;Title&#39;]}&lt;/a&gt;&lt;/h2&gt;<br>    &lt;p class=&quot;organization&quot;&gt;{row[&#39;name&#39;]} - {row[&#39;symbol&#39;]}&lt;/p&gt;<br>    &lt;p class=&quot;sentiment&quot;&gt;AI Sentiment(ranges from -1 to 1): {row[&#39;impact&#39;]}&lt;/p&gt;<br>    &lt;p class=&quot;sentiment&quot;&gt;AI Recommendation: {row[&#39;recommendation&#39;]}&lt;/p&gt;<br>  &lt;/div&gt;<br>  &lt;div class=&quot;sentiment-container&quot;&gt;<br>    &lt;!-- Sentiment Fill --&gt;<br>    &lt;div class=&quot;sentiment-slider&quot; style=&quot;width: {disp}%; background-color:{bg};&quot;&gt;&lt;/div&gt;  <br>  &lt;/div&gt;<br>&lt;/div&gt;</pre><ul><li>The layout is defined using a &lt;div&gt; with the class card. It includes an image section at the top, populated with the image link stored. Another &lt;div&gt; with the class &#39;content&#39; follows, where textual information such as the news title, company name, AI sentiment, and recommendation are displayed.</li><li>Towards the bottom of the card, there’s a sentiment-container class. In this section, the width and background color (bg) are dynamically set based on the calculated disp variable. bg selects one of the colors defined in the list variable bgs, providing a visual representation of the sentiment. This flexible combination of Python and HTML/CSS in Streamlit facilitates the creation of aesthetically pleasing and interactive user interfaces.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*0u6a3Ff4ZE4at1QW.png" /></figure><p>A very long time I wrote some CSS. So i used a bit of ChatGPT to code it up for me. Lets dive into our card layout CSS.</p><pre>.card {<br>    display: flex;<br>    flex-direction: column;<br>    position: relative;<br>    width: 100%;<br>    height: 500px;<br>    border: 1px solid #ccc;<br>    border-radius: 8px;<br>    overflow: hidden;<br>    margin: 10px;<br>}<br><br>/* Image Styling */<br>.card img {<br>    width: 100%;<br>    height: 50%;<br>    object-fit: cover;<br>}<br><br>/* Content Styling */<br>.card .content {<br>    flex-grow: 1;<br>    padding: 15px;<br>}<br><br>.card h2 {<br>    font-size: 1.2em;<br>    margin-bottom: 10px;<br>}<br><br>.card p {<br>    font-size: 0.9em;<br>    color: #21bdf5;<br>}<br><br>.card .organization {<br>    font-weight: bold;<br>}<br><br>/* Sentiment Styling */<br>.card .sentiment-container {<br>    display: flex;<br>    position: absolute;<br>    bottom: 0;<br>    left: 0;<br>    width: 100%;<br>    height: 25px;<br>    background-color: #f2f2f2;<br>}<br><br>.card .sentiment-slider {<br>    flex-grow: 1;<br>    background-color: #4CAF50;<br>}<br><br>.card .sentiment-label {<br>    font-size: 25px;<br>    color: #44e6f8;<br>}<br><br>/* Link Styling */<br>.card a {<br>    text-decoration: none;<br>    color: #007BFF;<br>}<br><br>/* Hover Effect */<br>.card:hover {<br>    box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);<br>    transform: translateY(-2px);<br>    transition: box-shadow 0.3s, transform 0.3s;<br>}</pre><p>Lets ask the owner of the code to give us a bit of explanation. I believe ChatGPT could provide a better explanation of the above code than what I could ever do. So here is the response <a href="https://chat.openai.com/share/77b72f06-dd10-4abc-b5f5-bf5f2d02af66">https://chat.openai.com/share/77b72f06-dd10-4abc-b5f5-bf5f2d02af66</a></p><p>Next, Lets build a progress bar that displays the progress while the download, scrapping and AI sentiment collection happens in the backend</p><pre>#Home.py<br>def progress_bar():<br>    progress_text = &quot;Operation in progress. Please wait.&quot;<br>    my_bar = st.progress(0, text=progress_text)<br><br>    for percent_complete,progress_text in nt.run(date.today()):<br>        time.sleep(0.01)<br>        my_bar.progress(percent_complete, text=progress_text)<br>    time.sleep(1)<br>    my_bar.empty()</pre><p>As we could see, my_bar.progress(percent_complete, text=progress_text) is set to show progress. Hence we will need to update some part of our code from previous article to return status regulary.</p><p>We’ve successfully implemented the progress bar, but how do we obtain regular updates when our code, as discussed in the previous article, only returns data at the end? Let’s delve into a Python concept known as generators, utilizing the power of the yield keyword.</p><h3>What is yield in Python?</h3><p>yield in Python is a key player in the world of generators. Unlike the conventional return keyword, which concludes the function and delivers a single result, yield allows the creation of generators, optimizing memory usage. This is particularly valuable when dealing with extensive lists, enabling the return of results incrementally instead of all at once.</p><p>By employing yield, a function becomes a generator. When the generator function is invoked, it doesn&#39;t execute immediately; instead, it returns a generator object. The function starts executing only when the next() method is called on the generator object. At each encounter with a yield statement during execution, the function produces a value, pauses, and retains its state. Subsequent calls to next() resume the function from where it left off, facilitating a step-by-step generation of results.</p><p>Generators, being memory-efficient, are ideal for scenarios where results need to be processed iteratively rather than in a single batch.</p><p>More about generators here — <a href="https://www.geeksforgeeks.org/python-yield-keyword/">https://www.geeksforgeeks.org/python-yield-keyword/</a></p><p>In the news_tech_trader.py file, which we developed in the previous article, let&#39;s focus on the utilization of the yield keyword within the run method.</p><pre>#news_tech_trader.py (developed in last article)<br>def run(self, day):<br>    results_list = []<br>    day = date.today() if not day else day<br>    extracted_news_file = f&#39;./data/news/{day}.csv&#39;<br>    df = pd.DataFrame()<br>    if not os.path.exists(extracted_news_file):<br>        extracted_news_file = news.extract_news()<br>    df = pd.read_csv(extracted_news_file)<br>    yield 5,&quot;Download Completed..&quot;<br>    df_len = len(df)<br>    i = 0<br>    if &#39;symbol&#39; not in df:<br>        for index,row in df.iterrows():<br>            text = factory.create_and_scrape(row[&#39;URL&#39;])<br>            if text is None or len(text)&lt;10:<br>                text = str(row[&#39;Title&#39;]) + &#39; &#39; + str(row[&#39;Description&#39;])  <br>            text = palm_interface.summarize(text)<br>            data = palm_interface.prompt(text)<br>            symbol = yfi.get_symbol_from_name(data[&#39;name&#39;],data[&#39;symbol&#39;]) if data else &quot;&quot;<br>            results_list.append({<br>            &#39;symbol&#39;: symbol if symbol else &quot;&quot;,<br>            &#39;name&#39;: data[&#39;name&#39;] if data else &quot;&quot;,<br>            &#39;impact&#39;: data[&#39;impact&#39;] if data else 0.0<br>            })<br>            i+=1<br>            percentage = i/df_len*100<br>            yield int(percentage), &quot;Scrapping and AI Analyzing in progress...&quot;<br>        df2 = pd.DataFrame(results_list)<br>        df = pd.concat([df, df2], axis=1)<br><br>    yield 100, &quot;Generating Recommendations&quot;</pre><p>This code employs the yield keyword to provide regular progress updates during the execution of the run method. The yielded values include the completion percentage and informative messages for the progress bar. If you need a detailed explanation, you can refer to the previous article where the code is thoroughly explained. The focus here is on the yield part, which ensures that progress is communicated back to the parent code for display in the progress bar.</p><p>ok.. finally continuing back to our Home.py where we put together all of these in action.</p><pre>button_click = st.button(f&#39;Download Latest {date.today()}&#39;)<br>if button_click:<br>    d = None<br>    progress_bar()<br>    d = date.today()<br>    button_click = False<br><br>d = st.date_input(label=&quot;:blue[Select a date]&quot;,format=&quot;YYYY-MM-DD&quot;)<br>if d:<br>    with st.expander(&quot;News&quot;):<br>        display_news_cards(d)</pre><ul><li>In this section of Home.py, we&#39;ve introduced a button field labeled <strong>Download Latest</strong> with the current date. Clicking the button triggers the progress_bar() function, initiating the actual execution of news download, scraping, and AI sentiment analysis.</li><li>Following that, a date picker field is provided using st.date_input. Upon selecting a date, an expander section is displayed, revealing the news cards for the chosen date using the display_news_cards function we developed earlier.</li><li>This interactive setup ensures a seamless user experience, allowing users to download the latest news, track the progress through a dynamic progress bar, and explore news cards based on their preferred date.</li></ul><p>We are done writing all the code… uff…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*xehXbf4JF2VC-IH0.jpg" /></figure><p>Complete code available in github — <a href="https://github.com/vishyarjun/news_based_stock_analyzer">https://github.com/vishyarjun/news_based_stock_analyzer</a></p><h3>Output of the implementation</h3><ol><li>Cards and AI sentiment view</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBZi8Oq7H7jo&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBZi8Oq7H7jo&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FBZi8Oq7H7jo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/19525a044d4c0c11c37d56020a3a026e/href">https://medium.com/media/19525a044d4c0c11c37d56020a3a026e/href</a></iframe><p>2. Download and live progress update</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XjlNkM_MRqnZkfDtVyiIbw.png" /></figure><h3>Some Final Thoughts…</h3><p>Having recently completed the development of my personal trading portal, which encompasses market analysis, backtesting capabilities, and now, the seamlessly integrated news aggregation portal, I must say it stands out as one of the most valuable additions. The portal also boasts exciting features such as integration with the Zerodha platform for both manual and automated buy/sell actions, collaboration with Finetuned Finance LLMs, and up-to-date macroeconomic indicators and graphs.</p><p>After immersing myself in Streamlit for the past six months, navigating its capabilities alongside the mentioned features, I’ve come to regard Streamlit as one of my favorite frameworks in recent times. While at work, where we cater to a user base of several thousand users, Streamlit is primarily employed for demos. In our production systems, we have not started using streamlit yet, although we are experimenting on it carefully on its possibilities.</p><p>In my forthcoming articles, I’m excited to share a separate series detailing my experiences in building features like Single Sign-On (SSO) and attempting to create a production-like system. I’ll delve into scaling considerations and explore performance testing, all accomplished using Streamlit. Stay tuned for insights into leveraging Streamlit beyond just demos, aiming for robust and scalable applications.</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Incase of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa515486ba09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python and LLM for Stock Market Analysis Part IV — ElasticSearch for Stock Symbol/Ticker accuracy]]></title>
            <link>https://levelup.gitconnected.com/python-and-llm-for-stock-market-analysis-part-iv-elasticsearch-for-stock-symbol-ticker-accuracy-1acda81eeec3?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/1acda81eeec3</guid>
            <category><![CDATA[stock-trading]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 04:56:48 GMT</pubDate>
            <atom:updated>2024-05-21T17:44:55.498Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python and LLM for Stock Market Analysis Part IV — ElasticSearch for Stock Symbol/Ticker accuracy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*hDtW1Ff1xfAU6Qmo.png" /></figure><blockquote>This post is a continution of our <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d">previous post</a> and to be read after completion the previous.</blockquote><p>Oftentimes, a trading system is required to read a multitude of finance articles, quarterly results, or, at the very least, the recent official announcements from companies.</p><p>In our last post, we aggregated news articles, used LLM to extract the symbol and sentiment of the news, and then combined it with technical indicators. While NLP and LLM have great potential to perform all of these tasks, it’s crucial to tie every news article back to the correct organization/symbol. This ensures that we can combine the news with its fundamentals and technical data from other sources.</p><h4>Is obtaining the accurate symbol an AI problem?</h4><p>While it is common to use LLM/NLP to identify a symbol from the stock’s name,</p><ul><li>this doesn’t constitute a complete AI problem statement; rather, it can be addressed using a Software Engineering approach.</li><li>Additionally, the symbol is not something we want hallucinations to impact, and we are aware that LLMs can adeptly create fictional information.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vGmaLaFP0kYVZSq7.jpg" /></figure><p><strong>Well, then If I give the complete organization symbol database to LLM and ask to look for errors would it not solve the problem for me without making things up?</strong></p><p>Thinking about cost, scalability and performance, LLM may not be an ideal option for this problem statement.</p><p>we will utilize LLM or NLP models to get the sentiment of the news articles and extract the name of the organization. We then employ elastic search to obtain a matching trading symbol.</p><h4>What is ElasticSearch?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*CdJ7fRD_SGA_HAC7.png" /></figure><p>ElasticSearch is an open-source full-text search engine library and is built on top of Apache Lucene. The benefits are plenty as it offers text-search (even if we do not have a exact matching text to search for). It’s called <strong>Fuzzy Search. </strong>Fuzzy search allows ElasticSearch to find documents that match a given search term even if there are spelling errors or slight variations in the terms. This is especially useful when LLM/NLP models doesn’t provide us with the exact name of the organization or when the source articles itself has only a partial name or name with spelling issues. There are other benefits. we will eventually look at it as we implement.</p><blockquote>Recently Elasticsearch introduced something called Elasticsearch Query Language (ES|QL). It transforms and simplifies data investigation. The ES|QL engine delivers advanced search capabilities, improving efficiency and accelerating resolution with search and streamlined workflows.</blockquote><ol><li>First thing we need is a data source for the list of stocks. we can get it from official NSE and BSE site.</li></ol><blockquote><a href="https://www.nseindia.com/market-data/securities-available-for-trading">https://www.nseindia.com/market-data/securities-available-for-trading</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u6acKCuvaO0dwA22_xoEVA.png" /></figure><p>2. Download the file and store it in a csv — <strong><em>EQUITY_L.csv</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0HInEABtEP6i0Q5kewbIA.png" /></figure><p>2. Next thing is we need a ElasticSearch instance. I usually run such instances as a Docker container. Follow the below link to install Elastic on Docker.</p><blockquote><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html</a></blockquote><p><strong><em>Note: Just one node is sufficient unless it is a production like environment.</em></strong></p><p>3. lets write a simple script to index the stock data into elasticsearch. we need a class that will be used for both loading the data and searching the data.</p><p>In the class definition we will set some basic details of elastic configs and index name.</p><pre>#elastic_interface.py<br>import pandas as pd<br>from elasticsearch import Elasticsearch<br>from elasticsearch.helpers import bulk<br>import json<br><br>class Elastic:<br>    def __init__(self):<br>        # Elasticsearch connection settings<br>        self.es_user = &#39;elastic&#39;<br>        self.es_password = &#39;mfb6RIIrWNrg7ors2LhA&#39;<br><br>        # Create Elasticsearch connection with authentication<br>        self.es = Elasticsearch(<br>            &quot;https://localhost:9200&quot;,<br>            http_auth=[self.es_user,self.es_password],<br>            verify_certs=False<br>        )<br>        # Specify your index (without doc_type for recent Elasticsearch versions)<br>        self.index_name = &#39;stocks&#39;<br><br><br></pre><pre><br>    def load_data(self):<br>        # Read CSV file into a DataFrame<br>        csv_file_path = &#39;EQUITY_L.csv&#39;<br>        df = pd.read_csv(csv_file_path)<br><br>        # Convert DataFrame to JSON with orient=&#39;records&#39;<br>        json_data = df.to_json(orient=&#39;records&#39;)<br><br>        # Convert JSON data to a list of dictionaries<br>        documents = json.loads(json_data)<br><br>        # Use the bulk API to index the data<br>        actions = [<br>            {&quot;_op_type&quot;: &quot;index&quot;, &quot;_index&quot;: self.index_name, &quot;_source&quot;: doc}<br>            for doc in documents<br>        ]<br><br>        success, failed = bulk(self.es, actions)<br>        print(f&quot;Successfully indexed {success} documents. Failed to index {failed} documents.&quot;)<br></pre><p>Finally execute the method by initializing the object</p><pre>elastic = Elastic()<br>#elastic.load_data()</pre><p>Uncomment the line elastic.load_data() and run the file by using python3 elastic_interface.py and comment it back. we dont need to load the data more than once (unless the container needs to be started again)</p><blockquote>Note: verify_certs=False should be considered only if you are running elasticsearch locally. if running on a server, especially a production-like environment, consider CA certificates.</blockquote><p><strong><em>A short explanation of the code:</em></strong></p><ol><li>Load the csv and convert to JSON.</li><li>Loop through the JSON and prepare data for indexing as follows</li></ol><ul><li>_op_type: The operation type, which can be &quot;index,&quot; &quot;create,&quot; &quot;update,&quot; or &quot;delete.&quot; In our case, we are using &quot;index&quot; to indicate that we want to index (insert or update) a document.</li><li>_index: The name of the index where the document should be stored.</li><li>_source: The actual document data.</li></ul><p>we are using bulk API of elasticsearch library to index the data, hence we do the step 2.</p><p>3. store the success and failed results in 2 parameters that bulk api returns by default.</p><p>The out of the above script will produce an output like below</p><pre>Successfully indexed 1941 documents. Failed to index [] documents.</pre><p>Now that our Elasticsearch is equipped with all the stock symbols and names, we are prepared for the main action: identifying stock symbols from the names.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZFI58ihsKAKZoJgv.jpg" /></figure><p>But before that, it’s important to note that Yahoo Finance itself provides an API to retrieve symbols from names. However, this API may expect a generic organization name and might not account for spelling errors or hallucinations that LLMs could introduce. Additionally, it appears that this API still has some bugs. For instance, passing Indian Bank returns the symbol for South Indian Bank. They are 2 different banks! Therefore, relying solely on the Yahoo Finance search API may not be foolproof, but it can serve as a good solution that we can enhance with our own Elasticsearch.</p><p>we are going to update the file yahoo_finance.py that we created in previous article. Lets add some dependencies first</p><pre>#yahoo_finance.py<br>from elastic_interface import elastic<br>import requests</pre><p>We will create a new method get_symbol_from_name that will use both yahoo finance api and elastic search to ensure the right symbol is identified.</p><pre>#yahoo_finance.py<br><br>    def get_symbol_from_name(self, name, symbol):<br>        try:<br>            yfinance = &quot;https://query2.finance.yahoo.com/v1/finance/search&quot;<br>            user_agent = &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36&#39;<br>            params = {&quot;q&quot;: name, &quot;quotes_count&quot;: 1, &quot;country&quot;: &quot;India&quot;}<br>            res = requests.get(url=yfinance, params=params, headers={&#39;User-Agent&#39;: user_agent})<br>            data = res.json()<br>            symbol = next((row[&#39;symbol&#39;] for row in data[&#39;quotes&#39;] if row[&#39;exchange&#39;] == &#39;NSI&#39; or row[&#39;exchange&#39;] == &#39;BSE&#39;),None)<br>            symbol = next(row[&#39;symbol&#39;] for row in data[&#39;quotes&#39;]) if not symbol else symbol <br>        except:<br>            symbol = elastic.perform_search(name) if data else &quot;&quot;<br>        symbol = str(symbol) + (&#39;.NS&#39; if symbol and not(symbol.endswith(&#39;.NS&#39;)) and not(symbol.endswith(&#39;.BO&#39;)) else &#39;&#39;)<br>        return symbol</pre><p>we are making an API requests to the url <a href="https://query2.finance.yahoo.com/v1/finance/search"><em>https://query2.finance.yahoo.com/v1/finance/search</em></a></p><blockquote>Note how we have exception scenario defaulted to elasticsearch. Everytime, Yahoo finance couldn’t find a symbol for a company, it takes the exception path that leads to the elasticsearch.</blockquote><p>let’s think about few scenarios here.</p><ol><li>When The article is about a company that is not listed in the market.</li><li>When The article is about a company that is not listed in the Indian market.</li><li>When The LLM has provided the name of the company, but it doesn’t an exact match with the Yahoo Finance database, or there are multiple companies with similar names with very less ways to uniquely identify.</li><li>We have loaded only the Indian equity list into Elasticsearch, and it cannot handle companies from other countries.</li></ol><p>Considering all the above factors, once we make the API call for search, we do the below</p><pre>symbol = next((row[&#39;symbol&#39;] for row in data[&#39;quotes&#39;] if row[&#39;exchange&#39;] == &#39;NSI&#39; or row[&#39;exchange&#39;] == &#39;BSE&#39;),None)<br>symbol = next(row[&#39;symbol&#39;] for row in data[&#39;quotes&#39;]) if not symbol else symbol</pre><h4><strong>What are we really doing here?</strong></h4><p>Line 1: If the company is listed in the Indian stock market (checking BSE or NSI), we take the symbol; otherwise, we set it to None.</p><p>Line 2: If the company is not listed in the Indian stock market but is listed in some other part of the world, we still take the symbol (we’ll explore why shortly). If not, we take the exception path.</p><p>It’s noteworthy that we intentionally don’t set a default None value in line 2. We want it to take the exception path when it reaches line 2, whereas in line 1, we don’t want that because we want line 2 to execute before it can go to the exception path.</p><p>Towards the end, we deliberately add a .NS to the symbol even if it’s not an Indian company. This is because Yahoo Finance will consider such a company non-existent and will not return any data. we don’t want yahoo to return technical data when the stock is not part of BSE or NSE as our focus here is only on stocks in NSE and BSE, so we follow this approach. As a reader, if you are looking to use this for other exchanges, you might have to slightly change this here.</p><p>Note we have also imported elastic object and consumed elastic.perform_search(name) in the above code.</p><p>lets write the definition of perform_search</p><pre>#elastic_interface.py<br><br>    def perform_search(self,search_term):<br>        if not search_term:<br>            return<br>        print(search_term)<br>        q = {<br>            &quot;query&quot;: {<br>          &quot;match&quot; : {<br>           &quot;NAME OF COMPANY&quot;: {<br>            &quot;query&quot;: search_term,<br>            &quot;fuzziness&quot;: &quot;AUTO&quot;<br>                   }<br>                  }<br>                 }<br>            }<br>        search_results = self.es.search(index=self.index_name, body=q)<br>       <br>        for hit in search_results[&#39;hits&#39;][&#39;hits&#39;]:<br>            return hit[&#39;_source&#39;][&#39;Symbol&#39;]</pre><p>This is the Elasticsearch way of querying, similar to an SQL query. As you can observe, we’ve employed a fuzzy search with a value matching the field NAME OF COMPANY. Fuzzy search allows room for mistakes and errors. In case INFOSYS is misspelled as INSOFYS, fuzzy search utilizes something called Levenshtein distance, allowing for such minimal errors and we will still get the right symbol from elasticsearch as a response.</p><p>This is particularly crucial when dealing with LLMs because for the same prompt, an LLM might respond differently at different times. It might say RELIANCE LTD once and RELIANCE INDUSTRIES LIMITED another time, and we cannot rely on an exact text match. This feature is akin to what helps us find the right movie even when we type a partial or incorrect term in a Netflix search.</p><p>While there are pros to this Elasticsearch approach, there are also cons. For instance, a news article about Boeing Ltd. is identified by Elasticsearch as Bombay Dyeing!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*lywjdhBlE4-oFZQr" /></figure><p>It’s funny and embarrassing at first. Overcoming this issue took some time (DM to know what it takes to avoid such issues). To avoid such things from happening, we should ensure our search criteria allows only little errors and it is not way too much.</p><p>One last change is to incorporate the changes for Yahoo Finance search and Elasticsearch in our main file that produces news-based recommendations.</p><p>Add the following line after we summarize and extract the result from LLM(refer previous article for clarity).</p><pre>#news_tech_trader.py<br>symbol = yfi.get_symbol_from_name(data[&#39;name&#39;],data[&#39;symbol&#39;]) if data else &quot;&quot;</pre><blockquote>Note Since this file is developed as part of our <a href="https://medium.com/@viswanathan.arjun/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d">previous article</a>, please refer to it or the <a href="http://news_based_stock_analyzer/">github</a>.</blockquote><p>Now that we are done writing the code, lets see it in action. Lets run the code we created in previous article again news_tech_trader.py by executing python3 news_tech_trader.py in a terminal.</p><p>The result looks something like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Og_fsBVWxG392c80YMdhQ.png" /></figure><p>Not bad? The quality of results have largely improved from our last article as we might see the symbol predictions and the technical data has less errors than how we had when we implemented it first. we have combined news articles and some technical indicators along with some sentiment prediction with LLM so far. Here is the complete code in <a href="https://github.com/vishyarjun/news_based_stock_analyzer">github</a>.</p><p>Next, we’re assembling a user-friendly UI so that manual execution in the terminal is not required. Additionally, we’ll transition from Google BARD to other LLM models, specifically trained on finance datasets.</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Incase of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1acda81eeec3" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/python-and-llm-for-stock-market-analysis-part-iv-elasticsearch-for-stock-symbol-ticker-accuracy-1acda81eeec3">Python and LLM for Stock Market Analysis Part IV — ElasticSearch for Stock Symbol/Ticker accuracy</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Sci-Fi Screens to Living Room Reality: My Meta Quest 3 Journey]]></title>
            <link>https://medium.com/@viswanathan.arjun/from-sci-fi-screens-to-living-room-reality-my-meta-quest-3-journey-349ebd5f0891?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/349ebd5f0891</guid>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[lifestyle]]></category>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[ar]]></category>
            <category><![CDATA[meta-quest-3]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 21:02:57 GMT</pubDate>
            <atom:updated>2024-01-08T00:27:42.404Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GfGsJdY4QXBROMKj.jpg" /></figure><p>2023 Thanksgiving marked the beginning of a transformative journey for me as I acquired the Meta Quest 3, complete with the Elite Strap. Little did I anticipate that my reality would undergo a profound shift in a mere two months.</p><p>My fascination with the virtual realm was fueled by those cinematic moments where scientists seamlessly manipulated virtual interfaces, gesturing with their hands and conjuring screens that materialized before them. The world of Virtual Reality (VR) and Mixed Reality (MR) has quietly entered a revolutionary phase, mirroring the disruptive impact witnessed in the realm of artificial intelligence by ChatGPT/OpenAI.</p><p>In much the same vein, Meta has etched its place in the landscape of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). Much like how ChatGPT/OpenAI stormed the AI scene, Meta has made significant strides in reshaping our perceptions of these immersive technologies.</p><p>The days of requiring expansive physical spaces for VR headsets to function are now a relic of the past. With the Meta Quest, or Oculus as it is alternatively known, the technology seamlessly integrates into our homes. By intelligently scanning the floor space and adeptly navigating obstacles, it customizes and adapts to the available space, eliminating the need for dedicated VR setups.</p><p>Here are my favorite list of Quest 3 Apps that I almost spend most of my days on, these days.</p><h4>Immersed</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*LpEEVTDy_ncTgDT5" /></figure><p>Who doesn’t appreciate the allure of working on expansive screens? Envision being surrounded by five screens of varying sizes, each positioned according to your preferences, and having the ability to effortlessly rearrange them with just the movement of your fingers. While physical screens have their merits, the freedom to rearrange them at will is a luxury that I find particularly exhilarating, thanks to Immersed VR. Best part is, it’s free to use.</p><p>The setup process is refreshingly straightforward; within a matter of seconds, my personalized five-screen workstation comes to life. As an active participant in the stock market, I allocate one screen exclusively for charts. Simultaneously, I develop content that necessitates a vertical screen for my VS Code, alongside a dedicated browser window. Of course, my primary focus remains on the central window, housing emails and essential tasks.</p><p>Immersed offers a unique social dimension through its public rooms. Despite using the free version, I find the “space” room particularly captivating. Here, I position my screens beside a large window that provides a breathtaking view of Earth. It creates an immersive environment, allowing me to engage with my work against the backdrop of a vast cosmic scene. The potential for social interaction in public rooms is intriguing, even though widespread adoption of VR workspaces has yet to materialize fully. However, I anticipate a surge in traction for this innovative way of working sooner rather than later.</p><h4>Can it replace my laptop for work?</h4><p>My personal take is NOT YET. One notable challenge with Immersed is the issue of battery consumption. The transition from 100% to 0% transpires within a mere hour, posing a limitation for extended usage periods. Additionally, navigating the virtual keyboard proves challenging at times. I often find myself squinting through the gap to locate my hands on the keyboard, an aspect that could benefit from further refinement. My eyes find it difficult to use the VR screens for long hours.</p><h4>Fury Table Tennis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hM5s1tRovZRpDRf3.jpg" /></figure><p>Among the array of apps available in the Meta Quest store, one of my standout acquisitions is the less mainstream yet highly impressive Fury Table Tennis. While other table tennis apps boast a $29.99 price tag and a considerable fan base, I opted for the less popular Fury Table Tennis due to its exceptional physics simulation.</p><p>What sets Fury apart is its ability to mimic the real-world dynamics of playing table tennis. The nuances of striking the ball, applying spin, executing top cuts — all mirror the authentic experience of using a physical racket. Having played both in real-life table tennis matches and within the virtual realm of this app, I can attest to the seamless and immersive experience it delivers.</p><p>However, the drawback lies in the limited online community for Fury Table Tennis, making it a less favorable choice for those seeking online multiplayer interactions. Despite this, for solo play and honing one’s skills, Fury Table Tennis stands out as a captivating choice. It has effortlessly become the app that commands the majority of my time within the Meta Quest ecosystem.</p><h4><strong>Creed Boxing</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W7-8fdi6BeZ9CQgC" /></figure><p>Embarking on my inaugural VR gaming journey, I encountered what has become one of the most exhilarating experiences in the virtual realm — Creed Boxing.</p><p>Picture this: I find myself standing inside a boxing ring, surrounded by a colossal setup reminiscent of the grandeur I used to witness in WWE matches during my childhood. A sea of spectators, numbering in the thousands, adds to the electrifying atmosphere. Across from me stands an imposing figure, an 8-pack behemoth known as Rhino, ready to unleash formidable punches.</p><p>In this immersive virtual world, I discover that I, too, possess a chiseled six-pack, and my persona is none other than Creed. The objective is simple: evade the opponent’s strikes and, at the opportune moment, deliver a knockout punch to emerge victorious.</p><p>Admittedly, my initial foray into the game saw me tasting defeat. However, as I grasped the nuances of the gameplay, victories began to stack up. Now, it has become a daily ritual post-work to step into the virtual ring, engage in intense punch-and-dodge sessions, and relish the satisfaction of knocking out my opponent.</p><p>What strikes me is the unexpected sense of tranquility that follows each gaming session. I’ve come to understand why boxing holds such allure for enthusiasts. The adrenaline surge during the match is complemented by a soothing calmness afterward.</p><p>Reflecting on this, I realize that this unique and exhilarating feeling would have remained unknown to me if not for the immersive power of VR. Creed Boxing has not only introduced me to the thrill of virtual gaming but has also unraveled the captivating world of boxing in a way I never anticipated.</p><h4>Population One</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zCeP2qtv0m_nfNaD.jpg" /></figure><p>In the realm of VR gaming, one title stands out as an absolute delight — Population: One.</p><p>If you’re a fan of the adrenaline-fueled battles in PUBG, this game takes the experience to a whole new level by immersing you directly into the action. Picture yourself soaring into a world where survival hinges on your shooting skills, echoing the familiar refrain of “winner winner chicken dinner.”</p><p>In this virtual universe, you’re not just a player; you’re an integral part of the game. Face your opponents head-on, engage in intense shootouts, and navigate the expansive landscape in a quest for survival. Much like PUBG, you’ll find yourself running for cover, scrounging for resources, and engaging in exhilarating firefights.</p><p>The dynamics are familiar — with the playing field gradually shrinking, emphasizing the need to stay within the safe zone. The thrill of the chase is real, whether you’re pursuing adversaries or making a mad dash for crucial supplies.</p><p>However, it’s worth noting that, like many intense VR experiences, playing Population: One for an extended period may lead to a touch of nausea. The immersive nature of the game, combined with its fast-paced action, can certainly leave a lasting impression.</p><p>In a sea of VR games, Population: One emerges as a standout, providing an unmatched level of engagement that bridges the virtual and real worlds. If you’re seeking the rush of PUBG with an added layer of immersion, this game is undoubtedly a must-try.</p><h4><strong>Meditation</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CeN0BgagvzFwX2oI.jpg" /></figure><p>In a twist I never anticipated, my Meta Quest 3 has become an unexpected ally in my meditation journey. Surprisingly, I find myself using this gadget more for mindfulness than for gaming or other activities.</p><p>The catalyst for this unanticipated usage was an intriguing article from Meta that highlighted the potential of VR meditation apps for health and wellness (link: <a href="https://www.meta.com/blog/quest/vr-meditation-apps-health-wellness/">Meta Blog</a>). Intrigued, and being someone who regularly incorporates meditation into my routine through various apps, I decided to explore the meditative capabilities of my Meta Quest 3.</p><p>For anyone who has tried traditional meditation, the challenge of keeping one’s eyes closed for an extended period, even just 15 or 20 minutes, is well-known. However, with the Liminal App on Meta Quest 3, a refreshing departure from this norm unfolds. Instead of closing our eyes, we are encouraged to keep them wide open, focusing on the visual stimuli presented to us.</p><p>The app offers different modes, and even as a free user with limited options, I found the experience surprisingly beneficial. The ability to maintain focus with eyes open adds a unique dimension to the meditation practice, making it a valuable tool for those who may find traditional methods challenging. The Liminal App on Meta Quest 3 has proven to be a delightful surprise in my exploration of this versatile device.</p><p>I’m still trying other apps on this space and might end up writing a separate article sharing my experience using each of them.</p><h4>IMAX Theatre Experience</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*yjeeOUBzGlfaMGKf" /></figure><p>As a dedicated movie enthusiast with little interest in traditional theaters, my childhood dream of owning a home theater came to life through the BigScreen VR app. Offering the immersive grandeur of an IMAX theater or the cozy ambiance of a home setting with a giant screen, BigScreen supports Prime Video and Disney, with Netflix expected to join soon. A groundbreaking feature includes communal movie-watching rooms, enabling shared experiences globally. Despite some quirks like hate speech in the community, the potential for shared cinematic moments across continents marks a powerful new era in virtual connectivity. Now, I can enjoy movies with family members in different parts of the world, transcending geographical distances.</p><h4>International Space Station — VR Tour</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8IDyP6IQfDBh88s8.jpg" /></figure><p>Whatever one might say about this app might look like an exaggeration, but it is just not.</p><p>Take a trip into orbit and experience life on board the International Space Station! In this Emmy-nominated simulation, learn how to move and work in zero-gravity using your controllers. Dock a space capsule, take a spacewalk, and let real NASA astronauts guide you on the ISS through informative videos and images.</p><p>As the makers of the game say, <strong><em>You will never look at the world same way again.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*26n-0TE___ydHAV9P4IJ2A.png" /></figure><p>Everytime I use this app, I remember the famous saying by <strong><em>carl sagan </em></strong><a href="https://www.goodreads.com/quotes/230027-look-again-at-that-dot-that-s-here-that-s-home-that-s"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><p>If not the other apps, this one is a must try!</p><h4>World Travel with Wander</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0zLgs7qp_HbxrLGy.jpg" /></figure><p>Wander is an immersive VR app that lets users virtually explore global destinations through the convenience of their living rooms. Similar to Google Maps, the app offers an enhanced Street View experience with features like favorites, voice search, and social connectivity. Users can navigate iconic locations, share experiences with friends, and even explore a scaled-down International Space Station. Despite minor usability concerns, such as Street View loading times, Wander provides an engaging VR exploration experience, utilizing both official and user-contributed images. The app’s ongoing development promises future improvements, making it a unique and evolving platform for virtual travel.</p><h4>My VR workout routine</h4><ol><li>Litesport</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PveWp_3GR58iIvpb" /></figure><p>I use Litesport a lot because its free version is good. Ofcourse the paid version has access to more workout routines. A 15 minute punching session with one of the virtual trainer will easily burn more well above 70 calories. this is already part of my daily workout routine.</p><p>2. FIT XR</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8JepZjlxSyDY1lNG.jpg" /></figure><p>This has less free workouts and more paid workouts, but graphics and usability wise, I find it more entertaining than the other one I use. it has Zumba and other type of workouts as well.</p><h4>First Encounters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FhlNsS4Y-_sanFUu" /></figure><p>First Encounter on the Meta Quest 3 is a low-key tech demo showcasing the mixed reality capabilities of the device. The app subtly merges the digital and physical worlds, introducing users to a quiet yet intriguing blend. The environment undergoes subtle transformations, like a spaceship entering through the wall, providing a realization of stepping into mixed reality. Engaged in a subdued adventure, users shoot down threats breaking through transformed walls, offering a quiet revelation of mixed reality’s presence. First Encounter serves as an understated introduction to the Meta Quest 3’s mixed reality potential, gently initiating users into the immersive possibilities without overwhelming the senses.</p><h4>Epic Roller Coasters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pTNB3SQFd9eTRHBC.jpg" /></figure><ul><li><strong><em>Motion Sickness Alert — </em></strong>Venturing into the world of adventure rides, I decided to try the Epic Roller Coaster app, only to find myself battling motion sickness that left me unable to open my eyes for an hour. Surprisingly, my recent real-life experience on challenging roller coasters at Busch Gardens in Tampa, Florida, didn’t evoke the same discomfort.</li><li>In preparation for this article, I sought feedback from friends and their children by experimenting with this app. Interestingly, while it might not be my cup of tea, I discovered that kids absolutely love it! Virtual roller coasters have become one of my favorite ways to entertain youngsters, providing them with a thrilling experience without leaving the comfort of their virtual seats.</li></ul><h4>Is it just a Gaming console?</h4><p>While my initial purchase of the Meta Quest 3 was driven by the desire to indulge in gaming, it didn’t take long for me to recognize that this exceptional product transcends the realm of a mere gaming console. It revealed itself to be a versatile tool, offering an array of experiences beyond the gaming landscape.</p><p>Certainly, it’s a captivating space where new developments unfold daily. As part of my routine, I’ve made it a habit to read at least a couple of articles daily to stay abreast of the latest happenings in the Meta Quest world. For a regular dose of updates, I frequent <a href="https://www.meta.com/blog/quest/">Meta’s Quest blog</a>.</p><h4><strong>Addiction?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MnTm0tT1Ae0PgyL4" /></figure><p>As an adult immersed in a 50+ hour workweek, stumbling upon this gadget has been nothing short of a revelation, rivaling the excitement I felt when first introduced to my trusty PS4 years ago. Over the past two months, this remarkable device has become the focal point of my activities, leading me to practically abandon my phone and TV. While I can confidently assert that I’m no longer addicted to gaming, the Meta Quest 3 has seamlessly integrated into various aspects of my life.</p><p>It has evolved beyond a gaming console, becoming a versatile tool for work, socializing, working out, meditating, learning, watching, and even traveling, all within the confines of my home. The beauty lies in its ability to keep me connected with the outside world even as I delve into its virtual realms. With a battery life that can stretch up to a satisfying three hours on a full charge, my reliance on this gadget has reached a point where I’m contemplating acquiring a second Meta Quest 3, ensuring a seamless transition while one charges. The addiction, it seems, is not to a specific activity but to the boundless possibilities this device presents.</p><h4><strong>Are my eyes safe?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*G2RLV33_L0EkJDPo.jpg" /></figure><ul><li>As someone who have constantly being told by parents that TV screens and mobile screens are the reason why I’m wearing power glasses, using this huge VR screens that goes 360 degrees for straight to 2 to 3 hours and I feel a bit of pain around my eyes sometimes. on a longer run, I’m not sure on the impact that it may cause on sightpower.</li><li>people around me notices and hints me that my eyes looks tired and I. clearly find it difficult on my eyes if I continuously use it for several hours.</li><li>One good thing is unlike quest 2, Quest 3 has space for me to wear my power glasses, this allows me to use my glass while I use the VR headset.</li><li>Another option is to use the VR prescription lenses — this is specifically made for Meta quest 3 and cost around 50$. While I haven’t personally tried this, my friends who used it suggested that its good. <a href="https://www.zennioptical.com/p/vr-inserts-meta-quest-3/VR70003?skuId=VR7000301">https://www.zennioptical.com/p/vr-inserts-meta-quest-3/VR70003?skuId=VR7000301</a></li></ul><h4>My next big VR App Idea…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lONOF_vKevcRzQ95.jpg" /></figure><ul><li>Prepare people for things that they are afraid of. For example, personally I’m very scared of a MRI machines and Unfortunately, I had to put myself through that machine few times. If I had a VR setup that made me prepare for an experience of MRI machine, I’m sure i would have come out of the fear sooner than later.</li><li>Personally I will think in the lines of bringing some experiences to people that they may not attempt to get in real life.</li><li>A Virtual MRI or surgical experience for patients before they get the actual experience. This will help the patients prepare mentally.</li></ul><p><strong><em>If you are a VR developer and interested in building this idea, Please DM me. I would love to collaborate as I have started the prototyping for this.</em></strong></p><p>Thanks for reading it this far. If you find this post interesting or useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Incase of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=349ebd5f0891" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python and LLM for Market Analysis — Part III — Allow your trading system to react for daily news]]></title>
            <link>https://levelup.gitconnected.com/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/3310b073af8d</guid>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Sat, 09 Dec 2023 22:39:21 GMT</pubDate>
            <atom:updated>2024-05-21T17:44:58.151Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python and LLM for Market Analysis — Part III — Allow your trading System to react for Daily News</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7IgHMVndc_0mDd7p.jpeg" /></figure><p>If you’re a day trader, swing trader, or an active follower of business news, you’re likely aware of the significant impact that news can have on the markets — sometimes it’s exhilarating to ride those movements, and other times it’s frustrating if we miss the right moment to act.</p><p>While Fundamental Analysis provides a solid foundation for strategic decision-making, combining it with Technical Analysis enhances our approach. Taking it a step further, integrating current market conditions, particularly news, can provide a more comprehensive perspective. This article explores the integration of these factors into our trading decisions.</p><p>For those of us who engage in part-time trading, manually sifting through multiple sources of business news, gauging market sentiment, and making daily decisions can be challenging. While this manual approach offers valuable learning experiences, there’s a need for scalability. This is where we turn to the power of Python, LLM models, and data science to streamline the process. In this article, we’ll delve deep into this fascinating intersection.</p><ol><li>Extract Latest news articles list from news aggregator APIs.</li><li>Reach out to each news source and scrape complete articles.</li><li>Summarize the article using LLM.</li><li>Perform Sentiment Analysis using Finance LLM models (Bloomberg, PaLM Finance model)</li><li>Combine them with Technical indicators and build a recommender system.</li></ol><h4>First things First!</h4><p><strong>Are Language Models (LLMs) truly finance experts when provided with the right data? </strong>The ongoing research in this field suggests that they are not there yet. While there are some finetuned models showing promise, their reliability is not absolute. A noteworthy research paper shedding light on this is available at <a href="https://arxiv.org/pdf/2310.12664.pdf">https://arxiv.org/pdf/2310.12664.pdf</a>.</p><p>Now, the question arises: Why should we consider using LLMs for finance? In my perspective, the potential lies in the future. As time progresses, we anticipate the development of highly sophisticated models that can offer expert insights in finance. As part of this series, we’ll take a closer look at this concept. Towards the conclusion, we’ll embark on the journey of finetuning our own Finance LLM model using a dataset comprising 50,000 business news articles. This hands-on approach will showcase how a finetuned model can outperform generic models.</p><p>In the realm of financial markets, predicting reactions can be exceptionally challenging. For instance, positive results announcements may lead to profit bookings and a subsequent price fall. Conversely, a negative result, if less negative than expected, might cause a price surge. The scenarios are diverse and complex. Only models specifically trained for such nuanced cases can make more accurate judgment calls in these situations.</p><h4><strong>So what are we building?</strong></h4><p>Take a deep look at below. Our final file structure looks like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/1*BE2Yh-fGRbrUMOhxmbYwmw.png" /></figure><h4>News Aggregator API</h4><p>first we need a way to collect all the daily news at one place. This is where <a href="http://newsapi.org/">newsapi.org</a> comes in handy. we are particularly interested in their business news. Fortunately newsapi has both REST APIs and python library that we can install and use. Lets install this library using</p><pre>pip3 install newsapi-python</pre><p>Register your account and get their API Key. While the service costs for a business usage, their developer version has limited features and is free to use.</p><p>Let’s create a simple python class that extracts news articles.</p><pre>#news_api.py<br>class NewsApi:<br>    def __init__(self):<br>        self.token = os.getenv(&quot;NEWS_API_TOKEN&quot;)<br>        self.country = &#39;in&#39;<br>        self.category=&#39;business&#39;<br>        self.csv_location = &#39;./news/&#39;<br>        self.filename = f&#39;news_{date.today()}.csv&#39;<br>        self.newsapi = NewsApiClient(api_key=self.token)</pre><pre>    def extract_news(self):<br>        page = 1 <br>        total_news_count = 0<br>        all_articles = []<br>        try:<br>            while True:<br>                top_headlines = self.newsapi.get_top_headlines(<br>                                            country=self.country,<br>                                            language=&#39;en&#39;,<br>                                            category=self.category,<br>                                            page=page<br>                                            )<br>                if top_headlines.get(&#39;status&#39;)==&#39;ok&#39;:<br>                    for row in top_headlines.get(&quot;articles&quot;,None):<br>                        article = [row[&#39;publishedAt&#39;],row[&#39;title&#39;],row[&#39;description&#39;],row[&#39;url&#39;]]<br>                        all_articles.append(article)<br>                page+=1<br>                <br>                if len(all_articles)&gt;=top_headlines[&#39;totalResults&#39;]:<br>                    break<br>            self.all_articles = pd.DataFrame(all_articles,columns=[&#39;Date&#39;,&#39;Title&#39;,&#39;Description&#39;,&#39;URL&#39;])<br>            self.all_articles.to_csv(f&#39;{self.csv_location}{self.filename}&#39;)<br>        except Exception as e:<br>            print(e)</pre><p>Long story short, we start with page 1, extract all the news in a page and go to next page. we collect the news and store it in <strong><em>all_articles</em></strong> list variable. Finally we store it in a csv file. while the API results have few other columns, we are only interested in Date, Title, Description and the URL.</p><h4>Extract Complete News Articles</h4><p>When exploring newsapi.org, you might observe that news articles are truncated beyond a certain character limit. However, a saving grace is that the link to the original source is provided. This presents a crucial need to extract the complete news content. By doing so, we empower Large Language Models (LLMs) to read the entire news, enabling them to make more informed judgments on the sentiment of the news.</p><p>In our quest to harness the full potential of LLMs, the extraction of complete news articles becomes a pivotal step. This not only ensures a comprehensive understanding of the content but also allows for a more accurate analysis of sentiments, ultimately enhancing the efficacy of our trading decisions.</p><blockquote><strong><em>Is it okay to scrape news articles from the website?</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N4N2P8JM9Tds7UdR.jpg" /></figure><blockquote><em>The answer is it depends on the website’s terms and conditions. But most of the news articles can be scraped since all the information is intentionally made available to the public. All public data can be scraped as long as your scraping method does not harm the data or website owner.</em></blockquote><p>There are already really good medium articles written about scrapping news sites. I will provide the links below.</p><ol><li><a href="https://dorianlazar.medium.com/scraping-medium-with-python-beautiful-soup-3314f898bbf5">https://dorianlazar.medium.com/scraping-medium-with-python-beautiful-soup-3314f898bbf5</a></li><li><a href="https://realpython.com/beautiful-soup-web-scraper-python/">https://realpython.com/beautiful-soup-web-scraper-python/</a></li><li><a href="https://medium.com/@poojan.s/stock-market-news-scraper-using-beautifulsoup-bc7db5c75f99">https://medium.com/@poojan.s/stock-market-news-scraper-using-beautifulsoup-bc7db5c75f99</a></li></ol><blockquote>Note that each news site is different and we will need to scrape differently for different source. This is a classic use case for <a href="https://realpython.com/factory-method-python/">FactoryPattern</a>.</blockquote><p>Although I have skipped writing about news scraping, here is the my completed code for scrapping from different news articles. Watch out my <a href="https://github.com/vishyarjun/news_based_stock_analyzer">github</a> as I actively keep adding more news scraping at every update.</p><h4>Summarization and Sentiment Analysis with LLM</h4><p>While I’m personally using a PaLM2 for summarization and BloombergGPT from Huggingface Hub that I have fine tuned with a decent business news dataset for getting news sentiment, in this article I will be using PaLM2 from Google for both the purposes. with recent talks on Gemini, the best known multimodal built by Google, PaLM2 is definitely worth a try!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*00o4QENfsrrXM5rW" /></figure><blockquote>Note: This article is intended to show technical possibility of using LLM for helping trading decisions. Readers are requested to perform thorough testing and perform due diligence before using it for making realtime decisions.</blockquote><p>Let’s start by building a simple PaLM2 interface, that takes the user query and extract results from LLM through Rest API. next we will build a simple prompt that can help us extract the data in the way we want.</p><ul><li>First we will need the API key. Follow this link and get one. <a href="https://developers.generativeai.google/products/palm">https://developers.generativeai.google/products/palm</a></li><li>Make sure to store the API KEY in .env file as PALM2_API_TOKEN</li></ul><p>Installing palm2 client is quite simple</p><pre>pip3 install google-generativeai</pre><pre>#palm_interface.py<br>class PalmInterface:<br>    def prompt(self,query):<br>        try:<br>            payload = self.build_input(query)<br>            defaults = { &#39;model&#39;: &#39;models/text-bison-001&#39; }<br>            response = palm.generate_text(**defaults, prompt=payload)<br>            json_response = json.loads(response.result)<br>            return json_response<br>        except Exception as e:<br>            print(e)<br>            return None</pre><p>Time for some prompt engineering…</p><pre>def summarize(self,full_news):<br>        template3 = f&quot;You excel in succinctly summarizing business and finance-related news articles.\<br>        Upon receiving a news article, your objective is to craft a concise and accurate summary while \<br>        retaining the name of the company mentioned in the original article. The essence of the article \<br>        should be preserved in your summary. A job well done in summarizing may earn you a generous tip.\<br>        Please proceed with the provided full news article. {full_news}&quot;<br>        try:<br>            defaults = { &#39;model&#39;: &#39;models/text-bison-001&#39; }<br>            response = palm.generate_text(**defaults, prompt=template3)<br>            return response.result<br>        except Exception as e:<br>            print(e)<br>            return None</pre><p>Prompt Engineering…If this term is hitting your ears for the first time, think of Prompting as the Yoda of manifestation. The clearer your requests to the universe, paired with an unwavering belief, the more likely your desires will land in your lap. :-) But hey, it’s not just that — let’s take a breather, digest this PJ, and then promptly forget it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*5-6vOaFbO1PW2mHX.jpg" /></figure><p>This time a few-shot prompting…</p><pre>def build_input(self,article):<br>    article_1 = &quot;Tech giant Apple Inc. reports record-breaking quarterly earnings, surpassing market expectations and driving stock prices to new highs. Investors express optimism for the company&#39;s future prospects.&quot;<br>    article_2 = &quot;Alphabet Inc., the parent company of Google, faces a setback as regulatory concerns lead to a sharp decline in share prices. The market reacts negatively to uncertainties surrounding the company&#39;s antitrust issues.&quot;<br>    prompt = f&quot;&quot;&quot;<br>    Few-shot prompt:<br>    Task: Analyze the impact of the news on stock prices.<br>    Instructions: As a seasoned finance expert specializing in the Indian stock market, you possess a keen understanding \<br>    of how news articles can influence market dynamics. In this task, you will be provided with a news article \<br>    or analysis. Upon thoroughly reading the article, if it contains specific information about a company&#39;s \<br>    stock, please provide the associated Stock Symbol (NSE or BSE Symbol), the Name of the stock, and the \<br>    anticipated Impact of the news.The Impact value should range between -1.0 and 1.0, with -1.0 signifying \<br>    highly negative news likely to cause a significant decline in the stock price in the coming days/weeks, \<br>    and +1.0 representing highly positive news likely to lead to a surge in share price in the next few days/weeks.\<br>    Your response must be strictly in the JSON format.Consider the following factors while determining the impact: \<br>    The magnitude of the news, The sentiment of the news,Market conditions at the date of the news, Liquidity \<br>    of the stock, The sector in which the company operates, The JSON response should include the keys: symbol, \<br>    name, and impact. Do not consider indices such as NIFTY. If the news is not related to the stock market or any \<br>    specific company, leave the values blank. Do not invent values; maintain accuracy and integrity in your response.\<br>        Examples:<br>    1. Article: &quot;{article_1}&quot;<br>    Response: {{&quot;symbol&quot;: &quot;AAPL&quot;, &quot;name&quot;: &quot;Apple Inc.&quot;, &quot;impact&quot;: 0.9}}<br><br>    2. Article: &quot;{article_2}&quot;<br>    Response: {{&quot;symbol&quot;: &quot;GOOGL&quot;, &quot;name&quot;: &quot;Alphabet Inc.&quot;, &quot;impact&quot;: -0.5}}<br><br>    3. Article: &quot;{article}&quot;<br>    Response: <br>    &quot;&quot;&quot;<br>    return prompt</pre><p>well there is definitely better prompting techniques available than the one above.</p><p>A good read for prompting techniques — , check out these and modify as needed.</p><p><a href="https://developers.generativeai.google/guide/prompt_best_practices">https://developers.generativeai.google/guide/prompt_best_practices</a></p><p>Some examples to get inspired from <a href="https://developers.generativeai.google/prompt-gallery">https://developers.generativeai.google/prompt-gallery</a></p><h4>Develop Trading Strategy</h4><p>So far we have seen things in bits and pieces. It’s time to connect them together.</p><p>we are going to develop a very simple strategy using a news sentiment and RSI.</p><p><strong>Strong Buy</strong> — If a news has positive sentiment and current RSI &lt; 35 (over sold zone)</p><p><strong>Buy</strong> — if a news has a positive sentiment</p><p><strong>Sell</strong> — if a news has a negative sentiment</p><p><strong>Strong Sell </strong>— If a news has a negative sentiment and Current RSI &gt; 75 (Over bought zone)</p><blockquote>The strategy is pretty basic and shouldn’t be used as is, the intent of this post is to combine news sentiment and technical indicators and not to recommend a strategy itself. readers are requested to develop their own proven strategy and back test it. Sometimes a bad news on a fundamentally strong companies will give us opportunity to buy. negative sentiments can sometimes become buying opportunities too.</blockquote><p>Let’s put this strategy in action. A simple class with 4 categories.</p><pre>#news_tech_trader.py<br>from news_api import news<br>import pandas as pd<br>from palm_interface import palm_interface<br>from yahoo_finance import yfi<br>import os<br>from datetime import date<br>from news_scrapper import factory<br>class NewsTrader:<br>    def __init__(self):<br>        self.strong_buy = []<br>        self.buy = []<br>        self.strong_sell = []<br>        self.sell = []</pre><p>The imports might confuse you a bit. don’t worry. take a look at the <a href="https://github.com/vishyarjun/news_based_stock_analyzer">github</a> and it will look fine.</p><p>A helper method to obtain RSI for a given Symbol.</p><pre>    def get_rsi(self, symbol):<br>        row = yfi.download_data(symbol,&#39;./data/&#39;)<br>        print(row)<br>        if row is None:<br>            return 50.0<br>        else:<br>            return row[&#39;RSI14&#39;] if row[&#39;RSI14&#39;] else 50.0</pre><p>Since we have already discussed about download historical data, calculating the RSI etc in the earlier part of our series, we will skip that topic in this article. Refer to <a href="https://github.com/vishyarjun/news_based_stock_analyzer">github</a> for how this is done. Also note how we are defaulting the RSI to 50 when RSI is not available. we are setting it to a neurtral value making it neither over sold nor over bought.</p><p>Now connecting the pieces together…</p><pre>def run(self):<br><br>    # PART - I - Check below explanation<br>    results_list = []<br>    extracted_news_file = f&#39;./data/news/{date.today()}.csv&#39;<br>    df = pd.DataFrame()<br>    if os.path.exists(extracted_news_file):<br>        df = pd.read_csv(extracted_news_file)<br>    else:<br>        extracted_news_file = news.extract_news()<br>        df = pd.read_csv(extracted_news_file)<br>        <br>    # PART - II - Check below explanation<br>    if &#39;symbol&#39; not in df:<br>        for index,row in df.iterrows():<br>            text = factory.create_and_scrape(row[&#39;URL&#39;])<br>            if text is None or len(text)&lt;10:<br>                print(&#39;scrape not successful!&#39;)<br>                text = str(row[&#39;Title&#39;]) + &#39; &#39; + str(row[&#39;Description&#39;])<br>            text = palm_interface.summarize(text)<br>            data = palm_interface.prompt(text)<br>            results_list.append({<br>            &#39;symbol&#39;: data[&#39;symbol&#39;] if data else &quot;&quot;,<br>            &#39;name&#39;: data[&#39;name&#39;] if data else &quot;&quot;,<br>            &#39;impact&#39;: data[&#39;impact&#39;] if data else 0.0<br>            })<br>        df2 = pd.DataFrame(results_list)<br>        df = pd.concat([df, df2], axis=1)<br>        df.to_csv(extracted_news_file, index=False)<br>    <br>    # PART - III, IV - Check below explanation<br>    for index, row in df.iterrows():<br>        if not pd.isna(row[&#39;symbol&#39;]) and len(row[&#39;symbol&#39;])&gt;0:<br>            rsi = self.get_rsi(row[&#39;symbol&#39;])<br>            if row[&#39;impact&#39;] &gt; 0.4:<br>                if rsi &lt;=35:<br>                    self.strong_buy.append(row[&#39;symbol&#39;])<br>                else:<br>                    self.buy.append(row[&#39;symbol&#39;])<br>            elif row[&#39;impact&#39;] &lt; -0.4:<br>                if rsi &gt;=75:<br>                    self.strong_sell.append(row[&#39;symbol&#39;])<br>                else:<br>                    self.sell.append(row[&#39;symbol&#39;])</pre><p>Don’t get overwhelmed by long lines of code. its actually easier than how it looks. here is whats happening..</p><ol><li>Read the news CSV file if it exists. if not download the news articles in a csv first.</li><li>Iterate through each article, Scrape the full article as needed and inference the palm API. concat the inference results with the news df itself. store it back to the same file</li><li>iterate through each article again. get the RSI value of each stock by calling <strong><em>get_rsi</em></strong>.</li><li>Depending on the category it falls, store in a list. For some of the articles the Symbol may not be available or incorrect. Its just fine for now, in one of the coming articles we will be looking at how to get rid of such issues by using ElasticSearch and ensure the extraction results from LLM are proper.</li></ol><p>Complete code is available on <a href="https://github.com/vishyarjun/news_based_stock_analyzer">Github</a>. Please star or fork the project if you like.</p><h4>Conclusion</h4><p>One of my personal favorite quote goes like</p><blockquote><em>Great investment opportunities come around when excellent companies are surrounded by unusual circumstances that cause the stock to be misappraised.</em></blockquote><blockquote>~ <strong>Warren Buffett</strong></blockquote><p>And what better way to understand a company’s current circumstances than through its latest news?</p><p>Our strategy implementation here is refreshingly simple, yet the real excitement unfolds when we blend the power of news and technical analysis with fundamental insights. In the upcoming articles, we’ll leverage the Screener API to add a layer of complexity in real-time. For instance, consider the lasting impact of news such as positive revenue or an increase in the order book — it often extends beyond a single day in the market. Conversely, negative news can influence stock performance for weeks. So, why limit ourselves to just one day of news? In an engaging exercise for our readers, we encourage exploring multiple days of news for each company, averaging sentiment, combine the volume and experiment to find that sweet spot. Trust me, it’s a fascinating journey.</p><p>In the next set of articles, we’ll delve into fine-tuning Language Models (LLMs) with historical news data, aligning them with actual price movements and considering macroeconomic conditions. We’ll explore the process of publishing and testing the model to assess its performance. Additionally, we’ll venture into scraping Twitter feeds for listed companies, integrating essential fundamental data such as Debt-to-Equity ratio (D/E), Return on Equity (ROE), Return on Capital Employed (ROCE), Price-to-Earnings ratio (P/E), and more. To enhance the recommendation system further, we’ll incorporate volume movement analysis. Get ready for an exciting journey as we build a more valuable and comprehensive recommendation system together.</p><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Incase of queries or details, Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3310b073af8d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/python-and-llm-for-market-analysis-part-iii-allow-your-trading-system-to-react-for-daily-news-3310b073af8d">Python and LLM for Market Analysis — Part III — Allow your trading system to react for daily news</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Write Your own jq tool in python]]></title>
            <link>https://medium.com/@viswanathan.arjun/write-your-own-jq-tool-in-python-9c4462716dc7?source=rss-2c22a3e01083------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c4462716dc7</guid>
            <category><![CDATA[tdd]]></category>
            <category><![CDATA[jq]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arjun]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 04:29:00 GMT</pubDate>
            <atom:updated>2023-11-06T04:29:00.733Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*rlZebvQrnjdYVZ3GP_aEog.png" /></figure><p>jq is a lightweight and powerful command-line tool used for handling JSON parsing, making it an indispensable tool for JSON data manipulation. It can parse, filter, transform, and pretty-print JSON data with ease. In this article, we will delve into a Python implementation of a jq-like tool that accomplishes two key tasks:</p><ol><li>Parsing command-line arguments and implementing JSON data filtering.</li><li>Pretty-printing JSON data, character by character.</li></ol><p>This challenge was originally posted on <a href="https://codingchallenges.fyi/challenges/challenge-jq/">Write Your own jq</a> in Coding Challenges. It stands as a classic algorithmic problem, and tackling it provides valuable insights into real-time product development. I recommend reading this article after thoroughly understanding the problem statement or follow along for a comprehensive understanding of our implementation.</p><p><strong><em>Test-Driven Development (TDD): </em></strong>This project is an excellent example of Test-Driven Development (TDD) in action. Before attempting to find a solution, we need to think about and establish test cases. This approach is crucial because we’ll be addressing a wide range of scenarios, ensuring a robust and reliable solution.</p><p>Below are some of our test cases</p><pre>curl -s &#39;https://dummyjson.com/quotes?limit=2&#39; | ccjq<br>curl -s &#39;https://api.github.com/repos/CodingChallegesFYI/SharedSolutions/commits?per_page=3&#39; | ccjq &#39;.[0]&#39;<br>curl -s &#39;https://dummyjson.com/quotes?limit=2&#39; | ccjq &#39;.quotes&#39;<br>curl -s &#39;https://dummyjson.com/quotes?limit=2&#39; | ccjq &#39;.code&#39;<br>curl -s &#39;https://api.github.com/repos/CodingChallegesFYI/SharedSolutions/commits?per_page=3&#39; | ccjq &#39;.[0] | .commit.message&#39;<br>curl -s &#39;https://dummyjson.com/quotes?limit=2&#39; | ccjq &#39;[.quotes[].quote]&#39;</pre><p>Lets take the first command for an example. curl Command: It initiates an HTTP request to a specific URL (&#39;https://dummyjson.com/quotes?limit=2&#39;). The -s flag suppresses the progress meter. The result of this curl command serves as input for the next component.</p><ol><li>Pipe Symbol (|): The pipe symbol (|) signifies the transfer of output from the previous command to the subsequent one.</li><li>ccjq Command: ccjq is our custom Python implementation of jq. It processes the data received from curl. It&#39;s important to note that this ccjq command can have optional filtering criteria provided as arguments.</li></ol><p>So, at this point, it’s clear that the ccjq command will need to handle two key inputs:</p><ul><li>A JSON output obtained from the curl command.</li><li>Optional filtering criteria, which may be specified in various formats.</li></ul><p>Before delving into the technical intricacies of our Python implementation, it’s essential to address a fundamental question: How do we enable a Python program to execute seamlessly when triggered via a command in the terminal?</p><p>Traditionally, we’re accustomed to running Python scripts using the python my_file.py syntax. However, in this context, our objective is to ensure that a Python program can be invoked as a command from the terminal. This requires configuration within your shell environment, typically achieved by modifying the .bash_profile or .zshrc files, depending on your shell of choice. use nano or vim editor and edit the file</p><pre>sudo nano ~/.bash_profile</pre><p>or</p><pre>sudo nano ~/.zshrc</pre><p>add the following part to the bottom of the file.</p><pre>ccjq() {<br>    # Read the input from stdin<br>    input=$(cat)<br><br>    # Execute your Python program, passing the input and the filter as arguments<br>    python3 &#39;/Users/Arjun/Documents/jq clone/ccjq.py&#39; &quot;$input&quot; &quot;$1&quot;<br>}</pre><p>This is a simple script that says when ccjq command is used, execute the python file ccjq.py, look at how we pass two arguments to this,</p><p>$input which is output of the curl command being passed into ccjq</p><p>$1 which is what user adds along with ccjq command for the filters. one example coud be <em>ccjq ‘[.quotes[].quote]’.</em></p><p>Now that our basic setup is ready, let’s dive into the fun part, the algorithm.</p><h4>Part I — Parsing the filter argument</h4><p>while the actual jq tools deals with variety of scenarios, in this implementation we are going to focus on handling below argument types</p><pre>ccjq &#39;[.quotes[].quote]&#39;<br>ccjq &#39;[.quotes[1].quote]&#39;<br>ccjq &#39;.quotes[1].quote&#39;<br>ccjq &#39;.codingchallenge&#39;<br>ccjq &#39;.codingchallenge?&#39;<br>ccjq &#39;.[&quot;codingchallenge&quot;]&#39;<br>ccjq &#39;.[&quot;codingchallenge&quot;]?&#39;<br>ccjq &#39;.[0]&#39;<br>ccjq &#39;.quotes&#39;<br>ccjq &#39;.[0] | .commit.message&#39;</pre><p>Lets observe each of the argument more carefully</p><p>1. <strong><em>ccjq ‘[.quotes[].quote]’</em></strong><br> — This `jq` command starts with an array filter `[…]`, which is used to create a new array.<br> — It then uses the `.quotes[].quote` path to select and extract the “quote” field from all elements in the “quotes” array. The `[]` notation is used to iterate through all elements in the “quotes” array.</p><p>2. <strong><em>ccjq ‘[.quotes[1].quote]’</em></strong><br> — This `jq` command creates a new array and extracts the “quote” field from the second element (index 1) of the “quotes” array. It uses the path `.quotes[1].quote`.</p><p>3. <strong><em>ccjq ‘.quotes[1].quote’</em></strong><br> — This `jq` command directly selects and extracts the “quote” field from the second element (index 1) of the “quotes” array without creating a new array.</p><p>4. <strong><em>ccjq ‘.codingchallenge’</em></strong><br> — This `jq` command selects and extracts the value of the “codingchallenge” field in the JSON object.</p><p>5. <strong><em>ccjq ‘.codingchallenge?’</em></strong><br> — The `?` operator is used to check if the “codingchallenge” field exists. If it exists, it selects and extracts its value. If it doesn’t exist, it produces no output.</p><p>6. <strong><em>ccjq ‘.[“codingchallenge”]’</em></strong><br> — This is an alternative way to select and extract the value of the “codingchallenge” field using square brackets notation.</p><p>7. <strong><em>ccjq ‘.[“codingchallenge”]?’</em></strong><br> — Similar to the previous command, this checks if the “codingchallenge” field exists and extracts its value if it does. question mark means if its not present, return null</p><p>8. <strong><em>ccjq ‘.[0]’</em></strong><br> — This command selects and extracts the first element of the top-level array.</p><p>9. <strong><em>ccjq ‘.quotes’</em></strong><br> — This command selects and extracts the entire “quotes” array.</p><p>10. <strong><em>ccjq ‘.[0] | .commit.message’</em></strong><br> — This command first selects the first element of the top-level array and then selects the “commit” object from that element and, finally, extracts the “message” field from the “commit” object.</p><p>Now let’s take our observation to next level, by closely looking how does each argument starts and how it ends, there by deriving how we can write our algorithm</p><ol><li>Pipe symbols are used to separate filtering, output of one filter will given to next level.</li><li>it always starts with a dot(.) symbol and it can nest to many levels</li><li>if it starts with open square bracket [ and ends with ] then a list result is expected to be provided. carefully note we have . after [ and also its vice versa — example 1 vs example 6. both are expected to different things</li><li>if it starts with open square bracket [ and ends with ],then we have 2 possibilities it can either be something like [0] or something like [&quot;codingchallenges&quot;] or even just [] meaning loop through every possibility in the given array.</li><li>Usually filters are separated by | or . but in case of quotes[1], we have 2 filters to be applied. extract quotes attribute and extract its 1st index.</li><li>if it ends with ? , it just means return null if it doesn’t have any valid data.</li><li>there are cases where a filter doesn’t starts with [ but can end with ] example .quotes[1]</li></ol><p>In a good algorithmic problem, listing such observation is extremely critical, only after this step, we may be able to able to write an optimal solution. Hence my recommendation to readers who are at early stages of dealing with algorithmic problems is, dedicate some good amount of time for this step. Sometimes we may not be able to observe all of it at one glance. hence we may need to go back and forth between listing our observation and solving it and back and forth.</p><p>Now with all the observations, let’s think about a parsing solution.</p><pre>    def parse_argument(self,ccjq_arg):<br>        return_as_list = True if ccjq_arg.startswith(&#39;[&#39;) and ccjq_arg.endswith(&#39;]&#39;) else False<br>        ccjq_arg = ccjq_arg[1:-1] if return_as_list else ccjq_arg<br>        data = json.loads(self.json_str)<br>        for filter in ccjq_arg.split(&#39;|&#39;):<br>            data = self.apply_filter(data,filter)<br>        return [data] if return_as_list else data</pre><p>our first method</p><ul><li>to capture if a list is expected as output, if so enclose our filtered data in a list</li><li>split the filter criteria that may have pipe | symbol and loop through each criteria. as you may have observed, we are calling apply_filter to deal with next level of filters</li></ul><pre>    def apply_filter(self,data,filter): <br>        if filter==&#39;.&#39;:<br>            return data<br>        for single_filter in filter.split(&#39;.&#39;)[1:]:<br>            single_filter = single_filter.strip()<br>            if single_filter.endswith(&#39;?&#39;):<br>                single_filter = single_filter[:-1]<br>            <br>            if single_filter.startswith(&#39;[&#39;) and single_filter.endswith(&#39;]&#39;):<br>                single_filter = single_filter[1:-1]<br>                data = self.extract_data(data,single_filter)<br>            <br>            elif str(single_filter).endswith(&#39;]&#39;):<br>                filter1, filter2 = single_filter.strip(&#39;]&#39;).split(&#39;[&#39;)<br>                data = self.extract_data(data,filter1)<br>                if filter2:<br>                    data = self.extract_data(data,filter2)<br>            else:<br>                data = self.extract_data(data,single_filter)<br>        return data</pre><p>on the next level of filter, we split the filter criteria by dots. our aim here is to simplify the filter in such a way that we can use it in python json and perform extraction, hence we are going to remove special characters and keep only the text and integer attributes.</p><ul><li>ends with ?</li><li>starts with [ and ends with ]</li><li>doesn’t start with [ but ends with ]</li><li>[] with text filter and [] with integer index filter</li></ul><p>Finally the actual extraction of data happens inside the method extract_data</p><pre>    def extract_data(self,data,criteria):<br>        if criteria.isdigit():<br>            criteria = int(criteria)<br>        if isinstance(criteria,int):<br>            data = data[criteria] if 0 &lt;= criteria &lt; len(data) else None<br>        else:<br>            if isinstance(data,list):<br>                data = [row.get(criteria,None) for row in data]<br>            else:<br>                data = data.get(criteria,None)<br>        return data</pre><p>The code is self explanatory, Look at how we deal with different data and criteria types here.</p><ul><li>criteria can be a string</li><li>criteria can be a integer(index)</li><li>data can be a list</li><li>data can be a json dict.</li><li>criteria index can be within or out of datalimits</li></ul><h4>Part II — Pretty Printing</h4><p>I was almost tempted to use Python’s built-in pretty-printing features at this point. However, I decided to embrace the challenge of pretty-printing as well. So, we’ll work with JSON data in string form, and focus on managing spacing, handling brackets, and adding line breaks to achieve the tidy and organized JSON formatting that jq is famous for.</p><p>Let’s start with listing our observations again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wkU8ZnfiTSrFqBkqnNlWyw.png" /></figure><ul><li>After every open brace or close brace, there is a line break.</li><li>After every open brace, indentation level increases.</li><li>After every close brace, indentation level decreases.</li><li>If the next character is a close brace, then there is always a line break.</li><li>comma can be a last character in a line, but if its between text then such comma shouldn’t be considered as end of a line.</li></ul><blockquote><strong><em>Note: make as many observations as possible here, validate it with different examples, this exercise is the core for our problem solving. And If I have missed something important, highlight it in my medium or linkedin post. would love to know your thoughts.</em></strong></blockquote><p>Using above observation, we are going to iterate character by character of a json string and pretty print it.</p><p>Lets start with a class</p><pre>class CCJQ:<br>    def __init__(self,json_str):<br>        self.comma = &#39;,&#39;<br>        self.spl_char = [&#39;{&#39;,&#39;}&#39;,&#39;[&#39;,&#39;]&#39;]<br>        self.open = [&#39;{&#39;,&#39;[&#39;]<br>        self.close = [&#39;}&#39;,&#39;]&#39;]<br>        self.quotes = [&#39;\&#39;&#39;,&#39;\&quot;&#39;]<br>        self.level = 0<br>        self.json_str = json_str<br>        self.default_space = 2</pre><p>Pay attention to how we’re establishing definitions for open, close, spl_charand default indentation levels. One crucial variable, level adapts continuously as we process each line and character.</p><p>As you’ve likely realized, we require a helpful spacer method to ensure proper spacing for each line.</p><pre>   def leave_space(self):<br>        return &quot; &quot; * self.level</pre><p>Now, it’s time to delve into our primary implementation for pretty printing. Dealing with extensive lines of code can be quite a hassle. Therefore, we’ll divide the entire code into manageable parts for a smoother and more organized process.</p><pre>    def custom_pprint(self):<br>        idx = 0<br>        curr = &#39;&#39;<br>        while idx &lt; len(self.json_str):<br>            # all the logic comes here, refer part I, II below<br>            idx+=1<br>        print(curr)<br>        return 0</pre><p>Our high-level code structure takes on this form: we systematically iterate through each character, storing it in curr. For a detailed breakdown of what transpires within the while loop, let’s look at each part below.</p><p><strong><em>Part-I — handling quotes</em></strong></p><pre>if  self.json_str[idx] in  self.quotes:<br>                text =  &#39;&#39;<br>                idx+=1<br>                while self.json_str[idx] not in self.quotes:<br>                    text+= self.json_str[idx]<br>                    idx+=1<br>                curr+=&#39;\&#39;&#39;+text+&#39;\&#39;&#39; </pre><p>As we anticipate, the text will always be enclosed between single or double quotes. So, whenever we encounter a quote, we gather all the text until we find its matching closing quote. That’s the essence of our approach in this section.</p><p><strong><em>Part-II — Open braces</em></strong></p><pre>elif self.json_str[idx] in self.open:<br>     curr+= self.json_str[idx]<br>     self.level+=self.default_space<br>     if idx+1 &lt; len(self.json_str):<br>         curr += &#39;\n&#39;<br>         curr+=self.leave_space()</pre><p>Whenever we come across an open brace, we include it in the ‘curr’ variable, which holds all the text. We then add 2 spaces for subsequent lines. Our trusty ‘self.level’ keeps tabs on how we space each line. We’ve set our default indentation to 2 spaces, but you can modify it to your liking. After adding the open brace, we introduce a line break and apply the proper spacing to the next line, achieved with the assistance of our ‘leave_space()’ utility.</p><p><strong><em>Part-III — close braces and else</em></strong></p><pre>elif self.json_str[idx] in self.close:<br>    if self.json_str[idx-1] not in self.close:<br>        curr += &#39;\n&#39;<br>    self.level-=self.default_space<br>    curr+=self.leave_space()<br>    curr+= self.json_str[idx]<br>    if idx+1 &lt; len(self.json_str) and self.json_str[idx+1]!=self.comma:<br>        curr += &#39;\n&#39;<br>        curr+=self.leave_space()<br>else:<br>     curr+= self.json_str[idx]</pre><p>We’re executing similar actions as we do with open braces in this section. However, when dealing with a closing brace, there’s a nuance to be mindful of. If a comma follows the closing brace, we avoid a line break and instead add the comma on the same line as the closing brace before proceeding to the next line break. This specific scenario is managed in the next part of our implementation.</p><p>In the else case, if the character isn’t one of the special characters, we straightforwardly append it to curr and continue with the process.</p><h4><strong>Part IV — handling commas</strong></h4><pre>if self.json_str[idx] == self.comma:<br>    curr += &#39;\n&#39;<br>    if self.json_str[idx-1]!=self.close:<br>        curr+=self.leave_space()</pre><p>Following a comma, a line break is consistently applied, as we’ve noticed in our observations. When we previously implemented the handling of a closing brace, we’d already addressed spacing for the subsequent lines. Here’s an interesting bit: a comma can occur both with and without a preceding closing brace. We cleverly avoid redundant spacing when encountering a closing brace.</p><blockquote>The overall code does a little bit more than what we discuss here, the context of this post is to only cover the core logic of this implementation. Feel free to take a look at the complete code in <a href="https://github.com/vishyarjun/jq_python/">github</a>.</blockquote><p>Thanks for reading it this far. If you find this post useful, please leave a clap or two, or if you have suggestions or feedbacks, please feel free to comment, It would mean a lot to me!</p><p>Please feel free to connect with me on <a href="https://www.linkedin.com/in/arjunvv/">LinkedIn</a> or <a href="https://twitter.com/vishyarjun">X(formerly twitter)</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c4462716dc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>