<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Victor on Medium]]></title>
        <description><![CDATA[Stories by Victor on Medium]]></description>
        <link>https://medium.com/@vicpal?source=rss-d2e2a5174563------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*36S_f7UAyGhcX974YsMMLA.jpeg</url>
            <title>Stories by Victor on Medium</title>
            <link>https://medium.com/@vicpal?source=rss-d2e2a5174563------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:31:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vicpal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Passing the AWS Cloud Practitioner certification exam using only free materials]]></title>
            <link>https://medium.com/@vicpal/passing-the-aws-cloud-practitioner-certification-exam-using-only-free-materials-9cf1a36f0df6?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cf1a36f0df6</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[anki]]></category>
            <category><![CDATA[cloud-practitioner-exam]]></category>
            <category><![CDATA[free]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 05:23:26 GMT</pubDate>
            <atom:updated>2025-03-07T05:23:26.803Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIvM6-pdIEV7ItT-tWP7bQ.jpeg" /><figcaption>Obligatory AI Image</figcaption></figure><h3>Introduction</h3><p>I recently passed the <a href="https://aws.amazon.com/certification/certified-cloud-practitioner/?ch=sec&amp;sec=rmg&amp;d=1">AWS Cloud Practitioner certification exam</a> with a score of 838 using only free materials. For reference a passing score is &gt;700. The entire process took me 32 hours spread over one month. While I could have compressed this into about 10 days at 3 hours of study per day, I preferred a slower pace to better retain the information.</p><h3><strong>Materials</strong></h3><p><a href="https://explore.skillbuilder.aws/learn/courses/134/aws-cloud-practitioner-essentials/lessons">Official AWS free course</a></p><p><a href="https://d1.awsstatic.com/training-and-certification/docs-cloud-practitioner/AWS-Certified-Cloud-Practitioner_Exam-Guide.pdf">Official AWS Cloud Practitioner Exam guide</a></p><p><a href="https://apps.ankiweb.net/">Anki</a> (flashcards with memory — free desktop version)</p><p><a href="https://chatgpt.com/">ChatGPT</a> / <a href="https://claude.ai/new">Claude</a> (free versions only)</p><p><a href="https://github.com/kananinirav/AWS-Certified-Cloud-Practitioner-Notes/blob/master/practice-exam/practice-exam-1.md">Practice Tests found on Github</a></p><h3>Background</h3><p>Although I have a data science degree, I had little to no experience with cloud tools before preparing for this exam. I learned something (?) about AWS at some point, but had only faint memories of it when I started this journey.</p><p>Why does that matter? AWS recommends having 6 months of exposure to the cloud before taking this exam. However, I discovered that practical cloud experience isn’t necessary to pass. In fact, I came across a <a href="https://www.youtube.com/watch?v=s8byh4MCJYw">YouTuber who mentioned that despite years of experience they still couldn’t pass this test</a> without dedicated preparation. But for him it only took 5 days of prep to pass given he was already getting a 50% on the mock exams.</p><p>This guide is for people starting at 0 or near 0 knowledge of the cloud and who are looking to take the test after studying 20–30 hours.</p><h3>Study Plan Breakdown</h3><p>Here’s how I spent my 32 hours of preparation:</p><ul><li><strong>12.5 hours</strong>: Learning core material from the official free AWS course + note-taking</li><li><strong>2.5 hours</strong>: Studying topics not covered in the official course but included in the exam guide</li><li><strong>13.5 hours</strong>: Reviewing material using Anki flashcards</li><li><strong>3.5 hours</strong>: Completing 7 practice tests (30 minutes each)</li></ul><h3>Weekly Schedule</h3><p>My study plan was spread across four weeks:</p><ul><li><strong>Week 1</strong>: Official AWS free course + Review</li><li><strong>Week 2</strong>: Official AWS free course + Review</li><li><strong>Week 3</strong>: Official AWS free course + Review</li><li><strong>Week 4</strong>: Practice Tests found on GitHub + Review</li></ul><h3>Study Resources</h3><h4>Official AWS Free Course</h4><p>The official AWS free course provided the foundation of my learning. This resource covered the majority of exam topics and is designed specifically for the Cloud Practitioner certification. It is very well done course with videos and blurbs. It contains 11 modules and I recommend it to everyone as the start point for learning about AWS. Here’s how the modules worked out time-wise:</p><ul><li><strong>Mod 1–3</strong>: 4 hours</li><li><strong>Mod 4–6</strong>: 6 hours</li><li><strong>Mod 7–11</strong>: 2.5 hours</li></ul><p>As you can see, the modules are a bit middle heavy with the final modules being much faster to finish. Either that or I got a little hasty at the end.</p><p>Amazon claims these modules take about 7 hours to complete. I assume they mean 7 hours without taking any notes or pausing to replay things that were hard to understand at first.</p><p>Since I was using an Anki flashcard deck to review the common concepts irrespective of the concepts I was currently learning, I felt more comfortable with the final topics more than I did with the beginning topics. This probably also added to why I was able to go through the final modules a little faster. I’ll talk about the flashcards in detail a bit later.</p><h4>Official AWS Cloud Practitioner Exam Guide</h4><p>Before we talk about the exam guide, let me tell you about the free <a href="https://explore.skillbuilder.aws/learn/courses/14050/aws-certified-cloud-practitioner-official-practice-question-set-clf-c02-english">official practice question set</a>. After having studied for 3 weeks and practicing all the major concepts everyday, I was expecting a passing score of 80% on this exam.</p><p>Nope.</p><p>I did terrible on this exam getting a 65% percent. But there were only 20 questions, so if there were more questions I may have got a passing 70%.</p><p>Anyways…</p><p>The reason I did not pass this practice exam was because there were topics covered in this practice exam that were not covered in any of the modules (at least to the best of my goldfish memory anyways 🐠 — there aren’t any emojis of goldfish apparently so this fish will do).</p><p>So where did these mysterious topics come from? Well, if you take a look at the <a href="https://d1.awsstatic.com/training-and-certification/docs-cloud-practitioner/AWS-Certified-Cloud-Practitioner_Exam-Guide.pdf">Official AWS Cloud Practitioner Exam guide</a> you’ll notice that near the end (page 15 of 22), there is a list of “in-scope services”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKq5Yrkrs7i8laJUAX5x8A.png" /><figcaption>In-scope services covered on the exam but which may not be covered in the free course modules</figcaption></figure><p>These in-scope services are plentiful and about 1/3 (?) of them are nowhere to be seen in the official free course. So, the exam guide was essential for identifying topics not covered in the free course. I then used ChatGPT, Claude, and Anki to deal with all of these topics in about 2.5 hours. We’ll talk about that in the Claude section.</p><h3>Practice Tests from GitHub</h3><p>In the final week, I focused on practice tests found on GitHub. These were great for:</p><ul><li>Identifying knowledge gaps</li><li>Getting comfortable with the exam format</li><li>Building test-taking stamina</li><li>Learning to deal with ambiguous questions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9Yw0hJyPm6gqpp72bWvyg.png" /><figcaption>Practice exam 1 on GitHub</figcaption></figure><p>Overall, I ended up taking exams 1–4 and 21–23. Each exam took me about 30 minutes to complete the 50 questions. But how did I know if these exams were even credible?</p><blockquote>“I have been scoring 70–80% on these tests and scored 790 in the exam.”</blockquote><p>Source: Some random <a href="https://www.reddit.com/r/AWSCertifications/comments/1bxihxd/aws_cloud_practitioner_exam/">post on Reddit</a>.</p><p>I was also scoring between 70~80% on these exams as well and ended with a score well above 700, so there is now 100% more evidence for this method.</p><h3>Anki Flashcards</h3><h4>What’s Anki?</h4><p><a href="https://apps.ankiweb.net/">Anki</a> is a desktop, smart-flashcard app that uses spaced repetition to optimize learning. It’s free on the desktop so as long as you have a browser on your phone, you can probably run it anywhere for free. I’ve been using Anki for 17 years 👴.</p><p>The “smart” part of Anki is that it shows you cards at increasing intervals as you demonstrate mastery, which significantly improves long-term retention. To put it simply, the flashcards you get wrong are shown more often and for cards you get right, they appear less frequently.</p><h4>Proper Use</h4><p>I downloaded shared decks (flashcards someone else built) and created my own flashcards for key concepts, services, and definitions. Effective flashcards follow these principles:</p><ul><li>One concept per card</li><li>Clear, concise questions</li><li>Answers that require recall rather than recognition</li></ul><p>What’s the last bit about recall vs recognition? Well, bad flashcards are ones with multiple choices on them or just “true/false” as the answer. To create strong memories, you have to link a concept to a mini story or, in this case, a microservice. We’ll talk more about this concept later as it directly impacted how I made flashcards from the resources, i.e., the <a href="https://d1.awsstatic.com/training-and-certification/docs-cloud-practitioner/AWS-Certified-Cloud-Practitioner_Exam-Guide.pdf">Official AWS Cloud Practitioner Exam guide</a> and the <a href="https://github.com/kananinirav/AWS-Certified-Cloud-Practitioner-Notes/blob/master/practice-exam/practice-exam-1.md">Practice Tests found on Githu</a>b.</p><h4>Shared Flashcard Decks</h4><p>While creating your own cards is fine, why not use what someone else has already built? Using a <a href="https://ankiweb.net/shared/decks">shared deck</a> from the Anki community saved time while still helping me identify knowledge gaps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asBU426sguNwwBAq9VrzuQ.png" /><figcaption>Downloadable flashcards created by the Anki community for AWS in this case</figcaption></figure><p>I just downloaded <a href="https://ankiweb.net/shared/info/449552125">the first deck</a> as it had a nice number of cards and even though that deck is from 2021, the content has not really changed that much. It also used an interesting style of recall, the Cloze method, which is basically just a fill-in-the-blank style of recall. I also like that the author of this deck says, “I used it to study and pass on the first attempt. I studied for 20 hours — it was sufficient”.</p><p>To download a deck, <a href="https://apps.ankiweb.net/">first download Anki</a>. Then go the <a href="https://ankiweb.net/shared/decks">shared decks</a>. Click on the deck you are interested in and then there is a download button in bright blue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJf3db_kRZd8ym1GXRFOKA.png" /><figcaption>Big blue button is how you get shared decks</figcaption></figure><p>Then you open the download and it loads itself into Anki like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-IerSovhm1-1vVwgvzWVA.png" /><figcaption>How Anki looks</figcaption></figure><p>You then click on the deck title to start learning / reviewing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y3Dx6axLG1sMvuheNE7bqA.png" /><figcaption>Front of flashcard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LqsB0fbwH0Q7IQbi_woX3w.png" /><figcaption>Back of flashcard</figcaption></figure><h3>My Own Decks</h3><p>After going through 3 weeks of reviewing the core material, I then wanted to review new material from practice exams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yqYNcZg-YZn260-o-nElQA.png" /></figure><p>However, the style of the practice exam mimics the style of the real exam layout which is multiple choice. Multiple choice is not good for learning efficiently, so I had to manipulate the questions. Back in my day making/manipulating 50 flashcards would have a been an intensive, laborious, and long exercise… in patience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*w9-EUjaJbrNW4r9Lj5Rq9w.jpeg" /><figcaption>Revealing my age</figcaption></figure><p>But now with ChatGPT and Claude, I can literally make flashcards out of anything within minutes and perfectly formatted for Anki to ingest.</p><h3>ChatGPT &amp; Claude</h3><p>I only used the free versions to generate flashcards in minutes. Below are the two strategies I used.</p><h4>Prompting</h4><p>I was first using ChatGPT to scrape (scrape == take the information I want) from the GitHub site. To do that, I needed to enable “Search” and “Reason”.</p><blockquote>Note: I did try this without “Reason”ing but the output was almost always a hallucination (i.e., fake news).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdSKK4mHA-1Cpy1cbN7CFQ.png" /><figcaption>ChatGPT reasoning through my prompt</figcaption></figure><p>The Prompt:</p><p><em>Get all questions and their correct answers from this page: </em><a href="https://github.com/kananinirav/AWS-Certified-Cloud-Practitioner-Notes/blob/master/practice-exam/practice-exam-4.md"><em>https://github.com/kananinirav/AWS-Certified-Cloud-Practitioner-Notes/blob/master/practice-exam/practice-exam-4.md</em></a><em> Do not include answer choices, only the questions and the correct answers. The output should be an anki formatted csv file with an example output like this: “Where can you store files in AWS? (Choose TWO)”,”Amazon EFS, Amazon EBS”.</em></p><p>The one issue I had with ChatGPT is that it would never give me a proper csv file that I could download — sometimes it would cut 15 of the questions, sometimes it would just give me 10 of them etc. So instead, I just asked for a csv file <em>output</em> so that I could cut and paste into Claude. Claude was much better at giving me a working file that I could easily download. If you’re paying for ChatGPT, you might have better luck and not need Claude. Also, if you’re paying for ChatGPT you might not see the Reason or Search button.</p><p>The overall workflow was as follows: (1) ChatGPT scrapes and delivers text I can copy and paste. (2) Then I copy-paste that information into Claude requesting a downloadable file in a format that Anki can read, i.e., a .txt or .csv file type.</p><h4>Claude-only prompting</h4><p>For some reason, the above method stopped working after 2–3 days. ChatGPT would say it doesn’t have access to the internet for whatever reason. And then I realized I could easily get the information I needed without scraping. I’d just need to copy-paste the GitHub page. To do that, I put the GitHub in Code mode (instead of Preview mode — see upper left hand corner of the following image). Then I just did a Ctrl+A within in the practice exam area and copy-pasted to Claude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eK2zkK7paearqmAi8UiYvQ.png" /><figcaption>GitHub in Code mode instead of Preview mode</figcaption></figure><p>Now my prompt was:</p><p><em>Get all questions and their correct answers. Do not include answer choices, only the questions and the correct answers. The output should be an anki formatted csv file with an example output like this: “Where can you store files in AWS? (Choose TWO)”,”Amazon EFS, Amazon EBS”. — — — — — — — — — — — — — — — — — — — — — — — — # Practice Exam …</em></p><p>Finally, this blog is almost over. But wait, there’s more!</p><p>Remember that about 1/3 of concepts that you may be asked about on the exam are not covered in the official course. So for that, I simply copy and pasted from the <a href="https://d1.awsstatic.com/training-and-certification/docs-cloud-practitioner/AWS-Certified-Cloud-Practitioner_Exam-Guide.pdf">Official AWS Cloud Practitioner Exam guide</a> (page 15 of 22) any “in-scope services” I was not very familiar with. Here’s an example prompt I used in Claude to make the flashcards for these concepts and then I put the output into Anki:</p><p><em>Output a csv file. Make 2 Anki importable flashcards per topic::: Amazon EventBridge • AWS Step Functions • Amazon Simple Email Service • AWS Billing Conductor • AWS Local Zones • AWS Outposts • AWS Wavelength (Amazon SES)::: Example output: What is Amazon Athena? — serverless interactive query service — analyze data stored in Amazon S3 using standard SQL</em></p><h3>Bonus: Importing the CSV or TXT files into Anki</h3><p>How do I put the CSV files into Anki? File &gt; Import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*mChLIL7qw-Rj6DSVI4060A.png" /></figure><p>Then, make sure the “Field separator” is correctly chosen and that you have a front (1) and back (2) for each card:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJbCJjbsN7ndeTCfnWOMUw.png" /><figcaption>Importing txt or csv files with the <strong>correct</strong> separator chosen</figcaption></figure><p>If the correct separator is not chosen, you will end up with all your text on the front of the card which is not useful:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vwbP2fqCSHm5IQ7JKEqkjQ.png" /><figcaption>Importing txt or csv files with the <strong><em>incorrect</em></strong> separator chosen</figcaption></figure><h3>Applying This Method to Other Certifications</h3><p>This approach can be adapted for other certifications with similar documentation available. For example, the Google Cloud Professional Data Engineer certification:</p><ul><li><a href="https://services.google.com/fh/files/misc/professional_data_engineer_exam_guide_english.pdf">GCP DE Documentation</a></li><li><a href="https://ankiweb.net/shared/decks?search=GCP">Anki Shared Flashcard Decks for GCP</a></li></ul><p>The principles remain the same:</p><ol><li>Study official materials first</li><li>Create and review flashcards regularly</li><li>Use Claude, etc. to fill knowledge gaps</li><li>Take practice tests to refine your understanding</li></ol><h3>The end &amp; good luck with your cert!</h3><h3>About the Author</h3><p>I’m Victor Palacios, the Director of Data Science Partnerships at the University of San Francisco. My background includes undergraduate coursework in learning and metacognition. Long, Long, <em>Long</em> ago I created and taught a course at the undergraduate level about learning effectively. I received my master’s degree in Data Science and currently mentor and teach in the Masters in Data Science program at the University of San Francisco.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cf1a36f0df6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[mAIstory: Fine-tuning Gemma for storytelling]]></title>
            <link>https://medium.com/usf-datascience/maistory-fine-tuning-gemma-for-storytelling-013b3cdc38f0?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/013b3cdc38f0</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kagglex]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 18:57:59 GMT</pubDate>
            <atom:updated>2024-11-14T18:03:47.547Z</atom:updated>
            <content:encoded><![CDATA[<h4>A small language model, synthetic data, fine-tuning, and LoRA — what could go wrong!?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gn6ltEpM9UHxmyb29NRWvQ.jpeg" /></figure><p>As part of my <a href="https://www.kaggle.com/kagglex">KaggleX mentorship program</a>, I decided to fine-tune a small language model, specifically Gemma2–2B-IT (hereafter, simply “Gemma”), to create an interactive storytelling companion for children. <strong><em>And</em></strong> I wanted to share all the fun(ny?) conversations I had with the bot before it did what I intended it to. <a href="https://colab.research.google.com/drive/1DfnSFomJjAzCQMVQwwHR9htdEknlcd3_">Here is the code.</a></p><h3>What We’ll Build</h3><p>We’re going to create a storytelling chatbot that can engage in back-and-forth storytelling conversations while maintaining child-friendly responses. Our model will:<br>- Understand and respond to children’s questions<br>- Offer choices within stories<br>- Generate age-appropriate content<br>- Maintain conversation context</p><h3>Important Elements</h3><p>There were three items I wanted this bot to be able to handle specifically for children.<br>1. <strong>Branching Decisions:</strong> stories should be short and tailored to whatever the child wants to do.<br>2. <strong>Handle Ambiguity:</strong> the bot should be able to handle one word replies or even one word beginnings, i.e., instead of “tell me a story about a dragon”, a child (or lazy adult) should just be able to write one word such as “dragon” and have the bot understand the intent — to be entertained with a story about a dragon.<br>3. <strong>Handle Interruptions: </strong>the bot should be able to handle being interrupted by the child to explain a word and then proceed with the story.</p><p>Interestingly, building such a bot was a bit harder than expected.</p><h3>Final Product Example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nDR39Ddy1EVMY62XXhWCw.png" /><figcaption>Trimmed the text a bit on the right, but you get the gist</figcaption></figure><h3>The Dataset</h3><p>Without fine-tuning our model, what happens when we ask for a story?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ctAOcz6VMbToGKhy" /><figcaption>//Not// Fine-tuned Gemma model</figcaption></figure><p>Notice the story is tween focused. So to accomplish the goals set out above I <em>could</em> try prompt engineering, but what would be the fun in that? I want to fine-tune!</p><p>Originally, I found a <a href="https://huggingface.co/datasets/ajibawa-2023/Children-Stories-Collection">dataset for children’s stories on HuggingFace</a> which I thought would be perfect for fine-tuning the Gemma model. So I fine-tuned using 1000 examples (<a href="https://www.linkedin.com/pulse/how-many-data-points-necessary-fine-tuning-model-premai-tqc3f/">why 1000?</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SB_7e67D-RJ0L8AY" /><figcaption>Fine-tuned Gemma model</figcaption></figure><p>Notice now that the model is a bit more child-focused, so I was heading in the right direction. Next I wanted to add memory. For example, if I tell the model my name is Victor, then I want the model to be able to recall such a fact later in our conversation. To do this, I stepped away from the Keras framework shown above and instead used <a href="https://huggingface.co/google/gemma-2-2b-it">HuggingFace</a>’s Gemma model and <a href="https://python.langchain.com/docs/how_to/chatbots_memory/">LangChain</a> for instantiating memory. If you’re not sure what conversational memory is, I thought <a href="https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/">this article did a great job explaining memory for language models</a>.</p><p>At this point I thought I was done. I just needed to do a little prompt engineering automagic with LangChain and my bot would do exactly what I wanted. I was wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8z9HyvXgiB9ANqTewJ0xg.png" /><figcaption>Apparently a phenomena known as degeneration for language models</figcaption></figure><p>As you can see, the conversation with Nutkin quickly spiraled into an infinite repetition loop. My KaggleX mentor, <a href="https://medium.com/u/d62fa8861970">George Williams</a>, then suggested that because the data I had trained on was not conversational bot data, but rather <a href="https://datascience.stackexchange.com/questions/66398/question-answering-qa-vs-chatbots">closer to Q&amp;A bot data</a> I might have to find training examples closer to what I wanted as output. This data, <em>of course</em>, did not exist.</p><p>Therefore, I used an LLM to create 200 synthetic interactive children’s stories. Why 200? Because that was all I had the patience for. Was 200 sufficient for fine-tuning? Absolutely not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jwZvC7ZblvmYs6ZmU93lzA.png" /></figure><p>This was just the first of many hilarious encounters with what I will coin as the lunatic language model. It was a bit of a <a href="https://www.imdb.com/title/tt8760708/">M3gan</a> moment when the model decided to answer my question with a question and one that felt slightly eerie as it decided to say, “human”.</p><p>On a more serious note, the issue I was having is probably the “instability” I had read about when a language model is fine-tuned with not enough examples. Thinking this might be a one-off, I pressed on trying instead to use my one word prompts. Which took a very strange turn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mg2bfKgU7pR1jh0u7jPOUA.png" /></figure><p>In this case, my model completely ignored the turn-taking task it was given and proceeded to become a liar (?). Yet still, stranger conversations would follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*bLSNfPKmG2f-0OTaQn53wg.png" /></figure><p>I finally decided to ask a question I knew was part of the fine-tuning and see what would happen then. The results were equally manic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*e-3TVsCy9FsmntllnjHdlg.png" /></figure><p>After playing with hyperparameter tuning and number of epochs, I finally had to throw in the towel and go back to the basics — more data. So, of course, I turned to an LLM and asked what I should do. The LLM’s solution was brilliant. All I had to do was transform my 200 examples into 925 training examples through story splitting. <a href="https://colab.research.google.com/drive/1DfnSFomJjAzCQMVQwwHR9htdEknlcd3_#scrollTo=yNXWhHWj44rb">All of my synthetic stories can be found in this Colab notebook</a>.</p><h3>Here’s a sample story from my dataset:</h3><pre># Story 143: The Sky Chef<br>    [<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;Tell me about cooking in the clouds!&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;You discover a kitchen floating in the sky! Make cloud cookies or rainbow cakes?&quot;,},<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;What are cloud cookies?&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;They&#39;re fluffy cookies that make you float! Cookies or cakes?&quot;,},<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;Cloud cookies sound yummy!&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;Add stardust sprinkles or moonbeam frosting to your cookies. Which topping?&quot;,},<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;What do stardust sprinkles do?&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;They make your wishes come true! Stardust or moonbeam?&quot;,},<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;Stardust sprinkles please!&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;Your cookies grant wishes! Share them with friends or open a sky bakery?&quot;,},<br>    ],</pre><h3>Here’s a sample split of the above story</h3><pre># Split 1<br>    [<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;Tell me about cooking in the clouds!&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;You discover a kitchen floating in the sky! Make cloud cookies or rainbow cakes?&quot;,},<br>    ],</pre><h3>And another split…</h3><pre># Split 2<br>    [<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;Tell me about cooking in the clouds!&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;You discover a kitchen floating in the sky! Make cloud cookies or rainbow cakes?&quot;,},<br>        {&quot;role&quot;: &quot;Child&quot;, &quot;text&quot;: &quot;What are cloud cookies?&quot;,},<br>        {&quot;role&quot;: &quot;Narrator&quot;, &quot;text&quot;: &quot;They&#39;re fluffy cookies that make you float! Cookies or cakes?&quot;,},<br>    ],</pre><p>If this is confusing, don’t worry. What we are doing is showing our model how to handle various cases. For instance, Split 1 teaches our model how to handle a child’s request for a story. Split 2 teaches the model how to handle a conversation with previous context. So on and so forth.</p><p>But that’s not even the best part of these splits. By doing this, we are taking our 200 stories and transforming them into 925 mini-stories that generate a working model!</p><p>Before I conclude this article, I wanted to add a small bit on how I created the dataset in case anyone is interested.</p><h3>Creating the Dataset</h3><p>To create the dataset, I used various LLMs like Google’s Gemini and Anthropic’s Claude. I had to work on my rules severals times, but eventually I settled on something close to this:</p><h4>My story-generation rules</h4><ol><li>Start with the child’s input or question about a scenario.</li><li>Offering choices between two options and explaining them when the child asks.</li><li>Building on the child’s choices to create a continuing adventure.</li><li>Including elements of magic, nature, imagination, and morality.</li><li>Ending with another choice that could lead to further adventure.</li><li>For the child’s input also incorporate direct questions about words such as “what is a spirit?” or “what’s a castle?”</li><li>Format the answers as code with each bracket ending with a comma, even the final bracket.</li><li>Randomly begin some input with one word, like “banana”, “lying”, or “dragon” and proceed to tell a relevant story.</li><li>Each line should be between 3 to 20 words.</li></ol><p>One thing to note here is that my Python format requirements are not specified in great detail because I fed that into my prompt as an example of what I was looking for — you can do the same with a simple copy-paste from the<a href="https://colab.research.google.com/drive/1DfnSFomJjAzCQMVQwwHR9htdEknlcd3_#scrollTo=yNXWhHWj44rb"> Google Colab notebook</a> as well. You can also try <a href="https://gretel.ai/">gretel.ai for dataset creation at scale</a> or <a href="https://github.com/argilla-io/distilabel">distilabel as well</a>.</p><h3>Challenges I Still Face</h3><p>1. <strong>System Prompt Adherence</strong>: The model sometimes ignores system instructions about story length or ending conditions.</p><p>2. <strong>Memory Management</strong>: GPU memory limitations will impact who can use my colab notebook — you will need <a href="https://colab.research.google.com/signup">A100 GPUs which currently cost $10 a month</a> to use but are worth it for anyone interested in fine-tuning a conversational bot. At the time of this writing (11/11/2024) <a href="https://cloud.google.com/vertex-ai">Google also offers $300 for free on VertexAI</a>.</p><h3>Future Improvements</h3><p>There are several ways we could enhance this project:</p><p>1. <strong>Better Prompt Engineering:</strong> To improve response consistency.</p><p>2. <strong>Expanded Dataset:</strong> For more diverse stories and interaction patterns.</p><p>3.<strong> Model Scaling</strong>: Test with larger models.</p><p>4.<strong> Improved Control</strong>: Implement better mechanisms for controlling story length and complexity and safety for children (i.e., no bombs or booze).</p><h3>Conclusion and Code</h3><p>Fine-tuning Gemma for interactive storytelling showed me the potential of smaller language models for specialized applications. It was also the most amount of laughs I’ve ever had doing data science. The instability is real.</p><p>While there are limitations to address (many, many limitations), these results demonstrate that it’s not hard to create engaging, personalized, child-friendly genAI storytellers — so long as you have enough examples.</p><p>The <a href="https://colab.research.google.com/drive/1DfnSFomJjAzCQMVQwwHR9htdEknlcd3_">complete code for this project is available here</a>.</p><p>For my next project, I’m building out something similar but with audio and visual components. Stay tuned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NsV86tG3kv7f4c02vW0yA.png" /></figure><h3>About the Author</h3><p>Victor Palacios, Director of Data Science Partnerships at the University of San Francisco (USFCA), earned his master’s degree in data science from USFCA and brings a unique background in natural language processing and translation to the field. His research interests span deception detection and healthcare analytics, complementing his role in the education sector. Currently, Victor oversees the university’s 9-month data science practicum program, where he matches students with industry partners working on real-world data challenges across many sectors — typically 50 organizations per year with over 100 unique data science problems per cohort.</p><h3>Obligatory Hash Tags</h3><p>#MachineLearning #GenAI #Coding #LLM #SLM #Gemma #Google #KaggleX #DataScience #Python #HuggingFace</p><h3>The Last Laugh</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWwOzhsHsOye5d8uuEd53Q.png" /></figure><blockquote>And they all lived happily ever after. Except the princess. The End.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=013b3cdc38f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/usf-datascience/maistory-fine-tuning-gemma-for-storytelling-013b3cdc38f0">mAIstory: Fine-tuning Gemma for storytelling</a> was originally published in <a href="https://medium.com/usf-datascience">USF-Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing the Top AI Chat Services for Resume Reviews]]></title>
            <link>https://medium.com/usf-datascience/comparing-the-top-ai-chat-services-for-resume-reviews-bbcdd92d9aae?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbcdd92d9aae</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[resume]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 20:18:00 GMT</pubDate>
            <atom:updated>2023-08-29T20:18:00.680Z</atom:updated>
            <content:encoded><![CDATA[<p>Will artificial intelligence (AI) steal your job? We won’t know that for some time, but for today let’s see how AI can help you get a job. In this article, we’ll compare the top free AI Chat Services and see which delivers the best bang for your (free) buck when asked to improve your resume.</p><h3><strong>Context</strong></h3><p>Unless you’ve been living under a rock, you’ve probably heard of <a href="https://chat.openai.com/">ChatGPT</a> — the most popular AI Chat Service known to humanity (as of August 2023, anyways). You may even have read articles about how to <a href="https://simplify.jobs/blog/how-to-use-chatgpt-to-write-and-improve-your-resume/">use it to write a resume</a>. Going one step further, Resume Builder even has an article entitled: <a href="https://www.resumebuilder.com/3-in-4-job-seekers-who-used-chatgpt-to-write-their-resume-got-an-interview/">3 in 4 Job Seekers Who Used ChatGPT to Write Their Resume Got an Interview</a>. Clearly, these services are a game-changer for job seekers.</p><p>In this article, we want to take those musings even further and compare the top free AI Chat Services and see which delivers the best result with the least effort.</p><p>We’ll compare our AI resume helpers using the following 3 parameters:</p><ul><li>2 resumes (an ideal resume and one full of errors)</li><li>3 free AI Chat Services — <a href="https://chat.openai.com/">ChatGPT</a>, <a href="https://bard.google.com/">Bard</a>, and <a href="https://claude.ai/chats">Claude</a></li><li>3 outputs (minimum) per service to check the variability of responses</li></ul><p>Based on the results of the above tests, we will crown a winner for best AI resume helper in this article.</p><h3><strong>About the Authors</strong></h3><p>Victor received his master’s degree in data science from the University of San Francisco. He has taught courses on data science, machine learning, and low-code data analytics tools such as KNIME. He works as the Director of Data Science Partnerships at the University of San Francisco. His guilty pleasures are deception detection and healthcare data science projects, but his love of the field originally blossomed from natural language processing. Victor is currently based in the United States, where all three AI Chat Services are available.</p><p>Mahantesh is a data scientist at KNIME. He has completed his Masters in Data Science from OVGU, Germany. He is actively working in the data science domain in the field of machine learning, natural language processing, deep learning, predictive modeling, and business analytics. He is currently based in Berlin, Germany, where only two of the three AI Chat Services are available — ChatGPT and Bard.</p><h3><strong>The Resumes</strong></h3><p>For this experiment, we used two data science resumes — one strong and one weak. The strong resume has the following characteristics:</p><ol><li>No typos.</li><li>Has been used to successfully land jobs for education meets data science roles.</li><li>Has clear, repetitive structure.</li><li>Is metric-oriented, meaning that most bullet points discuss a concrete number. For example, “Developed the target variable and improved baseline roc auc scores from 60% to 89%.”</li></ol><p>In contrast, the weak resume has many typos, an extra blank page, differing structure between roles, and does not include a LinkedIn profile. Finally, the weak resume includes irrelevant sections, such as hobbies that do not relate to data science, among many other errors. You can see each resume used below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*ay-6cYgp04Sx6peJ" /><figcaption>The Strong Resume</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*Ukp7PEaNt2Arw90D" /><figcaption><em>The Weak Resume</em></figcaption></figure><h3><strong>The AI Chat Services</strong></h3><p>Here is what Claude has to say about each AI Chat Service:</p><p><em>ChatGPT is an AI chatbot created by OpenAI that uses a large language model (LLM) to generate human-like conversations on many topics. It was released in November 2022 and quickly gained popularity for its advanced capabilities.</em></p><p><em>Claude is an AI assistant from Anthropic that uses a limited LLM focused on being helpful, harmless, and honest in its responses. It aims to improve safety in conversational AI.</em></p><p><em>BARD is Google’s new conversational AI chatbot, announced in February 2023, that utilizes LLMs to compete with ChatGPT. It aims to provide more up-to-date, high-quality responses than previous chatbots.</em></p><h3><strong>The Prompt</strong></h3><p>For each AI Chat Service, we use the same prompt: “Review my resume. Then, suggest improvements. Here is my resume: [resume text]”</p><h3><strong>The Prompt Output</strong></h3><p><strong><em>Strong Resume (SR) &amp; ChatGPT’s Response</em></strong></p><p><em>Overall, your resume looks impressive, and you have a strong background in data science with relevant experience and education. However, there are some areas where you can make improvements to enhance your resume further:</em></p><p><em>Formatting and Structure: Consider organizing your resume in a more visually appealing manner to make it easier to read. Use bullet points to break down your accomplishments and responsibilities for each job. Make sure the formatting is consistent throughout.</em></p><p><em>Summary/Objective Statement: Add a short summary or objective statement at the beginning of your resume to highlight your key skills and goals as a data scientist. This will help recruiters quickly grasp your expertise and what you’re looking for.</em></p><p><em>Reverse Chronological Order: List your professional experience and education in reverse chronological order, with the most recent positions and degrees first.</em></p><p><em>Quantify Achievements: Try to quantify your achievements with specific numbers or percentages wherever possible. For example, mention the number of students you taught, the size of the audience for webinars, or the revenue impact of your projects.</em></p><p><em>Impact-Oriented Language: Use strong action verbs and impact-oriented language to describe your accomplishments. Show how your work positively affected the organizations you were part of.</em></p><p><em>Remove Personal Information: For privacy reasons, consider removing your full address from the resume. It’s enough to mention your city and state.</em></p><p><em>Skills Section: Create a separate section for your technical skills and tools. You can categorize them under “Programming Languages,” “Data Analysis,” “Data Visualization,” etc., to provide a clearer overview of your expertise.</em></p><p><em>Professional Development: If you have participated in any data science-related workshops, conferences, or online courses, consider adding a section to highlight your commitment to continuous learning.</em></p><p><em>Publications and Research: Mentioning your published work and research projects in a separate section can add weight to your profile. Include your “Machine Learning-based Meal Detection” publication and any other relevant academic contributions.</em></p><p><em>References: It’s common practice to omit references from the resume. You can mention that references are available upon request.</em></p><p>Next, we’ll look at how those responses differ when we put in the exact same prompt three more times. Here is the<a href="https://chat.openai.com/share/a3e8e2e1-3214-4806-b6e4-8041aea9ebd7"> first SR ChatGPT reprompt</a>, the <a href="https://chat.openai.com/share/f025c5a6-762f-4d1c-bee5-9bf2985197b5">second SR ChatGPT reprompt</a>, and the <a href="https://chat.openai.com/share/9213b8ee-de07-459a-8ae0-37545376d19a">third SR ChatGPT reprompt</a>.</p><blockquote><strong><em>Why reprompt?</em></strong></blockquote><blockquote>AI Chat Service responses are not stable. That is, there is variability using the exact same prompt and in this article we would like to make sure our assessment is a fair one based on several samplings.</blockquote><p>On the positive side, most of the recommendations given were some of the most important. For us, quantifying achievements is the single most useful point to mention to anyone writing their resume, and we were happy to see that point mentioned in every single output given by ChatGPT. We did find it odd to mention quantifying achievements for this resume, given that almost every experience had some quantity mentioned. Notice also that in each answer given by ChatGPT, the number of items it suggested to fix were different (10, 11, 12, and 13, respectively). Also of worth, in the third reprompt it gave us a resume that we could copy and paste into a word processor if we chose, which could be useful. Overall, we like the variety of outputs given and would suggest everyone reprompt to see the colorful answers these AI Chat Services can produce.</p><p>On the negative side, we found it odd that in the third reprompt it suggested that we remove “Remove Generic Skills” such as Git, SQL, BeautifulSoup, and Selenium. We’re not sure in what world the above would be considered generic skills, and we believe that data scientists should be familiar with version control, know how to deal with databases, and be able to scrape their own data. When we interviewed data scientists, if we didn’t see such skills, we were worried. We also found it odd that it mentioned reverse chronological ordering in the strong resume, even though that was the strategy used throughout. It was either the case that ChatGPT did not understand that the strong resume was in reverse order or it simply defaulted to giving generic (but good) advice. For the other AI Chat Services, we’ll keep on eye on this theme — giving generally useful advice that is irrelevant for the resume being critiqued.</p><p><strong><em>Strong Resume (SR) &amp; Bard’s Response</em></strong></p><p><em>I can’t assist you with that, as I’m only a language model and don’t have the capacity to understand and respond.</em></p><p>Oddly, when we used Bard in the United States with our given prompt, a response was not given. We retried to make sure it was not a simple bug but to no avail. However, when we used Bard in Europe, it did give 3 distinct responses. Here is the <a href="https://g.co/bard/share/f1f0c47a9110">first SR Bard reprompt</a>, the <a href="https://g.co/bard/share/1441fdaa0ad9">second SR Bard reprompt</a>, and the <a href="https://g.co/bard/share/f2f1bea441fc">third SR Bard reprompt</a>.</p><p>On the positive side, Bard suggested checking the resume for grammatical and spelling errors, which ChatGPT did not. We also appreciated that only Bard mentioned looking for relevant keywords and putting them in the resume. Keywords help applicants beat the <a href="https://www.youtube.com/watch?v=rCQMidRqndY">ATS</a> (applicant tracking system) or simple rule-based counting metrics used by companies to quickly filter candidates.</p><p>On the negative side, Bard provided some generic suggestions on each section. Like Claude, it was also a bit on the shorter side of descriptions and ideas.</p><p><strong><em>Strong Resume (SR) &amp; Claude’s Response</em></strong></p><p><em>Here are a few suggestions to improve your resume:</em></p><p><em>Add a Profile/Summary section at the top that highlights your key skills, years of experience, and specialized expertise areas. This helps provide an overview of your qualifications.</em></p><p><em>Move Education section to the bottom. Experience and skills are more important.</em></p><p><em>Expand on your project descriptions by focusing on impact and quantifiable results. For example, how much revenue did you identify for the medical company? How much did your model improvement increase AUC score?</em></p><p><em>Be more specific about skills used for each project. Don’t just list Python — say Pandas, NumPy, Scikit-Learn, etc.</em></p><p><em>Remove coursework section under education. Focus on relevant projects instead.</em></p><p><em>Standardize your formatting and spacing. There are some inconsistencies in dates, indentation, etc.</em></p><p><em>Consider condensing listing of individual courses. Focus on degrees, major, university name.</em></p><p><em>Add key achievements under each role — presentations given, automation achieved, skills gained, etc. Quantify reach/impact where possible.</em></p><p><em>Make skills section more scannable — break into categories like Programming Languages, Data Science Libraries, Databases, etc.</em></p><p><em>Overall the content looks strong, with a mix of academic and professional projects. With some tweaks to polish up the formatting and organization, it will be a very solid data science resume. Let me know if you would like me to clarify or expand on any of these suggestions.</em></p><p>Next, we’ll look at how those responses differed when we put in the exact same prompt three more times. Here is the<a href="https://docs.google.com/document/d/1fxKmKv4whaaLlc8nYHe0c_-JpM3zSb0aHjcybBuKJiY/edit"> first SR Claude reprompt</a>, the <a href="https://docs.google.com/document/d/1AEIaVFblJots9r2pkfhKBWAgbrTOX7DcaZHEVHp1O5Y/edit">second SR Claude reprompt</a>, and the <a href="https://docs.google.com/document/d/1s9tffrP9RA2nbS0JFOaPbwKIp5mZJ6uyAQyaooCSOj4/edit">third SR Claude reprompt</a>.</p><p>On the positive side, these outputs are shorter in general which can save you time. These shorter responses make the experienced user aware to ask for more or less unlike with ChatGPT which gave us a lot to read with each output. We also enjoyed the specificity of the third reprompt which advised us to quantify and provided us with an example from the resume and then later told us to list out specific tools we used in each role. Finally, the reasoning provided was great for cases like the following: “Remove the coursework section under education. Coursework isn’t as relevant once you have work experience.” Here it was clear that in certain situations you want coursework (but not this one), and we appreciated being taught the reasoning behind the recommendation.</p><p>On the negative side, Claude is a bit more terse and tends to give shorter outputs than, say, ChatGPT, so if you didn’t know you can alter the verbosity of the AI Chat Service, you might be left feeling like these responses are lacking. It also tends to give less concrete, actionable material than ChatGPT as well. Comparatively speaking, we would have preferred a bit more verbosity and explanation, which we did find in one of ChatGPT’s responses but did not find as much in Claude’s. Finally, we found it puzzling that three out of four prompts told us to “Move Education section to the bottom”, but the third reprompt told us to do the opposite. In our opinion, put education on the top when you’re a new graduate or the college you went to is prestigious — otherwise, put it on the bottom if you have more relevant work experience. Since a human will only scan your resume for about 6 or <a href="https://www.prnewswire.com/news-releases/ladders-updates-popular-recruiter-eye-tracking-study-with-new-key-insights-on-how-job-seekers-can-improve-their-resumes-300744217.html">7 seconds</a>, you need to give them the most relevant information first.</p><p><strong><em>Weak Resume (WR) &amp; ChatGPT’s Response</em></strong></p><p>For the weak resume, we’ll just look at three outputs from each service. Here is the <a href="https://chat.openai.com/share/ad14219e-d5db-4f0d-9654-b84c22ed26cf">first WR ChatGPT output</a>, the <a href="https://chat.openai.com/share/ee17c2dc-26f7-4ef2-b072-dfbc10eebecb">second WR ChatGPT output</a>, and the <a href="https://chat.openai.com/share/e46487ad-c0c0-4a3a-a029-06cd02cd6678">third WR ChatGPT output</a>.</p><p>On the positive side, what we enjoyed about the first output was that it gave concrete examples from the weak resume and showed us how to improve it:</p><p><em>For example, instead of: “As a data scientist, We build models for classification, evaluate the results, and share my findings with stakeholders.”</em></p><p><em>You could write: “Developed machine learning models for classification, resulting in a 20% increase in accuracy compared to previous methods. Presented findings to stakeholders, enabling data-driven decision making and leading to a 15% improvement in business performance.”</em></p><p>This kind of output helps us learn how to improve the rest of the resume and helps us in case we need to update the resume again in the future. It also makes us aware that we need to be looking for projects that have a high impact to put on the resume in the first place! We also liked that in the third output was a fully rewritten (without grammatical or spelling error) version of the weak resume. This critique also felt very tailored to the specific resume it was reading. For instance it commented on the professional experience saying, “Professional Experience (KNIME): Elaborate on the educational materials and blog posts you created. Specify the number of blog posts, the topics covered, and any positive feedback or engagement received from the audience” which is exactly the kind of recommendation we would provide if this were a friend’s resume.</p><p>On the negative side, the outputs were not consistent in terms of quality of response and how tailored they were to the resume at hand. In our opinion, the quality of the first output is far better than either the second or third output. At times, it even gave strange feedback, such as the dreaded “References available upon request” line, which should not exist in a modern resume. It also suggested that we “​​Tailor for Each Job Application,” which isn’t feasible for the domain (data science) given that the number of applications needed to even get an interview is typically in the 200–300s according to the data from our students. Imagine how long it would take to tailor each application? The cost-performance tradeoff simply doesn’t exist. Instead, it should suggest creating 2–3 resumes tailored for a specific sector or data role. Also, note that none of the outputs told me about the spelling or grammatical mistakes we purposely made in the resume.</p><p><strong><em>Weak Resume (WR) &amp; Bard’s Response</em></strong></p><p>For the weak resume, we’ll just look at three outputs from each service. Here is the <a href="https://g.co/bard/share/467848bfeb6f">first WR Bard output</a>, the <a href="https://g.co/bard/share/ade4ddbfe887">second WR Bard output</a>, and the <a href="https://g.co/bard/share/a18bd560d5e6">third WR Bard output</a>.</p><p>On the positive side, it supplied specific recommendations for each section of the weak resume. Find below a sample of the particular suggestions offered by Bard.</p><ul><li><em>In your professional experience section, you could quantify your accomplishments whenever possible. For example, instead of saying “I built models for classification,” you could say “I built models for classification that achieved an accuracy of 95%.”</em></li><li><em>In your education section, you could include the names of any courses that are relevant to the jobs you are applying for.</em></li><li><em>In your hobbies and interests section, you could include any activities that demonstrate your skills or interests that are relevant to your professional goals. For example, if you are applying for a job in the healthcare industry, you could mention that you are a volunteer at a local hospital</em></li></ul><p>Also, it did suggest removing the Hobbies section of the resume as it doesn’t add relevance in the context of job application. Another positive aspect of Bard is that it provided a similar response when reprompted, which points towards the stability of the response.</p><p>On the negative side, Bard did not point out what spellings should be corrected, it simply suggested getting the resume checked for spelling mistakes. Also, it did not recommend adding LinkedIn or GitHub profile links which can be helpful for technical recruiters or technical round interviews.</p><p><strong><em>Weak Resume (WR) &amp; Claude’s Response</em></strong></p><p>For the weak resume, we’ll just look at three outputs from each service. Here is the <a href="https://docs.google.com/document/d/1p38YFNUuxw4clTi-Zwi7DJ3aysZ4oxoXoROHliKBB2s/edit">first WR Claude output</a>, the <a href="https://docs.google.com/document/d/1EktZ5TiM1lWXq5Fl35-lSyfgipthfGrwDigkM9-qw1Y/edit">second WR Claude output</a>, and the <a href="https://docs.google.com/document/d/1yygk3gEKWBW8HByYTIQEWatVlPqaqJ1WSlk0ERXkZ1E/edit">third WR Claude output</a>.</p><p>On the positive side, Claude told me to quantify the experiences, which should always be the first recommendation as most resumes I read forget this important detail. And just as with the strong resume, the brevity of responses was appreciated. We also appreciate that Claude can actually read PDFs instead of simply making up content which has been our experience with Bard when given documents.</p><p>On the negative side, the output seemed quite generic and didn’t seem as useful compared to output from the other AI Chat Services. As well, for Bard and ChatGPT, key points are bolded so that you can quickly get the gist, whereas with Claude that is not the case. In this sense, the product is not as polished in the presentation of output.</p><h3><strong>Crowning a Winner</strong></h3><p>To compare our results, we use a simple infographic with five key terms and stars (maximum of 3 stars and minimum of 1 star). The five key terms are defined below.</p><p><em>Out-of-Box Personalization</em> refers to whether or not the AI Chat Service made specific references to the resumes or simply output generic responses. If we gave more specific prompts, perhaps all services could give more personalized outputs, but <a href="https://www.deeplearning.ai/short-courses/">Prompt Engineering</a> is beyond the scope of this article.</p><p><em>Stability</em> refers to whether the 4 different outputs were relatively similar or had high variability.</p><p><em>Quality</em> refers to the usefulness of the responses — were the recommendations actually useful or not.</p><p><em>Precision</em> refers to whether the service noted the many mistakes we purposefully made in the weak resume, such as “end-to.end” or whether any service could find a grammatical/spelling flaw in the strong resume.</p><p><em>PDF Function </em>refers to whether the service’s PDF (or other document) reader worked and how well it worked. Note that ChatGPT does not have such functionality. For Bard, the functionality presumably exists but Bard (as of August 2023) usually gives completely unrelated output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6d-e3T9DENcooyCrYfN-zw.png" /></figure><p>Without further ado, the winner is: ChatGPT! If you’re not interested in prompt engineering and just want a tool to give you a vast amount of ideas to fix your resume with a simple prompt, then ChatGPT is for you.</p><h3><strong>Final Thoughts</strong></h3><p>One thing we were pleasantly surprised about was that none of the services suggested we change our names to another nationality which is currently a notorious topic <a href="https://www.businessinsider.com/student-uses-playrgound-ai-for-professional-headshot-turned-white-2023-8">with AI-image tools</a>. Given that a <a href="https://www.wired.com/story/ai-adversarial-attacks/">recent attack renders even chat services vulnerable</a> to all sorts of nefarious actors, we were relieved to see that we did not have any such negative outputs. We were also fairly surprised that overall, ChatGPT seemed to give the best responses even though <a href="https://arxiv.org/abs/2307.09009">Stanford researchers have seen ChatGPT’s outputs declining in quality</a> recently.</p><p>One other surprise to us was that none of the services pointed out specific errors like “end-to.end” (although some did mention checking for spelling errors). The take home message there being, if you are not going to copy paste the resume generated by the AI Chat Service you choose, then you need to ask the service to find and point out all grammatical and spelling errors in the resume explicitly.</p><p>On a personal note, Victor has had issues with very complex instructions given to ChatGPT and Bard but noticed that Claude is far better at following complex commands. For instance, if his instructions were long (say 10 distinct commands), ChatGPt and Bard simply ignore some of what is asked. In contrast, Claude can follow very long and complicated instructions well. To us, this work/article suggests that more open-ended, vague, simple questions are better for ChatGPT, but with better prompting, Claude may prove the superior service. We suggest that all users should try out at least 2 services and run their own experiments to see which is better for their particular task.</p><p>In the future, it might be helpful to compare the effects of using better prompts to be more specific about exactly what we want from each AI Chat Service. For instance, specifically stating that we want the AI to take the role of a data science manager and seeing if that has any effect on the output. We would also suggest explicitly stating that all grammatical and spelling errors be found in the resume, producing a copy-pasteable version of the resume, and citing sentences from the resume to show how they could be improved. One final instruction of interest might be to ask the AI Chat Service to explain its reasoning so that the resume writer can learn the “why” for all its recommendations. We leave that up to the reader to explore further.</p><p>Thanks for reading! May the luck of the LLMs ever be with you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbcdd92d9aae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/usf-datascience/comparing-the-top-ai-chat-services-for-resume-reviews-bbcdd92d9aae">Comparing the Top AI Chat Services for Resume Reviews</a> was originally published in <a href="https://medium.com/usf-datascience">USF-Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitor Job Applications with a LinkedIn Data App]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/monitor-job-applications-with-a-linkedin-data-app-b63cc0eaf66b?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/b63cc0eaf66b</guid>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[dataviz]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[stories]]></category>
            <category><![CDATA[knime]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Fri, 31 Mar 2023 07:46:02 GMT</pubDate>
            <atom:updated>2025-01-14T13:26:38.332Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA STORIES | LINKEDIN DATA APP | KNIME ANALYTICS PLATFORM</h4><h4>Create beautiful visualizations to get valuable insights from your LinkedIn data</h4><p><em>Co-author: </em><a href="https://www.linkedin.com/in/alida-brizzante-6519b8195/"><em>Alida Brizzante</em></a></p><figure><img alt="Monitor Job Applications with a LinkedIn Data App" src="https://cdn-images-1.medium.com/max/1024/0*jJjFMHi1QCwY22CT.gif" /><figcaption>Fig. 1. The gif shows what you can expect to see in the LinkedIn data app.</figcaption></figure><p>Many of us have felt the pain taking those first steps to start a career. At the beginning of the journey, you find yourself full of hope and positive vibes. This can soon start to fade once you realize that after hundreds of applications you’re still empty-handed.</p><p>One of the most useful features of LinkedIn is the simplicity with which you can apply for hundreds and thousands of jobs. But, keeping track of all your job applications is anything but straightforward! Monitoring them precisely not only allows you to unveil interesting patterns but also improves your job hunting skills.</p><p>That’s why we decided to build a browser-based <a href="https://www.knime.com/blog/deliver-data-apps-with-knime?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">data app</a> that visualizes insights about your LinkedIn job data in a meaningful, interactive way.</p><p>Today, we want to walk through the workflow behind the data app, show you how to request your LinkedIn data and use the data app with your own data!</p><h3>Now Use the Data App with Your Own LinkedIn Data</h3><p>The first page of the data app gives you the option to upload your own LinkedIn data. You have to request this file from LinkedIn, which you can do by following these steps:</p><ol><li>Open <a href="https://www.linkedin.com/feed/">LinkedIn </a>and click the profile icon (top-right).</li><li>Go to “Settings &amp; Privacy” and then “Data Privacy”.</li><li>Under the section “How LinkedIn uses your data”, click “Get a copy of your data”.</li><li>Select the “Download larger data archive, including connections, contacts, account history, and information we infer about you based on your profile and activity” option.</li></ol><p>Congrats! LinkedIn will now start preparing your data. You will receive your LinkedIn archive within 24 hours.</p><p>The files of interest are those referring to the connections (Connections.csv) and the job applications (Job Application.csv).</p><p>Please note that the files concerning the job applications will be stored in a dedicated folder. If you’ve applied for multiple jobs, you’ll have multiple files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhKQZHBA2VXsdrMD5nnVfQ.png" /><figcaption>Fig. 2. Example of the files contained in the LinkedIn folder once you download it.</figcaption></figure><h3>How We Built the LinkedIn Data App</h3><p>Our LinkedIn data app was built with the goal of enabling the user understand how their application process is going at a glance. We also wanted the data app to be customizable, interactive, easy to share, and consumable as a web application.</p><p>KNIME data apps are built using KNIME Analytics Platform, where we can collect, process and visualize data by drag-and-dropping and connecting nodes from the Node Repository (Fig. 3). We can define the desired level of customization, interactivity, and easily export our workflow to share it with others on the <a href="https://hub.knime.com/">KNIME Community Hub</a>. To make our KNIME data app available as a web application we’ll deploy it to the <a href="https://www.knime.com/knime-business-hub?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">KNIME Business Hub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jq8oFsDrQVbk77dFJ0_UeQ.png" /><figcaption>Fig. 3. Overview of the <a href="https://hub.knime.com/-/spaces/-/latest/~6WbbFLXx1ZV-13KF/">LinkedIn data application workflow</a>.</figcaption></figure><h4>Enable Data App Users to Upload LinkedIn Files</h4><p>When we use a browser-based data app, each “page” of the application is defined by the <a href="https://docs.knime.com/2022-12/analytics_platform_components_guide/index.html#introduction">components</a> containing nodes with an Interactive View. You can set a layout for each element in the page, with each <a href="https://docs.knime.com/2022-12/analytics_platform_components_guide/index.html#introduction">component</a> composite view organized using the <a href="https://docs.knime.com/2022-12/analytics_platform_components_guide/index.html#layouting">Node Usage and Layout window</a> in KNIME Analytics Platform.</p><p>The first page of the data app prompts the user to upload their LinkedIn personal files.</p><p>Our main goal here was to ensure workflow portability and generalizability, avoiding an instance-based workflow that would not work with other data. To this end, the workflow control features in KNIME, such as <a href="https://www.knime.com/nodeguide/control-structures/flow-variables">flow variables </a>in combination with the <a href="https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.switches.manualif.ManualIfNodeFactory">switch </a>and <a href="https://medium.com/low-code-for-advanced-data-science/the-wonderful-world-of-widgets-a-mini-guide-for-using-knime-widget-nodes-41f49a19147">widgets nodes</a> were crucial. Flow variables gave us flexibility. Instead of using predefined values, they change dynamically together with the data. And using switch nodes it was possible to create parallel branches depending on the flow variable’s value, introducing a little error handling logic.</p><p>Users can upload their personal files using the <a href="https://hub.knime.com/knime/extensions/org.knime.features.js.quickforms/latest/org.knime.js.base.node.widget.input.fileupload.FileUploadWidgetNodeFactory">File Upload Widget</a> node wrapped inside the File Upload component. The result is a simple web page to upload all the files you have in just a few clicks.</p><figure><img alt="Monitor Job Applications with a LinkedIn Data App" src="https://cdn-images-1.medium.com/max/1024/0*WxS7Uzb7xNlA96R1.gif" /><figcaption>Fig. 4. The first web page of the LinkedIn data app. The user can upload the connections and job applications files.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*KL3nwN2jl8Bo7olDYA5rXA.png" /><figcaption>Fig. 5. View of the nodes inside the File Upload component. The widgets nodes are used to upload the personal files.</figcaption></figure><p>Some of the uploaded files contain a disclaimer at the top, which can make auto-detection of the right file format troublesome. However, exploiting the configuration options provided by nodes in the KNIME’s file handling framework, we can easily customize the behavior of the <a href="https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.io.filehandling.csv.reader.CSVTableReaderNodeFactory">CSV Reader</a> node to allow for a graceful handling of unnecessary rows.</p><p>Each job application file can contain at most 200 observations. We know, however, that job-seekers often end up applying to more than 200 positions before they get hired. To account for that, we allow the user to upload up to 5 different job application files. Next, using the Concatenate node, we can create a concatenated table of all job applications for further processing.</p><h4>Engineer Features, Mine Texts, and Wrangle Dates</h4><p>After importing the data and solving data file readability issues, we process the data to extract insights job-seekers may look for. LinkedIn does not provide a comprehensive analysis of the job application history, especially when it comes to the questionnaire that most companies require for the job applications.</p><p>The data processing and preparation of the workflow addresses this issue. To discover hidden information in the data, we adopted a heterogeneous approach, using operations from text analysis, feature engineering, and date&amp;time manipulation. The complexity of the workflow was then wrapped into multiple <a href="https://www.knime.com/metanodes">metanodes </a>to ensure the workflow is easy to follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F3x-2eZkvNxrS0GWrbhBJw.png" /><figcaption>Fig. 6. View of the Processing Applications metanodes. It contains other metanodes dedicated to different processing operations.</figcaption></figure><p>Starting from the job position information, we used feature engineering techniques to extract a “detail score” out of the position title.</p><p>This score is built starting from the job position name’s length: if the position’s length is longer than the average, then a positive value normalized between 0 and 1 is assigned. The detail score helps applicants understand how broad or specific job positions on LinkedIn are adjusting their expectations. Additionally, an “urgency score” was also created filtering applications based on whether or not the urgency of the company to fill the job position is mentioned during the questionnaire.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNp6DB1KLC2lI4B4e92qRQ.png" /><figcaption>Fig. 7. View of the Urgency and Detail Scores metanode. It performs feature engineering operations on the job application’s Q&amp;A and position title.</figcaption></figure><p>As we mentioned above, the extraction of insights also involved applying text mining techniques to the applicant’s questions and answers. We focused especially on the most frequently asked questions and/or topics revolving around the term “salary”, computing the word frequency of that term and matching the question in which the term appeared (see Fig. 10).</p><p>We visualized the retrieved information in a word cloud for ease of interpretation (read more about it in the next section).</p><p>With the preprocessing steps implemented above, we can now get deeper and more informative insights into what the companies are interested in while hiring new employees.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUvk37qlCfizLsv1B1BMyA.png" /><figcaption>Fig. 8. View of the nodes inside the Questions and Answers metanode. Text analysis of the questionnaire and salary information.</figcaption></figure><h4>Provide Connections and Job Application Insights at a Glance</h4><p>The core of the data app is represented by two dashboards visualized as web pages:</p><ol><li>The first one elaborates a few insights about connections and job application data. The output is a short compact version of the visualizations;</li><li>The second page dives deeper into the job application and connections data, including additional options for interaction and granularity.</li></ol><p>The first page contains a “snapshot” visualization of your job application and connections data. This is extremely useful for visualizing at a glance some of the most insightful categories of the data provided by LinkedIn.</p><p>On the right side, you will find the visualization about the job applications while, on the left side, you can visualize the connections charts. Below there’s a short description of the meaning and usefulness of this dashboard:</p><ul><li><strong>Job positions of your connections </strong>. Having a network of connections working in a similar field to the one you are interested in might help you find a job more easily, or receive the right suggestions. Thus, it is useful to realize what are the most frequent job positions of your connections.</li><li><strong>Time trend of connections</strong>. It is important to monitor the always expanding network of people around your LinkedIn profile. A more dense and spread network will help you in your professional life.</li><li><strong>Job positions with more applications</strong>. While applying for many jobs during a short span of time, you may not be completely aware of all the job positions you applied to and with which frequency. This visualization allows you to keep track of the job positions of the application, letting you know both the variety of your applications and the availability of job positions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyHdQdvpb722nDicoqTMEA.png" /><figcaption>Fig. 9. The second page of the LinkedIn data app. It contains a short version of the main visualization dashboard, including some of the most interesting insights.</figcaption></figure><h4>Get a Deeper Dive into LinkedIn Data</h4><p>The last and main page of the data app capitalizes extensively on the interactivity of KNIME dashboards.</p><p>The data app user can change the settings and play with different visualizations. Below you can find some examples of the data app, with a special focus on the interactive parts.</p><p>The gif below shows how to interact with the data app to change the number of questions you want to visualize in the most frequently asked questions shown in the table.</p><figure><img alt="Monitor Job Applications with a LinkedIn Data App" src="https://cdn-images-1.medium.com/max/1024/0*ruzazABc_pQv09XG.gif" /><figcaption>Fig. 10. How to change the number of questions you want to visualize in the most frequently asked questions Table View.</figcaption></figure><p>The next gif shows how you can interact with the data app to select a time range, for example the number of job applications in the last month, year, or week.</p><figure><img alt="Monitor Job Applications with a LinkedIn Data App" src="https://cdn-images-1.medium.com/max/1024/0*IQ5V2GvhxPgmATo7.gif" /><figcaption>Fig. 11. The number of applications can be visualized per month, year or week. The user can change it dynamically thanks to the widget nodes.</figcaption></figure><p>This next gif shows how you can use the data app to visualize the most urgent job applications. The workflow finds those applications that mentioned the urgency of the application in the questionnaire.</p><figure><img alt="Monitor Job Applications with a LinkedIn Data App" src="https://cdn-images-1.medium.com/max/1024/0*bz0Fc2A-hx8vm1ym.gif" /><figcaption>Fig. 12. How to visualize the names of the companies (and the offered job position) that mentioned the urgency of the application during the questionnaire.</figcaption></figure><p>A more articulated visualization is the one about the network of connections, which displays the network of all your connections that work in the top 3 companies by number of employees (Fig. 16). This lets you visualize where the majority of your connections work. This is possible using just a few nodes thanks to the <a href="https://hub.knime.com/knime/extensions/org.knime.features.network/latest">KNIME Network Mining extension </a>.</p><p>If KNIME is one of the three companies represented in the network, all your connections working at KNIME will be visualized with the yellow-KNIME color instead of the default blue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svd6I235OLMWfH6zZ6tY0Q.png" /><figcaption>Fig. 13. Network visualization of the companies in which the majority of your connections work.</figcaption></figure><p>Last but not least, if you want to make your data app look even fancier and customizable, you can use the <a href="https://hub.knime.com/knime/spaces/Examples/latest/00_Components/Guided%20Analytics/Data%20App%20Flowchart~-o_CLuu7-qrfguqE">Data App Flowchart</a>. It’s a KNIME <a href="https://www.knime.com/verified-components">verified component</a> and adds a flowchart header to all your web pages, improving the aesthetics of your data app for a better user experience.</p><h3>Try these Tips &amp; Tricks in Your Data App</h3><p><strong>Include a high-level of interactivity</strong>: The design of the LinkedIn data app relies on a combination of charts, tables and text, where the user can enjoy a high level of interactivity, and the freedom to customize their visualizations, for instance changing the type of plot or the variables in it.</p><p><strong>Ensure a flexible design</strong>: Without diving deep into all the sections, a great example of the capabilities of KNIME Analytics Platform can be found in Fig. 17. In this section of the workflow, we used widgets nodes to let the user pick the number of questions that they want to visualize (<a href="https://hub.knime.com/knime/extensions/org.knime.features.js.quickforms/latest/org.knime.js.base.node.widget.reexecution.refresh.RefreshButtonWidgetNodeFactory">Refresh Button </a>and <a href="https://hub.knime.com/knime/extensions/org.knime.features.js.quickforms/latest/org.knime.js.base.node.widget.input.integer.IntegerWidgetNodeFactory">Integer Widgets</a>). Then, based on the number of rows the table has, two different tables are displayed. This describes perfectly the process of designing the data app in such a way that it would fit all different data instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*DxvHt3OMvmH6QJ5a-22KnA.png" /><figcaption>Fig. 14. View of some nodes in the main dashboard component. Visualization of the most frequent questions in the questionnaire.</figcaption></figure><p>In addition to the above, we list below some useful, advanced tricks to improve the final rendering of your dashboards.</p><ul><li><strong>Tag Clouds customization</strong>: The default look is already good enough, but playing around with the display options unveils a lot of features to create a tailored tag cloud. For instance, the font family of the words, the orientation of the words, and the word colors are all features that can be customized.</li></ul><blockquote>Learn how to <a href="https://www.knime.com/blog/how-to-assign-colors-to-bars-in-bar-chart">assign colors to bars in a bar chart</a>.</blockquote><ul><li><strong>Color assignment</strong>: It is possible to change the color of all charts using the <a href="https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.viz.property.color.ColorManager2NodeFactory">Color Manager</a> node. It may be a little tricky at the beginning but the result is definitely worth the effort. Take also into consideration the usage of loops ( <a href="https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.meta.looper.chunk.LoopStartChunkNodeFactory">Chunks Loop</a> nodes) to make the color assigning smoother and more flexible (Fig. 18).</li><li><strong>CSS styling</strong>: Sometimes the default styling options in your dashboard may require that extra kick to look as you envisioned them.. If this is your case, you can overwrite the styling options of graphs using a customizable CSS sheet included in a dedicated node ( <a href="https://hub.knime.com/knime/extensions/org.knime.features.js.views/latest/org.knime.js.base.node.css.editor.CSSEditorNodeFactory">CSS Editor </a>node). The customized CSS will be propagated to the view nodes via a flow variable.</li><li><strong>Widgets</strong>: The widget nodes allow you to do a lot of things that otherwise wouldn’t be possible: (i) providing input parameters to be used in downstream nodes; (ii) enhancing interactivity giving the user the options to input custom settings; (iii) including text outputs that can be customized using HTML tags (Fig. 19).</li></ul><blockquote>Learn more about <a href="https://medium.com/low-code-for-advanced-data-science/the-wonderful-world-of-widgets-a-mini-guide-for-using-knime-widget-nodes-41f49a19147">KNIME Widget nodes</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*y8SE7DJxDsvVI9AGfUX5rg.png" /><figcaption>Fig. 15. Nodes that create a Tag Cloud for the most applied job positions using a customized palette of colors.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-HFN5TSRwmYdE6fCMwShQ.png" /><figcaption>Fig. 16. Workflow related to a portion of the main dashboard. The widget nodes (Single Selection Widget) are used to introduce more interactivity, requiring the user to make a choice.</figcaption></figure><h3>The Power of Visualization for Quick Data Insight</h3><p>In this article, we discussed how to obtain beautiful data visualizations starting from simple raw data that anyone with a LinkedIn account can request. We went through all the steps build a publicly accessible, multi-page, interactive data app.</p><p>Creating a data app with KNIME is not only fun but also useful. Each visualization contains valuable information, either implicit or explicit. We hope you have fun using it in your job finding process.</p><p>Download and explore the workflow behind the data from the <a href="https://hub.knime.com/-/spaces/-/latest/~6WbbFLXx1ZV-13KF/">KNIME Community Hub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b63cc0eaf66b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/monitor-job-applications-with-a-linkedin-data-app-b63cc0eaf66b">Monitor Job Applications with a LinkedIn Data App</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet the Top KNinjas of Just KNIME It!]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/meet-the-top-kninjas-of-just-knime-it-6ab5d1c7ee5?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ab5d1c7ee5</guid>
            <category><![CDATA[low-code]]></category>
            <category><![CDATA[knime]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[stories]]></category>
            <category><![CDATA[community]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Fri, 23 Dec 2022 00:11:39 GMT</pubDate>
            <atom:updated>2022-12-23T00:13:34.196Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA STORIES | KNIME CHALLENGES | KNIME ANALYTICS PLATFORM</h4><h4>Celebrating our Top 4 KNinjas</h4><p><em>Co-author: </em><a href="https://www.linkedin.com/in/alinebessa/"><em>Aline Bessa</em></a></p><p>We have reached the end of the Just KNIME It! challenges and are now ready to announce our KNinjas!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0OWS7AA0UEC3XE6QGIBTQ.png" /><figcaption>The 4 Top KNinjas of the Just KNIME It! Challenges.</figcaption></figure><blockquote><a href="https://www.knime.com/just-knime-it?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">Just KNIME It!</a> is a series of weekly challenges that ran from January 25 to October 26, 2022, with the goal of helping KNIME users prove their KNIME knowledge and practice their workflow building skills. For this time period we issued a total of 40 challenges. Each week we published a challenge (levels easy to difficult), and gave a week for our community to solve it before publishing our own solution. Solutions were uploaded onto the personal KNIME Hub space with the tag #justknimeit-xx, where xx was the number of the challenge. For each challenge you’ll find a discussion <a href="https://forum.knime.com/c/just-knime-it/66">space on the KNIME Forum</a> plus a <a href="https://forum.knime.com/t/top-10-kninjas/43015">leaderboard </a>ranking participants by the number of submitted solutions.</blockquote><p>Chosen, based on four criteria — the number of submitted solutions, creativity, efficiency, and teaching value to others — the 4 top KNinjas are:</p><ol><li>Yasue Katsutaka (aka <a href="https://forum.knime.com/u/KNIMEST">KNIMEST</a> on the KNIME Forum)</li><li>Emiliano Amendola (aka <a href="https://forum.knime.com/u/eamendola">Eamendola</a> on the KNIME Forum)</li><li>Raffaello Barri (aka <a href="https://forum.knime.com/u/lelloba">lelloba</a> on the KNIME Forum)</li><li>Anil Kumar Sharma (aka <a href="https://forum.knime.com/u/AnilKS">AnilKS</a> on the KNIME Forum)</li></ol><p>Congratulations to these KNinjas! They have made it to the 4 KNinjas of the Just KNIME It! challenges.</p><p>Let’s get to know the winners. Read on to find out more about them, get pointers to their space on KNIME Community Hub, see selected challenges — the ones we feel stand out from other competitors, and as well as highlights on how they have contributed to the KNIME community.</p><h3><strong>Yasue Katsutaka </strong>( <strong>KNIME: </strong>@knimest , <strong>Twitter: </strong>@Note_of_KNIME )</h3><p>Yasue works as an Information System Manager for Medicinal Chemistry in Nara, Japan. In his current job, he manages the database used by the life science team for drug discovery research. A medicinal chemist for over 20 years, Yasue knows the importance of using data to drive drug discovery research.</p><h4>His Just KNIME It! Work</h4><p>Yasue completed all 40 Just KNIME It! challenges. You can find all of his JUST KNIME It! solutions in his <a href="https://hub.knime.com/knimest/spaces/Public/latest/KNIMEST_JP~p6WJ5q_Yt19hKcKC/">public space </a>on the KNIME Community Hub.</p><p>His solutions are always meticulous, with each node containing a clear description of what it will do, and all components named distinctly and meaningfully. Similarly, the dashboards contain all required explanations to understand what is depicted in them.</p><p>Other members of the KNIME team have noted that Yasue always does a great job and writes about each and every solution — a tall order to do on a weekly basis (psst… we will talk about his blog later in this article as well).</p><p>We would also like to praise Yasue’s incredible work because it can be used as educational material for new KNIME users as well. Overall, we believe Yasue’s fantastic workflows specialize in clean and helpful documentation, which is why he is one of our winners.</p><h4><strong>Selected Challenge</strong></h4><p>One solution we would like to feature is his workflow for <a href="https://forum.knime.com/t/solutions-to-just-knime-it-challenge-39/48205">Challenge 39</a>: Legislative Limits on Teaching.</p><p>In this challenge, KNIME users played the role of educational journalists investigating “educational gag orders” and were required to create a workflow that examined which states have the most gag orders, and which topics are most discussed overall. Yasue’s solution is shown in Fig. 1. Note the clear description and the comments under every node, as in all his other solutions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iu8xWoOakkkoPx4wdrkM2w.png" /><figcaption>Fig 1. Yasue’s annotated solution to challenge 39 of Just KNIME It!.</figcaption></figure><h4><strong>Extra Community Contribution</strong></h4><p><a href="https://hub.knime.com/knimest/spaces/Public/latest/KNIMEST_JP/JKI_039_Legislative_Limits_on_Teaching~Iq9sfNRL9NtI5VFt">Yasue’s solution</a> can be found not only on the <a href="https://hub.knime.com/">KNIME Community Hub</a>, but also on his <a href="https://note.com/knimesupportteam/n/nc9992cb90a51?magazine_key=m041672f62384#3a73491d-de5d-464c-b402-b0a57d6b5204">blog</a>. This blog — written in Japanese — has become a reference point for the KNIME newbies in Japan. Indeed, each article describes the chain of actions to bring the original data to the final target.</p><p>We were especially moved by his step-by-step instructions and guidance for each and every of the 40 challenges we presented. He also translated all the challenges so that even our Japanese-only-speaking community members can participate with ease. His work embodies the open-source spirit — taking work and making it available and accessible to an even broader audience in real-time.</p><p>Thank you, Yasue, for this wonderful contribution!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*_tnbmv_bo7-uNSYTLSNX_Q.png" /><figcaption>Fig 2. A picture of Yasue’s blog about all the Just KNIME It! Challenges.</figcaption></figure><h3><strong>Emiliano Amendola <em>(KNIME: @eamendola , Twitter: @eamendola5 )</em></strong></h3><p>Emiliano is a Computer Science Engineer from Asunción, Paraguay, and currently works as a Financial Data Analyst at the United Nations. In his career so far, he has mostly worked with banking and financial data, engaging in different data operations: from data cleaning and transformation to the application of machine learning models and the creation of dashboards. His interest in data analysis is connected to his senior years of his university studies, where he was particularly taken by all things data science.</p><h4><strong>His Just KNIME It! Work</strong></h4><p>Emiliano completed 37 Just KNIME It! challenges. You can find his solutions in his <a href="https://hub.knime.com/eamendola/spaces/Just%20KNIME%20it%20!/latest/~xkd9HssEnh8zVW9p/">dedicated space</a> on the KNIME Community Hub. Emiliano constantly blows us away with his amazing use of components to create beautiful works of art. Within the KNIME team, we each comment that his dashboards are some of the prettiest we’ve ever seen, the most elegant, and well documented. His workflows certainly specialize in beautiful dashboard design.</p><h4><strong>Selected Challenge</strong></h4><p>In representation of Emiliano’s submitted solutions, we have selected the workflow for <a href="https://forum.knime.com/t/solutions-to-just-knime-it-challenge-7/40541">Challenge 7</a>: Women in Government. In this challenge participants were asked to create one or more visualizations to inspect the percent share of seats held by women in local government, as reported by the <a href="https://hdr.undp.org/en">United Nations Development Program: Human Development Reports </a>. The purpose of this report was to “ensure women’s full and effective participation and equal opportunities for leadership at all levels of decision-making in political, economic and public life”. Emiliano’s <a href="https://hub.knime.com/eamendola/spaces/Just%20KNIME%20it%20!/latest/Challenge%2011%20-%20Mean%20Time%20to%20Close%20a%20Ticket~kV3QpmCIfFfSRgKw">solution </a>stood out for the unique use of layout, interactivity, and text size control — getting the right balance of all three is quite a feat, so he is clearly aesthetically gifted.</p><figure><img alt="Just KNIME It! KNinjas" src="https://cdn-images-1.medium.com/max/1020/0*jwwK0utV5mWduG4G.gif" /><figcaption>Fig. 3. Emiliano’s well-designed dashboard, made for challenge 7 of Just KNIME It!, as a GIF.</figcaption></figure><h4>Extra Community Contribution</h4><p>Not only for this challenge, but for almost any challenge with visual requirements, Emiliano constantly went above and beyond to offer excellent eye candy with his dashboards. And what we also love about this is that he brings his creations to LinkedIn and shows them off to the world.</p><p>Thanks for sharing your work with all of us in LinkedIn land, Emiliano!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EiNGY5vTisvmnz2wPMhusg.png" /><figcaption>Fig. 4. Emiliano’s LinkedIn post with an eye-pleasing solution to challenge 37 of Just KNIME It!</figcaption></figure><h3><strong>Raffaello Barri <em>(KNIME: @lelloba )</em></strong></h3><p>Raffaello graduated in Engineering Management, specializing in supply chain management two years ago. He then started working as a Business Process Specialist at Arcese in Milan, Italy. Raffaello discovered his interest in data-driven applications while working on his Master’s thesis, where he developed a logistics network. He encountered KNIME in August 2021 and quickly became passionate about the tool, using it not only for work but also for hobby projects.</p><h4><strong>His Just KNIME It! Work</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*UJmQ06NlFFSBxPfLY7Xu7w.png" /></figure><p>Raffaello completed all 40 Just KNIME It! challenges. You can find his solutions in his <a href="https://kni.me/s/_wKjnP3vpyRYs4y1">space on the KNIME Community Hub </a>. His solutions often make use of components to encapsulate and recycle useful functionalities. Raffaello is a master in <a href="https://www.knime.com/blog/best-practices-to-build-components">component</a> building, often even going to great lengths to find the most appropriate icon for them. And he also makes sure his workflows are clean and crisp, annotating each appropriately and making sure they are aesthetically pleasing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mCyhGAucuM72pKyHUiX_cg.png" /><figcaption>Fig. 5. Raffaello’s well-documented and visually appealing solution to challenge 9 of Just KNIME It!</figcaption></figure><h4><strong>Selected Challenge</strong></h4><p>In representation of Raffaello’s submitted solutions, we have selected the workflow for <a href="https://forum.knime.com/t/solutions-to-just-knime-it-challenge-31/46326">Challenge 31</a>: Extracting Keywords from a Website. Here, participants were tasked with extracting text from a recipe web page, performing a keyword search, and then creating a tag cloud of the keywords.</p><p>The solution should have shown the tag cloud of the keywords and produced a guess for the recipe! Looking at <a href="https://hub.knime.com/lelloba/spaces/Public/latest/Just%20Knime%20it!/justknimeit%20-%2031%20-%20Raffaello%20Barri~u1AyGdTdx4985klD">Lelloba’s solution </a>via the tag cloud below, can you guess what the ingredients will ultimately produce?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4cPzN1fd0Vw1VbzJhVbVg.png" /><figcaption>Fig. 6. Raffaello’s tag cloud solution to challenge 31 of Just KNIME It!</figcaption></figure><h4><strong>Extra Community Contribution</strong></h4><p>Apart from being a prime competitor in the Just KNIME It! challenges, Raffaello also completed the <a href="https://medium.com/low-code-for-advanced-data-science/data-exploration-in-66daysofdata-with-knime-a0092b78b87f">#66daysofdata</a> challenge, but away from the social media spotlight. He also spends time on the <a href="https://forum.knime.com/t/is-there-any-function-in-knime-same-as-text-to-columns-in-excel/48266">KNIME Forum helping others</a>. His posts on the Forum are clearly successful, having received over 200 likes! His dedication to KNIME and to KNIME users in need is truly something we love to see and reward!</p><p>Thank you for helping us make the KNIME community even better, Raffaello!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDqCpSHG3rLpmawbdBobeQ.png" /><figcaption>Fig. 7. Raffaello helping another KNIME user on the <a href="https://forum.knime.com/">KNIME Forum</a>.</figcaption></figure><h3><strong>Anil Kumar Sharma <em>(KNIME: @AnilKS , Twitter: @anilsharma2910 )</em></strong></h3><p>Anil is a DGM Purchase at Dabur India Limited (A leading FMCG Company) from Delhi NCR, India, and currently handles soft commodity sourcing for the CPG industry. His interests in data analytics mainly sparked within the scope of his job, where he usually deals with large amounts of diverse structured and unstructured data, and is responsible for the analytics and automation of various supply management processes.</p><h4><strong>His Just KNIME It! Work</strong></h4><p>Anil completed all 40 Just KNIME It! challenges. You can find his solutions in his <a href="https://kni.me/s/qYhDKtyJsGPND0EU">dedicated space</a> on the KNIME Community Hub. His solutions are efficient, to the point, and sometimes the very first solution we’d receive for a challenge! We really noticed that he’d often use the least number of nodes to reach the goal of the challenge — and in this case less is far more. His workflows specialize in compactness making them approachable and readable even for the newest of KNIME users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yE7tkqDEooPd_a5dcKlyBA.png" /><figcaption>Fig. 8. Anil’s elegant solution to challenge 27 of Just KNIME It!</figcaption></figure><h4><strong>Selected Challenge</strong></h4><p>In representation of Anil’s submitted solutions, we have selected the workflow for <a href="https://forum.knime.com/t/solutions-to-just-knime-it-challenge-27/45505">Challenge 27</a>: “Rare Blood Types”. Here the goal was to help a group of researchers find the number of citizens with O blood type per US state within some messy data. Anil was amongst the first challengers to submit a very practical and compact solution that quickly gets to the point and generates the needed graphics. Notice the beauty of just 7 nodes and one component!</p><p>He also approached the problem a little differently. Whereas in our solution we used the Regex Split node to extract the state from the address line into a separate column, Anil more strategically used the Cell Splitter node instead, making an approach that those without regex knowledge can easily follow.</p><h4><strong>Extra Community Contribution</strong></h4><p>Anil was also one of the participants of the first edition of the social media challenge <a href="https://medium.com/low-code-for-advanced-data-science/data-exploration-in-66daysofdata-with-knime-a0092b78b87f">#66daysofdata with KNIME</a>. He was the first user to complete the challenge, documenting his progress daily on Twitter. His tweets kept him and others well motivated to keep learning KNIME for 66+ days! Great job, Anil!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZK8FKJqzAtHUKRO2rrEcyQ.png" /><figcaption>Fig. 9. Anil’s animated bar chart tweet on Day 50 of the social media challenge “66 days of data with KNIME”.</figcaption></figure><h3>Congratulations to our newest KNinjas!</h3><p>Double congratulations indeed for having completed most of, if not all, the challenges and for having highlighted, each one of them, a different aspect of the versatility of the KNIME Analytics Platform. We at KNIME appreciate all your hard work and dedication, so thank you, top 4! You are the elites in our book and we look forward to seeing how you’ll make use of KNIME going forward!</p><p>P.S. The Just KNIME It! challenges have come to end… but… will there be a season 2? Stay tuned for what happens next with the Just KNIME It! challenges.</p><p>— — — — -</p><p>As first published on the KNIME Blog: <a href="https://www.knime.com/blog/top-kninjas-just-knime-it">https://www.knime.com/blog/top-kninjas-just-knime-it</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ab5d1c7ee5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/meet-the-top-kninjas-of-just-knime-it-6ab5d1c7ee5">Meet the Top KNinjas of Just KNIME It!</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Wonderful World of Widgets! A Mini-Guide for using KNIME Widget Nodes]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/the-wonderful-world-of-widgets-a-mini-guide-for-using-knime-widget-nodes-41f49a19147?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/41f49a19147</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[dashboard]]></category>
            <category><![CDATA[widget]]></category>
            <category><![CDATA[knime]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Wed, 10 Aug 2022 06:49:47 GMT</pubDate>
            <atom:updated>2022-08-10T06:49:47.837Z</atom:updated>
            <content:encoded><![CDATA[<h4>GETTING STARTED | WIDGET NODES | KNIME ANALYTICS PLATFORM</h4><h4>Create interactive data flows in just a few clicks</h4><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/max/1022/0*v_eETivPnPptjrNJ.gif" /><figcaption>Fig. 1. Here, the Column Filter widget and Refresh Button Widget enable us to interact with our data via and select the columns we wish to render as output.</figcaption></figure><p>So, what is a Widget node?</p><p>Use Widget nodes in your KNIME workflows to enable interaction with your data analysis by selecting, filtering, and entering values, and by applying new settings, all in an interactive view. This works both in <a href="https://www.knime.com/knime-analytics-platform?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">KNIME Analytics Platform</a> via <a href="https://docs.knime.com/latest/analytics_platform_components_guide/index.html#components-composite-views?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">Component Composite Views</a> and on the <a href="https://www.knime.com/knime-software/knime-webportal?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">KNIME WebPortal</a> via <a href="https://www.knime.com/data-apps?utm_source=medium&amp;utm_medium=organic&amp;utm_term=&amp;utm_content=&amp;utm_campaign=com_journal">Data Apps</a>. For example, above we interact with our data and select the columns we wish to render as output.</p><p>We enabled this interaction by using two widgets:</p><ol><li>The <a href="https://kni.me/n/Z6l9C4-_s7AKAtSx">Column Filter Widget</a>: The Column Filter Widget allows us to interactively choose the columns we want displayed.</li><li>The <a href="https://kni.me/n/EcMfHYYNQJgJmaas">Refresh Button Widget</a>: The Refresh Widget allows us to change our minds at any point and hit the yellow button to update our selection.</li></ol><p>These are just two kinds of widget nodes available in KNIME Analytics Platform. There are more kinds to get to know!</p><h3>Types of Widgets</h3><p>Widget nodes come in many flavors:</p><ul><li>Input widgets: input data — such as text</li><li>Selection widgets: select data — such as one or more variables</li><li>Output widgets: output data — such as an image</li><li>Filter widgets: filter data — such as one or more columns</li><li>Re-execution widget: re-execute downstream nodes — via the Refresh Button Widget</li></ul><p>Now, let’s look at each type and give you some examples of their uses.</p><h4><strong>Input Widgets</strong></h4><p>These widgets input different parameters into a workflow. For example, below we demonstrate the <a href="https://kni.me/n/cEoqAZ4LaFNAvWrK">Date&amp;Time Widget</a>, which allows you to interactively input a date to be used later.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/max/720/0*PzxdgLaLjGIdqBGt.gif" /><figcaption>Fig. 2: In this example, we used the Date&amp;Time Widget, which allows you to interactively input a date to be used later.</figcaption></figure><h4>Selection Widgets</h4><p>These widgets select input values from a given list. For example, you can choose a specific value or values of a chosen column to filter a data table.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/max/1022/0*BAM_tglGuqlro6Gh.gif" /><figcaption>Fig. 3. Using selection widgets in our underlying workflow we can enable interaction — here by choosing a specific value or values of a chosen column to filter a data table.</figcaption></figure><h4>Output Widgets</h4><p>These widgets produce links to download files, or display images or text.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*PHmrPcQ94KiD_fBS" /><figcaption>Fig. 4: A workflow showing how the Image Output widget can be used.</figcaption></figure><h4>Filter Widgets</h4><p>These widgets trigger interactive filtered views. For example, in the graphic below, we want to focus on all data above 6 and below 18.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/max/734/0*ttOo2ZhXCHbXvs2O.gif" /></figure><h4>Re-execution Widgets in KNIME Version 4.4</h4><p>The <a href="https://kni.me/n/EcMfHYYNQJgJmaas">Refresh Button Widget</a> node creates a button to re-execute downstream nodes of the workflow in a component’s composite view. You can decide which workflow segments should be re-executed by where you place the node. The (re-)execution is controlled by clicking the yellow button in the view.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*V9o6PklJA3WF9nCE" /><figcaption>Fig. 6: Clicking the yellow button refreshes the workflow segment controlled by the Refresh Button Widget node.</figcaption></figure><blockquote><strong><em>Note.</em></strong><em> If you’re still not sure why you would want a Refresh Button Widget node in your workflow, check out </em><a href="https://medium.com/low-code-for-advanced-data-science/eight-data-app-designs-with-the-refresh-button-5fa06f7d5185"><em>Eight Data App Designs with the Refresh Button</em></a><em>. It’s a great reference if you’re interested in seeing other examples of how powerful widgets can be.</em></blockquote><h4><strong>Re-execution in KNIME Version 4.5 and Above</strong></h4><p>What if you don’t want to click that yellow button every time you make a choice? Is there a way to automatically reload the page whenever a column or option is selected? There is in KNIME Analytics Platform version 4.5 and above!</p><p>Simply right-click the appropriate node, select “Configure” and look for the “Re-execution” tab. Select that tab and check the box next to “Re-execution on widget value change”. Here’s what that looks like:</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*lHGQWW1nVC0_FQwv" /><figcaption>Fig. 8: The automatic re-execution functionality option in a Widget node.</figcaption></figure><p>Also note there is a small icon that appears on a widget with re-execution enabled.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*y6rRiHnHm9AgWmUf" /><figcaption>Fig. 8: The small refresh icon in the upper-right corner appears on a widget with re-execution enabled.</figcaption></figure><p>So does this mean the Refresh Button Widget node is deprecated in these versions? Absolutely not. You can still opt to use the node instead of checking this option.</p><p>But why would you use the node instead of this awesomely automagical reloading? Think about it: What if you have a large amount of data, and reloading each time you make a selection is too time-consuming? In that case, the node would be the simpler method by far.</p><h3>Putting It All Together</h3><p>We’ve just learned about the different widget types, and have seen basic examples of their use. Did you know you can use widgets to build your first data app? Read all about it in the article: <a href="https://medium.com/low-code-for-advanced-data-science/how-to-create-an-interactive-dashboard-in-three-steps-with-knime-analytics-platform-c09d6d4f91b2">Build a Data App: Create an Interactive Dashboard in Three Steps with KNIME Analytics Platform</a>.</p><p>Now, let’s do something really challenging and novel: Let’s build a data app that makes regex interactive! You can download this workflow from the <a href="https://hub.knime.com/victor_palacios/spaces/Public/latest/Regex%20with%20Widgets~Sax0dvYR26tmiKwI">KNIME Hub</a> for free, if you want to follow along.</p><h4>What Is Regex?</h4><p>Regular expressions, commonly called regex, are a series of special characters that allow us to find specific text in a document. For example, let’s say I have 10,000 documents, and I want to locate a single date somewhere in there. I know I always use the same date format: YEAR-MONTH-DAY. For example: 2021–10–1. I can therefore use this regex to extract every instance of this date from every document: [0–9]{4}-[0–9]{1,2}-[0–9]{1,2}. The following figure describes how to read each part of this regex:</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*QHAIyiTi8NkeyEVO" /><figcaption>Fig. 9: An explanation of what each element in this regular expression means.</figcaption></figure><p>For the purposes of this article, this is all the regex you will need to know. (<em>Hooray!</em>)</p><h4><strong>Interactive Regex Workflow</strong></h4><p>Now let’s use several different widgets to make an interactive view, one in which we can enter a regex to extract the date we need from each document.</p><figure><img alt="A mini-guide for using KNIME widget nodes effectively" src="https://cdn-images-1.medium.com/proxy/0*MTDccmxAZ3YvDAdm" /><figcaption>Fig. 10: The interactive Regex workflow in just three nodes.</figcaption></figure><p>The trick with widgets is that they typically only behave the way you expect them to when you encapsulate them inside a component. This is done by selecting all nodes you want to put inside a component and then right-clicking to create that component. For more details, see <a href="https://www.analyticsvidhya.com/blog/2021/11/a-friendly-introduction-to-components-with-knime-analytics/">KNIME Tutorial — A Friendly Introduction to Components using KNIME Analytics Platform</a>. Let’s see what that gray node (the component) looks like on the inside:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zKS04WoqSOrsoGLVFHQ9Tw.png" /><figcaption>Fig. 11: The contents of a component which results in interactive regex.</figcaption></figure><blockquote><strong><em>Note.</em></strong><em> In this figure, we are using </em><a href="https://medium.com/low-code-for-advanced-data-science/knime-analytics-flow-variables-the-red-line-95df14ce94c9"><em>flow variables</em></a><em> (the red connections) to pass input parameters to other nodes.</em></blockquote><h4>Widgets used in the Interactive Regex Workflow</h4><p>Notice that we could have used two Refresh Button Widgets (placing one before the Column Filter Widget, as we did in previous examples) to create <a href="https://medium.com/low-code-for-advanced-data-science/eight-data-app-designs-with-the-refresh-button-5fa06f7d5185">sequential logic</a> (i.e., user selection impacts what appears on the screen next). But because of the new re-execution functionality in KNIME Analytics Platform 4.5, we can simply check the box for automatic re-execution in the configuration window of the <a href="https://kni.me/n/Z6l9C4-_s7AKAtSx">Column Filter Widget</a>.</p><p>Now let’s get to the real meat of the conversation: the superstar widgets. The <a href="https://kni.me/n/0it_BBA7j3jvAtZx">String Widget </a>node allows us to enter our regex interactively, and the <a href="https://kni.me/n/EcMfHYYNQJgJmaas">Refresh Button Widget </a>allows us to make changes dynamically without going back to the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tFjBCnOGE2Sy4jBBMGJ3-g.png" /><figcaption>Fig. 12: An interactive view showing modified regex to extract year and month only.</figcaption></figure><p>Finally, we added a <a href="https://kni.me/n/j7PCL2QTaG_QlYoA">Text Output Widget </a>which generates some fun text at the bottom of the interactive view. But you can use it for more serious things, like headers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WW5SjXprMCyGq18h0k0zhg.png" /><figcaption>Fig.13: Output from the Text Output Widget in an interactive view.</figcaption></figure><h3>Widgets for more Interactivity with Your Data Analysis</h3><p>On KNIME Analytics Platform and the KNIME WebPortal, use a widget node when you want to be able to interact with your data. Here we demonstrated the major types of widgets, and gave examples to demonstrate how they work. You can download a free workflow <a href="https://hub.knime.com/victor_palacios/spaces/Public/latest/Regex%20with%20Widgets~Sax0dvYR26tmiKwI">here</a> which we have built to let you play with widget nodes.</p><p>If you are looking for further documentation, see the <a href="https://docs.knime.com/2022-06/analytics_platform_components_guide/index.html#introduction">KNIME Components Guide</a> and if you would like to try some more <a href="https://hub.knime.com/knime/spaces/Examples/latest/09_Enterprise/02_WebPortal/03_Re-execution_Widgets_Examples~_mhOU2v4mgufL_Nl/">Re-execution Widget Examples </a>you can download them from the KNIME Hub. Thanks for reading!</p><p>— — — — -</p><p>As previously published on the KNIME Blog: <a href="https://www.knime.com/blog/mini-guide-widget-examples">https://www.knime.com/blog/mini-guide-widget-examples</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41f49a19147" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/the-wonderful-world-of-widgets-a-mini-guide-for-using-knime-widget-nodes-41f49a19147">The Wonderful World of Widgets! A Mini-Guide for using KNIME Widget Nodes</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Too Much Data or Not Enough? Solve with Statistical Sampling Strategies]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/too-much-data-or-not-enough-solve-with-statistical-sampling-strategies-50f498485c8f?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/50f498485c8f</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[theory]]></category>
            <category><![CDATA[sampling]]></category>
            <category><![CDATA[knime]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Tue, 03 May 2022 13:39:12 GMT</pubDate>
            <atom:updated>2023-08-11T21:05:22.853Z</atom:updated>
            <content:encoded><![CDATA[<h4>DATA SCIENCE THEORY | SAMPLING STRATEGIES | KNIME ANALYTICS PLATFORM</h4><h3>Too Much Data or Not Enough? Solve with Statistical Sampling</h3><h4><em>The pros and cons of statistical sampling methods plus how to find the right strategy</em></h4><p><em>Co-authors: </em><a href="https://www.linkedin.com/in/maarit-widmann-02641a170/"><em>Maarit Widmann</em></a><em> and </em><a href="https://www.linkedin.com/in/roberto-daniele-cadili-7422471b6/"><em>Roberto Cadili</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MroDXwEyXL5D4bot" /><figcaption>Photo by <a href="https://unsplash.com/@testalizeme?utm_source=medium&amp;utm_medium=referral">Testalize.me</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>.</figcaption></figure><p>Analyzing a population requires a representative sample of it that reflects the characteristics of the population. Otherwise the analysis of the sample won’t generalize to the population. At the same time, a carefully designed sample is enough to reveal the patterns and characteristics of the whole population, should that population be unreachable as a whole, too expensive to collect, or is evolving quickly.</p><p>On the other hand, we can also capitalize on statistical sampling methods to tackle the problem of the lack of rare examples and thus uneven data representation, which affects the performance and generalizability of machine learning models.</p><p>To ensure that our samples are representative and maximize the performance of a machine learning algorithm, in this article we want to:</p><ul><li>Explain some of the most popular statistical sampling techniques</li><li>Walk through a use case that highlights the effect of a sample design to model performance</li><li>Point you to a workflow that illustrates how some of these sampling strategies can be carried out in <a href="https://www.knime.com/knime-analytics-platform?utm_source=medium&amp;utm_medium=social&amp;utm_term=&amp;utm_content=&amp;utm_campaign=Community">KNIME</a>.</li></ul><blockquote><em>The </em><a href="https://hub.knime.com/victor_palacios/spaces/Public/latest/Sampling%20Strategies%20Comparison~ouHL6-Ca44-UK-Wt"><em>workflow</em></a><em> used throughout this article is freely available on the </em><a href="https://hub.knime.com/"><em>KNIME Hub </em></a><em>if you would like to play with the data or experiment with different strategies by yourself.</em></blockquote><h3>Five Designs for Samples to Represent Your Population</h3><p>Let’s first tackle the issue of designing a random sample that represents your population the best. We will introduce the following strategies, and give examples on which type of population they work the best for:</p><ul><li>Random sampling</li><li>Stratified sampling</li><li>Cluster sampling</li><li>Linear sampling</li><li>Take from the top</li></ul><h4>Random sampling</h4><p>You could say that <strong>simple random sampling </strong>is a bit like a lottery where we are randomly selecting a number from a box. For data scientists, it means we are randomly picking a data point from our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyuICpPb6xw3k67So4ctdw.png" /><figcaption>Fig. 1. Random sampling compared to lottery number drawings.</figcaption></figure><p>Put more technically, in simple random sampling we don’t introduce any bias to how we select a sample from the population. Every possible sample of a given size has the same chance of selection.</p><p>While simple random sampling is praised for its simplicity and lack of bias, it doesn’t guarantee the presence of all types of samples nor does it preserve the order of the data. If this is what you need, then we can use one of the sampling strategies below.</p><h4>Stratified sampling</h4><p>In stratified sampling, we consider that a feature partitions a heterogeneous population into homogeneous subgroups — called strata — and that our analysis will vary among the different subgroups. With stratified random sampling we can ensure that representatives of all subgroups are included in the sample.</p><p>In a sample extracted by stratified sampling, the distribution of the strata will be the same as in the population. For example, let’s say we randomly select 5 players from a pool of 3 goalkeepers (yellow in Figure 2) and 17 outfield players (gray in Figure 2). If we did this via stratified sampling, then we will likely have 3 ⁄ 20 *5 ≈ 1 goalkeeper and 17 ⁄ 20 *5 ≈ 4 outfield players — which sounds like a reasonable team!</p><figure><img alt="Statistical Sampling Strategies" src="https://cdn-images-1.medium.com/proxy/0*e8PMsPUCdOqmxaAi" /><figcaption>Fig. 2. Yellow and gray data points representing a strata in the population whose distribution should be preserved in the stratified random sample.</figcaption></figure><p>Stratified random sampling provides an unbiased and accurate representation of the a priori distribution of a feature in an underlying population. However, it can only preserve the distribution of a nominal feature. Otherwise we need to rely on other sampling strategies, such as cluster sampling.</p><h4>Cluster sampling</h4><p>Cluster sampling partitions the data into clusters that share similar characteristics such as latitude and longitude. It then samples whole clusters randomly from the population. All the members from these randomly selected clusters constitute the final sample.</p><p>If that sounds the same as stratified sampling, the <strong>difference </strong>between <strong>stratified </strong>and <strong>cluster sampling </strong>is that in stratified sampling we identify groups and then randomly sample from those groups. In cluster sampling, we identify groups and use <em>some of those groups </em>as our sample. In cluster sampling the groups can also overlap.</p><p>For example, suppose you want to investigate how well your product would do in a new country, let’s say Japan. Instead of performing stratified sampling for each town in Japan, we would carry out cluster sampling by identifying cities and then carrying out an experiment only in certain cities, but not all of them. This makes the sampling process much more efficient!</p><p>But cluster sampling has some drawbacks. The most important one is its reliance on the design of each cluster. The quality of the clusters and how accurately the sampled clusters represent the underlying population heavily determine the validity of our results. In practice, clusters often yield a partial representation of the population’s characteristics, as they tend to be more internally homogeneous than the population as a whole, hence potentially leaving out essential characteristics.</p><p>Last, we introduce two simple sampling techniques, <em>linear sampling </em>and <em>take from the top</em>.</p><h4>Linear sampling</h4><p>Linear sampling extracts a data point every <em>n </em>samples, for example, every 10th observation in the population. This sampling strategy is a simple way of extracting a sample that contains all types of data from the population, yet it only works if the data doesn’t show any pattern at regular intervals. This is often the case in time series data, for which you can then use the <em>take from the top </em>strategy, which we introduce next.</p><h4>Take from top</h4><p>The Take from top sampling strategy preserves a sample from the population as a chunk from the top. This strategy is used for <a href="https://www.knime.com/blog/building-a-time-series-analysis-application">time series</a> data, where the time order of observations in the population needs to be preserved in the extracted sample. Furthermore, when partitioning data for a time series model, this strategy ensures that the model is <em>trained </em>on past data (at the top) and <em>tested </em>on future data (at the bottom). However, this strategy doesn’t guarantee that the data represents the entire population and shouldn’t be used when the goal is to extract a random sample.</p><p>These statistical sampling techniques are formidable allies to constitute a representative sample of our population, but they can also tackle, for example, the problem of imbalanced datasets.</p><h3>Two Designs for Sampling Imbalanced Datasets</h3><p>Here we tackle the second issue with random sampling — imbalanced data. Imbalanced datasets pose a problem of representation, in that one group — the minority class — contains significantly fewer samples than the other group — the majority class. Class imbalance is especially detrimental to the construction of generalizable machine learning models as they learn to predict the majority class accurately but fail to capture the minority class. If adding new data is not a feasible option, to balance uneven datasets, we can use two sampling techniques: undersampling or oversampling.</p><ul><li><strong>Undersampling </strong>works by keeping all of the data in the minority class and decreasing the size of the data in the majority class to equal the size of the minority class. The most remarkable disadvantage of undersampling is the loss of potentially important information for the training of models, since data is removed without any consideration of what it is or how meaningful it might be for analysis.</li><li><strong>Oversampling</strong>, on the other hand, solves the problem of imbalanced datasets by increasing the number of data points in the minority class to equal the size of the majority class.</li></ul><p>We introduce two popular techniques for under- or oversampling called bootstrapping and synthetic minority oversampling technique (SMOTE).</p><h4>Bootstrapping</h4><p>This is simple random sampling but administered <em>with replacement, </em>that is, once a data point is picked it goes back into the population and thus may be chosen more than once. This technique can create duplicates of the data points in the minority class and also discard some representatives of the minority class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9E3LaQj0Kj614S5_Nu_bQ.png" /><figcaption>Fig. 3. Visualizing the oversampling of the minority class using bootstrapping.</figcaption></figure><p>The figure above shows a population with imbalance classes grey/yellow on the left. On the right is a bootstrap sample of the minority class (yellow), which contains duplicates due to the replacement.</p><h4>Synthetic Minority Oversampling Technique (SMOTE)</h4><p>This works by randomly picking a data point from the minority class and one of its k-nearest neighbors within the minority class. The synthetically generated data point lies then at a random point on the vector connecting these two data points in the feature space. It is worth mentioning that this technique does not create duplicates.</p><p>While oversampling techniques have the clear advantage of not losing potentially important information in our majority class, they can be computationally expensive and slow down the speed of our analysis.</p><blockquote><em>Learn more about oversampling:</em></blockquote><blockquote><a href="http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume16/chawla02a.pdf"><em>SMOTE: Synthetic Minority Over-sampling Technique</em></a></blockquote><blockquote><a href="https://www.youtube.com/watch?v=iPmNWjWoFIs&amp;t=2s"><em>Data Science Pronto! Bootstrapping </em></a><em>— KNIMETV, YouTube</em></blockquote><p>Now that we have talked about the uses for different sampling strategies, let’s try out these techniques on a healthcare dataset on <a href="https://www.kaggle.com/fedesoriano/stroke-prediction-dataset">Kaggle </a>for predicting stroke events.</p><h3>Example Workflow: Experiment with Statistical Sampling</h3><p>We’ve built a workflow that you can use to experiment with simple random sampling and stratified random sampling (<a href="https://kni.me/n/T0AhxSe0Yi42rQK8">Partitioning</a> node), undersampling (<a href="https://kni.me/n/nZoZFKRELGnk6y5i">Equal Size Sampling</a> node) and oversampling (<a href="https://kni.me/n/i1MQOr0EbxLu4c1B">Bootstrap Sampling</a> node and <a href="https://kni.me/n/YznaX_v45d3OzPEV">SMOTE </a>node).</p><p>The workflow draws on the kaggle <a href="https://www.kaggle.com/fedesoriano/stroke-prediction-dataset">Stroke Prediction Dataset </a>which represents 5110 rows with 11 clinical features such as body mass index, smoking status, age, gender, and glucose level. The task is to predict stroke (yes/no), which is a classification problem. We chose to build a <a href="https://www.knime.com/blog/fraud-detection-using-random-forest">Random Forest</a> model in KNIME.</p><p>The data is highly imbalanced as you can see in the Figure 4 generated by the <a href="https://kni.me/n/F0ZMkNAGudnmoAyE">Data Explorer</a> node. The histogram shows that most of the people (over 90%) have not had a stroke.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*UNz4YtQO6AWVkaYUSRl5ew.png" /><figcaption>Fig. 4. Visualizing class imbalance with the Data Explorer node.</figcaption></figure><blockquote><strong><em>Note.</em></strong><em> “Imbalanced data” is a term generally reserved for 90% or more of one class and 10% or less of another class, but that is a guideline — not a rule. In extreme cases like fraud detection, fraudulent transactions may constitute less than 1% of the data.</em></blockquote><h4>Introducing the workflow</h4><p>The workflow in Figure 5 experiments with different sampling strategies of the training and test sets (aka partitioning) and with different strategies to handle class imbalance in the training data (aka resampling). Finally, it compares their performances in the stroke prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-4Ztzk7POB5VXO_LmXmVA.png" /><figcaption>Fig. 5. Implementing different sampling strategies in a KNIME workflow.</figcaption></figure><p>The workflow implements random partitioning as the baseline at the top and stratified partitioning for the remaining models. Below we point to the nodes that implement the different sampling strategies:</p><ul><li>The <a href="https://kni.me/n/T0AhxSe0Yi42rQK8"><strong>Partitioning</strong></a><strong> </strong>node creates the training and test sets via the options for random, stratified, linear and take from top in its configuration dialog:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*4ImT6P_MLAmnyb5iKtq8wQ.png" /><figcaption>Fig. 6. Sampling technique choices in the Partitioning node.</figcaption></figure><ul><li>The <a href="https://kni.me/n/YznaX_v45d3OzPEV"><strong>SMOTE</strong></a><strong> </strong>node performs oversampling of the minority class on the training data. It allows us to choose the number of nearest neighbors, as well as whether to oversample the minority class only or to oversample each class equally.</li><li>The <a href="https://kni.me/n/nZoZFKRELGnk6y5i"><strong>Equal Size Sampling</strong></a><strong> </strong>node performs undersampling of the majority class by randomly discarding majority class data.</li><li>The <a href="https://kni.me/n/i1MQOr0EbxLu4c1B"><strong>Bootstrap Sampling</strong></a><strong> </strong>node allows for bootstrap sampling to obtain a certain sample size defined as % of the input data or as the absolute sample size. Additionally, we have the option to append the count of occurrences to track the number of duplicate rows for each originally unique data point.</li></ul><p>Finally, we want to check how much we can improve the model’s performance by selecting the most appropriate sampling method.</p><h4>Compare Statistical Sampling Strategy Results</h4><p>We can evaluate the models’ performances and interactively inspect the results by taking advantage of the <a href="https://kni.me/n/3-JGPq9anCe8LGG6">Binary Classification Inspector</a> node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XILTIAc6qPD0cwlqaUaAsA.png" /><figcaption>Fig. 7. Comparing different sampling techniques.</figcaption></figure><p>The top part of the view compares the models in terms of selected accuracy measures, such as area under the curve, overall accuracy, and sensitivity and also displays their ROC curves. You can see in the bar chart for the accuracy measures that if we do nothing to tackle class imbalance, that is using simple random partitioning (the blue bar) and stratified partitioning (the green bar) only, we obtain very similar — poor — results. In both cases, overall accuracy is almost 95% and the sensitivity is almost 0%. This means, the model is learning to classify only the majority class while having substantially no predictive ability for the minority class.</p><p>Instead, tackling class imbalance improves the model performance. In particular, using SMOTE on the training data (the red bar), we obtain an overall accuracy of 89% and a sensitivity score of 22%.</p><p>However, by far the best performing model is that the one where class imbalance was trained on undersampled training data (the purple bar), since it reaches a sensitivity value of 81%. If we take a closer look at the confusion matrix at the bottom part of the view, we can see that for 64 out of 75 individuals, stroke was correctly predicted. Although 479 individuals were mispredicted as patients who had a stroke when in fact they did not, in specific domains such as disease screening it is usually desirable to falsely alert healthy patients of a risk of suffering a stroke than missing a patient who is actually likely to suffer one.</p><h3>Lessons Learned: The Choice of Sampling Technique Depends on the Data</h3><p>Ultimately, the choice of sampling technique depends on the data and the problem you are trying to answer. Here we have illustrated several situations, explained the pros and cons of different sampling techniques, and highlighted how the sampling strategy can affect the accuracy of your model.</p><p>Download this <a href="https://kni.me/w/ouHL6-Ca44-UK-Wt">workflow</a> which we have built to try out and experiment with different strategies and review the results of each technique. Thank you for reading and happy data science-ing!</p><p><strong>References</strong></p><p>Illowsky, B., Dean, S. L., OpenStax College,, &amp; Rice University,. (2013). <a href="https://assets.openstax.org/oscms-prodcms/media/documents/IntroductoryStatistics-OP_i6tAI7e.pdf">Introductory Statistics</a>.</p><p>— — — — -</p><p>As first published on the KNIME Blog: <a href="https://www.knime.com/blog/statistical-sampling">https://www.knime.com/blog/statistical-sampling</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50f498485c8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/too-much-data-or-not-enough-solve-with-statistical-sampling-strategies-50f498485c8f">Too Much Data or Not Enough? Solve with Statistical Sampling Strategies</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Widget vs Configuration Nodes — What’s the Difference?]]></title>
            <link>https://medium.com/low-code-for-advanced-data-science/widget-vs-configuration-nodes-whats-the-difference-1ba47e858cd2?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ba47e858cd2</guid>
            <category><![CDATA[widget]]></category>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <category><![CDATA[knime]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Mon, 28 Feb 2022 09:01:30 GMT</pubDate>
            <atom:updated>2022-02-28T13:30:44.808Z</atom:updated>
            <content:encoded><![CDATA[<h4>GETTING STARTED | WIDGET &amp; CONFIGURATION | KNIME ANALYTICS PLATFORM</h4><h3>Widget vs Configuration Nodes — What’s the Difference?</h3><h4><strong><em>Everything you always wanted to know (but were afraid to ask)</em></strong></h4><p>Initially they seem to be doing very similar things: Both nodes enable you to provide input parameters for other nodes in the workflow and, unless they are in a component, they seem to provide the same service. Even their dialogs look identical! Just see for yourself in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ob3RrEg-oiCeVUhvXwf_qw.png" /><figcaption>Fig. 1. Widget node vs Configuration node configure options — they look virtually identical.</figcaption></figure><p>So let’s clarify the difference between Widget and Configuration nodes and when to use them. Here is a summary of the differences:</p><ul><li><a href="https://hub.knime.com/search?tag=Widgets"><em>Widget nodes</em></a><em> </em>enable you to interact with the data by selecting, filtering and entering values, and applying the new settings in <strong>both </strong><a href="https://www.knime.com/software-overview?utm_source=medium&amp;utm_medium=social&amp;utm_term=&amp;utm_content=&amp;utm_campaign=Community">KNIME Analytics Platform</a> <strong>and </strong>on the <a href="https://www.knime.com/knime-software/knime-webportal?utm_source=medium&amp;utm_medium=social&amp;utm_term=&amp;utm_content=&amp;utm_campaign=Community">KNIME WebPortal</a>. <a href="https://hub.knime.com/search?q=configuration&amp;tag=Configuration&amp;type=Node"><em>Configuration nodes</em></a><em> </em>enable interaction only on KNIME Analytics Platform.</li><li><em>Configuration nodes </em>implemented inside a component give users the ability to interact with the data to modify the output <strong><em>before </em></strong>running downstream nodes in KNIME Analytics Platform. Use them when you <strong>don’t want </strong>users on the KNIME WebPortal to provide input parameters for other nodes in the workflow.</li><li><em>Widget nodes </em>implemented inside a component give users the ability to interact with the data and modify the output that is then applied to other nodes <strong><em>after </em></strong>running downstream nodes in the <em>KNIME Analytics Platform </em>. Use them when you <strong>do want </strong>users on the KNIME WebPortal to provide input parameters for other nodes in the workflow.</li><li>Summing up: On the KNIME Analytics Platform, either will allow interaction — Configuration nodes before execution and Widget nodes after execution. On the KNIME WebPortal, use Configuration nodes when you want to protect your input parameters, and use the Widget nodes when you want to allow the manipulation of input parameters.</li></ul><p>See the table for where you can modify input parameters in each node with respect to software.</p><figure><img alt="Widget vs Configuration Nodes" src="https://cdn-images-1.medium.com/proxy/0*QJMKXWZ5Ehpqcice" /><figcaption>Fig. 2. Where input parameters can be modified with respect to software and nodes.</figcaption></figure><p>Now read on for a more detailed look at first Widget nodes and then Configuration nodes, when to use them, plus an example workflow that demonstrates both.</p><h4>What Are Widget Nodes and When Do I Use Them?</h4><p>You use Widget nodes to make it possible for users to interact with the workflow — enabling them to change the default settings of nodes that are inside components. For example you can use different Widget nodes to enable the user to add input like credentials, select something to be output by the workflow — a report, or an image, or filter data in a visualization within a certain date range or similar.</p><p>Put more precisely, Widget nodes allow you to set different variables in a so-called component “view”. This selection can be changed on the KNIME Analytics Platform and on the KNIME WebPortal.</p><p>Are There Different Types of Widget Nodes? Yes! Widget nodes come in many flavors:</p><ul><li>Input widgets: user inputs some data (like credentials)</li><li>Selection widgets: user selects some data (like a column)</li><li>Output widgets: user outputs some data (like an image)</li><li>Filter widgets: user interactively filters visualized data</li><li>Re-execution widget: user triggers re-execution of downstream nodes</li></ul><p>For even more details, see the official documentation in the <a href="https://docs.knime.com/2020-07/analytics_platform_components_guide/index.html#introduction">KNIME Components Guide</a>.</p><p><strong>An Example of When to Use a Widget Node</strong></p><p>Here are two examples of how a Widget node was used to allow the user to interact with a dashboard showing Netflix movies and TV shows. The user can look at movie data by year, selecting a certain range.</p><p>This first example shows the Netflix dashboard on your local installation of KNIME Analytics Platform — accessed when you right-click the component and select Interactive View.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7p0ayl-OCyL_4CnjOY6vw.png" /><figcaption>Fig. 3. Widget nodes enable the user to examine movie data by year via the Interactive View of the component.</figcaption></figure><p>Similarly, on the KNIME WebPortal you can use Widget nodes to enable other users to interactively select the parameters — here as above — they can select a certain range of years but via their browser on the WebPortal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EHp8RYn-SLvafnlwn-kcOQ.png" /><figcaption>Fig. 4. Widget nodes have been used to enable the user to interactively examine the movie data by year on the <strong>KNIME WebPortal — via a browser</strong>.</figcaption></figure><p>If you’d like to play with the free KNIME workflow associated with these visuals yourself, you can download it here: <a href="https://hub.knime.com/knime/spaces/Examples/latest/03_Visualization/02_JavaScript/14_Interactive_dashboard_netflix_shows~yK3enSZA775rBPCw">Create an interactive dashboard in 3 steps: Netflix dataset </a>. And if you’d like to read the original blog article you can check that out here: <a href="https://medium.com/low-code-for-advanced-data-science/how-to-create-an-interactive-dashboard-in-three-steps-with-knime-analytics-platform-c09d6d4f91b2">How to Create an Interactive Dashboard in Three Steps with KNIME Analytics Platform</a>.</p><h4>What Are Configuration Nodes and When Do I Use Them?</h4><p>A Configuration node enables you to set column selection configuration options within a component — giving you a dropdown menu, for example. Note that the options in — for example the dropdown menu — can be changed within KNIME Analytics Platform, but <strong>cannot </strong>be changed on the KNIME WebPortal.</p><p>Configuration nodes share the same “Configure…” options with Widget nodes, but with two important changes:</p><ol><li>In KNIME Analytics platform (when in a component) you are given choices for your parameters</li><li>On the KNIME WebPortal you cannot change the parameter settings</li></ol><p>Are There Different Types of Configuration nodes? Yes, but Configuration nodes come in less flavors:</p><ul><li>Input configurations: user inputs some data (like credentials)</li><li>Selection configurations: user selects some data (like a column)</li></ul><p>Note that output, filter, and re-execution only exist for Widget nodes and therefore Widgets offer a broader range of possibilities than their Configuration node counterparts.</p><p><strong>An Example of When to Use a Configuration Node</strong></p><p>In KNIME Analytics Platform, Configuration nodes that are encapsulated by components give us handy dropdown menus (and that is just one type of menu!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_fyPYK8epOpwtnJ_e8rpg.png" /><figcaption>Fig. 5. Convenient drop-down menu when selecting “Configure…” for a Configuration node encapsulated in a component.</figcaption></figure><p>Unlike Widget nodes, on the KNIME WebPortal you cannot use Configuration nodes to give yourself or other users the ability to interactively choose parameters.</p><h4>An Example Workflow Showing Both Widget and Configuration Nodes</h4><p>Now, let’s look at a workflow that has both Widget and Configuration nodes. We’ll touch on when you choose one over the other and how both can be used in a single workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FdvLcGuamMnooI-bphcLfQ.png" /><figcaption>Fig. 6. A workflow containing a configuration node and widget node of the same type (i.e., Value Selection).</figcaption></figure><p>In our <a href="https://kni.me/w/66Wov3tIkrd4nXlV">Widget Nodes vs Configuration Nodes </a>workflow (which you can download from the KNIME Hub to use yourself) we read in our “Customer data” and then pass some nominal value to the row filter. For example, select a product that a customer bought, such as a gold investment package.</p><p>We can see that the outputs are exactly the same, and we notice that even the “Configure…” options are similar.</p><p>However, when we put the Widget node or Configuration node <em>and </em>any other downstream nodes into components, the resulting components behave differently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5fboUjWTz8R2wBm3NL8mbw.png" /><figcaption>Fig. 7. The “Configure…” option for a Configuration component (left) versus a Widget component configuration (right).</figcaption></figure><p><em>Select Product Category via Configuration node</em></p><p>In the Configuration Component (left) you can see how we can select a Product Category. Select the Configuration Component, right-click it, and choose “Configure…”. Then the dialog box will open where you can change the product selection — or modify the configuration to set a totally different variable — “Marital Status” for example. Then this component’s output could be a subset of the data containing only married couples.</p><p><em>Select Product Category via Widget node</em></p><p>In the Widget Component, we can see that the “Component is not configurable”. So to select a product category here, we right-click the executed Widget Component to access the so-called “Interactive View”. Now, we can select from “Column” and “Value” (see fig. 8 below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Hduwfhi3tqa-91C14rlNw.png" /><figcaption>Fig. 8. The “Interactive View” option for a Widget component allows the user to select from a choice of input parameters.</figcaption></figure><h4>How To Decide When to Choose One Over the Other?</h4><p>To decide whether to use a Configuration or a Widget node ask yourself:</p><ol><li>Where will my workflow be used (in KNIME Analytics Platform or will it be deployed to the KNIME WebPortal)?</li><li>If it’s going to be used via the KNIME WebPortal, do I want my user to have a choice of input parameters?</li></ol><p>For example, I am building an application for a different team in my organization or for customers that will be accessible to them via the KNIME WebPortal and I only want them to have access to one category of data. In this case I would choose to use a Configuration node — ensuring that my parameters cannot be altered.</p><p>On the other hand, if I do want my customers to have some choice with respect to, for example, their product, then I would allow parameter choice by using a Widget node.</p><h4>When to Use Both Configuration and Widget Nodes</h4><p>But can I use a combination of Widget nodes and Configuration nodes? Of course! Let’s say we want to give customers access to a subset of the data e.g., married couples, and we don’t want them to be able to change this subset via the KNIME WebPortal. However, we do want them to have the freedom to interact with that subset.</p><p>For example — let’s image we want users on the KNIME WebPortal data to look at different investment products like gold, etc., but we are targeting a specific audience, the married couples. We therefore use the Configuration component to subset the data to only those people in our dataset who are married. Now, when customers interact with the WebPortal, they are free to look at data for different products but this data is restricted to that specific segment of the population.</p><p>To enable this, the associated workflow would look something like this:</p><figure><img alt="Widget vs Configuration Nodes" src="https://cdn-images-1.medium.com/proxy/0*BxpP5vXU_vX00Yu-" /><figcaption>Fig. 9. Using both a Configuration component and a Widget component to give parameter input choice for only a subset of the data.</figcaption></figure><h4>Are the Differences Clear?</h4><p>At the end of the day, when comparing and contrasting Configuration nodes vs Widget nodes, use Configuration nodes when you want to enable quick selection of parameters within KNIME Analytics Platform and use Widget nodes for the KNIME WebPortal. If you do not want to allow users to modify parameter selection on the KNIME WebPortal, use the Configuration nodes.</p><p>Note that Widget nodes give you options for a greater number of use cases which we’ll discuss in a separate article. For this article, we’ve stuck to the basics to show you a simple compare-and-contrast.</p><p>Download this free <a href="https://hub.knime.com/victor_palacios/spaces/Public/latest/Widget%20Nodes%20vs%20Configuration%20Nodes~66Wov3tIkrd4nXlV">workflow </a>which we have built to play with Widget versus Configuration nodes. Thanks for reading!</p><p>— — — — -</p><p>As previously published on the KNIME Blog: <a href="https://www.knime.com/blog/widget-vs-configuration-nodes-whats-the-difference">https://www.knime.com/blog/widget-vs-configuration-nodes-whats-the-difference</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ba47e858cd2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/low-code-for-advanced-data-science/widget-vs-configuration-nodes-whats-the-difference-1ba47e858cd2">Widget vs Configuration Nodes — What’s the Difference?</a> was originally published in <a href="https://medium.com/low-code-for-advanced-data-science">Low Code for Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PCA for Everyone]]></title>
            <link>https://medium.com/@vicpal/pca-for-everyone-bc88a690c042?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc88a690c042</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[titanic-dataset]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 03:41:07 GMT</pubDate>
            <atom:updated>2020-06-16T03:41:07.252Z</atom:updated>
            <content:encoded><![CDATA[<p>By Aneri Dand, Victor Palacios, and Sicheng Zhou</p><p>Here is a google colab notebook which anyone can interact with to learn PCA and apply PCA to the Titanic Dataset. (Click the lower left corner “pca_mini_project.ipynb” to access the file and then hit the “Open in Colab” on the Github Gist page to play with the data yourself.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*ymqP6RO5CM4lfGPPfjFIUA.png" /><figcaption>Created by the Authors</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a79b8246ff9cae89abed24e2dfa67199/href">https://medium.com/media/a79b8246ff9cae89abed24e2dfa67199/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc88a690c042" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Preview of Deception Detection with Natural Language Processing]]></title>
            <link>https://medium.com/@vicpal/a-preview-of-deception-detection-with-natural-language-processing-e62b090ece64?source=rss-d2e2a5174563------2</link>
            <guid isPermaLink="false">https://medium.com/p/e62b090ece64</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lies]]></category>
            <category><![CDATA[deception]]></category>
            <dc:creator><![CDATA[Victor]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 00:50:38 GMT</pubDate>
            <atom:updated>2020-05-21T22:15:56.145Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deception Detection with Natural Language Processing</h3><p>Here’s the code: <a href="https://github.com/Victor-Palacios/Deception_Detection_Capstone">https://github.com/Victor-Palacios/Deception_Detection_Capstone</a></p><p>Pamela Meyer, author of <em>Liespotting</em>, takes the stage for her TED talk, “How to spot a liar”. I ready my pencil, eager to take down every morsel of knowledge she has to offer.</p><p>Around 8 minutes into the talk, she uses the infamous Bill Clinton statement:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*Bgi83taW4qnw7mdWCcegag.png" /><figcaption><em>Adapted from: </em><a href="https://www.ted.com/talks/pamela_meyer_how_to_spot_a_liar?language=en#t-447811"><em>https://www.ted.com/talks/pamela_meyer_how_to_spot_a_liar?language=en#t-447811</em></a></figcaption></figure><p>She then points to two indicators of deception: “non-contracted denial” (i.e., “did not” versus “didn’t”) and distancing language (i.e., “that woman”). Furthermore, she notes that “qualifying language” (i.e., “In all candor”, “To tell you the truth”, etc.), repetition of the question in its entirety, and too much detail are also telltale signs of deception.</p><p>I first saw this video in 2013 and it inspired my master’s thesis work: identifying the neural correlates of deception for gesture-based and speech-based lies. Here’s a sample of that work: <a href="https://www.jcss.gr.jp/meetings/JCSS2014/proceedings/pdf/">https://www.jcss.gr.jp/meetings/JCSS2014/proceedings/pdf/</a> JCSS2014_P2–37.pdf.</p><p>Flash forward to 2019 and deception still fascinates me. Now I’m wondering if I can find any other signs of deception using natural language processing. Better yet, can I make a machine learning (predictive analytics) model that can detect deception in speech? This led to three questions:</p><ol><li>Can I extract a dataset of true and false statements to analyze verbal deception from the web?</li><li>Are certain patterns more common in either statement type (for example, less verbs, longer sentences, etc.)?</li><li>How well can machine learning differentiate truth from lie?</li></ol><p>Answers</p><p>Through “scraping” (data extraction from the web) via the programming language Python, I extracted and refined a dataset with over 1,000 data points from an original dataset of over 16,000 data points. I also found several statistically significant patterns: Character count and word count are statistically higher for truthful statements (p &lt; .042), while average word length is statistically higher for falsehoods (p = .003). I also found adjectives and nouns are more common in truth (p &lt; .006), whereas verbs are more common in deception (p = .040). Finally, machine learning models were able to detect deception with accuracy rates of 57–60% (7–10% better than the baseline chance of 50%).</p><p>But what does it all mean?</p><p>Why do liars use more verbs, while truth-tellers use more adjectives and nouns? It may be the case that liars want your attention to be focused on the purported actions that did not actually occur, while truth-tellers, because they’re recalling true memories, are prone to more detail in the form of descriptors and recounts of people, places and things. As for the machine learning models, they were able to detect deception 7–10% better than chance so they can still use some work.</p><p>What next?</p><p>There is still room for improvement such as creating new independent variables from the dataset. For example, we might examine “non-contracted denials” and “qualifying language”. We could also fine tune the classifiers or test other models. A more exciting area of research might be combining verbal deception detection with behavioral deception detection using machine learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e62b090ece64" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>