<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Data Folks Indonesia - Medium]]></title>
        <description><![CDATA[Artikel tentang seluruh yang berkaitan dengan data mulai dari machine learning, data analisis, data engineering, data science, business intelligence - Medium]]></description>
        <link>https://medium.com/data-folks-indonesia?source=rss----e743ed8d2403---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Data Folks Indonesia - Medium</title>
            <link>https://medium.com/data-folks-indonesia?source=rss----e743ed8d2403---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 16:29:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/data-folks-indonesia" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Regex and String Operations in PostgreSQL]]></title>
            <link>https://medium.com/data-folks-indonesia/regex-and-string-operations-in-postgresql-c89102a0372a?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/c89102a0372a</guid>
            <category><![CDATA[leetcode]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 04:24:08 GMT</pubDate>
            <atom:updated>2026-09-20T04:24:08.112Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*Pl95eIkKxwZctknJV3qT4A.png" /></figure><p>I found an interesting SQL problem on LeetCode, “First Letter Capitalization II” <a href="https://leetcode.com/problems/first-letter-capitalization-ii/">https://leetcode.com/problems/first-letter-capitalization-ii/</a>. The task is to transform a messy sentence into a consistent capitalization format, following three rules that depend on how each word starts and whether it contains hyphens. String manipulation is an overlooked part of SQL because it rarely comes up in practice, so my first instinct was to solve it in Python. This post shows how I solved it entirely in SQL, including the failed attempts that I took along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*5BiPka7PwF3m_4Wa9Yn-ow.png" /></figure><p>The table is very simple, content_id and content_text. Your job is to transform every word by applying these <strong>3 rules:</strong></p><ol><li>If the word starts that is not an alphabet, leave the entire word as is.</li><li>If the word consist of 2 or more non-empty parts of alphabet connected by hyphens -, convert every first char to uppercase and the remaining part to lower case</li><li>Otherwise, convert the first letter of the word to uppercase and the remaining alphabet letters to lower case.</li></ol><p>Given these three rules, this would be easier in Python, because I can think of familiar tools such as split() and re.sub(). Doing it in PostgreSQL is more challenging. Here’s my approach:</p><ol><li>Split each sentence into words, with one word per row.</li><li>Use CASE WHEN to decide which rule (A, B, or C) each word follows.</li><li>Put the transformed words back together into a sentence.</li></ol><h4>1st Step</h4><p>How do you split a sentence into rows? It turns out that PostgreSQL can do it with unnest(), joined to the table with LATERAL so that each sentence is paired with its own words. I also needed each word&#39;s position, so I could put the words back in the right order later. WITH ORDINALITY provides that position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*9QlAw4AT6C68DVQ_GClBRw.png" /></figure><pre>SELECT content_id, content_text, w, pos<br>FROM user_content, LATERAL unnest(string_to_array(content_text, &#39; &#39;)) WITH ORDINALITY AS t(w, pos)</pre><h4>2nd Step</h4><p>Now, we need to make condition statement to apply each rule.</p><p>First things first, CASE WHEN works like if statement, so a branch runs only when its condition is true. My first guess was that CASE WHEN w != &#39;^[a-zA-Z]&#39; would pick out the words that start with a non-letter, but it did not. The != operator compares two strings for exact equality, so it only asks whether the word is literally the text ^[a-zA-Z]. No word is, so the condition was true for every word. It runs out that PostgreSQL has separate operators for regular expressions that is ~. Using ~ is true when the pattern is found anywhere in the string, and !~ is true when it is not.</p><p>So, for the first rule you can set</p><pre>WHEN w !~ &#39;^[a-zA-Z]&#39; THEN w</pre><p>The second rule, the first letter of every hyphen-separated part of the word needs to be uppercase. The regex describes the whole word piece by piece. ^ marks the start, [a-zA-Z]+ is the first part (one or more letters), (-[a-zA-Z]+)+ is one or more groups of a hyphen followed by more letters, and $ marks the end.</p><p>To do the capitalizing, I used initcap() . It converts the first letter of each word in a string to uppercase and all remaining letters to lowercase. PostgreSQL defines a &quot;word&quot; as a sequence of alphanumeric chars, and any non-alphanumeric chars (a space, punctuations, a hyphen, or a symbol) separates words. So initcap() treats each hyphen-separated part as a word and capitalizes the letter after every hyphen. This is safe here because a 2nd rule word contains only letters and hyphens. The statement looks like this:</p><pre>WHEN w ~ &#39;^[a-zA-Z]+(-[a-zA-Z]+)+$&#39; THEN initcap(w)</pre><p>Last rule, is the fallback. The first letter becomes uppercase and the rest becomes lowercase. In this case, we can’t use initcap(), because it treats every non-alphanumeric char as a word boundary, and the test cases contain symbols such as @ in the middle of a word. For example, asd@wd would become Asd@Wd, not Asd@wd.</p><p>Instead, I built the result from smaller functions, left(w,1 ) takes the first char, upper() capitalizes it, substr(w, 2) takes everything from the second char onward, lower() lowercases it, and || joins the two parts. 1st and 2nd rule are checked first, so this goes in the ELSE condition</p><pre>ELSE upper(left(w, 1)) || lower(substr(w, 2))</pre><p>DONE!</p><h4>3rd Step</h4><p>The last step is to put the transformed words back together into sentences. string_agg() is an aggregate function that concatenates values from many rows into a single string, using the delimiter gives as its second arg. The word must come back in their original order, so I added ORDER BY pos inside the call, using the position from WITH ORDINALITY .</p><p>Then I put the CASE WHEN expression that applies the three rules inside string_agg() named the result converted_text, and group by content_id and content_text, the two columns I selected without aggregating them.</p><p>Here’s the full query</p><pre>SELECT<br>    content_id,<br>    content_text AS original_text,<br>    string_agg(CASE <br>        WHEN w !~ &#39;^[a-zA-Z]&#39; THEN w<br>        WHEN w ~ &#39;^[a-zA-Z]+(-[a-zA-Z]+)+$&#39; THEN initcap(w)<br>        ELSE upper(left(w, 1)) || lower(substr(w, 2))<br>    END, &#39; &#39; ORDER BY pos) as converted_text <br>FROM user_content, LATERAL unnest(string_to_array(content_text, &#39; &#39;)) WITH ORDINALITY AS t(w, pos)<br>GROUP BY content_id, content_text</pre><h3>Conclusion</h3><p>The problem showed me that a complex string transformation can be done entirely in PostgreSQL, without exporting a CSV file and processing it in Python. The pieces were unnest() with WITH ORDINALITY to split each sentence into ordered words, CASE WHEN with the ~ and !~ regex operators to apply the three rules, initcap() or upper(), lower(), left() and substr() to change the case and string_agg() to rebuild each sentence.</p><p>Lesson learned. First, = compares exact strings, while ~ searches for a pattern, so a regex needs ^ and $ when it must match the whole word. Second, passing the tests does not prove a query is correct, so edge cases such as it&#39;s-a-b are worth testing yourself.</p><p>Python is still my first choice for a quick cleanup, but when the data already lives in a DB, SQL can do the job without leaving it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c89102a0372a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/regex-and-string-operations-in-postgresql-c89102a0372a">Regex and String Operations in PostgreSQL</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Pixels to Pasture: Predicting Biomass with Frozen Vision Transformers]]></title>
            <link>https://medium.com/data-folks-indonesia/from-pixels-to-pasture-predicting-biomass-with-frozen-vision-transformers-7a613066052c?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/7a613066052c</guid>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:46:44 GMT</pubDate>
            <atom:updated>2026-08-30T02:51:04.024Z</atom:updated>
            <content:encoded><![CDATA[<p>How I aproached Kaggle competitions: <a href="https://www.kaggle.com/competitions/csiro-biomass/">CSIRO Image2Biomas 2025</a> to build an experimentation system.</p><p>The code is available on <a href="https://github.com/andreaschandra/kaggle-csiro-image2biomass#project-structure">Github</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rrhBUw_rIfbF1ocJyqXYYg.png" /></figure><h3>A Brief of The Competition</h3><blockquote>Build models that predict pasture biomass from images, ground-truth measurements, and publicly available datasets. Farmers will use these models to determine when and how to graze their livestock.</blockquote><p>The problem is multi label regression with 5 labels, the competition used weighted scoring.</p><p>The competition constrains:</p><ul><li>CPU / GPU Notebook to run ≤ 9 hours,</li><li>Internet disabled</li><li>Freely and publicly available data</li></ul><p>Timeline</p><ul><li>Start: Oct 28, 2025</li><li>Final Submission: Jan 28</li></ul><p><em>I joined in the middle of the compeition around mid December.</em></p><h3>Tech Stack</h3><p>I used Pytorch environment to train model and common computer vision library such as opencv, Pillow, albumentations and other popular libraries like transformers, and torchvision.</p><p>For code style, I simply used ruff for linter, formatter. Although, I previously used pylint, black, and isort. I found ruff make everything in a single library so that I don’t have to maintain the version in 3 libraries.</p><p>For training model, I have GTX 1660 Super 6GB. I know this is not enough to train transformers model. So, I decided to use vast.ai and rented more powerful GPU like RTX3080 up to RTX5090.</p><p>I used W&amp;B to track my experiments like training loss, validation loss, as well as R2 score which the competition required to calculate for leaderboard. I also put notes on what change on each experiments on W&amp;B, so that I don’t get confused. I also add commit hash to W&amp;B, so that I can roll back if I want to reproduce my experiment.</p><p>Huggingface platform: I used the public models like DINOv2 or Convnextv2 from huggingface model. I once tried to store my trained weight to huggingface, turned out this was not really helpful. Remember that kaggle competition notebook was not connected to the internet. So, I stored the weight to Kaggle models and then imported.</p><h3>Training flow</h3><p>Here’s the thing, I want to make automation as much as possible, so that everything is trackable, properlly logged, and scalable. The thing that was prevent me for doing that is GPU server. Since this is personal competition, I don’t have much credits to spawn kubernetes or running GPU server for all the time.</p><p>Hence, the flow was a bit painful. First, you develop code in your local like feature extraction, model architecture, evaluation, like everything that you write the code is in your local. Then push the changes to github, github will run code check. Now, since I used Vast.ai to train model. I need to ssh to GPU server and then run git pull in the workspace directory, after the code is updated, then make run to train the model.</p><p>Sometimes the unit test that you create is not perfect, sometimes bug just happen to anywhere, so you need to monitor for the first 20 minutes if the training process run correctly. I did a tiny scale of training in my local like using batch size 1, epoch 1, break the batch training to make sure everything is correct. But still, I needed to watch the training until it literally train the model. W&amp;B helps if the training was breaking via Email. so that I could look into it immediately.</p><p>GPU server is not cheap, I only allocated like around $50 for this competition, so I needed to switch ON/OFF. Fortunately, dataset download, feature extraction is automated, so I didn’t need to download that manually. I was thinking to store extracted features to Google Cloud Storage, it was indeed speed up the process, but you get billed from Google Cloud. So, I just created persistent disk in Vast.ai or I ran extraction process in my local and then uploaded to GPU server.</p><h3>Makefile</h3><p>makefile make everything easy, if you have a routine like a set of commands in terminal, I recommend you to create that.</p><p>Here’s mine</p><pre>install:<br> curl -LsSf https://astral.sh/uv/install.sh | sh<br> uv sync --all-extras<br><br>test:<br> pytest --tb=no --disable-warnings<br><br>run:<br> python src/csiro_biomass/main.py --config configs/train.yaml<br><br>format:<br> uv run ruff check --fix src/<br><br>check:<br> uv run ruff check src/<br><br>clean:<br> rm -rf __pycache__<br> rm -rf src/__pycache__<br> rm -rf src/csiro_biomass/__pycache__<br> rm -rf tests/__pycache__<br> rm -rf .pytest_cache<br> rm -rf .ruff_cache<br> rm -rf .mypy_cache<br> rm -rf models/*<br> rm -rf wandb/*</pre><h3>Project Management</h3><p>I use UV from astral.sh to manage dependencies</p><pre>[project]<br>name = &quot;csiro-biomass&quot;<br>version = &quot;0.1.0&quot;<br>description = &quot;CSIRO Image2Biomass Kaggle Competition&quot;<br>readme = &quot;README.md&quot;<br>requires-python = &quot;~=3.12.0&quot;<br>dependencies = [<br>    &quot;numpy&gt;=1.24.0&quot;,<br>    &quot;pandas&gt;=2.0.0&quot;,<br>    &quot;scikit-learn&gt;=1.3.0&quot;,<br>    &quot;matplotlib&gt;=3.7.0&quot;,<br>    &quot;seaborn&gt;=0.12.0&quot;,<br>    &quot;jupyter&gt;=1.0.0&quot;,<br>    &quot;ipykernel&gt;=6.25.0&quot;,<br>    &quot;tqdm&gt;=4.65.0&quot;,<br>    &quot;kagglehub&gt;=0.2.0&quot;,<br>    &quot;huggingface-hub&gt;=0.19.0&quot;,<br>    &quot;wandb&gt;=0.16.0&quot;,<br>    &quot;PyYAML&gt;=6.0.0&quot;,<br>    &quot;timm&gt;=1.0.22&quot;,<br>    &quot;gdown&gt;=5.2.1&quot;,<br>]<br><br>[project.optional-dependencies]<br>dev = [<br>    &quot;pytest&gt;=7.4.0&quot;,<br>    &quot;ruff&gt;=0.1.0&quot;,<br>    &quot;black&gt;=23.0.0&quot;,<br>    &quot;isort&gt;=7.0.0&quot;,<br>]<br>ml = [<br>    &quot;torch&gt;=2.0.0&quot;,<br>    &quot;torchvision&gt;=0.15.0&quot;,<br>    &quot;transformers&gt;=4.30.0&quot;,<br>]<br>cv = [<br>    &quot;opencv-python&gt;=4.8.0&quot;,<br>    &quot;pillow&gt;=10.0.0&quot;,<br>    &quot;albumentations&gt;=1.3.0&quot;,<br>]<br><br>[build-system]<br>requires = [&quot;hatchling&quot;]<br>build-backend = &quot;hatchling.build&quot;<br><br>[tool.uv]<br>dev-dependencies = [<br>    &quot;dotenv&gt;=0.9.9&quot;,<br>    &quot;pytest&gt;=7.4.0&quot;,<br>    &quot;pytest-cov&gt;=4.1.0&quot;,<br>    &quot;ruff&gt;=0.1.0&quot;,<br>]<br><br>[tool.ruff]<br>line-length = 100<br>target-version = &quot;py312&quot;<br><br>[tool.ruff.lint]<br>select = [&quot;E&quot;, &quot;F&quot;, &quot;I&quot;, &quot;N&quot;, &quot;W&quot;, &quot;UP&quot;]<br>ignore = [&quot;N812&quot;]</pre><h3>Notebooks</h3><p>I still used Jupyterlab for exploration and proof of concept before integrating it to my system. So that I don’t need to run the entire thing to check if the function works correctly. But, I didn’t do any experiment in the notebook.</p><h3>Training configurations</h3><p>This is the most important parts, since I built the experiment program, everything should be configurable like epoch, batch_size, kfolds, number of workers, etc in yaml file. so you don’t need to hardcoded in the program.</p><p>The configuration file, yaml file will be parsed to dataclass python.</p><h4><strong>General</strong></h4><p>This means you have general section where you can set seed, model directory, img_directory, etc</p><pre>@dataclass<br>class General:<br>    &quot;&quot;&quot;General configuration&quot;&quot;&quot;<br><br>    seed: int = 42<br>    competition: str = &quot;csiro_biomass&quot;<br>    model_dir: str = &quot;models/&quot;<br>    img_dir: str = None<br>    device: str = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>    kfolds: int = 1</pre><p>The rest of the section is something like this</p><pre>@dataclass<br>class FeatureExtractor:<br>    &quot;&quot;&quot;Feature extractor configuration.&quot;&quot;&quot;<br><br>    is_enabled: bool = False<br>    model: str = None<br>    pretrained_name: str = None<br><br><br>@dataclass<br>class Dataset:<br>    &quot;&quot;&quot;Dataset configuration.&quot;&quot;&quot;<br><br>    train: str = None<br>    test: str = None<br>    kfolds: int = 1<br>    use_train_augmentation: bool = True<br>    augmented: str = None<br>    aug_dir: str = None<br>    target_transform: bool = False<br><br><br>@dataclass<br>class Regressor:<br>    model: str<br><br><br>@dataclass<br>class Trainer:<br>    &quot;&quot;&quot;Trainer configuration.&quot;&quot;&quot;<br><br>    epoch: int = 100<br>    batch_size: int = 32<br>    shuffle: bool = True<br>    num_workers: int = 0<br><br><br>@dataclass<br>class Optimizer:<br>    &quot;&quot;&quot;Optimizer configuration.&quot;&quot;&quot;<br><br>    type: float = None<br>    params: dict = None<br><br><br>@dataclass<br>class Scheduler:<br>    &quot;&quot;&quot;Scheduler configuration.&quot;&quot;&quot;<br><br>    type: float = None<br>    params: dict = None<br><br><br>@dataclass<br>class Loss:<br>    &quot;&quot;&quot;Loss configuration.&quot;&quot;&quot;<br><br>    type: float<br>    params: dict<br><br><br>@dataclass<br>class Hugginface:<br>    &quot;&quot;&quot;Huggingface configuration.&quot;&quot;&quot;<br><br>    repo_id: str = None<br><br><br>@dataclass<br>class Wandb:<br>    &quot;&quot;&quot;Weights and Biases configuration.&quot;&quot;&quot;<br><br>    project: str = None<br>    entity: str = None<br><br><br>@dataclass<br>class Kaggle:<br>    &quot;&quot;&quot;Kaggle configuration.&quot;&quot;&quot;<br><br>    model_name: str = None<br><br><br>@dataclass<br>class Config:<br>    &quot;&quot;&quot;Main configuration.&quot;&quot;&quot;<br><br>    general: General<br>    feature_extractor: FeatureExtractor<br>    dataset: Dataset<br>    regressor: Regressor<br>    trainer: Trainer<br>    optimizer: Optimizer<br>    scheduler: Scheduler<br>    loss: Loss<br>    huggingface: Hugginface<br>    wandb: Wandb<br><br>    def to_dict(self) -&gt; dict:<br>        &quot;&quot;&quot;Convert config to dictionary recursively.&quot;&quot;&quot;<br>        return asdict(self)<br><br>    @classmethod<br>    def load_from_file(cls, path):<br>        &quot;&quot;&quot;Load from a YAML file.&quot;&quot;&quot;<br><br>        config_dict = read_yaml(path)<br>        return cls(<br>            general=General(**config_dict.get(&quot;general&quot;, {})),<br>            feature_extractor=FeatureExtractor(**config_dict.get(&quot;feature_extractor&quot;, {})),<br>            dataset=Dataset(**config_dict.get(&quot;dataset&quot;, {})),<br>            regressor=Regressor(**config_dict.get(&quot;regressor&quot;, {})),<br>            trainer=Trainer(**config_dict.get(&quot;trainer&quot;, {})),<br>            optimizer=Optimizer(**config_dict.get(&quot;optimizer&quot;, {})),<br>            scheduler=Scheduler(**config_dict.get(&quot;scheduler&quot;, {})),<br>            loss=Loss(**config_dict.get(&quot;loss&quot;, {})),<br>            huggingface=Hugginface(**config_dict.get(&quot;huggingface&quot;, {})),<br>            wandb=Wandb(**config_dict.get(&quot;wandb&quot;, {})),<br>        )</pre><h3>Finally</h3><p>Did I win the competition? absolutely not. Does this enhance my experience to build an experiment system, absolutely. This is my first attempt after a long time to build a comprehensive system from data to model to report. The code that I wrote in this repo is all by hand (turned out I already used Claude Code at that time but not heavy). that’s why it is a bit messy yet still structured.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a613066052c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/from-pixels-to-pasture-predicting-biomass-with-frozen-vision-transformers-7a613066052c">From Pixels to Pasture: Predicting Biomass with Frozen Vision Transformers</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Stop Being the Query Monkey: A Reflection & Survival Framework]]></title>
            <link>https://medium.com/data-folks-indonesia/how-to-stop-being-the-query-monkey-a-reflection-survival-framework-9f55d135b7da?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/9f55d135b7da</guid>
            <category><![CDATA[organizational-culture]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 09:35:07 GMT</pubDate>
            <atom:updated>2026-07-04T09:35:05.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dO1cQO4bEcKHtvhCu9ZW3g.jpeg" /></figure><p>For a stretch of my career, my job title said “Data Scientist”, but my actual job was like build dashboard, patch ETL pipelines, and aswer “can you pull the numbers for X” in Slack. While every model I wanted to ship got deprioritize for a report someone needed by end of day. Looking back, I wasn’t doing data science, I was a human query interface with a fancier title. *giggling*</p><p>If you work in data or BI in a tech company, you already know this pattern. Fintech like digital banking, stock securities, ewallet, is dense with reporting obligations, regulatory, risk, and audit, and that density creates gravity. It pulls every analytical role toward “run the query, format the table, send it over”. Do that long enough and it becomes your entire job description, whether or not it’s on your resume.</p><p>This isn’t a rant. It’s the framework I wish someone had handed me, the concrete moves that shift you from ticket-taker to someone whose name comes up when decision get made.</p><p>Why early and mid tech startup breeds query monkeys esp in fintech, 3 structural reasons:</p><ol><li>Regulatory reporting is relentless, in Indonesia context there are OJK, BI, IDX, and internal audit. These generate a constant stream of “just pull this number” request that feel urgent because compliance deadlines are real.</li><li>Risk-aversion rewards literalism. In most situation, deviating from what was asked feels riskier than just answering it. Concretely in banking, if someone asks for “NPL ratio by branch” and you suspect they actually need something broader, pushing back or reframing the request feels risky. You might be wrong, you might annoy a stakeholder, you might miss a deadline arguing about scope. More dangerously, you don’t have domain expertise either in digital product or the specific problem itself. So, analysts learn to stop questioning requests and start executing them.</li><li>Legacy systems fragment ownership. Core banking systems, data warehouses, and reporting tools are often stitched together by different teams over different decades. Nobody owns the full picture, so analyst become the connective tissue, manually, repeatedly, forever.</li></ol><p>Understanding why it happens matters, because the fix isn’t “work harder”, it’s “change your position in the workflow”.</p><blockquote>Usually, this happens a lot in the early stage of building the team. The worst part, instead of developing the data team from, the Head just went straight to build data science team as if it can do everything.</blockquote><h3><strong>The Framework</strong></h3><ol><li>Interrogate the request before you touch a query</li><li>Kill every recurring request withtin two cycles</li><li>Get upstream of the ticket</li><li>Build domain depth, not just tool depth</li><li>Automate your way out of the grunt work</li><li>Present findings, don’t just deliver tables</li><li>Track impact, not throughput</li></ol><h4>Interrrogate the Request Before You Touch a Query</h4><p>Every “can you get me X” has an unstated decision behind it. Before running anything, ask:</p><ul><li>What decision does this number inform</li><li>What will you do differently based on the answer?</li><li>Is X actually the right metric, or the one that’s easiest to ask for?</li></ul><p>In banking, this catches real problems. Someone ask for “current NPL ratio by branch”, the actual need if often “Which branches need intervention this quarter”, which might NPL trend + provisioning coverage + early warning indicators, not a static ratio. Asking the question upgrades you from typist to advisor in one move.</p><h4>Kill Every Recurring Request Within Two Cycles</h4><p>The single highest-leverage habit: if you get asked for the same (or similar) pull twice, that’s your signal to automate it. Third time it’s requested manually, that’s a process failure on your end, not theirs.</p><ul><li>First ask → answer it, note the pattern</li><li>Second ask → build a schedule query or lightweight dashboard (Looker, Power BI, Metabase, etc)</li><li>Third ask → point the to self-service, offer to walk the through it once</li></ul><p>This is the difference between doing work and eliminating work. Query monkeys accumulate recurring asks. Analyst who escape the trap systematically delete them.</p><h4>Get Upstream of The Ticket</h4><p>If requests only reach you as fully-formed tickets, you are structurally downstream. Someone else already decided what matters, and you are just executing. Fix your position, not just your output.</p><ul><li>Ask to sit in on planning or strategy meetings for the teams you support most (risk, collection, fraud, marketing, etc)</li><li>When you deliver a report, add one line of “here’s what I’d look at next”. This signals you think past the ask. Make the impression that you aren’t just query executor, you are already thinking about the decision behind the request</li><li>Volunteer for the messy, ill-defined problem nobody whats to scope, instead of only taking clean tickets.</li></ul><p>You can’t get pulled into decisions if decisions are always made before you are in the room</p><h4>Build domain depth, not just tool depth</h4><p>SQL and dashboarding tools are commodity skills in banking now. Every analyst has them. What’s scarce is someone who understands why a number matters. How a fraud model’s false-positive rate trade off against customer friction, how treasury liquidity ratios interact with regulatory buffers.</p><p>Pick on domain adjacent to your work e.g. credit risk, AML, collections, and then go deep enough that you can explain the business logic to a non-technical stakeholder without notes. That depth is what gets you invited to the table before the data is even pulled.</p><h4>Automate your way out of the grunt work</h4><p>Every hour spent manually re-running a query is an hour not spent interpretting one. This usually means:</p><ul><li>Scheduled ETL jobs instead of manual pulls (even a basic cron + script beats “I’ll run it Monday)</li><li>Anomaly/threshold alerts instead of manually eyeballing dashboard for issues</li><li>A lightweight semantic layer or metrics dictionary so “revenue” and “active customer” mean the same thing everywhere, killing the endless “which number is right” back-and-forth</li></ul><p>None of this requires fancy infra. A scheduled Python script writing to a shared table is often enough to retire a recurring manual task permanently.</p><h4>Present Findings, Don’t Just Deliver Tables</h4><p>The query monkey hands off a spreadsheet and lets someone else interpret it. The analyst who breaks out of that role hands off a narrative: what changed, what it means, what to do about it. Even a three-line summary above the table. “NPL ticked up 40bps in retail this month, concentrated in unsecured lending post-rate-hike; recommend tightening approval thresholds for that segment”. Changes how people perceive your role. You’re not the person who ran the query. You are the person who told them what it means.</p><h4>Track Impact, Not Throughput</h4><p>If you performance review material is “delivered 40 dashboards this quarter”, you are measuring yourself the way a query monkey gets measured. Track things closer to business outcomes instead:</p><ul><li>Risk flagged before it became a loss</li><li>Revenue or cost impact of a recommendation you made</li><li>Manual processes you eliminated (hours saved, requests killed)</li></ul><p>This reframing does double duty, it makes your case for pormotion concrete, and it forces you to actually spend time on work that produces this kind of evidence, which naturally pulls you away forom pure execution.</p><h3>Reality Check</h3><p>None of this happens by automatically or a single point of alignment. Some of it requres the managers to actually want you doing higher-leverage work, and some companies genuinely don’t have room for analyst to operate this way. The reporting load is simply too heavy, or the org itself doesn’t value it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f55d135b7da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/how-to-stop-being-the-query-monkey-a-reflection-survival-framework-9f55d135b7da">How to Stop Being the Query Monkey: A Reflection &amp; Survival Framework</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code Judo in Code Review]]></title>
            <link>https://medium.com/data-folks-indonesia/code-judo-in-code-review-def2c1fdcef6?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/def2c1fdcef6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding-best-practices]]></category>
            <category><![CDATA[code-review]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 10:22:36 GMT</pubDate>
            <atom:updated>2026-06-28T10:22:36.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KS-MyQ8EKf5oFhUu" /><figcaption>Photo by <a href="https://unsplash.com/@joshuajamias?utm_source=medium&amp;utm_medium=referral">Joshua Jamias</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Code Judo is a metaphor borrowed from the martial art — in Judo, you use your opponent’s weight and momentum against them, minimal effort for maximum effect.</p><p>Basically, you are doing “code judo” in code review when you try to find an alternative of implemented solution to make a change dramatically simpler, rather than adding new complexity to handle it.</p><p>Instead of fighting the codebase by bolting on more logic, you redirect it so the problem nearly solves itself.</p><blockquote>I found this term when reading a code agent skill from cursor plugins where the document specifically stated “code judo”</blockquote><p>Here’s the concrete examples of code judo</p><ul><li>Instead of adding a 5-branch conditional to handle 5 edge cases, change the data model so all 5 cases become the same case</li></ul><pre># 1. Replace multi-branch conditional with a dispatch table<br># ❌ Before<br>def process(action, data):<br>    if action == &quot;create&quot;: return create(data)<br>    elif action == &quot;update&quot;: return update(data)<br>    elif action == &quot;delete&quot;: return delete(data)<br>    elif action == &quot;archive&quot;: return archive(data)<br><br># ✅ After — adding new actions costs zero branches<br>HANDLERS = {&quot;create&quot;: create, &quot;update&quot;: update, &quot;delete&quot;: delete, &quot;archive&quot;: archive}<br>def process(action, data):<br>    return HANDLERS[action](data)</pre><ul><li>Instead of handling logic null value, add default value</li></ul><pre># 2. Replace scattered None-checks with a Null Object<br># ❌ Before — None guards spread across every callsite<br>def render(user):<br>    name = user.name if user else &quot;Guest&quot;<br>    avatar = user.avatar if user else DEFAULT_AVATAR<br>    role = user.role if user else &quot;viewer&quot;<br><br># ✅ After — callsites become unconditional<br>@dataclass<br>class AnonymousUser:<br>    name = &quot;Guest&quot;<br>    avatar = DEFAULT_AVATAR<br>    role = &quot;viewer&quot;<br><br>def render(user):<br>    name, avatar, role = user.name, user.avatar, user.role</pre><ul><li>Instead of a boolean flag that forks the function internally, use a format string that looks up the right function. This is way scalable and maintainable when you want to add more format. So, you don’t add more conditional statements inside the function</li></ul><pre># 3. Replace flag arguments with polymorphism<br># ❌ Before — boolean flag that forks the entire function body<br>def export(data, as_csv=False):<br>    if as_csv:<br>        return &quot;,&quot;.join(str(x) for x in data)<br>    else:<br>        return json.dumps(data)<br><br># ✅ After — open for extension, no internal branching<br>EXPORTERS = {<br>    &quot;csv&quot;: lambda data: &quot;,&quot;.join(str(x) for x in data),<br>    &quot;json&quot;: lambda data: json.dumps(data),<br>}<br>def export(data, fmt=&quot;json&quot;):<br>    return EXPORTERS[fmt](data)</pre><ul><li>Instead of sequential async calls one by one, use asyncio.gather to run them all at once</li></ul><pre># 4. Replace sequential awaits with parallel execution<br># ❌ Before — independent I/O serialized for no reason<br>async def get_dashboard(user_id):<br>    profile = await fetch_profile(user_id)<br>    orders  = await fetch_orders(user_id)<br>    notifs  = await fetch_notifications(user_id)<br>    return profile, orders, notifs<br><br># ✅ After — 3x faster, same logic<br>async def get_dashboard(user_id):<br>    return await asyncio.gather(<br>        fetch_profile(user_id),<br>        fetch_orders(user_id),<br>        fetch_notifications(user_id),<br>    )</pre><ul><li>Instead of repeated validation, use existing library to automatically checks</li></ul><pre># 5. Replace repeated validation conditionals with a declarative schema<br># ❌ Before — every field validated ad-hoc, grows with every new field<br>def validate(data):<br>    errors = []<br>    if not data.get(&quot;email&quot;): errors.append(&quot;email required&quot;)<br>    if not data.get(&quot;name&quot;): errors.append(&quot;name required&quot;)<br>    if len(data.get(&quot;password&quot;, &quot;&quot;)) &lt; 8: errors.append(&quot;password too short&quot;)<br>    return errors<br><br># ✅ After — adding a field = one line in the schema<br>from pydantic import BaseModel, constr<br><br>class UserInput(BaseModel):<br>    email: str<br>    name: str<br>    password: constr(min_length=8)<br># validation happens on instantiation; errors are automatic</pre><p>Sometimes, I is a bit challenging to review code manually (you may need experience and expertise to do so) because you constantly challenging the implementation of the function. However, I find it fun because there are multiple ways to do the same thing and considering the pros and cons, it will be more fun when you are doing pair programming, even if it will take longer time. But, hey it may have better code quality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=def2c1fdcef6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/code-judo-in-code-review-def2c1fdcef6">Code Judo in Code Review</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robot Avoiding Car using Arduino: First Attempt]]></title>
            <link>https://medium.com/data-folks-indonesia/robot-avoiding-car-using-arduino-first-attempt-434b51a09c3c?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/434b51a09c3c</guid>
            <category><![CDATA[arduino]]></category>
            <category><![CDATA[robotics]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Sat, 28 Jun 2025 00:33:17 GMT</pubDate>
            <atom:updated>2025-06-27T23:01:24.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O72TGhA9eiFKal6GZU6qiQ.jpeg" /></figure><p>Creating a robot that can detect and avoid obstacles is a classic Arduino project that teaches fundamental concepts in robotics. This car-avoiding robot uses sensors to detect objects in its path and automatically changes direction to navigate around them.</p><p>This simple project combines basic electronics, programming, and mechanical assembly to create an autonomous vehicle. This is the most recommended one for beginners learning Arduino due to concept coverage like sensor interface, motor control, and simple decision making based on conditional statements.</p><p>Here are the components that I use for building car-avoiding robot</p><h4><strong>Arduino Uno R3</strong></h4><ul><li>Main microcontroller board</li><li>Process sensor data and control motors</li><li>Provide digital and analog pins for connections</li></ul><h4>Sensor Shield V5</h4><ul><li>An expansion board that is placed on top of the Arduino board</li><li>Provides easy connections for sensors and motors</li><li>Protects broken pin due to human error (pushing too hard, bending, etc.)</li><li>pSimplifies wiring with labeled pins</li><li>Provide 3 main features for each pin: Voltage, Signal, Ground</li><li>Using this, you have 13 digital I/O pins</li></ul><h4>Motor Driver L298N</h4><ul><li>Controls DC motors with higher current requirements</li><li>Allows bidirectional motor control (forward / backward)</li><li>Acts as interface between Arduino and motors</li></ul><h4>Servo Motor SG90</h4><ul><li>Small, precise motor for rotating ultrasonic sensor</li><li>Enables 180-degree scanning motion</li></ul><h4>Ultrasonic Sensor HC-SR04</h4><ul><li>Measures distance to objects using sound waves</li><li>Has 4 pins: VCC, GND, Trigger, and Echo</li><li>Effective range 2–400cm</li><li>Provides accurate distance measurements for obstacle detection.</li><li>This kind of an eye for the robot to see the object.</li></ul><h4>DC Motors</h4><ul><li>Rotate the wheels</li><li>Connected via motor driver</li></ul><h4>4xAA Battery</h4><ul><li>To power the arduino via shield</li><li>Motor driver connect to shield</li><li>so that moth driver and arduino only using this power from battery</li></ul><h3>How it works</h3><p>The robot operates in a continuous loop:</p><p>During setup</p><ol><li>declare variable and values such as variable motor and its pin</li><li>set OUTPUT pin</li><li>Set the sensor direction to the front, then to the right, then to the left and back to the front.</li></ol><p>Loop:</p><ol><li>Detect: Sensor measures distances to nearby objects</li><li>Decide: Arduino processes distance data to find a clear path</li><li>Move: Motors turn wheels to move in the chosen direction</li><li>Repeat</li></ol><p>The ultrasonic sensor works like echolocation; it sends sound waves and measures how long they take to return after hitting objects. Shorter return times mean closer obstacles.</p><h3>Challenges</h3><p>The biggest challenge was inaccurate or messy documentation that I received from the store. I bought this in a packet for specific this project, even though there are two instructions that are very different from one another.</p><p>Common mistakes that I encountered</p><ul><li>Wrong pin</li><li>The configuration pin in the code is wrong</li><li>The wire is incorrect</li><li>Voltage (?)</li></ul><p><strong>Solution</strong>: double check here and there, read Arduino documentation and tutorials, do a bunch of trial and error.</p><p>Before building this car, I learn basic usage such as managing traffic light using Arduino. It only uses Arduino, breadboard, and light. With a simple delay timing, do the thing. Hence, I am a bit familiar with the pin and stuff.</p><p>The Arduino code can be found <a href="https://gist.github.com/andreaschandra/60bc43c34c020777d198e465a321b63b">here</a></p><h3>Conclusion</h3><p>Building this car-avoiding robot was challenging at first, but satisfying. The project taught essential Arduino skills, including sensor interfacing, motor control, and basic robotics programming.</p><p>Key takeaways for beginners like me:</p><ul><li>Start simple, build and test one component at a time. Make sure the hardware works.</li><li>Expect problems: documentation isn’t always accurate, expect low for the first trial</li><li>Use the forum; thousands of people are building this, and you are not the first.</li><li>Debug, debug, debug. The serial monitor is your best first step for finding issues</li><li>Be patient, trial and error are part of the learning process. I take two weeks off from this project, and one evening with simple adding enA and enB pins to do the magic 🌟</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=434b51a09c3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/robot-avoiding-car-using-arduino-first-attempt-434b51a09c3c">Robot Avoiding Car using Arduino: First Attempt</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chat With Your Document using LangChain, Chroma, and Claude]]></title>
            <link>https://medium.com/data-folks-indonesia/chat-with-your-document-using-langchain-chroma-and-claude-f6fd08386b3c?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6fd08386b3c</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[chroma]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[vectordb]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Fri, 27 Jun 2025 02:22:44 GMT</pubDate>
            <atom:updated>2025-06-27T03:23:31.653Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HW1ED--qcg92nPBmq5tJLQ.png" /></figure><p>Imagine having an intelligent assistant that can instantly answer questions about your documents, such as research papers, financial reports, announcements, etc. Instead of manually searching through hundreds of pages, you could simply ask “What are the key findings in the report?” or “How does the company perform this quarter?” and get precise, contextual answers immediately.</p><p>With Retrieval Augmented Generation (RAG), combining document search capabilities with conversational AI, you can create chatbots that don’t just provide generic responses, but actually understand and reference your specific content. Imagine a Google search that looks into your documents and points to the answer directly.</p><p>In this tutorial, we will build a simple document chatbot that leverages 4 components: <strong>LangChain</strong> for managing the chatbot, <strong>Chroma </strong>for document storage and retrieval, <strong>Huggingface</strong> for model embedding, and <strong>Claude</strong> for generating intelligence, context-aware responses. Unlike a simple QnA system, this chatbot will maintain conversation history, cite its sources, and provide a natural, interactive experience.</p><p>If you are looking to create an internal knowledge base for your team, build customer support automation, or simply want to make your document collections more accessible, this step-by-step guide will show you how to transform static documents into an interactive intelligence assistant.</p><p>By the end of this article, you will have a fully functional chatbot that can process PDFs and text files, remember previous conversations, and provide accurate answers with source citations.</p><p>let’s go</p><p><strong>Core Features:</strong></p><ul><li>Loads multiple document types (PDF, TXT)</li><li>Splits documents into searchable chunks</li><li>Creates vector embeddings for semantic search using HuggingFaceEmbeddings</li><li>Maintains conversation memory</li><li>Provides source citations for answers</li></ul><p><strong>Components:</strong></p><ol><li><strong>Document Processing</strong>: Loads and chunks your knowledge base</li><li><strong>Vector Storage</strong>: Creates searchable embeddings using ChromaDB</li><li><strong>Retrieval</strong>: Finds relevant document sections for each question</li><li><strong>Generation</strong>: Uses Anthropic Claude to generate contextual answers</li><li><strong>Memory</strong>: Keeps track of conversation history</li></ol><h4>How to build it</h4><p>Create pyproject.toml</p><pre>[project]<br>name = &quot;langc-explore&quot;<br>version = &quot;0.1.0&quot;<br>description = &quot;Add your description here&quot;<br>readme = &quot;README.md&quot;<br>requires-python = &quot;&gt;=3.11&quot;<br>dependencies = [<br>    &quot;langchain-community==0.3.1&quot;,<br>    &quot;chromadb&quot;,<br>    &quot;pypdf&quot;,<br>    &quot;python-dotenv&quot;,<br>    &quot;streamlit&quot;,<br>    &quot;langchain-anthropic&quot;,<br>    &quot;langchain-huggingface&quot;,<br>    &quot;sentence-transformers&quot;,<br>]</pre><p>use uv sync to install dependency</p><pre>from langchain_community.document_loaders import (<br>    PyPDFLoader,<br>    TextLoader,<br>    DirectoryLoader,<br>)<br><br>from langchain_anthropic import ChatAnthropic<br>from langchain_huggingface import HuggingFaceEmbeddings<br>from langchain_community.vectorstores import Chroma<br><br>from langchain.memory import ConversationBufferMemory<br>from langchain.chains import ConversationalRetrievalChain<br>from langchain_text_splitters.character import RecursiveCharacterTextSplitter<br>from langchain_core.messages import HumanMessage, AIMessage</pre><p>Initialize the LLM, memory, Vector DB, and Conversation chain.</p><p>We use HuggingFaceEmbeddings for transforming question to vector embedding so that we can calculate semantic similarity. The model all-mpnet-base-v2 is known for top semantic similarity, supports 100+ languages and optimized for sentence and paragraph.</p><pre>def __init__(self):<br>        self.embeddings = HuggingFaceEmbeddings(<br>            model_name=&quot;sentence-transformers/all-mpnet-base-v2&quot;<br>        )<br><br>        self.llm = ChatAnthropic(model=&quot;claude-3-5-haiku-latest&quot;)<br>        self.vector_store = None<br>        self.conversation_chain = None<br>        self.memory = ConversationBufferMemory(<br>            memory_key=&quot;chat_history&quot;, return_messages=True, output_key=&quot;answer&quot;<br>        )<br>        self.chat_history = []</pre><p>Let&#39;s load the documents that are uploaded by the user</p><pre>def load_documents(self, directory_path):<br>        &quot;&quot;&quot;Load documents from a directory&quot;&quot;&quot;<br><br>        loaders = [<br>            DirectoryLoader(directory_path, glob=&quot;**/*.pdf&quot;, loader_cls=PyPDFLoader),<br>            DirectoryLoader(directory_path, glob=&quot;**/*.txt&quot;, loader_cls=TextLoader),<br>        ]<br><br>        documents = []<br>        for loader in loaders:<br>            documents.extend(loader.load())<br><br>        return documents</pre><p>Convert long text into chunk and transform to vector embedding.</p><p>There is a good blog post for explaining how RecursiveCharacterTextSpliter Works</p><p><a href="https://dev.to/eteimz/understanding-langchains-recursivecharactertextsplitter-2846">Understanding LangChain&#39;s RecursiveCharacterTextSplitter</a></p><pre>def process_documents(self, documents):<br>        &quot;&quot;&quot;Split documents into chunks and create embeddings&quot;&quot;&quot;<br>        text_splitter = RecursiveCharacterTextSplitter(<br>            chunk_size=1000, chunk_overlap=200, length_function=len<br>        )<br>        chunks = text_splitter.split_documents(documents)<br><br>        self.vector_store = Chroma.from_documents(<br>            documents=chunks, embedding=self.embeddings, persist_directory=&quot;.chroma_db&quot;<br>        )<br><br>        return len(chunks)</pre><p>Let’s setup the conversation chain</p><pre>def setup_conversation_chain(self):<br>        &quot;&quot;&quot;Setup the conversational retrieval chain&quot;&quot;&quot;<br>        if self.vector_store is None:<br>            raise ValueError(<br>                &quot;Vector store is not initialized. Load and process documents first.&quot;<br>            )<br><br>        retriever = self.vector_store.as_retriever(<br>            search_type=&quot;similarity&quot;, search_kwargs={&quot;k&quot;: 3}<br>        )<br><br>        self.conversation_chain = ConversationalRetrievalChain.from_llm(<br>            llm=self.llm,<br>            retriever=retriever,<br>            memory=self.memory,<br>            return_source_documents=True,<br>            output_key=&quot;answer&quot;,<br>        )</pre><p>and finally, create a function for the user to pass the question</p><pre>def ask_question(self, question):<br>        if self.conversation_chain is None:<br>            raise ValueError(<br>                &quot;Conversation chain is not set up. Call setup_conversation_chain() first.&quot;<br>            )<br><br>        response = self.conversation_chain.invoke(<br>            {&quot;question&quot;: question, &quot;chat_history&quot;: self.chat_history}<br>        )<br><br>        self.chat_history.append(<br>            [HumanMessage(content=question), AIMessage(content=response[&quot;answer&quot;])]<br>        )<br><br>        print(f&quot;Chat History: {self.chat_history}\n\n&quot;)<br><br>        print(f&quot;Response: {response[&#39;source_documents&#39;]}\n\n&quot;)<br><br>        return response[&quot;answer&quot;], response[&quot;source_documents&quot;]</pre><h4>User Interface</h4><p>Streamlit is used for uploading documents and a chat interface</p><pre>import os<br>import streamlit as st<br>import shutil<br><br>from chatbot import KnowledgeBaseChatbot</pre><p>Make the main interaction</p><pre>def main():<br>    st.set_page_config(page_title=&quot;Knowledge Base Chatbot&quot;, page_icon=&quot;:robot_face:&quot;)<br>    st.title(&quot;Knowledge Base Chatbot&quot;)<br>    st.sidebar.title(&quot;📚 Document Management&quot;)<br><br>    if &quot;chatbot&quot; not in st.session_state:<br>        st.session_state.chatbot = KnowledgeBaseChatbot()<br><br>    if &quot;messages&quot; not in st.session_state:<br>        st.session_state.messages = []<br><br>    with st.sidebar:<br>        st.header(&quot;Upload documents&quot;)<br>        uploaded_files = st.file_uploader(<br>            &quot;Upload PDF or TXT files&quot;, type=[&quot;pdf&quot;, &quot;txt&quot;], accept_multiple_files=True<br>        )<br><br>        if st.button(&quot;Process Documents&quot;):<br>            if uploaded_files:<br>                temp_dir = &quot;temp_docs&quot;<br>                os.makedirs(temp_dir, exist_ok=True)<br><br>                for uploaded_file in uploaded_files:<br>                    with open(f&quot;{temp_dir}/{uploaded_file.name}&quot;, &quot;wb&quot;) as f:<br>                        f.write(uploaded_file.getbuffer())<br><br>                with st.spinner(&quot;Processing documents...&quot;):<br>                    documents = st.session_state.chatbot.load_documents(temp_dir)<br>                    chunk_count = st.session_state.chatbot.process_documents(documents)<br>                    st.session_state.chatbot.setup_conversation_chain()<br><br>                st.success(<br>                    f&quot;Processed {len(documents)} documents into {chunk_count} chunks.&quot;<br>                )<br><br>                shutil.rmtree(temp_dir)<br><br>    st.header(&quot;💬 Chat with your documents&quot;)<br><br>    for message in st.session_state.messages:<br>        with st.chat_message(message[&quot;role&quot;]):<br>            st.markdown(message[&quot;content&quot;])<br><br>    if prompt := st.chat_input(&quot;Ask a question about your documents&quot;):<br>        st.session_state.messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br>        with st.chat_message(&quot;user&quot;):<br>            st.markdown(prompt)<br><br>        if st.session_state.chatbot.conversation_chain is not None:<br>            with st.chat_message(&quot;assistant&quot;):<br>                with st.spinner(&quot;Thinking...&quot;):<br>                    try:<br>                        answer, sources = st.session_state.chatbot.ask_question(prompt)<br>                        st.markdown(answer)<br><br>                        if sources:<br>                            with st.expander(&quot;Source Documents&quot;):<br>                                for index, source in enumerate(sources, 1):<br>                                    st.write(f&quot;**Source {index}:**&quot;)<br>                                    st.write(<br>                                        source.page_content[:200] + &quot;...&quot;<br>                                    )  # Display first 200 chars<br>                                    st.write(&quot;---&quot;)<br><br>                        st.session_state.messages.append(<br>                            {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: answer}<br>                        )<br>                    except Exception as e:<br>                        st.error(f&quot;Error: {str(e)}&quot;)<br><br>        else:<br>            with st.chat_message(&quot;assistant&quot;):<br>                st.markdown(&quot;Please upload and process documents first.&quot;)</pre><h3>Conclusion</h3><p>It feels great to write a technical blog again after a while. This may be the MNIST for RAG Chatbot, for which you can find a similar tutorial elsewhere. Overall, the flow is something like this…</p><pre># The flow works like this:<br>Document → Embeddings (HuggingFace) → Vector Store (Chroma)<br>                                           ↓<br>Question → Embeddings → Similarity Search → Retrieved Context<br>                                           ↓<br>Context + Question + Memory → Claude → Answer<br>                                           ↓<br>                               Memory Storage → Next Question</pre><p>Using Streamlit for the interface makes this chatbot interactive and usable for everyone to try.</p><p>Thanks to <a href="https://medium.com/u/33e478999def">Affandy Fahrizain</a> <a href="https://medium.com/u/9e80c8c1e6b">Adhi Setiawan</a> and <a href="https://medium.com/u/c5465db64060">Alamsyah Hanza</a> for proofreading this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6fd08386b3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/chat-with-your-document-using-langchain-chroma-and-claude-f6fd08386b3c">Chat With Your Document using LangChain, Chroma, and Claude</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhance Your LLM’s Understanding: Document Parsing Tips You Need to Know]]></title>
            <link>https://medium.com/data-folks-indonesia/enhance-your-llms-understanding-document-parsing-tips-you-need-to-know-70d3dc2cd462?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/70d3dc2cd462</guid>
            <category><![CDATA[jina-ai]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[markdown]]></category>
            <dc:creator><![CDATA[Affandy Fahrizain]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 15:07:57 GMT</pubDate>
            <atom:updated>2024-11-21T15:07:57.562Z</atom:updated>
            <content:encoded><![CDATA[<h4>TLDR; In this article, we will show how to parse PDF and web pages into Markdown format — which can preserve document structure for better LLM understanding.</h4><figure><img alt="Generated by Microsoft Copilot" src="https://cdn-images-1.medium.com/max/1024/1*CuzYhajpHzdEL3orOg0kQg.png" /><figcaption>Image is generated with Microsoft Copilot</figcaption></figure><h3>Document Parsing</h3><p>In RAG, it is very common to upload our documents (i.e. PDF, spreadsheet, docs, or website) to the vector database so we can augment the LLM knowledge from it. One integral step is parsing the document itself, which the result must represent the document in the best way.</p><p>The easiest way to do that is to just read the file and get the contents. However, it is very possible to make the content lose its structure. Gratefully, we can mitigate this by parsing the document into a structured format — Markdown. To the best of my experience, most LLMs work best with the Markdown format, especially OpenAI’s family.</p><h3>PDF</h3><p>To parse PDF files into Markdown, we can leverage the extended version of PyMuPDF library, called <strong>PyMuPDF4LLM</strong>.</p><p>This library is designed specifically to produce the best output that works well with LLM. Without further ado, let’s see how we can use it to parse PDF documents.</p><p>First, we need to install the library itself.</p><pre>pip install pymupdf4llm</pre><p>We will use these sample PDFs from <a href="https://www.princexml.com/samples/">https://www.princexml.com/samples/</a>. Optionally, you may also use your own PDF file.</p><p>Fortunately, it is very easy to convert it into markdown using pymupdf4llm. In this article, we will only work with one of them. You can find the rest implementation on <a href="https://github.com/fhrzn/rags-archive">GitHub repo</a>.</p><pre>import pymupdf4llm<br><br># these files are downloaded from link above<br>filepath = {<br>    &quot;invoice1&quot;: &quot;assets/invoice.pdf&quot;,<br>    &quot;invoice2&quot;: &quot;assets/invoice2.pdf&quot;,<br>    &quot;brochure&quot;: &quot;assets/brochure.pdf&quot;,<br>    &quot;newsletter&quot;: &quot;assets/newsletter.pdf&quot;,<br>    &quot;textbook&quot;: &quot;assets/textbook.pdf&quot;,<br>}<br>newsletter = pymupdf4llm.to_markdown(filepath[&quot;newsletter&quot;], show_progress=False)<br>print(newsletter)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qQTjC8BRC70mghlFf0Y2A.png" /><figcaption>Side-by-side comparison between original PDF and PyMuPDF4LLM result</figcaption></figure><p>From the result above, we can see there is ----- token denoting different pages.</p><p>Although the parsed result is not perfect, the output’s structure is good enough as it also maintains the separation of each document part.</p><p>As a comparison, we shall look into <strong>Langchain PyPDFLoader</strong> implementation.</p><blockquote><em>💡 Make sure you already install langchain on your machine.</em></blockquote><pre>from langchain_community.document_loaders import PyPDFLoader<br><br>lc_newsletter = &quot;\n==================\n&quot;.join(doc.page_content for doc in list(PyPDFLoader(filepath[&quot;newsletter&quot;]).lazy_load()))<br>print(lc_newsletter)</pre><blockquote><em>💡 Note that langchain document loader’s implementation always return list of langchain </em><em>Document object – each page represented by an object. Therefore we join them with separator token </em><em>================== to denote different pages.</em></blockquote><p>Here is the result of Langchain PyPDFLoader:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liSSoc4jZf7yqJMK2J02pw.png" /><figcaption>Side-by-side comparison between original PDF and Langchain’s PyPDFLoader result</figcaption></figure><p>Pay attention to the output document structure that is not preserved, making it hard to identify which part originally belonged to. In addition, if we take a closer look, some captured words are strangely separated by random whitespace.</p><p>Although the LLM may still understand it, I believe giving better input representation will produce better output as well. Therefore, parsing into markdown format is a good choice to enhance LLM&#39;s understanding of our document.</p><p>Now we already know how to parse PDF into markdown format. What if I am telling you that you can do the same to webpages?</p><p>Let’s see how to do it.</p><h3>Webpage</h3><p>To parse the webpage into Markdown, we can utilize <a href="https://jina.ai/reader/">Jina AI Reader API</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NuhO0QCvbqyiE7x7" /><figcaption>Jina AI Reader API</figcaption></figure><p>Jina AI gives an API key we can use for the first 1M token processed. Once it reaches the limit, we need to top up if we intend to use the API key.</p><p>Afraid not, we can still use their service (for now) even without the API key. Simply do the request with omitting the API key. In this example, we will try to omit the API key.</p><blockquote><em>🚧 Please note the rate limiter is very tight when we are not using the API key. Make sure we are doing graceful requests, otherwise we will get </em><strong><em>Error 429 — Too many requests</em></strong></blockquote><p>To use it, we only need to perform a GET request to their endpoint.</p><p>Skip it if you have installed it already</p><pre>pip install requests</pre><p>We will try to parse one of my articles <a href="https://medium.com/data-folks-indonesia/exploring-visual-transformers-vit-with-huggingface-8cdda82920a0"><em>Exploring Vision Transformers (ViT) with 🤗 Huggingface</em></a>. You may also change it into any other websites.</p><pre>import requests<br>import os<br><br>BASE_URL = &quot;https://r.jina.ai&quot;<br>site = &quot;https://fhrzn.github.io/posts/building-conversational-ai-context-aware-chatbot/&quot;<br>url = os.path.join(BASE_URL, site)<br>resp = requests.get(url)<br>print(resp.text)</pre><p>And here is the parsed result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VM8A1bfyYQwNCA4djUZEnA.png" /><figcaption>Side-by-side comparison between online article and Jina AI Reader result</figcaption></figure><p>Please also pay attention, for some websites this technique might not work very well. That is probably caused by a firewall or Cloudflare protection. You may use the proxy to mitigate it.</p><p>Also, Jina AI Reader provides a lot more configurations. You may find it out here <a href="https://jina.ai/reader/#apiform">https://jina.ai/reader/#apiform</a>.</p><h3>Conclusion</h3><p>Maintaining document structures can ensure the quality of LLM responses when it getting asked about our documents. Therefore, choosing the right tools is essential. PyMuPDF4LLM ensures the parsed document output is given in markdown format, which is great for maintaining document structures.</p><p>On the other side, Langchain implementation is easy to use. Unfortunately, it lacks of ability to preserve document structure. One may extend Langchain API to create a PyMuPDF4LLM integration. So that can take benefit from both sides.</p><p>Should you have other opinions or feedback, please never hesitate to comment below!</p><h3>Let’s get Connected 🙌</h3><p>If you have any inquiries, comments, suggestions, or critics please don’t hesitate to reach out:</p><ul><li>Mail: <a href="/service/mailto:affahrizain@gmail.com">affahrizain@gmail.com</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/fahrizainn/">https://www.linkedin.com/in/fahrizainn/</a></li><li>GitHub: <a href="https://github.com/fhrzn">https://github.com/fhrzn</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70d3dc2cd462" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/enhance-your-llms-understanding-document-parsing-tips-you-need-to-know-70d3dc2cd462">Enhance Your LLM’s Understanding: Document Parsing Tips You Need to Know</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing Data I/O: Analyzing Joblib’s Compression Techniques]]></title>
            <link>https://medium.com/data-folks-indonesia/optimizing-data-i-o-analyzing-joblibs-compression-techniques-1fa019d411b7?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/1fa019d411b7</guid>
            <category><![CDATA[python-compression]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[joblib]]></category>
            <dc:creator><![CDATA[Ferdina Kusumah]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 15:07:19 GMT</pubDate>
            <atom:updated>2024-11-21T15:07:20.995Z</atom:updated>
            <content:encoded><![CDATA[<p>An In-Depth Comparison of Zlib, LZMA, and LZ4 for Optimal Data Compression Performance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J4inqUfgqlFWTQ_Q" /><figcaption>Photo by <a href="https://unsplash.com/@giorgiotrovato?utm_source=medium&amp;utm_medium=referral">Giorgio Trovato</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the realm of data processing, efficient compression is key to optimizing storage and improving I/O operations. Three widely-used compression methods Zlib, LZMA, and LZ4 each offer unique benefits and trade-offs.</p><p><strong>Zlib</strong> is known for its balance between compression speed and ratio, making it a versatile choice for general purposes. <strong>LZMA</strong> provides superior compression ratios, reducing file sizes significantly, but at the cost of slower processing speeds. <strong>LZ4</strong>, on the other hand, emphasizes speed, delivering rapid compression and decompression, though with larger resulting file sizes.</p><p><a href="https://joblib.readthedocs.io/en/stable/"><strong>Joblib</strong></a> is a powerful Python library designed for efficient management of computational tasks. It excels at saving and loading Python objects, particularly in scenarios involving large datasets or complex computations. Joblib’s integration of various compression methods makes it an invaluable tool for developers looking to optimize their data processing workflows.</p><p>This article delves into a comparative analysis of these compression methods by using <a href="https://joblib.readthedocs.io/en/stable/">joblib</a> dump and load, specifically focusing on their performance in terms of speed for dumping and loading data. By understanding the strengths and weaknesses of each, you can make informed decisions to optimize your data processing workflows.</p><p>Before we start i’ll show my device specification for doing this test.</p><h3><strong>Device Specification</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*rSLvCgM_elCIqwUVi9QxPQ.png" /></figure><p>With Ubuntu (22.04) and CPU 13th Gen Intel i5–13400F (16) @ 4.600GHz with 32Gb RAM.</p><h3><strong>Environment Specification</strong></h3><ul><li>Python version 3.11.9</li><li>Joblib version 1.4.2</li></ul><p>For testing scenario i’ll use this method for reading from Real world online csv file that is contains 4 Million data in row, then try to compress data, see the size of compression file and load data from compressed file. Compression method that i’ll use is:</p><ul><li>Compress and load data using raw data (No compression method)</li><li>Compress and load data using Zlib compression</li><li>Compress and load data using LZMA compression</li><li>Compress and load data using LZ4 compression</li><li>Summary for all compression</li></ul><p>Let’s start the test</p><h3><strong>Compression with raw data (No compression method)</strong></h3><pre>import os<br>from time import time<br><br>import joblib<br>import pandas as pd<br><br><br>def compress_raw_data(df: pd.DataFrame, pickle_path: str):<br>    &quot;&quot;&quot;Compress dataframe without any compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;wb&quot;) as f:<br>        joblib.dump(df, f)<br><br>    end_time = time() - start_time<br>    print(f&quot;Compression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>def file_size_raw(pickle_path: str):<br>    &quot;&quot;&quot;Get file size of raw compressed dataframe&quot;&quot;&quot;<br>    file_size = os.stat(pickle_path).st_size / (1024 * 1024)<br>    print(f&quot;File size {file_size:.2f} MB&quot;)<br>    return file_size<br><br><br>def decompress_raw_data(pickle_path: str):<br>    &quot;&quot;&quot;Decompress dataframe without any compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;rb&quot;) as f:<br>        joblib.load(f)<br><br>    end_time = time() - start_time<br>    print(f&quot;Decompression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>if __name__ == &quot;__main__&quot;:<br>    url = &quot;https://github.com/joblib/dataset/raw/main/kddcup.data.gz&quot;<br>    names = (<br>        &quot;duration, protocol_type, service, flag, src_bytes, &quot;<br>        &quot;dst_bytes, land, wrong_fragment, urgent, hot, &quot;<br>        &quot;num_failed_logins, logged_in, num_compromised, &quot;<br>        &quot;root_shell, su_attempted, num_root, &quot;<br>        &quot;num_file_creations, &quot;<br>    ).split(&quot;, &quot;)<br><br>    data = pd.read_csv(url, names=names)<br>    pickle_file_name = (<br>        os.path.dirname(os.path.abspath(__file__)) + &quot;/pickle_data.pickle&quot;<br>    )<br>    compress_raw_data(data, pickle_file_name)<br>    decompress_raw_data(pickle_file_name)</pre><p>And the result by doing compression without any compression method is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*SIV1bIUacQjp9vXNN086rA.png" /><figcaption>raw data compression</figcaption></figure><p>It’s showing compression took 0.58 seconds and file size is around787 MB and when it’s try to load data from compressed file it took 0.18 seconds</p><h3>Compression With Zlib Method</h3><p>Compression level by default is 3 , in general is good compromise between compression and speed.</p><pre>import os<br>from time import time<br><br>import joblib<br>import pandas as pd<br><br><br>def compress_zlib_data(df: pd.DataFrame, pickle_path: str):<br>    &quot;&quot;&quot;Compress dataframe with zlib compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;wb&quot;) as f:<br>        joblib.dump(df, f, compress=&quot;zlib&quot;)<br><br>    end_time = time() - start_time<br>    print(f&quot;Compression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>def file_size_zlib(pickle_path: str):<br>    &quot;&quot;&quot;Get file size of zlib compressed dataframe&quot;&quot;&quot;<br>    file_size = os.stat(pickle_path).st_size / (1024 * 1024)<br>    print(f&quot;File size {file_size:.2f} MB&quot;)<br>    return file_size<br><br><br>def decompress_zlib_data(pickle_path: str):<br>    &quot;&quot;&quot;Decompress dataframe with zlib compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;rb&quot;) as f:<br>        joblib.load(f)<br><br>    end_time = time() - start_time<br>    print(f&quot;Decompression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>if __name__ == &quot;__main__&quot;:<br>    url = &quot;https://github.com/joblib/dataset/raw/main/kddcup.data.gz&quot;<br>    names = (<br>        &quot;duration, protocol_type, service, flag, src_bytes, &quot;<br>        &quot;dst_bytes, land, wrong_fragment, urgent, hot, &quot;<br>        &quot;num_failed_logins, logged_in, num_compromised, &quot;<br>        &quot;root_shell, su_attempted, num_root, &quot;<br>        &quot;num_file_creations, &quot;<br>    ).split(&quot;, &quot;)<br><br>    data = pd.read_csv(url, names=names)<br>    pickle_file_name = (<br>        os.path.dirname(os.path.abspath(__file__)) + &quot;/pickle_data.pickle&quot;<br>    )<br>    compress_zlib_data(data, pickle_file_name)<br>    decompress_zlib_data(pickle_file_name)</pre><p>And the result by doing compression with Zlib compression method is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*CXw9s0XwyPqjikEWf8CFFg.png" /><figcaption>Zlib compression</figcaption></figure><p>It’s showing compression took 1.25 seconds and file size is around11 MB and when it’s try to load data from compressed file it took 0.65 seconds</p><h3>Compression With LZMA Method</h3><p>LZMA has good compression method, but at the cost of being very slow. Here i’ll use same compression method as Zlib to speed up our dump and load. Value is from 0–9 . Higher value means more compression, but also slower read and write times</p><pre>import os<br>from time import time<br><br>import joblib<br>import pandas as pd<br><br><br>def compress_lzma_data(df: pd.DataFrame, pickle_path: str):<br>    &quot;&quot;&quot;Compress dataframe with lzma compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;wb&quot;) as f:<br>        joblib.dump(df, f, compress=(&quot;lzma&quot;, 3))<br><br>    end_time = time() - start_time<br>    print(f&quot;Compression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>def file_size_lzma(pickle_path: str):<br>    &quot;&quot;&quot;Get file size of lzma compressed dataframe&quot;&quot;&quot;<br>    file_size = os.stat(pickle_path).st_size / (1024 * 1024)<br>    print(f&quot;File size {file_size:.2f} MB&quot;)<br>    return file_size<br><br><br>def decompress_lzma_data(pickle_path: str):<br>    &quot;&quot;&quot;Decompress dataframe with lzma compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;rb&quot;) as f:<br>        joblib.load(f)<br><br>    end_time = time() - start_time<br>    print(f&quot;Decompression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>if __name__ == &quot;__main__&quot;:<br>    url = &quot;https://github.com/joblib/dataset/raw/main/kddcup.data.gz&quot;<br>    names = (<br>        &quot;duration, protocol_type, service, flag, src_bytes, &quot;<br>        &quot;dst_bytes, land, wrong_fragment, urgent, hot, &quot;<br>        &quot;num_failed_logins, logged_in, num_compromised, &quot;<br>        &quot;root_shell, su_attempted, num_root, &quot;<br>        &quot;num_file_creations, &quot;<br>    ).split(&quot;, &quot;)<br><br>    data = pd.read_csv(url, names=names)<br>    pickle_file_name = (<br>        os.path.dirname(os.path.abspath(__file__)) + &quot;/pickle_data.pickle&quot;<br>    )<br>    compress_lzma_data(data, pickle_file_name)<br>    decompress_lzma_data(pickle_file_name)</pre><p>And the result by doing compression with LZMA compression method is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*IL1JWEWlsAC_Azw7McBxsg.png" /><figcaption>LZMA compression</figcaption></figure><p>It’s showing compression took 3.76 seconds and file size is around5 MB and when it’s try to load data from compressed file it took 0.89 seconds</p><h3>Compression With LZ4 Method</h3><p>LZ4 compression method is known to be one of the fastest available compression method but with a compression rate a bit lower than Zlib. You need to install that package manually <a href="https://pypi.org/project/lz4/">here</a>. Or execute this command pip install lz4</p><pre>import os<br>from time import time<br><br>import joblib<br>import pandas as pd<br><br><br>def compress_lz4_data(df: pd.DataFrame, pickle_path: str):<br>    &quot;&quot;&quot;Compress dataframe with lz4 compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;wb&quot;) as f:<br>        joblib.dump(df, f, compress=&quot;lz4&quot;)<br><br>    end_time = time() - start_time<br>    print(f&quot;Compression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>def file_size_lz4(pickle_path: str):<br>    &quot;&quot;&quot;Get file size of lz4 compressed dataframe&quot;&quot;&quot;<br>    file_size = os.stat(pickle_path).st_size / (1024 * 1024)<br>    print(f&quot;File size {file_size:.2f} MB&quot;)<br>    return file_size<br><br><br>def decompress_lz4_data(pickle_path: str):<br>    &quot;&quot;&quot;Decompress dataframe with lz4 compressions method&quot;&quot;&quot;<br>    start_time = time()<br>    with open(pickle_path, &quot;rb&quot;) as f:<br>        joblib.load(f)<br><br>    end_time = time() - start_time<br>    print(f&quot;Decompression took {end_time:.2f} seconds&quot;)<br>    return end_time<br><br><br>if __name__ == &quot;__main__&quot;:<br>    url = &quot;https://github.com/joblib/dataset/raw/main/kddcup.data.gz&quot;<br>    names = (<br>        &quot;duration, protocol_type, service, flag, src_bytes, &quot;<br>        &quot;dst_bytes, land, wrong_fragment, urgent, hot, &quot;<br>        &quot;num_failed_logins, logged_in, num_compromised, &quot;<br>        &quot;root_shell, su_attempted, num_root, &quot;<br>        &quot;num_file_creations, &quot;<br>    ).split(&quot;, &quot;)<br><br>    data = pd.read_csv(url, names=names)<br>    pickle_file_name = (<br>        os.path.dirname(os.path.abspath(__file__)) + &quot;/pickle_data.pickle&quot;<br>    )<br>    compress_lz4_data(data, pickle_file_name)<br>    decompress_lz4_data(pickle_file_name)</pre><p>And the result by doing compression with LZ4 compression method is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*xte-kAsK8mMfArNtPuNE8Q.png" /><figcaption>LZ4 compression</figcaption></figure><p>It’s showing compression took 0.23 seconds and file size is around20 MB and when it’s try to load data from compressed file it took 0.32 seconds</p><h3>Summary For All Compression Method</h3><p>This is script for comparing speed dump and load data from data</p><pre>import os<br><br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br><br>from lz4_data import compress_lz4_data, decompress_lz4_data<br>from lzma_data import compress_lzma_data, decompress_lzma_data<br>from raw_data import compress_raw_data, decompress_raw_data<br>from zlib_data import compress_zlib_data, decompress_zlib_data<br><br>url = &quot;https://github.com/joblib/dataset/raw/main/kddcup.data.gz&quot;<br>names = (<br>    &quot;duration, protocol_type, service, flag, src_bytes, &quot;<br>    &quot;dst_bytes, land, wrong_fragment, urgent, hot, &quot;<br>    &quot;num_failed_logins, logged_in, num_compromised, &quot;<br>    &quot;root_shell, su_attempted, num_root, &quot;<br>    &quot;num_file_creations, &quot;<br>).split(&quot;, &quot;)<br><br>df = pd.read_csv(url, names=names)<br>pickle_path = os.path.dirname(os.path.abspath(__file__))<br><br>N = 4<br>load_durations = (<br>    compress_raw_data(df, pickle_path + &quot;/pickle_raw.pickle&quot;),<br>    compress_lz4_data(df, pickle_path + &quot;/pickle_lz4.pickle&quot;),<br>    compress_zlib_data(df, pickle_path + &quot;/pickle_zlib.pickle&quot;),<br>    compress_lzma_data(df, pickle_path + &quot;/pickle_lzma.pickle&quot;),<br>)<br>dump_durations = (<br>    decompress_raw_data(pickle_path + &quot;/pickle_raw.pickle&quot;),<br>    decompress_lz4_data(pickle_path + &quot;/pickle_lz4.pickle&quot;),<br>    decompress_zlib_data(pickle_path + &quot;/pickle_zlib.pickle&quot;),<br>    decompress_lzma_data(pickle_path + &quot;/pickle_lzma.pickle&quot;),<br>)<br>ind = np.arange(N)<br>width = 0.5<br><br>plt.figure(1, figsize=(5, 4))<br>p1 = plt.bar(ind, dump_durations, width)<br>p2 = plt.bar(ind, load_durations, width, bottom=dump_durations)<br>plt.ylabel(&quot;Time in seconds&quot;)<br>plt.title(&quot;Dump and Load durations\n(lower better)&quot;)<br>plt.xticks(ind, (&quot;Raw&quot;, &quot;LZ4&quot;, &quot;Zlib&quot;, &quot;LZMA&quot;))<br>plt.legend((p1[0], p2[0]), (&quot;Dump duration&quot;, &quot;Load duration&quot;))<br>plt.show()</pre><p>And the result by comparing all compression method that is showing by chart is this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*3mYuz1Wqodb6ERTG8VOFnQ.png" /><figcaption>dump and load speed compression</figcaption></figure><p>We can see that LZ4 compression is better in speed compare to other compression method.</p><p>And now comparison by size that generated by each compression</p><pre>import os<br><br>import matplotlib.pyplot as plt<br>import numpy as np<br><br>from lz4_data import file_size_lz4<br>from lzma_data import file_size_lzma<br>from raw_data import file_size_raw<br>from zlib_data import file_size_zlib<br><br>pickle_path = os.path.dirname(os.path.abspath(__file__))<br><br>N = 4<br>file_sizes = (<br>    file_size_raw(pickle_path + &quot;/pickle_raw.pickle&quot;),<br>    file_size_lz4(pickle_path + &quot;/pickle_lz4.pickle&quot;),<br>    file_size_zlib(pickle_path + &quot;/pickle_zlib.pickle&quot;),<br>    file_size_lzma(pickle_path + &quot;/pickle_lzma.pickle&quot;),<br>)<br>ind = np.arange(N)<br>width = 0.5<br><br>plt.figure(1, figsize=(5, 4))<br>p1 = plt.bar(ind, file_sizes, width, log=True)<br>plt.ylabel(&quot;Size in MB&quot;)<br>plt.title(&quot;Size After Compression\n(lower better)&quot;)<br>plt.xticks(ind, (&quot;Raw&quot;, &quot;LZ4&quot;, &quot;Zlib&quot;, &quot;LZMA&quot;))<br>plt.legend((p1[0],), (&quot;File size&quot;,))<br>plt.show()</pre><p>And the result size by comparing all compression method that is showing by chart is this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*oIMMkofNFfD3OestZq_skg.png" /><figcaption>size compression comparison</figcaption></figure><p>We can see that LZMA is better than other compression method. It’s only take around5MB in size.</p><p>So in summary, the performance of three compression methods Zlib, LZMA, and LZ4 was evaluated using Joblib to compare their speed and efficiency for dumping and loading data.</p><blockquote>The results revealed that LZ4 provided the fastest processing times, making it ideal for scenarios where speed is crucial. LZMA achieved the best compression ratios, significantly reducing file sizes but with slower speeds. Zlib offered a balanced approach with moderate speed and compression.</blockquote><h3>Reference</h3><ul><li><a href="https://joblib.readthedocs.io/en/stable/">Joblib: running Python functions as pipeline jobs - joblib 1.4.2 documentation</a></li><li><a href="https://joblib.readthedocs.io/en/stable/auto_examples/compressors_comparison.html#sphx-glr-auto-examples-compressors-comparison-py">Improving I/O using compressors - joblib 1.4.2 documentation</a></li><li><a href="https://github.com/joblib/dataset">GitHub - joblib/dataset: Dataset for example and CI</a></li><li><a href="https://pypi.org/project/lz4/">lz4</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fa019d411b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/optimizing-data-i-o-analyzing-joblibs-compression-techniques-1fa019d411b7">Optimizing Data I/O: Analyzing Joblib’s Compression Techniques</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data to Deployment: Creating a Machine Learning Web Service with Flask]]></title>
            <link>https://medium.com/data-folks-indonesia/from-data-to-deployment-creating-a-machine-learning-web-service-with-flask-9212fa7363a0?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/9212fa7363a0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[iris-dataset]]></category>
            <dc:creator><![CDATA[Ferdina Kusumah]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 15:06:55 GMT</pubDate>
            <atom:updated>2024-11-21T15:06:58.214Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how to build and deploy a machine learning model as a web service using Python Flask, and the Iris dataset in a few simple steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O4QXQdNRY9qLBKFw" /><figcaption>Photo by <a href="https://unsplash.com/@agk42?utm_source=medium&amp;utm_medium=referral">Alex Knight</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the rapidly evolving world of technology, the ability to seamlessly integrate machine learning models into web applications is a highly sought-after skill. Whether you’re a data scientist looking to showcase your predictive models, or a web developer aiming to add intelligent features to your applications, understanding how to deploy machine learning models as web services is essential. In this article, we will walk through the entire process of creating a machine learning web service using Python Flask, and the well-known Iris dataset.</p><p>We’ll start by training a machine learning model with the Iris dataset, a classic in the field of data science, known for its simplicity and effectiveness. Once our model is trained and validated, we’ll use Flask, a lightweight web framework for Python, to create a web service that can handle requests and return predictions. This hands-on approach ensures that you’ll not only grasp the theoretical aspects but also gain practical experience in deploying machine learning models.</p><p>By the end of this guide, you’ll have a fully functional web service that can predict iris species from sepal and petal measurements. This journey from data to deployment will equip you with the skills to turn your machine learning projects into interactive web applications, opening up a world of possibilities for both personal projects and professional development. So, let’s dive in and transform our machine learning model into a web-based predictive service!</p><p>We will start with creating simple script to train your pre-built models that provided by scikit-learn</p><p>In this example i’ve used this spesification version:</p><ul><li>Python version 3.11</li><li>Flask version 3.0.3</li></ul><p>Let’s dive into code 🚀🚀</p><h3><strong>Activate the virtual environment</strong></h3><p>I’m encourage you to use virtual environment over directly in main root python, it’s keep your environment isolated, easy to maintain, and not breaking other code in future. Another things is that you can install different version of python by using virtual environment.</p><h3>Install required library</h3><p>Install this all required dependency.</p><ul><li>pip install sklearn</li></ul><p>This is library for machine learning, also included with pre-built iris dataset that we will used.</p><ul><li>pip install flask</li></ul><p>This is a famous web framework in python, that is mainly used for web development for python. It’s also stable and widely used</p><ul><li>pip install pickle</li></ul><p>This is library that help us to serialize and de-serialize python object when our machine learning model is ready after training. It’s bundle our final result.</p><h3>Write our machine learning code</h3><p>Create new file called train.py and put this code.</p><pre># 1. Import necessary libraries<br>import pickle<br><br>from sklearn.datasets import load_iris<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.model_selection import train_test_split<br><br># 2. Load the Iris dataset<br>iris = load_iris()<br>X, y = iris.data, iris.target<br><br># 3. Split the dataset into training and test sets<br>X_train, _, y_train, _ = train_test_split(X, y, test_size=0.3, random_state=42)<br><br># 4. Train a RandomForestClassifier<br>model = RandomForestClassifier(n_estimators=100)<br>model.fit(X_train, y_train)<br><br># 5. Save the model to a file<br>with open(&quot;iris_model.pkl&quot;, &quot;wb&quot;) as model_file:<br>    pickle.dump(model, model_file)</pre><p>This is our machine learning code, then we will go through each one of this step.</p><ul><li>In step 1, we imported all required library that already installed in our environment.</li><li>In step 2, we call pre-built iris dataset that already installed when we install sklearn library earlier. The data will looks like this</li></ul><pre>5.1,3.5,1.4,0.2,0<br>4.9,3.0,1.4,0.2,0<br>4.7,3.2,1.3,0.2,0<br>6.5,2.8,4.6,1.5,1<br>5.7,2.8,4.5,1.3,1<br>7.7,2.6,6.9,2.3,2<br>6.0,2.2,5.0,1.5,2<br>6.9,3.2,5.7,2.3,2</pre><p>Where column order is (left to right): sepal length, sepal width, petal length, petal width, and the prediction number, where 0=setosa , 1=versicolor , 2=virginica</p><p>Then we split the data, column 1–4 is our data that define as Xand 5 is our prediction defined as y</p><ul><li>In step 3, we will split our dataset to be prepared for training, with this scenario</li></ul><p>Param X is our matrix data that we will train, y is prediction result, test_size=0.3 means we will used 30% of our dataset to be the test set, and the remaining 70% will be as training set.</p><p>It will return 4 value, X_train , X_test , y_train , y_test for value X_test and y_test we will not used in this so i’m make it _ ignore variable. Only X_train and y_train that we will used for training.</p><ul><li>In step 4, we train our data that already prepared earlier in step 3 by using RandomForestClassifier</li></ul><p>There are many other classifier that we can use for training machine learning, in this case i choose RandomForestClassifier with n_estimators=100 means number of trees in the forest. Then we fit or train our dataset with this classifier to be final dataset.</p><ul><li>In step 5, we get the final result of our dataset that is ready to make prediction based on parameter: sepal length, sepal width, petal length, petal width and we save final result as pickle</li></ul><h3>Write our web service</h3><p>Then create another file app.py then put this code in it.</p><pre>import pickle<br><br>from flask import Flask, jsonify, request<br><br>app = Flask(__name__)<br><br># load model that already trained<br>with open(&quot;iris_model.pkl&quot;, &quot;rb&quot;) as model_file:<br>    model = pickle.load(model_file)<br><br># flower label prediction<br>FLOWER_LABEL = {0: &quot;setosa&quot;, 1: &quot;versicolor&quot;, 2: &quot;virginica&quot;}<br><br>@app.route(&quot;/&quot;, methods=[&quot;OPTIONS&quot;, &quot;GET&quot;])<br>def home():<br>    return jsonify({&quot;status&quot;: 200, &quot;message&quot;: &quot;services is up and running!&quot;})<br><br><br>@app.route(&quot;/prediction&quot;, methods=[&quot;OPTIONS&quot;, &quot;POST&quot;])<br>def prediction():<br>    sl = request.json[&quot;sepal_length&quot;]<br>    sw = request.json[&quot;sepal_width&quot;]<br>    pl = request.json[&quot;petal_length&quot;]<br>    pw = request.json[&quot;petal_width&quot;]<br><br>    # make predictions<br>    resp = model.predict([[sl, sw, pl, pw]])<br>    result = FLOWER_LABEL[resp[0]]<br>    return jsonify({&quot;status&quot;: 200, &quot;prediction&quot;: result})<br><br><br>if __name__ == &quot;__main__&quot;:<br>    app.run()</pre><p>This is our web service to serve our machine learning prediction, we read the pickle model right before service is running, i’ll keep this very simple to understand the flow, then we defined the label to make it easier after we make prediction.</p><p>In this web service there’s only 2 routes that available, one is for checking if our service is running and healthy. The other to make prediction based on parameter that provided.</p><p>Then run web services by execute command python app.py , you should see our service running like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*XpoHhrrSY6qBJt5FlnXlAA.png" /></figure><p>This is the example when we test in postman</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*Y2kyKlvhFGIdcdkWnzs_vg.png" /></figure><p>Now we integrate machine learning code to our web service 🎉🎉🎉.</p><p>Keep in mind, this is simple code that i used, should not put in production. In real case, we can put in intermediary storage like redis, s3 storage, or put in database. Also we can use worker to execute prediction if our model is large. But the concept should be same.</p><h3><strong>Reference</strong></h3><ul><li><a href="https://scikit-learn.org/stable/">scikit-learn</a></li><li><a href="https://flask.palletsprojects.com/en/3.0.x/">Welcome to Flask - Flask Documentation (3.0.x)</a></li><li><a href="https://docs.python.org/3/library/pickle.html">pickle - Python object serialization</a></li><li><a href="https://en.wikipedia.org/wiki/Iris_flower_data_set">Iris flower data set - Wikipedia</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9212fa7363a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/from-data-to-deployment-creating-a-machine-learning-web-service-with-flask-9212fa7363a0">From Data to Deployment: Creating a Machine Learning Web Service with Flask</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Machine Learning K-Nearest Neighbor using LLM]]></title>
            <link>https://medium.com/data-folks-indonesia/learn-machine-learning-k-nearest-neighbor-using-llm-5c9bfb8882cd?source=rss----e743ed8d2403---4</link>
            <guid isPermaLink="false">https://medium.com/p/5c9bfb8882cd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[k-nearest-neighbours]]></category>
            <dc:creator><![CDATA[AC]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 02:01:35 GMT</pubDate>
            <atom:updated>2024-01-03T02:01:35.357Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F0NaEsqVj-udY3kT" /><figcaption>Photo by <a href="https://unsplash.com/@nadineshaabana?utm_source=medium&amp;utm_medium=referral">Nadine Shaabana</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Chat agent based on LLM is no doubt is on the hype. LLM is build for multi-purposes. Some use it for brainstorm, explain concepts, polishing their text, and more. Here, I tried to learn machine learning using LLM called pi.ai by Inflection AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*ACsENAWEeoIXI1TRFmq5Vg.png" /><figcaption>Discover feature in pi.ai</figcaption></figure><p>Here I am, trying to verify if it is safe enough to learn something new that are really specific, and maintain our curiousity to learn something new.</p><p><em>Let’s learn machine learning using pi.ai</em></p><h3>Introduction</h3><p>I’m excited to explore a new topic with you. What would you like to learn about?</p><blockquote>machine learning</blockquote><p>Machine learning is a fascinating field, and there’s a lot to explore! Do you want to start from the basics or dive into a more specific topic? Machine learning encompasses everything from supervised and unsupervised learning, to neural networks and deep learning. What are you interested in?</p><blockquote>lets do supervised learning</blockquote><p>Great choice! Supervised learning is a type of machine learning where the model is trained on labeled data. In supervised learning, the training data is composed of input-output pairs, where the input is a set of features or attributes, and the output is the label or target variable.</p><p>The goal of supervised learning is to build a model that can generalize from the training data to make predictions on new, unseen data.</p><p>Some common supervised learning tasks include classification (categorizing data into different classes) and regression (predicting continuous values, such as prices or temperatures).</p><p>If you’d like, we can dig deeper into the algorithms, approaches, and applications of supervised learning. 😊</p><blockquote>lets continue</blockquote><p>Of course! One of the key components of supervised learning is the learning algorithm. These algorithms are responsible for taking the training data and using it to learn the relationship between the input and output variables.<br>Some common supervised learning algorithms include:</p><ul><li>Linear regression: A simple algorithm that learns a linear relationship between the input and output variables.</li><li>Logistic regression: A similar algorithm that is used for classification tasks, where the output variable is categorical.</li><li>Support vector machines (SVMs): A more complex algorithm that is often used for classification tasks.</li><li>Decision trees: An algorithm that learns a decision tree structure to classify data.</li></ul><p>Each of these algorithms has its own strengths and weaknesses, and the choice of algorithm will depend on the specific problem and dataset.</p><h3>K-Nearest Neighbor</h3><p><em>I wonder why pi is not list K-Nearest Neighbor</em></p><blockquote>Is K-Nearest Neighbor is supervised learning algorithm?</blockquote><p>Yes, K-Nearest Neighbor (KNN) is a supervised learning algorithm. It is a non-parametric, lazy learning algorithm that is used for both classification and regression tasks.</p><p>The basic idea behind KNN is that, given a new data point, the model will classify or predict its label by looking at the labels of its nearest neighbors in the training data. The “K” in KNN refers to the number of neighbors that are used to make the prediction.</p><p>The main advantage of KNN is that it is easy to understand and implement. However, it can be computationally expensive, as it requires the model to store all of the training data and compute the distance between each new data point and all of the training points.</p><p>KNN is often used for problems where there are few training data points or where the data is highly non-linear. It is also a popular choice for problems where interpretability is important, as the model’s predictions can be easily understood by looking at the nearest neighbors.</p><p><em>I ask everytime I found new term, this one, non-parametric</em></p><blockquote>why KNN is non-parametric?</blockquote><p>KNN is non-parametric because it does not assume any specific underlying distribution or functional form for the data. Unlike other supervised learning algorithms, such as linear regression or logistic regression, KNN does not require the data to be normally distributed or to have any particular functional form.</p><p>This means that KNN can handle data that is non-linear, non-Gaussian, or contains outliers, without making any assumptions about the underlying distribution.</p><p>The non-parametric nature of KNN allows it to be used for a wide variety of problems and datasets, without requiring extensive knowledge about the data or the problem domain.</p><p>However, this flexibility also means that KNN can be more prone to overfitting, as the model may be too flexible and capture random noise in the data rather than the underlying pattern.</p><p><em>This one is lazy learning</em></p><blockquote>why KNN is lazy algorithm?</blockquote><p>KNN is considered a lazy learning algorithm because it does not explicitly build a model during the training phase.</p><p>Unlike other supervised learning algorithms, such as decision trees or neural networks, KNN does not learn a model that generalizes from the training data. Instead, it simply stores the training data and uses it to make predictions when new data is presented.</p><p>This lazy approach means that KNN does not require any explicit training phase and can make predictions on new data immediately. However, it also means that KNN requires all of the training data to be stored in memory, which can become impractical for large datasets.</p><p>Another consequence of KNN’s laziness is that it is not as scalable as other algorithms. Because it has to store all of the training data in memory, it can become slow and computationally expensive for large datasets.</p><p>Overall, KNN’s lazy nature makes it simple and efficient to use, but it also has some limitations in terms of scalability and generalizability.</p><p><em>I can confirm that, the explaination from pi.ai is correct</em></p><h3>Method</h3><blockquote>how KNN works?</blockquote><p>Here’s a quick overview of how KNN works:</p><ul><li>First, the training data is divided into K nearest neighbors for each data point. This is typically done using some distance metric, such as Euclidean distance.</li><li>Then, when a new data point is presented, the K nearest neighbors are identified in the training data.</li><li>The class or value of the new data point is then predicted based on the majority class or value of its K nearest neighbors.</li></ul><p>For example, if the new data point is a customer record and the task is to predict whether the customer will churn or not, the K nearest neighbors would be the customer records with the closest attributes to the new record. The prediction would be based on the churn status of the majority of the K nearest neighbors.<br>This simple yet effective approach makes KNN a popular choice for many real-world problems, such as customer segmentation, fraud detection, and image classification. 😊</p><blockquote>what is euclidean distance?</blockquote><p>Euclidean distance is a metric used to measure the distance between two points in a multi-dimensional space. It is named after the ancient Greek mathematician Euclid, who developed the concept of distance in geometry.</p><p>Mathematically, the Euclidean distance between two points (x_1, y_1) and (x_2, y_2) in a two-dimensional space can be calculated using the following formula:</p><p>d = sqrt((x_2-x_1)² + (y_2-y_1)²)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NBx2eRr5wZJs7h2q.png" /><figcaption>Euclidean distance — Wikipedia</figcaption></figure><p>In this formula, d represents the Euclidean distance between the two points, and x_1, y_1, x_2, and y_2 represent the coordinates of the two points.</p><p>In a multi-dimensional space, the Euclidean distance is calculated by summing the squared difference between each coordinate of the two points, and then taking the square root of the sum.</p><p>Euclidean distance is a commonly used metric in machine learning and statistics, particularly in clustering and classification algorithms. 😊</p><p><em>I inserted the image for you to understand visually, pi could not generate the image</em></p><h3>Parameter K in KNN</h3><p>Choosing the value of K in KNN can be a challenging task, as it directly affects the performance of the algorithm. Here are some tips for deciding the best value of K:</p><ul><li>Start with a small value of K, such as 3 or 5, and gradually increase it until the performance stabilizes or starts to decline.</li><li>Use cross-validation to test different values of K and choose the one that performs best on the validation set.</li><li>Consider the size and dimensionality of the data. For small datasets, a lower value of K may be more appropriate, while for high-dimensional data, a higher value of K may be necessary to capture the underlying structure.</li><li>Experiment with different distance metrics, such as Euclidean distance or Manhattan distance, to see which works best for your data.</li><li>Keep in mind that the choice of K is a trade-off between accuracy and computational efficiency. A higher value of K may lead to more accurate predictions, but it can also be more computationally expensive.</li></ul><h3>K-NN Distances</h3><p>There are several distance metrics that can be used with KNN, each with its own strengths and weaknesses. Here are some of the most commonly used distance metrics:</p><ul><li>Euclidean distance: This is the most commonly used distance metric in KNN. It measures the straight-line distance between two points in multi-dimensional space.</li><li>Manhattan distance: Also known as city block distance, this metric calculates the distance between two points by summing the absolute value of the differences between their coordinates.</li><li>Minkowski distance: This metric is a generalization of both Euclidean and Manhattan distance, and can be tuned by adjusting a parameter called p.</li><li>Hamming distance: This metric is used for categorical data, and calculates the distance between two points based on the number of positions where their categorical values differ.</li></ul><p>Each of these distance metrics has different properties and is more suitable for certain types of data and tasks. It is important to experiment with different metrics to find the one that works best for your problem.</p><p><a href="https://www.analyticsvidhya.com/blog/2021/08/how-knn-uses-distance-measures/">How KNN Uses Distance Measures?</a></p><h3>K-Nearest Neighbor Pros and Cons</h3><blockquote>what are the pros and cons using KNN?</blockquote><p>KNN has several advantages and disadvantages, which can make it more or less suitable for different types of problems.</p><h4>Pros</h4><ul><li>KNN is simple to understand and implement, making it a good choice for beginners in machine learning.</li><li>It is non-parametric, which means it can handle non-linear data without requiring any assumptions about the underlying distribution.</li><li>KNN can be used for both classification and regression tasks.</li><li>It does not require any explicit training phase, making it fast and efficient for prediction tasks.</li></ul><h4>Cons:</h4><ul><li>KNN can be computationally expensive, as it requires storing all of the training data in memory.</li><li>It is prone to overfitting, as it does not generalize well to new data.<br>KNN is sensitive to the choice of K, and choosing the wrong K can lead to poor performance.</li><li>It does not work well for high-dimensional data, as the number of nearest neighbors increases exponentially with the number of dimensions.</li></ul><p>Overall, KNN is a useful and versatile algorithm, but it is not always the best choice for every problem. It is important to consider the specific characteristics of the data and the task at hand when choosing a machine learning algorithm.</p><h3>Evaluation Method</h3><p><em>This is general evaluation method for supervised learning, you can use for other algorithm too other than K-Nearest Neighbor</em></p><p>Evaluating supervised learning algorithms is an important step in the machine learning process, as it helps to determine the effectiveness of the model and identify areas for improvement. Here are some common methods for evaluating supervised learning algorithms:</p><ul><li>Accuracy: This is the most common metric used to evaluate supervised learning algorithms. It measures the percentage of correctly classified instances out of the total number of instances in the test dataset.</li><li>Precision and recall: These metrics are particularly useful for binary classification tasks, and measure the proportion of true positives and true negatives, respectively.</li><li>F-score: This metric combines precision and recall into a single score, and is often used to measure the performance of a model on imbalanced datasets.</li><li>ROC curve: The receiver operating characteristic curve is a plot of the true positive rate against the false positive rate, and is used to evaluate the trade-off between sensitivity and specificity.</li><li>Cross-validation: This is a technique that involves partitioning the dataset into multiple folds, training the model on each fold, and then evaluating its performance on the remaining fold. (edit: Cross-validation is under model selection, the way you evaluate your model into K Folds to determine if the mode is overfit or not)</li></ul><h3>Metrics Explaination</h3><p><em>In this section, I asked pi.ai to explain precision in layman term, which means this can be useful when you explain your model score to the non technical person</em></p><p>Precision score is a metric used to evaluate the performance of a classification model in machine learning. It can be explained in layman’s terms as follows:</p><p>Imagine you are a detective trying to catch a criminal. You have a list of suspects, and you want to figure out which ones are actually guilty. The precision score measures how good you are at correctly identifying the guilty suspects.</p><p>For example, let’s say you have a list of 10 suspects, and you correctly identify 3 of the 4 guilty suspects. Your precision score would be 3/4, or 75%. This means that you are pretty good at identifying the guilty suspects, but you still have some room for improvement.</p><p>In machine learning, the precision score is used to measure how accurate the model is at correctly classifying the positive class (guilty suspects), without including too many false positives (innocent suspects). A high precision score indicates that the model is good at correctly identifying the positive class, while a low precision score indicates that the model is generating too many false positives.</p><h3>Conclusion</h3><p>This is the second article I try to use LLM for learning something. I found it useful and the thing that drive our learning is purely our curiousity. Learning something from LLM is not enough, it may give false information, so you need to double check and confirm with other experts to understand fully and rightly about the method. But, it is indeed a good start. Other than that, you can go back to text book too, to understand the method deeply.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c9bfb8882cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-folks-indonesia/learn-machine-learning-k-nearest-neighbor-using-llm-5c9bfb8882cd">Learn Machine Learning K-Nearest Neighbor using LLM</a> was originally published in <a href="https://medium.com/data-folks-indonesia">Data Folks Indonesia</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>