<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by George Pearse on Medium]]></title>
        <description><![CDATA[Stories by George Pearse on Medium]]></description>
        <link>https://medium.com/@george.pearse?source=rss-b502c2f08a7c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rkMqBSuKZx5ImZMbzAq7lg.jpeg</url>
            <title>Stories by George Pearse on Medium</title>
            <link>https://medium.com/@george.pearse?source=rss-b502c2f08a7c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:58:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@george.pearse/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Missing Terminology of Data Centric AI]]></title>
            <link>https://medium.com/@george.pearse/the-missing-terminology-of-data-centric-ai-5a0afb488817?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a0afb488817</guid>
            <category><![CDATA[data-centric-ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Sat, 16 Dec 2023 15:19:26 GMT</pubDate>
            <atom:updated>2024-04-17T07:54:24.964Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Centric AI is part marketing gimmick, but also has its value in places. This article quickly goes through some terms the industry could benefit from, in order to understand the data problems presented by supervised learning at scale and speed.</p><h3>Data Viscosity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KW-gbpYD5943xh1JCkuvmA.png" /><figcaption>a viscous liquid</figcaption></figure><p>As a dataset gets larger, it becomes more and more difficult to alter it in order to train models to respond to requests from customers.</p><p>Early datapoints may be labelled with classes which have since become more specific.</p><blockquote><strong>“Oh could you just add X”</strong></blockquote><p>This can be an intimidating question if you’ve already scaled your dataset to many 10s of thousands of images in order to achieve your required accuracy, but with a dense set of informative images, and clean labels, it can be a much more manageable scenario.</p><p>Another symptom is that you’re also unable to experiment at the speed you once could, and are no longer able to iterate through ideas at the rate you feel should be possible. A potential solution is to always train on <strong>‘best subsets’</strong>, steadily removing data which may have become redundant due to over-representation.</p><h3><strong>Dataset Claustrophobia</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*lgguSv4PVTOAqenmJlsMAw.png" /><figcaption>looking claustrophobic</figcaption></figure><p>Your dataset is too large for the tools that you use for you to ever feel like you’re achieving a <strong>“birds eye view”</strong>, and have confidence that there’s not a large set of erroneous labels that you’re unaware of.</p><p>See my article on using ‘bulk’ <a href="https://medium.com/mlearning-ai/quick-nlp-labelling-with-bulk-327ccb62320">https://medium.com/mlearning-ai/quick-nlp-labelling-with-bulk-327ccb62320</a> for a potential remedy, but voxel51 <a href="https://voxel51.com/">https://voxel51.com/</a> also serves a similar purpose in the context of Computer Vision.</p><p>A core part of combating this, is to use tools with appropriate speed for the scale of your dataset. It may be worth training an additional fast, but less accurate model, just as a tool to sift through unannotated images, or to try to find blatent errors in the training set (provided that fast model has not memorised the training annotations).</p><h3><strong>Defensive Classes</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*MTABOD2t_4AfGwtuV-3e5w.png" /></figure><p>The client does not directly care about the class, but its existence helps the model to train, and allows you to understand errors at a larger scale.</p><p>It is a <strong>“defensive class”</strong>, because it’s main purpose is to prevent the model from getting confused between it, and a “<strong>product class”</strong>.</p><p>Many of these terms exist to address the under-appreciated downsides of scale. Scale is great for a large, well funded team, but has its drawbacks if you’re trying to move fast in a resource constrained environment.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a0afb488817" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to train a model with MMDetection]]></title>
            <link>https://medium.com/@george.pearse/how-to-train-a-model-with-mmdetection-8ab6a6fea3f0?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ab6a6fea3f0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mmdetection]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Sat, 16 Dec 2023 11:22:02 GMT</pubDate>
            <atom:updated>2023-12-16T16:32:38.079Z</atom:updated>
            <content:encoded><![CDATA[<p>MMDetection is an excellent tool, I’ve used Detectron2 and Pytorch-Lightning with the torch native faster_rcnn (<a href="https://pytorch.org/vision/main/models/faster_rcnn.html">https://pytorch.org/vision/main/models/faster_rcnn.html</a>), MMDetection is much more versatile and powerful than both, supports export to ONNX, and continues to receive signficant updates.</p><p>There’s also the excellent looking MMDeploy to simplify matching your research and deployment setups which I’ve yet to properly take for a spin (<a href="https://mmdeploy.readthedocs.io/en/latest/03-benchmark/supported_models.html">https://mmdeploy.readthedocs.io/en/latest/03-benchmark/supported_models.html</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*53MtcoMC36EzJUu8WOPIiA.png" /><figcaption>Object Detection, Instance Segmentation, and Panoptic Segmentation</figcaption></figure><p>But, the docs are awful, and it doesn’t yet seem to have the same community built around it as many other popular open-source tools.</p><p>You can spend weeks trying to custom implement functionality before realising it was there, perfectly implemented all along, just without a single mention in the docs.</p><p>An additional complexity, is that the <strong>MMStack </strong>has a lot of components, <strong>MMEngine,</strong> <strong>MMCV</strong>, and <strong>MMDetection</strong>, and it’s not always obvious where different functionality will be implemented, and so it’s hard to know where to look.</p><p>This is a rushed article but I still think it could save people a lot of time</p><h3>The Config System</h3><p>MMDetection is extremely config heavy, the framework takes a config.py style file, full of python dictionaries, and builds the data_loader, augmentations, model etc. from it. It looks a little like this:</p><pre>cfg = Config.fromfile(args.config)<br>runner = Runner.from_cfg(cfg)</pre><p>If (like me), you don’t want to need to clone the MMDetection repo to get started, and just want to use it as an API, you can instead take the below approach.</p><pre>from mmengine.hub import get_config<br><br>cfg = get_config(&#39;mmdet::swin/mask-rcnn_swin-t-p4-w7_fpn_amp-ms-crop-3x_coco.py&#39;)<br>runner = Runner.from_cfg(cfg)</pre><p>Both of these are creating all of the below objects (this is taken from the MMEngine docs). <a href="https://mmengine.readthedocs.io/en/latest/examples/train_seg.html">https://mmengine.readthedocs.io/en/latest/examples/train_seg.html</a></p><pre>num_classes = 32  # Modify to actual number of categories.<br><br>runner = Runner(<br>    model=MMDeeplabV3(num_classes),<br>    work_dir=&#39;./work_dir&#39;,<br>    train_dataloader=train_dataloader,<br>    optim_wrapper=dict(<br>        type=AmpOptimWrapper, optimizer=dict(type=AdamW, lr=2e-4)),<br>    train_cfg=dict(by_epoch=True, max_epochs=10, val_interval=10),<br>    val_dataloader=val_dataloader,<br>    val_cfg=dict(),<br>    val_evaluator=dict(type=IoU),<br>    custom_hooks=[SegVisHook(&#39;data/CamVid&#39;)],<br>    default_hooks=dict(checkpoint=dict(type=&#39;CheckpointHook&#39;, interval=1)),<br>)<br>runner.train()</pre><p>The docs also lack emphasis on how to neatly train on any dataset, and instead are better built for replicating papers on COCO. The only thing that needs to be editted to train on your own custom dataset, is the config for the dataloader.</p><p>The full config looks like the below, it expects three variables</p><ul><li>data_root (both the other paths are relative to this.</li><li>data_prefix (where the images are)</li><li>ann_file (path to the COCO annotations).</li></ul><p>Example functional values would be</p><pre>data_root = &lt;some-dataset-name&gt;<br>ann_file = &quot;annotations/train.json&quot;<br>data_prefix = dict(img=&quot;data/&quot;)</pre><p>Where the true paths are:</p><pre>data_root = &lt;some-dataset-name&gt;<br>ann_file = &lt;some-dataset-name&gt;/annotations/train.json<br>data_prefix = &lt;some-dataset-name&gt;/data</pre><pre>dict(<br>    batch_size=batch_size,<br>    num_workers=6,<br>    persistent_workers=True,<br>    sampler=sampler,<br>    batch_sampler=None,<br>    dataset=dict(<br>        type=&quot;CocoDataset&quot;,<br>        data_root=self.data_root, <br>        ann_file=ann_file,<br>        data_prefix=dict(img=&quot;data/&quot;),<br>        filter_cfg=dict(filter_empty_gt=filter_empty_gt),<br>        pipeline=pipeline,<br>        metainfo=self.metainfo,<br>    ),<br>)</pre><h3>How to Train a Model</h3><p>I tend to use typer instead of python’s argparse for command line scripts, more or less the same functionality, just a little less boiler plate.</p><pre>import typer<br>from mmengine.hub import get_config<br><br># This is just needed to fix the num_classes in a model <br># You&#39;d really hope that there&#39;s a better approach <br># But I&#39;m not sure, and have yet to see one. <br># After a good bit of looking.<br>def replace_nested_value(dictionary, key_to_replace, new_value):<br>    if isinstance(dictionary, dict):<br>        for key, value in dictionary.items():<br>            if key == key_to_replace:<br>                dictionary[key] = new_value<br>            elif isinstance(value, dict):<br>                replace_nested_value(value, key_to_replace, new_value)<br>            elif isinstance(value, list):<br>                for item in value:<br>                    replace_nested_value(item, key_to_replace, new_value)<br><br>def set_num_frozen_stages(model: dict, frozen_stages: int): <br>    model[&#39;backbone&#39;][&#39;frozen_stages&#39;] = frozen_stages<br>    return model<br><br>def main(<br>    data_root: str = &#39;../data&#39;,<br>    data_prefix: str = &#39;images&#39;,<br>    training_path: str = &#39;annotations/train.json&#39;,<br>    validation_path: str = &#39;annotations/validation.json&#39;<br>    model_architecture_family: str = &#39;swin&#39;,<br>    training_config: str = &quot;mask-rcnn_swin-t-p4-w7_fpn_amp-ms-crop-3x_coco.py&quot;,<br>    num_classes: int = 10,<br>    output_directory: str = &#39;work_dir&#39;,<br>    frozen_stages: int = 3,<br>):<br>    <br>    # Retrieves the model, its transformations, the corresponding<br>    # Optimizer etc.<br>    cfg = get_config(f&#39;mmdet::{model_architecture_family}/{training_config}&#39;)<br><br>    # Set the number of frozen stages in the model<br>    cfg.model = set_num_frozen_stages(cfg.model, frozen_stages)<br><br>    # Where the config file gets dumped to after it&#39;s been <br>    # Built to simplify reproducing the experiment<br>    <br>    if not os.path.exists(output_directory):<br>        os.mkdir(output_directory)<br><br>    cfg.work_dir = output_directory<br><br>    # Just overwriting the config dictionary values<br>    # The config object is very similar to a python<br>    # dictionary with a few small differences<br>    cfg.train_dataloader.dataset.data_root = data_root<br>    cfg.train_dataloader.dataset.ann_file = training_path<br>    cfg.train_dataloader.dataset.data_prefix = dict(img=data_prefix)<br><br>    cfg.val_dataloader.dataset.data_root = data_root<br>    cfg.val_dataloader.dataset.ann_file = training_path<br>    cfg.val_dataloader.dataset.data_prefix = dict(img=data_prefix)<br><br>    # Change the number of classes in the model<br>    # Often needs to modify an integer in multiple heads<br>    # e.g in both the Mask Head and a BBox Head <br>    # To cover all architectures, this needs to be generic<br>    # Wouldn&#39;t be surprising if there&#39;s a neater way to achieve<br>    # This with built in tooling, but I haven&#39;t found it yet<br>    replace_nested_value(cfg.model, &#39;num_classes&#39;, num_classes)<br><br>    runner = Runner.from_cfg(cfg)<br>    runner.train()<br><br>if __name__ == &#39;__main__&#39;:<br>   typer.run(main)</pre><h3>How to Run Inference</h3><p>The inference API is also awfully tucked away, and under documented.</p><pre>from mmdet.apis import DetInferencer<br><br>inferencer = DetInferencer(<br>    model=config_path,<br>    weights=weights_path,<br>)<br><br>results = inferencer(<br>    file_paths, # list of file_paths<br>    return_vis=False,<br>    batch_size=batch_size,<br>    no_save_pred=True,<br>    pred_score_thr=threshold,<br>    no_save_vis=True,<br>    out_dir=&quot;outputs&quot;,<br>)</pre><p>There’s a great list of all of the available config values here <a href="https://mmdetection.readthedocs.io/en/dev-3.x/user_guides/config.html">https://mmdetection.readthedocs.io/en/dev-3.x/user_guides/config.html</a></p><p>Please message me if you try any of this code and have any problems. I have written this quickly out of frustration and the knowledge that I wouldn’t write anything if I didn’t write quickly.</p><p>Error messages from MMDetection tend to be cryptic at best. You have been warned.</p><p>Let me know if you want more MMDetection tutorials and reach out directly with any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ab6a6fea3f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Engine Design]]></title>
            <link>https://medium.com/@george.pearse/data-engine-design-9b29a20ff9f0?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b29a20ff9f0</guid>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engine]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Tue, 20 Sep 2022 18:02:39 GMT</pubDate>
            <atom:updated>2023-06-08T16:13:30.978Z</atom:updated>
            <content:encoded><![CDATA[<p>Breaking down the systems of the top ML companies (with a strong Computer Vision focus).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*WLSb5hOdX48bmljqJzVqkw.png" /><figcaption>a big engine</figcaption></figure><p>Tesla created the term “Data Engine” to describe what is more generally known as a “Data Flywheel”. I’d summarise a “Data Flywheel” as:</p><blockquote>A system which uses model outputs to improve training datasets</blockquote><p>These training datasets can then be used to create better models, and so the ‘flywheel’ spins. A Data Engine has 3 main components or modules:</p><ul><li><strong>Active Learning</strong> — selecting the most valuable datapoints to label.</li><li><strong>Dataset Cleaning</strong> — ensuring that your Ground Truth labels are accurate and self-consistent.</li><li><strong>Auto-labelling </strong>— using your model predictions as a starting point for training labels. Instead of starting from scratch in your labelling tool, just correct the predictions of your current best model.</li></ul><p>Let’s dive into each in more depth.</p><h3>Active Learning</h3><p>Active Learning is the the problem of finding the most valuable datapoints to label, where value is defined by the per-datapoint model improvement against some ML metric. When you’re working on improving a weak baseline, the easiest instances may be the most valuable, but as your model improves, it quickly becomes the case that the hardest instances contain the most useful information to the model. Human intuition wrt. learning works perfectly, the only adaptation being the sheer volume of data required to teach an ML model. Meta recently released an excellent paper on “Data Pruning” which explains these concepts in great depth.</p><p><a href="https://arxiv.org/abs/2206.14486">https://arxiv.org/abs/2206.14486</a></p><p>See BAAL to get started.</p><p><a href="https://github.com/baal-org/baal">GitHub - baal-org/baal: Library to enable Bayesian active learning in your research or labeling work.</a></p><h3>Dataset Cleaning</h3><p>Dataset cleaning is the art of iteratively improving the quality of your ground truth labels. Your dataset probably suffers from two problems - random annotation mistakes and bias. In the case of random mistakes, it can be remedied by a very large dataset, but this is often wasteful (both of human time and GPU resource) and unnecessary.</p><p>The simplest approach is to stop training a model early, before it’s memorised the training set, and then use this model to run inference back over its own training set. The frames with the largest disagreement between the original labels and the model’s predictions are likely to include incorrect annotations. Send the top X of these to your labelling tool, and correct where appropriate.</p><p>See cleanlab to get started.</p><p><a href="https://github.com/cleanlab/cleanlab">GitHub - cleanlab/cleanlab: The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.</a></p><h3>Auto-Labelling</h3><p>Most annotation tools now make it very easy to send your model’s predictions as a starting point for annotation. This is particularly valuable for object detection, where the most valuable human input may be to correct one wrong prediction in a frame that includes one hundred correct predictions.</p><p>A particularly popular combination is to use Voxel51 and CVAT to support such a workflow. See Voxel’s docs for more detail.</p><p><a href="https://docs.voxel51.com/user_guide/annotation.html">Annotating Datasets - FiftyOne 0.21.0 documentation</a></p><h3>Conclusion</h3><p>This set-up is increasingly becoming available ‘off-the-shelf’. Companies like V7 labs and Scale AI provide much of the tooling to get you there, with only the help of a large pay check. In reality however, 90% of the gains can be achieved through a pretty simple set-up built around open-source tools like cleanlab, CVAT, Voxel Fiftyone and maybe the odd Streamlit app. Where the “paid for services” distinguish themselves is in environments where you must juggle and update many production models.</p><p>Make your Data Engine look more like a Tesla’s and less like a Fiat’s.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b29a20ff9f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick NLP Labelling with Bulk]]></title>
            <link>https://medium.com/@george.pearse/quick-nlp-labelling-with-bulk-327ccb62320?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/327ccb62320</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Sun, 18 Sep 2022 21:06:14 GMT</pubDate>
            <atom:updated>2022-09-19T10:46:46.324Z</atom:updated>
            <content:encoded><![CDATA[<p>An NLP POC in an afternoon.</p><blockquote>So you have an internal free-text dataset, but you don’t have labels, and you don’t know how easy it will be to make them, enter bulk.</blockquote><p><a href="https://github.com/koaning/bulk">GitHub - koaning/bulk: A Simple Bulk Labelling Tool</a></p><p>A great way to estimate how difficult a Machine Learning project will be, is to see if the clusters align with how you’d want to break down the classes. Or for a binary problem, if the class boundary you’d be interested in already at least partially aligns with a single cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*qg_RVh_2HydhMeQuGUN0PA.png" /><figcaption>clustering in bulk.</figcaption></figure><p>Bulk stands out for the quality of its EDA (exploratory data analysis), the scatter chart is made with bokeh and enables arbitrary zooming in and out. To go from a highly granular view, to the dataset ‘from 1 million miles out’. You can also colour by any set of keywords.</p><p>To colour by keywords you specify them from the terminal when running the program:</p><pre>python -m bulk text ready.csv --keywords &quot;F1,hair,sun,fight,rain&quot;</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.loom.com%2Fembed%2Fe6fbf050987c47d189805e2e02847c4a&amp;display_name=Loom&amp;url=https%3A%2F%2Fwww.loom.com%2Fshare%2Fe6fbf050987c47d189805e2e02847c4a&amp;image=https%3A%2F%2Fcdn.loom.com%2Fsessions%2Fthumbnails%2Fe6fbf050987c47d189805e2e02847c4a-1663534275806.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=loom" width="1152" height="864" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e5de2ec38b0ed66ca47d51f01ea137e/href">https://medium.com/media/2e5de2ec38b0ed66ca47d51f01ea137e/href</a></iframe><p>It differentiates itself from equivalent tools by the ability to then apply quick labels. Taking you out of EDA and into action. It is un-opinionated, meaning you can apply your own choice of dimensionality reduction (umap, t-sne, take your pick).</p><p><a href="https://arize.com/blog/t-sne-vs-umap/">SNE vs. t-SNE vs. UMAP: An Evolutionary Guide</a></p><p>This design choice also makes it much more performant than some equivalent tools like the tensorboard embedding projector. The browser only has to receive 2 dimensions worth of data (x and y coordinates), instead of the several thousand of the original vector, greatly reducing the required data transfer from server to client/browser.</p><p><a href="https://projector.tensorflow.org/">Embedding projector - visualization of high-dimensional data</a></p><p>After highlighting a section of the chart, you can export a named csv (named with the label) store of those records, ready to use to train a model.</p><h4>The Dataset</h4><p>I’ve chosen an open-source twitter dataset (from around 2009). The 30k sample (post dimensionality reduction, with raw tweets) used in the demo is available at:</p><p><a href="https://github.com/GeorgePearse/bulk-demo/blob/main/ready">https://github.com/GeorgePearse/bulk-demo/blob/main/ready</a> (1).csv</p><p>The biggest advantage of this dataset is that there’s a natural limit to the size of the free text in tweets. Short snippets are where the simple UI of bulk really shines.</p><h3>The Code</h3><p>The below code uses the SentenceTransformer API which is written on top of Hugging Face. I find Hugging Face unreasonably awkward to use, but this thin API is great for NLP POCs.</p><p><a href="https://www.sbert.net/">SentenceTransformers Documentation - Sentence-Transformers documentation</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9773d24e520e179d80c0edc46af291b8/href">https://medium.com/media/9773d24e520e179d80c0edc46af291b8/href</a></iframe><h3>The Pre-Trained Model</h3><p>Plucked from the sbert website, which will have plugged it from hugging-face.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZpdLpBWMsrf72-K32nCf6Q.png" /><figcaption>all-mpnet-base-v2 model card.</figcaption></figure><p>Huge credit to Vincent Warmerdam for creating bulk and a set of other well-defined, clever tools. Check out his GitHub at <a href="https://github.com/koaning">https://github.com/koaning</a>.</p><p>Please clap and follow if this content interests you. I’ll be trying to focus on longer form content and projects over the next few months. Closing the gap between hobby project and production application.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=327ccb62320" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vector Databases for Data-Centric AI (Part 2)]]></title>
            <link>https://medium.com/@george.pearse/vector-databases-for-data-centric-ai-part-2-ba995053ce05?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba995053ce05</guid>
            <category><![CDATA[data-centric-ai]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Fri, 26 Aug 2022 11:37:24 GMT</pubDate>
            <atom:updated>2022-08-27T09:56:30.941Z</atom:updated>
            <content:encoded><![CDATA[<p>Building applications with QDrant, Hugging-Face and Streamlit.</p><p>QDrant have created an excellent vector database and I suspect ML Engineers are only beginning to scratch its potential applications.</p><p>Vector databases support hybrid similarity search and provide a CRUD API to run updates to your datasets. They are a significant improvement upon first-wave Approximate Nearest Neighbour tools like Faiss and Annoy which enable very high performance in-memory vector search but little in the way of support for update flows, nor metadata filters.</p><p>I’ve already written about some of the applications of these tools here.</p><p><a href="https://medium.com/mlearning-ai/the-many-uses-of-a-vector-database-65cb6cc70b3f">Vector Databases for Data-Centric AI</a></p><blockquote><strong>Hybrid search</strong> is <strong>vector or “semantic” </strong>search combined with attribute filtering.</blockquote><p>The semantic search implemented by QDrant requires a list of positive and negative examples. Each positive datapoint is an example of what you want the responses to be similar to, each negative datapoint is an example of what you want the responses to be different to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*wPXZhyfKUE4iq3FoOMgbOA.png" /></figure><p>This allows you to build up arbitrarily complex decision boundaries within your feature space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*_oKEGDHUX9MX2ztNS1cXuQ.png" /></figure><p>An example QDrant query:</p><pre>{&quot;positive&quot;: [0],    <br>&quot;negative&quot;: [1],    <br>&quot;top&quot;: 10,   <br>&quot;with_payload&quot;: true}</pre><p><strong>This enables the interactive definition of classes:</strong></p><ul><li>Start with a single positive datapoint.</li><li>Look through the responses.</li><li>Add those that you consider to be similar to the list of positives</li><li>Add those you consider to be different to the list of negatives</li><li>Run that new query, and repeat.</li></ul><p>After a batch of labelling you would also be in a position to improve your embeddings and continue the process with a <em>‘better’ </em>separated dataset (I’ll be experimenting more with this next).</p><p>I’ve built a mini Streamlit application to support this flow and enable you to save each query once complete along with a CSV containing its results.</p><h3>QDrant-NLP Demo</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F743387784%3Fh%3Ddcc33ff258%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F743387784&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1495506897-3399cfee094db538033e8187c3639e079540c00850e7c532627ead8dd1a864b1-d_1280&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1200" frameborder="0" scrolling="no"><a href="https://medium.com/media/26baf28142ed0730f13cc7fc68176b9e/href">https://medium.com/media/26baf28142ed0730f13cc7fc68176b9e/href</a></iframe><h3>How to Run</h3><p><a href="https://github.com/GeorgePearse/QDrant-NLP">GitHub - GeorgePearse/QDrant-NLP: QDrant-NLP</a></p><p>Just clone the repo and run:</p><pre>docker-compose up</pre><p>I would like to increase the number of datasets this can be tried on, either with GPU backed lambda functions or by saving many example datasets to S3. So far I’ve only made a 6K subset of ag_news available. <a href="https://huggingface.co/datasets/ag_news">ag_news · Datasets at Hugging Face</a></p><p>This is the code snippet used to generate the embeddings via hugging-face:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cd3ec8a1dceba5b4058319be653e0cf/href">https://medium.com/media/0cd3ec8a1dceba5b4058319be653e0cf/href</a></iframe><h3>Where to Use</h3><p>Shout out to both Kern.AI (an excellent open-source NLP labelling tool)</p><p><a href="https://github.com/code-kern-ai/refinery">https://github.com/code-kern-ai/refinery</a></p><p>and Voxel51 (an excellent open-source Computer Vision analysis tool)</p><p><a href="https://github.com/voxel51/fiftyone">https://github.com/voxel51/fiftyone</a></p><p>for being early adopters of the technology in their platforms, but I don’t believe either have yet made use of all of the value it can provide.</p><h3>Other Reads in Data-Centric AI</h3><p><a href="https://eric-landau.medium.com/introduction-to-micro-models-or-how-i-learned-to-stop-worrying-and-love-overfitting-fd8fbe98e99b">Introduction to micro-models or: how I learned to stop worrying and love overfitting | by Eric Landau | Medium</a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba995053ce05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Floating on Open-Source]]></title>
            <link>https://medium.com/@george.pearse/floating-on-open-source-bd6ff3817afb?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd6ff3817afb</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Fri, 26 Aug 2022 09:32:53 GMT</pubDate>
            <atom:updated>2022-08-27T09:55:41.108Z</atom:updated>
            <content:encoded><![CDATA[<p>Many Machine Learning businesses are really just a tech team floating on a pile of open-source projects combined with a large, and expensive dataset. The amount of original technical work is low (and should be) and mostly involves bringing a few different APIs together and ensuring that they play nicely.</p><ul><li>Model Serving — (open-source e.g. TorchServe, FastAPI, Bento etc.)</li><li>Model Training — (Pytorch-Lightning, TensorFlow, Composer etc.)</li><li>Experiment Tracking — (MLFlow, TensorBoard etc.)</li></ul><blockquote>Open-source projects come and go, but you must ride the wave.</blockquote><p>The reality of this should be fully embraced. Your core responsibility is to pick the right tools, know how heavily you can afford to commit your stack to each, and know when to move on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cl3-VO9_NwgsD36f65eg7A.png" /><figcaption>Riding on Open-Source</figcaption></figure><p>This does have some technical consequences. Certain design patterns favour the ability to quickly chop-and-change the underlying implementation. The core to these techniques though is really just to avoid writing too much code that is specific to an API choice that may not last. Laszlo Sragner has already written well on the topic.</p><p><a href="https://laszlo.substack.com/p/you-only-need-2-design-patterns-to">You only need 2 Design Patterns to improve the quality of your code in a data science project (substack.com)</a></p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd6ff3817afb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick Business Intelligence with Streamlit and SQL]]></title>
            <link>https://medium.com/@george.pearse/quick-business-intelligence-with-streamlit-and-sql-cd6a9ba8a48f?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd6a9ba8a48f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Sat, 20 Aug 2022 13:59:44 GMT</pubDate>
            <atom:updated>2022-08-20T18:22:10.474Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quick Business Intelligence with Streamlit, SQL and Plotly</h3><p>Business Intelligence so simple you can edit the source code yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*fY_uq_quhywmYwl2uHaZog.png" /><figcaption>Streamlit and Plotly. A match made in heaven.</figcaption></figure><p>As soon as you have a few customers seriously using your product you need to get started with your analytics. Many would be tempted to start with a full blown Superset and Snowflake Modern Data Stack. Often this is a great solution, often it’s overkill.</p><p>If you’re analysing medium to large datasets it’s much faster to perform your transformations in SQL to minimise the amount of data that needs to be transferred over the network.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F741402075%3Fh%3D56271aaadf%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F741402075&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1491785838-7c19f6969b519ea160938387bbae07f20024704892685d65ca44d4b5b5052065-d_1280&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a78c5543ab42b09367ffeeef091ba260/href">https://medium.com/media/a78c5543ab42b09367ffeeef091ba260/href</a></iframe><p>And with the development of tools like DBT and Dagster that make scheduling transformations extremely easy, there’s no need for heavy or complex transformation and caching layers in your BI tool. DBT can manage the transformations and an orchestration layer can frequently materialise data in the structure it’s needed for graphing to keep things performant.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b737b6554acfcd9bd95a5b5910aff15d/href">https://medium.com/media/b737b6554acfcd9bd95a5b5910aff15d/href</a></iframe><p>The vast majority of plots just have an X axis value and a Y axis value, I would be tempted to simplify the interface so that the first column is always the X input and the second always the Y input.</p><p>Be warned, the happy path for this app is narrow. Building more and more into it and writing up each step as I go. Once it becomes a tidy and simple solution I’ll make it available on Docker Hub and pypi as the quickest route to effective and efficient business intelligence.</p><blockquote>The only low-code Business Intelligence solution on PyPi (maybe)</blockquote><p>I plan to add ways to save queries, save chart config, integrate DBT, connect to Snowflake, and access the results of your saved queries via API endpoints.</p><p>If this work interests you please clap and follow. I’m going to continue to create tooling that is easy to tweak and play with across Analytics and Machine Learning over the next few months.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd6a9ba8a48f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Start Learning SQL with Streamlit and SQLite]]></title>
            <link>https://medium.com/@george.pearse/how-to-start-learning-sql-with-streamlit-d3edad7494cd?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3edad7494cd</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Wed, 17 Aug 2022 06:33:25 GMT</pubDate>
            <atom:updated>2022-08-21T08:47:18.002Z</atom:updated>
            <content:encoded><![CDATA[<p>Learning SQL is easier than starting to learn SQL. Data Engineering with Streamlit and SQLite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*msyFliA6a5yI3HXmAeTWCA.png" /><figcaption>Streamlit and SQLite, a match made in heaven. Image by Author</figcaption></figure><p>For beginners trying to get into the data industry learning SQL is the obvious first step. This is easier said than done. While the statements that you need to know and understand are extremely simple. Database terminology is confusing and setting up a database is a piece of work you’re unlikely to be doing as a junior.</p><p>A more advanced solution is written up here:</p><p><a href="https://medium.com/me/stats/post/cd6a9ba8a48f">Stats for Quick Business Intelligence with Streamlit and SQL (medium.com)</a></p><h3>SQLite</h3><p>SQLite is the simplest DB you can get started with. It creates a local file on your laptop, against which you can run almost all the same queries that you would against a PostgresDB. Creating a connection simultaneously creates the DB and when running this demo you will see this file appear in the same directory.</p><h3>Streamlit</h3><p>If you’re trying to break into data you’ll also probably be learning python. Pandas and its table object, a DataFrame are synonymous with both Data Engineering and Data Science. In this demo pandas is only used to read in the CSV and store the data between that state and SQLite.</p><p>Streamlit is the fastest way to create UIs with python. Its greatest for one-directional workflows. It’s recent, improved multi-page set-up makes it extremely quick to put together a few, simple, interacting flows.</p><p>Streamlit could not rank more highly against the metrics I use to recommend frameworks for adoption.</p><p><a href="https://medium.com/p/a4cd273e0e18">A Framework for Picking Frameworks | by George Pearse | Aug, 2022 | Medium</a></p><h3>The Streamlit Code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8493e912f05a627fafc80f58cc77755/href">https://medium.com/media/b8493e912f05a627fafc80f58cc77755/href</a></iframe><h3><strong>The Python Packages</strong></h3><ul><li>sqlite3 and os are python in-built packages, they don’t need to be installed.</li><li>Both pandas and streamlit must be ‘pip installed’. Other blogs will already explain how to get started with this in detail.</li></ul><h3>Steps to Run</h3><p><strong>From the same folder as the code above run:</strong></p><pre>streamlit run streamlit_sqlite_demo.py</pre><p><strong>Click on the localhost link:</strong></p><p>This should take you to a webpage where you’ll now have 3 pages, navigable via the sidepanel which will enable you to create a SQLite database, upload a CSV, and then run SQL queries against it. Your page may have a different colour scheme, this can be changed via the settings button at the top right of the website.</p><p><strong>1.Pick a name for your database (end in .db) and click “Create database”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rFKDxVhXeNA1xkpkQU9SIw.png" /><figcaption>create database. Image by Author.</figcaption></figure><p><strong>2.Find a CSV to upload, and give the table a name.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hxxX1SW3IJ-bAhliD9nVWg.png" /><figcaption>upload data. Image by Author</figcaption></figure><p><a href="https://www.data.gov.uk/">https://www.data.gov.uk/</a> has a good store of CSVs to upload and run queries against. If you’re struggling to find an example (file must only have one row of headers) the below works, download the CSV named:</p><p><a href="https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/930473/_MoD__Senior_Officials_Expenses__Report_Q1_20-21.csv">MOD’s Senior Officials’ Expenses (2 Star and above) April to June 2020</a></p><p>from</p><p><a href="https://www.data.gov.uk/dataset/2dd8b18f-8fe6-44c8-8a20-2b92d16f7191/mod-senior-officials-business-expenses-hospitality-and-meetings-january-to-december-2020">MOD: senior officials’ business expenses, hospitality and meetings, January to December 2020 — data.gov.uk</a></p><p><strong>3.Select a DB that you’ve created and run a SQL query.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1g3ENg_h_zEsLlejKvMfyQ.png" /><figcaption>run query. Image by Author</figcaption></figure><pre>select * from &lt;table-name&gt;</pre><p>Is a good place to start.</p><p>This article does an excellent job of explaining how to publish your Streamlit App to Heroku.</p><p><a href="https://towardsdatascience.com/a-quick-tutorial-on-how-to-deploy-your-streamlit-app-to-heroku-874e1250dadd">A quick tutorial on how to deploy your Streamlit app to Heroku. | by Navid Mashinchi | Towards Data Science</a></p><p>The application shown in the screenshots is currently available</p><p><a href="https://georgepearse-sqlite-streamlit-app-app-l4kyr3.streamlitapp.com/">Streamlit (streamlitapp.com)</a></p><p>Hosted via Streamlit cloud.</p><p>If you find this interesting or helpful, please clap and follow. Let me know your thoughts, any questions, and all problems. Thanks!</p><p>Also looking for good publications to use. If you’re an editor and this content matches your reader’s interests please get in touch.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3edad7494cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Framework for Picking Frameworks]]></title>
            <link>https://medium.com/@george.pearse/a-framework-for-picking-frameworks-a4cd273e0e18?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4cd273e0e18</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 14:15:57 GMT</pubDate>
            <atom:updated>2022-08-17T08:07:18.908Z</atom:updated>
            <content:encoded><![CDATA[<p>How to choose your next Open-Source tool. 5 things to consider.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*ATAs39O0d3BZPxLsGeEqwQ.png" /><figcaption>choice making temp placeholder. image by author.</figcaption></figure><p>With the rise of the Modern Data Stack, images like this (below) have become commonplace. I would recommend a view of the original high-res image and article (link in the caption).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zwi8YvEyEorfp4nmBDFUog.png" /><figcaption>Serious credits to Matt Turck. <a href="https://mattturck.com/data2019/">A Turbulent Year: The 2019 Data &amp; AI Landscape — Matt Turck</a></figcaption></figure><p>What’s obvious is that it’s a congested space. The problem as a professional in the field is to know where to focus your attention. When each tool requires deployment to k8s how do you decide where to start?</p><p><strong>1.Time to POC</strong></p><p>With so many tools out there what really matters is how quickly you can come to an informed decision with respect to whether the tool will add value to your team. What matters here is the time to a POC, in this POC you’ll probably either realise it’s not for you, or reach the point where you want to convince your manager to take it further. It’s the MVP of demos.</p><p>For a pip installable package or a docker-compose service, the set-up process can be almost instant. A k8s helm chart can take significantly longer to set-up as you would actually use it in your team.</p><p><strong>2.Time to Value</strong></p><p>Once you pick a tool how long will it be until you’re moving faster than whatever you were using before. This is a combination of the learning curve and the level of efficiency that it adds. For a simple tool like DBT this time is very short, for a more complex tool like kubeflow it’s likely on the scale of months. This is highly dependent on how much of a change to workflows it causes.</p><p><strong>3.Size of Value</strong></p><p>Once you’re over the learning curve how much will you benefit? Will you consistently move faster across multiple projects. Will you be able to deliver higher quality products as a result?</p><p><strong>4.Quality and Momentum of the Development Team</strong></p><p>This is a factor I wouldn’t have considered a year ago, but that I take very seriously now. Some open-source packages and tools dissolve and disappear as fast as they appear. The speed of response to feedback and pull requests also varies wildly. If you see an active slack channel it’s a good sign. Commercial backing, even better. QDrant were excellent for me here, responsive and professional from the get go.</p><p><strong>5.How much will it teach you?</strong></p><p>Bit of a selfish one but if you’re planning in staying in a role for 2+ years your personal growth also matters to your employer. Getting heavily involved with a high growth package or tool can help you to grow as an ML Engineer or Software Engineer. Requesting features and reading the code required to construct the solutions (or contributing it and receiving feedbacks) is a great way to learn.</p><p>Similarly, some frameworks are highly opinionated. My decision to adopt Dagster taught me a lot about about Data Engineering principles in general because it is designed in a way that guides you towards a sustainable solution.</p><p>Others have absolutely excellent documentation. FastAPI really is in a category of its own in this respect.</p><p>Let me know if you find this interesting. Please follow if you’d like to read more similar content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4cd273e0e18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Active Learning for Deep Learning]]></title>
            <link>https://medium.com/@george.pearse/active-learning-for-deep-learning-312a1657c4c1?source=rss-b502c2f08a7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/312a1657c4c1</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[George Pearse]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 08:55:38 GMT</pubDate>
            <atom:updated>2022-08-20T19:08:38.071Z</atom:updated>
            <content:encoded><![CDATA[<p>How to choose what datapoints to label.</p><p>Data Centric AI is suddenly all the rage, big names in the ML community like Angrew Ng and James Zhou (Stanford) have come forward to try to end modelitis (throwing the latest and greatest model architecture at a problem). They’re pushing for a move towards a more sustainable and systematic approach to improving the performance of ML systems.</p><p>Yet very little is often said about the exact techniques that can be applied.</p><p>There are three main components to Data Centric AI:</p><ul><li><strong>Highly granular evaluation sets</strong> focused on a specific problem. In the context of Computer Vision this can be achieved with nearest neighbours over embeddings either with torch.cdist, annoy or faiss depending on the dataset size.</li></ul><p><a href="https://medium.com/mlearning-ai/the-many-uses-of-a-vector-database-65cb6cc70b3f">Vector Databases for Data-Centric AI</a></p><ul><li><strong>Dataset Cleaning.</strong> The removal of instances that are mislabelled or out of the relevant distribution. Often involves <strong><em>Data Valuation</em></strong> techniques, where many permutations of the dataset are sampled within the training set in order to determine their value for a specific task. Examples include Leave-One-Out (LOO), approximations of Data Shapley Values and Reinforcement Learning for Data Valuation.</li></ul><p><a href="https://ai.googleblog.com/2020/10/estimating-impact-of-training-data-with.html">Estimating the Impact of Training Data with Reinforcement Learning</a></p><ul><li><strong>Active Learning.</strong> The task of selecting which data-points to label in order to maximise the per data-point model improvement.</li></ul><p>This article will focus on Active Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6a5Lf7sB_B0xt4jmc4B5dw.png" /><figcaption><a href="https://product.hubspot.com/blog/bayesian-active-learning">https://product.hubspot.com/blog/bayesian-active-learning</a></figcaption></figure><p>Active Learning is most useful when labelling is expensive e.g. requires experts in the domain, and there’s a large pool of unlabelled instances to choose from.</p><p>Three components can be mixed and matched in Active Learning:</p><ul><li>Selecting data-points with a high model ‘uncertainty’.</li><li>Selecting data-points to label in order to be representative of the full set.</li><li>Selecting data-points in order to maximize diversity.</li></ul><p>The latter two may sound similar but representative sampling should be thought of as matching the distribution of the full population (labelled and unlabelled alike), while diversity sampling focuses on maximising the coverage of a given <strong>latent space</strong> (outliers are highly ranked by such a system).</p><blockquote>“A latent space, also known as a latent feature space or embedding space, is <strong>an embedding of a set of items within a manifold in which items which resemble each other more closely are positioned closer to one another in</strong> the latent space.”</blockquote><p>Due to the long training times of Deep Neural Networks and the insignificance of a single datapoint to the behaviour of a model, Active Learning tends to be applied in contexts in which a batch of data is submitted for labelling. This increases the importance of the diversity component. Uncertainty sampling techniques applied in batch form are likely to ‘oversolve’ a specific problem. For a model designed to detect bone fractures that is currently particularly weak at identifying wrist fractures, a model uncertainty based technique may only select wrist fractures, even if only a small number of examples is sufficient to correct for the problem.</p><p>It’s like a student failing a mock exam partially due to a long division problem only studying long division from then on to the detriment of any other weaknesses. <a href="https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html">https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html</a> This article excellently explains this problem and shows an uncertainty based technique being outperformed by random selection in the batch setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKn5PsSZhvvWtM1YqD-e9g.png" /><figcaption>Random selection outperforms BALD when selecting data-points in batch. <a href="https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html">https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html</a></figcaption></figure><p>The best techniques in the batch setting tend to be a combination of uncertainty sampling and diversity sampling. If you need to apply Active Learning to your own problem. The most mature package for Active Learning in the context of Deep Learning is BAAL.</p><p><a href="https://github.com/baal-org/baal">GitHub - baal-org/baal: Library to enable Bayesian active learning in your research or labeling work.</a></p><p>This is a very active area and new tools are popping up frequently. You can check <a href="https://github.com/stars/GeorgePearse/lists/active-learning">https://github.com/stars/GeorgePearse/lists/active-learning</a> for a list of repos worth a further look in this area.</p><p>Let me know your thoughts. Please click follow if the content interests you. I’m currently looking for my next role.</p><h3>References</h3><ul><li><a href="https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html">https://oatml.cs.ox.ac.uk/blog/2019/06/24/batchbald.html</a></li></ul><h3>Further Reading</h3><ul><li><a href="https://towardsdatascience.com/advanced-active-learning-cheatsheet-d6710cba7667">https://towardsdatascience.com/advanced-active-learning-cheatsheet-d6710cba7667</a>. Written by the author of Human-in-the-Loop Machine Learning (highly recommended).</li><li><a href="https://jacobgil.github.io/deeplearning/activelearning">https://jacobgil.github.io/deeplearning/activelearning</a>. Excellent blog post covering the latest and greatest of Active Learning along with the situations in which it’s most useful.</li><li><a href="https://devblog.pytorchlightning.ai/active-learning-made-simple-using-flash-and-baal-2216df6f872c">https://devblog.pytorchlightning.ai/active-learning-made-simple-using-flash-and-baal-2216df6f872c</a>. Written by the PyTorch Lightning team.</li><li><a href="https://medium.com/@ODSC/active-learning-your-models-new-personal-trainer-a89722c0db5a">https://medium.com/@ODSC/active-learning-your-models-new-personal-trainer-a89722c0db5a</a></li></ul><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=312a1657c4c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>