<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alberto Andreotti on Medium]]></title>
        <description><![CDATA[Stories by Alberto Andreotti on Medium]]></description>
        <link>https://medium.com/@albertoandreotti?source=rss-6c2287b0d455------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FCrsh5-89rt_LJGmmSLCBQ.jpeg</url>
            <title>Stories by Alberto Andreotti on Medium</title>
            <link>https://medium.com/@albertoandreotti?source=rss-6c2287b0d455------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:46:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@albertoandreotti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[De-identifying Pathology Slides at Scale: A Serverless AWS Batch Pipeline for SVS Whole-Slide…]]></title>
            <link>https://medium.com/john-snow-labs/de-identifying-pathology-slides-at-scale-a-serverless-aws-batch-pipeline-for-svs-whole-slide-dcdb11a49357?source=rss-6c2287b0d455------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcdb11a49357</guid>
            <category><![CDATA[hipaa]]></category>
            <category><![CDATA[pathology]]></category>
            <category><![CDATA[gdpr]]></category>
            <dc:creator><![CDATA[Alberto Andreotti]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 10:13:07 GMT</pubDate>
            <atom:updated>2026-09-08T10:13:07.974Z</atom:updated>
            <content:encoded><![CDATA[<h3>De-identifying Pathology Slides at Scale: A Serverless AWS Batch Pipeline for SVS Whole-Slide Images</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNd43FEirLdCjaNRR-3XJQ.png" /><figcaption>Synthetic pathology image. For illustration purposes only.</figcaption></figure><p>Digital pathology is generating data faster than most healthcare organizations can safely share it. A single whole-slide image (WSI) can weigh in at several gigabytes, and every one of them is a potential HIPAA or GDPR incident waiting to happen — not because of the tissue in the image, but because of the metadata, barcodes, and handwritten labels riding along with it.</p><p>In this post we’ll walk through a production-grade, serverless de-identification pipeline for <strong>SVS pathology slides</strong>, built on <strong>AWS Batch</strong> and <strong>John Snow Labs’ Visual NLP</strong>. We’ll cover the compliance problem it solves, the anatomy of an SVS file, the architecture we deploy, and a full hands-on walkthrough — from `docker build` to a redacted slide landing back in S3 — using the actual infrastructure code from <a href="https://github.com/JohnSnowLabs/visual-nlp-workshop/tree/master/products/awsbatch/svs">this repository.</a></p><h3>1. The de-identification problem in healthcare</h3><h4>What is PHI?</h4><p>Protected Health Information (PHI) is any data that can tie a medical record back to an identifiable person: name, date of birth, medical record number, accession number, physician name, facility name, a photographed ID sticker — even a barcode. In imaging workflows specifically, PHI has a nasty habit of hiding in places engineers don’t expect: embedded file metadata, a scanner’s auto-generated label image, or text burned directly into pixels.</p><h4>HIPAA</h4><p>In the US, the HIPAA Privacy Rule defines 18 categories of identifiers (the “Safe Harbor” list) that must be removed for data to be considered de-identified — names, geographic subdivisions smaller than a state, dates tied to an individual, medical record numbers, biometric identifiers, and full-face photographic images among them. Digital pathology slides can trip several of these at once: a label image is effectively a photograph, and it often contains a patient sticker. Also, the header of the file can contain tags like patient identifiers, technician’s name, institution, or even dates.</p><h4>GDPR</h4><p>In the EU, health data is a “special category” of personal data under Article 9 of GDPR, subject to stricter processing conditions than ordinary PII. Pseudonymization and anonymization are explicitly called out as risk-mitigation techniques, but GDPR’s bar for “anonymous” data (i.e., data that falls outside the regulation entirely) is high: re-identification must be reasonably impossible, not just inconvenient.</p><h4>Why you need to care</h4><p>Two things make this an engineering problem, not just a legal one:</p><ul><li>Scale. A single pathology lab can generate thousands of slides a week. Manual review doesn’t scale, and neither does a one-off script running on someone’s laptop.</li><li>PHI hides in unstructured places. Unlike a database column you can just drop, PHI in imaging data can be a photograph of a sticker, a watermark on a scanned label, or a free-text field a technician typed six years ago. Finding it requires actual computer vision and NLP.</li></ul><p>That’s the gap we will close with our pipeline: OCR + NER to <strong>find</strong> PHI in image content, and tags, applied automatically, at the moment data lands in S3.</p><h3>2. DICOM and SVS: where PHI actually lives</h3><p><a href="https://github.com/JohnSnowLabs/visual-nlp-workshop/tree/master/products/awsbatch">This repository</a> ships two parallel pipelines — `dicom/` and `svs/` — because DICOM and SVS files hide PHI in structurally different ways.</p><h4>DICOM</h4><p>In an extreme simplification, a DICOM file is a header of tagged attributes, followed by pixel data. The header is structured: `PatientName` is tag `(0010,0010)`, `PatientID` is `(0010,0020)`, `PatientBirthDate` is `(0010,0030)`, and so on. Many of these tags can be removed using rules, but some require NLP.</p><p>What is even harder is <strong>burned-in PHI</strong>: text baked directly into the pixel data by the imaging modality itself — ultrasound annotations, ID overlays on radiology exports, ruler and timestamp burn-ins. That text isn’t in a tag you can clear; it has to be <strong>found</strong> visually and then redacted. All that without corrupting the file’s encoding, and without destroying valuable information present in the file.</p><h4>SVS (Aperio whole-slide images)</h4><p>An SVS file is a <strong>pyramidal, multi-resolution tiled TIFF</strong>. It contains several encoded layers: the full-resolution tissue image, one or more downsampled levels for fast viewing, and — critically — a couple of small auxiliary images the scanner adds automatically:</p><ul><li><strong>Label image</strong> — a photo of the physical glass slide’s label, which is frequently a printed or handwritten patient/accession sticker.</li><li><strong>Macro image</strong> — a low-res overview of the entire slide including its edges, sometimes capturing handwriting on the slide itself.</li><li><strong>Header/metadata fields</strong> (e.g. `ImageDescription`) — free-text fields the scanner populates, which can carry accession numbers, operator names, or facility identifiers.</li></ul><p>In other words: SVS PHI is almost never in the tissue region itself — it’s in the small label/macro auxiliary images and the header text. That distinction is exactly what shapes the pipeline design in section 4: we clean the header, the auxiliary images, and only run the (expensive) OCR+NER stage on tiles that actually contain detected text.</p><h3>3. Introducing AWS Batch, and our architecture</h3><h4>Why AWS Batch</h4><p>Pathology de-identification is a bursty, <strong>resource-heavy batch workload</strong>, not a request/response service: files arrive in batches (a scanner’s daily export, a bulk migration), each job needs meaningful CPU and memory for a few minutes, and there are long idle stretches in between. That profile is a poor fit for an always-on server and a good fit for AWS Batch, which handles provisioning, queuing, and scaling of compute for you — you submit jobs, it finds capacity.</p><h4>The proposed architecture</h4><p>Let’s take a quick look at our architecture,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUh_BE1tl07guy-l-8QiiQ.png" /></figure><p>1. <strong>Amazon S3</strong> — the source of truth. Slides land under s3://&lt;bucket&gt;/&lt;folder&gt;/.<br>2. <strong>EventBridge</strong> — watches the bucket via S3 → EventBridge notifications, filtered specifically on object keys ending in `_READY`. Nothing runs until that dummy file marker appears — this lets you upload a whole batch of files first and trigger processing exactly once, on your own schedule.<br>3. <strong>Lambda</strong> (check trigger_batch_job.py) — the dispatcher. It targets the folder where the dummy `_READY` marker was dropped, and submits the job to AWS Batch. <br>4. <strong>AWS Batch</strong> — a managed EC2 compute environment allocates capacity on demand (0 to 64 vCPUs, `c7a.4xlarge` instances, `BEST_FIT_PROGRESSIVE` allocation) and runs the container as a job. Nobody pays for idle EC2 capacity between batches.<br>5. <strong>The container</strong> — downloads each input file from S3, runs the Visual NLP de-identification pipeline, and uploads the result to `&lt;folder&gt;_output/`. On a per-file failure, it writes `_FAILURE_{filename}` (containing the traceback) to the output prefix instead of silently dropping the file.</p><p><strong>Everything after the initial `_READY` marker write is event-driven — there’s no polling, no cron, and no idle compute cost.</strong></p><h3>4. Practical walkthrough: deploying the SVS pipeline</h3><p>This section follows the actual runbook in `svs/README.md`, ordered for a fresh deploy into an account that’s never run this before.</p><p><strong>Note</strong>: if you’re interested in Dicom, the steps are almost the same, follow dicom/README.md on the repository.</p><h4>The SVS de-identification pipeline itself</h4><p>Worth understanding before you deploy anything, because it explains <strong>why </strong>the container is built the way it is. For each input `.svs` file (`svs/docker/app.py`):</p><p>1. <strong>remove_phi</strong> — strips PHI-bearing header/metadata fields from the file.<br>2. <strong>svs_to_tiles</strong> — breaks the pyramidal image apart, extracting the label/macro auxiliary images (and a thumbnail) at an automatically-selected level.<br>3. A small Spark pipeline runs Text Detection over the tiles, and <strong>only tiles where text was actually detected are kept</strong> — this is the optimization that keeps cost down, since most of a whole-slide image is tissue with no text at all.<br>4. The surviving tiles go through the full clinical NER pipeline, which returns bounding-box coordinates for identified PHI text.<br>5. <strong>redact_phi_in_tiles</strong> draws over those coordinates. By default it redacts the tiles in place; set `CREATE_NEW_SVS_FILE=true` to instead assemble a brand-new de-identified `.svs` file, leaving the original untouched (slower, but non-destructive).</p><h4>Prerequisites</h4><p>Let’s make sure we have all these things working before we get started.</p><p>- AWS CLI configured for the target account — confirm with `aws sts get-caller-identity`.<br>- Docker, to build the image.<br>- Node.js (for `npx aws-cdk`) and Python 3.10+.<br>- A <strong>Visual NLP license/keys JSON file</strong>. <br>- IAM permissions on your user for <strong>AmazonEC2ContainerRegistryFullAccess</strong>, plus the CDK bootstrap policy set documented in cdk/README.md.</p><h4>Create the ECR repository (by hand, first)</h4><p>We create the repo *before* deploying the stack, and point the stack at it via context (rather than letting CDK own it), so that `cdk destroy` never touches the image and pushing a new build never forces a redeploy:</p><pre><br>aws ecr create-repository --repository-name deid-pipeline<br></pre><p>Note the `repositoryUri` from the output, e.g. `123456789012.dkr.ecr.us-east-1.amazonaws.com/deid-pipeline`.</p><h4>Build and push the container image</h4><p>Building the container is simple,</p><pre>REGION=&lt;like &#39;us-east-1&#39;&gt;<br>REPO_URI=&lt;repositoryUri from step 4.2&gt;<br>aws ecr get-login-password --region ${REGION} | docker login --username AWS --password-stdin &quot;${REPO_URI%%/*}&quot;<br><br><br>docker build --secret id=license,src=path/to/your-license.json -t deid-container docker<br>docker tag deid-container:latest &quot;${REPO_URI}:latest&quot;<br>docker push &quot;${REPO_URI}:latest&quot;</pre><h4>Bootstrap and deploy the CDK stack</h4><p>An AWS CDK(Cloud Development Kit) stack represents <strong>a collection of AWS resources</strong> that you define using CDK constructs.</p><p>The CDK app (`cdk/deid_pipeline/deid_stack.py`) <strong>provisions everything:</strong> a VPC (public subnets only, no NAT gateway — instances get a public IP for ECR/S3 access, keeping cost down), the Batch compute environment/queue/job definition, the Lambda trigger, the EventBridge rule, and a Secrets Manager secret holding `SPARK_OCR_LICENSE`.</p><pre>cd cdk<br>python3 -m venv .venv &amp;&amp; source .venv/bin/activate<br>pip install -r requirements.txt<br><br>npx aws-cdk bootstrap aws://123456789012/us-east-1   # once per account/region<br><br>npx aws-cdk deploy \<br>  -c ecr_repository_name=deid-pipeline \<br>  -c existing_bucket_name=pathology-images-123456789012-us-east-1 \<br>  -c license_file=./existing_license.json<br></pre><p><strong>A couple of context values worth understanding:</strong></p><p>- <strong>bucket_name vs. existing_bucket_name</strong> — pass `bucket_name` to have the stack create a fresh, stack-owned bucket (destroyed on `cdk destroy`); pass `existing_bucket_name` to import a bucket you already have (e.g. one shared with another workload). <strong>Pass at most one.</strong><br>- <strong>license_file</strong> is read locally at CDK synth time, and its `SPARK_OCR_LICENSE` field is written straight into the Secrets Manager secret the stack creates — there’s no separate “upload the secret” step. The stack file itself is generic and safe to commit; <strong>the JSON license is never committed</strong>.</p><p>Under the hood, two separate IAM roles matter for the running job:</p><p>- The <strong>job (task) role</strong> is what the container itself assumes — granted `read/write` on the data bucket, nothing else.<br>- The <strong>execution role</strong> is what ECS/Batch uses <strong>before</strong> your code even starts — to pull the image from ECR and resolve the `SPARK_OCR_LICENSE` secret from Secrets Manager, injecting it as an environment variable the container picks up at `nlp.start()`.</p><p><strong>Note the stack outputs — you’ll need `BucketName` and `JobQueueArn` for the next step.</strong></p><h4>Test end-to-end</h4><p>Let’s put this thing to work!</p><pre>BUCKET=&lt;BucketName from previous step&gt;<br>aws s3 cp slide1.svs s3://$BUCKET/testfolder/<br>aws s3 cp slide2.svs s3://$BUCKET/testfolder/</pre><pre># Nothing runs yet — this is the trigger:<br>aws s3api put-object --bucket $BUCKET --key testfolder/_READY --body /dev/null</pre><pre># Watch it work:<br>aws batch list-jobs --job-queue &lt;JobQueueArn from previous step&gt;<br>aws s3 ls s3://$BUCKET/testfolder_output/</pre><p>To tail the Lambda dispatcher’s logs live:</p><pre>FN=$(aws cloudformation describe-stacks --stack-name DeidPipelineStack \<br>  --query &quot;Stacks[0].Outputs[?OutputKey==&#39;TriggerFunctionName&#39;].OutputValue&quot; --output text)<br>aws logs tail /aws/lambda/$FN --follow<br></pre><p>If a file fails to process, <strong>you won’t get silence</strong> — you’ll get `testfolder_output/_FAILURE_slide1.svs` containing the full traceback, while the rest of the batch continues.</p><h4>Teardown</h4><p>Once we’re done, destroying the stack is easy,</p><pre>cd cdk<br>npx aws-cdk destroy</pre><p>Because the ECR repository was created outside the stack, `cdk destroy` never touches it or the image inside it — redeploying later doesn’t require rebuilding. Delete it explicitly if you’re done with it entirely:</p><pre>aws ecr delete-repository --repository-name deid-pipeline --force</pre><h3>5. Summary</h3><p>PHI in pathology imaging can be pretty intricate — it lives in scanner-generated label photos, macro overviews, and free-text metadata fields, which means finding it requires real OCR and NER. The pipeline in this post handles that automatically: data on S3 connects to a fully event-driven chain — EventBridge → Lambda → AWS Batch — spins up exactly the compute it needs, runs John Snow Labs’ Visual NLP de-identification pipeline, and writes a redacted slide back out, scaling to zero the moment the queue is empty.</p><p>The same architectural pattern (`S3 → EventBridge → Lambda → AWS Batch → container → S3`) also powers the companion DICOM pipeline in this repo, adapted for DICOM’s structured-tag-plus-burned-in-pixel PHI model instead of SVS’s tiled-image-plus-label model.</p><p>As an additional reminder, all the code referenced in this post — the CDK stack, the Lambda trigger, the Dockerfile, and the pipeline entrypoint — lives in <a href="https://github.com/JohnSnowLabs/visual-nlp-workshop/tree/master/products/awsbatch">this repository</a>.</p><p>- `svs/README.md` — the ordered deploy runbook this walkthrough follows<br>- `svs/cdk/` — the CDK app (`deid_pipeline/deid_stack.py`, `lambda/trigger_batch_job.py`)<br>- `svs/docker/` — the container source (`app.py`, `installer.py`, `Dockerfile`)<br>- `dicom/` — the parallel DICOM pipeline, same architecture</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcdb11a49357" width="1" height="1" alt=""><hr><p><a href="https://medium.com/john-snow-labs/de-identifying-pathology-slides-at-scale-a-serverless-aws-batch-pipeline-for-svs-whole-slide-dcdb11a49357">De-identifying Pathology Slides at Scale: A Serverless AWS Batch Pipeline for SVS Whole-Slide…</a> was originally published in <a href="https://medium.com/john-snow-labs">John Snow Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DICOM de-identification at scale in Visual NLP — Part 3.]]></title>
            <link>https://medium.com/john-snow-labs/dicom-de-identification-at-scale-in-visual-nlp-part-3-61cecc3adb56?source=rss-6c2287b0d455------2</link>
            <guid isPermaLink="false">https://medium.com/p/61cecc3adb56</guid>
            <dc:creator><![CDATA[Alberto Andreotti]]></dc:creator>
            <pubDate>Tue, 29 Oct 2024 16:02:49 GMT</pubDate>
            <atom:updated>2025-09-01T13:32:50.915Z</atom:updated>
            <content:encoded><![CDATA[<h3>DICOM de-identification at scale in Visual NLP — Part 3.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RadMANAfO2wlQAVF.png" /></figure><p>This post will delve into the utilization of <strong>Visual NLP</strong> to manipulate <strong>pixel</strong> and <strong>overlay data</strong> within DICOM images.</p><p>In the following examples, we will work with these two transformers: <strong>DicomToImageV3</strong>, which is responsible for extracting frame images, and <strong>DicomDrawRegions</strong>, which draws rectangle regions on top of Dicom frames and proves useful in building de-identification pipelines.</p><h3>DicomToImageV3</h3><p><strong>DicomToImageV3</strong> will extract images from the <strong>pixel</strong> and <strong>overlay data</strong> in the Dicom file, and will return them in a Spark DataFrame following Visual NLP’s <strong>Image</strong> structure.</p><p>Image representation on the dataframe can take multiple forms in Visual NLP. Typically, it’s either a grayscale or RGB image in uncompressed format.</p><p>On the other hand, DICOM files come with a vast variety of color spaces(or photometric interpretations), compression techniques, and many other metadata elements that help specify how the binary data contained in the file should be interpreted. Each combination of these settings, is called a Transfer Syntax, according to the DICOM standard.</p><p>Back to the color spaces, these are some typical input values you can expect to deal with in DicomToImageV3, MONOCHROME2, RGB, YBR, YBR FULL, YBR FULL 422, PALETTE COLOR. After this point, everything will happen automatically in the subsequent stages in the pipeline, so there’s not that much you have to worry about.</p><p>Now, let’s extract frames from one of the test DICOM files:</p><pre>dicom_to_image = DicomToImageV3() \<br>    .setInputCols([&quot;content&quot;]) \<br>    .setOutputCol(&quot;image&quot;) \<br>    .setKeepInput(False)<br><br>result = dicom_to_image.transform(dicom_df)<br><br>results.show()</pre><pre>+--------------------+---------+-------+--------------------+-------------------+------+<br>|               image|exception|pagenum|                path|   modificationTime|length|<br>+--------------------+---------+-------+--------------------+-------------------+------+<br>|{file:/Users/nmel...|         |      0|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      1|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      2|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      3|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      4|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      5|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      6|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      7|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      8|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |      9|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>|{file:/Users/nmel...|         |     10|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>+--------------------+---------+-------+--------------------+-------------------+------+</pre><p>There will be a separate row with an image for each DICOM frame. Then, the <strong><em>pagenum</em></strong> column will contain the number of the frame. Let’s display frames as images using <strong><em>display_images</em></strong> function:</p><pre>display_images(result, limit=2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*rOiwYlv1NHHcce0c.png" /></figure><p><strong>DicomToImageV3</strong> supports up to 1000 frames or more, depending on the size of the frames and the available memory. When working with a large number of frames during debugging of the pipeline, it is useful to extract only a limited number of frames. To do this, you can set the <strong><em>frameLimit</em></strong> parameter:</p><pre>dicom_to_image = DicomToImageV3() \<br>    .setInputCols([&quot;content&quot;]) \<br>    .setOutputCol(&quot;image&quot;) \<br>    .setFrameLimit(1) \<br>    .setKeepInput(False)<br><br>result = dicom_to_image.transform(dicom_df)<br><br>result.show()</pre><pre>+--------------------+---------+-------+--------------------+-------------------+------+<br>|               image|exception|pagenum|                path|   modificationTime|length|<br>+--------------------+---------+-------+--------------------+-------------------+------+<br>|{file:/Users/nmel...|         |      0|file:/Users/nmeln...|2023-08-20 14:17:23|426776|<br>+--------------------+---------+-------+--------------------+-------------------+------+</pre><p>For handling big files (2 GB or more) we need to use the <strong>path</strong> column as input instead of<strong> </strong>the content column. This forces to load Dicom files directly from the file system instead of passing through the DataFrame.</p><pre>dicom_to_image = DicomToImageV3() \<br>  .setInputCols([“path”]) \<br>  .setOutputCol(“image”) \<br>  .setKeepInput(False)</pre><h4>Handling Big Images</h4><p>So far, our assumption has been that images are of a reasonable size, and we will always have enough resources to handle them. In practice, many times images are too big, and we can get into trouble if we just decode them and feed them into models.</p><p>In DicomToImageV3, we have two parameters to help us with that,</p><ul><li>scale: this is a scaling factor to be applied across the width and height of the frames before extracting them from the source dicom file. Default value is ‘1’, which means no scaling. Using a value of, for example, .5, can reduce the effective size of the image by a factor of 4X, and thus reduce the memory requirements and processing times.</li><li>compressionMode: this parameter will configure compression in the images that are loaded into the output dataframe. Possible values are `enabled`, `disabled` or `auto`. When set to auto, compression will only be applied when the resulting uncompressed images go above certain threshold, leaving images uncompressed otherwise.</li><li>compressionThreshold: this is the size in megapixels of the resulting output images above which compression will be applied to the image before returning them in the output dataframe.</li></ul><h3>DicomDrawRegions</h3><p>We will use<strong> DicomDrawRegions</strong> to draw regions on top of the frames of our DICOM files. We will be masking both <strong>pixel </strong>and <strong>overlay data. </strong>We’ll do it this way because our Text Detection considered the entire image(pixel + overlay) when the text was detected, so some text may be coming from the overlay as well.</p><p>In this example, we will explore simple de-identification, in the sense that we will get rid of all the text that is present in the image. There is, however, the possibility of a more targeted process, in which we detect specific PHI entities like patient names or diagnoses, and only remove those.</p><p>Back to our simplified approach, we will have to start our pipeline by detecting the text on the image. Let’s get started!</p><p>We already can extract frame images using <strong>DicomToImageV3</strong>. In order for us to compare results with the original images, we will set the <strong><em>keepInput </em></strong>parameter to True. This will make thecontent field available after the call to DicomToImageV3.</p><pre>dicom_to_image = DicomToImageV3() \<br>    .setInputCols([&quot;content&quot;]) \<br>    .setOutputCol(&quot;image&quot;) \<br>    .setKeepInput(True)</pre><p>Next, we need to detect text. We can use <strong>ImageTextDetectorV2 </strong>here<strong>:</strong></p><pre>text_detector = ImageTextDetectorV2 \<br>    .pretrained(&quot;image_text_detector_v2&quot;, &quot;en&quot;, &quot;clinical/ocr&quot;) \<br>    .setInputCol(&quot;image&quot;) \<br>    .setOutputCol(&quot;regions&quot;) \<br>    .setScoreThreshold(0.5) \<br>    .setTextThreshold(0.2) \<br>    .setSizeThreshold(10)</pre><p>This comes with no mystery: it will receive an image, and it will return text regions in the form of bounding boxes, more specifically on the regions column.</p><p>And as final step, we draw solid rectangles using <strong>DicomDrawRegions:</strong></p><pre>draw_regions = DicomDrawRegions() \<br>    .setInputCol(&quot;path&quot;) \<br>    .setInputRegionsCol(&quot;regions&quot;) \<br>    .setOutputCol(&quot;dicom_cleaned&quot;) \<br>    .setRotated(True) \<br>    .setKeepInput(True)</pre><p>We put all the pieces together by creating our Spark ML Pipeline and calling it:</p><pre>pipeline = PipelineModel(stages=[<br>     dicom_to_image,<br>     text_detector,<br>     draw_regions<br>])<br><br>result = pipeline.transform(dicom_df)<br><br>result.show()</pre><pre>+--------------------+----------+--------------------+--------------------+<br>|       dicom_cleaned| exception|                path|             content|<br>+--------------------+----------+--------------------+--------------------+<br>|[52 75 62 6F 20 4...|          |file:/Users/nmeln...|[52 75 62 6F 20 4...|<br>+--------------------+----------+--------------------+--------------------+</pre><p>Let’s display original and cleaned DICOMS using the <strong><em>display_dicom() </em></strong>function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*br9sN_iMQr1Xj6YT.png" /></figure><p>We can see our results to the right, and we verify that all the text is gone!.</p><h4>Controlling the Output Compression</h4><p>At this point, we are almost ready to go! We have our images and overlay taken care of, and the only thing missing is to write back to DICOM so we can share those files without any risk.</p><p>One detail is missing, however, and that is the output compression type. Typically, you wouldn’t have to worry about it, as DicomDrawRegions would try to honor the original compression type present in the input file as much as possible. This default behavior is intended to avoid any unwanted information loss, for example when you pass from a lossless compression approach to a lossy one.</p><p>Some users, on the other hand will prefer to pick the compression algorithm manually, for example because they are more concerned about storage. In that case, you can pick the compression algorithm yourself. These are the supported options,</p><ul><li><strong>RLELossless</strong>: RLELossless: based on Run-Length Encoding Lossless, this method is employed for lossless compression.</li><li><strong>JPEGBaseline8Bit</strong>: JPEGBaseline8Bit is a specific variant of JPEG compression. It adheres to baseline compression with 8 bits per color channel, typically resulting in a 24-bit color depth for RGB medical images. This compression method is inherently lossy.</li><li><strong>JPEGLSLossless</strong>: JPEGLSLossless is a DICOM compression method that represents a lossless image compression standard based on the JPEG-LS (Lossless JPEG) standard.</li></ul><p>We can choose compression by setting the <strong>compression</strong> parameter and force compression of <strong>pixel data</strong> for files without compression by setting the <strong>forceCompress</strong> parameter to <strong>True</strong>:</p><pre>draw_regions = DicomDrawRegions() \<br>    .setInputCol(&quot;content&quot;) \<br>    .setInputRegionsCol(&quot;regions&quot;) \<br>    .setOutputCol(&quot;dicom_cleaned&quot;) \<br>    .setRotated(True) \<br>    .setCompression(DicomCompression.RLELossless) \<br>    .setForceCompress(True)</pre><h4>Getting the files out of the pipeline</h4><p>The last stage in today’s post is storing to a DICOM file on disk. To retrieve the name of the original file from the <strong>path</strong> column, let’s define a UDF function:</p><pre>def get_name(path, keep_subfolder_level=0):<br>    path = path.split(&quot;/&quot;)<br>    path[-1] = path[-1].split(&#39;.&#39;)[0]<br>    return &quot;/&quot;.join(path[-keep_subfolder_level-1:])</pre><p>To save the DataFrame with the cleaned DICOM files using the <strong><em>binaryFormat</em></strong> datasource to the <strong><em>output_path</em></strong>, we need to specify a few options:</p><ol><li>The <strong>‘type’</strong> of the file, ‘dicom’ in this case.</li><li>The ‘<strong>field’</strong> that contains the DICOM file, which is actually the column on the dataframe.</li><li>A ‘<strong>prefix’</strong> for the files.</li><li>The ‘<strong>nameField</strong>’ column, which contains the name of the file.</li><li>And an ‘outputPath’, that, as you guessed, is the location on disk for output files.</li></ol><pre>from pyspark.sql.functions import *<br><br>output_path = &quot;./deidentified/&quot;<br><br>result.withColumn(&quot;fileName&quot;, udf(get_name, StringType())(col(&quot;path&quot;))) \<br>    .write \<br>    .format(&quot;binaryFormat&quot;) \<br>    .option(&quot;type&quot;, &quot;dicom&quot;) \<br>    .option(&quot;field&quot;, &quot;dicom_cleaned&quot;) \<br>    .option(&quot;prefix&quot;, &quot;ocr_&quot;) \<br>    .option(&quot;nameField&quot;, &quot;fileName&quot;) \<br>    .mode(&quot;overwrite&quot;) \<br>    .save(output_path)</pre><p>You can find the Jupyter notebook with the full code <a href="https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/jupyter/Dicom/SparkOcrDicomRemoveText.ipynb">here</a>.</p><p>In this post, we have constructed the simplest de-identification pipeline to remove all text in DICOM images. In the next post, we will create a more complex pipeline using <strong>NER</strong> de-identification models with <strong>Spark NLP </strong>and<strong> Spark NLP for Healthcare</strong>.</p><h3>Links</h3><ul><li><a href="https://github.com/JohnSnowLabs/spark-ocr-workshop">Visual NLP Workshop</a></li><li><a href="https://nlp.johnsnowlabs.com/docs/en/ocr">Visual NLP Documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61cecc3adb56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/john-snow-labs/dicom-de-identification-at-scale-in-visual-nlp-part-3-61cecc3adb56">DICOM de-identification at scale in Visual NLP — Part 3.</a> was originally published in <a href="https://medium.com/john-snow-labs">John Snow Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Question Answering in Visual NLP: A Picture is Worth a Thousand Answers]]></title>
            <link>https://medium.com/spark-nlp/question-answering-in-visual-nlp-a-picture-is-worth-a-thousand-answers-535bbcb53d3c?source=rss-6c2287b0d455------2</link>
            <guid isPermaLink="false">https://medium.com/p/535bbcb53d3c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Alberto Andreotti]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 21:28:41 GMT</pubDate>
            <atom:updated>2023-04-13T21:28:41.111Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oMDWa-Urq7PB0Lxc" /><figcaption>Photo by <a href="https://unsplash.com/@cdd20?utm_source=medium&amp;utm_medium=referral">愚木混株 cdd20</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Lights, camera, action! Welcome to the future of information extraction with Visual NLP by John Snow Labs, where “OCR-Free” multi-modal AI models are stealing the show. Imagine a world where computer vision and NLP join forces to extract high-accuracy information from forms, tables, or diagrams, regardless of image quality or the need for training. Today, we’ll dive into the thrilling world of Document Visual Question Answering and introduce you to the Donut model, a state-of-the-art AI model that’s taking the stage by storm.</p><h4><strong>A New Era of Information Extraction: Document Visual Question Answering</strong></h4><p>Gone are the days of relying solely on traditional OCR systems to extract information from images. Visual Question Answering represents a new generation of AI models that combine computer vision and NLP to provide higher accuracy information extraction. This dynamic duo can handle poor quality images and deliver results without training or tuning models, thanks to zero-shot learning. The result? Faster, more accurate extraction, and happier end users and developers.</p><h4><strong>Meet the Donut Model: Your New BFF for Visual NLP</strong></h4><p>The Donut model (<a href="https://arxiv.org/abs/2111.15664">https://arxiv.org/abs/2111.15664</a>) is the new kid on the block, and it’s making quite a splash. This innovative model achieves state-of-the-art accuracy in various benchmarks, revolutionizing the way we think about AI-powered information extraction.</p><p>The Donut model’s unique approach allows it to understand both the visual and textual elements of a document, combining these aspects to answer questions directly from an image. This groundbreaking method paves the way for more accurate, efficient, and reliable data extraction.</p><p>For example, consider this image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*b2a4eSC92CbuaEm52yo3kw.png" /></figure><p>How would you answer these two questions:</p><ol><li>When is the coffee break?</li><li>Who is giving the introductory remarks?</li></ol><p>The answers are obvious from the image — even though they are not written there. As a human, you have the common sense to know that this is part of an agenda to an event, so with all likelihood the coffee break is between 11:14 to 11:39am, and Lee A. Waller is giving the introductory remark. This requires a combination of:</p><ul><li>Seeing — you see the two columns align.</li><li>Reading — you can’t answer the questions without knowing how to read.</li><li>Common sense — you can tell that this is an agenda for an event without further hints, and that on an agenda times state when an event will happen, and names refer to speakers.</li></ul><p>The donut model can give you a correct answer to these questions, directly from the image, without any training or tuning, even without OCR!. It also does that in four languages.</p><h4><strong>Visual NLP + Donut Model = A Match Made in Heaven</strong></h4><p>Integrating the Donut model with John Snow Labs’ Visual NLP (<a href="https://www.johnsnowlabs.com/visual-nlp/">https://www.johnsnowlabs.com/visual-nlp/</a>) creates a powerful combination that brings numerous concrete benefits to the table:</p><ul><li><strong>Efficient handling of complex documents</strong>: The combined strengths of the Donut model and Visual NLP allow for efficient processing and understanding of long, intricate, and unusually formatted documents, making data extraction more reliable. Easily combine the Donut with other models like Table Recognition and Document Classifiers to apply VQA on the specific sections you care about.</li><li><strong>Superior accuracy</strong>: The integration of the Donut model with Visual NLP leads to exceptional accuracy in data extraction, reducing the prevalence of errors and improving overall data quality. Apply Image Enhancement before VQA, and have access to purpose specific fine-tuned models.</li><li><strong>Streamlined workflows</strong>: The integration simplifies the implementation process, allowing developers to focus on other tasks while leveraging the power of the Donut model within Visual NLP. Leverage Visual NLP serving options like LightPipelines or Data Streaming.</li><li><strong>Enhanced scalability</strong>: By combining the Donut model with Visual NLP’s powerful distributed processing capabilities, users can scale up their data extraction processes to handle large volumes of documents more efficiently. Scale up computation through Apache Spark to millions of records without changing your pipeline.</li><li><strong>Better adaptability</strong>: The combination of Donut and Visual NLP provides improved adaptability, allowing users to tackle a wider range of extraction tasks and handle different types of documents with ease. Use out-of-the-box data ingestion layers to consume DocX, PDFs, and many others!.</li></ul><h4><strong>Donut Model in Action: A Picture-Perfect Example</strong></h4><p>Ready to see the Donut model in action? Let’s take a look at the code required to run Donut inside Visual NLP and explore a concrete example of an image that the model can answer questions from directly.</p><p><strong>Define the pipeline</strong></p><pre>binary_to_image = BinaryToImage()\<br>    .setOutputCol(&quot;image&quot;) \<br>    .setImageType(ImageType.TYPE_3BYTE_BGR)<br><br>visual_question_answering = VisualQuestionAnswering()\<br>    .pretrained(&quot;docvqa_donut_base_opt&quot;, &quot;en&quot;, &quot;clinical/ocr&quot;)\<br>    .setInputCol([&quot;image&quot;])\<br>    .setOutputCol(&quot;answers&quot;)\<br>    .setQuestionsCol(&quot;questions&quot;)<br><br># OCR pipeline<br>pipeline = PipelineModel(stages=[<br>    binary_to_image,<br>    visual_question_answering<br>]).setOutputCol(“image”)</pre><p><strong>Call the pipeline</strong></p><pre>%%time<br>from pyspark.sql.functions import explode<br>results = pipeline.transform(image_and_questions).cache()<br>results.select(results.answers).show(truncate=False)<br>+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <br>|answers<br>+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <br>|[ When is the Coffee Break? -&gt; 11:34 to 11:39 a.m., <br>   Who is giving the Introductory Remarks? -&gt; lee a. waller, trrf vice presi- ident|<br>+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - <br>CPU times: user 12.1 ms, sys: 10.7 ms, total: 22.8 ms</pre><h4><strong>And there’s more!</strong></h4><p>This was a super simplified example for us to build intuition around the topic, if you want to explore more complex use cases, I encourage you to take a look at the two notebooks listed here,</p><p><a href="https://github.com/JohnSnowLabs/spark-ocr-workshop/tree/master/webinars">spark-ocr-workshop/webinars at master · JohnSnowLabs/spark-ocr-workshop</a></p><p><a href="https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/webinars/VisualQuestionAnsweringOnInvoices.ipynb">VisualQuestionAnsweringOnInvoices.ipynb</a> explains how to process invoices out of a collection of documents. It first classifies the documents into different categories, it keeps the invoices and then asks some questions about the total amount billed.</p><p><a href="https://github.com/JohnSnowLabs/spark-ocr-workshop/blob/master/webinars/VisualQuestionAnsweringOnTables.ipynb">VisualQuestionAnsweringOnTables.ipynb</a>: similarly this one deals with documents from the Australian Stock Exchange, trying to automatically determine the amount of shares a company’s director has acquired or sold.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=535bbcb53d3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spark-nlp/question-answering-in-visual-nlp-a-picture-is-worth-a-thousand-answers-535bbcb53d3c">Question Answering in Visual NLP: A Picture is Worth a Thousand Answers</a> was originally published in <a href="https://medium.com/spark-nlp">spark-nlp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training a Contextual Spell Checker for Italian Language.]]></title>
            <link>https://medium.com/data-science/training-a-contextual-spell-checker-for-italian-language-66dda528e4bf?source=rss-6c2287b0d455------2</link>
            <guid isPermaLink="false">https://medium.com/p/66dda528e4bf</guid>
            <dc:creator><![CDATA[Alberto Andreotti]]></dc:creator>
            <pubDate>Wed, 10 Mar 2021 17:46:11 GMT</pubDate>
            <atom:updated>2021-03-12T21:00:31.904Z</atom:updated>
            <content:encoded><![CDATA[<h3>Training a Contextual Spell Checker for Italian Language</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5EUZI-W_6gM9jBAAgcGkOA.jpeg" /><figcaption>Image credit: Unsplash.</figcaption></figure><h4>Introduction</h4><p>In a <a href="https://medium.com/spark-nlp/applying-context-aware-spell-checking-in-spark-nlp-3c29c46963bc">previous post</a>, we have introduced Spell Checking, and the need for considering contextual clues when generating corrections. We also discussed the availability of pre-trained models for Spell Checking in the Spark-NLP library. Although, pre-trained models are convenient for many situations, sometimes you need to target a specific domain like finance or a specific language like German. In this post we’re going to explore training a Conextual Spell Checker from scratch.</p><p>We will delve into the training process and data modeling involved in creating a Contextual Spell Checking model. We will be using generic Italian as our target language, but these steps can be applied to any other domain or language of your interest.</p><p>We will be using the <a href="http://www.corpusitaliano.it/">PAISÀ Corpus</a> as our training dataset, and we will add custom word classes to handle things like places, and names; finally we will see how to use the GPU with Spark-NLP to speed up training. Let’s get started!.</p><h4>Code for this article</h4><p>You can find the entire code for this article along with some additional examples, in the following notebook,</p><p><a href="https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/blogposts/5.TrainingContextSpellChecker.ipynb">Training a Context Spell Checker — Italian.</a></p><h3>Data Preparation</h3><p>You’re going to be downloading the corpus file from <a href="https://clarin.eurac.edu/repository/xmlui/bitstream/handle/20.500.12124/3/paisa.raw.utf8.gz?sequence=1&amp;isAllowed=y">this URL</a>. Next, we will be applying some cleaning to the raw text to get rid of some noise.</p><p>Let’s get started by adding some basic imports,</p><pre>from sparknlp.annotator import *<br>from sparknlp.common import *<br>from sparknlp.base import *<br>import sparknlp<br>spark = sparknlp.start()</pre><p>Let’s follow by loading the corpus data.</p><pre>from pyspark.sql.functions import *<br>paisaCorpusPath = “/path/to/paisa.raw.utf8”</pre><pre># do some brief DS exploration, and preparation to get clean text<br>df = spark.read.text(paisaCorpusPath)<br>df = df.filter(~col(‘value’).contains(‘&lt;/text’)).\<br> filter(~col(‘value’).contains(‘&lt;text’)).\<br> filter(~col(‘value’).startswith(‘#’)).\<br> limit(10000)</pre><pre>df.show(truncate=False)</pre><p>And this is what we get,</p><pre>+--------------------+<br>|               value|<br>+--------------------+<br>|Davide Guglielmin...|<br>|Avete partecipato...|<br>|Otto mesi di rita...|<br>|Il prezzo dei big...|<br>|Dopo la “Shopping...|<br>|Il nome è tutto u...|<br>|Ovvero? Ovvero do...|<br>|Il patto ha lo sc...|<br>|La tre giorni org...|<br>|Quanto pagate per...|<br>|Per contro, nell’...|<br>|Non è poco, ovvia...|<br>|Dopo una lunga as...|<br>|La serata sarà un...|<br>|I primi 4 negozi ...|<br>|La Chaingang Rota...|<br>|La Chaingang Rota...|<br>|Tra le molte rami...|<br>|Un primo corollar...|<br>|E&#39; proprio nel di...|<br>+--------------------+<br>only showing top 20 rows</pre><p>We have 7555997 paragraphs there, each containing a few sentences, and as training does not require labeled data, that’s all we need.</p><h3>Basic Pipeline Setup</h3><p>Now that we have our data in good shape, we’re going to setup a basic pipeline to do our processing. This is the typical Spark-NLP pipeline with data flowing as annotations inside annotators.</p><pre>assembler = DocumentAssembler()\<br> .setInputCol(“value”)\<br> .setOutputCol(“document”)\</pre><pre>tokenizer = RecursiveTokenizer()\<br> .setInputCols(“document”)\<br> .setOutputCol(“token”)\<br> .setPrefixes([“\””, ““”, “(“, “[“, “\n”, “.”, “l’”, “dell’”, “nell’”, “sull’”, “all’”, “d’”, “un’”])\<br> .setSuffixes([“\””, “””, “.”, “,”, “?”, “)”, “]”, “!”, “;”, “:”])</pre><pre>spellChecker = ContextSpellCheckerApproach().\<br>    setInputCols(&quot;token&quot;).\<br>    setOutputCol(&quot;corrected&quot;).\<br>    setLanguageModelClasses(1650).\<br>    setWordMaxDistance(3).\<br>    setEpochs(10).\<br>    addVocabClass(&#39;_NAME_&#39;, names).\<br>    updateRegexClass(&#39;_DATE_&#39;, date_regex)</pre><p>We start with our DocumentAssembler for which we feed the data that we have prepared in previous step. Then we continue with our RecursiveTokenizer, in which we setup some special options to deal with some details of the Italian language.</p><p>Finally, we have our Spell Checker. The first two method calls are the usual ones for connecting inputs and outputs in a pipeline. Then we have a setting for the language model inside the Spell Checker setLanguageModelClasses(), this one is dependent on the vocabulary size, and the model will use it to control the factoring in the language model.</p><p>Basically, the idea behind factoring is that we won’t treat words in isolation, but rather we will group them assigning each word a specific class, and an id within the class. This helps speeding up the internal processing both in training and during inference.</p><p>Following we have setWordMaxDistance(3), which simply tells the Spell Checker that our errors are going to be at an edit distance of 3, at most.</p><p>The number of epochs is the typical gradient descent parameter that you know from neural networks, and it will be used by the neural language model inside the Spell Checker.</p><p>Last but not least, we have the addition of a special word class, in this case for treating names. Let’s get into more detail there!.</p><h3>Adding special classes</h3><p>This section is optional and explains the addVocabClass() and addRegexClass() calls that we made at the end of the annotator setup code. If you don’t want to add special classes or you want to come back later, just comment out those two lines.</p><p>At this point we have a corpus which is fairly big enough to give us good exposure to different flavors of the Italian language.</p><p>Still, we may want to have another mechanism for building our model vocabulary outside what we’ve seen in the corpus. This is where special classes come into play.</p><p>With this, we can unlock the power of knowledge contained in specific datasets like name dictionaries or gazetteers.</p><p>They will not only allow us to teach the spell checker to preserve some words, but also to propose corrections, and unlike the words in the main vocabulary, you will be able to update them once the model is trained. For example, if after the model is trained you still want to be able to support a new name, you can do so by just updating the names class, without re-training everything from scratch.</p><p>When you train a Spell Checker from scratch as we’re doing here you get two special classes predefined: _DATE_, and _NUM_, for handling dates and numbers respectively. You can override them and also add more classes.</p><p>Following our example about names, we will add Italian names as a special class in our Context Spell Checker in Spark-NLP,</p><pre>import pandas as pd<br>import io<br>import requests</pre><pre># Get a list of common Italian names<br>url=”<a href="https://gist.githubusercontent.com/pdesterlich/2562329/raw/7c09ac44d769539c61df15d5b3c441eaebb77660/nomi_italiani.txt">https://gist.githubusercontent.com/pdesterlich/2562329/raw/7c09ac44d769539c61df15d5b3c441eaebb77660/nomi_italiani.txt</a>&quot;<br>s=requests.get(url).content</pre><pre># remove the first couple of lines (which are comments) &amp; capitalize first letter<br>names = [name[0].upper() + name[1:] for name in s.decode(‘utf-8’).split(‘\n’)[7:]]</pre><pre># visualize<br>names</pre><p>This is what we get(truncated),</p><pre>[&#39;Abaco&#39;,<br> &#39;Abbondanza&#39;,<br> &#39;Abbondanzia&#39;,<br> &#39;Abbondanzio&#39;,<br> &#39;Abbondazio&#39;,<br> &#39;Abbondia&#39;,<br> &#39;Abbondina&#39;,<br> &#39;Abbondio&#39;,<br> &#39;Abdelkrim&#39;,<br> &#39;Abdellah&#39;,<br> &#39;Abdenago&#39;,<br> &#39;Abdon&#39;,<br> &#39;Abdone&#39;,<br> &#39;Abela&#39;,<br> &#39;Abelarda&#39;,<br> &#39;Abelardo&#39;,<br> &#39;Abele&#39;,</pre><p>And this is what we passed in the <em>names </em>variable<em> </em>to the Spell Checker during annotator setup in previous listing.</p><p>If you paid attention on how we set up the annotator, you may also have noticed that we configured a second special class for numbers using the updateRegexClass() function call.</p><p>This is going to serve the same purpose of adding additional sources for correction candidates, but this time for dates, so we’ll need to pass a regex describing the dates that we want to cover.</p><p>We will focus on a specific format here, but you can add your own as well,</p><p><strong>format:</strong> dd/mm/yyyy</p><p><strong>regex:</strong></p><p>([0–2][0–9]|30|31)\\/(01|02|03|04|05|06|07|08|09|10|11|12) \\/(19|20)[0–9]{2}</p><p>It’s important to understand what happens behinds the scenes with the regex we’ve just created. First the regex we created is a finite regex, in the sense that the vocabulary, or set of strings it matches is finite. This is important because the Spell Checker will enumerate the regex, and create a Levenshtein Automaton with it.</p><h4>Training the Model</h4><p>Now that we have our pipeline ready we will just call fit(), and pass our dataset.</p><pre>pipeline = Pipeline(<br> stages = [<br>    assembler,<br>    tokenizer,<br>    spellChecker<br> ])<br>model = pipeline.fit(df)</pre><h4>Using the GPU</h4><p>You will notice that the training process is not very fast. That’s because so far we’ve only used CPU. We can significantly speedup training by just replacing the spark-nlp library with spark-nlp-gpu.</p><p>You can do this with almost no change just by passing the right flag to the Spark-NLP start() function like this,</p><pre>sparknlp.start(gpu=True)</pre><h4>Playing with the model</h4><p>Let’s see what this model can do!.</p><pre>lp.annotate(“sonno Glorea ho lasciatth la paterte sul tavolo acanto allu fruttu”)[‘corrected’]</pre><pre>[&#39;sono&#39;,<br> &#39;Gloria&#39;,<br> &#39;ho&#39;,<br> &#39;lasciato&#39;,<br> &#39;la&#39;,<br> &#39;patente&#39;,<br> &#39;sul&#39;,<br> &#39;tavolo&#39;,<br> &#39;accanto&#39;,<br> &#39;alla&#39;,<br> &#39;frutta&#39;]</pre><p>We can see that the model is indeed generating corrections that consider the context, for example ‘allu’ was corrected as ‘alla’, when other options like ‘alle’, or ‘allo’ were also at the same edit distance of 1 from the input word, however the model took the right choice. We can also see the names lexicalization helping the model to understand and correct the proper name ‘Gloria’.</p><p>In general, this is what we can expect from this model, to take the right choice when multiple similar, equally distant candidates, are possible. The model can take some more risky decisions like actually replace input words that are present in the vocabulary.</p><p>This is more the job of a grammar check, think of Grammarly, for example. Still, if you want to explore how much you can stretch this model to perform these corrections there are more examples in the companion notebook for this article.</p><h4>Access the model through the Model Hub</h4><p>Although this model is not perfect, I have uploaded it to the <a href="https://nlp.johnsnowlabs.com/models">Model Hub</a> and made it available to the community in case somebody finds it useful. You can access it in the following manner,</p><pre>ContextSpellChecker.pretrained(“italian_spell”, “it”)</pre><p>You can as well train your own models, and share them with the community following <a href="https://modelshub.johnsnowlabs.com/">these steps</a>.</p><h4>A word on Tokenization</h4><p>This section is optional but recommended for those trying to fine tune their models for a particular language or domain.</p><p>The tokenization part is crucial in many NLP tasks, and Spell Checking is not exception. The units that you’ll get out of tokenization will be very important as those will define the level at which your model will detect errors, generate corrections and leverage the context for ranking different corrections.</p><p>Here, we’re using the RecursiveTokenizer, which is called like that because it will continue to apply its tokenization strategy recursively, that is, to the results of previous tokenization steps, until the strategy cannot be applied anymore.</p><p>The default tokenization in the RecursiveTokenizer is to split on white space. To tokenize further you can specify infix patterns, prefix patterns and suffix patterns.</p><p>Infix patterns are characters that can be found in the middle of a token and on which you want to split. Accordingly, prefix patterns, and suffix patterns are characters that you will like to split on and will appear in the beginning, and in the end of each token, respectively.</p><p>There are more options for this RecursiveTokenizer that you can find in the documentation. Also there are other tokenizers available to you in the Spark-NLP library.</p><h4>Conclusion</h4><p>We’ve explored how we can train our own Spell Checker, how to prepare our data and pipeline, how to provide rules for special classes, how to use the GPU, and how to adjust tokenization to better match our particular needs.</p><p>The Spark-NLP library provides some reasonable default behaviors, but can also be adjusted for specific domains and use cases.</p><p>Hope you can start training and sharing your own models, in bocca al lupo!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66dda528e4bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/training-a-contextual-spell-checker-for-italian-language-66dda528e4bf">Training a Contextual Spell Checker for Italian Language.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applying Context Aware Spell Checking in Spark NLP]]></title>
            <link>https://medium.com/spark-nlp/applying-context-aware-spell-checking-in-spark-nlp-3c29c46963bc?source=rss-6c2287b0d455------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c29c46963bc</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spelling-correction]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Alberto Andreotti]]></dc:creator>
            <pubDate>Thu, 21 May 2020 03:57:42 GMT</pubDate>
            <atom:updated>2021-03-10T18:01:04.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPH63No_aw_JmbNba-KVaw.jpeg" /><figcaption>Image credit: Pexels.</figcaption></figure><h3>Introduction</h3><p>Today we are exploring Spell Checking, a very important task in any serious NLP pipeline that needs to deal with noisy, incorrect data that has been generated in the wild.</p><p>Take for example the case of tweets, instant messaging, blog posts, OCR, or any other user generated text content. Being able to rely on correct data, without spelling problems reduces vocabulary sizes at different stages in the pipeline, and improves the performance of all the models in the pipeline.</p><p>More specifically, we’re going to be exploring Spark-NLP’s ContextSpellChecker annotator, a special class of Spell Checker that benefits from contextual information to both detect errors and produce the best corrections.</p><p>Spark NLP is a Natural Language Processing(NLP) library that allows you to run NLP algorithms and models in a distributed environment. Being one of the most widely used open source libraries for NLP, and the only one that allows to run in parallel in a distributed cluster of computers, Spark NLP provides state of the art models for many different tasks from Tokenization, Sentence Detection, and Name Entity Recognition to Contextual Embeddings like BERT or ELMO. You may be interested in looking at the <a href="https://nlp.johnsnowlabs.com/docs/en/models">complete list of models</a>.</p><h3>Spark NLP Pipelines &amp; annotations</h3><p>Next we’re going to explore a simple Spark NLP pipeline. For this, we are going to focus on Figure 1. A pipeline is just a sequence of <em>annotators</em>, each performing a different task on our input data. In our example, we have four Annotators: DocumentAssember, RecursiveTokenizer, ContextSpellChecker, and Finisher. Let’s explore each of those.</p><p>The <strong>DocumentAssembler</strong>, will take your input text, and will create a number of <em>annotations</em> inside a Spark Dataframe to represent your data. Annotations are just an alternative representation of your data in which that is useful for attaching metadata to it. A full list of members of the Annotation structure can be seen in Figure 2.</p><p>You’re going to have different flavors for these annotations at every stage of the pipeline. For example, annotations taken out of the DocumentAssembler will contain the text for the document, typically a paragraph, or a line, together with some indexes describing the beginning and end of the text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/1*IuXY4v95cVkn6WjYav5OJA.png" /><figcaption>Figure 1: A typical Spark NLP pipeline.</figcaption></figure><p>Following with our example, we have our next annotator, the <strong>RecursiveTokenizer</strong> that will split your document(s) into a set of tokens according to some rules that will depend on what you want to accomplish in the following annotators.</p><p>Next, we have our <strong>ContextSpellChecker</strong> that will take each of these tokens, detect if there’s any error, correct them and return clean tokens at the output.</p><p>Finally, our last annotator, the <strong>Finisher</strong>, will take the corrected text and will transform the set of annotations into a set of strings. This will enable other applications, for example, to display the text to users as reading through the annotation structure can be confusing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*-233wZ_PdX6dbH7popxgUQ.png" /><figcaption>Figure 2: The structure of an Annotation.</figcaption></figure><p>These structures can get more complex than this, according to different use cases, but at least we now count on a good understanding that will guide us through the rest of the topics. If you’re interested in knowing more, please check <a href="https://youtu.be/fsS057SNFtg">this great tutorial</a> freely available on Youtube.</p><h3><strong>Spell Checking Task</strong></h3><p>Spell Checking is a task for which we don’t see much activity, when compared for example to classification or translation in NLP. Interestingly, the task itself sometimes is defined in different ways according to the use case. However, it seems there’s a clear consensus that the task consists basically in doing two things,</p><ol><li>Detecting which words need correction.</li><li>Proposing a correction or a list of candidate corrections for the word.</li></ol><p>The first point is straightforward, you need to be accurate when detecting those words that need correction; this will allow you to both preserve correct words, and will also allow for faster correction times.</p><p>The second point can lead to some discussion. For Spell Checking in the context of digital writing tools like word processors or the keyboard in your cell phone, the task is defined as precision@k, meaning that we are satisfied with the model producing a suggestion list of k elements containing the right answer.</p><p>On the other hand, Spark-NLP is typically used on complete chunks of text, meaning that we don’t have a human in the loop while we produce corrections. This has both a good side and a bad one. On the good side, we can use all the text surrounding a word to produce a correction(before and after each word), and on the bad side we only have one shot to produce the right correction.</p><p>OK, too much talking so far, let’s get warm with an example,</p><blockquote>“I will call my <strong><em>siter</em></strong>.”</blockquote><blockquote>“Due to bad weather, we had to move to a different <strong><em>siter</em></strong>.”</blockquote><blockquote>“We travelled to three <strong><em>siter</em></strong> in the summer.”</blockquote><p>The appropriate corrections there, starting from the first sentence, would be {sister, site, and sites}. So how did we know that?: <strong>Contextual Information</strong>.</p><p>If you look carefully, all three corrections are at edit distance 1 from the right answer,</p><blockquote>siter -&gt; insert ‘s’ -&gt; sister</blockquote><blockquote>siter -&gt; remove ‘r’ -&gt; site</blockquote><blockquote>siter -&gt; replace ‘r’ by ‘s’ -&gt; sites</blockquote><p>And also the correction words {sister, site, and sites} have a very similar uni-gram probability. So traditional spell checking methods are going to be in trouble to make the right choice here.</p><h3><strong>Spell Checking in Spark-NLP</strong></h3><p>There are currently three spell checkers in Spark NLP: SymmetricDelete, NorvigSweeting, and ContextSpellChecker. The first two do not take the context words into considerations, but are still a reasonable option if performance is a priority.</p><p>Going back to our ContextSpellChecker, we all have witnessed how Deep Learning changed the field of NLP in the last couple of years. Deep Learning is great, but it comes with a caveat, it’s difficult to customize your pretrained models to adjust for specific datasets without full retrains.</p><p>The approach that the ContextSpellChecker takes is to blend the best of two worlds; benefit from the great results of Deep Learning, and still allow some level of customization like in traditional methods. This allows for configurable, off-the-shelf, Contextual Spell Checking to be possible.</p><p>Let’s have a look!</p><h3><strong>Getting Started</strong></h3><p>At this point, we have a good idea of why the context is important when proposing corrections for words during Spell Checking. Now let’s see how we can materialize this in Spark-NLP.</p><p>As it’s usual, let’s start by building a pipeline; a <em>spell correction pipeline</em>. This pipeline is going to include an instance of the DocumentAssembler, RecursiveTokenizer, <strong>ContextSpellChecker</strong> and Finalizer.</p><p>We will use pretrained models from our library for each of these annotators.</p><pre>from sparknlp.annotator import *<br>from sparknlp.common import *<br>from sparknlp.base import *<br>from sparknlp_jsl.annotator import *</pre><pre>spark = sparknlp.start()</pre><pre>documentAssembler = DocumentAssembler()\<br>.setInputCol(“text”)\<br>.setOutputCol(“document”)</pre><pre>tokenizer = RecursiveTokenizer()\<br>.setInputCols([“document”])\<br>.setOutputCol(“token”)\<br>.setPrefixes([“\””, “(“, “[“, “\n”])\<br>.setSuffixes([“.”, “,”, “?”, “)”,”!”, “‘s”])</pre><pre>spellModel = ContextSpellCheckerModel\<br>.pretrained()<br>.setInputCols(“token”)\<br>.setOutputCol(“checked”)</pre><pre>finisher = Finisher()\<br>.setInputCols(“checked”)</pre><pre>pipeline = Pipeline(stages = [<br>documentAssembler,<br>tokenizer,<br>spellModel,<br>finisher<br>])</pre><pre># let&#39;s create an empty dataframe just to call fit()<br>empty_ds = spark.createDataFrame([[&quot;&quot;]]).toDF(&quot;text&quot;)<br>lp = LightPipeline(pipeline.fit(empty_ds))</pre><p>As usual, we built our pipeline as a sequence of annotators, and we fed the output of one annotator to the input of the next. The RecursiveTokenizer comes by default with a set of rules to handle English, however you can define your own rules, to handle other languages or other scenarios. The ContextSpellChecker is coming pretrained from the model repository.</p><p>Finally, notice the use of the LightPipeline, this is a convenience class that allows us to play with the models without having to deal with Spark Dataframes.</p><p>OK, so we built the pipeline, let’s see what we can do with it!</p><pre>lp.annotate(“Plaese alliow me tao introdduce myhelf, I am a man of waelth und tiaste”)</pre><pre>&gt;&gt;</pre><pre>{‘checked’: [‘Please’, ‘allow’, ‘me’, ‘to’, ‘introduce’,                              ‘myself’, ‘,’, ‘I’, ‘am’, ‘a’, ‘man’, ‘of’, ‘wealth’, ‘and’, ‘taste’]}</pre><p>Not bad, right?</p><h3>How ContextSpellChecker works ?</h3><p>Spell Checking is a sequence to sequence mapping problem. Given an input sequence, potentially containing a certain number of errors, ContextSpellChecker will rank correction sequences according to three things,</p><ol><li>Different correction candidates for each word — <strong>word level</strong>.</li><li>The surrounding text of each word, i.e. it’s context — <strong>sentence level</strong>.</li><li>The relative cost of different correction candidates according to the edit operations at the character level it requires — <strong>subword level.</strong></li></ol><p>Let’s explore these different levels, and let’s see how we can customize the pretrained model in Spark-NLP to better fit our particular needs.</p><h3>Word Level Corrections</h3><p>Continuing with our pretrained pipeline, let’s try to see how corrections work at the word level. Each ContextSpellChecker model that you will find in Spark-NLP library comes with two sources for word candidates:</p><ul><li>a general vocabulary that is built from the training corpus during model training(and remains immutable during the life of the model), and</li><li>special classes for dealing with special types of words like numbers or dates. These are configurable, and you can modify them so they can adjust better to your data.</li></ul><p>The general vocabulary is learned during training, and cannot be modified, however, the special classes on a pretrained model can be updated. This means you can modify how existing classes produce corrections, but not the number or type of the classes. Let’s see how we can accomplish this.</p><p>First, let’s go back to our pretrained model, and check which classes it has been trained with,</p><pre>spellModel.getWordClasses()</pre><pre>&gt;&gt;</pre><pre>[‘(_AGE_,RegexParser)’,<br>‘(_LOC_,VocabParser)’,<br>‘(_DATE_,RegexParser)’,<br>‘(_NAME_,VocabParser)’,<br>‘(_NUM_,RegexParser)’]</pre><p>We have five predefined classes,</p><ul><li>AGE: age tokens like ‘21-year-old’.</li><li>LOC: tokens representing locations like a city, state, country, etc.</li><li>DATE: tokens representing dates like ‘Jan-03&#39;.</li><li>NAME: tokens representing names and surnames.</li><li>NUM: tokens representing numbers, like 22 or twenty-two.</li></ul><p>of two different types: some are vocabulary based and others are regex based, let’s see what this means,</p><ul><li><strong>Vocabulary based classes</strong> can propose correction candidates from the provided vocabulary, for example a dictionary of names.</li><li><strong>Regex classes</strong> are defined by a regular expression, and they can be used to generate correction candidates for things like numbers. Internally, the Spell Checker will enumerate your regular expression and build a fast automaton, not only for recognizing the word(number in this example) as valid and preserve it, but also for generating a correction candidate. Thus the regex should be a finite regex(it must define a finite regular language).</li></ul><p>Now let’s see this in action with an example. Suppose that you have a new friend from Poland whose name is Jowita, let’s see how the pretrained Spell Checker does with this name.</p><pre># Foreign name without errors<br>sample = &#39;We are going to meet Jowita in the city hall.&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;Moita&#39;, &#39;in&#39;, &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;.&#39;]}</pre><p>Well, the result is not very good, that’s because this model has been trained mainly with American English texts. At least, the surrounding words are helping to obtain a correction that is a name. We can do better, let’s see how.</p><h3>Updating a predefined word class</h3><h4>Vocabulary Classes</h4><p>In order for the Spell Checker to be able to preserve words, like a foreign name, we have the option to update existing classes so they can recognize more words as part of the class. In order to do that, we’re going to make use of the updateVocabClass() method, that will receive the name of the class, ‘_NAME_’ in our example,</p><pre># add some more, in case we need them<br>spellModel.updateVocabClass(&#39;_NAME_&#39;, [&#39;Monika&#39;, &#39;Agnieszka&#39;, &#39;Inga&#39;, &#39;Jowita&#39;, &#39;Melania&#39;], True)</pre><pre># Let&#39;s see what we get now<br>sample = &#39;We are going to meet Jowita at the city hall.&#39;<br>lp.annotate(sample)</pre><pre>{&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;Jowita&#39;, &#39;in&#39;, &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;.&#39;]}</pre><p>We included Jowita, the name of our foreign friend. Much better, right?</p><p>Now suppose that we want to be able to not only preserve the word, but also to propose meaningful corrections to the name of our foreign friend. Let’s see what happens when the ContextSpellChecker receives ‘Jovita’ instead of Jowita.</p><pre># Foreign name with an error<br>sample = &#39;We are going to meet Jovita in the city hall.&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;Jowita&#39;, &#39;in&#39;, &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;.&#39;]}</pre><p>Here, we were able to add the new word, Jowita, to the name class and have the ContextSpellChecker propose corrections for it, so the right correction, Jowita was obtained. Furthermore, the new word has been treated as a name, that meaning that the model used information about the typical context for names in order to produce the best correction.</p><h4>Regex Classes</h4><p>We can do something similar for classes defined by regexes. We can add a regex, to for example deal with a special format for dates, and again, that will not only preserve the date with the special format, but also be able to correct it, in case errors appear.</p><pre># Date with custom format<br>sample = &#39;We are going to meet her in the city hall on february-3.&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;her&#39;,  &#39;in&#39;,  &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;on&#39;, &#39;February&#39;, &#39;.&#39;]}</pre><pre># simple example with 3 months<br>spellModel.updateRegexClass(&#39;_DATE_&#39;, &#39;(january|february|march)-[0-31]&#39;)<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;her&#39;,  &#39;in&#39;,  &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;on&#39;, &#39;february-3&#39;, &#39;.&#39;]}</pre><p>Now our date wasn’t destroyed!</p><pre># now check that it produces good corrections to the date<br>sample = &#39;We are going to meet her in the city hall on febbruary-3.&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;We&#39;, &#39;are&#39;, &#39;going&#39;, &#39;to&#39;, &#39;meet&#39;, &#39;her&#39;,  &#39;in&#39;,  &#39;the&#39;, &#39;city&#39;, &#39;hall&#39;, &#39;on&#39;, &#39;february-3&#39;, &#39;.&#39;]}</pre><p>And the model produces good corrections for the special regex class. Remember that each regex that you enter to the model must be finite. In all these examples the new definitions for our classes didn’t prevent the model to continue using the context to produce corrections. Let’s see why being able to use the context is important.</p><h3>Sentence Level Corrections</h3><p>The Spell Checker can leverage the context of words for ranking different correction sequences. Let’s take a look at some examples,</p><pre># check for the different occurrences of the word &quot;ueather&quot;<br>example2 = [&quot;During the summer we have the best ueather.&quot;,\<br>            &quot;I have a black ueather jacket, so nice.&quot;,\<br>            &quot;I introduce you to my sister, she is called ueather.&quot;]<br>lp.annotate(example2)</pre><pre>&gt;&gt;</pre><pre>[{&#39;checked&#39;: [&#39;During&#39;, the&#39;, &#39;summer&#39;, &#39;we&#39;, &#39;have&#39;, &#39;the&#39;,   &#39;best&#39;, &#39;weather&#39;, &#39;.&#39;]},<br> {&#39;checked&#39;: [&#39;I&#39;, &#39;have&#39;, &#39;a&#39;, &#39;black&#39;, &#39;leather&#39;, &#39;jacket&#39;,<br>   &#39;,&#39;, &#39;so&#39;, &#39;nice&#39;, &#39;.&#39;]},<br> {&#39;checked&#39;: [&#39;I&#39;, &#39;introduce&#39;, &#39;you&#39;, &#39;to&#39;, &#39;my&#39;, &#39;sister&#39;, &#39;,&#39;,<br>   &#39;she&#39;, &#39;is&#39;, &#39;called&#39;, &#39;Heather&#39;, &#39;.&#39;]}]</pre><p>Now we’ve learned how the context can help to pick the best possible correction, and why it is important to be able to leverage the context even when the other parts of the Spell Checker were updated.</p><p>Unfortunately, the only way to customize how the context is used in ContextSpellChecker is to train the language model by training a new model from scratch. If you want to be able to train your custom language model, please keep an eye on my next posts.</p><h3>Advanced: Subword level corrections</h3><p>Another fine tuning that our Spell Checker accepts is to assign different costs to different edit operations that are necessary to transform a word into a correction candidate. So, why is this important? Errors can come from different sources,</p><ul><li>Homophones are words that sound similar, but are written differently and have different meaning. Some examples, {there, their, they’re}, {see, sea}, {to, too, two}. You will typically see these errors in text obtained by Automatic Speech Recognition(ASR).</li><li>Characters can also be confused because of looking similar. So a 0(zero) can be confused with a O(capital o), or a 1(number one) with an l(lowercase l). These errors typically come from OCR.</li><li>Input device related, sometimes keyboards cause certain patterns to be more likely than others due to letter locations, for example in a QWERTY keyboard.</li><li>Last but not least, ortographic errors, related to the writter making mistakes. Forgetting a double consonant, or using it in the wrong place, interchanging letters(i.e., ‘becuase’ for ‘because’), and many others.</li></ul><p>The goal is to continue using all the other features of the model and still be able to adapt the model to handle each of these cases in the best possible way. Let’s take the case of substitutions,</p><pre># sending or lending ?<br>sample = &#39;I will be 1ending him my car&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;I&#39;, &#39;will&#39;, &#39;be&#39;, &#39;sending&#39;, &#39;him&#39;, &#39;my&#39;, &#39;car&#39;]}</pre><pre># let&#39;s make the replacement of an &#39;1&#39; for an &#39;l&#39; cheaper<br>weights = {&#39;1&#39;: {&#39;l&#39;: .1}}<br>spellModel.setWeights(weights)<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;I&#39;, &#39;will&#39;, &#39;be&#39;, &#39;lending&#39;, &#39;him&#39;, &#39;my&#39;, &#39;car&#39;]}</pre><p>What happenede here? by default, the cost of any of the edit operations {substitution, deletion, insertion, exchange} is 1, by providing custom weights for individual edit operations we can change the cost for a candidate word for which that edit operation is involved, in this case we’re making the replacement with an l in lending cheaper than with an s in sending, thus lending is preferred.</p><p>The <em>weights</em> dictionary that we passed to the model is part of edit matrix, which is a larger structure, and as discussed earlier, the most appropriate content for this matrix depends on your application.</p><p>Unfortunately, sometimes, assembling this matrix by hand could be challenging. This is something to be soon included like an option during training for the Context Spell Checker. Stay tuned on new releases!</p><h3>Advanced: The Mysterious tradeoff parameter</h3><p>When producing a correction, a contextual spell checker has to balance two forces:</p><ul><li>The context information: by which the model wants to change words based on the surrounding words.</li><li>The word and subword information: by which the model wants to preserve as much an input word as possible to avoid destroying it.</li></ul><p>Changing input words that are in the vocabulary for others that seem more suitable according to the context is one of the most challenging tasks in spell correction. This is because you run into the risk of destroying existing ‘good’ words.</p><p>The models that you will find in the Spark-NLP library have already been configured in a way that balance these two forces and produces good results in most of the situations. But your dataset can be different from the one used to train the model. So we encourage the user to play a bit with the hyper parameters, and for you to have an idea on how it can be modified, we’re going to see the following example,</p><pre>sample = &#39;have you been two the falls?&#39;<br>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;have&#39;, &#39;you&#39;, &#39;been&#39;, &#39;two&#39;, &#39;the&#39;, &#39;falls&#39;, &#39;?&#39;, &#39;.&#39;]}</pre><p>Here ‘two’ is clearly wrong, probably a typo, and we would expect the model to be able to choose the right correction candidate according to the context. In order for the model to rely more on the context and less on word information, we have the setTradeoff() method.</p><p>You can think of the tradeoff as how much a single edition(insert, delete, etc) operation affects a candidate correction sequence whe compared to other sequences.<br>So the lower the tradeoff, the less we care about the edit operations in the word, and the more we care about the word fitting properly into its context. The range for this parameter will depend on how well the model was trained, but it’s still a reasonable rule is to expect it to be between 5 and 25. <br>Let’s play a bit with this parameter to relax the importance the model puts on individual words in our example,</p><pre>spellModel.getTradeoff()</pre><pre>&gt;&gt;</pre><pre>10.0</pre><pre># let&#39;s decrease the influence of word-level errors<br>spellModel.setTradeoff(5.0)</pre><pre>empty_ds = spark.createDataFrame([[&quot;&quot;]]).toDF(&quot;text&quot;)<br>lp = LightPipeline(pipeline.fit(empty_ds))</pre><pre>lp.annotate(sample)</pre><pre>&gt;&gt;</pre><pre>{&#39;checked&#39;: [&#39;have&#39;, &#39;you&#39;, &#39;been&#39;, &#39;to&#39;, &#39;the&#39;, &#39;falls&#39;, &#39;?&#39;, &#39;.&#39;]}</pre><p>Now we can see that the right correction was produced.</p><h3>Performance Issues</h3><p>The discussion about performance revolves around <em>error detection</em>. The more errors the model detects the more work it will have to do generating word candidates for corrections and ranking sequences.</p><p>Basically the error detection stage of the model can decide whether a word needs a correction or not; with two reasons for a word to be considered as incorrect,</p><ul><li><strong>The word is OOV:</strong> the word is out of the vocabulary.</li><li><strong>The context:</strong> the word doesn’t fit well within its neighbouring words.</li></ul><p>The only parameter that we can control at this point is the second one, and we do so with the setErrorThreshold() method that contains a threshold score above which the word will be considered suspicious and a good candidate for being corrected.</p><p>The parameter that comes with the pretrained model has been set so you can get both a decent performance and accuracy. For reference, this is how the F-score, and time varies in a sample dataset for different values of the errorThreshold,</p><pre>fscore | totaltime | threshold <br>------------------------------<br>52.69  |  405s     |   8f <br>52.43  |  357s     |  10f<br>52.25  |  279s     |  12f <br>52.14  |  234s     |  14f</pre><p>You can trade some minor points in accuracy for a nice speedup.</p><h3>Conclusion</h3><p>We have explored the Spell Checking problem, and we have seen why the context words are important for obtaining good corrections. Then we have seen how to build our own Spark NLP pipeline for spell checking. We have explored different mechanisms for customizing the ContextSpellChecker, and briefly touched on some advanced topics.</p><p>Finally, we explored how performance can be improved by compromising some minor points in accuracy.</p><h3>Next Steps</h3><p>Stay tuned for new blogposts where we will be exploring how to train your own models and how to use additional customization!.</p><p>Update: the new article about training a Spell Checker is out,<br><a href="https://medium.com/@albertoandreotti/training-a-contextual-spell-checker-for-italian-language-66dda528e4bf">https://medium.com/@albertoandreotti/training-a-contextual-spell-checker-for-italian-language-66dda528e4bf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c29c46963bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/spark-nlp/applying-context-aware-spell-checking-in-spark-nlp-3c29c46963bc">Applying Context Aware Spell Checking in Spark NLP</a> was originally published in <a href="https://medium.com/spark-nlp">spark-nlp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>