Once again, IBM watsonx.ai teamed up with Watson NLP for a major upgrade of the Watson Data Science NLP capabilities. Read “teamed up” as: we grabbed the latest and greatest models from our NLP colleagues and sprinkled them generously across all our data science offerings:

• IBM Cloud Pak for Data (CPD) 4.7
• watsonx.ai
• IBM Cloud Pak for Data as a Service

Our goal was to tackle the three witches of NLP: bias, high compute expense and limited adaptability. Let’s find out how we fared…

“But wait, what’s Slate?”, you ask? Slate is a family of encoder-only large language models models built by IBM. You can’t use them for generative AI (i.e., they won’t write this blog entry for me) but they are fast and effective for enterprise NLP tasks like sentiment analysis, entity extraction and classification.

Fine-tune Slate models on your data — but try our already fine-tuned models FIRST

Our “base” Slate model is called pretrained-model_slate.153m.distilled_many_transformer_multilingual_uncased .
That’s quite a mouthful, so let’s tease the name apart: It’s a Slate model, i.e., an encoder-only model similar to RoBERTa , with 153 million parameters. We distilled knowledge from a larger Slate model into this one to keep it reasonably small, which improves inference times. Finally, it was trained on a multilingual, uncased corpus of data.

You can fine-tune this Slate model on your own data to create custom models for

• classification
• entity extraction
• sentence and document sentiment detection
• targeted sentiment detection

Here’s an example how to create a fine-tuned classification model:

import watson_nlp
from watson_nlp.blocks.classification.transformer import Transformer
from watson_core.data_model.streams.resolver import DataStreamResolver
training_data_file = "train_data.json"

# Step 1: create datastream from training data
data_stream_resolver = DataStreamResolver(target_stream_type=list, expected_keys={'text': str, 'labels': list})
train_stream = data_stream_resolver.as_data_stream(training_data_file)

# Step 2: load pre-trained Slate model
pretrained_model_resource = watson_nlp.load('pretrained-model_slate.153m.distilled_many_transformer_multilingual_uncased ')

# Step 3: Fine-train your model, based on the Slate model
classification_model = Transformer.train(train_stream, pretrained_model_resource)

Each fine-tuning follows the same three steps:

• Step 1: Prepare a JSON file with the training data. The format will differ, depending on your fine-tuning use case. For classification, each entry contains the text to classify, and its classification labels (plural, as we support multi-class classification).
• Step 2: Load the “base” Slate model.
• Step 3: Start training.

For all fine-tuning use cases:

• Expect to bring 1000 examples or more. This is significantly less data than what prior approaches required, especially for models that work across multiple languages. Still, it’s worth checking whether a simpler approach works just as well: Training an SVM ensemble can yield good classification results, and will be faster to train and apply to new data. Dictionaries and regular expressions don’t require any training data for entity detection, and are a good alternative to detect “simpler”, more regular, entities.
• Use a GPU environment for training. In watsonx.ai and Cloud Pak for Data as a Service, that’s the GPU notebook environments. They have the Watson NLP library pre-installed, just like the NLP Runtime 23.1 and NLP Runtime 22.2 environments. In Cloud Pak for Data, it’s the GPU environment that you can install as an add-on.

Fine-tuned NLP models available out-of-the-box

Before you embark on your fine-tuning exercise, check out the Slate models that we have already fine-tuned for several use-cases:

• entity extraction: entity-mentions_transformer-workflow_multilingual_slate.153m.distilled
• relationship detection: relations_transformer-workflow_multilingual_slate.153m.distilled
• sentence and document sentiment detection: sentiment-aggregated_transformer-workflow_multilingual_slate.153m.distilled
• targeted sentiment detection: targets-sentiment_transformer-workflow_multilingual_slate.153m.distilled

To make using them even easier, these models are available as workflows. They include all necessary preprocessing as part of the model. You just invoke the run method with your input text, like so:

import watson_nlp
# Load Target Sentiment model for English
targets_sentiment_model = watson_nlp.load('targets-sentiment_transformer-workflow_multilingual_slate.153m.distilled')

# Run the target sentiment model on the input text
targets_sentiments = targets_sentiment_model.run('The rooms are nice, but the bed was not very comfortable.')

# Print the targets with the associated sentiment
print(targets_sentiments)

In our previous NLP environment (NLP Runtime 22.2), you had to know that targeted sentiment required the output of syntax analysis, and add an additional intermediate step:

import watson_nlp

# Load Syntax and the Target Sentiment model for English
syntax_model = watson_nlp.load('syntax_izumo_en_stock')
targets_sentiment_model = watson_nlp.load('targets-sentiment_sequence-bert_multi_stock')

# Run the syntax model on the input text
syntax_result = syntax_model.run('The rooms are nice, but the bed was not very comfortable.')

# Run the targets sentiment model on the syntax results
targets_sentiments = targets_sentiment_model.run(syntax_result)

# Print the targets with the associated sentiment
print(targets_sentiments)

If you’re using NLP in Runtime 22.2 today, consider upgrading to Runtime 23.1 — you can remove the intermediate step from your code, and you’ll get an improved targeted sentiment model!

And if you’re missing the call to watson_nlp.download() in above code snippet — that step is no longer necessary in watsonx.ai and Cloud Pak for Data as a Service! We “pre-load” the NLP models now on our cloud environment for faster load times. You can still keep the download() method in your Runtime 22.2 notebooks, but remove that call when switching to Runtime 23.1 — it’s no longer supported there.

Reduced bias through careful training data selection and filtering

All large language models are trained on incredible amounts of data. Before trusting these models to work on your enterprise data, wouldn’t it be nice if:

• Your model provider carefully vetted and balanced all data sources that went into the model, and ensured that the sources’ intellectual property is respected according to its license?
• Your model provider removed hate, abuse and profanity (aka “HAP”) from the data before it went into training the model?

If you say “yes” to both, read on. Otherwise, I have a picture for you:

Above notebook snippet runs sentiment analysis on the same three sentences, on two models. One of them is our “Slate” sentiment model. I hope you can spot the difference — it’s our training data selection and HAP filtering at work!

Foundational models that do not require GPUs

Slate models are fairly powerful and small enough to run effectively in runtime environments as small as 1 CPU and 8G RAM — whether using them for batch inference through Jupyter Notebooks in watsonx.ai and Watson Studio or for online inference through Python functions in Watson Machine Learning on CPD.

To further improve their runtime characteristics, we offer our fine-tuned Slate foundation models in two variants:

• GPU-optimized: These models run in both GPU and CPU environments, with a significantly higher throughput in GPU environments (up to an order of magnitude).
• CPU-optimized: These models run only on CPUs. They need more memory to run, compared to the GPU-optimized variant. However, their throughput is higher, compared to running the GPU-optimized variant on a CPU. These models have a -cpu suffix, e.g. targets-sentiment_transformer-workflow_multilingual_slate.153m.distilled-cpu

So, if you have a GPU available, use our GPU-optimized models. In all other cases, use the CPU-optimized models (when available). Here’s an easy switch you can include in your notebooks to load the right model, based on the environment:

gpu_available = False
try:
    hw_spec = os.environ['RUNTIME_HARDWARE_SPEC']
    if 'num_gpu' in json.loads(hw_spec):
        gpu_available = True
except:
    pass

entity_model_name = 'entity-mentions_transformer-workflow_multilingual_slate.153m.distilled'
if gpu_available:
    entity_model = watson_nlp.load(entity_model_name)
else:
    entity_model = watson_nlp.load(entity_model_name + '-cpu')

This optimization is not limited to our fine-tuned models: if you fine-tune a Slate model, you can save the resulting model as either GPU-optimized or CPU-optimized — or you save the model twice, once per variant. Just specify the cpu_format parameter when saving the model:

from ibm_watson_studio_lib import access_project_or_space
wslib = access_project_or_space({...})

wslib.save_data('<model name>', 
                data=my_finetuned_model.as_bytes(cpu_format=True), 
                overwrite=True)

Let Watson NLP de-witch your text analytics challenges

So, did we achieve our goals? Let’s see…

• Bias: While there’s always more to be done, the new release of our NLP capabilities improves trust and transparency in our models. And further good news: we’ll bring the same level of trust to additional models, step by step.
• High compute expense: We tackled this from two angles — providing out-of-the-box fine-tuned models for specific use cases, so you don’t have to spend resources on fine-tuning yourself and making CPU-optimized models available that reduce the need for GPUs.
• Limited adaptability: In NLP Runtime 22.2, we only supported custom classification models. We now expanded this to custom entity, sentiment and targeted sentiment models, all based on the same Slate model.

I think I leave the verdict to you — try out our new NLP capabilities in watsonx.ai and let us know what we did well, and what we can improve!

We put our Watson NLP capabilities into IBM Cloud Pak for Data 4.6— at no extra charge. Here are the noteable changes and enhancements compared to Watson NLP in Watson Studio Notebooks in IBM Cloud that you’ve been using since January ’22 (right?). If you’re looking for a “Watson NLP Primer” before diving into this post: check here and there.

NLP in Watson Studio Notebooks: completely behind your firewall

The Watson NLP Python library is part of all Notebook runtimes in IBM Cloud Pak for Data (CPD) by default, whether you start your Notebook with Runtime 22.1 or Runtime 22.2. It’s also included in our JupyterLab runtimes — so you can use NLP in a Git-Based Project and in Python scripts, not just Notebooks. We keep the NLP capabilities between IBM Cloud Pak for Data and IBM Cloud Pak for Data as a Service in sync — so notebooks developed in the cloud will also run in CPD, and vice versa.

One side note: the NLP versions within Runtime 22.1 and Runtime 22.2 are different. Runtime 22.2 contains some NLP API updates, and newer built-in models. So, if you start your text analysis journey now, I recommend you start with Runtime 22.2 right away. Our documentation clearly states the changes between 22.1 and 22.2 (good news: there aren’t many).

One enhancement when running NLP in Cloud Pak for Data: you do not need to “download” NLP models before using them. For example, to load the English syntax model in CPD, use

# Load Syntax for English
syntax_model = watson_nlp.load('syntax_izumo_en_stock')

In CPD as a service, you need to download the model into your runtime first:

# Load Syntax for English
syntax_model = watson_nlp.load(watson_nlp.download('syntax_izumo_en_stock'))

In CPD, the built-in NLP models (the so-called “blocks”) are installed in a directory within the file system of your CPD cluster. This way, NLP works in so-called air-gapped CPD environments that don’t have internet access: everything you need for text analytics is right in your CPD installation, no further downloads required at runtime.

Speaking of installation: as the built-in NLP models are fairly large, they’re only installed when you explicitly request them —check our install docs on Runtime 22.2 and Runtime 22.1, respectively.

Deploy NLP models in Watson Machine Learning

The new feature I like most about Watson NLP in CPD: you can now take NLP models and deploy them in Watson Machine Learning. This includes both built-in and custom NLP models.

With this enhancement, you can now take your NLP models and wrap them into a web service to integrate them into your application (through a so-called online deployment), or apply the models regularly to batches of incoming data (aka batch deployment).

More often than not, you have to apply multiple models to the incoming text data. For example, you run a language detection model first, then the appropriate syntax model, followed by a sentiment model. This is why CPD allows you to deploy Python functions and Python scripts that contain your custom logic alongside the actual model invocation. Here’s an example:

def detect_sentiment():
    import watson_nlp
    
    syntax_model = watson_nlp.load('syntax_izumo_en_stock')
    sentiment_model = targets_sentiment_model = watson_nlp.load('targets-sentiment_sequence-bert_multi_stock')
    
    def score(input):
        scoring_prediction_out = []
        for input_data_row in input["input_data"][0]["values"]: 
            scoring_prediction_row = []
            for input_data in input_data_row:            
                syntax_result = syntax_model.run(input_data, 
                                                     parsers=('token', 'lemma', 'part_of_speech'))
                targets_sentiments = targets_sentiment_model.run(syntax_result)
                scoring_prediction_row.append(targets_sentiments.to_dict())
                
            scoring_prediction_out.append(scoring_prediction_row)
            
        # Score using the pre-defined model
        scoring_response = {
            'predictions': [{'fields': ['nlp_prediction'], 
                             'values': scoring_prediction_out
                            }]
        }

        return scoring_response
    
    return score

The aptly named detect_sentiment() is a Python function for sentiment detection that you can deploy in Watson Machine Learning. It follows the same structure as all Python functions you’ll create for NLP:

First, you load the relevant NLP models. This can be both built-in models, as shown in the example above, and custom models that you created. Built-in models are directly available via the load() method. Custom models need to be promoted into a deployment space first, from the project you created them in. This is explained in detail in our sample NLP project — check out the notebook Deploy Custom Classification Model once you imported the project into CPD.

Note: load these models outside the score() method. This way, WML will load these models only once, when the function is deployed, to ensure that the actual inference within score() is as fast as possible.

Then, you create the score() method to do the inference. This is very similar to applying NLP models within notebooks: you apply them in the required order on the input data. In our example, it’s first syntax analysis, then targeted sentiment extraction.

Lastly, you turn the NLP results into a Python dict, collect the results for each row of the incoming data, and return these results. You can then deploy the function directly from your Notebook, or from the CPD UI.

To test an online deployment of your NLP models, simply type in or upload some samples into our UI and check the results, as shown below:

To get started, check out our documentation on NLP capabilities and how to deploy NLP models. My recommendation: import our sample project, and execute its notebooks. They (hopefully) get you started on all capabilities of NLP in Cloud Pak for Data, including the creation of custom models and model deployment. Note: the directory contains the same project in two flavors: one for the NLP version in Runtime 22.1, and one for Runtime 22.2.

Including Watson NLP into Cloud Pak for Data is another milestone in our path to integrate NLP into the AI lifecycle. And we won’t stop there: watch this space for new innovations, new models, and new capabilities with Watson NLP and Cloud Pak for Data.

Managing machine learning features in Cloud Pak for Data today

A feature is a piece of data that serves as the input or target for a machine learning model. Data scientists spend a lot of effort constructing features such as In-Store Customer Revenue Last Month from ‘raw’ data sources. These features can be reused in multiple machine learning models — if they’re discoverable across the enterprise.

IBM Cloud Pak for Data provides a large variety of tools to construct features, including Jupyter notebooks, SPSS Modeler flows or Watson Pipelines. With its myriad of data connectors, it’s easy to store these features in the backend of your choice — you’re not tied to the backend of a particular “feature store”. And by adding these features to a catalog in Watson Knowledge Catalog, all data scientists can find them and use them in projects. So, we’re done, right? Not quite…

Feature Groups: Curated Data Assets for Machine Learning

We just enhanced feature management in IBM Cloud Pak for Data as a Service with Feature Groups. A Feature Group is a data asset that contains features for machine learning. Feature Groups extend the existing feature management capabilities in three ways:

1. Feature metadata that helps build models
2. Easily find features across the platform
3. Use feature metadata in Jupyter notebooks

Creating feature groups in projects and catalogs

When you open a data asset in one of your projects or catalogs, you will notice a new tab Feature group, with a small “beta” icon on the side. Initially, this tab will be empty. Press the New feature group button, select the data asset columns, enter some feature metadata, and your screen will start looking like this:

There are various metadata attributes you can set for each feature. I want to highlight three of them. See our documentation for the full list.

Role: Most features serve as Input for a machine learning model. If your feature group is curated training data, specify one or more features as Target, to make the “goal” of this feature group clear. Use the Identifier role for features that are not to be used as input for modeling, but that identify a particular row of data.

Recipe: If your feature represents a complex formula (such as customer lifetime value) or is the result of some advanced data wrangling, store the actual formula or the code snippet in the recipe. That way, others can reproduce your feature, and it’s transparent how a particular feature was derived.

Fairness information: For input features that may be susceptible to bias (such as Age or Gender), provide reference and monitored groups. For target features, provide favorable and unfavorable outcomes. IBM AutoAI and open source frameworks such as AIF360 use this information to detect and mitigate bias in the feature group and in the models built from it.

Finding feature groups

When searching for data assets in Cloud Pak for Data, you will notice a new filter option: Contains Feature Group. Use this filter to narrow your results from dozens of data assets that contain your search term down to feature groups, as shown below:

Starting with AutoAI, you will be able to select feature groups as input in Watson Studio model building tools:

The AssetFrame: create and use feature groups in Notebooks

Metadata is most useful in the tool you’re currently working with. That’s why we’ve created a new library in our Python notebooks called AssetFrame. It provides easy access to data assets and their feature group metadata. Here’s a quick look at what it can do:

Initialize an AssetFrame with the name of a data asset:

from assetframe_lib import AssetFrame
af = AssetFrame.from_data_asset("Retail Customers")

Show a sample of the data asset:

Have you noticed? The syntax is similar to pandas dataframes, but the result has two key improvements:

• You don’t need to load the full data asset into your notebook first — the AssetFrame retrieves the data “on the fly”, using the data connectivity capabilities of Cloud Pak for Data.
• The data is enriched with feature metadata: each column contains the role, monitored and reference groups are highlighted, as are favorable and unfavorable outcomes.

Create a feature group from a pandas data frame:

af = AssetFrame.from_pandas(dataframe=credit_risk_df, name="Retail Customers")
af.create_default_features()

risk_feat = af.get_feature("Risk")
risk_feat.set_roles(["Target"])

risk_feat.set_favorable_labels("No Risk")
risk_feat.set_unfavorable_labels("Risk")

af.to_data_asset()

The code snippet above takes your feature-engineered pandas dataframe credit_risk_df, turns all columns into features, and provides additional metadata for the target column Risk. It then creates a new data asset in your project containing the feature group metadata and your pandas dataframe.

Next steps

Should you turn all your data assets into feature groups? No. Here are good candidates for feature groups:

• Data for a specific use case — for example, manually annotated training data for supervised learning. You plan to reuse the data to build multiple models for the same use case.
• Data about a specific entity —for example, a customer, product information or monthly sales. You plan to reuse the data to build models for different use cases, typically joining it with other data first.
• Data that is the result of feature construction, and contains “higher-level” attributes such as customer lifetime value.

To get started with feature groups, see our documentation on the Feature Group UI and the AssetFrame to find out their complete capabilities.

We have also added a new Sample Project in the Watson Studio Gallery: Creating and using feature store data. In this project, you will find more examples of the AssetFrame library, including how to use a Feature Group to train an AutoAI model.

Start making your feature data “stand out” through feature groups — we’d love to hear your feedback!

I like the topic modelling capabilities of scikit-learn, especially Non-Negative Matrix Factorization, aka NMF. Many data scientists use LDA, but I found NMF to produce more clearly distinguishable topics when the underlying documents share some key terms (for example, social media posts about the same brand or product).

But be it LDA or NMF, the quality of topic models improves by reducing the number of words the algorithm has to consider. Stopword filtering is a start, but with Watson NLP, you can do even better!

Consider the excellent topic modelling example from scikit-learn. Using the Kulbach-Leibler divergence on a newsgroup dataset, NMF already produces some fairly good topics — but some topics contain terms like don or ve,or a lot of numbers, which makes the topics harder to understand.

These topics are the result of running the following parts of the topic modelling example:

Now, let’s “clean up” up this model. Fortunately, the Syntax block of Watson NLP integrates easily into scikit-learn. It allows me to

• Limit my analysis to words that have a particular part-of-speech, such as nouns or adjectives. For my topic models, I often use only nouns or proper nouns.
• Use the lemma of a word, instead of its surface form that appears in the text. This way, both singular (“mouse”) and plural forms (“mice”) of a word get mapped to the same form (“mouse”). Unlike word stems that stemming algorithms produce, lemmas are actual words that users of the topic model immediately understand.

To use Watson NLP in scikit-learn, I first create the custom tokenization function watson_tokenizer in my Notebook:

This function receives the text to tokenize. It then runs Watson NLP’s syntax model for English to obtain the individual tokens, the lemma of each token and its part-of-speech. Then, it collects the tokens with the part-of-speech tags Noun or Proper Noun. If the token has a lemma, it returns the lemma — otherwise, it returns the surface form. Running it yields

To integrate watson_tokenizer into scikit-learn, I specify it as tokenizer when creating the scikit-learn Vectorizer that is used to pre-process the text for topic modeling.

Then, I fit the NMF model based on the new IF-IDF values:

The change is minimal — but the result is arguably better:

To repeat: both results were achieved by running the scikit-learn topic modelling example within IBM Watson Studio, on the same document sample (2000 documents), with the same number of features (1000). All I changed was using a custom tokenizer instead of the scikit-learn stopword list.

This approach not only works for NMF topic modelling: I can use Watson NLP’s syntax block as a custom tokenizer in every scikit-learn Vectorizer that accepts a tokenizer attribute. This way, I can also improve LDA models, document clustering or text classification with the higher-quality text features from Watson NLP. However, for text classification, I recommend using Watson NLP all the way.-)

So, to give your topic models a nicer tan, go to IBM Cloud Pak for Data as a Service, launch a notebook with the NLP environment — and Welcome to Club T(r)opicana!

Clean up your topic models with Watson NLP was originally published in IBM Data Science in Practice on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since January, thousands of Watson Studio Notebooks have used the Watson NLP library for high-quality Natural Language Processing in more than 20 languages. I’m glad our users weren’t put off by my blog post on that subject.

It’s time for the next step: as of today, the Watson NLP integration is no longer in “beta” status, and ready to take on your production workload!

Deeper sentiment insights

We added a new model for sentence-level sentiment detection, based on BERT, with improved quality. We’ve also added a utility method that makes it easy to compute document-level sentiment, based on the sentiment of the individual sentences.

And my favorite: we included a new model to detect targeted sentiment. Targeted sentiment is an innovation straight out of IBM Research: it not only determines the sentiment (positive or negative) of a particular phrase, but tells you exactly what the sentiment is about. For example, the sentence “The location was great, but the rooms were dirty.” will yield two pieces of targeted sentiment: (location, SENT_POSITIVE) and (rooms, SENT_NEGATIVE) — without the need to specify targets in advance!

This allows for deep sentiment insights without any custom configuration, as the following example shows. I ran a set of hotel reviews through targeted sentiment — and the result clearly shows me the areas the hotel needs to improve on:

Support for GPUs

The Watson NLP library is now also included in all our GPU environments by default. This is specifically relevant if you want to train your own classification models based on CNN — you will see that the training time per epoch is reduced, compared to our CPU environments. Running on GPUs may also speed up inference time, i.e., when you apply your custom model to new data.

Getting Started

Using Watson NLP models is still easy as 1–2–3–4:

1. Start a Python notebook with the DO + NLP Runtime 22.1 on Python 3.9, or the GPU Runtime 22.1 on Python 3.9
2. Import the watson_nlp library
3. Load the block that contains the model you need
4. Apply the block to your text

We updated the block catalog in our documentation with all models you can use out-of-the box, and instructions of how to create your own models, including code samples.

We also updated our sample Project in the Watson Studio Gallery: Text Analysis with Watson Natural Language Processing. This project contains sample notebooks that show you how to work with each model, and examples how to further analyze and visualize the results for insights. Most of these notebooks are also available outside the project, directly in the Gallery. Just search for “NLP” in the Gallery to get them all.

Migrating from the NLP beta version

The good news: most of your notebooks will continue to work “as is” once you switched to the new environments. Here’s what you need to do:

• The NLP beta version will be supported for another two weeks. To ensure your notebooks continue to work after that, switch to an environment that contains the new NLP library.
• If you use the Default NLP beta environment: Change the environment of your Notebook from Default Python 3.8 + Watson NLP XS (beta) to DO + NLP Runtime 22.1 on Python 3.9 or GPU Runtime 22.1 on Python 3.9.
• If you use a custom NLP environment: Create a new custom NLP environment, using the software versions DO + NLP Runtime 22.1 on Python 3.9 or GPU Runtime 22.1 on Python 3.9. Change the environment of your notebook to this new custom environment.
• Sentiment analysis: Use the new blocks for sentence sentiment and targeted sentiment. The document sentiment block from the NLP beta is deprecated.
• Tone classification: Use the block name ensemble_classification-workflow_en_tone-stock instead of ensemble_classification-wf_en_tone-stock.
• Emotion classification: Use the block name ensemble_classification-workflow_en_emotion-stock instead of ensemble_classification-wf_en_emotion-stock.
• Custom classification models: We recommend retraining your models with the new NLP library. This will allow you to benefit from quality improvements we made, especially in the ensemble classifier.

As always, we’re looking forward to your feedback, and are excited to bring more innovations from IBM Research and IBM Software to your data science experience!

Watson Natural Language Processing: now generally available in IBM Watson Studio Notebooks was originally published in IBM Data Science in Practice on Medium, where people are continuing the conversation by highlighting and responding to this story.

High-Quality Natural Language Processing in more than 20 Languages

Call center records, customer complaints, social media posts, problem reports… text is everywhere in the enterprise and beyond. Most of the time, text data doesn’t stand alone: it’s part of a data record with many structured columns. And now, it’s easier than ever to extract meaning and structure from text and combine it with structured data. We’re launching Watson Natural Language Processing Environments for Watson Studio Notebooks. This capability is available in all Watson Studio plans in IBM Cloud Pak for Data as a Service, including our free trial.

The Watson Natural Language Processing Environment gives you instant access to pre-trained, high-quality text analysis models for over 20 languages in your notebooks. These models are created and maintained by experts in IBM Research and IBM Software and evaluated for quality for each language. The following models are provided out-of-the-box:

• Syntax Analysis: Identify tokens, their base forms (aka lemmas) and their part of speech. Run dependency parsing to identify the subject, verb, and objects of a sentence.
• Noun Phrase extraction: Identify noun phrases like “windscreen wiper” or “front bumper” from your text.
• Keyword retrieval: Rank noun phrases according to their relevance in a document.
• Entity Extraction: Extract 20 entity types from the text, including Organization, Person, Job Title, Location, Phone Number, Email Address, Measure, Money, Date and Time.
• Sentiment: Identify the sentiment of a text as positive, negative, or neutral.
• Tone: Classify a text as excited, frustrated, impolite, polite, sad, satisfied, or sympathetic.
• Emotion: Classify the emotion expressed in a text as anger, disgust, fear, joy, or sadness.

It bears repeating: these models don’t just work for English. We provide models that work in Arabic, Chinese, Czech, Danish, Dutch, German, English, Finnish, French, Hebrew, Hindi, Italian, Japanese, Korean, Norwegian, Portuguese, Romanian, Russian, Slovak, Spanish and Turkish — and for some models, we support even additional languages.

We placed special attention to the data used to train these models. This includes the quality of the data, the data provenance, and its intellectual property. The value for you: the models we built from this data can be used in your production environment, without worrying about license or intellectual property infringements — unlike some models you may find for open-source text analysis frameworks.

In case our built-in models don’t cover all your analysis requirements — feel free to build your own custom models, using:

• A high-performance matcher for dictionaries and regular expressions. Thanks to our built-in syntax analysis, you can put lemmas into the dictionary, and we match all inflected forms automatically. So, you add “mouse” — we find “mice”.
• GloVE, USE and BERT embeddings
• Custom classifiers with your own training data. You can use SVM, CNN and BERT approaches — or simply create an ensemble classifier that combines various models

You can store these custom models in your Watson Studio project, and reuse in other notebooks.

Integrated into the Cloud Pak for Data Experience

Extracting information from text is just the first step. To deliver insights, you want to integrate the results into your data fabric — and automate the insights going forward. Fortunately, Watson Studio has got you covered:

Turn text analysis into text mining

Once the NLP results are in your notebook, use the full array of “structured” analytics approaches to identify patterns across documents. Or correlate the results with other, structured columns in your data. This yields additional insights that go beyond a single piece of text.

For example, run syntax analysis to identify noun phrases that occur in car problem descriptions, and then use association rule analysis to identify the key noun phrases for a particular car model.

Or correlate the results of syntax analysis and sentiment analysis to identify the aspects that drive positive or negative sentiment for a particular hotel

Visualize and share insights

Visualize results right in your notebook (like the charts you see above) and share a link to your notebook with your stakeholders or create dashboards to let them interactively explore the results.

Productize and Automate

Run your notebook as a job to apply text analysis regularly on new data. Use AI Orchestration to combine the notebook with other tools, such as AutoAI.

Getting Started

Using Watson NLP models is easy as 1–2–3–4:

1. Start a Python notebook with the Default Python 3.8 + Watson NLP XS (beta) environment
2. Import the watson_nlp library
3. Load the block that contains the model you need
4. Apply the block to your text

Our extensive documentation includes a block catalog with all models you can use out-of-the box, and instructions how to create your own models, including code samples.

We also added a new Sample Project in the Watson Studio Gallery: Text Analysis with Watson Natural Language Processing. This project contains sample notebooks that show you how to work with each model, and examples how to further analyze and visualize the results for insights — including the ones from this blog.

We’re looking forward to your feedback — so, just plug in your own data and get going!