<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Roxane Bois on Medium]]></title>
        <description><![CDATA[Stories by Roxane Bois on Medium]]></description>
        <link>https://medium.com/@roxane.bois?source=rss-340374b4ecda------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V0TCJlEzzMwCI1LjVCWPqg.png</url>
            <title>Stories by Roxane Bois on Medium</title>
            <link>https://medium.com/@roxane.bois?source=rss-340374b4ecda------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:16:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@roxane.bois/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Grooming Detection Part. 3: A BERT Multiclass Model Experiment]]></title>
            <link>https://medium.com/besedo-engineering/grooming-detection-part-3-a-bert-multiclass-model-experiment-ddbd4ecda3c5?source=rss-340374b4ecda------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddbd4ecda3c5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[annotations]]></category>
            <category><![CDATA[training]]></category>
            <category><![CDATA[grooming]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Roxane Bois]]></dc:creator>
            <pubDate>Thu, 12 Jan 2023 15:31:45 GMT</pubDate>
            <atom:updated>2023-01-12T15:31:45.355Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Imagery of a little girl holding hands with a robot" src="https://cdn-images-1.medium.com/max/1024/1*-6HhemojYulOWvXm8Ku3gg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@askkell">Andy Kelly</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>In our last blog posts, we shared a Rule-base Matching solution to help you detect grooming in texts. This blog post is an experiment of grooming detection using a pre-trained deep learning model: BERT.</p><h3>What is BERT?</h3><p>BERT is an encoder model with an attention mechanism that permits generalization over data regarding language. Its model architecture works in two times:</p><p>(i) a <strong>pre-trained</strong> stage where the model learns language distribution on a large amount of data. This learning is based on two unsupervised tasks:</p><ul><li>a <strong>Masked Language Model (MLM)</strong> task where some tokens are randomly masked into sentences and predicted by the model. For instance: <em>“Artificial Intelligence aims to [MASK] human behaviour.”</em></li><li>a <strong>Next Sentence Prediction (NSP)</strong> task where the model predicts if two sentences follow each other using a binary response (yes/no). For instance, given the sentence <em>“Artificial Intelligence aims to simulate human behaviour.”</em>, the machine has to predict if the following sentence is the next: <em>“Example tasks in which this is done include speech recognition, computer vision, natural language processing.</em>”</li></ul><p>(ii) a <strong>fine-tuning</strong> stage where the pre-trained model classifies texts on a specific supervised task. An example of grooming detection will be to predict if a text contains grooming based on X labels.</p><p>To understand BERT processing and Transformers models in-depth, we suggest you check this excellent <a href="http://jalammar.github.io/illustrated-bert/">blog post</a>.</p><p>Tons of versions of BERT have already been developed in the community. The classic <a href="https://huggingface.co/bert-base-uncased">BERT Base</a> and <a href="https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad">BERT Large</a> models were trained on unpublished books and the English Wikipedia. Still, we also have models trained in other languages like <a href="https://huggingface.co/camembert-base">CamemBERT</a> or <a href="https://huggingface.co/bert-base-multilingual-cased">BERT Multilingual</a> and specific models like <a href="https://huggingface.co/google/mobilebert-uncased">MobileBERT</a>, <a href="https://huggingface.co/distilbert-base-uncased">DistilBERT</a>, and others. Beyond BERT, <a href="https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8">here</a> is a blog post that can help you understand the differences between some well-known Transformers models (RoBERTa, DistilBERT, XLNet).</p><h3>Grooming Detection Model with BERT</h3><p>If our main solution is rule-based right now to detect grooming, we wanted to test the performances of a pre-trained Deep Learning model on this task. We tested the easiest solution: a prediction text by text with only content body and labels to predict.</p><h3>Data Description</h3><p>We will use open-access data from <a href="https://zenodo.org/record/3713280">PAN12</a> and <a href="https://github.com/danielafe7-usp/BF-PSR-Framework/tree/main/Data">PJZC</a> (only the one-on-one conversations between groomers and non-groomers) plus additional client data. As a reminder, our client data contains social media profiles, dating website profiles, and a public in-game chat. The following table gathers information about each dataset we have.</p><figure><img alt="total client: 90 617, total open-access:151 605, total client+open-access: 242 222. Split ratio: 70, 15, 15." src="https://cdn-images-1.medium.com/max/725/1*JGlOGsxXMoqwGcvR8WocMw.png" /><figcaption>Number of texts in each split of our datasets (70%, 15%, 15%)</figcaption></figure><p>Our data has been re-annotated for our specific needs regarding client data and our Rule-Based Matching solution. We identified grooming as multiple topics occurring in the text that are easiest to detect in production and regarding our various data types. The combination of those topics may contain grooming risks. The annotation task has been defined as the following:</p><blockquote>👉 We define grooming as a process to approach, persuade and engage a child/teen (a victim) in sexual activities by using Internet.</blockquote><p>We have <strong>6 labels:</strong> 5 labels for <strong>risk of grooming</strong> (sexual ; underage ; contact information ; approach ; other) and 1 label for <strong>non-risk of grooming</strong> (ok).</p><ul><li>underage: texts where there might be a child or a teen involved in the conversation (ages below 21, age-related terms, age differences, about school or puberty stages)</li><li>sexual: texts about sex and sexual interests (sexual terms, body parts, sexual fantasies, sexual orientations, photo/nude sending, first-time discussions, sexual actions reframed as acceptable)</li><li>contact information: texts containing information about someone (full name, location, phone number) or contact requests (on other platforms, for instance)</li><li>approach: texts of people wanting or asking to meet someone in real life (haste of seeing someone, meeting organization)</li><li>other: another case, but the message is still suspicious and may contain grooming</li><li>ok: the text contains none of the above and doesn’t seems suspicious regarding grooming.</li></ul><p>By analyzing the other texts once a first annotation was done, we found out they contained more explicit grooming cases. Another label should be added to those cases. For now, as we had very few texts with that label, we only used them to enrich our grooming Matcher (see our previous blog post).</p><p>As we started this project with those 6 labels, we did not design our classification task with the presence/absence of grooming in a binary way. Instead, we designed a multiclass prediction with the labels defined above.</p><p>Know that annotation has been a long iterative process but still contains errors regarding its complexity. Before model training, we end up with a Cohen Kappa of 0.86 on open-access data and 0.82 on client data. These Cohen Kappa has been computed on a representative sub-sample of 1 544 texts from open-access data and 1 467 texts from client data.</p><h3>Configurations</h3><p>We tried a simple BERT-base model to predict grooming with the above classes. As a reminder, we classify texts and not conversations here. We decided to train three different models on three datasets: the first one only contains client data, the second one only has open-access data, and the third one is a mix of both (we call it “total” here). We fine-tuned each model regarding the macro-f1 score and also the f1 scores of classes that were not ok. We have the same test set for each model containing client and open-access data.</p><h3>Results</h3><p>The results of each model can be found in the table below. The model containing client and open-access data has better scores and better predicts each label. However, the client model would be even more interesting for prediction on client data only.</p><figure><img alt="CLIENT: f1 “ok”=0.96, f1 “sexual”=0.62, f1 “underage”=0.30, f1 “contact info”= 0.57, f1 “approach”=0.00 (8 texts in train). Macro avg.f1=0.61*. OPEN-ACCESS: f1 “ok”=0.95, f1 “sexual”=0.69, f1 “underage”=0.38, f1 “contact info”=0.39, f1 “approach”=0.33. Macro-avg.f1=0.55. TOTAL: f1 “ok”=0.97, f1 “sexual”=0.73, f1 “underage”=0.56, f1 “contact info”= 0.74, f1 “approach”=0.43. Macro avg.f1=0.69*." src="https://cdn-images-1.medium.com/max/1024/1*Dk16_h7rrYP0sjeoXn-j2A.png" /><figcaption>BERT results on our client data and open-access data</figcaption></figure><p>*<em>without </em><em>approach. This label was underrepresented in val (N=1), and test (N=2) sets: we thus decided it could be deleted from client data. The original F1 score was 0.49.</em></p><p>By checking the results on each dataset (client or open-access), we understand that the client data model has difficulty generalizing on open-access data. Surprisingly, the opposite is false: the open-access data model generalizes better regarding the scores and mismatches review.</p><p>By checking 100 mismatches from each prediction to better understand each model, we found that 50% of the texts were annotation errors and not prediction errors. This is encouraging as the machine is learning the classes well at some point. Here are some examples of correct predictions with wrong labels:</p><p>Open-access model: y_true = ok, y_pred = contact info</p><ul><li><em>plz send me contact no</em></li><li><em>kik me on my username x</em></li></ul><p>Client model: y_true = ok, y_pred = contact info</p><ul><li><em>I’m japanese but speak english and turkish &lt;youtube link&gt;</em></li><li><em>&lt;facebook link&gt;</em></li></ul><p>Total model: y_true = ok, y_pred = sexual terms</p><ul><li><em>you are hot</em></li><li><em>a bit nsa action with a hot lady</em></li></ul><p>Total model: y_true = ok, y_pred = child presence</p><ul><li><em>19/1.82/ …</em></li><li><em>… 16 years, teenwolf, witchcraft, rock, queen, nirvana</em></li></ul><p>The scores show that every model has less difficulty on ok labels: The remaining annotation errors are mostly in this class. This class contains a bit of every other class. That is why it can easily be predicted over others. Therefore, the ok scores are not that representative of model performances compared to other labels.</p><h3>Conclusion</h3><p>We shared with you some experimental yet encouraging results of a BERT model for grooming risk detection. A BERT model appears to generalize well on topics related to grooming.</p><p>For future work, a better way to define the classification task would be to design a multilabel classification. For instance, predict the presence/absence of underage, sexual, contact information and approach predict ok in case every previous label is absent.</p><p>Another idea would be to binary predictgrooming/non-grooming texts by merging the existing labels in 2 classes defined in our annotation task. Unfortunately, we lack important contexts to do so because we are processing very short chat texts at the message level and not full conversations. Plus, it is difficult to say whether a message contains grooming or not, even for a human (explicit cases are very rare). Defining the major steps in the grooming phenomenon (age-related conversations, sexual approaches, …) is an adaptive way to overcome this complexity of automation.</p><p>To push predictions, think of adding lots of False Positive in your non-grooming texts, as in PAN12 and PJZC. Namely, you can add locations that are not approaches, sexual interests that are not related to grooming, endearment between BFFs, and age-related conversations that are not related to grooming in your negative label 0.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddbd4ecda3c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/besedo-engineering/grooming-detection-part-3-a-bert-multiclass-model-experiment-ddbd4ecda3c5">Grooming Detection Part. 3: A BERT Multiclass Model Experiment</a> was originally published in <a href="https://medium.com/besedo-engineering">Besedo Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grooming Detection Part. 2: A Rule-base Matching Solution]]></title>
            <link>https://medium.com/besedo-engineering/grooming-detection-part-2-a-rule-base-matching-solution-fb800bfd9ab4?source=rss-340374b4ecda------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb800bfd9ab4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[grooming]]></category>
            <category><![CDATA[filters]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[spacy]]></category>
            <dc:creator><![CDATA[Roxane Bois]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 15:16:12 GMT</pubDate>
            <atom:updated>2023-01-11T15:16:12.969Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PkNcZb70Ncat-Ej7Ql3fvA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@asterfolio">Asterfolio</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>In a previous blog post, we introduced grooming detection with a state-of-the-art review and some existing models and datasets. Today, we will present an example of a Rule-base matching solution to detect grooming in chat messages.</p><h3>What is Rule-base Matching?</h3><p>Rule-base Matching aims to detect words and linguistic patterns in text. Detecting turns of phrases with words and patterns is an effortless yet efficient and precise way to filter texts. This solution alone can already give great results for some use cases, but they are even greater combined with model training or another kind of automation.</p><p>It can be done with regular expressions, but regex can be very time-consuming if you have lots of patterns and different topics you want to detect. Lexicons, dictionaries, and Natural Language Processing (NLP) pipelines can be very handy in helping you match specific features very fast. Here, we use <a href="https://spacy.io/usage/rule-based-matching">spaCy Matchers</a> to catch words and patterns not only on their surface form but also on linguistic features kept in spaCy tokens: we mainly use POS-tags and lemmas, but morphological and syntactic traits are also available.</p><h3>Task Definition</h3><p>Before developing any solution, you must define your task and dataset to find a better way to detect grooming.</p><h4>Know your Use-case</h4><p>Do you want to detect grooming in chat data or social media profiles? If you have chat data: is that one-on-one or a public chat? Is it a teenager community (in-game chat, for instance) or an adult community (like dating websites)? You can consider your detection based on the overall context by defining your task. For instance, if you are in an adult community, you can first detect underage users. If this is a teenage community, maybe sexual content or age-related conversations can give you a hint. That way, you will know how many things you want to detect and have a general idea of what to put in your Matcher(s).</p><p>You also need a general idea of how precise your Matcher will be. If it is used alone, you can start with general patterns and slowly precise them in the testing stage by checking what it catches in data. Suppose you have some human resources and expertise to review your filter results before making a final decision. In that case, you may be careful not to miss any suspicious texts (especially with grooming), so they can be sorted in a second step (i. e. target a high <strong>Recall</strong>). In that case, your Matcher can be more generic, depending on the time you have to review the results.</p><p>If your Matcher occurs in a specific pipeline, this is trickier:</p><ul><li>If it is placed in the first processing steps (for instance, as data augmentation to training a model on top of it), try to find a nice balance between <strong>Precision</strong> and <strong>Recall</strong> by focusing on the <strong>F1 score</strong>.</li><li>If it occurs in the last processing steps, maybe you can develop a more <strong>Precise</strong> filter with very specific words and patterns to filter “good” and “bad” results.</li></ul><p>Some more insights about Matcher’s evaluation will be given in the “<strong>How to Evaluate a Matcher</strong>” section down below.</p><h4>Linguistic Features and Data Exploration</h4><p>If you are lucky enough to have annotated data (or courageous enough to annotate it 😉), an optional yet very interesting step before developing your Matcher would be to explore your data. It can help you gain great information about what you want to detect. In our Grooming use case, you can start by splitting your dataset between grooming and non-grooming texts (depending on your labels). By only reading a sample of each label set, you can better understand what happens in your data. When you read it, try to focus on what regularly stands out of each set and gather some similarities in semantic fields. For instance, you can bring “<em>baby</em>,” “<em>bby</em>,” “<em>little princess</em>,” etc., into things related to endearment in grooming chats. Do not hesitate to bring out lots of examples into various categories: the more categories you have, the better it will be to merge or delete them if you need to.</p><p>You can also do some data analysis using NLP tools. For instance, extract linguistic features from each set of texts with <a href="https://spacy.io/">spaCy</a> or <a href="https://www.nltk.org/">NLTK</a> libraries and then compute some basic frequencies, word clouds, or even bi-grams and tri-grams analysis. Here is <a href="https://python-course.eu/applications-python/python-wordcloud-tutorial.php">a tutorial to create word clouds</a>. If you gather linguistic features in lists, here is a quick code to get frequencies using NLTK.</p><pre>#Import your libraries <br>from nltk import word_tokenize<br>from nltk.util import ngrams<br>from nltk import everygrams<br>from nltk.probability import FreqDist<br>nltk.download(&#39;punkt&#39;)<br><br>#Create your big token list. <br>#You can also build lists of spaCy POS tags, lemmas, NER etc.<br>tokens = []<br>for text in df_grooming[&quot;text&quot;]:<br>    text = nltk.word_tokenize(text)<br>    for token in text:<br>        tokens.append(token)<br><br>#Create your lists of unigrams, bigrams, trigrams and everygrams<br>bigrams = list(ngrams(tokens, 2))<br>trigrams = list(ngrams(tokens, 3))<br>everygrams = list(everygrams(tokens, 1, 5)) <br>#everygrams is a combination of every possible ngrams from 1 to 5 tokens<br><br>#Count list frequencies<br>freq_dict = FreqDist()<br>for gram in tokens: # &lt;-- you can change &quot;tokens&quot; with every list you have<br>    freq_dict[gram] += 1<br><br>#Print the 10 most common words<br>freq_dict.most_common(10)</pre><p>You can also compute collocates with different measures. Useful code to do so is mentioned in <a href="https://www.nltk.org/howto/collocations.html">NLTK documentation</a>.</p><p>You can also check occurrences of those words, bigrams, trigrams, and collocates to understand better what happens. At least some other features can be extracted very easily. For instance, think of text length in terms of the number of words, the number of symbols in texts (like “<em>?</em>” for questions, “<em>@</em>” for emails …), and things like that.</p><h3>Matcher Development</h3><p>Okay! Now that you know better what you want to detect and how to let’s jump into the development part.</p><h4>Words and Patterns Creation</h4><p>Once you know a bit better what you want to add to your filter, you just have to implement some patterns. <a href="https://medium.com/besedo-engineering/spacy-matchers-guidelines-c38207f884d7">This previous blog </a>post will teach you everything you need to know about spacy Matchers and how to create, export, and reuse your Matchers properly.</p><p>You can also enrich your filter by applying Word Embeddings methods. As an example, <a href="https://fasttext.cc/docs/en/support.html">FastText</a> processes out of vocabulary words (oov) like “<em>s3x</em>” instead of “<em>sex</em>.” It can be an interesting model to check for chat data that is often not normalized. <a href="https://techblog.ezra.com/different-embedding-models-7874197dc410">This blog post</a> can help you apply Word Embeddings methods to some words of your Matcher to find new, similar words. If you have some data normalization problems, this <a href="https://medium.com/besedo-engineering/nlp-attacks-part-1-why-you-shouldnt-trust-your-text-classification-models-37947853b1a5">blog post on NLP attacks</a> will suggest some solutions.</p><h4>Iterative Testings</h4><p>Once your Matcher is all set, you can finally test it on a simple prompt or a whole dataset. Again, all the code is available <a href="https://medium.com/besedo-engineering/spacy-matchers-guidelines-c38207f884d7">here</a>.</p><p>The next steps are kind of a rolling process: you test your patterns on your data, check the results and modify your Matcher regarding if you miss some word/patterns or if there is so much noise in the matches. This process is endless, and you need to decide when to stop, i. e. when you find the scores sufficient.</p><figure><img alt="words and patterns creation/modification &gt; test on data &gt; metrics computation &gt; (mis)matches checking &gt; repeat." src="https://cdn-images-1.medium.com/max/766/1*TcD1Zr_GW3NW990uAYfcwA.png" /><figcaption>The iterative process of filter development</figcaption></figure><h3>How to Evaluate a Matcher?</h3><p>Talking about evaluation now, how to properly check the quality of your Matcher? What metrics may be important?</p><h4>Data Annotation… or not</h4><p>What is great with Rule-Base Matching is that you don’t need annotated data to develop and test it. By checking the matched texts, you can already compute a <strong>Precision</strong> score. However, suppose you want a full understanding of your Matcher efficiency on your data. In that case, it is better to apply it to a representative sub-sample or even a whole dataset that has been annotated. That way, you will also have access to a <strong>Recall</strong> and an <strong>F1 score</strong> and thus have a solid quality check for your solution. Here is <a href="https://www.google.com/search?q=how+to+annotate+data+for+nlp&amp;client=ubuntu&amp;hs=wVs&amp;sxsrf=ALiCzsYIM3Gtkx5HkSpG3HUYyeRSF83wew%3A1671111443108&amp;ei=EyObY9mrBq2WlwTM27MI&amp;ved=0ahUKEwiZgcmf3_v7AhUty4UKHcztDAEQ4dUDCA8&amp;uact=5&amp;oq=how+to+annotate+data+for+nlp&amp;gs_lcp=Cgxnd3Mtd2l6LXNlcnAQAzIFCAAQogQyBQgAEKIEMgUIABCiBDIFCAAQogQ6CggAEEcQ1gQQsAM6CAghEMMEEKABOgcIIxCwAhAnOgoIIRDDBBAKEKABSgQIQRgASgQIRhgAUPoDWJ0IYJ0KaAFwAXgAgAGKAYgBywOSAQMxLjOYAQCgAQHIAQjAAQE&amp;sclient=gws-wiz-serp#fpstate=ive&amp;vld=cid:629def4b,vid:OqdPoWmRPBU">a quick guideline for data annotation for NLP</a>. You can also check <a href="https://medium.com/besedo-engineering/sentiment-analysis-part-3-data-annotation-d0845d1a8a1b">this blog post</a> on which we already went through the text annotation process.</p><h4>Useful Metrics</h4><p>The image below is a reminder of what are <strong>False Positives (FP)</strong>, <strong>False Negatives (FN), True Positives (TP),</strong> and <strong>True Negatives (TN)</strong>. The FP and FN represent what your Matcher is classifying wrong (on the picture, “False” is depicted in red and grey), while the TP and TN represent good results (the green parts are “True”). The circle of selected items represents the texts caught by the Matcher. The overall square represents the total amount of data. If data is not annotated, you will only have access to the circle (= the Matcher results).</p><figure><img alt="The circle contains a green part (TP) and a red part (FP). It is inside a bigger square that contains a green part (TN) and a grey part (FN). Precision = TP/(TP+FP). Recall = TP/(TP+FN)." src="https://cdn-images-1.medium.com/max/1024/1*ol_rZL0wd7U8xoOrErEDVQ.png" /><figcaption>A visualization of True Positive, True Negatives, False Positives and False Negatives</figcaption></figure><p>On that basis, you can compute a <strong>Precision</strong> and a <strong>Recall</strong> as shown in the image above. The <strong>F1-score </strong>is calculated based on those Precision and Recall scores:</p><figure><img alt="F1-Score = 2* ((Precision*Recall)/(Precision+Recall))" src="https://cdn-images-1.medium.com/max/458/1*fRlUJjap-r0_tzlWunNIyw.png" /></figure><p>An <strong>Accuracy</strong>, which is more a general score of what has been well classified, can also be computed with the following:</p><figure><img alt="Accuracy = (TP+TN)/(TP+TN+FP+FN)" src="https://cdn-images-1.medium.com/max/370/1*wwFWwP21t8be0ivoPwpu2Q.png" /></figure><h4>Mismatches Checking</h4><p>In addition to the scores, check some mismatches manually as a last step. It is the best way to understand your Matcher&#39;s strengths and limitations fully. For instance, if you do not have annotated data, you can take 100 matched texts to review and check which ones are TP or FP. If you have annotated data, you can quickly check each sub-sample of FN, FP, TN, TP ; focusing on the wrong results to improve later on.</p><h3>Use-Case: Detecting Grooming Risks in Social Media, Dating Profiles, and In-Game chats</h3><p>Our client data contains social media profiles, dating profiles, and in-game chats. To detect grooming on those, we developed 4 Matchers thanks to 100 000 texts from our clients + the 160 310 grooming texts in PAN12+PJZC (see our previous blog post):</p><ul><li>grooming contains explicit grooming formulations, public denunciations, and people searching for underage: <em>perv, pervert, you jerk, i’m looking for a young lady, are you home alone …</em></li><li>sexual contains explicit sexual terms: <em>ass, cock, vagina, fuck …</em></li><li>it is completed by sexual-related which contains more implicit sexual content that can be related to grooming. This Matcher contains subcategories like:</li><li>body parts: <em>hips, tongue …</em></li><li>photo sending: <em>send (0:2) photo, pic of you …</em></li><li>sexual emojis: 🍑, 😈 …</li><li>sexual fantasies: <em>in bed with you|me, cuddle (0:2) you|me …</em></li><li>sexual orientations: <em>gayteen, with a girl</em></li><li>sexual actions re-framed as acceptable: <em>touch it, take your/my flower, …</em></li><li>undressing: <em>take off your|my, pyjama, …</em></li><li>underage-21 detects underage users (less than 21 years old). This original Matcher contains pattern variations: <em>I be (0:2) 8:21 years? old?.</em></li><li>a second version of this Matcher underage-21_v2, has been developed to increase the recall and match some more things, even if there is a bit more noise in the results (variations of the pattern: <em>8:21 years old?).</em></li></ul><p>We detect not always proper grooming but suspicious messages that may contain grooming. Those suspicious texts are then sent to a manual queue to be reviewed. However, because moderators can’t verify many texts, we want to keep a high Precision. The combination of some filters can be interesting for some clients and use cases: for instance, underage-21 + sexual can detect underage users looking for sexual favours (useful in dating websites).</p><p>Here are the main results of the Matchers on a representative subset (N=3 011) of our client + open-access data. This subset has been re-annotated regarding if they contain underage users, sexual terms, or none of those.</p><figure><img alt="Grooming: 7 matches. Sexual: 313 matches, 563 texts, Acc=0.90, P=0.91, R=0.51, F1=0.65. Sexual related: 270 matches, 563 texts, Acc=0.82, P=0.55, R=0.26, F1=0.35. Underage 21: 67 matches, 266 texts, Acc=0.93, P=0.97, R=0.25, F1=0.39. Underage 21 (v2): 106 matches, 266 texts, Acc=0.94, P=0.85, R=0.34, F1=0.48." src="https://cdn-images-1.medium.com/max/1024/1*gEOlYzxtgWlNxB6dyuXvzA.png" /><figcaption>Our filter results to detect grooming risks</figcaption></figure><p>We can see that the sexual and the two underage filters are very precise, which is what we targeted here. Unfortunately, the sexual-related Matcher still contains lots of FP, but it allows us to catch some ambiguous texts that could contain grooming. Its <strong>Recall</strong> is very low because some of those texts are already caught by the sexual Matcher. The second version of underage seems to be better knowing we have manual review behind: despite the lower <strong>Precision</strong>, it stays acceptable regarding the higher number of matches and <strong>Recall</strong>. The grooming filter catches a very small amount of text. If we read them, they do not always imply grooming (it contains patterns related to <em>sugar babies</em> that an underage user does not always produce). This last filter works well on open-access data but needs some investigations to adapt to the client type containing too little chat data.</p><h3>Conclusion</h3><p>In this blog post, we shared the importance of task definition regarding Grooming Detection, as it is hard to handle. We also shared with you a way to develop and evaluate a Rule-base Matching solution applied to this topic.</p><p>Our first results are working well as we have human expertise behind them to review the results. Detecting grooming risks is already a great improvement as a human should review not every text.</p><p>However, if Rule-Base Matching is great for explicit turns of phrases that can be modelled in some words and patterns, it can’t generalize a phenomenon as you need to know exactly what to put in your Matcher. As we saw in our previous blog post, grooming is a phenomenon full of implied messages and formulations that are hard to catch. Language not only passes through words but also through context and world knowledge that helps us understand the meaning of a specific word or text. For instance, if a groomer says, “<em>Did you touch it in the shower. Lol,</em>” we know “<em>it</em>” refers to a sexual body part, but the machine does not have this knowledge. Detecting those kinds of texts by adding this linguistics knowledge to machines is the hard part of NLP.</p><p>As an extension to this solution, we trained a <a href="https://huggingface.co/docs/transformers/model_doc/bert">BERT</a> model (known for its language knowledge representation) to see to what extent the grooming phenomenon could be generalized by a transformer model used alone. This solution will be presented in our next and final blog post about grooming detection. <em>Stay tuned!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb800bfd9ab4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/besedo-engineering/grooming-detection-part-2-a-rule-base-matching-solution-fb800bfd9ab4">Grooming Detection Part. 2: A Rule-base Matching Solution</a> was originally published in <a href="https://medium.com/besedo-engineering">Besedo Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grooming Detection Part. 1: How to detect grooming in chat data?]]></title>
            <link>https://medium.com/besedo-engineering/grooming-detection-part-1-how-to-detect-grooming-in-chat-data-a1310694eaeb?source=rss-340374b4ecda------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1310694eaeb</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[grooming]]></category>
            <category><![CDATA[linguistics]]></category>
            <dc:creator><![CDATA[Roxane Bois]]></dc:creator>
            <pubDate>Tue, 10 Jan 2023 15:58:09 GMT</pubDate>
            <atom:updated>2023-01-10T15:58:09.115Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Imagery of a masked gamer on his computer" src="https://cdn-images-1.medium.com/max/1024/1*c6q6FOp112Ip4gCVu3pfDg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jakeschu">Jake Schumacher</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>The growing evolution of technology permits us to connect more easily with people worldwide. However, this broad connection is not without risks. Nowadays, young users are more and more connected online at an early age. Thus, it is important to protect them from bad encounters that can lead to grooming.</p><p>Grooming is a communication process in which an offender tries to engage an underage user in a relationship via the web, most of the time to perpetrate molestation [1][5]. At Besedo, user protection is a priority, especially regarding child protection. If chat message moderation presents sensitive yet important questions, we want to be prepared to handle this kind of danger for our clients and their users.</p><p>This blog post series about CSAM Detection aims to raise awareness about that phenomenon and suggest some technical tips to deal with it. This first blog post is specifically about significant state-of-the-art information about grooming and will give an overview of the existing datasets for grooming detection. <em>Let’s go!</em></p><h3>Two approaches in grooming detection</h3><p>In the state of the Art, we can find two kinds of papers about grooming detection: papers from the<strong> forensic community </strong>and papers from the <strong>data science community</strong>. The table below sums up the strengths and weaknesses of both, mentioned by Milon-Flores and Cordeiro in [2].</p><figure><img alt="Forensic approach: (+) behavioral features (-) not automated + analysis of finite conversations, when it is too late. Preventive approach: (+) automation, fast, in real time (-) filtered data, information lost in preprocessing." src="https://cdn-images-1.medium.com/max/1024/1*zUSVhcPAUVxMNAtGixX2nQ.png" /><figcaption>Pros and Cons of the Forensic Approach vs. the Preventive Approach to Detect Grooming</figcaption></figure><p>Even if our application is on the data science side, we found it very important also to gain information from forensics. Let’s go deeper with forensic studies first, as they can bring great insights into offenders’ and victims’ behaviours — which is what we want to detect in our data.</p><h3>How do offenders approach minors online?</h3><p>To summarize every piece of information we found in SOTA, we will bring you a worst-case scenario of what grooming implies. Several studies talk about 3 main stages in the grooming communication process, but we added a 4th regarding the high risks incurred by this last stage. We will use a male character here for convenience.</p><p><strong>1) Breaking the ice</strong>: Grooming can start as a normal conversation, with a simple “<em>Hello, what’s your name?</em>”. In that “breaking the ice” stage, the offender tries to gather personal information about his targeted victim: a (full) name, age, and location. This information seems commonplace but sharing them is dangerous for the victim. If the offender is an expert, he may even isolate the victim in a less moderated and private chat to remain discreet: <em>“Do you have Discord?”.</em> He can also pretend he’s a child: in that case, unfortunately, the offender is nearly impossible to detect…</p><p><strong>2) The deceptive relationship</strong>: After a few exchanges, a relationship takes place. The offender tries to get his victim’s trust. They may talk about parents, school, friends, and hobbies. He manipulates his victim so that nobody knows they are talking to an older person: isolation and secrecy are a big part of grooming. Exclusivity can occur with some “flirtatious” texts and imitating a strong, fusional relationship.</p><p><strong>3) Sexual intentions</strong>: Once the trust is built, discussions can effortlessly come around sexuality. Groomers tend to discuss some awful things like puberty stages and sexual fantasies. The offender might text like a young person discovering relationships and sexuality and/or ask lots of inappropriate questions (”<em>Have you ever kissed a boy?</em>”, <em>“what would you do if I do that?”</em>). They can also use sexual actions reframed as acceptable as they are talking to younger people (i. e., they are saying some things without clearly saying them, using some metaphors and analogies). Some very raw and harsh sexual messages can be found too.</p><p><strong>4) Real-life approach</strong>: the worst-case scenario would include a real-life meeting representing huge physical, emotional and psychological risks for the victim: sexual molestation and child abuse, which can also be recorded and shared online in paedophiles’ communities (leading to CSAM).</p><p>Of course, the conversations between offenders and underage are not always as tricky. Sometimes, groomers are sexually explicit in the first place, and it is thus easier to detect and stop them. But those forensic studies globally teach us that contact information, contact request on external chats, sexual words, and arranging of a real-life meeting may be key features to detect grooming in texts. In addition to that, know that in public chats, denunciations of other people may help to detect grooming too <em>“You perv,” “He’s a pedo,” “she’s like 14” </em>[4].</p><h3>Existing grooming datasets</h3><p>Most of the time, grooming appears in chat data, which is difficult to collect. In addition, child abuse is a very sensitive subject and a rare phenomenon in real-life data. For model training, an important study regarding grooming, say a representative dataset, would contain less than 4% of grooming conversations [1]. Now, where to find grooming data?</p><ul><li><a href="https://zenodo.org/record/3713280"><strong>PAN12</strong></a> is the most well-known grooming dataset. It was made for a grooming detection challenge in 2012. PAN12 contains binary chat messages labelled on the chat and message levels. It contains true negatives from <a href="https://netsplit.de/">IRC logs</a> (a computer science forum), false positives from <a href="https://www.omegle.com/">Omegle</a> (that may contain consensual sexual conversations), and true positives from <a href="http://www.perverted-justice.com/?con=full"><strong>Perverted Justice’s Archives</strong></a>, which seems to be the only grooming data publicly available since then.</li></ul><blockquote>📌 <strong>Perverted Justice’s Archives</strong> contains 1 on 1 conversations between real groomers and people pretending they are underage victims. Almost every grooming study is based on this data. However, keep in mind this dataset contains some biases: <br>· The dataset is ten years old and English language evolves very fast <br>· It is not real victims answering to the groomers but volunteers acting like<br>· Regarding PAN12: Despite the chat messages from Omegle as false positives, there is a huge gap between computer science conversations and one on one personal conversation containing grooming. <br>· Plus, it is possible that Omegle contains grooming?</blockquote><ul><li>In early 2022, Milon-Flores and Cordeiro [2] constituted a new grooming dataset using the same method as PAN12: <a href="https://github.com/danielafe7-usp/BF-PSR-Framework/tree/main/Data"><strong>PJZC</strong></a>. The authors took newer <a href="https://krijnhoetmer.nl/irc-logs/">IRC logs </a>data (2013–2022) as true negatives, “chit chats” conversations as false positives and the last grooming chats from PJ’s Archives (2013–2014) as true positives. This dataset is also binary annotated but only on the chat level.</li></ul><figure><img alt="PAN12 grooming texts (label on chat level) / PAN12 total texts: 139 740/2 669 388. PJZC grooming texts (label on chat level) / PJZC total texts: 20 570/372 161." src="https://cdn-images-1.medium.com/max/1024/1*oRFMYxRCobMQN1TG4DXPYQ.png" /><figcaption>Description of Open-access Grooming Datasets PAN12 and PJZC</figcaption></figure><p>Knowing this information, we think a nice and representative open-access dataset to start grooming detection would be the ones used in Milon-Flores and Cordeiro’s study, which mixes PJZC and PAN12. Its content is described in the table above. It is covering a wide period (2012–2022, but only 2012–2014 for grooming chats) without overlapping.</p><blockquote>💡 If you have some resources and time for data annotation, know that recent chats from teens communities can be interesting to detect grooming. For instance, we can think about <a href="https://discord.com/"><em>Discord</em></a> or <a href="https://fr.reddit.com/"><em>Reddit</em></a> which is not so moderated, or even <a href="https://www.tiktok.com/"><em>Tik Tok</em></a>’s and <a href="https://www.instagram.com/"><em>Instagram</em></a>’s comments. In-game or live platforms chats, social medias and dating websites in general might be interesting data, but their access is often restricted.</blockquote><h3>Processing chat messages</h3><p>About data science papers now, we find topics such as (e)SPD coming for (early) Sexual Predator Detection. Those papers aim to detect sexual predators in one-on-one chat messages, including groomers. An important question to tackle in your grooming detection is: should you classify messages or conversations? A clear decision should be made as chat messages are often very short, and more context is often needed to understand the overall meaning.</p><p>Suppose you decide to classify messages in the first place. In that case, the challenge with chat data is to add context to that only message, as we often can’t make a solid decision (say if this is grooming or not) based on only one text: the previous texts could also be taken into account. The field of Early Text Classification (ETC) proposes some solutions to represent this context. It focuses on dynamic representations of features and/or texts to process chat data. For instance, a quick and suitable representation of text would be to concatenate texts: this is the solution Milon-Flores and Cordeiro [2] used. If you do so, you will have the following data representation:</p><figure><img alt="“index”, “text”, “label”\n”0&quot;,”Hello baby, how are you?”,”grooming”\n”1&quot;, ”Hello baby, how are you? Hello”, ”grooming”\n”2&quot;, ”Hello baby, how are you? Hello Are you home alone?”, “grooming”" src="https://cdn-images-1.medium.com/max/819/1*otD-Vq2iTKIqcHy5g7PmLQ.png" /><figcaption>Chat messages with concatenated texts</figcaption></figure><p>But this solution can imply biases because the previous text is processed inside the actual text. Some context may be added using metadata as well :</p><figure><img alt="“index”, ”text”, ”chat history*”, “user logs**”, “label”\n”0&quot;, ”Hello baby, how are you?”, “”, “”, “grooming”\n”1&quot;, “Hello”, “Hello baby, how are you?”, “”, “grooming”\n”2&quot;, “Are you home alone?”, “Hello baby, how are you? Hello”, “Hello baby, how are you?”, “grooming”" src="https://cdn-images-1.medium.com/max/1024/1*dCOWVNMLV6-D__-2YRBQaw.png" /><figcaption>Chat messages with additional metadata</figcaption></figure><p>*<em>chat history: history of the overall conversation</em></p><p>**<em>user logs: history of the user messages</em></p><p>That way, all information can be considered in the training process, but not all the texts from the chat history and the user logs might be interesting for the final decision. Another solution would be to add or replace those additional columns with specific features representing the text or the history/logs content.</p><h3>Grooming Features</h3><p>Some Data Science studies use grooming features inspired by forensic studies to detect grooming. Those features can be detected with a rule-based matching solution. The results of the rule-based matching solution can be used for text augmentation. A random example is provided in the table below.</p><figure><img alt="“index”, “text”, “sexual_count”, “approach_count”, “parents_count”, “spans”, “label”\n”74&quot;, “I pick you up at 1:30”, “0”, “2”, “0”, “0”, “[“pick you up”, “1:30”]”, “grooming”" src="https://cdn-images-1.medium.com/max/1024/1*UZlCH0_UY65Ze2Rf_JRzKw.png" /><figcaption>Chat messages with grooming features</figcaption></figure><p>A rule-based matching solution can be used alone to filter potential grooming messages and then reviewed by human experts: no model is necessarily needed. Those features can also help in data annotation. Here is a non-exhaustive list of features that can help in grooming detection, with some examples. They come from several studies mentioned in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*KfUpRcxl-yw7QIlLIcnnYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*GooqF1UWHAFfYxuSRLYjfQ.png" /></figure><p>Know that some things may be prioritized in your task definition regarding grooming features. You can pick in the table the most interesting features regarding your data and what you want to detect. As an example:</p><ul><li>explicit grooming denunciation may be sufficient to detect grooming in public chats or live platforms (e. g.: <em>“Ok… pedo.”</em>)</li><li>On dating websites, people searching for underage partners (e. g.: <em>“I’m 35, (…) looking for a younger girl”) </em>and underage users interested in sexual exchanges — or not (e. g.: <em>“16yo gayteen 👅 Chat Hot </em>👀<em>”</em>) could be prioritized. You can also pay attention to the <em>sugar daddy/sugar mommy</em> phenomenon if <em>sugar babies</em> are underage.</li></ul><h3>[Bonus] Existing Open-access Models to Detect Grooming</h3><p>In addition to those datasets and grooming features, two models are already trained to detect grooming. They both classify conversations regarding if they contain grooming (label=1) or not (label=0).</p><ul><li>a CNN model trained on PAN12: <a href="https://github.com/kenjenlee/online-grooming-detector">https://github.com/kenjenlee/online-grooming-detector</a></li><li>some BERT models (Bert-base, Bert-large, and MobileBert) trained on <a href="https://paperswithcode.com/dataset/panc">PANC</a> [3]: <a href="https://gitlab.com/early-sexual-predator-detection/eSPD-lab/-/tree/master/">https://gitlab.com/early-sexual-predator-detection/eSPD-lab/-/tree/master/</a></li></ul><h3>Conclusion</h3><p>As we just saw, detecting grooming in data is a very challenging task. Not only it appears in chat data which contains very short texts with very little context, but there are few open-access resources around that sensitive subject online. PAN12 and PJZC are two datasets you can use in your grooming detection task: they contain Perverted Justice’s Archives of one-on-one conversations between real groomers and volunteers pretending they are victims.</p><p>To process chat messages, many configurations are possible. Important decisions would be to (i) choose to develop a rule-based solution, a classification model, both, or even another automation (ii) decide if you want to process entire chats or only messages, and (iii) predict the presence/absence of grooming or some more detailed features. For instance, contact information/requests, sexual content, age-related conversations, and real-life approaches may be key points of grooming. These are at least the steps in the communication process where underage users face huge risks.</p><p>We hope this blog post helped you understand grooming and how to start your detection project! In the next blog posts, we will share two examples of grooming detection and their results: Part. 2 presents a rule-base matching solution we tried and Part. 3 will jump into a BERT model experiment. <em>Stay tuned!</em></p><h3>References</h3><p>[1] Escalante and al. (2013). Sexual predator detection in chats with chained classifiers [<a href="https://www.researchgate.net/publication/304933788_Sexual_predator_detection_in_chats_with_chained_classifiers">paper</a>]</p><p>[2] Milon-Flores and Cordeiro. (2022). How to take advantage of behavioural features for the early detection of grooming in online conversations [<a href="https://www.sciencedirect.com/science/article/abs/pii/S0950705121011187">paper</a>]</p><p>[3] Vogt, Leser and Akbik. (2021). Early Detection of Sexual Predators in Chats [<a href="https://aclanthology.org/2021.acl-long.386.pdf">paper</a>]</p><p>[4] Lykoulas et Patsakis. (2020). Large-scale analysis of grooming in modern social networks. [<a href="https://arxiv.org/pdf/2004.08205.pdf">paper</a>]</p><p>[5] Gunawan and al. (2016). Detecting online child grooming conversation. [<a href="https://www.researchgate.net/publication/317692642_Detecting_online_child_grooming_conversation">paper</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1310694eaeb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/besedo-engineering/grooming-detection-part-1-how-to-detect-grooming-in-chat-data-a1310694eaeb">Grooming Detection Part. 1: How to detect grooming in chat data?</a> was originally published in <a href="https://medium.com/besedo-engineering">Besedo Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>