<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sofiene Jenzri on Medium]]></title>
        <description><![CDATA[Stories by Sofiene Jenzri on Medium]]></description>
        <link>https://medium.com/@sof.jenzri?source=rss-35c116e48896------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MAqQH1iU_rv3lUa3uEOvXg.png</url>
            <title>Stories by Sofiene Jenzri on Medium</title>
            <link>https://medium.com/@sof.jenzri?source=rss-35c116e48896------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:47:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sof.jenzri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I improved emails intention detection accuracy by leveraging attachments insights.]]></title>
            <link>https://medium.com/@sof.jenzri/how-i-improved-emails-intention-detection-accuracy-by-leveraging-attachments-insights-4d86aa4e18c0?source=rss-35c116e48896------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d86aa4e18c0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[contact-center]]></category>
            <category><![CDATA[email]]></category>
            <dc:creator><![CDATA[Sofiene Jenzri]]></dc:creator>
            <pubDate>Mon, 01 Aug 2022 11:24:23 GMT</pubDate>
            <atom:updated>2022-08-01T11:24:23.988Z</atom:updated>
            <content:encoded><![CDATA[<p>Today I will share and explain a model architecture for email categorization that combines email body and attached documents to accurately predict email’s intentions. My baseline would be a vanilla classification model that takes only the email body as input.</p><h3>Introduction and use cases</h3><p>Incoming email ingestion and processing can be a time-consuming and repetitive task especially for businesses that have large customer exposure. <br>Surprisingly, spontaneous emails remain a major communication channel for businesses, despite their tremendous digitalization efforts to streamline their customer relation processes (which include delivering website contact forms, chatbots etc.). <br>Use cases are very diverse and span across various industries. Furthermore, email senders are not limited to enterprises’ customers. Vendors, partners and public administrations also send emails daily. <br>Here are a few recurrent use cases:</p><ul><li>Customer claims management for insurance companies.</li><li>Customer complaints handling for enterprises in Telecom and utilities.</li><li>Various mortgage-related requests processing for banks.</li><li>Booking-related emails for travel industry companies.</li></ul><p>Automating the processing of incoming emails generally consists of (but not limited to) two major steps:</p><ol><li><strong>Email triage</strong>: Here we try to categorize the email by understanding its intention(s) and potentially reroute the email to a specialized business unit for handling.</li><li><strong>Information extraction</strong>: Depending on the identified intention, extracting a few pieces of information can be necessary for the downstream processes such as logging a customer case in a CRM system for tracking.</li></ol><p>In this article, I am going to focus on the intention detection part, and I am going to depict a few real-life scenarios to illustrate the challenges that the data scientist can encounter while designing an ML system to solve this task.</p><h3>ML Problem Framing</h3><p><strong>Scenario 1</strong>: Let’s imagine I am a health insurance customer, and I would like to claim a health reimbursement for medicine that I bought. I would then fill in the insurance reimbursement form and attach it to an email along with a pharmacy receipt and a bank ID document (to make sure the money is wired to the right bank account). I don’t bother to write a detailed email because I consider the attachments self-explanatory vis-à-vis my request.<br>Here is what my email would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*sDEw0nYUeWDZvb_2e5HKmQ.png" /><figcaption>Figure 1: insurance email example.</figcaption></figure><p>Let’s jump to the other side and put ourselves in the claim handler shoes. The later won’t get a precise understanding of the email’s intention until he opens the attachments and reads through them.</p><p>Hence, an effective email intention detection system should grasp knowledge from both email body and attachments content.</p><p><strong>Scenario 2</strong>: Let’s imagine I am customer of a bank and I have just moved to a new address. At the same time, I am unhappy about 50% extra service fees I paid last month for no evident reason. In that case, I would send an email to ask for address modification; and in the very same email I would inquire (and complain) about the unexpected extra service fee. <br>Here is what my email would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*EQvhWIy9Utg63ICCHx5OWg.png" /><figcaption>Figure 2: bank’s contact center email example</figcaption></figure><p>In this case, a multi-class classification system won’t be enough to handle the intention detection, so we can frame it as a multi-label classification problem.<br>However, in many real-life scenarios multi-label classification systems may not be the most effective choice to solve the intention detection task because of two reasons:</p><ol><li>It can be an overkill: Emails are in most cases about one main intention, sometimes they hold a secondary intention and in rare cases a third one.</li><li>It is hard to find labeled data that cover all multi-label combinations.</li></ol><p>An intermediate solution system can be a multi-task multi-class classification system that can predict the main intention and detect residual secondary and third intentions. Deep learning frameworks allow designing multi-output networks to achieve this. I will share in the next sections a detailed architecture of a DNN as an example of multi-task multi-class classification system.</p><h3>Architecture overview</h3><p>The ML system that I am presenting combines two sets of features: email body embeddings and a dense vector of features representing knowledge of the nature of the attachments. One way to obtain these attachment features is to train and apply a document classifier which outputs probability distributions over a predefined list of document types.</p><p>Let’s have a look at the model architecture and then share a few implementation highlights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y02Sgf2j1aFcYiedgnCqXQ.png" /><figcaption>Figure 3: Architecture overview of an intention detection ML model.</figcaption></figure><p>This architecture supposes two training steps (see <em>Figure 3</em>):</p><ol><li>Train the attached documents classifier (yellow layer)</li><li>Train the main model (green layers)</li></ol><p>The model outputs two classifications:</p><ol><li>The email main intention: a multi-class classification task. The email in <em>Figure 2</em> would be classified in “Address change” category as a main intention.</li><li>Presence of a request (which requires human handling): a binary classification task. The email in <em>Figure 2</em> would be classified as True because it does require a manual handling.</li></ol><p>PS: The model’s input features can be enriched with additional sources of information such as email subject, attachments names and other email metadata (sender category, email date etc.) which, depending on the use case, can bring predictive insights.</p><h3>Implementation</h3><h4><strong>Email body embeddings</strong></h4><p>There are multiple, more or less sophisticated, techniques that NLP offers to generate a vector representation out an email body text. <br>A “simple” approach would be to use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">TF-iDF</a> + dimension reduction using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA">PCA</a>. <br>There are more recent techniques such as Word2Vec and <a href="https://radimrehurek.com/gensim/models/doc2vec.html">Doc2Vec </a>that use shallow neural networks to learn word and text embeddings respectively. <br><a href="https://fasttext.cc/docs/en/support.html">Fasttext </a>offers an improved version of the initial Word2Vec algorithm and can be used to generate pretrained text embeddings as well.<br>However the most powerful and state-of-the-art technique uses transformers-based pretrained language models, like <a href="https://huggingface.co/docs/transformers/model_doc/bert">BERT</a>, to retrieve “context-aware“ text embeddings. This is what I have used in my experimentations as shown in <em>Figure 3</em>.<strong><br></strong>I used HuggingFace&#39;s pretrained <a href="https://huggingface.co/models">models </a>and <a href="https://huggingface.co/docs/transformers/index">APIs </a>to retrieve the email body text embeddings.<br>More precisely, since I tested my model on a French dataset, I used Camembert’s API that outputs bare hidden states. I average-pooled the last hidden states to obtain the text embeddings. But you can also try to use the second or third to last layers which are theoretically less dependent on the text corpus that the Language Model was trained on.</p><h4><strong>Attachments classifier</strong></h4><p>Before explaining how we obtain vector representations of attached documents, let me emphasis the motivation of this approach. After all, one might think why don’t we just concatenate the attachments’ text to the email body text, hence, ending up with a single input text which is much simpler. There are two main reasons why I think this method can lead to poor performance:</p><ol><li>Attachments text can be very large with a lot of superfluous content that is likely to overwhelm the more important subtle details present in the email body.</li><li>All what matters for the intention detection is the nature or type of the document, not the detailed content. And given a specific business process, documents usually fall into a predefined set of types.</li></ol><p>That’s why we need to train an attachments classifier to generate a dense vector representation of attachment types.</p><p><strong>A preliminary step of OCR</strong> applied to the attachment might be necessary if the document is an image or a scanned PDF.<strong><br></strong>I used UiPath’s <a href="https://docs.uipath.com/document-understanding/docs/introduction">Document Understanding</a> Framework which offers a prebuilt document classifier that I trained on my documents. This framework is an end-to-end document processing solution that natively offers OCR capabilities.<br>You can test this framework by signing on to the <a href="https://community.uipath.com/">community</a> edition of the UiPath platform.</p><p>Otherwise, you can build and train your TF-iDF document classifier and use an opensource OCR engine such as Tesseract for non-native PDFs.</p><p>Let N be the number of document types that we expect in our use case; and DC be the trained attachments classifier. <br>Given an email which has X attachments, DC will output for each attachment a stochastic vector holding the probability distribution over the document types. Since the final attachments vector representation needs to of a fixed length, in this case N (independent of X), we use max-pooling over the attachments dimension. <br>Here is a code snippet for generating the attachments vector representation for a given email:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/689477877c8b5c52746ee663396e92f6/href">https://medium.com/media/689477877c8b5c52746ee663396e92f6/href</a></iframe><h4>Classification model</h4><p>I used Tensorflow’s functional API to build a multi-branch neural network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6004930b2afa8d75235474ab292ed5c/href">https://medium.com/media/f6004930b2afa8d75235474ab292ed5c/href</a></iframe><p>I printed out the architecture using the Keras API <em>tf.keras.utils.plot_model</em>. Here is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*Q-xQLTiQOgdLXMWTgcaD2A.png" /><figcaption>Figure 4: Output of <em>tf.keras.utils.plot_model.</em></figcaption></figure><p>In the architecture of <em>Figure 4,</em> we supposed that we have as input:</p><ul><li>768-dimension email text embeddings</li><li>8 attachments document types</li></ul><p>and as an output:</p><ul><li>7 main intentions</li><li>1 secondary intention (request for information)</li></ul><h3>Training and evaluation</h3><p>I trained my model on a proprietary dataset of mortgage-related emails.<br>The dataset is comprised of 1100 emails dispatched unevenly across 7 main intentions.<br>The DNN that I trained has 227k trainable parameters (the green layers in <em>Figure 3</em>). I trained it for 50 epochs with batch size of 32. <br>I used a CPU which is enough given the dataset size. However, having a GPU can be of a great help to generate the email text embeddings and/or to apply OCR to the attachments; these two steps being the main bottlenecks of this ML system at both train and inference time.</p><p>I achieved 87% weighted F1-score average on the main intention classification task. When compared to a vanilla email-text-only classification model (which does not use the attachments), my model results in a 10% increase of weighted F1-score average.</p><h3>Summary</h3><p>Automating the incoming email ingestion is a challenging task. In most real-world scenarios, including insights from the email’s attached documents can significantly help understanding the sender’s intention.</p><p>An important part of the intention detection ML system (that was not covered in this article) is the data preprocessing. Indeed, it is worth emphasizing the importance of data preparation and cleaning, in this case email preprocessing and pruning, attachments selection etc., which greatly impact the model’s performance.</p><p>I hope you found this article useful. Please share your feedback/questions in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d86aa4e18c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>