<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sinequa on Medium]]></title>
        <description><![CDATA[Stories by Sinequa on Medium]]></description>
        <link>https://medium.com/@sinequa?source=rss-c033ea71a12a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*xYi7zT4tWD2dlS5SAtA96A.png</url>
            <title>Stories by Sinequa on Medium</title>
            <link>https://medium.com/@sinequa?source=rss-c033ea71a12a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:11:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sinequa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Azure AD/Azure Graph — Getting an App Registration as Owner of another one]]></title>
            <link>https://sinequa.medium.com/azure-ad-azure-graph-getting-an-app-registration-as-owner-of-another-one-60c0a25cfcb6?source=rss-c033ea71a12a------2</link>
            <guid isPermaLink="false">https://medium.com/p/60c0a25cfcb6</guid>
            <dc:creator><![CDATA[Sinequa]]></dc:creator>
            <pubDate>Wed, 22 Feb 2023 22:56:31 GMT</pubDate>
            <atom:updated>2023-02-23T08:03:00.181Z</atom:updated>
            <content:encoded><![CDATA[<h3>Azure AD/Azure Graph — Getting an App Registration as Owner of another one</h3><p><em>Written by Romain Leonard, Chief Information Security Officer @ </em><a href="https://www.sinequa.com/"><em>Sinequa</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OWQct20HYjDcHBe4Dk8PQ.jpeg" /></figure><p>For some reason, we have an App Registration which is used to create other App Registrations and manage them.</p><p>When the Owner App creates the Owned App, everything is fine and the Owner App is automatically set as the owner of the newly created App.</p><p>The issue we had was with an App created during a POC, we wanted to manage the same way as the other ones.</p><h3>The Issue</h3><p>Neither the Interface, the command line, nor the Graph API allows you to set an App Registration as the Owner of another App.</p><h3>Pre-Requisites</h3><p>To illustrate this Article, I created the following 2 apps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D3VN5m4QwSuIvLcs" /><figcaption>The Owner</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VdeP31dwByzIHnKR" /><figcaption>The Owned</figcaption></figure><h3>Failures</h3><h4>Using the Portal</h4><p>While trying to add the Ownership through the portal, the MyOwnerApp cannot be found in the Interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LGLH9Mu0LKeD4Rq4" /></figure><h3>Using the Command Line</h3><p>The following command line was built with both Apps’ object ids:</p><blockquote>az ad app owner add — id 68d9a681-c32e-448c-9809-f6c5f9f5c065 — owner-object-id d78d52dc-f188–48ae-9968-d493fd2ebf62</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1azLzfv6GtI0DgxV" /></figure><p>The following command line was built with Owned App’s object ids and the Owner App Id:</p><blockquote>az ad app owner add — id 68d9a681-c32e-448c-9809-f6c5f9f5c065 — owner-object-id bd05e895–46f4–41e5–9f79–8ac8f0db8301</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IMdfj6dOvf5UC54o" /></figure><h3>Using the GraphAPI</h3><p>By using the documentation, we tried to do the following call to the API:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q5EUOmXk6lBJdynZ" /><figcaption>Doc: <a href="https://learn.microsoft.com/en-us/graph/api/application-post-owners?view=graph-rest-1.0&amp;tabs=http#request">https://learn.microsoft.com/en-us/graph/api/application-post-owners?view=graph-rest-1.0&amp;tabs=http#request</a></figcaption></figure><h3>The Solution</h3><p>By looking at the Apps which were owned by our Owner App, we identified that the application was not the Owner, it is the associated ServicePrincipal which is.</p><p>So first, we found a way to get the associated Service Principal ID of our owner App:</p><blockquote>az rest — URL “https://graph.microsoft.com/v1.0/servicePrincipals(appId=&#39;bd05e895-46f4-41e5-9f79-8ac8f0db8301&#39;)&quot; — query ‘id’ — output tsv</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*be390Z-XQKYIk-32" /></figure><blockquote>az ad app owner add — id 68d9a681-c32e-448c-9809-f6c5f9f5c065 — owner-object-id eb8c8758–18e0–46c1-b707-ad8de59c86ca</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wzgZ_Su6P8jsN-WO" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/0*qgsJ2Ffj3isklcd1" /></figure><p><strong>About the Author</strong></p><p>Romain Leonard is the Chief Information Security Officer at Sinequa, where he leads all the activities related to the security of Sinequa. Romain is passionate about cyber security and intelligent search technologies. He has over 10 years of experience working in the cybersecurity area where he held various positions, such as penetration tester, security engineer, and CISO.</p><p><strong>About Sinequa</strong></p><p>Sinequa’s Search Cloud brings organizations of all sizes the most complete enterprise search ever. Customers employ the Sinequa Search Cloud to connect all content (both text and data), derive meaning, learn from user interactions, and present information in context. This solves content chaos and informs employees through a single, secure interface. They get the knowledge, expertise, and insights needed to make informed decisions and do more, faster. Sinequa helps these organizations accelerate innovation, reduce rework, foster collaboration, ensure compliance, and increase productivity. For more information visit <a href="https://cts.businesswire.com/ct/CT?id=smartlink&amp;url=https%3A%2F%2Fwww.sinequa.com%2F&amp;esheet=52614385&amp;newsitemid=20220323005136&amp;lan=en-US&amp;anchor=www.sinequa.com&amp;index=4&amp;md5=a28f7ecec0d79fd01a3db529f8606e38">www.sinequa.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60c0a25cfcb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Automation Runbook in Python: A Definitive Guide]]></title>
            <link>https://sinequa.medium.com/how-to-create-a-python-runbook-1aa586dc6d26?source=rss-c033ea71a12a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1aa586dc6d26</guid>
            <dc:creator><![CDATA[Sinequa]]></dc:creator>
            <pubDate>Fri, 21 Oct 2022 17:02:52 GMT</pubDate>
            <atom:updated>2022-10-25T07:44:56.344Z</atom:updated>
            <content:encoded><![CDATA[<h3>Azure Automation Runbook in Python and User Assigned Managed Identity</h3><p><strong>How to execute a Python runbook with a User Assigned Managed Identity.</strong></p><p><em>Written by Romain Leonard, Chief Information Security Officer @ </em><a href="https://www.sinequa.com/"><em>Sinequa</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gD5l0uEtQu_XSyg5zstg8g.jpeg" /></figure><p>PowerShell Runbooks are great, but when running multiple instances over multiple subscriptions, the lack of subscription parameters on some commands, such as the remote execution one, causes issues.</p><p>On the other hand, with the Python SDK, there is no issue with multiple subscriptions, but … While executing PowerShell runbooks with a user Assigned Managed Identity is well documented, I could not find any information on Python.</p><h3>Background</h3><h3>The problem</h3><p>We have an automation account that is used to schedule some actions on the VMs.</p><p>This automation account is central and must command VMs in multiple subscriptions.</p><h3>Our issues</h3><p>Some jobs fail because the Powershell context of one instance overrides the context of concurrent ones.</p><h3>Identified solutions</h3><p>We identified a few possible solutions to solve these problems:</p><ol><li>Running the runbooks at different times for each subscription: Painful</li><li>Creating an automation account per subscription: Painful</li><li>Using Python runbooks : Should be fun</li></ol><h3>First Python runbooks</h3><h3>Using and Importing Azure CLI</h3><p>The Azure documentation (<a href="https://learn.microsoft.com/en-us/azure/automation/python-3-packages">here</a>) contains a runbook that can import a package and its dependencies. If you are new to Python runbooks, you might struggle with it, as I did.</p><p>It is pretty easy to use; they just forgot to explain how to set the prerequisites. You need to set a Connection and a Certificate to point to an App Registration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/0*IOGMa_A3QVqz5F_I" /></figure><p>So, I simplified it and allowed it to be run by stage directly from my environment.</p><pre>#!/usr/bin/env python3<br><br># imports<br>import requests<br>import sys<br>import pip<br>import os<br>import re<br>import json<br>import getopt<br>import time<br><br>import azure.identity<br><br># Constantes<br>PYPI_ENDPOINT = &quot;<a href="https://pypi.org/simple">https://pypi.org/simple</a>&quot;<br>FILENAME_PATTERN = &quot;[\\w]+&quot;<br>DOWNLOAD_DIR=&quot;downloaded&quot;<br><br><br>def get_azure_token():<br>    creds = azure.identity.AzureCliCredential()<br>    return creds.get_token(&quot;<a href="https://management.azure.com/">https://management.azure.com/</a>&quot;).token</pre><pre><br>def get_packagename_from_filename(packagefilename):<br>    match = re.match(FILENAME_PATTERN, packagefilename)<br>    return match.group(0)<br><br><br>def resolve_download_url(https://medium.com/packagename,%20packagefilename):<br>    response = requests.get(&quot;%s/%s&quot; % (PYPI_ENDPOINT, packagename))<br>    urls = re.findall(r&#39;href=[\&#39;&quot;]?([^\&#39;&quot; &gt;]+)&#39;, str(response.content))   <br>    for url in urls:<br>        if packagefilename in url:<br>            return(url)<br><br><br>def download_package_with_dependencies(packagename):<br>    pip.main([&#39;download&#39;, &#39;-d&#39;, DOWNLOAD_DIR, packagename])<br><br><br>def get_pkgs_from_downloaded_packages():<br>    out = {}<br><br>    for filename in os.listdir(DOWNLOAD_DIR):<br>        pkgname      = get_packagename_from_filename(filename)<br>        out[pkgname] = filename<br><br>    return out<br><br><br># TODO handle pagination<br>def send_webservice_list_module_request(subid, rg, acct, token):<br>    request_url = &quot;https://management.azure.com/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Automation/automationAccounts/%s/python3Packages/?api-version=2018-06-30&quot; \<br>                  % (subid, rg, acct)<br><br>    headers     = {&#39;Content-Type&#39; : &#39;application/json&#39;, &#39;Authorization&#39; : &quot;Bearer %s&quot; % token}<br>    r = requests.get(request_url, headers=headers)<br>    if str(r.status_code) not in [&quot;200&quot;, &quot;201&quot;]:<br>        raise Exception(&quot;Error listing packages for Automation account. Error code is {0}&quot;.format(str(r.status_code)))<br><br>    out  = []<br>    data = json.loads(r.text)<br>    for pkg in data[&#39;value&#39;]:<br>        out.append(pkg[&#39;name&#39;])<br><br>    return out<br><br><br>def send_webservice_import_module_request(subid, rg, acct, token, pkgname, pkgurl, retry=True):<br>    request_url = &quot;https://management.azure.com/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Automation/automationAccounts/%s/python3Packages/%s?api-version=2018-06-30&quot; \<br>                  % (subid, rg, acct, pkgname)<br><br>    requestbody = { &#39;properties&#39;: { &#39;description&#39;: &#39;uploaded via automation&#39;, &#39;contentLink&#39;: {&#39;uri&#39;: &quot;%s&quot; % pkgurl} } }<br>    headers     = {&#39;Content-Type&#39; : &#39;application/json&#39;, &#39;Authorization&#39; : &quot;Bearer %s&quot; % token}<br>    r = requests.put(request_url, data=json.dumps(requestbody), headers=headers)<br>    print (&quot;    Request status for package %s was %s&quot; % (pkgname, str(r.status_code)))<br>    if str(r.status_code) not in [&quot;200&quot;, &quot;201&quot;]:<br>        if retry:<br>            print(&#39;Waiting 20s before retry&#39;)<br>            time.sleep(20)<br>            send_webservice_import_module_request(subid, rg, acct, token, pkgname, pkgurl, False)<br>        else:<br>            raise Exception(&quot;Error importing package {0} into Automation account. Error code is {1}&quot;.format(pkgname, str(r.status_code)))<br>    time.sleep(6)<br><br><br>if __name__ == &#39;__main__&#39;:<br>    if len(sys.argv) &lt; 9:<br>        raise Exception(&quot;Requires Subscription id -s, Automation resource group name -g, account name -a, and module name -g as arguments. \<br>                        Example: -s xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx -g contosogroup -a contosoaccount -m pytz &quot;)<br><br>    # Process any arguments sent in<br>    subid   = None<br>    rg      = None<br>    acct    = None<br>    pkgname = None<br><br>    opts, args = getopt.getopt(sys.argv[1:], &quot;s:g:a:m:&quot;)<br>    for o, i in opts:<br>        if o == &#39;-s&#39;:  <br>            subid = i<br>        elif o == &#39;-g&#39;:  <br>            rg = i<br>        elif o == &#39;-a&#39;: <br>            acct = i<br>        elif o == &#39;-m&#39;: <br>            opkgname = i<br><br>    print(&#39;== Downloading packages&#39;)<br>    download_package_with_dependencies(opkgname)<br><br>    print(&#39;\n== Identifying packages url&#39;)<br>    tpkgs = get_pkgs_from_downloaded_packages()<br>    #for k,v in tpkgs.items():<br>    #    print(&#39;  - %s: %s&#39; % (k, v))<br><br>    print(&#39;\n== Getting azure token&#39;)<br>    token = get_azure_token()<br><br>    print(&#39;\n== List already deployed packages&#39;)<br>    cpkgs = send_webservice_list_module_request(subid, rg, acct, token)<br><br>    print(&#39;\n== Compare package lists&#39;)<br>    pkgs = set(tpkgs.keys()) - set(cpkgs)<br><br>    print(&#39;\n== Importing package to the automation account&#39;)<br>    e = []<br>    i = 1<br>    l = len(pkgs)<br>    for pkgname in sorted(pkgs):<br>        print(&#39; -&#39;, pkgname)<br>        if i % 5 == 0:<br>            print(&quot;===&gt; %d / %d&quot; % (i, l))<br>        i += 1<br><br>        pkgurl = resolve_download_url(https://medium.com/pkgname,%20tpkgs[pkgname])<br>        if pkgurl is None:<br>            e.append(pkgname)<br>            print(&#39;!!! Unable to identify URL for %s&#39; % pkgname)<br>            continue<br><br>        send_webservice_import_module_request(subid, rg, acct, token, pkgname, pkgurl)<br><br>    print(&#39;\n== Packages with errors&#39;)<br>    print(e)</pre><p>Note: The import must be done from a Windows computer with the right python version :(</p><p>I used the following runbook:</p><pre>#!/usr/bin/env python3<br># -*- coding: utf-8 -*-<br><br># Import<br>from azure.cli.core import get_default_cli<br><br># Functions<br>def az_cli_run_safe(args):<br> cli = get_default_cli()<br> cli.invoke(args)<br><br> if cli.result.result:<br>     return cli.result.result<br> elif cli.result.error:<br>     raise cli.result.error<br> return True<br><br>def az(cmd):<br> return az_cli_run_safe(cmd.split(&#39; &#39;))<br><br># Script<br>az(&quot;login --identity&quot;)<br>az(&quot;login --identity -u /subscriptions/XXXXXXXXXX/resourcegroups/test-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXXXX-id&quot;])</pre><p>And… It failed. The identity is not found.</p><p>Therefore, I created a credential object in my automation account, and it worked!</p><pre>#!/usr/bin/env python3<br># -*- coding: utf-8 -*-<br><br># Import<br>from azure.cli.core import get_default_cli<br>import automationassets<br><br># Functions<br>def az_cli_run_safe(args):<br> cli = get_default_cli()<br> cli.invoke(args)<br><br> if cli.result.result:<br>     return cli.result.result<br> elif cli.result.error:<br>     raise cli.result.error<br> return True<br><br>def az(cmd):<br> return az_cli_run_safe(cmd.split(&#39; &#39;))<br><br># Script<br>creds = automationassets.get_automation_credential(&#39;TESTRLE&#39;)<br>az_cli_run_safe([&quot;login&quot;, &quot;--service-principal&quot;, &quot;-u&quot;, creds.get(&#39;username&#39;), &quot;-p&quot;, creds.get(&#39;password&#39;),  &quot;--tenant&quot;, &quot;XXXXXXXXXXXXXXXX&quot;])<br>az(&quot;account show&quot;)</pre><p>This was the first success, but it did not satisfy our needs since it does not use the Managed Identity, and the import of the dependencies is painful and not very reliable.</p><h3>Using the documentation</h3><p>The Azure documentation contains a method called “<a href="https://learn.microsoft.com/en-us/azure/automation/python-3-packages#use-a-package-in-a-runbook">get_automation_runas_credential</a>.”</p><p>I tried to run it from scratch, assuming it was a piece of code to use my Managed Identity like with our Powershell Workbooks.</p><p>Later on, I found some more <a href="https://learn.microsoft.com/en-us/azure/automation/learn/automation-tutorial-runbook-textual-python-3#add-authentication-to-manage-azure-resources">documentation</a>, where there is a light explanation on how to set it up.</p><p>This code allows you to use a Service Principal or an App Registration configured to authenticate with a certificate. You need to set:</p><ol><li>A connection object:</li></ol><pre>runas_connection = automationassets.get_automation_connection(&quot;AzureRunAsConnection&quot;)</pre><p>2. The certificate to your App Registration</p><pre>cert = automationassets.get_automation_certificate(&quot;AzureRunAsCertificate&quot;)</pre><p>Anyway, this still was not what we needed.</p><h3>Using the Python SDK</h3><p>I created a test workbook where I tested a different type of authentication methods of the Python SDK:</p><pre># Working<br># Requires to set a credential object in the automation account<br><br>from azure.common.credentials import ServicePrincipalCredentials<br>creds       = automationassets.get_automation_credential(&#39;TESTRLE&#39;)<br>credentials = ServicePrincipalCredentials(creds.get(&#39;username&#39;), creds.get(&#39;password&#39;), tenant=TENANT)</pre><pre># Not working<br># Exception: &#39;DefaultAzureCredential&#39; object has no attribute &#39;signed_session&#39;<br># Possible explanation: API components version missmatch<br><br>from azure.identity           import DefaultAzureCredential<br>credentials = DefaultAzureCredential(exclude_shared_token_cache_credential=True)</pre><pre># Not working<br># Exception: &#39;ManagedIdentityCredential&#39; object has no attribute &#39;signed_session&#39;<br># Possible explanation: API components version missmatch<br><br>from azure.identity import ManagedIdentityCredential <br>credentials = ManagedIdentityCredential()</pre><p>At this point, I did not have a workaround for the recurring exception.</p><pre>Exception: &#39;ManagedIdentityCredential&#39; object has no attribute &#39;signed_session&#39;</pre><p>One of the many possible fixes I found later is:</p><pre># <a href="https://github.com/Azure/msrest-for-python/blob/master/msrest/authentication.py">https://github.com/Azure/msrest-for-python/blob/master/msrest/authentication.py</a><br>class AzureCredsUtils():<br>    __slots__ = [&#39;_token&#39;]<br>    def __init__(self, creds, scope=&quot;<a href="https://management.azure.com/">https://management.azure.com/</a>&quot;):<br>        self._token = creds.get_token(scope).token<br><br>    def signed_session(self, session = None):<br>        session = session or requests.Session()<br>        session.headers[&#39;Authorization&#39;] = &quot;Bearer %s&quot; % self._token<br>        return session<br><br>creds = DefaultAzureCredential(exclude_shared_token_cache_credential=True)<br>fixed = AzureCredsUtils(creds, &#39;<a href="https://graph.windows.net/&#39;">https://graph.windows.net/&#39;</a>)</pre><h3>The graal</h3><p>I stumbled across <a href="https://learn.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation#get-access-token-for-system-assigned-managed-identity-using-http-get">this piece of documentation</a>.</p><p>I created a python runbook to get this token and it worked!</p><pre># Getting env vars<br>endPoint = os.getenv(&#39;IDENTITY_ENDPOINT&#39;)+&quot;?resource=<a href="https://management.azure.com/">https://management.azure.com/</a>&quot;<br>identityHeader = os.getenv(&#39;IDENTITY_HEADER&#39;)<br><br># Getting access token from local endpoint<br>headers = {<br> &#39;X-IDENTITY-HEADER&#39;: identityHeader,<br> &#39;Metadata&#39;: &#39;True&#39;<br>}<br>response = requests.get(endPoint, headers=headers)<br><br># Checking the result<br>if response.status_code != 200:<br> raise Exception(&#39;Unable to get the Managed identity token&#39;)<br><br># Returning the credential object<br>token = response.json()<br>print(token)</pre><p>I looked in the documentation on how I could set a token for the Azure SDK.</p><p>I found the <a href="https://learn.microsoft.com/en-us/python/api/msrest/msrest.authentication.basictokenauthentication?view=azure-python">perfect component for it</a>.</p><h3>Final solution</h3><p>Could someone from Microsoft put this in the documentation?</p><p>This solution works with the best practice for Azure identity management, the Managed Identities. The code is much easier than the get_automation_runas_credential found in the documentation. This does not require importing any package into the automation account.</p><pre>#!/usr/bin/env python3<br># -*- coding: utf-8 -*-<br><br>## Params (in order)<br># subscription_id<br># resource_group_name<br># vm_name<br><br>###############################################################<br>### Imports<br>###############################################################<br>import os<br>import sys<br>import requests<br><br>from azure.common.credentials import BasicTokenAuthentication<br>from azure.mgmt.compute       import ComputeManagementClient<br><br><br>###############################################################<br>### Naming the parameters<br>###############################################################<br>subscription_id     = sys.argv[1]<br>resource_group_name = sys.argv[2]<br>vm_name             = sys.argv[3]<br><br><br>###############################################################<br>### Methods<br>###############################################################<br>def authenticate_with_identity():<br> # Getting env vars<br> endPoint       = os.getenv(&#39;IDENTITY_ENDPOINT&#39;)+&quot;?resource=<a href="https://management.azure.com/">https://management.azure.com/</a>&quot;<br> identityHeader = os.getenv(&#39;IDENTITY_HEADER&#39;)<br><br> # Getting access token from local endpoint<br> headers = {<br>  &#39;X-IDENTITY-HEADER&#39;: identityHeader,<br>  &#39;Metadata&#39;: &#39;True&#39;<br> }<br> response    = requests.get(endPoint, headers=headers)<br><br> # Checking the result<br> if response.status_code != 200:<br>  raise Exception(&#39;Unable to get the Managed identity token&#39;)<br><br> # Returning the credential object<br> token = response.json()<br> return BasicTokenAuthentication(token)<br><br><br>###############################################################<br>### Script<br>###############################################################<br>## Credentials<br>credentials = authenticate_with_identity()<br>client      = ComputeManagementClient(credentials, subscription_id)<br><br>run_command_parameters = {<br> &#39;command_id&#39;: &#39;RunPowerShellScript&#39;, # For linux, don&#39;t change it<br> &#39;script&#39;: [<br>  &#39;type C:\\Test\\test.txt&#39;<br>    ]<br>}<br>poller = client.virtual_machines.run_command(<br> resource_group_name,<br> vm_name,<br> run_command_parameters<br>)<br><br>result = poller.result()  # Blocking till executed<br>print(result.value[0].message)  # stdout/stderr</pre><p><strong>About the Author</strong></p><p>Romain Leonard is the Chief Information Security Officer at Sinequa, where he leads all the activities related to the security of Sinequa. Romain is passionate about cyber security and intelligent search technologies. He has over 10 years of experience working in the cybersecurity area where he held various positions, such as penetration tester, security engineer, and CISO.</p><p><strong>About Sinequa</strong></p><p>Sinequa’s Search Cloud brings organizations of all sizes the most complete enterprise search ever. Customers employ the Sinequa Search Cloud to connect all content (both text and data), derive meaning, learn from user interactions, and present information in context. This solves content chaos and informs employees through a single, secure interface. They get the knowledge, expertise, and insights needed to make informed decisions and do more, faster. Sinequa helps these organizations accelerate innovation, reduce rework, foster collaboration, ensure compliance, and increase productivity. For more information visit <a href="https://cts.businesswire.com/ct/CT?id=smartlink&amp;url=https%3A%2F%2Fwww.sinequa.com%2F&amp;esheet=52614385&amp;newsitemid=20220323005136&amp;lan=en-US&amp;anchor=www.sinequa.com&amp;index=4&amp;md5=a28f7ecec0d79fd01a3db529f8606e38">www.sinequa.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1aa586dc6d26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meta-Learning for Keyphrase Extraction]]></title>
            <link>https://sinequa.medium.com/meta-learning-for-keyphrase-extraction-54ec960cd2ba?source=rss-c033ea71a12a------2</link>
            <guid isPermaLink="false">https://medium.com/p/54ec960cd2ba</guid>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[keyphrase-extraction]]></category>
            <category><![CDATA[multilingual]]></category>
            <category><![CDATA[meta-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <dc:creator><![CDATA[Sinequa]]></dc:creator>
            <pubDate>Mon, 06 Sep 2021 13:17:05 GMT</pubDate>
            <atom:updated>2021-09-06T13:17:05.038Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A strategy to boost zero-shot predictions</em></p><p>KeyPhrase Extraction (KPE) is the process of extracting relevant chunks of words from a document to best capture and represent its content. Having keyphrases helps the reader get the gist of the document in a glance and browse quickly through many documents. Hence it is very valuable for the user experience of search engines to enhance previews (showing the beginning of a document while staying on the main results page).</p><p>In this article we outline what needs to be done to build a keyphrase extractor that performs well not only on in-domain data, but also in a zero-shot scenario where keyphrases need to be extracted from data that have a different distribution (either a different domain or a different type of documents).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*niFJ7Ar24495zhu_rVVq2w.png" /></figure><h3>An ill-defined task</h3><p>The definition of what is or isn’t a “good” keyphrase is very fuzzy for humans, since many datasets report having a very low matching rate of keyphrases between annotators <a href="http://www.lrec-conf.org/proceedings/lrec2014/pdf/1037_Paper.pdf">[1]</a>. Yet no annotator is really assigning wrong keyphrases; then how come humans can be in such disagreement? The main reasons are:</p><ol><li>There is a vast number of potentially relevant keyphrases for a given document (especially if the document is several paragraphs long or more), yet only a few (~ 3–10 on usual datasets) of the most salient are selected at a time, and</li><li>Humans do not really agree on the exact definition of what a “good” keyphrase should be.</li></ol><p>When discussing the matter, reviewing each other’s opinion and working on a common definition, their F1-score (common agreement on the chosen keyphrases) can go up three times as high <a href="https://www.aclweb.org/anthology/2020.law-1.7/">[2]</a> on some easy-to-annotate datasets. A common definition can be something like:</p><ul><li>Keyphrases are between one and five words long,</li><li>Keyphrases should represent the main topics and ideas of the document,</li><li>Keyphrases should each convey new information and not be synonyms of one another.</li></ul><p>One of the great challenges of research is comparing models between one another, as different metrics can cause the same models to be evaluated and ranked differently. The main metric for this article is F1@5 (F1-score on the exact match of the top 5 extracted keyphrases), with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*uLe1HqoI5l_mmn7cnJ0AAw.png" /></figure><p>Using the same metric, the scores of a model for different datasets can vary a lot, either because the number of annotated keyphrases is different from the number of keyphrases extracted by the model (i.e., if there are more annotated keyphrases than extracted keyphrases, recall will decrease and precision will increase), or simply because it’s harder to extract keyphrases from the documents in some datasets. Oftentimes, models tend to work well on a dataset for which they were tailored (through either feature engineering or model parameter tuning) and have poorer performance on a slightly differing dataset.</p><h3>Scarcity of data</h3><p>When benchmarking the state-of-the-art models, we were struck by how little data is available for training. We found only 39 datasets that were created with proper human annotation, there are some curated dataset lists available online <a href="https://github.com/boudinfl/ake-datasets,%20https://github.com/LIAAD/KeywordExtractor-Datasets">[3]</a>. Most datasets are extracted from scientific papers from natural sciences, and some important domains such as law are not represented at all. Moreover, 25 of those datasets are in English, and most contain only a few hundred or thousand documents, which is limiting for supervised training. Luckily, some are quite big:</p><ul><li>KPTimes <a href="https://arxiv.org/abs/1911.12559">[4]</a>: 289,923 news articles, license terms not entirely clear <a href="https://github.com/ygorg/KPTimes#license">[5]</a>,</li><li>OpenKP (MS MARCO) <a href="https://arxiv.org/abs/1911.02671">[6]</a>: 148,124 web pages, limited to academic use, and</li><li>KP20K <a href="https://arxiv.org/abs/1704.06879">[7]</a>: 547,090 science paper abstracts, limited to academic use.</li></ul><p>As you will see, having a bigger dataset is both necessary and sufficient for proper training and boosting performance.</p><h3>Creating a new keyphrase dataset</h3><p>Due to the scarcity of data, especially in languages other than English, we turned to HAL <a href="https://hal.archives-ouvertes.fr/">[8]</a> which is “an open archive where authors can deposit scholarly documents from all academic fields”. When uploading an article, the author can specify a set of keyphrases. HAL offers an API <a href="https://api.archives-ouvertes.fr/docs/search">[9]</a> to crawl the available articles including authors, title, abstract and, most importantly, the keyphrases.</p><p>We decided to only keep the keyphrases that were present in the abstract and we only kept articles that had at least three of those present keyphrases. Unsurprisingly, most of the gathered data was in English and French (as it is a French initiative), but some other languages were also available. The number of documents per language are:</p><ul><li>English: 206,365</li><li>French: 119,185</li><li>Spanish: 1,550</li><li>Portuguese: 379</li><li>Italian: 228</li><li>German: 96</li></ul><h3>Model choice</h3><p>To solve the keyphrase extraction problem, we can choose from two fundamentally different approaches — supervised or unsupervised learning.</p><p>Unsupervised learning has the nice property of not depending on the training data distribution and, therefore, being theoretically more robust to unseen data. However, it seriously lags in performance when compared to supervised approaches. Moreover, our extensive study of SIFRank <a href="https://ieeexplore.ieee.org/abstract/document/8954611/">[10]</a> (well-performing unsupervised model based on ELMo <a href="https://arxiv.org/abs/1802.05365">[11]</a> embeddings) showed that robustness was not that good and that SIFRank often produced unsatisfactory keyphrases (either not very salient or not self-contained), which is problematic for any practical use.</p><p>The supervised approach requires an annotated dataset with golden keyphrases. Using these labels, an algorithm can teach a model to extract keyphrases by minimizing some loss function that, in turn, maximizes the F1 score on exact matches. This method works well with transformer-based models such as BERT.</p><p>This is why we decided to go with a BERT-based, state-of-the-art method called BERT-JointKPE <a href="https://arxiv.org/abs/2004.13639">[12]</a>. We will give a brief overview of how the method works, but you should check out their great paper if you want to dive deeper. On a high level, the model steps are as follows:</p><ol><li>The raw text is passed into a BERT model to extract one embedding per token.</li><li>The token embeddings are then fed to a custom CNN with various window sizes (from 0 to the maximum length of a keyphrase) which will create an n-gram representation for every possible n-gram.</li><li>Those n-gram representations are then fed to the chunking network, which will select which n-grams are keyphrases (chunks) thanks to a linear binary classification layer.</li><li>Those chunks are in turn fed into the ranking network which will rank them according to their saliency and assign a score to each of them using a linear layer.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*BPPSFLnsM6SichSj" /><figcaption>BERT-JointKPE’s Schema</figcaption></figure><p>The training is performed jointly on the combined loss of the chunking and ranking network, the total loss is simply the sum of the chunking and ranking loss. For the chunk loss, the cross-entropy is computed for every chunk where the chunk label is 1 if the chunk represents a keyphrase and 0 otherwise. For the ranking loss, the hinge loss like in pairwise learning to rank is computed on exact matches with the ground truth by minimizing the score of non-keyphrases minus the score of real keyphrases.</p><h3>Meta-Learning</h3><p>Our goal was not only to reach high scores for datasets that were used during training, but we also wanted to reach good scores for previously unseen data from other domains. For that purpose, we performed multi-source domain adaptation, i.e., training the model on various datasets to reach good performance in several types of domains (science, law, etc) or of text structure (tweets, articles, emails, etc).</p><p>When trained on at least two datasets with different documents and different annotators, performance on both a validation set from a known distribution and on zero-shot data goes up significantly. This phenomenon is called meta-learning <a href="https://arxiv.org/abs/2004.05439">[13]</a>.</p><p>As the reader will probably have noticed, no F1 score was reported for our experiences. We are collecting them, rendering them in pretty vizualizations and running extended experiences at the moment. For now the reader will have to trust us but no worries a post is coming soon with a lot more details.</p><p>When faced with different types of distributions, the model has to learn beyond recognizing patterns in a single kind of documents; it has to learn the general principle of “what is a good keyphrase”. Indeed, varying the type and source of training data reduces the confusion factors. For example, if all documents are scientific papers, the model will learn that rather complicated words are usually significant and people’s names are insignificant; but in a news article, it may be that the name of a person is indeed important. When learning on both types of data, the model will have to acknowledge that not all documents are the same. Hence when faced with a new type of document, it will not assume it is exactly like the training distribution, and therefore the zero-shot performance is expected to rise. There are four types of parameters to vary in training distributions:</p><ol><li>Annotator (prior on what is a good keyphrase),</li><li>Structure of the documents (tweets, articles, emails, etc.),</li><li>Topic of the document (science, law, politics, etc.),</li><li>Language.</li></ol><p>But what is more surprising is that the performance on any of the datasets used in training also rises, sometimes very sharply. How does mixing the distribution with another slightly different distribution help? This is because instead of learning to recognize a distribution, the algorithm learns the concept of keyphrases as a whole. The more diverse the added data is, the more this effect is noticeable. However, we noticed that we do not need many different datasets; using five datasets is barely more helpful than using only two. We also noticed that not all datasets are equally useful for meta-learning. Here are our main findings:</p><ol><li>The larger the dataset, the more useful it is for meta-learning (i.e., more data means better performance).</li><li>Using a dataset from a certain domain improves zero-shot performance especially for another dataset from the same domain.</li><li>Small datasets (even if they have a very different distribution) can make a real difference (e.g., 10,000 samples from various languages).</li><li>As small datasets are easily overfitted, they should be combined with larger datasets. This way we can reap the benefits of diverse training data while avoiding the risk of overfitting.</li><li>In order to avoid small datasets being outweighed by larger datasets, repeat the data from the smaller datasets to match the size of the biggest dataset. Obviously there are limits to this approach, it should only be done if the datasets do not differ in size by more than an order of magnitude.</li></ol><p>Combining all these findings (i.e., using a large dataset like KPTimes and some smaller datasets of various topics and types of text like emails, science papers and medical papers) allows us to train a single model that performs very well in a wide array of domains, and is competitive to unsupervised models in zero-shot scenarios. This is even more impressive given that most models we compare with only work well on a single dataset — their training dataset.</p><p>We also conducted experiments using datasets in different languages. First, we trained a multi-lingual BERT-Base model only on English datasets, and the second model was trained on English datasets plus the HAL datasets in the other available languages. While the zero-shot performance of both models for English data did not differ significantly, the performance for non-English data was, as expected, much better.</p><h3>Influence of model size</h3><p>Additionally, we tested the effect of reducing the size of BERT on model performance (while performing the same training on the same data) as switching from BERT-Base to smaller models can reduce inference times substantially which is highly desirable. It appears that using a BERT-Small <a href="https://arxiv.org/abs/1908.08962">[14]</a> barely lowers the overall performance despite using only 22% of the weights used by BERT-Base. Remarkably, the reduction in performance on zero-shot datasets is much weaker but still noticeable.</p><h3>Post-processing tricks</h3><p>In the course of our research, we also explored a number of techniques that can make the results of the keyphrases extracted by the model more beneficial to users. The goal was to narrow down the result set to only the best keyphrases, and to ensure that each keyphrase brings new value and (hopefully) isn’t just a different way of saying the same thing as an earlier keyphrase. Here are some tips we suggest to maximize the value of a keyphrase system:</p><ul><li>Threshold: Only accept keyphrases with a high confidence score to avoid predicting keyphrases of low quality.</li><li>Syntactic diversity: If a keyphrase is included in another keyphrase, then the longer keyphrase should convey more information. We delete the shorter keyphrase until all combinations have been explored and trimmed. This biases the results towards longer keyphrases but is very efficient for diversity.</li><li>Merge overlapping keyphrases: If two keyphrases overlap (e.g., the last word of a keyphrase matches the first word of another keyphrase), then it is probable that they represent two pieces of an important keyphrase, like a long name. It is desirable to merge them into a single keyphrase, however, an upper bound to limit the maximum length of a keyphrase created in such a manner should be applied (e.g., do not create keyphrases with more than eight words).</li><li>Remove stopwords at beginning or end: It can happen that the model predicts a keyphrase that contains a stopword at the beginning or the end, they are most likely not helpful and should be removed. Note that stopwords inside the keyphrase should be kept as they could very well be part of a valid keyphrase.</li></ul><h3>Concluding remarks</h3><p>In order to achieve automatic KPE with good enough performance to be of practical use, we need to take into account the behavior of models on unseen data. In order to do that, some tricks can be very useful. If you know what the target distribution will look like, it is often very efficient to fine-tune on this distribution (or something close to it) in order to trigger transfer learning. But if you wish to have a more universal model, then robustness is crucial. A great way to achieve a robust model is to train it on various datasets to enable meta-learning, which also enhances performance. Tests showed great results on random types of data, including noisy, out-of-domain and more complicated datasets.</p><h3>Examples</h3><p>Wondering what kind of keywords our model is capable of extracting? Here are some examples from different domains:</p><p>Wikipedia article:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*7Q3zz9ywfjhiDu59OKuDrg.png" /></figure><p>Indeed job offer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*1-t90NYJZ-JGtZ3rIN8b7Q.png" /></figure><p>Enron Email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*DmDX6XM-BB_pAUfwL1yRwQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54ec960cd2ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying long textual documents (up to 25 000 tokens) using BERT]]></title>
            <link>https://sinequa.medium.com/classifying-long-textual-documents-up-to-25-000-tokens-using-bert-9d2dd55ca060?source=rss-c033ea71a12a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d2dd55ca060</guid>
            <category><![CDATA[long-texts]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Sinequa]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 18:02:58 GMT</pubDate>
            <atom:updated>2020-12-08T18:02:58.652Z</atom:updated>
            <content:encoded><![CDATA[<h3>What do we want to achieve?</h3><p>We want to classify texts into predefined categories which is a very common task in NLP. For many years, the classical approach for simple documents was to generate features using <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> and combine it with <a href="https://en.wikipedia.org/wiki/Logistic_regression">logistic regression</a>. Formerly we used to rely on this stack at Sinequa for textual classification, and, spoiler alert, with the model presented here we have beaten our baseline from 5% to 30% for very noisy and long documents datasets. This former approach had two main issues: the feature sparsity that we tackled via compression techniques and the word-matching issue that we tamed leveraging Sinequa’s powerful linguistic capacities (mainly through our homegrown tokenizer).</p><p>Later on, the pandora box of language models (pre-trained on humongous corpora in an unsupervised fashion and fine-tuned on downstream supervised tasks) was opened and TF-IDF based techniques were not state of the art anymore. Such language models could be <a href="https://arxiv.org/pdf/1301.3781.pdf">word2vec</a> combined with LSTMs or CNNs, ELMo, and most importantly the Transformer (in 2017: <a href="https://arxiv.org/pdf/1706.03762.pdf">https://arxiv.org/pdf/1706.03762.pdf</a>).</p><p><a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a> is a Transformer based language model that has gained a lot of momentum in the last couple of years since it beat all NLP baselines by far and came as a natural choice to build our text classification.</p><h3>What is the challenge then?</h3><p>Transformer based language models such as BERT are really good at understanding the semantic context (where bag-of-words techniques fail) because they were designed specifically for that purpose. As explained in the introduction, BERT outperforms all NLP baselines, but as we say in the scientific community, “no free lunch”. This extensive semantic comprehension of a model like BERT offers comes with a big caveat: it cannot deal with very long text sequences. Basically, this limitation is 512 tokens (a token being a word or a subword of the text) which represent more or less two or three Wikipedia paragraphs and we obviously don’t want to consider only such a small sub-part of a text to classify it.</p><p>To illustrate this, let’s consider the task of classifying comprehensive product reviews into positive or negative reviews. The first sentences or paragraphs may only contain a description of the product and it would likely require to go further down the review to understand whether the reviewer actually likes the product or not. If our model does not encompass the whole content, it might not be possible to make the right prediction. Therefore, one requirement for our model is to capture the context of a document while managing correctly long-time dependencies between the sentences at the beginning and the end of the document.</p><p>Technically speaking, the core limitation is the memory footprint that grows quadratically with the number of tokens along with the use of pre-trained models that come with a fixed size determined by Google (&amp; al.). This is expected since each token is “attentive” [https://arxiv.org/pdf/1706.03762.pdf] to every other token and therefore requires a [N x N] attention matrix, with [N] the number of tokens. For example, BERT accepts a maximum of 512 tokens which hardly qualifies as long text. And going beyond 512 tokens rapidly reaches the limits of even modern GPUs.</p><p>Another problem that arises using Transformers in a production environment is the very slow inference due to the size of the models (110M parameters for BERT base) and, again, the quadratic cost. So, our goal is not only to find an architecture that fits into memory during the training but to find one that also responds reasonably fast during inference.</p><p>The last challenge we address here is to build a model based on various feature types: long text of course, but also additional textual metadata (such as title, abstract …) and categories (location, authors …).</p><h3>So, how to deal with really long documents?</h3><p>The main idea is to split the document into shorter sequences and feed these sequences into a BERT model. We obtain the CLS embedding for each sequence and merge the embeddings. There are a couple of possibilities to perform the merge, we experimented with:</p><ul><li>Convolutional Neural Networks (CNN)</li><li>Long Short-Term Memory Networks (LSTM)</li><li>Transformers (to aggregate Transformers, yes :) )</li></ul><p>Our experiments on different standard text classification corpora showed that using additional Transformer layers to merge the produced embeddings works best without introducing a large computational cost.</p><h4>Want the formal description, right?</h4><p>We consider a text classification task with <em>L</em> labels. For a document <em>D</em>, its tokens given by the <a href="https://medium.com/@makcedward/how-subword-helps-on-your-nlp-model-83dd1b836f46">WordPiece tokenization</a> can be written <em>X =( x₁, …, xₙ) </em>with <em>N</em> the total number of token in <em>D</em>. Let K be the maximal sequence length (up to 512 for BERT). Let <em>I</em> be the number of sequences of <em>K </em>tokens or less in <em>D</em>, it is given by I=⌊ N/K ⌋.</p><p>Note that if the last sequence in the document has a size lower to <em>K</em> it will be padded with 0 until the <em>Kᵗʰ</em> index. Then if <em>s</em>ⁱ with <em>i</em><strong><em>∈ </em></strong><em>{1, .., I},</em> is the <em>i</em>-th sequence with <em>K</em> elements in <em>D</em>, we have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*V0wWXH6zln0DoAIVbr0R5g.png" /></figure><p>We can note that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*d17-6ACYbiHMPf7fv0CA9Q.png" /></figure><p>BERT returns the CLS embedding but also an embedding per token.</p><p>Let define the embeddings per token returned by BERT for the i-th sequence of the document such as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*9HKnHCbpjFlQgkWj2F2pvA.png" /></figure><p>where <em>CLS</em> is the embedding of the special token inserted in front of each text sequence fed to BERT, it is generally considered as an embedding summarizing the full sequence.</p><p>To combine the sequences, we only use <em>CLSᵢ</em> and do not use <em>y</em>. We use <em>t</em> transformers <em>T₁, …,Tₜ</em> to obtain the final vector to feed to the last dense layer of the network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*2kvmbKrSz_RFYnZpaDT0RA.png" /></figure><p>where <strong>∘</strong> is the function composition operation.</p><p>Given the last dense layer weights <em>W ∈ </em>ℝᴸˣᴴ where <em>H</em> is the hidden size of the transformer and bias <em>b</em> <em>∈ </em>ℝᴸ</p><p>The probabilities <strong><em>P</em></strong> <em>∈ </em>ℝᴸ are given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*Kjf7I0D_MlKqfdKTuWd9GA.png" /></figure><p>Finally, applying <em>argmax </em>on the vector <em>P</em> returns the predicted label. For a summary of the above architecture, you can have a look at figure 1.</p><p>The architecture above enables us to leverage BERT for the text classification task bypassing the maximum sequence length limitation of transformers while at the same time keeping the context over multiple sequences. Let’s see how to combine it with other types of features.</p><h3>How to deal with metadata?</h3><p>Oftentimes, a document comes with more than just its content. There can be metadata that we divide into two groups, textual metadata, and categorical metadata.</p><h4>Textual Metadata</h4><p>By textual metadata, we mean short text that has (after tokenization) a relatively small number of tokens. This is required to fit entirely into our language model. A typical example of such metadata would be titles or abstracts.</p><p>Given a document with <em>M</em> metadata annotation. Let</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*XqJF_Jy_I-2CgydI2ER4IQ.png" /></figure><p>be the CLS embeddings produced by BERT for each metadata. The same technique as above is used to get the probability vector as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*gMufVs1TnMb0lJP1K2GBxg.png" /></figure><h4>Categorical Metadata</h4><p>Categorical metadata can be a numerical or textual value that represents a category. Numerical values can be the number of pages whereas textual values can be the publisher name or a geo-location.</p><p>A common way to deal with such features is to implement <a href="https://export.arxiv.org/pdf/1606.07792">the Wide and Deep architecture</a>. Our experiments showed that results yielded by the deep part of this network were sufficiently good and the wide part was not required.</p><p>We encode the categorical metadata in a single cross-category vector using one-hot encoding. This encoding is then passed into an embedding layer that learns a vector representation for each distinct category. The last step is to apply a pooling layer on the resulting embedding matrix.</p><p>We considered max, average and min pooling and found that using average pooling worked best for our test corpora.</p><h3>How does the complete architecture look?</h3><p>Hope you stuck around until now, the following figure will hopefully make things a lot clearer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Bjuwmq0gIgacPF1ZrjvUw.png" /><figcaption>Figure 1</figcaption></figure><p>There are three sub-models, one for text, another for textual metadata, and the last one for categorical metadata. The output of the three sub-models is merely concatenated into a single vector before passing it through a dropout layer and finally into the last dense layer with a softmax activation for the classification.</p><p>You probably have noticed that there are multiple BERT instances depicted in the architecture, not only for the text input but also for the textual metadata. As BERT comes with many parameters to train, we decided not to include a separate BERT model per sub-model, but instead share the weights of a single model in between the sub-models. Sharing weights certainly reduces the RAM used by the model (enabling training with larger batch-size, so accelerating training in a sense) but it does not change the inference-time since there will still be as many BERT executions no matter whether their weights are shared or not.</p><h3>What about inference time?</h3><p>By now, you must have guessed that including that many invocations of the BERT model do not come for free. And it is true that it is computationally expensive to run inference of such a model. However, there are a couple of tricks to improve inference times. In the following, we focus on CPU inference as this is very important in production environments.</p><p>A couple of notes for the conducted experiments:</p><ul><li>We consider a simplified model only containing a text feature.</li><li>We limited the tokens that we used per document to 25,600 tokens which correspond roughly to around 130,000 characters if the document contains English text.</li><li>We perform the experiments with documents that have the above described maximum length. In practice, documents have varying sizes and as we use dynamic size tensors in our model, inference times are considerably faster for short documents. As a rule of thumb, when using a document that is half as long reduces the inference time by 50 %.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZGpcgQCedPnwk8c1i_FMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WlLLPfS2aWT5h9aLpmcrUQ.png" /></figure><p>References</p><ol><li><a href="https://arxiv.org/abs/2006.04152">https://arxiv.org/abs/2006.04152</a>, <a href="https://arxiv.org/pdf/2001.08950.pdf">https://arxiv.org/pdf/2001.08950.pdf</a></li><li><a href="https://blog.tensorflow.org/2020/04/tfrt-new-tensorflow-runtime.html">https://blog.tensorflow.org/2020/04/tfrt-new-tensorflow-runtime.html</a></li><li><a href="https://www.tensorflow.org/xla?hl=fr">https://www.tensorflow.org/xla?hl=fr</a></li><li><a href="https://medium.com/microsoftazure/accelerate-your-nlp-pipelines-using-hugging-face-transformers-and-onnx-runtime-2443578f4333">https://medium.com/microsoftazure/accelerate-your-nlp-pipelines-using-hugging-face-transformers-and-onnx-runtime-2443578f4333</a></li></ol><h3>What else is there to do?</h3><h3>Linear Transformers</h3><p>Building a Transformer-like architecture that does not come with the quadratic complexity in time and memory is currently a very active field of research. There are a couple of candidates that are definitely worth trying once pre-trained models will be released:</p><ul><li>Linformer [<a href="https://arxiv.org/pdf/2006.04768.pdf">https://arxiv.org/pdf/2006.04768.pdf</a>]</li><li>BigBird [<a href="https://arxiv.org/pdf/2007.14062.pdf">https://arxiv.org/pdf/2007.14062.pdf</a>]</li><li>Reformers [<a href="https://arxiv.org/pdf/2001.04451.pdf">https://arxiv.org/pdf/2001.04451.pdf</a>] (only O(N log(N)) complexity)</li><li>Performers [<a href="https://arxiv.org/pdf/2009.14794.pdf">https://arxiv.org/pdf/2009.14794.pdf</a>]</li><li>etc…</li></ul><p>A preliminary test with the very promising <a href="https://arxiv.org/pdf/2004.05150.pdf">Longformer </a>model could not be executed successfully. We tried to train a LongFormer model using the TensorFlow implementation of <a href="https://huggingface.co/transformers/model_doc/longformer.html">Hugging Face</a>. However, it appears that the implementation is not yet memory-optimized as it was not possible to train it even on a large GPU with 48 GB of memory.</p><p>Inference time is the cornerstone of any ML project that needs to run in production, so we do plan to use such “linear transformers” in the future in addition to pruning and quantization.</p><h3>Are we done yet?</h3><p>Yes, thanks for sticking with us until the end. If you have questions or remarks about our model, feel free to comment. We would love to hear from you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d2dd55ca060" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Query Intent: Few-Shot Learning & Out-of-domain]]></title>
            <link>https://sinequa.medium.com/query-intent-few-shot-learning-out-of-domain-14bc4a16ec48?source=rss-c033ea71a12a------2</link>
            <guid isPermaLink="false">https://medium.com/p/14bc4a16ec48</guid>
            <category><![CDATA[few-shot-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[query-intent]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sinequa]]></dc:creator>
            <pubDate>Wed, 29 Apr 2020 17:49:51 GMT</pubDate>
            <atom:updated>2020-04-29T22:02:36.760Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>As with other Machine Learning models, the very common problem with Query Intent is lack of data. This problem is even more important<strong> </strong>when we want to provide a training service, not a trained model, to our clients.</p><p>In the real-word use-case, clients do not have many example queries for each intent.</p><p>Moreover, most existing Query Intent models are built for Intent detection (<strong>In-distribution</strong> detection) but are incapable of recognizing the Non-Intent (<strong>Out-of-distribution</strong> detection). To jointly address these issues, we developed a Query Intent model that can be trained on a few data, without requiring any additional Non-Intent example queries. Experiment results show that our model surpasses recently proposed techniques on the <strong>In-distribution </strong>(ID) detection problem and highly improves the performance on <strong>Out-of-distribution</strong> (OOD) detection.</p><h3>Proposed Approach</h3><p>Recently a new <strong>Joint Intent Classification and Slot Filling</strong> model<strong> </strong>(<a href="https://arxiv.org/abs/1902.10909"><strong>[2]</strong></a> Qian Chen et al., 2019), based on <strong>BERT</strong> (Bidirectional Encoder Representations from Transformers) (<a href="https://arxiv.org/abs/1810.04805"><strong>[1]</strong></a><strong> </strong>Devlin et al., 2018) has been introduced. It outperforms the state-of-the-art methods for intent detection and achieves significant improvement on intent classification accuracy and slot filling F1 on several public benchmark datasets. Unfortunately, these public datasets used in this paper are generally large, moreover there have not been efforts to explore this model for Non-Intent detection.</p><p>Several techniques have been proposed to address the problem of <strong>Out-of-distribution </strong>detection. Here are some remarkable baseline methods that do not require any OOD datasets for training and related comments extracted from the article <a href="https://arxiv.org/abs/1909.03862"><strong>[4]</strong></a>:</p><ol><li><strong>MSP </strong><a href="https://arxiv.org/abs/1610.02136"><strong>[5]</strong></a>: A text classifier with a Softmax output layer is trained, and the maximum Softmax output is used as the detection score. The training objective of this classifier does not include the entropy regularization (ER) term.</li><li><strong>Entropy</strong>: The Shannon entropy of the predicted distribution for each input is used as the detection score. A higher entropy means higher uncertainty of the prediction, which indicates that the input sample may be from the OOD data.</li><li><strong>ODIN </strong><a href="https://arxiv.org/abs/1706.02690"><strong>[6]</strong></a>: The temperature scaling and input perturbation technique is applied to the text classifier as obtained in the previous <strong>MSP </strong>baseline, and the maximum Softmax output is used as the detection score. Note that small perturbations of each input are added to the last feature layer in this baseline.</li></ol><blockquote>These approaches above follow the softmax-based concept, which has been demonstrated to work surprisingly well in the image classification tasks. However it seems to have difficulties in Query Intent where the model tends to make a high softmax score even for OOD query. This behavior has been shown in some recent works such as <a href="https://arxiv.org/abs/1412.1897"><strong>[8]</strong></a> and <a href="https://arxiv.org/abs/1610.08401"><strong>[9]</strong></a>.</blockquote><p>Our experiments on the model “Joint Intent Classification and Slot Filling” shows that, for OOD queries, the softmax score could be as high as the ID queries, higher than 0.95 in a lot of cases. That’s because these OOD queries are quite close to an intent in ID dataset but completely different from the rest.</p><p>About the recent paper <strong>Early-Layer Output</strong> (<a href="https://arxiv.org/abs/1910.10307"><strong>[3]</strong></a> V. Abdelzad et al., 2019), the hypothesis underlying their approach is that there exists a latent space, represented by the output of a layer called the optimal OOD discernment layer, in which represented features are discriminative enough to allow separating distributions of ID and OOD datasets.</p><p>In this section, we propose a novel method based on the <strong>Joint Intent Classification and Slot Filling</strong> model and the <strong>Early-Layer Output </strong>model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ENMzMZsTSgnICM_wBko7nQ.png" /><figcaption><em>Figure 1. A high-level view of the proposed model for intent detection.</em></figcaption></figure><h4>Model:</h4><p>For the <strong>ID</strong> problem, due to the different sizes of client data and public datasets, the training is split into two phases. The first phase is the training on public datasets that are supposed to be large, and the second one performs on client extended data (i.e. merged with a small part of public datasets, the reason for this extension is very simple, just to keep a good score on public datasets). These two phases are very similar to the original idea of BERT, our Query Intent model is first <strong>pre-trained </strong>on public datasets and then <strong>fine-tuned </strong>on the client data.</p><p>For the <strong>OOD</strong> problem, our main insight is based on <a href="https://arxiv.org/abs/1910.10307"><strong>[3]</strong></a>, that there is a latent space in which the distributions of ID and OOD datasets are well separated, and that’s the before-last layer of the Joint Intent Classification and Slot Filling model. Our approach relies indeed on the assumption that BERT is good enough for classification and OOD problems, and that the output layer for BERT contains relevant information for both problems. The high-level view of our model is shown as in <em>Figure 1</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XISNkfSN819PoWocjlys_A.png" /><figcaption><em>Figure 2. A figure extracted from the paper </em><strong><em>[3]</em></strong><em>. From a well-chosen layer, the distributions of ID and OOD datasets are separated.</em></figcaption></figure><h4>Joint Intent Classification and Slot Filling:</h4><p>Let <strong>N </strong>be the number of intents. For each query input <strong>x</strong>, the intent prediction probabilities are computed by softmax as:</p><figure><img alt="Softmax" src="https://cdn-images-1.medium.com/max/979/1*8fouBUV0fTY8fL__Br3Zeg.png" /></figure><p>where <strong>h0</strong> is the hidden state of the first classification token [CLS] of BERT, <strong>Wi </strong>(resp. <strong>bi</strong>) is the weight matrix (resp. bias vector) of the intent classifier layer (i.e the before-last dense layer in Figure 1).</p><p>The slot prediction probabilities for the word <strong>xn</strong> of the query <strong>x </strong>can be similarly obtained through:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*ib2qVOoGHyCsh8zqlIYzeQ.png" /></figure><p>where <strong>hn</strong> is the hidden state corresponding to the <strong>first</strong> <strong>sub-token </strong>of the word <strong>xn</strong> and <strong>Ws </strong>(resp. <strong>bs</strong>) is the weight matrix (resp. bias vector) of the slot classifier layer.</p><p>In order to learn the intent classifier and the slot classifier parameters jointly, we define a loss function, which is the linear combination of <strong>Li</strong> (cross-entropy between the predicted intent and the correct intent) and <strong>Ls</strong> (average cross-entropy between the predicted slots and the correct slots):</p><figure><img alt="Loss function" src="https://cdn-images-1.medium.com/max/283/1*vy2GAZ10e8wqhfwVOc5LHA.png" /></figure><p>where alpha is a new parameter to be acquired during the training phase.</p><blockquote>We fix alpha = 3 for all experimental evaluations.</blockquote><h4>Confidence threshold estimation</h4><p>The thresholds are computed only when the model is completely trained. Using the output of the before-last layer, we calculate the corresponding threshold for each intent.</p><p>Assume that our model is trained on the dataset <strong>D</strong> (of <strong>N</strong> intents). To compute the thresholds, we feed the output of the before-last dense layer into an exponential layer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*kqIukchltKbv8GuK_Tn6JQ.png" /></figure><p>Then, the confidence thresholds are given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*_23QMrC546SLoXraEpHX2A.png" /></figure><p>Experiment results show that the optimized value of delta is 0.3.</p><p>By fixing delta, the thresholds are computed on the training dataset that we have used on the second phase, therefore no need to have a new dataset.</p><p>Once the confidence thresholds are computed, we can use them to perform Non-Intent detection. For a new input query q, the predicted intent is obtained as follows:</p><pre>set k = argmax(exp_q) <br>set sigmoid_k = 1 / (1 + threshold(k) / exp_q(k)) <br>* if (softmax_q(k) * sigmoid_k &gt;= 0.5) <br>----&gt; intent k <br>* otherwise <br>----&gt; none intent</pre><p><strong>[1]</strong> J. Devlin, Ming-Wei C., K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/abs/1810.04805">https://arxiv.org/abs/1810.04805</a></p><p><strong>[2]</strong> Q. Chen, Z. Zhuo, W. Wang. BERT for Joint Intent Classification and Slot Filling. <a href="https://arxiv.org/abs/1902.10909">https://arxiv.org/abs/1902.10909</a></p><p><strong>[3]</strong> V. Abdelzad, K. Czarnecki, R. Salay, T. Denoundeny, S. Vernekary, B. Phan. Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output. <a href="https://arxiv.org/abs/1910.10307">https://arxiv.org/abs/1910.10307</a></p><p><strong>[4]</strong> Y. Zheng, G. Chen, M. Huang. Out-of-domain Detection for Natural Language Understanding in Dialog Systems. <a href="https://arxiv.org/abs/1909.03862">https://arxiv.org/abs/1909.03862</a></p><p><strong>[5]</strong> D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. <a href="https://arxiv.org/abs/1610.02136">https://arxiv.org/abs/1610.02136</a></p><p><strong>[6]</strong> S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. <a href="https://arxiv.org/abs/1706.02690">https://arxiv.org/abs/1706.02690</a></p><p><strong>[7]</strong> G. W. Taylor, T. DeVries. Learning Confidence for Out-of-Distribution Detection in Neural Networks. <a href="https://arxiv.org/abs/1802.04865">https://arxiv.org/abs/1802.04865</a></p><p><strong>[8]</strong> Anh N., J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. <a href="https://arxiv.org/abs/1412.1897">https://arxiv.org/abs/1412.1897</a></p><p><strong>[9]</strong> Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. <a href="https://arxiv.org/abs/1610.08401">https://arxiv.org/abs/1610.08401</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14bc4a16ec48" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>