<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Clemens Wolff on Medium]]></title>
        <description><![CDATA[Stories by Clemens Wolff on Medium]]></description>
        <link>https://medium.com/@clemens.wolff?source=rss-b134c3fd06b1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T0cLoC7aBxOiBe5Z1pvwJw.png</url>
            <title>Stories by Clemens Wolff on Medium</title>
            <link>https://medium.com/@clemens.wolff?source=rss-b134c3fd06b1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:59:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@clemens.wolff/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using the ambassador pattern for reliable pipes-and-filters text processing]]></title>
            <link>https://medium.com/@clemens.wolff/using-the-ambassador-pattern-for-reliable-pipes-and-filters-text-processing-9ebbea8f82c4?source=rss-b134c3fd06b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ebbea8f82c4</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Clemens Wolff]]></dc:creator>
            <pubDate>Wed, 06 Jun 2018 19:27:46 GMT</pubDate>
            <atom:updated>2019-01-20T09:15:22.059Z</atom:updated>
            <content:encoded><![CDATA[<p>In our last <a href="https://medium.com/@clemens.wolff/leveraging-microservices-for-relationship-based-text-exploration-9efd2a54f7ae">article</a>, <a href="https://medium.com/u/b6ae4f9ac99">Alex Anikiev</a>, <a href="https://medium.com/u/1951c98d933c">Arman Rahman</a> and I introduced a text exploration pipeline based on microservices. The pipeline contains a variety of processing steps such as extracting plain text from a document, cleaning up and canonicalizing the text, extracting insights, and so forth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*_C5UsQCI1LLjVmcbfCSyoQ.png" /><figcaption>Overview of the text exploration pipeline from the previous <a href="https://medium.com/@clemens.wolff/leveraging-microservices-for-relationship-based-text-exploration-9efd2a54f7ae">article</a>.</figcaption></figure><p>In essence, the system described in the previous article implements a <a href="http://www.enterpriseintegrationpatterns.com/patterns/messaging/PipesAndFilters.html">pipes-and-filters</a> architecture orchestrated by HTTP calls: each microservice receives an input document via a request, performs a stateless document transformation and then sends its output to one or more independent downstream processing steps. This architecture has several advantages including speed of development and ease of maintenance: frameworks such as <a href="https://www.npmjs.com/package/express">Express</a>, <a href="http://sanic.readthedocs.io/en/latest/">Sanic</a> or <a href="http://sparkjava.com/">Spark</a> enable us to quickly spin up web services and single-responsibility processing layers require less code to be understood or re-tested in the face of changes to any particular processing step.</p><p>However, the system has one big flaw: an orchestrator process has to manage the HTTP requests to the various services and pipe data between the inflight calls. This not only introduces a single point of failure but also makes it difficult to cleanly implement reliability features such as retries or handling intermittent failures. A natural way to improve system reliability is to introduce <a href="http://www.enterpriseintegrationpatterns.com/patterns/messaging/Introduction.html">task queues</a> between the processing steps, but this usually requires extensive code changes.</p><p>In this article we’ll show how we improved the reliability of the pipes-and-filters architecture via task queues without any service-side code changes by leveraging the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/ambassador">ambassador container pattern</a>, thus unlocking the best of both worlds: an easy to develop text processing system that also is fault tolerant.</p><h3>Definition</h3><p>Before diving deeper into how we made our text processing pipeline more reliable, let us first introduce the ambassador container on a conceptual level. Brendan Burns’ <a href="https://azure.microsoft.com/en-us/resources/designing-distributed-systems/en-us/">book</a> <em>Designing Distributed Systems</em> defines the ambassador container pattern as follows:</p><blockquote>An ambassador container brokers interactions between the application container and the rest of the world. […] The two containers are tightly linked in a symbiotic pairing that is scheduled to a single machine.</blockquote><p><em>Designing Distributed Systems</em> also provides concrete usage examples for the ambassador container pattern such as <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/sharding">service sharding</a>: when the application container makes a request to an external service, the ambassador handles the logic required to dispatch the request to the correct shard of the sharded service. This allows the application container to remain simple since it’s not burdened with the sharding logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNyRdvGBjZiKNRVteVqvyw.png" /><figcaption>Diagram demonstrating the use of the ambassador container pattern for service sharding</figcaption></figure><p>In the example above, the ambassador container hides the sharded nature of the dependent service from the application: the application does not need to be aware that it’s talking to a sharded service such as for example a <a href="https://redis.io/topics/cluster-tutorial">Redis cluster</a> where data grew too large to fit into the memory of a single instance.</p><p>Following this example, we can use a variation of the ambassador container to improve reliability for our pipes-and-filters text processing architecture by transparently adding task queues to the system. Our text processing application container remains unchanged as a HTTP service and the ambassador container takes care of the integration with a queuing system:</p><ul><li>Continuously, the ambassador container listens for messages on an input task queue.</li><li>When a message is received from the queue, the ambassador converts it into a web request to the text processing container and executes the request.</li><li>The ambassador container waits for the response from the text processing service and on receipt sends the appropriate messages to notify one or more downstream task queues of the result.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5cc6p_E1zKpFdXoTgcCOQ.png" /><figcaption>Diagram showing an ambassador container for injecting queues into the text processing pipeline</figcaption></figure><p>Note that the addition of the queues by construction improves the reliability of the system. For one, task queue implementations can be set up for high availability, thus removing the orchestrator as a single point of failure. Additionally, the task queues also protect against intermittent outages of the text processing containers: a message will only get deleted from the queue once the processing step is complete and all downstream messages have been enqueued; so if the processing of a message crashes part way through, it will simply be retried by the next available queue listener.</p><h3>Implementation</h3><p>We decided to implement the ambassador container pattern described in the previous section on top of <a href="https://azure.microsoft.com/en-us/services/service-bus/">Azure ServiceBus</a>, a highly scalable, fully managed distributed task queue. We chose ServiceBus instead of other managed queues offerings such as <a href="https://azure.microsoft.com/en-us/services/storage/queues/">Storage Queues</a> since it supports more advanced features such as <a href="https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues">dead-lettering</a> (which automatically purges unprocessable messages from the queue to avoid wasting resources on futile processing attempts) as well as <a href="https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-outages-disasters">geo-replication and failover</a>.</p><p>We additionally decided to implement the ambassador container in C# since (unlike the <a href="https://github.com/Azure/azure-sdk-for-node/tree/master/lib/services/serviceBus">Node SDK</a> or <a href="https://github.com/Azure/azure-sdk-for-python/tree/master/azure-servicebus">Python SDK</a>) the <a href="https://github.com/Azure/azure-service-bus-dotnet">DotNet Core SDK</a> for ServiceBus exposes a push-style API via <a href="https://www.amqp.org/">AMQP</a>. This simplifies the queue integration code by removing the need for implementing a message polling loop.</p><p>Setting up a simple tool that listens for ServiceBus messages, creates a post request and forwards the response to one or more downstream queues is fairly straight forward, as demonstrated by the snippet below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/534d8c1f12d8fe2aa6649da30556cd47/href">https://medium.com/media/534d8c1f12d8fe2aa6649da30556cd47/href</a></iframe><p>The snippet above necessarily will have to contain some domain-specific logic to convert between a ServiceBus message and a HTTP request (omitted here for brevity), however, if the text processing steps all expose a unified interface such as “expect a posted plain text file and return a text response”, the same ambassador container can be reused for all the processing microservices. Beyond the conversion logic, the code skeleton above is fully general and will enable any type of application to leverage transparent ServiceBus queues for improved reliability.</p><h3>Deployment</h3><p>After implementing the ambassador pattern as described in the previous section, we’re now ready to deploy the container. First, we can use a simple multi-stage Dockerfile to build and run the ambassador container:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6c400917e5da1f60d8b270aae3762d3/href">https://medium.com/media/c6c400917e5da1f60d8b270aae3762d3/href</a></iframe><p>Next, we can use the <a href="https://docs.docker.com/compose/compose-file/#depends_on">depends_on</a> directive in <a href="https://docs.docker.com/compose/overview/">Docker-Compose</a> to connect the ambassador with its service on a single machine:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f165ae0412fceda3ba382a9d0953517d/href">https://medium.com/media/f165ae0412fceda3ba382a9d0953517d/href</a></iframe><p>Alternatively, we can use <a href="https://kubernetes-v1-4.github.io/docs/user-guide/pods/multi-container/">multi-container pods</a> in <a href="https://kubernetes.io/">Kubernetes</a> to orchestrate the ambassador next to its service in a cluster context:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b75769b64bcf94d9d1bf08b476bf22f6/href">https://medium.com/media/b75769b64bcf94d9d1bf08b476bf22f6/href</a></iframe><h3>Conclusion</h3><p>In this article we discussed how to leverage the <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/ambassador">ambassador container pattern</a> and <a href="https://azure.microsoft.com/en-us/services/service-bus/">Azure ServiceBus</a> to improve the reliability of a pipes-and-filters text processing system by transparently introducing task queues. Additionally, the pattern can also be used for leveraging the ServiceBus push-style AMQP API from languages where the ServiceBus SDK does not implement AMQP, such as Node or Python.</p><p>The approach described in this article can be reused in any system where fault tolerance and automatic retries are important but where the application code should not be burdened with these concerns.</p><h3>Resources</h3><ul><li><a href="https://azure.microsoft.com/en-us/resources/designing-distributed-systems/en-us/">Brendan Burns’ “Designing Distributed Systems” book</a></li><li><a href="https://azure.microsoft.com/en-us/services/service-bus/">Overview page for Azure ServiceBus</a></li><li><a href="https://github.com/Azure/azure-service-bus-dotnet">C# SDK for Azure ServiceBus</a></li><li><a href="https://github.com/paolosalvatori/ServiceBusExplorer">Service Bus Explorer: useful debugging tool for Azure ServiceBus</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ebbea8f82c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leveraging microservices for relationship-based text exploration]]></title>
            <link>https://medium.com/@clemens.wolff/leveraging-microservices-for-relationship-based-text-exploration-9efd2a54f7ae?source=rss-b134c3fd06b1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9efd2a54f7ae</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[linked-data]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Clemens Wolff]]></dc:creator>
            <pubDate>Thu, 12 Apr 2018 13:22:53 GMT</pubDate>
            <atom:updated>2018-06-06T19:32:10.810Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/b6ae4f9ac99">Alex Anikiev</a>, <a href="https://medium.com/u/1951c98d933c">Arman Rahman</a> and I recently worked on building a text exploration tool which enables users to quickly dive into a collection of documents and use knowledge buried in the corpus to get insights on new topics and questions. For example, one might ingest a set of articles about science and then automatically establish links between scientists and their research domains.</p><p>A core part of the document exploration tool we built is a visualization that shows how various entities in the corpus relate to each other and that lets the user interactively explore further connections from each of those entities. This type of visualization enables the user to explore cross-document and cross-corpus linked ideas which otherwise may be difficult to discern.</p><p>In this article we’ll show how to use open information extraction and the resource description framework to build such an interactive entity-graph exploration widget. We’ll also show how microservices enable us to build a solution that leverages best-of-breed libraries across a wide variety of language ecosystems with minimal amounts of custom code.</p><h3>Architecture</h3><p>There are three main components to our relationship-based text exploration widget:</p><ol><li>A <em>relationship extraction component</em> enables a user to ingest a new document into the system and runs all analysis steps required for going from raw documents to interesting links and connections.</li><li>A <em>relationship storage component</em> takes the extracted relationships and stores them in a manner that can be efficiently and flexibly queried.</li><li>A <em>relationship visualization component</em> enables a user to explore the corpus of documents ingested into the system.</li></ol><p>We’ll explore each of these components in more detail in the following sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wDDYo4bwGKe4AUKhh5Eu9Q.png" /><figcaption>Abstract text exploration architecture</figcaption></figure><h3>Relationship extraction</h3><p>Taking an unstructured document and extracting meaningful relationships from it is <a href="http://www.cse.iitd.ac.in/~mausam/papers/ijcai16a.pdf">an active area of research</a> in the natural language processing community. <a href="https://en.wikipedia.org/wiki/Information_retrieval">Information retrieval</a> can be used to extract insights from documents but it’s often hard to structure or interconnect the outputs to create more complete knowledge representations. <a href="https://en.wikipedia.org/wiki/Information_extraction">Information extraction</a> can be used to map documents onto rich structured ontologies but creating and maintaining comprehensive ontologies requires substantial amounts of expensive manual effort. A promising way to approach the structured relationship extraction problem in the absence of formal ontologies is <a href="https://en.wikipedia.org/wiki/Open_information_extraction">open information extraction</a> (OpenIE).</p><p>OpenIE systems take unstructured input sentences and produce structured subject-relation-argument triples that encode the core pieces of information contained in the sentences. For example, for a sentence such as “Albert Einstein was born in Ulm and died in Princeton”, an OpenIE system might produce the subject-relation-argument triples (“Albert Einstein”, “was born in”, “Ulm”) and (“Albert Einstein”, “died in”, “Princeton”). The information captured in these triples is inherently more machine consumable than the original sentences since multiple facts can now be linked to specific actors and a network of relationships can be established automatically. For example, with the extracted relationships above, we can now create connections between the entities “Albert Einstein”, “Ulm” and “Princeton” via the “was born in” and “died in” predicates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dzlalpijSQR-F1ItYBcChg.png" /><figcaption>Overview of the OpenIE process</figcaption></figure><p>Several great OpenIE systems have been published, for example by <a href="https://nlp.stanford.edu/software/openie.html">Stanford University</a>, <a href="https://www.researchgate.net/publication/261265209_Open_Information_Extraction_via_Contextual_Sentence_Decomposition">Freiburg University</a> and the <a href="https://github.com/dair-iitd/OpenIE-standalone">University of Washington</a>. In this section, we’re going to show how to use <a href="https://github.com/aculich/clausie">ClausIE</a>, the OpenIE system created by the <a href="http://resources.mpi-inf.mpg.de/d5/clausie/clausie-www13.pdf">Max Planck Institute</a> since it is published under a relatively easy-to-integrate open source <a href="https://github.com/aculich/clausie/blob/16c70b4bc8de22e65439fe46eade8f511066a388/clausie_license.html">license</a> and offers near state of the art performance.</p><p>ClausIE is implemented in Java so to turn it into a microservice consumable from other ecosystems, we used <a href="http://sparkjava.com/">Spark</a> to wrap the OpenIE functionality. Spark is a lightweight web framework for Java, comparable to Ruby’s Sinatra or Python’s Flask. Spark’s high-level API makes it easy to implement a simple route to wrap the ClausIE system:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffcf86cc36c75160da11958857d09993/href">https://medium.com/media/ffcf86cc36c75160da11958857d09993/href</a></iframe><p>Note that in the example above we’re explicitly enabling multipart requests in Spark to enable efficient processing of large text documents in a streaming fashion which reduces memory requirements for the service.</p><p>The ClausIE OpenIE microservice is published on <a href="https://github.com/c-w/ClausIE-Server">Github</a> and can be run via <a href="https://www.docker.com/what-docker">Docker</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba6d845dd200145a45e430bbfb0d0939/href">https://medium.com/media/ba6d845dd200145a45e430bbfb0d0939/href</a></iframe><p>As shown in the example above, the open information extraction endpoint expects a text file as input where each line corresponds to one sentence. To prepare arbitrary documents to fit this format, we use two additional microservices for document pre-processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MnxrR5jqraEEjrAbJ9OYnQ.png" /><figcaption>Relationship extraction pipeline overview</figcaption></figure><p>First, we use <a href="https://tika.apache.org/">Apache Tika</a> to extract plain text content from a wide range of document formats including docx, pdf and <a href="https://tika.apache.org/1.8/formats.html">many more</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85a28a9ca69465a3a2cb65d40b0e26e6/href">https://medium.com/media/85a28a9ca69465a3a2cb65d40b0e26e6/href</a></iframe><p>Note the “accept” header in the code snippet above. This header is required when calling the Apache Tika service to ensure that the document parser returns plain text output as opposed to say XML or HTML; failure to specify the header will lead to undefined behavior.</p><p>After turning raw documents into plain text, we then use a simple <a href="http://sanic.readthedocs.io/en/latest/">Sanic</a> service that leverages the excellent text-processing tools available in Python, such as <a href="https://www.nltk.org/">NLTK</a>, to easily parse the extracted document text, split it into sentences and perform rich cleanups such as discarding poorly converted sentences and so forth:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28a0dd6948cc965dfe4ea51535661733/href">https://medium.com/media/28a0dd6948cc965dfe4ea51535661733/href</a></iframe><p>The cleanup service can then be run via Docker, just like the OpenIE server and the Apache Tika server:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c469f7ead7353b60a68b4d148e1a5551/href">https://medium.com/media/c469f7ead7353b60a68b4d148e1a5551/href</a></iframe><p>In addition to the pre-processing steps described above, relationship pruning post-processing steps are often beneficial to ensure that the OpenIE extractions are of high quality for downstream use. OpenIE is domain independent by design which means that adding domain-specific processing steps will improve the salience of the extracted relationships for the particular context of the ingested documents, for example by canonicalizing multiple written forms of the same logical entity to one root textual string to increase the density of the OpenIE relationship graph for that entity. We’ll cover the topic of OpenIE post-processing in a future article.</p><p>Note that the pre-processing, relationship extraction and post-processing steps are each <a href="https://en.wikipedia.org/wiki/Embarrassingly_parallel">embarrassingly parallel</a>: every document can be processed independently of the rest of the corpus. Given that the services introduced above are already containerized, it is easy to scale the ingestion process, for example by deploying the pipeline to a <a href="https://azure.microsoft.com/en-us/services/container-service/">managed Kubernetes cluster</a> and setting up a <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">horizontal pod autoscaler</a>.</p><h3>Relationship storage</h3><p>In the previous section we described three microservices that enable us to extract semantically interesting subject-relation-argument triples from arbitrary documents. Ingesting a document via the pipeline takes a few minutes of processing so it’s impractical to run the pipeline live each time a user makes a query. As such, we need to define a method to store the produced relationships at ingestion time, as well as a flexible querying mechanism to retrieve them at exploration time.</p><p>A natural fit for storing triple information that represents linked facts is the <a href="https://en.wikipedia.org/wiki/Resource_Description_Framework">resource description framework</a> (RDF). RDF is a data format that was created as part of research into the semantic web to enable machine-consumable description of information with support for complex querying and inference. RDF is supported by a mature ecosystem of tools and libraries which make it easy to implement functionality on top of it.</p><p>We can convert the <a href="http://jsonlines.org/">JSONL</a> output of our OpenIE microservice to RDF in <a href="https://en.wikipedia.org/wiki/Turtle_(syntax)">Turtle format</a> using Python’s <a href="https://rdflib.readthedocs.io/en/stable/">RDFlib</a>. Turtle is an efficient textual storage format for RDF triples that can be ingested by most RDF-based tools:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2d765dd418133c2b548cdb504d11c98/href">https://medium.com/media/f2d765dd418133c2b548cdb504d11c98/href</a></iframe><p>We can then store the RDF relationships in the <a href="https://jena.apache.org/documentation/tdb/index.html">Apache Jena</a> database. Apache Jena is a <a href="https://www.w3.org/wiki/LargeTripleStores#Jena_TDB_.281.7B.29">high performance</a> single-node RDF data-store. The <a href="https://jena.apache.org/documentation/fuseki2/">Fuseki</a> REST interface makes it easy to interact with Apache Jena:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dedcf0ccedd43b45b98f2694c635073a/href">https://medium.com/media/dedcf0ccedd43b45b98f2694c635073a/href</a></iframe><p>Similarly, the Fuseki REST interface also makes it easy to query our RDF database using <a href="https://en.wikipedia.org/wiki/SPARQL">SPARQL</a>. SPARQL is a powerful SQL-like querying language that enables complex queries across RDF relationships. For example, to find all relationships about victories one could run the following query:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/333cac0af68505322022fba4cbd47ce4/href">https://medium.com/media/333cac0af68505322022fba4cbd47ce4/href</a></iframe><p>Note that RDF inherently stores relationships as pure graphs. As such, duplicate relationships are not supported. To preserve duplicates across a collection of documents, the snippets above can be adapted to use <a href="https://www.w3.org/TR/rdf-primer/#reification">reification</a> or a <a href="https://jena.apache.org/tutorials/sparql_datasets.html">named graph</a> per ingested document.</p><p>Reification is the process of introducing meta-statements to the RDF store that claim provenance of each factual statement, such as (“document X”, “said that”, “relationship Y”) where relationship Y is a pointer to a statement like (“Albert Einstein”, “was born in”, “Ulm”). The disadvantage of this approach is that for each fact stored in the RDF database, multiple meta-facts now have to be stored which drastically increases storage size. Named graphs, on the other hand, solve the provenance issue by assigning statements to explicit sub-graphs in the RDF store at ingestion time, similar to having shards in a traditional SQL database. SPARQL then enables retrieval of facts from one or more named graphs at querying time via the <a href="https://www.w3.org/TR/rdf-sparql-query/#namedGraphs">FROM NAMED statement</a>.</p><p>If a use-case makes it necessary to preserve duplicates inside of a single document (e.g. to visualize how often particular entity relationship occurred in the document), it’s easiest to mirror the relationships to another datastore such as <a href="https://www.postgresql.org/">PostgreSQL</a> in addition to storing them in Apache Jena. This polyglot storage solution enables in-document duplicate-based queries executed against PostgreSQL in addition to graph-based SPARQL queries executed against Apache Jena.</p><p>Tools like <a href="http://aksw.org/Projects/Sparqlify.html">Sparqlify</a> could be used to run SPARQL queries directly against PostgreSQL and remove the need for maintaining two datastores but this is not recommended for a production setup since SPARQL to SQL translation often only supports a limited subset of the SPARQL querying language which limits query flexibility.</p><h3>Relationship visualization</h3><p>One advantage of using a technology like RDF to store and query the entity relationships extracted from our documents is that we can draw from a rich ecosystem of user interaction tools that have been built in this area.</p><p>For example, we can leverage the <a href="https://yasqe.yasgui.org/">YASQE</a> library to easily create a rich user interface for writing SPARQL queries with code completion, syntax highlighting, query execution, and so forth:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f5a4648b932cfd55e9d7bab1203df9b2/href">https://medium.com/media/f5a4648b932cfd55e9d7bab1203df9b2/href</a></iframe><p>Given that the result of a SPARQL query always describes a graph, we can implement a simple generic re-usable visualization for the query response using the <a href="https://js.cytoscape.org/">Cytoscape</a> graphing library:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dde89e6320b958305c5ccec9db26d43/href">https://medium.com/media/1dde89e6320b958305c5ccec9db26d43/href</a></iframe><p>Note that the snippet above assumes that the SELECT statement in the user’s SPARQL queries will always contain a subject clause, an argument clause and a relation clause.</p><p>We can use the query editor and graph visualization shown above to for example find all relationships that start with the entity “Albert Einstein” (as in previous examples in this article):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*hPsN8asQcaCSElx3N7df2g.png" /><figcaption>Visualization of SPARQL query editor and response graph</figcaption></figure><p>Cytoscape also enables us to easily post-process the displayed graph. For example, to only display interesting dense relationships that span more than two entities, we could execute the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ec68e95b7031d0aee73d983b2a1a762/href">https://medium.com/media/0ec68e95b7031d0aee73d983b2a1a762/href</a></iframe><h3>Conclusion</h3><p>In this article we introduced a text exploration pipeline that visualizes relationships between entities within a document and across a collection of documents.</p><p>The pipeline leverages a number of open source tools such as <a href="https://tika.apache.org/">Apache Tika</a>, <a href="https://jena.apache.org/documentation/fuseki2/">Apache Jena Fuseki</a>, <a href="https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/software/clausie/">ClausIE</a>, <a href="https://www.nltk.org/">NLTK</a>, <a href="http://yasqe.yasgui.org/">YASQE</a> and <a href="http://js.cytoscape.org/">Cytoscape</a>. Using Docker and a microservices approach makes it easy to integrate these services that otherwise live in their own ecosystems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_C5UsQCI1LLjVmcbfCSyoQ.png" /><figcaption>Overview of end-to-end text exploration pipeline</figcaption></figure><p>We’ve demonstrated how to leverage powerful tools from the semantic web community such as RDF and SPARQL for natural language processing explorations. We also introduced <a href="https://github.com/c-w/ClausIE-Server">ClausIE-Server</a>, an open source microservice for open information extraction. If your application needs to make sense of unstructured text, give ClausIE-Server a try!</p><p>In future articles we’ll cover enhancements to the text exploration tool such as how to improve the reliability of the pipeline using <a href="https://medium.com/@clemens.wolff/using-the-ambassador-pattern-for-reliable-pipes-and-filters-text-processing-9ebbea8f82c4">transparent task queues</a>, how to improve the quality of the generated relationships via coreference resolution, named entity recognition and domination theory or how to increase the dynamism of the visualization by auto-generating SPARQL queries based on user interactions with the entity relationship graph.</p><h3>Resources</h3><ul><li><a href="https://github.com/c-w/ClausIE-Server">Dockerized open information extraction microservice based on ClausIE</a></li><li><a href="https://github.com/LogicalSpark/docker-tikaserver">Dockerfile for Apache Tika document content extraction service</a></li><li><a href="https://github.com/stain/jena-docker/tree/master/jena-fuseki">Dockerfile for Apache Jena Fuseki linked data service</a></li><li><a href="http://js.cytoscape.org/">Documentation for easy graph visualization via cytoscape.js</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9efd2a54f7ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>