<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhijith Nair on Medium]]></title>
        <description><![CDATA[Stories by Abhijith Nair on Medium]]></description>
        <link>https://medium.com/@abhi1993nair?source=rss-308382bf640e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*f8-nH1VQgmpcU70A.</url>
            <title>Stories by Abhijith Nair on Medium</title>
            <link>https://medium.com/@abhi1993nair?source=rss-308382bf640e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:24:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhi1993nair/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ontology: Teach the model, not the Parser]]></title>
            <link>https://medium.com/@abhi1993nair/ontology-teach-the-model-not-the-parser-a469c47de6f2?source=rss-308382bf640e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a469c47de6f2</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[ontology]]></category>
            <dc:creator><![CDATA[Abhijith Nair]]></dc:creator>
            <pubDate>Sun, 16 Aug 2026 03:23:28 GMT</pubDate>
            <atom:updated>2026-08-16T03:23:28.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qyc0mZryl9gaWV6KGAP9hw.png" /></figure><p><em>Second in a series on ontologies.</em> <em>Part 1: </em><a href="https://medium.com/REPLACE-WITH-PART-1-URL"><em>Ontology — the 2,000-year-old idea your data pipeline needs</em></a></p><p>Part 1 opened on a file every team has. Four fields, three suppliers, twelve different names for them, and a mappings.py that somebody updates every quarter and that breaks quietly when a vendor renames a column. It closed on a line that sounds like a caveat and is actually a design brief:</p><blockquote><strong><em>The ontology is not the vocabulary you download. It is the constraints you add to it.</em></strong></blockquote><p>This post takes that literally, on real data, and counts the cost in triples.</p><h3>So you went and downloaded the standard</h3><p>Sensible. For product data that standard is the <strong>GS1 Web Vocabulary</strong> — from the people behind the barcode. 18,749 triples, free, maintained by somebody who is not you.</p><p>Then you point it at an actual supplier export and <strong>nothing maps.</strong> Not one field out of twenty-four.</p><p>That is not a bug, and it is not unusual. GS1 has a property for nearly everything in that file. It simply does not know that the column called sugars is its sugarsPerNutrientBasis, that fiber is its fibrePerNutrientBasis spelled the other way round, that traces and allergens are one relation separated by a code, or that mg and g are the same dimension a thousand apart. A standard names things in its own language. Your data arrives in somebody else&#39;s.</p><p>Part 1 measured something else about that file, too: it <strong>declares 56 classes and refers to 123</strong>. So the vocabulary is not only speaking a different language from your data, it is incomplete in its own terms.</p><p>The usual next move is a mapping file. Then a parser. Then a second parser when the next supplier signs — which is the file Part 1 started with, rebuilt one storey up.</p><p><strong>This post does the other thing. Every gap gets closed with triples, and the Python is never touched again.</strong></p><h3>The claim</h3><p>A vocabulary you did not write, cannot edit and do not control can still catalogue a dataset it has never seen — if you extend it with <em>axioms</em> rather than code. The cataloguer below contains zero dataset knowledge. Every time it learns a new field, a new unit or a new value shape, the ontology grows and the code does not.</p><p>The data is a real Open Food Facts export: <strong>1,515 products, 851 columns.</strong> Nothing here is invented. Where GS1 has no property for something, we say so and show the gap.</p><p>By the end, the object that catalogued a cereal export reads a biscuit export it has never opened — <strong>24 fields out of 24, no new code and no new triples</strong>. The total cost of everything added to GS1 comes to <strong>288 triples</strong>. And the mapping document the industry-standard tooling would have wanted turns out to be something you can generate from the ontology rather than write by hand.</p><h3>How to read the code</h3><p>Everything below is triples. If you have not written RDF before, this is the whole of the syntax you need, and it is genuinely small.</p><p><strong>A triple is a sentence: subject, predicate, object.</strong> g.add((s, p, o)) is the only way anything gets into a graph, and a graph is a set of those sentences. There are no tables and no columns, so adding a fact never changes the shape of anything already there - which is why an ontology can grow all post without a migration.</p><p><strong>Names are URLs.</strong> gs1:sugarsPerNutrientBasis is shorthand for https://ref.gs1.org/voc/sugarsPerNutrientBasis. The Namespace objects in the next cell are that shortening: gs1: is GS1&#39;s, ext: is ours, d: is the catalogued data. Using URLs as names is the whole reason two vocabularies written by strangers can meet without a coordination meeting.</p><p><strong>Three kinds of thing can sit in a triple.</strong> A URIRef is a named thing. A Literal is a value - a string, a number, a language-tagged piece of text. A BNode is a thing that exists but needs no name: <em>the sugars measurement on this product</em> is a real node carrying a value and a unit, and nothing outside needs to link to it.</p><p>Four predicates do nearly all the work here, and each is a <strong>claim about the world</strong>, not a rule for a validator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*XQ6rzpDHc3H8tPF2gMJBAA.png" /></figure><p>The right-hand column is the point, and it is the thing most easily misread by anyone arriving from JSON Schema or Pydantic. These do not check anything. A reasoner reads them forwards and <em>adds the facts they imply</em>: rdfs:domain never complains that your record is not a food product, it concludes that it is.</p><p>Three more names appear constantly. owl:Class, owl:ObjectProperty and owl:DatatypeProperty declare what a name refers to - a category, a link between two things, or a link from a thing to a value. And skos:altLabel is nothing more than &quot;another name for this&quot;, which turns out to be the hinge the whole cataloguer turns on.</p><p>The right-hand column is the point, and it is the thing most easily misread by anyone arriving from JSON Schema or Pydantic. These do not check anything. A reasoner reads them forwards and <em>adds the facts they imply</em>: rdfs:domain never complains that your record is not a food product, it concludes that it is.</p><p>Three more names appear constantly. owl:Class, owl:ObjectProperty and owl:DatatypeProperty declare what a name refers to - a category, a link between two things, or a link from a thing to a value. And skos:altLabel is nothing more than &quot;another name for this&quot;, which turns out to be the hinge the whole cataloguer turns on.</p><pre>import re<br>from rdflib import Graph, Namespace, Literal, BNode, URIRef, RDF, RDFS, OWL, XSD<br>from rdflib.namespace import SKOS<br>from owlviz import draw_record, draw_property_tree # Custom Package<br><br>GS1  = Namespace(&quot;https://ref.gs1.org/voc/&quot;)     # the standard<br>EXT  = Namespace(&quot;http://example.org/gs1x/&quot;)     # our extension - GS1 stays untouched<br>DATA = Namespace(&quot;http://example.org/data/&quot;)     # the catalogued records<br><br>import textwrap<br><br>def wrapped(items, indent=&quot;  &quot;, width=72):<br>    &quot;&quot;&quot;One long list on one long line is unreadable. Reflow it.&quot;&quot;&quot;<br>    return textwrap.fill(&quot;, &quot;.join(str(x) for x in items), width=width,<br>                         initial_indent=indent, subsequent_indent=indent)</pre><h3>The dataset</h3><p>851 columns is the problem statement. Before modelling anything, look at how they are distributed — because the answer decides what an ontology is even for here.</p><pre>import pandas as pd<br>import collections<br><br>df = pd.read_csv(&quot;openfoodfacts_export.csv&quot;, sep=&quot;\t&quot;, dtype=str,<br>                 keep_default_na=False, na_values=[&quot;&quot;])<br><br>fill = {c: df[c].notna().sum() for c in df.columns}<br>nut  = [c for c in df.columns if c.startswith(&quot;nutrition.&quot;)]<br><br>print(f&quot;{len(df):,} products, {len(df.columns):,} columns &quot;<br>      f&quot;({sum(1 for c in df.columns if fill[c] == 0)} entirely empty)&quot;)<br>print(f&quot;  nutrition : {len(nut):3d} columns&quot;)<br>print(f&quot;  metadata  : {len(df.columns) - len(nut):3d} columns&quot;)<br><br>paths = collections.Counter()<br>for c in nut:<br>    m = re.match(r&quot;nutrition\.input_sets\.(.+?)\.nutrients\.&quot;, c)<br>    if m:<br>        paths[m.group(1)] += fill[c]<br>print(&quot;\nnutrient values by input set:&quot;)<br>for k, v in paths.most_common(4):<br>    print(f&quot;  {k:34s} {v:7,}&quot;)</pre><pre>1,515 products, 851 columns (2 entirely empty)<br>  nutrition : 644 columns<br>  metadata  : 207 columns<br><br>nutrient values by input set:<br>  estimate.as_sold.100g               41,428<br>  packaging.as_sold.serving           22,368<br>  packaging.as_sold.100g              20,185<br>  packaging.prepared.serving              83</pre><p>Two facts worth keeping.</p><p><strong>644 of the 851 columns are nutrition</strong>, and they are not one table — they are the same nutrients repeated across input sets, bases and serving definitions. The largest single source is estimate.as_sold.100g, which is Open Food Facts&#39; own estimate rather than anything printed on a label.</p><p><strong>The remaining 207 columns are product metadata</strong> — names in fifty languages, ingredients in thirty, allergens, categories, countries, brands, packaging. GS1 has properties for nearly all of it, which is the point of using a standard somebody else maintains.</p><h3>Provenance</h3><p>Both exports are Open Food Facts data, used under the Open Database Licence (ODbL) — attribution to Open Food Facts contributors, and any derived database published on the same terms. gs1Voc.ttl is the GS1 Web Vocabulary as published. The checksums pin the exact files this post was run against, because every number below is measured from them.</p><pre>import sys, os, hashlib, datetime, platform<br>import owlrl, pyshacl, graphviz<br><br>def md5(path, chunk=1 &lt;&lt; 20):<br>    h = hashlib.md5()<br>    with open(path, &quot;rb&quot;) as f:<br>        for block in iter(lambda: f.read(chunk), b&quot;&quot;):<br>            h.update(block)<br>    return h.hexdigest()[:12]<br><br>import rdflib<br>print(f&quot;python     {sys.version.split()[0]}   {platform.system()} {platform.release()}&quot;)<br>for mod in (rdflib, owlrl, pyshacl, pd, graphviz):<br>    print(f&quot;{mod.__name__:11s}{getattr(mod, &#39;__version__&#39;, &#39;n/a&#39;)}&quot;)<br><br>print()<br>for path, what in [(&quot;gs1Voc.ttl&quot;, &quot;GS1 Web Vocabulary (Turtle, as published)&quot;),<br>                   (&quot;openfoodfacts_export.csv&quot;, &quot;Open Food Facts export - cereal (ODbL)&quot;),<br>                   (&quot;openfoodfacts_export_biscut.csv&quot;, &quot;Open Food Facts export - biscuit (ODbL)&quot;),<br>                   (&quot;shapes.ttl&quot;, &quot;our SHACL shapes&quot;)]:<br>    st = os.stat(path)<br>    print(f&quot;{path}&quot;)<br>    print(f&quot;    {st.st_size / 1e6:7.2f} MB   md5:{md5(path)}   &quot;<br>          f&quot;{datetime.date.fromtimestamp(st.st_mtime)}   {what}&quot;)</pre><pre>python     3.11.14   Windows 10<br>rdflib     7.4.0<br>owlrl      7.1.4<br>pyshacl    0.31.0<br>pandas     2.3.3<br>graphviz   0.21<br><br>gs1Voc.ttl<br>       1.27 MB   md5:b2dea5788b88   2026-08-10   GS1 Web Vocabulary (Turtle, as published)<br>openfoodfacts_export.csv<br>       3.06 MB   md5:c18e929418c0   2026-08-11   Open Food Facts export - cereal (ODbL)<br>openfoodfacts_export_biscut.csv<br>       0.06 MB   md5:13d024a028a3   2026-08-12   Open Food Facts export - biscuit (ODbL)<br>shapes.ttl<br>       0.00 MB   md5:873618321ca1   2026-08-15   our SHACL shapes</pre><h3>Load GS1 exactly as published</h3><p>No conversion, no editing, no subsetting.</p><p>GS1 <em>does</em> ship a unit property. gs1:unitCode has domain QuantitativeValue, range xsd:string, a skos:exactMatch to schema:unitCode, and a comment pointing at UN/ECE Recommendation 20 - the list where GRM means gram. What GS1 does not ship is that list. Nothing in the file says a gram is a thousandth of a kilogram, or that grams and kilojoules are not comparable quantities.</p><p>So we keep GS1’s property and supply the ten individuals it refers to and never defines. That is the first gap, and a small one.</p><pre>def load_vocabulary(ttl=&quot;gs1Voc.ttl&quot;):<br>    &quot;&quot;&quot;GS1 as published, plus the unit code list it names but never supplies.&quot;&quot;&quot;<br>    g = Graph(); g.parse(ttl, format=&quot;turtle&quot;)<br>    g.bind(&quot;gs1&quot;, GS1); g.bind(&quot;ext&quot;, EXT); g.bind(&quot;skos&quot;, SKOS)<br><br>    # a unit code is a thing, not a string: it has a dimension and a conversion<br>    g.add((EXT.UnitCode, RDF.type, OWL.Class))<br>    for p, rng in ((&quot;toBase&quot;, XSD.float), (&quot;dimension&quot;, XSD.string),<br>                   (&quot;code&quot;, XSD.string)):<br>        g.add((EXT[p], RDF.type, OWL.DatatypeProperty))<br>        g.add((EXT[p], RDFS.domain, EXT.UnitCode))<br>        g.add((EXT[p], RDFS.range, rng))<br><br>    # a measurement carries what the label declared - gs1:value with gs1:unitCode,<br>    # GS1&#39;s own pair - and the same quantity normalised, which is ours<br>    for p, rng, kind in ((&quot;unitCodeRef&quot;, EXT.UnitCode, OWL.ObjectProperty),<br>                         (&quot;baseValue&quot;,   XSD.float,    OWL.DatatypeProperty)):<br>        g.add((EXT[p], RDF.type, kind))<br>        g.add((EXT[p], RDFS.domain, GS1.QuantitativeValue))<br>        g.add((EXT[p], RDFS.range, rng))<br><br>    for code, pref, base, dim, alts in [<br>        (&quot;GRM&quot;, &quot;g&quot;,    1.0,     &quot;mass&quot;,   [&quot;g&quot;, &quot;gr&quot;, &quot;gram&quot;, &quot;grams&quot;]),<br>        (&quot;KGM&quot;, &quot;kg&quot;,   1000.0,  &quot;mass&quot;,   [&quot;kg&quot;, &quot;kilogram&quot;]),<br>        (&quot;MGM&quot;, &quot;mg&quot;,   0.001,   &quot;mass&quot;,   [&quot;mg&quot;, &quot;milligram&quot;]),<br>        (&quot;MC&quot;,  &quot;\u00b5g&quot;,   1e-6,    &quot;mass&quot;,   [&quot;\u00b5g&quot;, &quot;ug&quot;, &quot;mcg&quot;, &quot;microgram&quot;]),<br>        (&quot;ONZ&quot;, &quot;oz&quot;,   28.3495, &quot;mass&quot;,   [&quot;oz&quot;, &quot;ounce&quot;]),<br>        (&quot;MLT&quot;, &quot;ml&quot;,   1.0,     &quot;volume&quot;, [&quot;ml&quot;, &quot;millilitre&quot;]),<br>        (&quot;LTR&quot;, &quot;l&quot;,    1000.0,  &quot;volume&quot;, [&quot;l&quot;, &quot;litre&quot;, &quot;liter&quot;]),<br>        (&quot;E14&quot;, &quot;kcal&quot;, 1.0,     &quot;energy&quot;, [&quot;kcal&quot;, &quot;cal&quot;]),<br>        (&quot;KJO&quot;, &quot;kJ&quot;,   0.239,   &quot;energy&quot;, [&quot;kj&quot;, &quot;kjoule&quot;]),<br>        (&quot;P1&quot;,  &quot;%&quot;,    1.0,     &quot;ratio&quot;,  [&quot;%&quot;, &quot;percent&quot;]),<br>    ]:<br>        u = EXT[code]<br>        g.add((u, RDF.type, EXT.UnitCode))<br>        g.add((u, EXT.code, Literal(code)))               # the UN/ECE Rec 20 code<br>        g.add((u, SKOS.prefLabel, Literal(pref)))<br>        g.add((u, EXT.toBase, Literal(base, datatype=XSD.float)))<br>        g.add((u, EXT.dimension, Literal(dim)))<br>        for a in alts:<br>            g.add((u, SKOS.altLabel, Literal(a)))<br><br>    # the mapping vocabulary itself, declared rather than merely used - GS1 is<br>    # criticised below for exactly this, so our own extension should not do it<br>    for cls in (EXT.FieldMapping, EXT.TagList):<br>        g.add((cls, RDF.type, OWL.Class))<br>    for p, rng, kind in ((&quot;sourceField&quot;,     XSD.string,   OWL.DatatypeProperty),<br>                         (&quot;languageTag&quot;,     XSD.string,   OWL.DatatypeProperty),<br>                         (&quot;mapsTo&quot;,          RDF.Property, OWL.ObjectProperty),<br>                         (&quot;valueShape&quot;,      OWL.Class,    OWL.ObjectProperty),<br>                         (&quot;tagClass&quot;,        OWL.Class,    OWL.ObjectProperty),<br>                         (&quot;tagCodeProperty&quot;, RDF.Property, OWL.ObjectProperty),<br>                         (&quot;fixedProperty&quot;,   RDF.Property, OWL.ObjectProperty),<br>                         (&quot;fixedValue&quot;,      OWL.Thing,    OWL.ObjectProperty)):<br>        g.add((EXT[p], RDF.type, kind))<br>        g.add((EXT[p], RDFS.domain, EXT.FieldMapping))<br>        g.add((EXT[p], RDFS.range, rng))<br><br>    # the common parent GS1 never declared for its 43 nutrient properties<br>    g.add((EXT.nutrientPerBasis, RDF.type, OWL.ObjectProperty))<br>    return g<br><br><br>onto = load_vocabulary()<br>print(f&quot;{len(onto):,} triples - GS1 as published, plus &quot;<br>      f&quot;{len(list(onto.subjects(RDF.type, EXT.UnitCode)))} units&quot;)</pre><pre>18,867 triples - GS1 as published, plus 10 units</pre><h3>One triple, four inferences</h3><p>Before any of our work, check what the published vocabulary does on its own.</p><p>Assert a single field on a record with <strong>no type at all</strong>, run RDFS closure, and ask what the record is.</p><p><em>Closure</em> is the one piece of machinery worth naming. owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(g) takes the four predicates above, writes down every fact that follows from them, and repeats until nothing new appears. It rewrites the graph in place, so afterwards the inferred triples sit beside the asserted ones and no query can tell them apart - a convenience and a trap in equal measure, and one this post returns to.</p><pre>import owlrl<br><br>probe = Graph(); probe += onto<br>x, m = DATA[&quot;unknown_thing&quot;], BNode()<br>probe.add((x, GS1.energyPerNutrientBasis, m))      # ONE triple. no rdf:type anywhere.<br>probe.add((m, GS1.value, Literal(250.0)))<br><br>print(&quot;asserted types:&quot;, [str(o) for o in probe.objects(x, RDF.type)] or &quot;NONE&quot;)<br><br>owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(probe)<br>local = lambda u: str(u).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br>keep  = lambda n: [local(o) for o in probe.objects(n, RDF.type)<br>                   if str(o).startswith(str(GS1))]<br><br>print(&quot;inferred record :&quot;, sorted(keep(x)))<br>print(&quot;inferred measure:&quot;, sorted(keep(m)))</pre><pre>asserted types: NONE<br>inferred record : [&#39;FoodBeverageTobaccoProduct&#39;, &#39;Product&#39;]<br>inferred measure: [&#39;NutritionMeasurementType&#39;, &#39;QuantitativeValue&#39;]</pre><p>The record is a FoodBeverageTobaccoProduct and therefore a Product. The value is a NutritionMeasurementType and therefore a QuantitativeValue. Four types that nobody asserted, derived from one field name.</p><p>None of our axioms were loaded. This is rdfs:domain, rdfs:range and rdfs:subClassOf in the published file, doing exactly the job they were written for. <strong>The vocabulary already knows what things are.</strong> That is what we are building on.</p><h3>What GS1 gives, and what it withholds</h3><p>It knows <em>what a thing is</em>. It has far less to say about <em>how measurements relate to each other</em>.</p><pre>nutrients = sorted(p for p in set(onto.subjects(RDF.type, OWL.ObjectProperty))<br>                   if str(p).endswith(&quot;PerNutrientBasis&quot;))<br>with_parent = [p for p in nutrients if list(onto.objects(p, RDFS.subPropertyOf))]<br><br>print(f&quot;*PerNutrientBasis properties      : {len(nutrients)}&quot;)<br>print(f&quot;  ...with a declared domain+range : &quot;<br>      f&quot;{sum(1 for p in nutrients if list(onto.objects(p, RDFS.domain)) and list(onto.objects(p, RDFS.range)))}&quot;)<br>print(f&quot;  ...with any subPropertyOf parent: {len(with_parent)}&quot;)<br>print(f&quot;\nowl:disjointWith axioms in GS1    : {len(list(onto.triples((None, OWL.disjointWith, None))))}&quot;)<br>print(f&quot;skos:altLabel anywhere on a nutrient: &quot;<br>      f&quot;{sum(len(list(onto.objects(p, SKOS.altLabel))) for p in nutrients)}&quot;)</pre><pre>*PerNutrientBasis properties      : 43<br>  ...with a declared domain+range : 43<br>  ...with any subPropertyOf parent: 0<br><br>owl:disjointWith axioms in GS1    : 0<br>skos:altLabel anywhere on a nutrient: 0</pre><p>Every nutrient is fully typed and <strong>not one of them has a parent.</strong> Sugars, added sugars, glucose and lactose are 43 unrelated properties. There is no “nutrient” concept to ask a question against, and no way to roll anything up.</p><p>Nor does a single nutrient carry an altLabel. GS1 has 25 in the whole file, on things like bestBeforeDate and grossArea - none on a nutrient. That is the other half of the problem: a standard names properties in its own language, and no supplier writes fibrePerNutrientBasis in a spreadsheet.</p><p>Both gaps are additive. Neither requires editing GS1.</p><h3>The cataloguer</h3><p>This class contains no dataset knowledge and no field names. It reads five things from the ontology at runtime:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLQP7B581JK6x-G5KW9dGw.png" /></figure><p>Everything it can do is a consequence of what the ontology says. After this cell it never changes again.</p><pre>QTY = re.compile(r&quot;^\s*(?:(\d+)\s*[x\u00d7*]\s*)?(\d+(?:[.,]\d+)?)\s*([a-zA-Z%\u00b5/\u00b2]+)?\.?\s*$&quot;)<br><br><br>class Cataloguer:<br>    &quot;&quot;&quot;Maps arbitrary records onto an ontology. Contains no dataset knowledge.&quot;&quot;&quot;<br><br>    def __init__(self, onto: Graph):<br>        self.g     = onto<br>        self.field = self._field_index()     # source field name -&gt; property<br>        self.mapf  = self._mapping_index()   # source field name -&gt; FieldMapping<br>        self.unit  = self._unit_index()      # unit spelling     -&gt; unit individual<br>        self.range = self._range_index()     # property          -&gt; expected shape<br><br>    def _field_index(self):<br>        ix = {}<br>        for p in set(self.g.subjects(RDF.type, OWL.DatatypeProperty)) | \<br>                 set(self.g.subjects(RDF.type, OWL.ObjectProperty)):<br>            for lbl in self.g.objects(p, SKOS.altLabel):<br>                ix[str(lbl).strip().lower()] = p<br>        return ix<br><br>    def _mapping_index(self):<br>        &quot;&quot;&quot;ext:FieldMapping nodes describe columns that need more than a name.&quot;&quot;&quot;<br>        ix = {}<br>        for fm in self.g.subjects(RDF.type, EXT.FieldMapping):<br>            src = str(next(self.g.objects(fm, EXT.sourceField), &quot;&quot;)).strip().lower()<br>            if src:<br>                ix[src] = fm<br>        return ix<br><br>    def _unit_index(self):<br>        &quot;&quot;&quot;Spelling -&gt; unit; and the base unit of each dimension, read not assumed.&quot;&quot;&quot;<br>        ix, base_of = {}, {}<br>        for u in self.g.subjects(RDF.type, EXT.UnitCode):<br>            factor = float(next(self.g.objects(u, EXT.toBase), 1.0))<br>            dim    = str(next(self.g.objects(u, EXT.dimension), &quot;&quot;))<br>            pref   = str(next(self.g.objects(u, SKOS.prefLabel), &quot;&quot;))<br>            code   = str(next(self.g.objects(u, EXT.code), &quot;&quot;))<br>            if factor == 1.0:                       # the base unit says so itself<br>                base_of[dim] = (u, code)<br>            for a in list(self.g.objects(u, SKOS.altLabel)) + [Literal(pref)]:<br>                ix[str(a).strip().lower()] = (u, factor, dim, code)<br>        self.base_unit = base_of<br>        return ix<br><br>    def _range_index(self):<br>        &quot;&quot;&quot;Property -&gt; expected shape. First range by IRI where several are declared.&quot;&quot;&quot;<br>        props = set(self.field.values()) | {<br>            next(self.g.objects(fm, EXT.mapsTo)) for fm in self.mapf.values()}<br>        ix = {}<br>        for p in props:<br>            declared = sorted(self.g.objects(p, RDFS.range), key=str)<br>            if declared:<br>                ix[p] = declared[0]<br>        return ix<br><br>    def _measure(self, raw, cls):<br>        &quot;&quot;&quot;&#39;28.9 mg&#39; -&gt; a node carrying what was declared AND what it normalises to.&quot;&quot;&quot;<br>        m = QTY.match(str(raw).strip())<br>        if not m:<br>            return None<br>        count = int(m.group(1)) if m.group(1) else 1<br>        num   = float(m.group(2).replace(&quot;,&quot;, &quot;.&quot;))<br>        tok   = (m.group(3) or &quot;&quot;).strip().lower()<br>        hit   = self.unit.get(tok)<br><br>        n, sub = BNode(), Graph()<br>        sub.add((n, RDF.type, cls))<br>        if hit is None:                        # a number in a unit nobody declared<br>            sub.add((n, GS1.value, Literal(num * count, datatype=XSD.float)))<br>            return n, sub<br><br>        u, factor, dim, code = hit<br><br>        # as declared: gs1:value and gs1:unitCode agree, so a GS1 reader is right<br>        sub.add((n, GS1.value, Literal(num * count, datatype=XSD.float)))<br>        sub.add((n, GS1.unitCode, Literal(code)))<br>        sub.add((n, EXT.unitCodeRef, u))<br>        # and the same quantity in the dimension&#39;s base unit, which is ours.<br>        # The base unit itself is not stored - it is the one UnitCode of this<br>        # dimension whose ext:toBase is 1.0, and the graph already says which.<br>        sub.add((n, EXT.baseValue, Literal(num * count * factor, datatype=XSD.float)))<br>        return n, sub<br><br>    def _taglist(self, raw, fm):<br>        &quot;&quot;&quot;&#39;en:gluten,en:nuts&#39; -&gt; one node per tag, shaped by the FieldMapping.&quot;&quot;&quot;<br>        cls   = next(self.g.objects(fm, EXT.tagClass), None)<br>        codep = next(self.g.objects(fm, EXT.tagCodeProperty), None)<br>        fixp  = next(self.g.objects(fm, EXT.fixedProperty), None)<br>        fixv  = next(self.g.objects(fm, EXT.fixedValue), None)<br><br>        out, sub = [], Graph()<br>        for tag in str(raw).split(&quot;,&quot;):<br>            tag = tag.strip()<br>            if not tag or tag == &quot;-&quot;:<br>                continue<br>            n = BNode()<br>            if cls is not None:<br>                sub.add((n, RDF.type, cls))<br>            sub.add((n, codep or RDFS.label, Literal(tag)))<br>            if fixp is not None and fixv is not None:<br>                sub.add((n, fixp, fixv))<br>            out.append(n)<br>        return out, sub<br><br>    def catalog(self, records, id_field=&quot;id&quot;, type_=GS1.FoodBeverageTobaccoProduct):<br>        g = Graph(); g.bind(&quot;gs1&quot;, GS1); g.bind(&quot;ext&quot;, EXT); g.bind(&quot;d&quot;, DATA)<br>        rep = {&quot;records&quot;: 0, &quot;mapped&quot;: set(), &quot;unmapped&quot;: set(), &quot;dropped&quot;: set()}<br><br>        for rec in records:<br>            s = DATA[str(rec[id_field]).strip()]<br>            g.add((s, RDF.type, type_))<br>            rep[&quot;records&quot;] += 1<br><br>            for key, val in rec.items():<br>                if key == id_field or val in (None, &quot;&quot;, &quot;-&quot;):<br>                    continue<br>                k  = key.strip().lower()<br>                fm = self.mapf.get(k)<br>                prop = (next(self.g.objects(fm, EXT.mapsTo)) if fm is not None<br>                        else self.field.get(k))<br>                if prop is None:<br>                    rep[&quot;unmapped&quot;].add(key); continue<br>                rep[&quot;mapped&quot;].add(key)<br><br>                if fm is not None and next(self.g.objects(fm, EXT.valueShape), None) == EXT.TagList:<br>                    nodes, sub = self._taglist(val, fm)<br>                    g += sub<br>                    for n in nodes:<br>                        g.add((s, prop, n))<br>                    continue<br><br>                rng = self.range.get(prop)<br>                if rng is not None and (rng, RDF.type, OWL.Class) in self.g:<br>                    made = self._measure(val, rng)          # range is a NODE<br>                    if made:<br>                        n, sub = made<br>                        g += sub<br>                        g.add((s, prop, n))<br>                    else:<br>                        rep[&quot;dropped&quot;].add(key)             # shape did not parse<br>                elif rng == RDF.langString:<br>                    # the tag is declared, not assumed; &quot;en&quot; only as a fallback<br>                    tag = str(next(self.g.objects(fm, EXT.languageTag), &quot;en&quot;)) \<br>                          if fm is not None else &quot;en&quot;<br>                    g.add((s, prop, Literal(str(val), lang=tag)))<br>                else:<br>                    g.add((s, prop, Literal(val)))<br>        return g, rep<br><br><br>print(&quot;cataloguer defined - this cell never changes again&quot;)</pre><pre>cataloguer defined - this cell never changes again</pre><h3>The feed</h3><p>Real columns, real values, renamed only to strip Open Food Facts’ path prefixes — the way a supplier would actually send them. Nothing invented.</p><pre>NUT = &quot;nutrition.input_sets.packaging.as_sold.100g.nutrients.&quot;<br>META = [&quot;quantity&quot;, &quot;serving_size&quot;, &quot;product_name_en&quot;, &quot;ingredients_text_en&quot;,<br>        &quot;brands&quot;, &quot;allergens&quot;, &quot;allergens_tags&quot;, &quot;traces_tags&quot;, &quot;countries_tags&quot;]<br><br><br>def build_feed(df, n=6, require=&quot;allergens_tags&quot;, langs=False):<br>    &quot;&quot;&quot;Rename OFF columns to bare field names. Values untouched.&quot;&quot;&quot;<br>    cols = {&quot;code&quot;: &quot;id&quot;}<br>    if langs:                       # offer every translated column, not just _en<br>        for c in df.columns:<br>            if c.startswith((&quot;product_name_&quot;, &quot;ingredients_text_&quot;)):<br>                cols[c] = c<br>    for c in df.columns:<br>        if c.startswith(NUT) and c.endswith(&quot;.value&quot;):<br>            cols[c] = c[len(NUT):-len(&quot;.value&quot;)]<br>    for c in META:<br>        if c in df.columns:<br>            cols[c] = c<br><br>    units = {c[len(NUT):-len(&quot;.unit&quot;)]: c<br>             for c in df.columns if c.startswith(NUT) and c.endswith(&quot;.unit&quot;)}<br><br>    sub = df[[c for c in cols if c in df.columns]].rename(columns=cols)<br>    have = df[[c for c in units.values() if c in df.columns]]<br>    if require and require in sub:<br>        sub = sub.loc[sub[require].notna()]<br><br>    recs = []<br>    for i, r in (sub if n is None else sub.head(n)).iterrows():<br>        rec = {k: v for k, v in r.items() if isinstance(v, str) and v.strip()}<br>        for nutrient, ucol in units.items():          # fold the unit onto the value<br>            if nutrient in rec and isinstance(df.at[i, ucol], str):<br>                rec[nutrient] = f&quot;{rec[nutrient]} {df.at[i, ucol]}&quot;<br>        recs.append(rec)<br>    return recs<br><br><br>FEED = build_feed(df)<br>print(f&quot;{len(FEED)} records, {len(FEED[0])} fields on the first one\n&quot;)<br>for k, v in list(FEED[0].items())[:12]:<br>    print(f&quot;  {k:24s} {str(v)[:52]}&quot;)</pre><pre>6 records, 11 fields on the first one<br><br>  id                       0884912002372<br>  energy-kcal              375 kcal<br>  added-sugars             21.4285714285714 g<br>  iron                     28.9285714285714 mg<br>  serving_size             1 cup (56 g)<br>  product_name_en          Cranberry Almond Crunch<br>  ingredients_text_en      Whole grain wheat, sugar, dried cranberries (cranber<br>  brands                   great-grains<br>  allergens                gluten, nuts<br>  allergens_tags           en:gluten,en:nuts<br>  countries_tags           en:united-states,en:world</pre><h3>What GS1 alone can catalogue</h3><p>Before adding anything: run the cataloguer against the published vocabulary. GS1 has properties for almost every field in this feed — but no altLabel telling it that sugars means sugarsPerNutrientBasis.</p><pre>g0, rep0 = Cataloguer(onto).catalog(FEED)<br>print(f&quot;{rep0[&#39;records&#39;]} records -&gt; {len(g0)} triples&quot;)<br>print(f&quot;  mapped   : {len(rep0[&#39;mapped&#39;])}&quot;)<br>print(f&quot;  UNMAPPED : {len(rep0[&#39;unmapped&#39;])} fields&quot;)</pre><pre>6 records -&gt; 6 triples<br>  mapped   : 0<br>  UNMAPPED : 24 fields</pre><p>Nothing maps. The vocabulary is right there, fully typed, and completely unreachable — because a standard names things in its own language and a feed names them in someone else’s.</p><p>That is the entire gap. Everything below closes it with triples.</p><h3>Nutrients: altLabels only</h3><p>Twenty-two of these properties already exist in GS1, with domains and ranges declared. We add <strong>no properties</strong>. We add the names this feed happens to use, plus one axiom: a common parent, so that “all nutrients” becomes a thing you can ask for.</p><p>Note energy-kcal and energy-kj: two source columns, one GS1 property. The unit carries the difference, which is only possible because units are modelled.</p><pre>GS1_NUTRIENTS = {<br>    &quot;energyPerNutrientBasis&quot;:        [&quot;energy-kcal&quot;, &quot;energy-kj&quot;, &quot;energy&quot;],<br>    &quot;fatPerNutrientBasis&quot;:           [&quot;fat&quot;],<br>    &quot;saturatedFatPerNutrientBasis&quot;:  [&quot;saturated-fat&quot;, &quot;saturated fat&quot;],<br>    &quot;transFatPerNutrientBasis&quot;:      [&quot;trans-fat&quot;, &quot;trans fat&quot;],<br>    &quot;carbohydratesPerNutrientBasis&quot;: [&quot;carbohydrates&quot;, &quot;carbohydrate&quot;],<br>    &quot;sugarsPerNutrientBasis&quot;:        [&quot;sugars&quot;],<br>    &quot;starchPerNutrientBasis&quot;:        [&quot;starch&quot;],<br>    &quot;fibrePerNutrientBasis&quot;:         [&quot;fiber&quot;, &quot;fibre&quot;],       # GS1 spells it fibre<br>    &quot;proteinPerNutrientBasis&quot;:       [&quot;proteins&quot;, &quot;protein&quot;],<br>    &quot;saltPerNutrientBasis&quot;:          [&quot;salt&quot;],<br>    &quot;sodiumPerNutrientBasis&quot;:        [&quot;sodium&quot;],<br>    &quot;cholesterolPerNutrientBasis&quot;:   [&quot;cholesterol&quot;],<br>    &quot;calciumPerNutrientBasis&quot;:       [&quot;calcium&quot;],<br>    &quot;ironPerNutrientBasis&quot;:          [&quot;iron&quot;],<br>    &quot;potassiumPerNutrientBasis&quot;:     [&quot;potassium&quot;],<br>    &quot;magnesiumPerNutrientBasis&quot;:     [&quot;magnesium&quot;],<br>    &quot;phosphorusPerNutrientBasis&quot;:    [&quot;phosphorus&quot;],<br>    &quot;zincPerNutrientBasis&quot;:          [&quot;zinc&quot;],<br>    &quot;vitaminAPerNutrientBasis&quot;:      [&quot;vitamin-a&quot;],<br>    &quot;vitaminCPerNutrientBasis&quot;:      [&quot;vitamin-c&quot;],<br>    &quot;vitaminDPerNutrientBasis&quot;:      [&quot;vitamin-d&quot;],<br>    &quot;vitaminEPerNutrientBasis&quot;:      [&quot;vitamin-e&quot;],<br>}<br><br><br>def teach_nutrients(g):<br>    for name, alts in GS1_NUTRIENTS.items():<br>        p = GS1[name]<br>        for a in alts:<br>            g.add((p, SKOS.altLabel, Literal(a)))            # names, not properties<br>        g.add((p, RDFS.subPropertyOf, EXT.nutrientPerBasis))  # the one axiom<br>    return g<br><br><br>before = len(onto)<br>onto = teach_nutrients(onto)<br>print(f&quot;+{len(onto) - before} triples: &quot;<br>      f&quot;{sum(len(v) for v in GS1_NUTRIENTS.values())} altLabels, &quot;<br>      f&quot;{len(GS1_NUTRIENTS)} subPropertyOf, 0 new properties&quot;)</pre><pre>+51 triples: 29 altLabels, 22 subPropertyOf, 0 new properties</pre><h3>Metadata: eighty columns, two properties</h3><p>productName and ingredientStatement are declared rdfs:range rdf:langString in GS1. The export carries product names in <strong>fifty</strong> languages and ingredients in <strong>thirty</strong> - eighty columns in a spreadsheet, two properties in RDF, distinguished by language tag.</p><p>The cataloguer already handles this. It has always handled it; nothing had told it these columns existed. One of the seven is a trap we walked into. gs1:brandName has rdfs:domain gs1:Brand, not gs1:Product, so by the same rule that classifies beverages further down, asserting it on the record types all 1,515 products as gs1:Brand as well - and with zero owl:disjointWith in the file, nothing objects. gs1:brandOwner, domain gs1:Product, is the property that belongs here.</p><pre>GS1_METADATA = {<br>    &quot;productName&quot;:         [&quot;product_name_en&quot;, &quot;product_name&quot;],<br>    &quot;ingredientStatement&quot;: [&quot;ingredients_text_en&quot;, &quot;ingredients&quot;],<br>    &quot;allergenStatement&quot;:   [&quot;allergens&quot;],<br>    &quot;brandName&quot;:           [&quot;brands&quot;, &quot;brand&quot;],<br>    &quot;netContent&quot;:          [&quot;quantity&quot;, &quot;net_content&quot;, &quot;pack size&quot;],<br>    &quot;servingSize&quot;:         [&quot;serving_size&quot;, &quot;serving&quot;],<br>    &quot;gpcCategoryCode&quot;:     [&quot;gpc&quot;, &quot;gpc_category_code&quot;],<br>}<br><br><br>def teach_metadata(g):<br>    for name, alts in GS1_METADATA.items():<br>        for a in alts:<br>            g.add((GS1[name], SKOS.altLabel, Literal(a)))<br>    return g<br><br><br>before = len(onto)<br>onto = teach_metadata(onto)<br>for name in GS1_METADATA:<br>    rng = next(onto.objects(GS1[name], RDFS.range), None)<br>    print(f&quot;  {name:22s} range = {str(rng).rsplit(&#39;/&#39;, 1)[-1].rsplit(&#39;#&#39;, 1)[-1]}&quot;)<br>print(f&quot;\n+{len(onto) - before} triples&quot;)</pre><pre>  productName            range = langString<br>  ingredientStatement    range = langString<br>  allergenStatement      range = langString<br>  brandName              range = langString<br>  netContent             range = QuantitativeValue<br>  servingSize            range = QuantitativeValue<br>  gpcCategoryCode        range = string<br><br>+14 triples</pre><h3>Allergens: one shape, several families</h3><p>allergens_tags and traces_tags look like two fields. In GS1 they are <strong>one relation and a code</strong>:</p><pre>gs1:hasAllergen  -&gt;  AllergenDetails<br>                       |- gs1:allergenType                   -&gt; the allergen<br>                       &#39;- gs1:allergenLevelOfContainmentCode -&gt; contains / may contain</pre><p>That is the regulation’s own model, and it is why traces is not a separate concept. To express it we need more than a name per column, so the mapping itself becomes a node in the ontology - ext:FieldMapping - carrying the value shape, the class to build, and any fixed code. One honest caveat about this path: gs1:allergenType is an owl:ObjectProperty whose rdfs:range is gs1:AllergenTypeCode , and what we store against it is the literal&quot;en:gluten&quot; — an Open Food Facts tag, not a GS1 code. The queries below work; the catalogue is not conformant on this relation until those tags are aligned to that code list, and nothing in the vocabulary will tell you so.</p><p>The same shape then serves countries, which are a comma-separated tag list too.</p><pre>def teach_taglists(g):<br>    &quot;&quot;&quot;A tag list is a value SHAPE. Declaring it once serves several columns.&quot;&quot;&quot;<br>    for src, prop, cls, codep, fixed in [<br>        (&quot;allergens_tags&quot;, GS1.hasAllergen, GS1.AllergenDetails,<br>             GS1.allergenType, (GS1.allergenLevelOfContainmentCode, GS1.CONTAINS)),<br>        (&quot;traces_tags&quot;,    GS1.hasAllergen, GS1.AllergenDetails,<br>             GS1.allergenType, (GS1.allergenLevelOfContainmentCode, GS1.MAY_CONTAIN)),<br>        (&quot;countries_tags&quot;, GS1.countryOfOrigin, GS1.Country, None, None),<br>    ]:<br>        fm = EXT[f&quot;fm_{src}&quot;]<br>        g.add((fm, RDF.type, EXT.FieldMapping))<br>        g.add((fm, EXT.sourceField, Literal(src)))<br>        g.add((fm, EXT.mapsTo, prop))<br>        g.add((fm, EXT.valueShape, EXT.TagList))<br>        g.add((fm, EXT.tagClass, cls))<br>        if codep is not None:<br>            g.add((fm, EXT.tagCodeProperty, codep))<br>        if fixed is not None:<br>            g.add((fm, EXT.fixedProperty, fixed[0]))<br>            g.add((fm, EXT.fixedValue, fixed[1]))<br>    return g<br><br><br>before = len(onto)<br>onto = teach_taglists(onto)<br>print(f&quot;+{len(onto) - before} triples - 3 field mappings, 1 value shape&quot;)</pre><pre>+21 triples - 3 field mappings, 1 value shape</pre><h3>Same cataloguer, everything maps</h3><p>The Python has not been touched since it was defined.</p><pre>g1, rep1 = Cataloguer(onto).catalog(FEED)<br>print(f&quot;{rep1[&#39;records&#39;]} records -&gt; {len(g1)} triples  (was {len(g0)})&quot;)<br>print(f&quot;  mapped   : {len(rep1[&#39;mapped&#39;])} fields&quot;)<br>print(f&quot;  UNMAPPED : {sorted(rep1[&#39;unmapped&#39;]) or &#39;none&#39;}&quot;)<br>print(f&quot;  dropped  : {sorted(rep1[&#39;dropped&#39;]) or &#39;none&#39;}   (matched a property, value shape did not parse)&quot;)</pre><pre>6 records -&gt; 290 triples  (was 6)<br>  mapped   : 22 fields<br>  UNMAPPED : [&#39;added-sugars&#39;, &#39;alcohol&#39;]<br>  dropped  : [&#39;serving_size&#39;]   (matched a property, value shape did not parse)</pre><h3>Where GS1 has nothing</h3><p>Two fields survive unmapped, and no altLabel will fix them: <strong>GS1 has no property for added sugars, and none for alcohol as a nutrient.</strong> It offers percentageOfAlcoholByVolume - a bare float, one of only twenty in 557 properties, inconsistent with every other nutrient in the vocabulary.</p><p>The same is true of water, lactose, glucose, fructose, the fibre fractions and choline. Caffeine exists only as isDecaffeinated, a code, not a quantity.</p><p>So declare them. In our namespace, never GS1’s — and hang each one under the GS1 property it specialises, so the standard’s own hierarchy absorbs them.</p><pre><br>EXT_NUTRIENTS = [<br>    # (name,                      aliases,               GS1 parent to compose with)<br>    (&quot;addedSugarsPerNutrientBasis&quot;, [&quot;added-sugars&quot;, &quot;added sugars&quot;],<br>                                    GS1.sugarsPerNutrientBasis),<br>    (&quot;waterPerNutrientBasis&quot;,       [&quot;water&quot;],                 EXT.nutrientPerBasis),<br>    (&quot;lactosePerNutrientBasis&quot;,     [&quot;lactose&quot;],   GS1.sugarsPerNutrientBasis),<br>    (&quot;glucosePerNutrientBasis&quot;,     [&quot;glucose&quot;],   GS1.sugarsPerNutrientBasis),<br>    (&quot;fructosePerNutrientBasis&quot;,    [&quot;fructose&quot;],  GS1.sugarsPerNutrientBasis),<br>    (&quot;alcoholPerNutrientBasis&quot;,     [&quot;alcohol&quot;],               EXT.nutrientPerBasis),<br>    (&quot;caffeinePerNutrientBasis&quot;,    [&quot;caffeine&quot;],              EXT.nutrientPerBasis),<br>]<br><br><br>def teach_gaps(g):<br>    for name, alts, parent in EXT_NUTRIENTS:<br>        p = EXT[name]<br>        g.add((p, RDF.type, OWL.ObjectProperty))<br>        g.add((p, RDFS.domain, GS1.FoodBeverageTobaccoProduct))<br>        g.add((p, RDFS.range, GS1.NutritionMeasurementType))<br>        g.add((p, RDFS.subPropertyOf, parent))       # added sugars ARE sugars<br>        for a in alts:<br>            g.add((p, SKOS.altLabel, Literal(a)))<br>    return g<br><br><br>before = len(onto)<br>onto = teach_gaps(onto)<br>g2, rep2 = Cataloguer(onto).catalog(FEED)<br><br>print(f&quot;+{len(onto) - before} triples: {len(EXT_NUTRIENTS)} properties GS1 lacks\n&quot;)<br>print(f&quot;{rep2[&#39;records&#39;]} records -&gt; {len(g2)} triples&quot;)<br>print(f&quot;  mapped   : {len(rep2[&#39;mapped&#39;])} fields&quot;)<br>print(f&quot;  UNMAPPED : {sorted(rep2[&#39;unmapped&#39;]) or &#39;none&#39;}&quot;)<br>print(f&quot;  dropped  : {sorted(rep2[&#39;dropped&#39;]) or &#39;none&#39;}&quot;)</pre><pre>+36 triples: 7 properties GS1 lacks<br><br>6 records -&gt; 308 triples<br>  mapped   : 24 fields<br>  UNMAPPED : none<br>  dropped  : [&#39;alcohol&#39;, &#39;serving_size&#39;]</pre><p>Nothing is unmapped now. Two values are <strong>dropped</strong>, and the difference matters: a dropped field found its property but its value did not fit the shape the ontology declared.</p><p>serving_size is 1 cup (56 g) and alcohol carries the unit % vol - a spelling with a space in it. Both fail the quantity pattern. Neither is silently discarded: the cataloguer reports them, because a pipeline that cannot say <em>I did not understand this</em> is the one that quietly corrupts your numbers.</p><h3>One record, drawn</h3><p>Blue is a property GS1 already had. Orange is one we declared because GS1 had nothing for it. Green is a unit.</p><p>Each measurement shows the figure <strong>as the label declared it</strong>, next to the gs1:unitCode declared with it. The unit node carries the conversion, so 28.93 mg and 0.0289 g are both readable off the picture and neither is pretending to be the other.</p><pre>s1 = list(g2.subjects(RDF.type, GS1.FoodBeverageTobaccoProduct))[0]<br>draw_record(g2, s1, onto,<br>            new_props=[EXT[n] for n, _, _ in EXT_NUTRIENTS],<br>            title=&quot;One catalogued record&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fV7m3w1uuMmkdJTIbiQlmA.png" /></figure><h3>The property hierarchy we created</h3><p>GS1 shipped 43 nutrient properties with no parent. One axiom each gives them a common ancestor; the orange nodes are the seven we had to declare ourselves. This is what makes the next cell possible.</p><pre>draw_property_tree(onto, EXT.nutrientPerBasis, new_ns=EXT, data=g2, limit=15,<br>                   title=&quot;Nutrients under a parent GS1 never declared&quot;)<br><br># root on the left, children flowing right - the same reading order as the<br># record figures above. BT would fan these 25 children into one unreadable row.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*yMXeudRWZtj5TiXSt2JxpQ.png" /></figure><h3>The axiom that pays for itself</h3><p>gs1:sugarsPerNutrientBasis is asserted only where the feed said sugars. Run RDFS closure and ask again - the parent property collects every child without a rule enumerating them.</p><pre>closed = Graph(); closed += g2; closed += onto<br><br>def count(g, p):<br>    return len(list(g.triples((None, p, None))))<br><br>before_n = count(closed, EXT.nutrientPerBasis)<br>owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(closed)<br>after_n  = count(closed, EXT.nutrientPerBasis)<br><br>print(f&quot;ext:nutrientPerBasis   before {before_n:4d}  -&gt;  after {after_n:4d}&quot;)<br>print(f&quot;gs1:hasAllergen        {count(closed, GS1.hasAllergen)} assertions&quot;)<br>print(&quot;\nnothing enumerated the nutrients - subPropertyOf did it&quot;)</pre><pre>ext:nutrientPerBasis   before    0  -&gt;  after   30<br>gs1:hasAllergen        12 assertions<br><br>nothing enumerated the nutrients - subPropertyOf did it</pre><h3>Now ask it something</h3><p>Six records prove the mechanism. The argument needs the whole catalogue.</p><pre>FEED_ALL = build_feed(df, n=None, require=None)<br>g_all, rep_all = Cataloguer(onto).catalog(FEED_ALL)<br>print(f&quot;{len(FEED_ALL):,} products -&gt; {len(g_all):,} triples&quot;)<br>print(f&quot;  unmapped: {sorted(rep_all[&#39;unmapped&#39;])[:6]} ... ({len(rep_all[&#39;unmapped&#39;])} fields)&quot;)</pre><pre>1,515 products -&gt; 61,160 triples<br>  unmapped: [&#39;beta-glucan&#39;, &#39;biotin&#39;, &#39;carbohydrates-total&#39;, &#39;choline&#39;, &#39;copper&#39;, &#39;fluoride&#39;] ... (24 fields)</pre><p>Everything still unmapped is either a genuine hole in GS1 or a name we have not taught it yet. Genuine holes: beta-glucan, choline, carbohydrates-total — no property, no altLabel, nothing to point at. Not holes: biotin, copper and fluoride, all three of which GS1 ships and which are unmapped only because we never gave them a name. Either way the field is reported, not guessed.</p><h3>Reason over a slice, not the whole vocabulary</h3><p>Closure materialises triples, so it is worth being precise about which axioms the question needs. Sub-property, sub-class, domain, range and the unit individuals come to 1,485 triples out of 18,989. The rest of GS1 is irrelevant to this question and can stay out of memory.</p><pre>schema = Graph()<br>for p in (RDFS.subPropertyOf, RDFS.subClassOf, RDFS.domain, RDFS.range):<br>    for t in onto.triples((None, p, None)):<br>        schema.add(t)<br>for u in onto.subjects(RDF.type, EXT.UnitCode):       # the query walks into units<br>    for t in onto.triples((u, None, None)):<br>        schema.add(t)<br><br>closed = Graph(); closed += g_all; closed += schema<br>print(f&quot;schema slice : {len(schema):,} of {len(onto):,} triples&quot;)<br>print(f&quot;before closure: {len(closed):,}&quot;)<br>owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(closed)<br>print(f&quot;after closure : {len(closed):,}&quot;)</pre><pre>schema slice : 1,485 of 18,989 triples<br>before closure: 62,645<br>after closure : 115,277</pre><h3>The peak</h3><p>ext:nutrientPerBasis is the parent property we declared. Count how many times the catalogued data actually <em>uses</em> it.</p><pre>asserted = len(list(g_all.triples((None, EXT.nutrientPerBasis, None))))<br>derived  = len(list(closed.triples((None, EXT.nutrientPerBasis, None))))<br>print(f&quot;asserted in the data : {asserted}&quot;)<br>print(f&quot;available to query   : {derived:,}&quot;)</pre><pre>asserted in the data : 0<br>available to query   : 6,133</pre><p><strong>Zero, and six thousand.</strong></p><p>No record carries that property. No column in the CSV corresponds to it. No line of Python mentions a nutrient by name. It exists only as 29 subPropertyOf triples - 25 that name it directly, and four more hanging off gs1:sugarsPerNutrientBasis, which is itself one of the 25 - and it now answers a question the dataset cannot answer about itself: <em>every nutrient measurement in this catalogue, whatever it is called.</em></p><p>That is the difference between a schema and an ontology. A schema describes the columns you have. An ontology lets you address concepts the columns never named.</p><h3>Traversal: three declared relations, no join written</h3><p>The query below walks measurement to unit to dimension. Nowhere does it mention grams, kilojoules, or any nutrient. It asks the ontology what kind of quantity each measurement is, and the ontology answers from the unit code list.</p><p>If SPARQL is new: it is pattern matching over triples. ?name is a variable, each line inside WHERE { } is a triple with variables in some of its three positions, and the engine returns every set of bindings that makes all the patterns true at once. <strong>A variable used twice is a join</strong> - ?m below is the same node on two consecutive lines, and that is what walks the graph from a product to a measurement to a unit. Everything else in these queries is SQL&#39;s vocabulary doing SQL&#39;s job: SELECT, COUNT, GROUP BY, ORDER BY, FILTER.</p><pre>Q_DIM = &quot;&quot;&quot;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT ?dimension (COUNT(?m) AS ?measurements)<br>WHERE {<br>  ?product ext:nutrientPerBasis ?m .     # any nutrient, at any depth<br>  ?m       ext:unitCodeRef      ?u .     # the unit it was recorded in<br>  ?u       ext:dimension        ?dimension .<br>}<br>GROUP BY ?dimension ORDER BY DESC(?measurements)&quot;&quot;&quot;<br><br>for r in closed.query(Q_DIM):<br>    print(f&quot;  {str(r.dimension):8s} {int(r.measurements):6,}&quot;)</pre><pre>  mass      5,312<br>  energy      821</pre><p>Mass and energy, separated — which is what stops you summing a kilojoule into a gram total. A DataFrame will perform that addition without complaint.</p><h3>The same query, a different answer</h3><p>Here is the sharpest version of the claim. Run one query, unchanged, against two graphs that differ by <strong>four triples</strong> — the subPropertyOf axioms saying that added sugars, lactose, glucose and fructose are sugars.</p><pre>Q_SUGAR = &quot;&quot;&quot;<br>PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT (COUNT(?m) AS ?measurements) (ROUND(AVG(?v)) AS ?mean_g)<br>WHERE { ?product gs1:sugarsPerNutrientBasis ?m . ?m ext:baseValue ?v }&quot;&quot;&quot;<br><br>FOLDED = [EXT.addedSugarsPerNutrientBasis, EXT.lactosePerNutrientBasis,<br>          EXT.glucosePerNutrientBasis,     EXT.fructosePerNutrientBasis]<br><br>for label, keep in ((&quot;without the 4 axioms&quot;, False), (&quot;with them&quot;, True)):<br>    s = Graph(); s += schema<br>    if not keep:<br>        for p in FOLDED:<br>            s.remove((p, RDFS.subPropertyOf, None))<br>    c = Graph(); c += g_all; c += s<br>    owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(c)<br>    for r in c.query(Q_SUGAR):<br>        print(f&quot;  {label:22s} measurements={int(r.measurements):4d}  mean={r.mean_g} g&quot;)</pre><pre>  without the 4 axioms   measurements= 578  mean=16 g<br>  with them              measurements= 682  mean=15 g</pre><p>The query text is identical. The knowledge moved out of the question and into the model — which is the whole point, because the next question inherits it for free.</p><h3>Crossing families</h3><p>Allergens and nutrients arrived from different columns, in different shapes, through different mechanisms — a tag list with a containment code, and a quantity with a unit. Once catalogued they are one graph, and joining them is a matter of naming both in the same pattern.</p><p>One new bit of syntax: [ gs1:allergenType &quot;en:gluten&quot; ] means <em>some node whose </em>allergenType<em> is </em>en:gluten. It is a blank node written inline, used here because the query has to pass through the allergen node to reach the product and never needs to refer to it again.</p><pre>Q_GLUTEN = &quot;&quot;&quot;<br>PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT (COUNT(DISTINCT ?p) AS ?products) (ROUND(AVG(?v)) AS ?mean_sugar)<br>WHERE {<br>  ?p gs1:hasAllergen [ gs1:allergenType &quot;en:gluten&quot; ] .<br>  ?p gs1:sugarsPerNutrientBasis ?m . ?m ext:baseValue ?v .<br>}&quot;&quot;&quot;<br>for r in closed.query(Q_GLUTEN):<br>    print(f&quot;  contains gluten : {int(r.products):4d} products, mean sugars {r.mean_sugar} g&quot;)<br><br>for r in closed.query(Q_GLUTEN.replace(<br>        &#39;?p gs1:hasAllergen [ gs1:allergenType &quot;en:gluten&quot; ] .&#39;,<br>        &#39;FILTER NOT EXISTS { ?p gs1:hasAllergen [ gs1:allergenType &quot;en:gluten&quot; ] }&#39;)):<br>    print(f&quot;  gluten-free     : {int(r.products):4d} products, mean sugars {r.mean_sugar} g&quot;)</pre><pre>contains gluten :  213 products, mean sugars 14 g<br>gluten-free     :  410 products, mean sugars 15 g</pre><p>No join key was declared, no foreign key exists, and the two families were never designed to be combined. They compose because both were mapped onto the same vocabulary — which is the 1991 interoperability argument arriving, thirty-five years later, in a blog post.</p><h3>Which food is it?</h3><p>Every record catalogued so far is typed FoodBeverageTobaccoProduct and nothing more specific. That is not GS1 being vague - it follows from which properties we chose to map.</p><pre>FOOD_CLASSES = [&quot;Beverage&quot;, &quot;FruitsVegetables&quot;, &quot;MeatPoultry&quot;, &quot;Seafood&quot;,<br>                &quot;MilkButterCreamYogurtCheeseEggsSubstitutes&quot;]<br><br>disc = {}<br>for prop, _, dom in onto.triples((None, RDFS.domain, None)):<br>    name = str(dom).rsplit(&quot;/&quot;, 1)[-1]<br>    if name in FOOD_CLASSES:<br>        disc.setdefault(name, []).append(str(prop).rsplit(&quot;/&quot;, 1)[-1])<br><br>print(&quot;GS1 properties that pin a record to ONE food class:&quot;)<br>for k in sorted(disc, key=lambda x: -len(disc[x])):<br>    print(f&quot;  {k[:26]:28s} {len(disc[k])}  e.g. {&#39;, &#39;.join(sorted(disc[k])[:3])}&quot;)<br>print(f&quot;\n  total discriminating properties: {sum(len(v) for v in disc.values())}&quot;)</pre><pre>GS1 properties that pin a record to ONE food class:<br>  Beverage                     9  e.g. alcoholicBeverageSubregion, beverageVintage, isCarbonated<br>  MilkButterCreamYogurtChees   7  e.g. cheeseFirmness, cheeseMaturationPeriodDescription, fatInMilkContent<br>  FruitsVegetables             5  e.g. isPittedStoned, isSeedless, isShelledPeeled<br>  MeatPoultry                  4  e.g. anatomicalForm, bonelessClaim, meatPoultryType<br>  Seafood                      4  e.g. catchZone, fishType, freshOrSeawaterFarmed<br><br>  total discriminating properties: 29</pre><p>Twenty-nine of them, and asserting any one settles the question — because rdfs:domain is a claim about what the subject <em>is</em>, not a hint.</p><pre>schema_only = Graph()<br>for p in (RDFS.subClassOf, RDFS.domain, RDFS.range):<br>    for t in onto.triples((None, p, None)):<br>        schema_only.add(t)<br><br>for prop in [GS1.percentageOfAlcoholByVolume, GS1.fatInMilkContent,<br>             GS1.isSeedless, GS1.fishType, GS1.meatPoultryType]:<br>    c = Graph(); c += schema_only<br>    probe = DATA[&quot;probe&quot;]<br>    c.add((probe, prop, Literal(&quot;x&quot;)))<br>    owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(c)<br>    got = sorted(str(o).rsplit(&quot;/&quot;, 1)[-1] for o in c.objects(probe, RDF.type)<br>                 if str(o).startswith(str(GS1)))<br>    print(f&quot;  {str(prop).rsplit(&#39;/&#39;, 1)[-1]:30s} -&gt; {got}&quot;)</pre><pre>  percentageOfAlcoholByVolume    -&gt; [&#39;Beverage&#39;, &#39;FoodBeverageTobaccoProduct&#39;, &#39;Product&#39;]<br>  fatInMilkContent               -&gt; [&#39;FoodBeverageTobaccoProduct&#39;, &#39;MilkButterCreamYogurtCheeseEggsSubstitutes&#39;, &#39;Product&#39;]<br>  isSeedless                     -&gt; [&#39;FoodBeverageTobaccoProduct&#39;, &#39;FruitsVegetables&#39;, &#39;Product&#39;]<br>  fishType                       -&gt; [&#39;FoodBeverageTobaccoProduct&#39;, &#39;Product&#39;, &#39;Seafood&#39;]<br>  meatPoultryType                -&gt; [&#39;FoodBeverageTobaccoProduct&#39;, &#39;MeatPoultry&#39;, &#39;Product&#39;]</pre><h3>The choice we made, and what it cost</h3><p>We mapped the feed’s alcohol column to ext:alcoholPerNutrientBasis - a property we invented, whose domain is the broad FoodBeverageTobaccoProduct. GS1 already had percentageOfAlcoholByVolume, whose domain is Beverage.</p><p>Same column, same values. One classifies the product; the other does not.</p><pre>onto_bev = Graph(); onto_bev += onto<br>onto_bev.remove((EXT.alcoholPerNutrientBasis, SKOS.altLabel, None))<br>onto_bev.add((GS1.percentageOfAlcoholByVolume, SKOS.altLabel, Literal(&quot;alcohol&quot;)))<br><br>g_bev, _ = Cataloguer(onto_bev).catalog(FEED_ALL)<br>c = Graph(); c += g_bev<br>for p in (RDFS.subClassOf, RDFS.domain, RDFS.range):<br>    for t in onto_bev.triples((None, p, None)):<br>        c.add(t)<br>owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(c)<br><br>for cls in (&quot;FoodBeverageTobaccoProduct&quot;, &quot;Beverage&quot;):<br>    n = len(set(c.subjects(RDF.type, GS1[cls])))<br>    print(f&quot;  inferred {cls:32s} {n:5d}&quot;)</pre><pre>  inferred FoodBeverageTobaccoProduct        1515<br>  inferred Beverage                            31</pre><p>Thirty-one products classify themselves as beverages — every one in this feed that states an alcohol content. No category column was read, no classifier trained, no threshold chosen. One altLabel pointed at a different property and the taxonomy did the rest.</p><p>(The export holds 250 products with an alcohol value across all three input sets; this feed reads only packaging.as_sold.100g, which is where the other 219 are missing from. Widen the feed and the same axiom classifies them too.)</p><p>The trade-off is real and worth naming: percentageOfAlcoholByVolume is a bare float - one of the twenty in 557 properties - so mapping there gains the classification and loses the unit node. Assert both properties and you keep both.</p><p><strong>One caution.</strong> Two properties now answer to the name alcohol, and the field index is a dictionary built by iterating a set: last writer wins, non-deterministically. That is a live bug, so check for it.</p><pre>from collections import Counter<br><br>def name_clashes(graph, label):<br>    claims = Counter()<br>    for p in set(graph.subjects(RDF.type, OWL.ObjectProperty)) | \<br>             set(graph.subjects(RDF.type, OWL.DatatypeProperty)):<br>        for lbl in graph.objects(p, SKOS.altLabel):<br>            claims[str(lbl).strip().lower()] += 1<br>    clash = {k: v for k, v in claims.items() if v &gt; 1}<br>    print(f&quot;{label:26s} field names claimed twice: {len(clash)}&quot;)<br>    for k, v in sorted(clash.items()):<br>        print(f&quot;    {k!r} claimed by {v} properties&quot;)<br><br>name_clashes(onto, &quot;as mapped above&quot;)<br>onto_bev.add((EXT.alcoholPerNutrientBasis, SKOS.altLabel, Literal(&quot;alcohol&quot;)))<br>name_clashes(onto_bev, &quot;if we assert both&quot;)</pre><pre>as mapped above            field names claimed twice: 0<br>if we assert both          field names claimed twice: 1<br>    &#39;alcohol&#39; claimed by 2 properties</pre><h3>Four more axioms, and a trap</h3><p>GS1 splits fat the same way it splits sugars: saturatedFat, transFat, monounsaturatedFat and polyunsaturatedFat exist, and none of them has a parent. Four triples fix the retrieval problem.</p><pre>FAT_PARTS = [&quot;saturatedFat&quot;, &quot;transFat&quot;, &quot;monounsaturatedFat&quot;, &quot;polyunsaturatedFat&quot;]<br><br>fat_schema = Graph(); fat_schema += schema<br>for part in FAT_PARTS:<br>    fat_schema.add((GS1[part + &quot;PerNutrientBasis&quot;], RDFS.subPropertyOf,<br>                    GS1.fatPerNutrientBasis))<br><br>Q_FAT = &quot;&quot;&quot;PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>SELECT (COUNT(*) AS ?measurements) WHERE { ?p gs1:fatPerNutrientBasis ?m }&quot;&quot;&quot;<br><br>for label, sch in ((&quot;as GS1 ships it&quot;, schema), (&quot;with 4 axioms&quot;, fat_schema)):<br>    c = Graph(); c += g_all; c += sch<br>    owlrl.DeductiveClosure(owlrl.RDFS_Semantics).expand(c)<br>    for r in c.query(Q_FAT):<br>        print(f&quot;  gs1:fatPerNutrientBasis  {label:16s} {int(r.measurements):5d}&quot;)</pre><pre>  gs1:fatPerNutrientBasis  as GS1 ships it    581<br>  gs1:fatPerNutrientBasis  with 4 axioms     1202</pre><p>581 to 1,202 from four triples. Now the trap, and it applies to the sugars result earlier too.</p><pre>Q_BOTH = &quot;&quot;&quot;PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT (COUNT(DISTINCT ?p) AS ?n) WHERE {<br>  ?p gs1:sugarsPerNutrientBasis ?a ; ext:addedSugarsPerNutrientBasis ?b }&quot;&quot;&quot;<br>for r in g_all.query(Q_BOTH):<br>    print(f&quot;products stating BOTH total sugars and added sugars: {int(r.n)}&quot;)</pre><pre>products stating BOTH total sugars and added sugars: 56</pre><p>Fifty-six products declare total sugars <em>and</em> added sugars. After closure both measurements sit under gs1:sugarsPerNutrientBasis, because that is exactly what we asserted - added sugars are a kind of sugars.</p><p>rdfs:subPropertyOf<strong> means &quot;is a kind of measurement of&quot;. It does not mean &quot;adds up to&quot;.</strong> Saturated fat is a component of total fat, not an addition to it. Sum the rolled-up property and you double-count every product that declared both.</p><p>So the roll-up is right for <em>retrieval</em> — find every sugar-related measurement, whatever the source called it — and wrong for <em>totalling</em>. The earlier query went from 578 measurements to 682 and the mean moved from 16 g to 15 g; that is more measurements retrieved, not a better estimate of total sugar.</p><p>This is the judgement boundary, in the middle of our own result. The axiom is correct and the aggregation built on it can still be wrong.</p><h3>A physical check the ontology makes expressible</h3><p>Energy is not independent of the macronutrients: roughly 4 kcal per gram of carbohydrate and protein, 9 per gram of fat. Predict it, compare with what the label declares, and any large gap means one of the two is wrong - or that the label follows a convention this arithmetic does not model, since US labels count fibre inside total carbohydrate and EU labels do not.</p><p>Two things make the arithmetic legal. ext:baseValue puts every mass in grams whatever the label used, and gs1:unitCode &quot;E14&quot; selects the products that declared their energy in kcal rather than kJ. The query runs against the <strong>asserted</strong> graph, deliberately - after the fat roll-up above, the closed graph would count saturated fat as fat again.</p><pre>ATWATER = &quot;&quot;&quot;<br>PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT %s WHERE {<br>  ?p gs1:energyPerNutrientBasis        [ ext:baseValue ?kcal ; gs1:unitCode &quot;E14&quot; ] ;<br>     gs1:carbohydratesPerNutrientBasis [ ext:baseValue ?carb ] ;<br>     gs1:proteinPerNutrientBasis       [ ext:baseValue ?prot ] ;<br>     gs1:fatPerNutrientBasis           [ ext:baseValue ?fat ] .<br>  BIND (4*?carb + 4*?prot + 9*?fat AS ?predicted)<br>  BIND (ABS(?kcal - ?predicted) AS ?gap)<br>  %s<br>}&quot;&quot;&quot;<br><br>summary = ATWATER % (&quot;(COUNT(*) AS ?checked) (ROUND(AVG(?gap)) AS ?mean_gap)&quot;, &quot;&quot;)<br>worst   = ATWATER % (&quot;?p ?kcal ?predicted&quot;, &quot;FILTER (?gap &gt; 300)&quot;)<br><br>for r in g_all.query(summary):<br>    print(f&quot;products checkable: {int(r.checked)}   mean absolute gap: {r.mean_gap} kcal&quot;)<br>print()<br><br>for r in g_all.query(worst):<br>    print(f&quot;  {str(r.p).rsplit(&#39;/&#39;, 1)[-1]:16s} &quot;<br>          f&quot;declared {float(r.kcal):9.1f}   predicted {float(r.predicted):9.1f}&quot;)</pre><pre>products checkable: 564   mean absolute gap: 80 kcal<br><br>  7613287476678    declared    1713.3   predicted     382.3<br>  5998400312029    declared      77.0   predicted     380.1<br>  0056022204247    declared     351.0   predicted    3895.0<br>  5010525093031    declared     378.0   predicted      60.5<br>  3111900664470    declared       0.0   predicted     381.0<br>  00002312         declared    2121.0   predicted   30989.0</pre><p>Products whose declared energy is impossible given their own macronutrients — one claims 2,121 kcal per 100 g against a predicted 30,989, another declares zero against a predicted 381.</p><p>No column in the export says “this row is inconsistent”. The check exists because energy, carbohydrate, protein and fat were mapped onto properties whose units are known, and 4/4/9 is arithmetic over comparable quantities. In the CSV those four values are strings in four columns with units in four more.</p><h3>Validation: what SHACL adds, and one way to get it wrong</h3><p>OWL infers. SHACL checks.</p><p>Its vocabulary fits in a paragraph. A <strong>shape</strong> is a bundle of constraints. Its <strong>target</strong> says which nodes they apply to — sh:targetClass gs1:NutritionMeasurementType means every node of that class, and those nodes are the shape&#39;s <strong>focus nodes</strong>. Each constraint names an sh:path, the property to look down, and then something to require of whatever is found there: sh:minCount, sh:maxCount, sh:datatype, sh:class, sh:in. A <em>sequence path</em> written ( gs1:energyPerNutrientBasis ext:unitCodeRef ) means follow the first property, then the second, and check whatever you land on.</p><p>shapes.ttl is all SHACL Core, deliberately: a nutrition measurement needs exactly one xsd:float value, a gs1:unitCode string and a unit drawn from the code list - and a product&#39;s energy must be recorded in an energy unit, which is that sequence path with sh:in.</p><p>Run it twice: once on the catalogued data alone, once with the unit individuals included.</p><pre>import re, collections<br><br>shapes = Graph(); shapes.parse(&quot;shapes.ttl&quot;, format=&quot;turtle&quot;)<br><br>def check(graph, label):<br>    ok, _, txt = validate(graph, shacl_graph=shapes, advanced=True, inference=&quot;none&quot;)<br>    msgs = re.findall(r&quot;Message: (.+)&quot;, txt)<br>    print(f&quot;{label:34s} conforms={ok}  violations={len(msgs):,}&quot;)<br>    for m, n in collections.Counter(msgs).most_common(3):<br>        print(f&quot;     {n:6,}  {m[:66]}&quot;)<br>    return collections.Counter(msgs)<br><br>m0 = check(g_all, &quot;data alone&quot;)<br>print()<br><br>with_units = Graph(); with_units += g_all<br>for u in onto.subjects(RDF.type, EXT.UnitCode):<br>    for t in onto.triples((u, None, None)):<br>        with_units.add(t)<br>m1 = check(with_units, &quot;data + the unit code list&quot;)</pre><pre>data alone                         conforms=False  violations=6,133<br>      6,133  value present but no unit from the code list<br>data + the unit code list          conforms=True  violations=0</pre><p>Six thousand violations, then none — and <strong>the data never changed.</strong></p><p>Before reading either number, ask what the shapes could even have looked at. A shape whose target class has no instances validates trivially; a constraint whose path does not occur in the graph does not fail, it disappears.</p><p>(The helper below walks rdf:first and rdf:rest because a sequence path is stored as an <strong>RDF list</strong> - a linked list, one cell at a time, ending at rdf:nil. That is how RDF represents anything ordered, and it is the one piece of the syntax that looks stranger than it is.)</p><pre>SH = Namespace(&quot;http://www.w3.org/ns/shacl#&quot;)<br><br>def path_iris(node):<br>    &quot;&quot;&quot;sh:path is an IRI, or an RDF list when it is a sequence path.&quot;&quot;&quot;<br>    if isinstance(node, URIRef):<br>        return [node]<br>    out, cur = [], node<br>    while cur is not None and cur != RDF.nil:<br>        head = next(shapes.objects(cur, RDF.first), None)<br>        if isinstance(head, URIRef):<br>            out.append(head)<br>        cur = next(shapes.objects(cur, RDF.rest), None)<br>    return out<br><br>local = lambda t: str(t).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br><br>print(f&quot;{&#39;target class&#39;:34s} {&#39;alone&#39;:&gt;8s} {&#39;+units&#39;:&gt;8s}   focus nodes&quot;)<br>for cls in sorted(set(shapes.objects(None, SH.targetClass)), key=str):<br>    a = len(set(g_all.subjects(RDF.type, cls)))<br>    b = len(set(with_units.subjects(RDF.type, cls)))<br>    print(f&quot;  {local(cls):32s} {a:8,} {b:8,}{&#39;   &lt;== nothing to validate&#39; if not b else &#39;&#39;}&quot;)<br><br>seen = {p for node in shapes.objects(None, SH.path) for p in path_iris(node)}<br>print(f&quot;\n{&#39;predicate a constraint depends on&#39;:34s} {&#39;alone&#39;:&gt;8s} {&#39;+units&#39;:&gt;8s}&quot;)<br>for p in sorted(seen, key=local):<br>    a = len(list(g_all.triples((None, p, None))))<br>    b = len(list(with_units.triples((None, p, None))))<br>    print(f&quot;  {local(p):32s} {a:8,} {b:8,}&quot;)<br><br>for cls in sorted({o for o in shapes.objects(None, SH[&quot;class&quot;])}, key=str):<br>    a = len(set(g_all.subjects(RDF.type, cls)))<br>    b = len(set(with_units.subjects(RDF.type, cls)))<br>    print(f&quot;\n  sh:class {local(cls):23s} {a:8,} {b:8,}&quot;<br>          f&quot;{&#39;   &lt;== the 6,133 explained&#39; if not a else &#39;&#39;}&quot;)</pre><pre>target class                          alone   +units   focus nodes<br>  FoodBeverageTobaccoProduct          1,515    1,515<br>  NutritionMeasurementType            6,133    6,133<br><br>predicate a constraint depends on     alone   +units<br>  energyPerNutrientBasis                821      821<br>  unitCode                            7,353    7,353<br>  unitCodeRef                         7,353    7,353<br>  value                               7,423    7,423<br><br>  sh:class UnitCode                       0       10   &lt;== the 6,133 explained</pre><p>ext:UnitCode occurs <strong>zero</strong> times in the first graph and ten in the second. That single line is the whole of the difference: the constraint says the unit must be an ext:UnitCode, nothing in the validated graph said ext:GRM was one, so all 6,133 measurements failed. The report was confident, detailed, and entirely an artefact of what had been loaded.</p><p>The same silence cuts the other way, and it cut us. An earlier version of shapes.ttl had six constraints and three of them could never match anything: two SPARQL constraints selected on gs1:unitCode, which the cataloguer was not emitting at the time, and one traversed ext:nutrientPerBasis, which exists only after closure and was being run against the un-closed graph. All three reported success, and they reported it <em>fast</em> - which was the only visible symptom, and looks like good news.</p><h3>The two checks Core cannot express</h3><p>Kilojoules and kilocalories are the same quantity twice, so a declared pair must sit near the 4.184 ratio. And nothing can weigh more than 100 g per 100 g. Both are arithmetic across nodes, and SHACL will express both — with sh:sparql.</p><p>It should not, here. A SPARQL constraint is evaluated <strong>once per focus node</strong>: 1,515 products, 1,515 query executions, minutes of wall clock. The same two checks as plain SPARQL over the whole graph take about three seconds. The dead versions in the old shapes.ttl were fast for exactly the reason they were useless.</p><p>Note which figure each one reads. The ratio check compares gs1:value - what the label <em>declared</em> - because comparing normalised values would make the ratio 1 by construction and test nothing. The weight check reads ext:baseValue, because 100 g is a claim about grams whatever unit the source used.</p><pre>Q_RATIO = &quot;&quot;&quot;<br>PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>SELECT ?p ?kj ?kc WHERE {<br>  ?p gs1:energyPerNutrientBasis ?a, ?b .<br>  ?a gs1:value ?kj ; gs1:unitCode &quot;KJO&quot; .        # as declared, not normalised<br>  ?b gs1:value ?kc ; gs1:unitCode &quot;E14&quot; .<br>  FILTER (?kc &gt; 0 &amp;&amp; (?kj / ?kc &lt; 4.0 || ?kj / ?kc &gt; 4.4))<br>} ORDER BY ?p&quot;&quot;&quot;<br><br>Q_OVER = &quot;&quot;&quot;<br>PREFIX gs1: &lt;https://ref.gs1.org/voc/&gt;<br>PREFIX ext: &lt;http://example.org/gs1x/&gt;<br>SELECT ?p ?prop ?v WHERE {<br>  ?p ?prop ?m .<br>  ?m a gs1:NutritionMeasurementType ;<br>     ext:baseValue   ?v ;                        # grams, whatever the label said<br>     ext:unitCodeRef ?u .<br>  ?u ext:dimension &quot;mass&quot; .<br>  FILTER (?v &gt; 100)<br>} ORDER BY DESC(?v)&quot;&quot;&quot;<br><br>ratio = list(with_units.query(Q_RATIO))<br>over  = list(with_units.query(Q_OVER))<br><br>print(f&quot;declared kJ/kcal outside 4.0-4.4 : {len(ratio):3d} products&quot;)<br>for r in ratio[:4]:<br>    print(f&quot;     {local(r.p):16s} kJ={float(r.kj):8.1f}  kcal={float(r.kc):8.1f}&quot;<br>          f&quot;  ratio={float(r.kj) / float(r.kc):6.2f}&quot;)<br><br>print(f&quot;\nmass nutrient over 100 g/100 g   : {len(over):3d} measurements&quot;)<br>for r in over[:4]:<br>    print(f&quot;     {local(r.p):16s} {local(r.prop):32s} {float(r.v):8.1f} g&quot;)<br><br>both = {str(r.p) for r in ratio} &amp; {str(r.p) for r in over}<br>print(f&quot;\nfailing both checks              : {sorted(local(x) for x in both)}&quot;)</pre><pre>declared kJ/kcal outside 4.0-4.4 :  14 products<br>     00002312         kJ=   122.0  kcal=  2121.0  ratio=  0.06<br>     3858891292960    kJ=  1611.0  kcal=   364.0  ratio=  4.43<br>     4000540083248    kJ=   754.0  kcal=   450.0  ratio=  1.68<br>     4260686523132    kJ=  1573.0  kcal=   314.0  ratio=  5.01<br><br>mass nutrient over 100 g/100 g   :  10 measurements<br>     00002312         fatPerNutrientBasis                2321.0 g<br>     00002312         carbohydratesPerNutrientBasis      2312.0 g<br>     5900020000637    calciumPerNutrientBasis             466.0 g<br>     6033000086761    saltPerNutrientBasis                450.0 g<br><br>failing both checks              : [&#39;00002312&#39;]</pre><p>Fourteen products declare a kJ figure that cannot be their kcal figure, and ten measurements claim more than 100 g of something per 100 g. 00002312 fails both - and it is the record the Atwater check already flagged, declaring 2,121 kcal against a predicted 30,989. Three independent checks, one bad row, and no column in the export that says so.</p><p>That is the third time in this series the same failure has appeared: Post 1’s class list came back quietly short, the dimension query returned no rows, and now a validator reports 6,133 problems that do not exist while an earlier version stayed silent about the ones it could not see. Every one is the same shape — <strong>the graph did not contain what the question referenced, and nothing raised an error.</strong></p><p>Reasoning does not remove that failure mode. It moves it: instead of a wrong number you get an empty result or an impossible one, which is easier to notice and much easier to explain. That is the honest summary of what an ontology buys you here.</p><h3>What it cost</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O2cgzEvLMBKB0yniGsYthg.png" /></figure><p>No new Python since the cataloguer was defined. No edit to gs1Voc.ttl. A table of counts is exactly the kind of thing that drifts away from the code under it, so count them instead of trusting them. The count has a trap of its own, and it caught this cell. t not in gs1_published compares against a second parse of the same file, and rdflib mints fresh identifiers for blank nodes on every parse; GS1’s eight owl:unionOf classes are RDF lists built from blank nodes, 67 triples of them, so every one came back as ours and the total read 307. Nothing we add uses a blank node, which is why excluding them is exact rather than approximate.</p><pre>gs1_published = Graph(); gs1_published.parse(&quot;gs1Voc.ttl&quot;, format=&quot;turtle&quot;)<br>ours = Graph()<br>for t in onto:<br>    if t not in gs1_published and not any(isinstance(x, BNode) for x in t):<br>        ours.add(t)<br><br>rows = [<br>    (&quot;skos:altLabel on GS1 properties&quot;,<br>     sum(1 for s, _, _ in ours.triples((None, SKOS.altLabel, None))<br>         if str(s).startswith(str(GS1)))),<br>    (&quot;rdfs:subPropertyOf&quot;,  len(list(ours.triples((None, RDFS.subPropertyOf, None))))),<br>    (&quot;ext: properties declared&quot;,<br>     len({s for s in set(ours.subjects(RDF.type, OWL.ObjectProperty)) |<br>          set(ours.subjects(RDF.type, OWL.DatatypeProperty))<br>          if str(s).startswith(str(EXT))})),<br>    (&quot;ext:UnitCode individuals&quot;, len(set(ours.subjects(RDF.type, EXT.UnitCode)))),<br>    (&quot;ext:FieldMapping nodes&quot;,   len(set(ours.subjects(RDF.type, EXT.FieldMapping)))),<br>]<br>for label, n in rows:<br>    print(f&quot;  {label:34s} {n:5d}&quot;)<br>print(f&quot;\n  every triple we added to GS1        {len(ours):5,}&quot;)<br>print(f&quot;  GS1 as published, untouched        {len(gs1_published):5,}&quot;)</pre><pre>  skos:altLabel on GS1 properties       43<br>  rdfs:subPropertyOf                    29<br>  ext: properties declared              21<br>  ext:UnitCode individuals              10<br>  ext:FieldMapping nodes                 3<br><br>  every triple we added to GS1          240<br>  GS1 as published, untouched        18,749</pre><h3>A file it has never seen</h3><p>Everything so far was built against one export. The claim was that a vocabulary plus axioms would catalogue a dataset nobody had modelled for. Time to test it rather than assert it.</p><p>openfoodfacts_export_biscut.csv is a separate download: 53 biscuit products, 359 columns. No cell above has read it, and no axiom was written with it in mind.</p><pre>bis = pd.read_csv(&quot;openfoodfacts_export_biscut.csv&quot;, sep=&quot;\t&quot;, dtype=str,<br>                  keep_default_na=False, na_values=[&quot;&quot;])<br><br>A, B = set(df.columns), set(bis.columns)<br>print(f&quot;cereal export : {len(A):4d} columns&quot;)<br>print(f&quot;biscuit export: {len(B):4d} columns&quot;)<br>print(f&quot;  shared      : {len(A &amp; B):4d}&quot;)<br>print(f&quot;  new here    : {len(B - A):4d}  {sorted(B - A)}&quot;)</pre><pre>cereal export :  851 columns<br>biscuit export:  359 columns<br>  shared      :  358<br>  new here    :    1  [&#39;ingredients_text_fa&#39;]</pre><p>One column in this file has never been seen before. Now run the cataloguer — the same object, holding the same ontology, with nothing added for biscuits.</p><pre>bis_feed = build_feed(bis, n=None, require=None)<br>g_bis, rep_bis = Cataloguer(onto).catalog(bis_feed)<br><br>print(f&quot;{len(bis_feed)} biscuits -&gt; {len(g_bis):,} triples&quot;)<br>print(f&quot;  mapped   : {len(rep_bis[&#39;mapped&#39;])} / {len(rep_bis[&#39;mapped&#39;]) + len(rep_bis[&#39;unmapped&#39;])} fields&quot;)<br>print(f&quot;  UNMAPPED : {sorted(rep_bis[&#39;unmapped&#39;]) or &#39;none&#39;}&quot;)<br>print(f&quot;  dropped  : {sorted(rep_bis[&#39;dropped&#39;]) or &#39;none&#39;}&quot;)<br>print()<br>print(wrapped(sorted(rep_bis[&quot;mapped&quot;])))</pre><pre>53 biscuits -&gt; 1,256 triples<br>  mapped   : 24 / 24 fields<br>  UNMAPPED : none<br>  dropped  : [&#39;quantity&#39;, &#39;serving_size&#39;]<br><br>  allergens, allergens_tags, brands, calcium, carbohydrates,<br>  cholesterol, countries_tags, energy-kcal, energy-kj, fat, fiber,<br>  ingredients_text_en, iron, potassium, product_name_en, proteins,<br>  quantity, salt, saturated-fat, serving_size, sodium, sugars,<br>  traces_tags, trans-fat</pre><p><strong>Twenty-four of twenty-four. Nothing unmapped.</strong></p><p>Energy, fat, saturated fat, trans fat, carbohydrates, sugars, fibre, protein, salt, sodium, cholesterol, calcium, iron, potassium, allergens, traces, countries, brand, product name, ingredients — all of it, on a file the ontology had never opened, with no line of code and no new triple.</p><p>That is the entire argument of this post, in one cell. The axioms were written about <em>concepts</em>, and biscuits are made of the same concepts as cereal.</p><pre>s_bis = list(g_bis.subjects(RDF.type, GS1.FoodBeverageTobaccoProduct))[0]<br>draw_record(g_bis, s_bis, onto,<br>            new_props=[EXT[n] for n, _, _ in EXT_NUTRIENTS],<br>            title=&quot;A biscuit, catalogued by an ontology built for cereal&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaYbWfv4Uz9SY0IcKmdkcA.png" /></figure><h3>Where it actually stops</h3><p>Two fields were dropped, and neither is a naming problem. Both found their property and failed on the <em>shape of the value</em>.</p><pre>for col in (&quot;quantity&quot;, &quot;serving_size&quot;):<br>    vals = bis[col].dropna()<br>    bad = [v for v in vals if not QTY.match(str(v).strip())]<br>    print(f&quot;{col:14s} {len(bad):2d} of {len(vals):2d} unparseable&quot;)<br>    for v in bad[:5]:<br>        print(f&quot;                 {v!r}&quot;)</pre><pre>quantity        2 of 16 unparseable<br>                 &#39;400g*4&#39;<br>                 &#39;50g g&#39;<br>serving_size    9 of 17 unparseable<br>                 &#39;4 pieces (17 g)&#39;<br>                 &#39;4 Pieces (17 g)&#39;<br>                 &#39;2 portion (16 g)&#39;<br>                 &#39;1 biscuit (51 g)&#39;<br>                 &#39;5 cookies (55 g)&#39;</pre><p>1 biscuit (51 g), 5 cookies (55 g), 4 pieces (17 g), 400g*4.</p><p>The cereal export said 1 cup (56 g). Biscuits say the same thing with a different container noun, and one supplier writes a multipack backwards. Field <em>names</em> transferred perfectly across the two datasets; value <em>spellings</em> did not, and no amount of vocabulary fixes that - a quantity parser is a parser.</p><p>The honest reading: the ontology carried the meaning across, and the string handling stayed exactly as fragile as it was. Neither value was silently discarded, which is the only reason we can see the problem at all.</p><h3>The other stop, and a bug it exposes</h3><p>This file carries twelve populated language columns. Only two are offered to the cataloguer, because only product_name_en and ingredients_text_en have altLabels.</p><pre>langs = [c for c in bis.columns<br>         if c.startswith((&quot;product_name_&quot;, &quot;ingredients_text_&quot;)) and bis[c].notna().sum()]<br>print(f&quot;populated language columns: {len(langs)}&quot;)<br>print(wrapped(langs))<br><br>known = {str(l) for p in (GS1.productName, GS1.ingredientStatement)<br>         for l in onto.objects(p, SKOS.altLabel)}<br>print(f&quot;\n  reachable today: {sorted(set(langs) &amp; known)}&quot;)<br>print(&quot;  unreachable    :&quot;)<br>print(wrapped(sorted(set(langs) - known), indent=&quot;    &quot;))</pre><pre>populated language columns: 12<br>  product_name_ar, product_name_en, product_name_es, product_name_fr,<br>  product_name_it, product_name_pl, product_name_ro, product_name_tr,<br>  ingredients_text_en, ingredients_text_fa, ingredients_text_fr,<br>  ingredients_text_ro<br><br>  reachable today: [&#39;ingredients_text_en&#39;, &#39;product_name_en&#39;]<br>  unreachable    :<br>    ingredients_text_fa, ingredients_text_fr, ingredients_text_ro,<br>    product_name_ar, product_name_es, product_name_fr, product_name_it,<br>    product_name_pl, product_name_ro, product_name_tr</pre><p>Adding the other ten looks like ten triples — exactly the kind of thing this post says should be free. It is not, and the reason is worth seeing.</p><pre>probe = Graph(); probe += onto<br>probe.add((GS1.ingredientStatement, SKOS.altLabel, Literal(&quot;ingredients_text_fr&quot;)))<br><br>g_fr, _ = Cataloguer(probe).catalog(<br>    [{&quot;id&quot;: &quot;probe&quot;, &quot;ingredients_text_fr&quot;: &quot;Farine de ble, sucre, beurre&quot;}])<br><br>for _, p, o in g_fr:<br>    if p == GS1.ingredientStatement:<br>        print(f&quot;stored   : {o!r}&quot;)<br>        print(f&quot;lang tag : {o.language}   &lt;-- French text, tagged English&quot;)</pre><pre>stored   : rdflib.term.Literal(&#39;Farine de ble, sucre, beurre&#39;, lang=&#39;en&#39;)<br>lang tag : en   &lt;-- French text, tagged English</pre><p>The altLabel worked. The language tag did not, because Cataloguer._catalog hardcodes lang=&quot;en&quot; on every rdf:langString. Ten new columns would enter the graph as ten confident lies about what language they are in.</p><p>So this one is <strong>not</strong> free, and pretending otherwise would be the exact failure the rest of this post is about. The fix belongs in the mapping vocabulary — an ext:languageTag on a FieldMapping, read at catalogue time the way ext:TagList already is - which is one more thing the ontology says and one more thing the code stops assuming.</p><p>Which is the honest summary of the whole exercise. A new dataset in the same universe costs nothing: 24 fields out of 24, first try. What it costs instead is every assumption you baked into the code before you had a second dataset to prove it wrong.</p><h3>Making the other ten free</h3><p>The bug above was a hardcoded lang=&quot;en&quot;. The fix is not to hardcode more languages - it is to stop the code deciding at all. Cataloguer now reads ext:languageTag from the field mapping and falls back to en only when nothing says otherwise, so every run above behaved identically.</p><p>Now declare the mappings. One FieldMapping per translated column, generated from the languages the file actually contains.</p><pre>def teach_languages(g, frame):<br>    &quot;&quot;&quot;Every product_name_* / ingredients_text_* column, tagged by its suffix.&quot;&quot;&quot;<br>    n = 0<br>    for prop, prefix in ((GS1.productName, &quot;product_name_&quot;),<br>                         (GS1.ingredientStatement, &quot;ingredients_text_&quot;)):<br>        for col in frame.columns:<br>            if not col.startswith(prefix):<br>                continue<br>            tag = col[len(prefix):]<br>            if not tag.isalpha() or len(tag) &gt; 3:      # skip _tags, _all and friends<br>                continue<br>            fm = EXT[f&quot;fm_{col}&quot;]<br>            g.add((fm, RDF.type, EXT.FieldMapping))<br>            g.add((fm, EXT.sourceField, Literal(col)))<br>            g.add((fm, EXT.mapsTo, prop))<br>            g.add((fm, EXT.languageTag, Literal(tag)))<br>            n += 1<br>    return g, n<br><br><br>onto, n_lang = teach_languages(onto, bis)<br>print(f&quot;{n_lang} language mappings declared - still zero lines of parsing code&quot;)</pre><pre>12 language mappings declared - still zero lines of parsing code</pre><p>Re-catalogue the biscuits, this time offering every translated column.</p><pre>bis_feed_all = build_feed(bis, n=None, require=None, langs=True)<br>g_bis2, rep_bis2 = Cataloguer(onto).catalog(bis_feed_all)<br><br>print(f&quot;fields offered : {len(rep_bis2[&#39;mapped&#39;]) + len(rep_bis2[&#39;unmapped&#39;])}&quot;)<br>print(f&quot;  mapped       : {len(rep_bis2[&#39;mapped&#39;])}&quot;)<br>print(f&quot;  UNMAPPED     : {sorted(rep_bis2[&#39;unmapped&#39;]) or &#39;none&#39;}&quot;)<br><br># sorted, because iterating a Graph is unordered and this sample would<br># otherwise show a different literal per language on every run<br>seen = {}<br>for _, p, o in sorted(g_bis2, key=lambda t: (str(t[1]), str(t[2]))):<br>    if p in (GS1.productName, GS1.ingredientStatement) and o.language not in seen:<br>        seen[o.language] = (str(p).rsplit(&quot;/&quot;, 1)[-1], str(o)[:38])<br>print(&quot;\none literal per language tag actually produced:&quot;)<br>for tag in sorted(seen):<br>    prop, text = seen[tag]<br>    print(f&quot;  @{tag:3s} {prop:20s} {text}&quot;)</pre><pre>fields offered : 34<br>  mapped       : 34<br>  UNMAPPED     : none<br><br>one literal per language tag actually produced:<br>  @ar  productName          OTTE BiscutS  NGzari<br>  @en  ingredientStatement  49% sesame seeds, glucose syrup, sugar<br>  @es  productName          OTTE BiscutS  NGzari<br>  @fa  ingredientStatement  آرد گندم روغن قنادی و آردی ،آفتابگردان<br>  @fr  ingredientStatement  Framboise (60%), sucre de canne, poudr<br>  @it  productName          Dore BISCUT<br>  @pl  productName          Happy days butter biscuts<br>  @ro  ingredientStatement  49% _seminte de susan_, sirop de gluco<br>  @tr  productName          Bisküvili Sütlü Çikolata</pre><p>Ten more fields, correctly tagged, and the only thing that changed is the contents of the ontology. ingredients_text_fa - the single column in this file that had never been seen before - arrives tagged @fa, because the mapping says so rather than because anyone taught the code about Farsi.</p><p>That is the third value shape the ontology now describes: quantities with units, tag lists with codes, and text with a language. The cataloguer knows how to <em>follow</em> a shape and nothing about which shapes exist.</p><h3>What ships</h3><p>Two files, and the difference between them is the whole argument. One is everything we added to a vocabulary we do not control. The other is a dataset nobody modelled for, expressed in it.</p><p>The extension is larger here than the 240 triples counted back at <em>What it cost</em>, because the twelve language mappings were declared after that point. The total below is the honest one.</p><pre>delta = Graph()<br>delta.bind(&quot;gs1&quot;, GS1); delta.bind(&quot;ext&quot;, EXT); delta.bind(&quot;skos&quot;, SKOS)<br>for t in onto:<br>    if t not in gs1_published and not any(isinstance(x, BNode) for x in t):<br>        delta.add(t)<br>delta.serialize(&quot;phase2a_extension.ttl&quot;, format=&quot;turtle&quot;)<br><br>catalogue = Graph()<br>catalogue.bind(&quot;gs1&quot;, GS1); catalogue.bind(&quot;ext&quot;, EXT); catalogue.bind(&quot;d&quot;, DATA)<br>catalogue += g_all<br>catalogue.serialize(&quot;phase2a_catalogue.ttl&quot;, format=&quot;turtle&quot;)<br><br>print(f&quot;gs1Voc.ttl              {len(gs1_published):7,} triples   as published, unmodified&quot;)<br>print(f&quot;phase2a_extension.ttl   {len(delta):7,} triples   everything we added&quot;)<br>print(f&quot;phase2a_catalogue.ttl   {len(catalogue):7,} triples   &quot;<br>      f&quot;{len(set(g_all.subjects(RDF.type, GS1.FoodBeverageTobaccoProduct))):,} products as GS1&quot;)</pre><pre>gs1Voc.ttl               18,749 triples   as published, unmodified<br>phase2a_extension.ttl       288 triples   everything we added<br>phase2a_catalogue.ttl    61,160 triples   1,515 products as GS1</pre><h3>Isn’t this RML?</h3><p>Largely, yes — and the answer is worth more than the technique.</p><p>R2RML has been a W3C Recommendation since 2012 for relational sources, and RML extends it to CSV, JSON and XML. A mapping is itself RDF: an rr:TriplesMap with an rml:logicalSource, an rr:subjectMap, and an rr:predicateObjectMap per column. The engines are mature - <strong>Morph-KGC</strong> in Python, built on pandas and using mapping partitions to keep large sources in memory; RMLMapper in Java; SDM-RDFizer; and Ontop if you want virtual RDF over SQL instead of materialisation. The specification is being modularised by the W3C Knowledge Graph Construction Community Group into RML-Core, RML-IO, RML-FNML, RML-CC, RML-Star and RML-LV, with RML-Core and RML-Star reaching Final Community Group Report in October 2025 and a Working Group charter now in W3C review to take them further.</p><p>If you are doing this in production, use them. A hand-written interpreter is not what you want on the critical path.</p><p>So what is the cataloguer? <strong>It is an RML engine with the mapping document taken out and replaced by the ontology.</strong></p><p>The difference is not expressive power, and it is worth being precise about that because the obvious claim is wrong. An rr:predicateMap is a <em>term map</em>, so a predicate can perfectly well be computed - from rr:template or rml:reference. But it is computed from the <strong>data</strong>, not from a <strong>vocabulary</strong>. Nothing in RML resolves sugars against skos:altLabel, reads rdfs:range to decide the value is a node rather than a literal, or consults a unit code list to normalise mg into g. RML-FNML will let you call a function that does all three - at which point you have written the cataloguer and put it inside RML.</p><p>The difference is <strong>where the knowledge lives</strong>, and it has one consequence, which is the consequence this whole post has been about. An RML mapping is per-source: the column names sit in rml:reference, so a new file with new names is a new mapping document. The ontology is not per-source. That is why the biscuits worked.</p><p>Which is testable. If the ontology really is upstream of the mapping, then the mapping is a <em>projection</em> of it — and a projection can be computed. Below, three of them, from one ontology nobody edits in between.</p><pre>RML = Namespace(&quot;http://semweb.mmlab.be/ns/rml#&quot;)   # the modular respec uses<br>RR  = Namespace(&quot;http://www.w3.org/ns/r2rml#&quot;)     # http://w3id.org/rml/ instead<br>QL  = Namespace(&quot;http://semweb.mmlab.be/ns/ql#&quot;)<br><br><br>def generate_rml(onto, columns, source, subject_col=&quot;id&quot;):<br>    &quot;&quot;&quot;Write the RML mapping this ontology implies for these columns.<br><br>    Reads exactly what the Cataloguer reads - altLabel, FieldMapping, range -<br>    and emits a standard mapping document instead of triples.<br>    &quot;&quot;&quot;<br>    cat = Cataloguer(onto)<br>    g = Graph()<br>    for p, ns in ((&quot;rml&quot;, RML), (&quot;rr&quot;, RR), (&quot;ql&quot;, QL), (&quot;gs1&quot;, GS1), (&quot;ext&quot;, EXT)):<br>        g.bind(p, ns)<br><br>    tm, ls, sm = URIRef(&quot;http://example.org/map/#Product&quot;), BNode(), BNode()<br>    g.add((tm, RDF.type, RR.TriplesMap))<br>    g.add((tm, RML.logicalSource, ls))<br>    g.add((ls, RML.source, Literal(source)))<br>    g.add((ls, RML.referenceFormulation, QL.CSV))<br>    g.add((tm, RR.subjectMap, sm))<br>    g.add((sm, RR.template, Literal(&quot;http://example.org/data/{%s}&quot; % subject_col)))<br>    g.add((sm, RR[&quot;class&quot;], GS1.FoodBeverageTobaccoProduct))<br><br>    emitted, needs_more = [], []<br>    for col in sorted(columns):<br>        if col == subject_col:<br>            continue<br>        fm   = cat.mapf.get(col.strip().lower())<br>        prop = (next(onto.objects(fm, EXT.mapsTo)) if fm is not None<br>                else cat.field.get(col.strip().lower()))<br>        if prop is None:<br>            continue<br><br>        rng = cat.range.get(prop)<br>        if rng is not None and (rng, RDF.type, OWL.Class) in onto:<br>            needs_more.append(col)      # a node, not a literal - see below<br>            continue<br><br>        pom, om = BNode(), BNode()<br>        g.add((tm, RR.predicateObjectMap, pom))<br>        g.add((pom, RR.predicate, prop))<br>        g.add((pom, RR.objectMap, om))<br>        g.add((om, RML.reference, Literal(col)))<br>        if rng == RDF.langString:                    # the ontology knew the tag<br>            tag = str(next(onto.objects(fm, EXT.languageTag), &quot;en&quot;)) if fm is not None else &quot;en&quot;<br>            g.add((om, RR.language, Literal(tag)))<br>        emitted.append((col, prop))<br>    return g, emitted, needs_more<br><br><br>def cols_of(feed):<br>    return {k for rec in feed for k in rec}<br><br><br>runs = [(&quot;cereal&quot;,           FEED_ALL,      &quot;openfoodfacts_export.csv&quot;),<br>        (&quot;biscuit&quot;,          bis_feed,      &quot;openfoodfacts_export_biscut.csv&quot;),<br>        (&quot;biscuit +langs&quot;,   bis_feed_all,  &quot;openfoodfacts_export_biscut.csv&quot;)]<br><br>made = {}<br>for label, feed, src in runs:<br>    g, emitted, more = generate_rml(onto, cols_of(feed), src)<br>    made[label] = (g, emitted, more)<br>    print(f&quot;  {label:16s} {len(cols_of(feed)):3d} columns -&gt; &quot;<br>          f&quot;{len(emitted):2d} predicateObjectMaps, {len(more):2d} need more than a reference&quot;)<br><br>made[&quot;cereal&quot;][0].serialize(&quot;phase2a_mapping_cereal.rml.ttl&quot;, format=&quot;turtle&quot;)<br>made[&quot;biscuit +langs&quot;][0].serialize(&quot;phase2a_mapping_biscuit.rml.ttl&quot;, format=&quot;turtle&quot;)<br><br>a = {c for c, _ in made[&quot;cereal&quot;][1]}<br>b = {c for c, _ in made[&quot;biscuit&quot;][1]}<br>print(f&quot;\n  in both mappings   : {sorted(a &amp; b)}&quot;)<br>print(f&quot;  cereal only        : {sorted(a - b) or &#39;none&#39;}&quot;)<br>print(f&quot;  biscuit only       : {sorted(b - a) or &#39;none&#39;}&quot;)<br><br>lang = {c for c, _ in made[&quot;biscuit +langs&quot;][1]} - b<br>print(f&quot;\n  appear only once the language mappings exist: {len(lang)}&quot;)<br>print(wrapped(sorted(lang), indent=&quot;    &quot;))</pre><pre>cereal            64 columns -&gt;  4 predicateObjectMaps, 35 need more than a reference<br>biscuit           25 columns -&gt;  4 predicateObjectMaps, 20 need more than a reference<br>biscuit +langs    35 columns -&gt; 14 predicateObjectMaps, 20 need more than a reference<br><br>in both mappings   : [&#39;allergens&#39;, &#39;brands&#39;, &#39;ingredients_text_en&#39;, &#39;product_name_en&#39;]<br>  cereal only        : none<br>  biscuit only       : none<br><br>appear only once the language mappings exist: 10<br> ingredients_text_fa, ingredients_text_fr, ingredients_text_ro,<br> product_name_ar, product_name_es, product_name_fr, product_name_it,<br> product_name_pl, product_name_ro, product_name_tr</pre><p>Read the first two rows together. The biscuit export shares 358 of its 359 columns with the cereal one, and the two mappings come out <strong>identical</strong> — no column is cereal-only, none is biscuit-only. Nothing in either document is about the file.</p><p>The third row is where the document changes, and it is worth being clear about why: not because a new file arrived, but because ten more columns became <em>reachable</em>. The ext:languageTag mappings were already in the ontology. Offer the translated columns and the projection picks them up.</p><p>So the mapping document is a function of two things — the ontology, and the columns you hand it. Only one of those carries meaning, and it is the one that did not change across all three calls.</p><pre># the object map generated for the one column no cell had ever seen before<br>g_bis_rml = made[&quot;biscuit +langs&quot;][0]<br>tm = URIRef(&quot;http://example.org/map/#Product&quot;)<br><br>frag = Graph()<br>for p, ns in ((&quot;rml&quot;, RML), (&quot;rr&quot;, RR), (&quot;gs1&quot;, GS1)):<br>    frag.bind(p, ns)<br><br>for pom in g_bis_rml.objects(tm, RR.predicateObjectMap):<br>    om = next(g_bis_rml.objects(pom, RR.objectMap))<br>    if (om, RML.reference, Literal(&quot;ingredients_text_fa&quot;)) in g_bis_rml:<br>        frag.add((tm, RR.predicateObjectMap, pom))<br>        for p, o in g_bis_rml.predicate_objects(pom):<br>            frag.add((pom, p, o))<br>            if isinstance(o, BNode):<br>                for p2, o2 in g_bis_rml.predicate_objects(o):<br>                    frag.add((o, p2, o2))<br><br>print(frag.serialize(format=&quot;turtle&quot;).strip())</pre><pre>@prefix gs1: &lt;https://ref.gs1.org/voc/&gt; .<br>@prefix rml: &lt;http://semweb.mmlab.be/ns/rml#&gt; .<br>@prefix rr: &lt;http://www.w3.org/ns/r2rml#&gt; .<br><br>&lt;http://example.org/map/#Product&gt; rr:predicateObjectMap [ rr:objectMap [ rml:reference &quot;ingredients_text_fa&quot; ;<br>                    rr:language &quot;fa&quot; ] ;<br>            rr:predicate gs1:ingredientStatement ] .</pre><p>rr:language &quot;fa&quot; was not written by anyone. It is ext:languageTag from a FieldMapping, projected into a mapping document, for the one column in the biscuit export that nothing above had ever seen.</p><p>And that file is standard RML, which is checkable rather than assertable. <strong>Morph-KGC 2.10 parses </strong>phase2a_mapping_biscuit.rml.ttl<strong>, reports &quot;15 mapping rules retrieved&quot;, and materialises 30 triples from a two-row CSV carrying those column names - including </strong>gs1:ingredientStatement &quot;...&quot;@fa<strong>.</strong> A language tag that began as a triple in an ontology came back out of a third-party engine, correct, with no code of ours in between.</p><p>That is the architecture, and it is the right way round:</p><blockquote><strong><em>an ontology → the axioms, constraints and field mappings the new data needs → generated RML → an engine that scales.</em></strong></blockquote><p>The mapping document RML wants is a projection of the ontology, and a projection can be computed. What cannot be computed is the judgement that went into the ontology first: that fiber and fibre are the same concept, that added sugars are a kind of sugars, that MGM is a thousandth of a GRM, that traces_tags and allergens_tags are one relation separated by a containment code. Put that judgement in a mapping document and it is worth one file. Put it in the ontology and it is worth every file you are ever sent.</p><p>Now the honest half, and it is the same boundary as everywhere else here. Most of the mapped columns are counted and skipped, because rml:reference yields a literal and a measurement is a node carrying a declared value, a declared unit code and a normalised figure. RML expresses the first two natively - a second TriplesMap and a join, or the raw export&#39;s separate .value and .unit columns. The third needs RML-FNML and a function, because normalising mg to g is arithmetic over the unit code list, and no vocabulary performs arithmetic.</p><p>The generator reads rdfs:range to decide which columns are which. It is not making that call - it is reporting one the ontology already made.</p><h3>What we have not used</h3><p>Everything so far rests on four of GS1’s mechanisms. It ships more, and it withholds more. This is the whole inference surface, counted.</p><pre>gs1_only = Graph(); gs1_only.parse(&quot;gs1Voc.ttl&quot;, format=&quot;turtle&quot;)   # no additions of ours<br><br>def count(pred=None, cls=None):<br>    if cls is not None:<br>        return len(set(gs1_only.subjects(RDF.type, cls)))<br>    return len(list(gs1_only.triples((None, pred, None))))<br><br>rows = [<br>    (&quot;rdfs:domain&quot;,              count(RDFS.domain),        &quot;USED    types every record&quot;),<br>    (&quot;rdfs:range&quot;,               count(RDFS.range),         &quot;USED    picks the value shape&quot;),<br>    (&quot;rdfs:subClassOf&quot;,          count(RDFS.subClassOf),    &quot;USED    inherited by closure&quot;),<br>    (&quot;rdfs:subPropertyOf&quot;,       count(RDFS.subPropertyOf), &quot;61 of 62 hang off linkType&quot;),<br>    (&quot;owl:FunctionalProperty&quot;,   count(cls=OWL.FunctionalProperty), &quot;UNUSED  at most one value&quot;),<br>    (&quot;skos:*Match -&gt; schema.org&quot;, count(SKOS.exactMatch) + count(SKOS.closeMatch)<br>                                 + count(SKOS.broadMatch) + count(SKOS.relatedMatch),<br>                                                            &quot;UNUSED  route to JSON-LD&quot;),<br>    (&quot;owl:unionOf&quot;,              count(OWL.unionOf),        &quot;UNREACHED not on any path we map&quot;),<br>    (&quot;owl:Restriction&quot;,          count(cls=OWL.Restriction), &quot;ABSENT  no defined classes&quot;),<br>    (&quot;owl:disjointWith&quot;,         count(OWL.disjointWith),   &quot;ABSENT  nothing can contradict&quot;),<br>    (&quot;owl:InverseFunctionalProperty&quot;, count(cls=OWL.InverseFunctionalProperty),<br>                                                            &quot;ABSENT  the axiom duplicates need&quot;),<br>    (&quot;owl:inverseOf&quot;,            count(OWL.inverseOf),      &quot;ABSENT&quot;),<br>    (&quot;owl:TransitiveProperty&quot;,   count(cls=OWL.TransitiveProperty), &quot;ABSENT&quot;),<br>]<br>print(f&quot;{&#39;mechanism&#39;:33s} {&#39;as published&#39;:&gt;13s}   status&quot;)<br>for name, n, status in rows:<br>    print(f&quot;{name:33s} {n:13d}   {status}&quot;)<br><br>ours_sub = len(list(onto.triples((None, RDFS.subPropertyOf, None)))) - count(RDFS.subPropertyOf)<br>print(f&quot;\nproperty-hierarchy axioms we had to write ourselves: {ours_sub}&quot;)</pre><pre>mechanism                          as published   status<br>rdfs:domain                                 576   USED    types every record<br>rdfs:range                                  558   USED    picks the value shape<br>rdfs:subClassOf                             115   USED    inherited by closure<br>rdfs:subPropertyOf                           92   61 of 62 hang off linkType<br>owl:FunctionalProperty                       41   UNUSED  at most one value<br>skos:*Match -&gt; schema.org                    85   UNUSED  route to JSON-LD<br>owl:unionOf                                   8   UNREACHED not on any path we map<br>owl:Restriction                               0   ABSENT  no defined classes<br>owl:disjointWith                              0   ABSENT  nothing can contradict<br>owl:InverseFunctionalProperty                 0   ABSENT  the axiom duplicates need<br>owl:inverseOf                                 0   ABSENT<br>owl:TransitiveProperty                        0   ABSENT<br><br>property-hierarchy axioms we had to write ourselves: 29</pre><p><strong>Two routes are sitting in the file, unused, and one is a trap.</strong></p><p>The 85 SKOS mappings to schema.org are a publication route: the same catalogued data re-expressed as the vocabulary search engines read. Note what kind of route it is — SKOS mapping properties carry no OWL semantics, so no reasoner walks them for you. It is a rewrite you perform, not an inference you receive. That is Post 1’s opening arriving as an output instead of an anecdote.</p><p>Twenty-nine discriminating domains classify records into five food classes, and we reach exactly one of them.</p><p>The trap is owl:FunctionalProperty, on 41 properties including gtin and netContent. It looks like a free duplicate check. It is the opposite. Functional means <em>at most one value</em>, and under the open-world assumption a second value is not an error - a reasoner concludes the two values denote the same thing. Assert two different gs1:netContent nodes on one product - a functional object property - and OWL does not object, it identifies them. On a functional datatype property the axiom bites the other way: two distinct literals cannot be merged, so the graph is simply inconsistent. The axiom that catches duplicates is owl:InverseFunctionalProperty - two products sharing a GTIN are the same product - and GS1 declares <strong>zero</strong> of those. The duplicate check you want is sh:maxCount 1 in SHACL, which is a different language for a reason.</p><p>owl:unionOf appears 8 times, all of it on organisation and location properties (address, contactPoint, glnType, geo, eligibleQuantity), and none of them is on any path this feed touches. The cataloguer takes the first declared rdfs:range when a property has several, which is safe here only because nothing we map has more than one.</p><p><strong>Three routes require authoring, because GS1 declines to supply them.</strong> Disjointness — zero axioms, so nothing in this vocabulary can contradict anything. Class definitions by condition — zero restrictions, so there is no “a Beverage is anything with alcohol content”. And value-threshold classification, which is not an OWL job at all.</p><p><strong>One route is untouched data.</strong> categories_tags carries 1,515 rows of en:breakfast-cereals-style terms: a complete parallel taxonomy. Aligning it to GS1&#39;s classes is the Sneaker equivalentClass Footwear move from Post 1, and it would classify all 1,515 products rather than the 31 an alcohol figure happens to reach.</p><h3>Where the boundary is</h3><p>An ontology is not a universal decoder, and the honest edges are worth stating.</p><p><strong>Vocabulary.</strong> It deciphers only what it has a term for. Anything with no property and no altLabel is reported as unmapped, never guessed. GS1 genuinely has no property for water, added sugars, lactose, glucose, fructose, fibre fractions or choline; it models caffeine as isDecaffeinated, a code rather than a quantity. Those need declaring, and declaring them is a judgement.</p><p><strong>Shape.</strong> Scalar-plus-unit, literal, language-tagged string, coded node. Not nested structures, not time series, not free text needing NLP.</p><p><strong>Expressivity — and this is where the language you pick decides what you get.</strong></p><p>The sentence to carry away: <strong>OWL infers what must be true; SHACL reports what is not.</strong> Under OWL’s open-world assumption a missing value is <em>unknown</em>, not <em>absent</em> — so a cardinality axiom does not flag two values, it concludes they are the same thing. That is why the constraints in shapes.ttl are SHACL and not OWL.</p><p><strong>Judgement.</strong> Every axiom is a modelling decision. addedSugars being a kind of sugars is a call somebody made, and if it is wrong it is now wrong in every query at once. The roll-up&#39;s power and its risk are the same mechanism.</p><h3>References</h3><h3>Specifications</h3><p>RDF Schema 1.1. W3C Recommendation, 25 February 2014. → <a href="https://www.w3.org/TR/rdf-schema/">w3.org/TR/rdf-schema/</a></p><p>RDF 1.1 Concepts and Abstract Syntax. W3C Recommendation, 25 February 2014. → <a href="https://www.w3.org/TR/rdf11-concepts/">w3.org/TR/rdf11-concepts/</a></p><p>RDF 1.1 Turtle. W3C Recommendation, 25 February 2014. → <a href="https://www.w3.org/TR/turtle/">w3.org/TR/turtle/</a></p><p>SPARQL 1.1 Query Language. W3C Recommendation, 21 March 2013. → <a href="https://www.w3.org/TR/sparql11-query/">w3.org/TR/sparql11-query/</a></p><p>OWL 2 Web Ontology Language Profiles (2nd Edition). W3C Recommendation, 11 December 2012. → <a href="https://www.w3.org/TR/owl2-profiles/">w3.org/TR/owl2-profiles/</a></p><p>Shapes Constraint Language (SHACL). W3C Recommendation, 20 July 2017. → <a href="https://www.w3.org/TR/shacl/">w3.org/TR/shacl/</a></p><p>SKOS Simple Knowledge Organization System Reference. W3C Recommendation, 18 August 2009. → <a href="https://www.w3.org/TR/skos-reference/">w3.org/TR/skos-reference/</a></p><h3>Knowledge graph construction</h3><p>R2RML: RDB to RDF Mapping Language. W3C Recommendation, 27 September 2012. → <a href="https://www.w3.org/TR/r2rml/">w3.org/TR/r2rml/</a></p><p>Dimou, A., Vander Sande, M., Colpaert, P., Verborgh, R., Mannens, E., Van de Walle, R. (2014). <em>RML: A Generic Language for Integrated RDF Mappings of Heterogeneous Data.</em> LDOW 2014 (Linked Data on the Web), WWW workshop.</p><p>Iglesias-Molina, A., et al. (2023). <em>The RML Ontology: A Community-Driven Modular Redesign After a Decade of Experience in Mapping Heterogeneous Data to RDF.</em> ISWC 2023, LNCS. → <a href="https://doi.org/10.1007/978-3-031-47243-5_9">doi.org/10.1007/978–3–031–47243–5_9</a></p><p>RML-Core. W3C Knowledge Graph Construction Community Group, Final Community Group Report, 31 October 2025. → <a href="https://w3id.org/rml/core/spec">w3id.org/rml/core/spec</a></p><p>Arenas-Guerrero, J., Chaves-Fraga, D., Toledo, J., Pérez, M. S., Corcho, O. (2024). <em>Morph-KGC: Scalable knowledge graph materialization with mapping partitions.</em> Semantic Web 15(1), 1–20. → <a href="https://doi.org/10.3233/SW-223135">doi.org/10.3233/SW-223135</a></p><h3>Vocabularies, data and units</h3><p>GS1 Web Vocabulary. → <a href="https://ref.gs1.org/voc/">ref.gs1.org/voc/</a></p><p>Open Food Facts. Product data by Open Food Facts contributors, used under the Open Database Licence (ODbL) v1.0. → <a href="https://world.openfoodfacts.org/">world.openfoodfacts.org</a> · <a href="https://opendatacommons.org/licenses/odbl/1-0/">opendatacommons.org/licenses/odbl/1–0/</a></p><p>UN/ECE Recommendation 20, <em>Codes for Units of Measure Used in International Trade</em> — the list gs1:unitCode points at and does not ship. → <a href="https://unece.org/trade/uncefact/cl-recommendations">unece.org/trade/uncefact/cl-recommendations</a></p><h3>Tools</h3><p>rdflib, owlrl, pySHACL, pandas and Graphviz. Exact versions, and md5 checksums of every input file, are printed at the top of this post - because a number measured against an unspecified version of a moving file is not a measurement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a469c47de6f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ontology: Structured meaning without a model]]></title>
            <link>https://medium.com/@abhi1993nair/ontology-structured-meaning-without-a-model-f02b74a9a6c8?source=rss-308382bf640e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f02b74a9a6c8</guid>
            <category><![CDATA[ontology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mapping]]></category>
            <dc:creator><![CDATA[Abhijith Nair]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 06:19:57 GMT</pubDate>
            <atom:updated>2026-08-11T07:19:13.220Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARA8qekDu-weaX-Ribqs2g.png" /></figure><p>Search <em>what is an ontology</em> and the first answer you get is: <strong>a formal, explicit specification of a shared conceptualization.</strong></p><p>That one sentence has ended more engineers’ interest in the subject than anything else in computing. It isn’t wrong. It’s just useless — true in the way that <em>a two-wheeled human-powered vehicle</em> is true, and about as much help when you’re trying to stay upright.</p><p>So let’s start somewhere else. Let’s start with a file you have already written.</p><pre>Supplier A          Supplier B          Supplier C<br>----------          ----------          ----------<br>sku                 item_code           article_no<br>size_eu             euSize              Size (EU)<br>net_wt              weight_grams        Nett Wt<br>barcode             ean13               GTIN</pre><p>Four fields. Three suppliers. Twelve names.</p><p>You have something that handles this. It’s called mappings.py, or field_map.json, or — if you&#39;ve been at the company long enough — mappings_v2_FINAL.py. Somebody updates it every quarter. It breaks when a vendor renames a column. It breaks quietly, and you hear about it from a customer.</p><p>Supplier D signs next month. That’s a two-week ticket, and you already know roughly which two weeks.</p><p>Here’s the part worth saying out loud: <strong>that file is trying to be an ontology.</strong> It’s just a bad one — a flat list of string pairs, with no structure, no way to check itself, and no way to answer a question nobody wrote a row for.</p><p>This post is about what the good version looks like, and how far it gets you before any AI is involved.</p><h3>What is an ontology?</h3><p>Let’s go back to that definition, because it isn’t useless. It’s compressed. <em>A formal, explicit specification of a shared conceptualization</em> — four words doing four separate jobs:</p><ul><li><strong>formal</strong> — a machine can check it, not just read it</li><li><strong>explicit</strong> — written down, not living in a senior engineer’s head</li><li><strong>shared</strong> — more than one system agrees to it</li><li><strong>conceptualization</strong> — about the concepts, not how they’re stored</li></ul><p>That last one is where engineers slide off. An ontology is not a database schema. A schema says how bytes are laid out. An ontology says what things <em>are</em> — and it stays true whether the data is sitting in Postgres, a CSV, or the body of an email.</p><p>The way in is a ladder you have already climbed twice.</p><p><strong>A list has items.</strong> Your twelve field names, flat. No structure, nothing derivable.</p><p><strong>A taxonomy adds hierarchy.</strong> Product → WearableProduct → Footwear. Better — you can roll things up. But it&#39;s still just nesting, and you have this already. It&#39;s your category tree.</p><p><strong>An ontology adds relationships and constraints between concepts.</strong> Not only that Footwear sits under Product, but that upper material applies to footwear and nothing else; that a product carries at most one Global Trade Item Number (GTIN); that footwear and food are mutually exclusive. That third rung buys you something the first two can’t. Give a machine a single fact — <em>this record has an upper material</em> — and it concludes: therefore footwear, therefore a wearable product, therefore a product. Three types it was never told, from one line it was.</p><p>Now hang a rule off the top of that chain: every product needs a GTIN. The machine applies it to a record nobody classified, using a rule nobody wrote for shoes. The rule says <em>product</em>. The record never said <em>product</em>. Nothing in the pipeline was taught anything about footwear. Worth noticing which half of that came free. The inheritance was already in the vocabulary. The rule was not — somebody had to state it. That distinction comes back at the end of this post.</p><p>One honest note: in practice “ontology,” “taxonomy,” and “vocabulary” get used loosely and interchangeably, and arguing about it is a good way to lose an afternoon. The boundary that matters isn’t the label — it’s whether a machine can derive something you didn’t state.</p><h3>Where this came from</h3><p>Search <em>what is an ontology</em> and you land in philosophy. That’s fair, but it isn’t where the software came from. The actual lineage runs in five beats.</p><h3>Aristotle, c. 350 BCE</h3><p>The <em>Categories</em>, part of the <em>Organon</em>, sets out ten kinds of thing that can be said to exist — substance, quantity, quality, relation, and six more. That’s where the word comes from, and why it still carries philosophical weight.</p><p>It is not where the software comes from. Moving on.</p><h3>1984 — Cyc, and the cautionary tale</h3><p>Doug Lenat starts Cyc at the Microelectronics and Computer Technology Corporation in Austin. The plan: hand-encode the common-sense knowledge every human has and no computer had. That objects fall. That people can’t be in two places at once. That a dead thing stays dead.</p><p>Decades of work, millions of assertions, and Cyc still exists commercially. It is also the field’s standing warning — encoding everything by hand is slower than the world changes.</p><h3>1991–1993 — the actual origin</h3><p>This is the beat worth knowing, because the motivation isn’t the one people assume.</p><p>Expert systems worked. That was the problem. Every lab built its own knowledge base, in its own representation, and none could talk to any other. Nothing was reusable. Everything got rebuilt from scratch.</p><p>The Knowledge Sharing Effort, funded by the US Defense Advanced Research Projects Agency (DARPA), existed to fix exactly that. Its founding paper is Neches, Fikes, Finin, Gruber, Patil, Senator and Swartout, <em>Enabling Technology for Knowledge Sharing</em> (AI Magazine 12(3), 1991).</p><p>Out of it came Thomas Gruber’s <em>A Translation Approach to Portable Ontology Specifications</em> (1993) — the paper carrying the definition everyone quotes. Its stated purpose was to support the sharing and reuse of formally represented knowledge among AI systems, by defining the common vocabulary that shared knowledge is represented in.</p><p><strong>Ontologies were invented to solve an integration problem, not a reasoning problem.</strong> Reasoning came along for the ride.</p><p>One detail that ages well: the same paper introduced Ontolingua, a translation layer, because even in 1993 the formats couldn’t read each other. A few cells from now we convert this vocabulary from Turtle into RDF/XML — two ways of writing down the same Resource Description Framework (RDF) data — because the Python library that reads one will not read the other. Same problem, thirty-three years later.</p><h3>2001–2009 — the standards era</h3><p>May 2001. Berners-Lee, Hendler and Lassila publish <em>The Semantic Web</em> in Scientific American. The vision: a web where machines process meaning, not just markup. Widely read, and widely mocked afterwards for over-promising — but it moved the money and the working groups.</p><p>The pieces already existed. Two rival ontology languages had been converging: OIL (Ontology Inference Layer, European) and DAML-ONT (DARPA Agent Markup Language, American, October 2000). They merged into DAML+OIL — first in December 2000, then extended in March 2001 with XML Schema datatypes — a semantic markup language built on RDF and RDF Schema with richer modelling primitives. Lucent Technologies submitted it to the World Wide Web Consortium (W3C), which published it as a W3C Note on 18 December 2001. The Web Ontology Working Group convened that year to turn it into a standard.</p><p><strong>10 February 2004 — OWL 1,</strong> the Web Ontology Language. Everything lands on the same day: RDF Concepts and Abstract Syntax, RDF/XML Syntax, RDF Schema, and the six OWL documents. OWL is explicitly a revision of DAML+OIL, incorporating the lessons of its design and application.</p><p>The design decision worth understanding is the three sublanguages, because it makes the expressiveness-versus-decidability trade-off visible:</p><ul><li><strong>OWL Lite</strong> — a classification hierarchy and simple constraints</li><li><strong>OWL DL (Description Logic)</strong> — maximum expressiveness that still guarantees the computation finishes</li><li><strong>OWL Full</strong> — maximum freedom and syntactic licence, no computational guarantees</li></ul><p>OWL Full lets you mix freely with RDF Schema and enforces no separation between classes, properties, individuals and data values. OWL DL requires that separation. The reason OWL DL exists at all is that tool builders had powerful reasoning systems which only work under those restrictions.</p><p>That’s the whole tension in one line: you can say anything, or you can guarantee an answer. <strong>OWL DL is where the reasoner at the end of this post lives.</strong></p><p>It also explains something we hit in the next section, loading the GS1 vocabulary — GS1 being the standards body behind the barcode. A Description Logic tool will only hand you the classes an ontology actually declares, while a vocabulary written in the looser OWL Full style is free to refer to plenty it never declares. Nothing errors. The list is just short. A 2004 design decision, still shaping what your loader returns in 2026.</p><p><strong>15 January 2008 — SPARQL 1.0</strong> — the SPARQL Protocol and RDF Query Language, one of computing’s more determined recursive acronyms — becomes a W3C Recommendation. Ontologies become queryable, not just declarative. For most engineers, this is when the stack turns usable.</p><p><strong>27 October 2009 — OWL 2</strong>, a W3C Recommendation compatible with the 2004 version. Its contribution was replacing Lite/DL/Full with three profiles named EL, QL and RL — each a fragment of the language chosen so that one specific reasoning task stays tractable. Same trade-off, made practical instead of philosophical.</p><h3>2011–2012 — it stops being academic</h3><p>schema.org launches on 2 June 2011 — Bing, Google and Yahoo, with Yandex joining that November. (Most write-ups say all four launched it. They didn’t.)</p><p>The scale it started at is worth quoting: 297 classes and 187 relations at launch, grown to 638 classes and 965 relations within five years. In November 2012 the GoodRelations e-commerce ontology was merged into schema.org’s core — which is why Offer and price exist in your product markup at all.</p><p>Google introduces the Knowledge Graph terminology in 2012, seeded by Freebase, acquired when Google bought Metaweb in 2010.</p><p>Thirty years from a DARPA reusability problem to the markup on your product page.</p><h3>Why anyone bothered</h3><p>Go back to 1991 and look at what was actually broken.</p><p>It wasn’t that machines couldn’t reason. Expert systems reasoned fine. MYCIN picked antibiotics for blood infections about as well as the Stanford faculty it was scored against — and was never used on a patient. XCON configured VAX orders for Digital every day and saved them millions a year. The problem was that every one of them was an island. A team spent three years encoding a domain, and the team next door couldn’t use a line of it: different representation, different vocabulary, different assumptions about what a “patient” or a “component” was.</p><p>Not incompatible formats. Incompatible meanings.</p><p>So the money went to the sharing problem. That’s what the DARPA Knowledge Sharing Effort was for, and it’s why Gruber’s definition contains the word <em>shared</em>. It isn’t decoration. It’s the entire point.</p><p>Ontologies were built to make knowledge portable between systems. The reasoning came along for the ride — and the reasoning is the part everyone remembers, which is why most introductions to this subject open in the wrong place.</p><p><strong>You have this problem. Today.</strong></p><p>Two teams in the same company ship services that both have a User. One means <em>has logged in at least once</em>. The other means <em>a row exists in the table</em>. Nobody wrote either definition down, and the reconciliation job has been quietly wrong for eight months.</p><p>A company gets acquired. Their status field has seven values; yours has four. Three map cleanly, two are the same thing under different names, and two describe a state your system has no concept of. Somebody decides what happens to those two in a meeting, and the decision survives only in a migration script.</p><p>A partner sends you a Swagger file. status: string tells you the shape and nothing about the meaning. You infer the rest from example payloads and a support thread, and you find out you inferred wrong when a customer hits a case you never saw.</p><p>Every one of these is the 1991 problem: two systems, the same concepts, different vocabularies, and no formal statement of what either one actually means. It isn’t getting easier, either — a growing share of the schemas arriving at your door were generated by a model with no more idea what status means than you have.</p><p>And your solution is the file from the top of this post. mappings.py. Four hundred rows of string pairs, with four properties worth naming.</p><p><strong>It carries no reasoning.</strong> size_eu → size, and nothing else. It can&#39;t tell you that a size belongs to a wearable product, that a wearable product is a product, that a product needs a GTIN. Every consequence gets written out separately, by a person, who has to remember.</p><p><strong>It can’t be checked.</strong> Nothing detects that a row is wrong. It’s a dict — every entry is equally valid to the interpreter. The first review any row ever gets is a customer complaint.</p><p><strong>It doesn’t compose.</strong> Three suppliers, three mapping files. Mapping A to B tells you nothing about mapping B to C. Supplier four is a fourth file.</p><p><strong>It can’t answer anything you didn’t anticipate.</strong> A field arrives that nobody wrote a row for, and the system has no opinion. Not <em>I don’t know</em> — no opinion at all.</p><p>That file is trying to be an ontology. It’s a flat, unstructured, uncheckable one, and everyone maintaining it knows it’s wrong without having a name for what it should be instead.</p><h3>Java first, Python later</h3><p>If you went looking for ontology tooling before roughly 2015, you found Java. Not because the community was conservative, but because of what the tooling actually has to do.</p><h3>Why Java got there first</h3><p>A reasoner is not a parser. Deciding whether one class subsumes another in OWL DL means running a <strong>tableau algorithm</strong> — a search that builds candidate models and tries to derive a contradiction, backtracking through branches and merging nodes when cardinality constraints force it. Worst case, it is exponential. Getting one correct is a PhD; getting one fast is a career.</p><p>So they were written where the researchers were, and the researchers were in Java:</p><ul><li><strong>Protégé</strong> (Stanford) — the ontology editor, and still the reference tool</li><li><strong>Apache Jena</strong> — the RDF and SPARQL framework</li><li><strong>The OWL API</strong> (Manchester) — the manipulation library nearly every other tool is built on</li><li><strong>The reasoners</strong> — HermiT (Oxford), Pellet, FaCT++ (Manchester), and ELK for the EL profile</li></ul><p>That is not a legacy stack. It’s an unusually deep one — twenty-five years of correctness work on algorithms where “mostly right” is worthless.</p><h3>Python arrived through data science</h3><p>Python got here from a different direction. Not the Semantic Web community, but the people who had graphs and data and wanted both in a notebook.</p><p><strong>rdflib</strong> is the foundation. Parse and serialize Turtle, JSON-LD, RDF/XML and N-Triples; hold a graph in memory; run SPARQL over it. It’s the tool you will spend the most time in, and it’s pure Python.</p><p><strong>Owlready2</strong>, written by Jean-Baptiste Lamy, does something different: it makes OWL classes behave like Python classes. class Sneaker(Thing) and sku.gtin = &quot;8901234567892&quot; are ordinary Python that happens to be editing an ontology. If you want a book, his <em>Ontologies with Python</em> (Apress) is the one.</p><p><strong>pySHACL</strong> validates data against Shapes Constraint Language (SHACL) shapes. <strong>LinkML</strong> goes the other way — write a YAML schema, generate Python dataclasses, JSON Schema and OWL from it. That last one is the pragmatic route for teams who are never going to write RDF by hand.</p><h3>The punchline</h3><p>Python didn’t replace Java here.</p><p>Owlready2 ships a Java .jar. Call sync_reasoner() and a Java Virtual Machine starts underneath your import, runs HermiT, and hands the inferred triples back to Python. Same for sync_reasoner_pellet().</p><p>That is why the setup below asks for a Java Runtime Environment (JRE), and why a “pure Python” reasoner isn’t a realistic option — nobody is going to reimplement twenty-five years of tableau optimization to avoid a dependency.</p><p>It isn’t a failure of Python. It’s the reason you can use any of this today: the hard part was solved somewhere else, correctly, and you get to call it inside a with block. You will write zero Java in this post, and Java will run.</p><h3>What to install</h3><ul><li><strong>parse Turtle, run SPARQL</strong> — rdflib</li><li><strong>OWL classes as Python objects, reasoning</strong> — Owlready2 (+ a JRE)</li><li><strong>validate data against shapes</strong> — pySHACL</li><li><strong>schema-first: YAML → Python + OWL</strong> — LinkML</li><li><strong>edit an ontology by hand</strong> — Protégé (desktop)</li></ul><pre>pip install rdflib owlready2 pyshacl</pre><h3>Reading a real ontology: GS1</h3><p>GS1 are the barcode people. Every product you have ever scanned carries one of their identifiers, which means they have spent fifty years doing the unglamorous work of agreeing what a “product” is across every retailer and supplier on earth. Their Web Vocabulary is the result, published as OWL in both Turtle and JSON-LD, free to download.</p><p>It is a real standard, not a teaching example: 18,749 triples, 557 properties, and a class hierarchy that runs from Product down through WearableProduct to Footwear. Long names, inconsistent conventions, written for people who already know the standard. Exactly the kind of vocabulary that defeats string matching.</p><p><strong>The first thing that goes wrong is the file format.</strong> Owlready2 cannot parse Turtle. Hand it the download and you get NTriples parsing error (or unrecognized file format), which sends you looking for a corrupt file. The file is fine. It&#39;s the wrong serialisation.</p><p>rdflib parses Turtle and writes RDF/XML; Owlready2 reads RDF/XML. Two libraries, one translation step — which is Ontolingua’s problem from 1993, still in your terminal.</p><pre>from rdflib import Graph<br>from owlready2 import get_ontology<br><br># Owlready2 reads RDF/XML, not Turtle. rdflib does the translation.<br>g = Graph()<br>g.parse(&quot;gs1Voc.ttl&quot;, format=&quot;turtle&quot;)<br>g.serialize(destination=&quot;gs1Voc.owl&quot;, format=&quot;xml&quot;)   # one-off: the file persists<br>onto = get_ontology(&quot;gs1Voc.owl&quot;).load()<br>print(f&quot;{len(g):,} triples parsed&quot;)</pre><pre>18,749 triples parsed</pre><h3>The second thing that goes wrong is quieter</h3><p>Now ask the loaded ontology for its classes. You get a list, no warnings, and nothing that suggests you should look any closer.</p><p>Count it three ways instead: how many classes the file <em>declares</em>, how many it <em>refers to</em> as a domain or a superclass, and how many Owlready2 hands back.</p><pre>from rdflib import RDF, RDFS, OWL, URIRef<br>declared = {s for s in g.subjects(RDF.type, OWL.Class)} | \<br>           {s for s in g.subjects(RDF.type, RDFS.Class)}<br>referenced = declared | set(g.subjects(RDFS.subClassOf, None)) \<br>                      | set(g.objects(None, RDFS.subClassOf)) \<br>                      | set(g.objects(None, RDFS.domain))<br># named classes only - drop the anonymous class expressions<br>declared   = {c for c in declared   if isinstance(c, URIRef)}<br>referenced = {c for c in referenced if isinstance(c, URIRef)}<br>print(f&quot;declared as a class    : {len(declared)}&quot;)<br>print(f&quot;referenced as a class  : {len(referenced)}&quot;)<br>print(f&quot;returned by classes()  : {len(list(onto.classes()))}&quot;)<br>print(f&quot;referenced, undeclared : {len(referenced - declared)}&quot;)<br>local = lambda u: str(u).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br>print(&quot;\nundeclared, sample:&quot;, sorted(local(u) for u in (referenced - declared))[:6])</pre><pre>declared as a class    : 56<br>referenced as a class  : 123<br>returned by classes()  : 56<br>referenced, undeclared : 67<br><br>undeclared, sample: [&#39;AllergenTypeCode&#39;, &#39;AnatomicalFormCode&#39;, &#39;BusinessEntityType&#39;, &#39;CertificationStatus&#39;, &#39;CheeseFirmnessCode&#39;, &#39;ColourCodeListCode&#39;]</pre><p>Owlready2 returns 56, and it is right: 56 is exactly what the file declares. The loader isn’t dropping anything.</p><p>The gap is in the vocabulary. GS1 refers to 123 classes and declares 56 of them. The other 67 — AllergenTypeCode, CheeseFirmnessCode, ColourCodeListCode and the rest of the code lists — are used as the domain of a property or the parent of a class without the file ever saying <em>this is a class</em>. Legal in OWL Full, invisible to a Description Logic tool, and reported by nobody.</p><p>This is worth sitting with, because it is the whole post in miniature. Nothing failed. No exception, no warning, no null. You asked a question, got a confident answer, and the answer was complete only with respect to a rule nobody told you about. If you had built a mapping table off that list, it would be missing two-thirds of the vocabulary’s concepts and would look finished.</p><h3>The hierarchy is the mechanism</h3><p>ancestors() returns every superclass of a class, and it is the only fact the rest of this post runs on. Two classes are compatible when one lies on the other&#39;s chain — meaning one individual could be both. Anything else is two different kinds of thing, and no single record can be both at once.</p><pre>for name in (&quot;Product&quot;, &quot;WearableProduct&quot;, &quot;Footwear&quot;):<br>    c = onto.search_one(iri=f&quot;*/{name}&quot;)<br>    print(f&quot;{name:16s} -&gt; {[a.name for a in c.ancestors() if hasattr(a, &#39;name&#39;)]}&quot;)</pre><pre>Product          -&gt; [&#39;Product&#39;, &#39;Thing&#39;]<br>WearableProduct  -&gt; [&#39;Product&#39;, &#39;WearableProduct&#39;, &#39;Thing&#39;]<br>Footwear         -&gt; [&#39;Product&#39;, &#39;WearableProduct&#39;, &#39;Thing&#39;, &#39;Footwear&#39;]</pre><h3>Domains: where a property is allowed to live</h3><p>rdfs:domain states which class a property belongs to, and it is the hinge for everything that follows. isSeedless is the example that lands: its domain is FruitsVegetables, because seedlessness is meaningless anywhere else.</p><p>Read that carefully. Asserting isSeedless does not merely describe a product — it <em>commits</em> the record to that branch of the taxonomy. A property is not a label you attach. It is a claim about what the thing is.</p><p>That is not a rare trick, either. 481 of GS1’s 557 properties have a domain more specific than Product, so most of the vocabulary carries typing information whether or not anyone reads it that way.</p><pre>seedless = onto.search_one(iri=&quot;*isSeedless&quot;)<br>print(&quot;isSeedless domain:&quot;, seedless.domain)<br>narrowing = [p for p in onto.properties()<br>             if p.domain and p.domain[0] is not onto.Product]<br>print(f&quot;properties whose domain is narrower than Product: {len(narrowing)}&quot;)</pre><pre>isSeedless domain: [gs1Voc.FruitsVegetables]<br>properties whose domain is narrower than Product: 481</pre><h3>Domains that aren’t a single class</h3><p>Not every domain is one named class. GS1 uses unions, so a domain entry can be a class expression rather than a class, and printing one raw is unreadable. Two small helpers fix that: one to render an expression, one to flatten it to the named classes involved so membership tests still work.</p><pre>from owlready2 import ThingClass, And, Or, Not, Restriction<br><br>def label(c):<br>    &quot;&quot;&quot;Readable name for a domain entry — named class or class expression.&quot;&quot;&quot;<br>    if c is None:                    return &quot;-&quot;<br>    if isinstance(c, ThingClass):    return c.name<br>    if isinstance(c, Or):            return &quot; OR &quot;.join(label(x) for x in c.Classes)<br>    if isinstance(c, And):           return &quot; AND &quot;.join(label(x) for x in c.Classes)<br>    if isinstance(c, Not):           return f&quot;NOT {label(c.Class)}&quot;<br>    if isinstance(c, Restriction):   return f&quot;{c.property.name} restriction&quot;<br>    return str(c)<br><br>def named(c):<br>    &quot;&quot;&quot;Flatten to the named classes involved, so membership tests work.&quot;&quot;&quot;<br>    if isinstance(c, ThingClass):    return {c}<br>    if isinstance(c, (Or, And)):     return {n for x in c.Classes for n in named(x)}<br>    if isinstance(c, Not):           return named(c.Class)<br>    return set()<br><br>shown = 0<br>for p in onto.properties():<br>    if p.domain and onto.Product not in named(p.domain[0]):<br>        print(f&quot;{p.name:32s} {label(p.domain[0])}&quot;)<br>        shown += 1<br>        if shown == 15:<br>            print(f&quot;... and {len(narrowing) - shown} more&quot;)<br>            break</pre><pre>isOnlyWithPaymentCard            Discount<br>faxNumber                        ContactPoint<br>allergenSpecificationAgency      FoodBeverageTobaccoProduct<br>awardPrizeCode                   AwardPrizeDetails<br>postalName                       PostalAddress<br>alcoholicBeverageSubregion       Beverage<br>transactionDate                  Transaction<br>addressSuburb                    PostalAddress<br>preparationInstructions          FoodAndBeveragePreparationInformation<br>isOnlyWithRetailerPaymentCard    Discount<br>maximumDiscountAmount            Discount<br>convenienceLevelPercent          FoodAndBeveragePreparationInformation<br>minPrice                         PriceSpecification<br>priceCurrency                    PriceSpecification<br>streetAddressLine4               PostalAddress<br>... and 466 more</pre><h3>Seeing the shape</h3><p>Graphviz, purely to make the next two figures legible. If the binary isn’t on PATH these cells fail while everything else still runs — nothing below depends on them.</p><h3>Two figures</h3><p>The first is the class hierarchy under Product. The second adds property domain edges: a solid edge means the property narrows to a subclass, which is exactly the narrowing the resolver later exploits.</p><pre>from graphviz import Digraph<br>from owlready2 import default_world<br><br>PALETTE = {<br>    &quot;class_fill&quot;: &quot;#E8F0FE&quot;, &quot;class_line&quot;: &quot;#1A4D8F&quot;,<br>    &quot;root_fill&quot;:  &quot;#1A4D8F&quot;, &quot;root_text&quot;:  &quot;white&quot;,<br>    &quot;prop_fill&quot;:  &quot;#F1F8E9&quot;, &quot;prop_line&quot;:  &quot;#4C7A1F&quot;,<br>    &quot;far_fill&quot;:   &quot;#FFF3E0&quot;, &quot;far_line&quot;:   &quot;#B35C00&quot;,<br>    &quot;map_line&quot;:   &quot;#B00060&quot;,<br>}<br><br>def descendants(root, limit=40):<br>    &quot;&quot;&quot;Root plus its subclasses, breadth-first, capped.&quot;&quot;&quot;<br>    out, queue = [], [root]<br>    while queue and len(out) &lt; limit:<br>        c = queue.pop(0)<br>        if c in out:<br>            continue<br>        out.append(c)<br>        queue += sorted(c.subclasses(), key=lambda x: x.name)<br>    return out<br><br>def draw(root, limit=40, properties=True, mappings=True,<br>         rankdir=&quot;BT&quot;, title=None, palette=None, dpi=None):<br>    &quot;&quot;&quot;Render an ontology branch. Returns a Digraph, which Jupyter shows inline.&quot;&quot;&quot;<br>    pal = {**PALETTE, **(palette or {})}<br>    g = Digraph(&quot;onto&quot;, format=&quot;svg&quot;)<br>    g.attr(rankdir=rankdir, bgcolor=&quot;white&quot;, splines=&quot;polyline&quot;,<br>           nodesep=&quot;0.35&quot;, ranksep=&quot;0.55&quot;, pad=&quot;0.4&quot;)<br>    if dpi:<br>        g.attr(dpi=str(dpi))<br>    g.attr(&quot;node&quot;, fontname=&quot;Helvetica&quot;, fontsize=&quot;11&quot;,<br>           shape=&quot;box&quot;, style=&quot;rounded,filled&quot;, penwidth=&quot;1.4&quot;)<br>    g.attr(&quot;edge&quot;, fontname=&quot;Helvetica&quot;, fontsize=&quot;9&quot;,<br>           color=&quot;#5F6B7A&quot;, arrowsize=&quot;0.7&quot;)<br>    if title:<br>        g.attr(label=f&quot;\n{title}&quot;, labelloc=&quot;t&quot;,<br>               fontname=&quot;Helvetica-Bold&quot;, fontsize=&quot;15&quot;, fontcolor=&quot;#1A2733&quot;)<br>    classes = descendants(root, limit)<br>    ids = {c: f&quot;c{i}&quot; for i, c in enumerate(classes)}<br>    # classes + subClassOf<br>    for c in classes:<br>        is_root = c is root<br>        g.node(ids[c], c.name,<br>               fillcolor=pal[&quot;root_fill&quot;] if is_root else pal[&quot;class_fill&quot;],<br>               color=pal[&quot;class_line&quot;],<br>               fontcolor=pal[&quot;root_text&quot;] if is_root else &quot;#12283D&quot;)<br>        for parent in c.is_a:<br>            if parent in ids and parent is not c:<br>                g.edge(ids[c], ids[parent], label=&quot;  subClassOf&quot;)<br>    # properties attached to their domain<br>    # SOLID = domain narrows below root (the interesting case). DASHED = on root.<br>    if properties:<br>        for p in root.namespace.ontology.properties():<br>            if not p.domain:<br>                continue<br>            targets = named(p.domain[0]) &amp; set(classes)<br>            if not targets:<br>                continue<br>            narrows = root not in named(p.domain[0])<br>            pid = f&quot;p_{p.name}&quot;<br>            g.node(pid, p.name, shape=&quot;ellipse&quot;,<br>                   fillcolor=pal[&quot;prop_fill&quot;], color=pal[&quot;prop_line&quot;],<br>                   fontcolor=&quot;#2C4A12&quot;, penwidth=&quot;1.8&quot; if narrows else &quot;1.0&quot;)<br>            for t in targets:<br>                g.edge(pid, ids[t], label=&quot;  domain&quot;, color=pal[&quot;prop_line&quot;],<br>                       style=&quot;solid&quot; if narrows else &quot;dashed&quot;)<br>    # equivalentClass links.<br>    # Owlready2 does NOT mirror equivalent_to: set it on A and B.equivalent_to<br>    # stays empty. So scan every loaded ontology, in both directions.<br>    if mappings:<br>        inside, pairs = set(classes), set()<br>        for o in default_world.ontologies.values():<br>            for c in o.classes():<br>                for eq in getattr(c, &quot;equivalent_to&quot;, []):<br>                    if not isinstance(eq, ThingClass):<br>                        continue<br>                    if c in inside and eq not in inside:   pairs.add((c, eq))<br>                    elif eq in inside and c not in inside: pairs.add((eq, c))<br>        for local, far in sorted(pairs, key=lambda p: p[1].name):<br>            fid = f&quot;f_{id(far)}&quot;<br>            g.node(fid, f&quot;{far.namespace.name}:{far.name}&quot;,<br>                   fillcolor=pal[&quot;far_fill&quot;], color=pal[&quot;far_line&quot;],<br>                   fontcolor=&quot;#5C2E00&quot;, style=&quot;rounded,filled,dashed&quot;)<br>            g.edge(ids[local], fid, label=&quot;  equivalentClass&quot;, dir=&quot;both&quot;,<br>                   color=pal[&quot;map_line&quot;], fontcolor=pal[&quot;map_line&quot;],<br>                   style=&quot;bold&quot;, penwidth=&quot;1.6&quot;, constraint=&quot;false&quot;)<br>    return g</pre><pre>draw(onto.Product, properties=False, mappings=False,<br>     title=&quot;GS1 Product hierarchy&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvOdtV6SgCHeTi7kQIejjQ.png" /><figcaption>GS1 Product hierarchy</figcaption></figure><h3>One axiom connects a supplier’s catalog</h3><p>Everything so far has been reading. This is the first thing we write.</p><p>A supplier sends you a catalog full of things they call Sneaker. GS1 has no such class — it has Footwear. The mappings.py answer is a row: Sneaker → Footwear, one string pointing at another, carrying nothing.</p><p>The ontology answer is an axiom. Sneaker and Footwear are declared <em>equivalent</em>: not similar, not mapped, but the same class under two names. From that moment every fact GS1 states about footwear applies to the supplier&#39;s class, and every fact the supplier states about a sneaker is a fact about footwear. Nobody enumerated the consequences. There is nothing to keep in sync.</p><p>Note where the class is defined. Sneaker lives in the supplier&#39;s own ontology, at their own IRI — you are not editing GS1, and you are not asking the supplier to adopt it. Two vocabularies stay separate and become interoperable anyway, which is the 1991 problem being solved exactly as intended.</p><pre>from owlready2 import Thing<br>acme = get_ontology(&quot;http://acme.example/catalog#&quot;)<br>with acme:<br>    class Sneaker(Thing): pass<br>acme.Sneaker.equivalent_to = [onto.Footwear]<br>draw(onto.Footwear, title=&quot;One axiom links a supplier catalog to GS1&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwaF5BNheLdGs2zyy9GYvw.png" /><figcaption>Axiom Links</figcaption></figure><p>The figure shows the result: the supplier’s class sitting inside the GS1 hierarchy, connected by the one statement we made.</p><p>Be clear about what this does and does not buy you. It is genuinely compositional — a second supplier’s Trainer needs one axiom, not a second mapping file, and the two suppliers are now related to each other through GS1 without anyone writing a Sneaker → Trainer rule. That is the property mappings.py could never have.</p><p>What it does not do is tell you whether the equivalence is <em>true</em>. A person decided that a sneaker is footwear. The machine propagates that decision faithfully and will propagate a wrong one just as faithfully. Structure removes the bookkeeping, not the judgement.</p><h3>From an email to a typed record</h3><p>Now the harder case, and the one you actually have.</p><p>The supplier above was cooperative enough to send a catalog with class names in it. More often what arrives is this: a human being typing field names from memory, in an email, at the end of the day. No schema, no class, no agreement on what anything is called.</p><p>The rest of this blog works one such message. The goal is not to guess well. It is to be <em>honest about what the message determines</em> — and the mechanism for that is the structure we have just spent the post reading. Lexical similarity gets a vote on what a line might mean. The vocabulary’s own domains decide what survives.</p><p>Every layer from here reports one of three states, and no layer is allowed to collapse the middle one into a plausible single answer:</p><ul><li>RESOLVED — exactly one reading survives</li><li>AMBIGUOUS — several survive; the message genuinely does not say</li><li>UNRESOLVED — nothing survives; the vocabulary has no home for this</li></ul><p>That third column is the whole discipline. A pipeline that returns AMBIGUOUS is more useful than one that returns a confident wrong answer, and considerably harder to build, because at every layer the tempting shortcut is to pick the best-scoring option and move on.</p><h3>Reading the message</h3><p>The first step is deliberately dumb. Find lines that look like key: value or key - value, normalize the key, and keep everything else to one side.</p><p>Two details matter more than they look.</p><p><strong>Lines that aren’t field-shaped are kept, not dropped.</strong> “Hi team” and “Regards, John” are not parse failures — they are correctly identified as carrying no field. A pipeline that silently discards what it cannot parse has no way to tell you it discarded something.</p><p>synonyms()<strong> is here as a control.</strong> It pulls WordNet lemmas for the key, and it never enters the scoring. Watch what it returns for colour when we run it: emblazon, gloss, people of colour. That is surface variation, not domain meaning. It is in the code so you can see, concretely, what the dictionary approach has to offer — and then watch structure do the work instead.</p><pre>import re, nltk<br>from nltk.stem import WordNetLemmatizer<br>from nltk.corpus import wordnet as wn<br># one-time, conda users may already have these<br>for pkg in (&quot;punkt&quot;, &quot;punkt_tab&quot;, &quot;wordnet&quot;, &quot;omw-1.4&quot;):<br>    nltk.download(pkg, quiet=True)<br>LEM = WordNetLemmatizer()<br>STOP = {&quot;the&quot;,&quot;a&quot;,&quot;an&quot;,&quot;of&quot;,&quot;for&quot;,&quot;and&quot;,&quot;is&quot;,&quot;are&quot;,&quot;this&quot;,&quot;that&quot;,<br>        &quot;please&quot;,&quot;hi&quot;,&quot;hello&quot;,&quot;dear&quot;,&quot;thanks&quot;,&quot;regards&quot;,&quot;attached&quot;,&quot;find&quot;}<br># Two shapes of field line, and nothing more:<br>#   PAIR  -&gt;  &quot;Colour: black&quot;   or  &quot;Colour = black&quot;<br>#   DASH  -&gt;  &quot;Nett wt - 780 g&quot;<br># Anything that matches neither is kept as leftover, never discarded.<br>PAIR = re.compile(r&quot;^\s*([A-Za-z][A-Za-z0-9 ._/&amp;-]{0,40}?)\s*[:=]\s*(.+?)\s*$&quot;)<br>DASH = re.compile(r&quot;^\s*([A-Za-z][A-Za-z0-9 ._/&amp;]{0,40}?)\s+-\s+(.+?)\s*$&quot;)<br><br># Lemmatize, not stem: &quot;Numbers&quot; -&gt; &quot;number&quot;, so a plural key and a<br># singular property name collapse to the same string. This is the entire<br># linguistic contribution of the pipeline.<br>def normalize(phrase):<br>    &quot;&quot;&quot;Lowercase, strip punctuation, lemmatize, drop stopwords.&quot;&quot;&quot;<br>    toks = re.findall(r&quot;[a-z0-9]+&quot;, phrase.lower())<br>    return &quot; &quot;.join(LEM.lemmatize(t) for t in toks if t not in STOP)<br><br># The control condition. WordNet gives surface variants of a word, which is<br># not the same thing as domain meaning - it has never heard of GS1. Computed<br># and displayed so you can see what it offers, then never scored on.<br>def synonyms(phrase):<br>    &quot;&quot;&quot;WordNet lemmas - surface variation, NOT domain meaning.&quot;&quot;&quot;<br>    out = set()<br>    for syn in wn.synsets(phrase.replace(&quot; &quot;, &quot;_&quot;)):<br>        for l in syn.lemmas():<br>            out.add(l.name().replace(&quot;_&quot;, &quot; &quot;).lower())<br>    return out<br><br>def mentions(text):<br>    &quot;&quot;&quot;[{raw_key, key, value, synonyms}] plus lines that aren&#39;t field-shaped.&quot;&quot;&quot;<br>    found, leftover = [], []<br>    for line in text.splitlines():<br>        if not line.strip():<br>            continue<br>        m = PAIR.match(line) or DASH.match(line)<br>        if not m:<br>            leftover.append(line.strip()); continue<br>        raw_key, value = m.group(1).strip(), m.group(2).strip().rstrip(&quot;.,;&quot;)<br>        key = normalize(raw_key)<br>        if not key:<br>            leftover.append(line.strip()); continue<br>        found.append({&quot;raw_key&quot;: raw_key, &quot;key&quot;: key, &quot;value&quot;: value,<br>                      &quot;synonyms&quot;: synonyms(key)})<br>    return found, leftover</pre><p>normalize lowercases, strips punctuation, lemmatizes each token and drops stopwords, so Article Numbers and article number collapse to the same key. That is the entire linguistic contribution of this pipeline.</p><p>Now run it on a message of the kind that actually arrives.</p><pre>EMAIL = &quot;&quot;&quot;<br>Hi team,<br>Please add these to the catalogue.<br>Article Numbers : 8901234567892<br>EU Sizes: 42<br>Nett wt - 780 g<br>Colour: black<br>Regards, John<br>&quot;&quot;&quot;<br>ms, rest = mentions(EMAIL)<br>for m in ms:<br>    print(f&quot;{m[&#39;raw_key&#39;]:18s} -&gt; {m[&#39;key&#39;]:16s} = {m[&#39;value&#39;]}&quot;)<br>print(&quot;not field-shaped:&quot;, rest)</pre><pre>Article Numbers    -&gt; article number   = 8901234567892<br>EU Sizes           -&gt; eu size          = 42<br>Nett wt            -&gt; nett wt          = 780 g<br>Colour             -&gt; colour           = black<br>not field-shaped: [&#39;Hi team,&#39;, &#39;Please add these to the catalogue.&#39;, &#39;Regards, John&#39;]</pre><p>Four field-shaped lines found, three lines correctly set aside.</p><p>Look at what the keys became: article number, eu size, nett wt, colour. Not one of them is a GS1 property name. nett wt is not even a word. No dictionary contains it, no synonym list will reach it, and a fuzzy match against 557 property names will return something plausible for every single one of these — which is the problem, not the solution.</p><h3>Proposing candidates</h3><p>Two independent kinds of evidence, and it matters that they are kept apart.</p><p><strong>Key evidence</strong> is lexical similarity between the normalized key and the property’s name and label — half character-level ratio, half token overlap. This is a <em>proposal</em> mechanism and nothing more. It has no idea what anything means and will happily rank a wrong property first, which it is about to do twice.</p><p><strong>Value evidence</strong> is what the value proves regardless of what the human called it. 8901234567892 is thirteen digits whose last one is exactly the GS1 check digit for the first twelve. That is not a resemblance, it is arithmetic, so gtin scores 1.0 on a line whose key never says &quot;GTIN&quot;. A boolean-shaped value scores weakly, because looking boolean is suggestive and nothing more.</p><p>Everything below the threshold is discarded; the top few survive per line.</p><pre>import re<br>from difflib import SequenceMatcher<br>from rdflib import Graph, RDFS, RDF, OWL<br><br>def load_props(ttl):<br>    &quot;&quot;&quot;Read properties AND their domains out of the ontology.&quot;&quot;&quot;<br>    g = Graph(); g.parse(ttl, format=&quot;turtle&quot;)<br>    props = {}<br>    for p in set(g.subjects(RDF.type, OWL.DatatypeProperty)) | \<br>             set(g.subjects(RDF.type, OWL.ObjectProperty)):<br>        local = str(p).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br>        lbl = next((str(o) for o in g.objects(p, RDFS.label)), None)<br>        dom = next((str(o).rsplit(&quot;/&quot;, 1)[-1] for o in g.objects(p, RDFS.domain)), None)<br>        props[local] = {&quot;label&quot;: lbl or local, &quot;domain&quot;: dom}<br>    return props<br><br># GS1 property names are camelCase; message keys are prose. Splitting on the<br># case boundary puts both into the same shape before comparing.<br>def split_camel(s):<br>    return re.sub(r&quot;(?&lt;!^)(?=[A-Z])&quot;, &quot; &quot;, s).lower()<br><br># Half character-level similarity, half token overlap. Character ratio alone<br># rewards coincidental spelling; token overlap alone misses &quot;colour&quot;/&quot;color&quot;.<br># This is a PROPOSAL score - it never decides anything on its own.<br>def key_score(key, prop, meta):<br>    &quot;&quot;&quot;Lexical similarity to the property&#39;s own name and label.&quot;&quot;&quot;<br>    best = 0.0<br>    for s in {split_camel(prop), (meta[&quot;label&quot;] or &quot;&quot;).lower()}:<br>        s_norm = normalize(s) or s<br>        char = SequenceMatcher(None, key, s_norm).ratio()<br>        kt, st = set(key.split()), set(s_norm.split())<br>        token = len(kt &amp; st) / len(kt | st) if kt | st else 0<br>        best = max(best, 0.5 * char + 0.5 * token)<br>    return best<br><br># The GS1 check digit: weight the first 12 digits 1,3,1,3..., sum, and the<br># 13th digit must complete the total to a multiple of 10. This is arithmetic,<br># not resemblance - it either holds or it doesn&#39;t.<br>def gtin_ok(s):<br>    d = re.sub(r&quot;\D&quot;, &quot;&quot;, s)<br>    if len(d) != 13:<br>        return False<br>    tot = sum(int(c) * (3 if i % 2 else 1) for i, c in enumerate(reversed(d[:12])))<br>    return int(d[-1]) == (10 - tot % 10) % 10<br><br>BOOL = {&quot;yes&quot;: True, &quot;no&quot;: False, &quot;true&quot;: True, &quot;false&quot;: False}<br><br># What the VALUE proves, independent of what the human called the field.<br># A passing check digit is proof (1.0). Looking boolean is a hint (0.5).<br>def value_evidence(value):<br>    &quot;&quot;&quot;What the VALUE points to, independent of whatever the human called it.&quot;&quot;&quot;<br>    hits = {}<br>    if gtin_ok(value):<br>        hits[&quot;gtin&quot;] = 1.0                # check digit = proof<br>    if value.strip().lower() in BOOL:<br>        hits[&quot;isSeedless&quot;] = 0.5          # boolean-shaped, weak<br>    return hits<br><br>def candidates(text, ttl, threshold=0.45, top=4):<br>    props = load_props(ttl)<br>    ms, leftover = mentions(text)<br>    out = []<br>    for m in ms:<br>        scored = {}<br>        for p, meta in props.items():<br>            s = key_score(m[&quot;key&quot;], p, meta)<br>            if s &gt;= threshold:<br>                scored[p] = (&quot;key&quot;, round(s, 2))<br>        for p, s in value_evidence(m[&quot;value&quot;]).items():<br>            if p not in scored or s &gt; scored[p][1]:<br>                scored[p] = (&quot;value&quot;, s)<br>        ranked = sorted(scored.items(), key=lambda kv: -kv[1][1])[:top]<br>        out.append({**m, &quot;candidates&quot;: ranked,<br>                    &quot;domains&quot;: {p: props[p][&quot;domain&quot;] for p, _ in ranked}})<br>    return out, leftover</pre><p>One thing worth noticing in load_props: a property can declare more than one domain, and this code keeps the first one it happens to see. GS1&#39;s gtin, for instance, declares both gs1:Product and schema:Product — an intersection, not a choice. It does not change the outcome here, but it is the kind of shortcut that quietly loses information, and it deserves a comment rather than silence.</p><p>Now the candidates for our four lines.</p><pre>out, leftover = candidates(EMAIL, &quot;gs1Voc.ttl&quot;)<br>for m in out:<br>    print(f&quot;{m}&quot;)<br>    #print(f&quot;{m[&#39;raw_key&#39;]:18s} -&gt; {m[&#39;key&#39;]:16s} = {m[&#39;value&#39;]}&quot;)<br>print(&quot;not field-shaped:&quot;, rest)</pre><pre>{&#39;raw_key&#39;: &#39;Article Numbers&#39;, &#39;key&#39;: &#39;article number&#39;, &#39;value&#39;: &#39;8901234567892&#39;, &#39;synonyms&#39;: set(), &#39;candidates&#39;: [(&#39;gtin&#39;, (&#39;value&#39;, 1.0)), (&#39;hasSerialNumber&#39;, (&#39;key&#39;, 0.5)), (&#39;faxNumber&#39;, (&#39;key&#39;, 0.5))], &#39;domains&#39;: {&#39;gtin&#39;: &#39;Product&#39;, &#39;hasSerialNumber&#39;: &#39;Product&#39;, &#39;faxNumber&#39;: &#39;ContactPoint&#39;}}<br>{&#39;raw_key&#39;: &#39;EU Sizes&#39;, &#39;key&#39;: &#39;eu size&#39;, &#39;value&#39;: &#39;42&#39;, &#39;synonyms&#39;: set(), &#39;candidates&#39;: [(&#39;size&#39;, (&#39;key&#39;, 0.61)), (&#39;servingSize&#39;, (&#39;key&#39;, 0.48))], &#39;domains&#39;: {&#39;size&#39;: &#39;WearableProduct&#39;, &#39;servingSize&#39;: &#39;FoodBeverageTobaccoProduct&#39;}}<br>{&#39;raw_key&#39;: &#39;Nett wt&#39;, &#39;key&#39;: &#39;nett wt&#39;, &#39;value&#39;: &#39;780 g&#39;, &#39;synonyms&#39;: set(), &#39;candidates&#39;: [], &#39;domains&#39;: {}}<br>{&#39;raw_key&#39;: &#39;Colour&#39;, &#39;key&#39;: &#39;colour&#39;, &#39;value&#39;: &#39;black&#39;, &#39;synonyms&#39;: {&#39;colorize&#39;, &#39;colouring&#39;, &#39;distort&#39;, &#39;colourize&#39;, &#39;vividness&#39;, &#39;semblance&#39;, &#39;people of colour&#39;, &#39;coloring&#39;, &#39;colouring material&#39;, &#39;coloration&#39;, &#39;colouration&#39;, &#39;colorise&#39;, &#39;colour&#39;, &#39;emblazon&#39;, &#39;discolour&#39;, &#39;color in&#39;, &#39;colour in&#39;, &#39;coloring material&#39;, &#39;discolor&#39;, &#39;colourise&#39;, &#39;tinge&#39;, &#39;color&#39;, &#39;people of color&#39;, &#39;gloss&#39;}, &#39;candidates&#39;: [(&#39;colourCodeValue&#39;, (&#39;key&#39;, 0.6)), (&#39;colourCode&#39;, (&#39;key&#39;, 0.6)), (&#39;colourDescription&#39;, (&#39;key&#39;, 0.5))], &#39;domains&#39;: {&#39;colourCodeValue&#39;: &#39;ColourCodeDetails&#39;, &#39;colourCode&#39;: &#39;Product&#39;, &#39;colourDescription&#39;: &#39;Product&#39;}}<br>not field-shaped: [&#39;Hi team,&#39;, &#39;Please add these to the catalogue.&#39;, &#39;Regards, John&#39;]</pre><p>This is the honest starting position, and it is a mess.</p><p>Article Numbers proposes gtin (on value evidence, 1.0), hasSerialNumber and faxNumber. Colour proposes colourCodeValue, colourCode and colourDescription. EU Sizes proposes size and servingSize. And Nett wt proposes nothing at all — no property name in GS1 is lexically close enough to nett wt to clear the threshold.</p><p>Two of these are plainly wrong, and notice that the scores do not know it. faxNumber scores exactly as well as hasSerialNumber, because &quot;Article Numbers&quot; and &quot;Fax Number&quot; genuinely do share a token. colourCodeValue scores <em>highest</em> of the three colour candidates — it is, lexically, the best match to &quot;Colour&quot; in the entire vocabulary.</p><p>A synonym dictionary would stop here and pick those. The next step is where they die, and nothing lexical is involved.</p><h3>Coherence: what the structure eliminates</h3><p>Here is the move the whole post has been building toward.</p><p>Every mention now has a shortlist. Take the cartesian product of those shortlists — every possible reading of the whole message at once, including the reading where a line goes unmatched — and throw away any reading that would require a single record to be two incompatible things.</p><p>“Incompatible” has a precise meaning, and it is the only fact this uses: compatible(a, b) asks whether one class lies on the other&#39;s ancestor chain. If it does, one individual could be both. If it doesn&#39;t, they are different kinds of thing and no record is both at once. That is ancestors(), from the section above, doing all the work.</p><p>Then keep the readings that explain the most mentions, because a reading that leaves everything unmatched is trivially coherent and useless.</p><pre>from itertools import product as cartesian<br>from rdflib import Graph, RDFS, RDF, OWL<br>_TAX = {}<br><br>def taxonomy(ttl):<br>    &quot;&quot;&quot;ancestors[class] = {itself + every superclass}. Cached - GS1 is big.&quot;&quot;&quot;<br>    if ttl in _TAX:<br>        return _TAX[ttl]<br>    g = Graph(); g.parse(ttl, format=&quot;turtle&quot;)<br>    local = lambda u: str(u).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br>    parent, classes = {}, set()<br>    for s, _, o in g.triples((None, RDFS.subClassOf, None)):<br>        parent.setdefault(local(s), set()).add(local(o))<br>        classes |= {local(s), local(o)}<br>    for t in (RDFS.Class, OWL.Class):<br>        classes |= {local(c) for c in g.subjects(RDF.type, t)}<br>    anc = {}<br>    def walk(c, seen):<br>        if c in anc: return anc[c]<br>        if c in seen: return {c}                    # cycle guard<br>        seen = seen | {c}<br>        out = {c}<br>        for p in parent.get(c, ()):<br>            out |= walk(p, seen)<br>        anc[c] = out<br>        return out<br>    for c in classes:<br>        walk(c, frozenset())<br>    _TAX[ttl] = anc<br>    return anc<br><br># The only semantic test in the pipeline. Two classes are compatible when one<br># is an ancestor of the other, because then a single individual could belong<br># to both. Off each other&#39;s chains means two different kinds of thing.<br>def compatible(a, b, anc):<br>    &quot;&quot;&quot;Can ONE individual be both? Only if they lie on a common chain.&quot;&quot;&quot;<br>    if a is None or b is None: return True<br>    return b in anc.get(a, {a}) or a in anc.get(b, {b})<br><br># Enumerate every reading of the whole message at once - the cartesian product<br># of the per-line shortlists, with (None, None) allowing a line to go<br># unmatched - then discard any reading that needs one record to be two<br># incompatible things. Whatever survives is what the message actually permits.<br>def resolve(text, ttl, top=3, max_combos=50_000):<br>    ms, leftover = candidates(text, ttl)<br>    anc = taxonomy(ttl)<br>    for m in ms:<br>        m[&quot;candidates&quot;] = m[&quot;candidates&quot;][:top]     # cap: GS1 gives many<br>    options = [[(p, m[&quot;domains&quot;].get(p)) for p, _ in m[&quot;candidates&quot;]] + [(None, None)]<br>               for m in ms]<br>    viable, n = [], 0<br>    for combo in cartesian(*options):<br>        n += 1<br>        if n &gt; max_combos: break<br>        chosen = [(p, d) for p, d in combo if p]<br>        if not chosen: continue<br>        doms = [d for _, d in chosen if d]<br>        if all(compatible(a, b, anc)<br>               for i, a in enumerate(doms) for b in doms[i+1:]):<br>            viable.append(combo)<br>    if viable:                                       # explain the most mentions<br>        best = max(sum(1 for p, _ in c if p) for c in viable)<br>        viable = [c for c in viable if sum(1 for p, _ in c if p) == best]<br>    out = []<br>    for i, m in enumerate(ms):<br>        survivors = {c[i][0] for c in viable if c[i][0]}<br>        killed = [{&quot;property&quot;: p, &quot;implies&quot;: m[&quot;domains&quot;].get(p),<br>                   &quot;reason&quot;: &quot;incoherent with the rest of the message&quot;}<br>                  for p in {p for p, _ in m[&quot;candidates&quot;]} - survivors]<br>        status = (&quot;RESOLVED&quot; if len(survivors) == 1<br>                  else &quot;AMBIGUOUS&quot; if survivors else &quot;UNRESOLVED&quot;)<br>        out.append({&quot;raw_key&quot;: m[&quot;raw_key&quot;], &quot;value&quot;: m[&quot;value&quot;],<br>                    &quot;status&quot;: status, &quot;options&quot;: sorted(survivors),<br>                    &quot;implies&quot;: sorted({m[&quot;domains&quot;].get(p) for p in survivors<br>                                       if m[&quot;domains&quot;].get(p)}),<br>                    &quot;killed&quot;: killed})<br>    types = sorted({d for c in viable for _, d in c if d},<br>                   key=lambda d: -len(anc.get(d, {d})))<br>    # Record type per READING, not pooled across readings. If the readings<br>    # disagree, say so - do not silently pick the most specific.<br>    per_reading = set()<br>    for c in viable:<br>        doms = [d for _, d in c if d]<br>        if doms:<br>            per_reading.add(max(doms, key=lambda d: len(anc.get(d, {d}))))<br>    if len(per_reading) == 1:<br>        rec_type, rec_status = per_reading.pop(), &quot;RESOLVED&quot;<br>    elif per_reading:<br>        rec_type, rec_status = sorted(per_reading), &quot;AMBIGUOUS&quot;<br>    else:<br>        rec_type, rec_status = None, &quot;UNRESOLVED&quot;<br>    for o in out:<br>        o[&quot;record_type&quot;] = rec_type<br>        o[&quot;record_type_status&quot;] = rec_status<br>    return out, leftover<br></pre><p>Two candidates die on the real vocabulary, and the second is the one worth the price of the post.</p><p>faxNumber implies ContactPoint. Every other reading of this message puts the record under Product, and a product is not a contact point, so any reading containing faxNumber is discarded whole. Wrong concept, killed structurally.</p><p>colourCodeValue implies ColourCodeDetails. This one is subtler: the concept is right. ColourCodeDetails is a GS1 container class that groups a colour code with its issuing agency — it is genuinely about colour. It is simply not the <em>product</em>. The record cannot be both a product and a code-details structure, so it goes.</p><p>Right concept, wrong level, and lexically the single best match in the file. No dictionary could have caught that. The map caught it without being asked.</p><p><strong>The record type has to be derived per reading, not pooled across readings.</strong> Pooling looks reasonable, produces a plausible answer, and no test catches it, because there is nothing to compare against — it would simply report the most specific branch it saw anywhere and sound confident. That is the same failure this pipeline exists to prevent, reappearing one layer down. A three-state contract is only honest if every layer respects it.</p><h3>Which lines could decide</h3><p>Before looking at the verdict, ask the vocabulary a different question — one that needs no message at all.</p><p>Which properties have a domain sitting under exactly one branch? Those are the lines that, if a message happened to contain them, would settle the question. They can be listed in advance, from the vocabulary alone, which means you can tell a supplier <em>which field to add</em> rather than waiting to see what they send.</p><pre>from rdflib import Graph, RDFS, RDF, OWL<br>def discriminators(ttl, branches):<br>    &quot;&quot;&quot;Properties whose domain sits under exactly ONE branch.<br>    These are the lines in a message that can settle the question.<br>    Run it on your real vocabulary to find which ones exist.<br>    &quot;&quot;&quot;<br>    g = Graph(); g.parse(ttl, format=&quot;turtle&quot;)<br>    anc = taxonomy(ttl)<br>    local = lambda u: str(u).rsplit(&quot;/&quot;, 1)[-1].rsplit(&quot;#&quot;, 1)[-1]<br>    # A property discriminates when its domain sits under exactly ONE of the<br>    # branches. Those are the lines that could settle the question - and they<br>    # are knowable from the vocabulary alone, before any message arrives.<br>    out = {b: [] for b in branches}<br>    for p in set(g.subjects(RDF.type, OWL.DatatypeProperty)) | \<br>             set(g.subjects(RDF.type, OWL.ObjectProperty)):<br>        dom = next((local(o) for o in g.objects(p, RDFS.domain)), None)<br>        if not dom: continue<br>        hits = [b for b in branches if b in anc.get(dom, {dom})]<br>        if len(hits) == 1:<br>            lbl = next((str(o) for o in g.objects(p, RDFS.label)), local(p))<br>            out[hits[0]].append((local(p), lbl, dom))<br>    return out<br><br>discriminators(&quot;gs1Voc.ttl&quot;, [&quot;WearableProduct&quot;, &quot;FoodBeverageTobaccoProduct&quot;])</pre><pre>{&#39;WearableProduct&#39;: [(&#39;isThermal&#39;, &#39;Is Thermal&#39;, &#39;WearableProduct&#39;),<br>  (&#39;isMaternity&#39;, &#39;Is Maternity&#39;, &#39;Clothing&#39;),<br>  (&#39;clothingCut&#39;, &#39;Clothing Cut&#39;, &#39;Clothing&#39;),<br>  (&#39;consumerLifestage&#39;, &#39;Consumer Lifestage&#39;, &#39;WearableProduct&#39;),<br>  (&#39;isWaterproof&#39;, &#39;Is Waterproof&#39;, &#39;WearableProduct&#39;),<br>  (&#39;sportingActivityType&#39;, &#39;Sporting Activity Type&#39;, &#39;WearableProduct&#39;),<br>  (&#39;collarType&#39;, &#39;Collar Type&#39;, &#39;Clothing&#39;),<br>  (&#39;footwearUpperType&#39;, &#39;Footwear Upper Type&#39;, &#39;Footwear&#39;),<br>  (&#39;seasonCalendarYear&#39;, &#39;Season Calendar Year&#39;, &#39;WearableProduct&#39;),<br>  (&#39;targetConsumerGender&#39;, &#39;Target Consumer Gender&#39;, &#39;WearableProduct&#39;),<br>  (&#39;isPatterned&#39;, &#39;Is Patterned&#39;, &#39;WearableProduct&#39;),<br>  (&#39;seasonParameter&#39;, &#39;Season Parameter&#39;, &#39;WearableProduct&#39;),<br>  (&#39;size&#39;, &#39;Size&#39;, &#39;WearableProduct&#39;),<br>  (&#39;isWearableItemDisposable&#39;,<br>   &#39;Is Wearable Item Disposable&#39;,<br>   &#39;WearableProduct&#39;),<br>  (&#39;targetConsumerAge&#39;, &#39;Target Consumer Age&#39;, &#39;WearableProduct&#39;),<br>  (&#39;textileMaterialContent&#39;, &#39;Textile Material Content&#39;, &#39;WearableProduct&#39;),<br>  (&#39;footwearFasteningType&#39;, &#39;Fastening Type&#39;, &#39;Footwear&#39;),<br>  (&#39;styleDescription&#39;, &#39;Style Description&#39;, &#39;WearableProduct&#39;),<br>  (&#39;seasonName&#39;, &#39;Season Name&#39;, &#39;WearableProduct&#39;),<br>  (&#39;textileMaterial&#39;, &#39;Has Textile Material&#39;, &#39;Clothing&#39;),<br>  (&#39;upperMaterialType&#39;, &#39;Has Upper Material&#39;, &#39;Footwear&#39;)],<br> &#39;FoodBeverageTobaccoProduct&#39;: [(&#39;pantothenicAcidPerNutrientBasis&#39;,<br>   &#39;Pantothenic Acid Per Nutrient Basis Quantity&#39;,<br>...</pre><p>upperMaterialType (&quot;Has Upper Material&quot;) has domain Footwear. servingSize requires FoodBeverageTobaccoProduct. The two branches are disjoint in practice, so a message containing the first cannot coherently contain the second.</p><p>That is the payoff in one line: add Upper material: leather to the email and EU Sizes: 42 resolves to size — not because anything learned that leather relates to shoes, but because servingSize became an impossible route. A line that never mentions size decides what the size line means.</p><h3>The result</h3><p>Four lines in, four verdicts out — plus the eliminations, each carrying the class that killed it.</p><pre>out, leftover = resolve(EMAIL, &quot;gs1Voc.ttl&quot;)<br>rt, rs = out[0][&quot;record_type&quot;], out[0][&quot;record_type_status&quot;]<br>print(f&quot;record type : {rt}  [{rs}]\n&quot;)<br>for m in out:<br>    print(f&quot;{m[&#39;raw_key&#39;]:16s} {m[&#39;status&#39;]:11s} {m[&#39;options&#39;]}&quot;)<br>    for k in m[&quot;killed&quot;]:<br>        print(f&quot;{&#39;&#39;:16s}   killed {k[&#39;property&#39;]} (implies {k[&#39;implies&#39;]})&quot;)<br>print(&quot;\nnot field-shaped:&quot;, leftover)</pre><pre>record type : [&#39;FoodBeverageTobaccoProduct&#39;, &#39;WearableProduct&#39;]  [AMBIGUOUS]<br><br>Article Numbers  AMBIGUOUS   [&#39;gtin&#39;, &#39;hasSerialNumber&#39;]<br>                   killed faxNumber (implies ContactPoint)<br>EU Sizes         AMBIGUOUS   [&#39;servingSize&#39;, &#39;size&#39;]<br>Nett wt          UNRESOLVED  []<br>Colour           AMBIGUOUS   [&#39;colourCode&#39;, &#39;colourDescription&#39;]<br>                   killed colourCodeValue (implies ColourCodeDetails)<br><br>not field-shaped: [&#39;Hi team,&#39;, &#39;Please add these to the catalogue.&#39;, &#39;Regards, John&#39;]</pre><p>Read it honestly.</p><p>Article Numbers is AMBIGUOUS between gtin and hasSerialNumber. Both have domain Product, so coherence has nothing to separate them — they are equally compatible with everything else in the message. Colour is AMBIGUOUS between colourCode and colourDescription for the same reason.</p><p>Nett wt is UNRESOLVED. The pipeline says so rather than attaching it to the nearest-sounding property.</p><p>And the record type is AMBIGUOUS across WearableProduct and FoodBeverageTobaccoProduct, because &quot;Size: 42, Colour: black, barcode 8901234567892&quot; genuinely describes both a shoe and a food product. The message does not say. The system does not pretend otherwise.</p><h3>Making a bad mapping impossible</h3><p>Everything so far has been the resolver’s own arithmetic — my compatible() function walking ancestor chains in Python. That works, but it is my code deciding what the vocabulary means.</p><p>A reasoner does the same job from the axioms themselves, and it can do something compatible() cannot: reject an assertion outright. Not score it low. Reject it, because holding it would make the ontology contradict itself.</p><p>Below is the same ambiguity the resolver reported for EU Sizes: 42 — a record that could be a wearable product or a food product — handed to HermiT instead. Twice: once with the vocabulary exactly as GS1 ships it, and once with a single axiom added.</p><pre>from owlready2 import (Thing, AllDisjoint, sync_reasoner,<br>                       OwlReadyInconsistentOntologyError)<br># An &quot;incoming record&quot; is just an individual: a named thing with claims on it.<br># is_a is the individual&#39;s asserted types - here, deliberately, BOTH of the<br># branches the resolver could not separate.<br>record = Thing(&quot;incoming_record&quot;, namespace = acme)<br>record.is_a = [onto.WearableProduct, onto.FoodBeverageTobaccoProduct]<br># sync_reasoner() starts a JVM, runs HermiT over the whole world, and writes<br># whatever it derives back into the Python objects. It raises only when the<br># ontology is *inconsistent* - when no possible world satisfies every axiom.<br>try:<br>    with onto:<br>        sync_reasoner(debug = 0)<br>    print(&quot;accepted - nothing in GS1 forbids a wearable food product&quot;)<br>except OwlReadyInconsistentOntologyError:<br>    print(&quot;rejected&quot;)</pre><p>Accepted. The record claims to be a shoe and a snack simultaneously, and the reasoner has no objection — because GS1 never says those two branches are exclusive. Recall the count from earlier: the vocabulary ships <strong>zero</strong> owl:disjointWith axioms. It was written to describe what exists, not to forbid what doesn&#39;t.</p><p>So state the missing fact. One axiom.</p><pre># AllDisjoint asserts that no individual may belong to more than one of these<br># classes. This is stronger than &quot;they are different classes&quot; - it says the<br># intersection is necessarily empty.<br>with onto:<br>    AllDisjoint([onto.WearableProduct, onto.FoodBeverageTobaccoProduct])<br># Same record, same reasoner, one more axiom in the world.<br>try:<br>    with onto:<br>        sync_reasoner(debug = 0)<br>    print(&quot;accepted&quot;)<br>except OwlReadyInconsistentOntologyError:<br>    print(&quot;rejected - the record cannot be both, and the reasoner proved it&quot;)</pre><p>Rejected.</p><p>Look at what did <em>not</em> happen. Nothing compared strings. Nothing scored a similarity. No threshold was tuned, no human reviewed it, and no model was asked for an opinion. The record was refused because holding it, together with the axioms already present, is a contradiction — and that verdict is deterministic. Run it a thousand times and it rejects a thousand times, for a reason you can print.</p><p>That is a different kind of answer from anything earlier . The resolver could say <em>these two readings both survive</em>. The reasoner can say <em>this claim is impossible</em>.</p><p><strong>And the honest caveat, which is the point of the whole post.</strong> That veto exists only because we asserted the disjointness ourselves. GS1 gave us the hierarchy, the domains and the labels — a rich, carefully maintained description of what products are. It did not give us a single statement about what a product cannot be. Published vocabularies almost never do.</p><p>So the ontology is not the file you download. It is the file you download plus the constraints you add to it, and the constraints are where the enforcement lives. One line of AllDisjoint bought a structural veto that no amount of matching, scoring or prompting could have produced.</p><h3>What survives, and the two ways out</h3><p>What is left is not failure. It is the message being genuinely underdetermined, reported as such. Three mechanisms reduce it, and they do different jobs.</p><p><strong>Coherence resolves cross-branch ambiguity.</strong> One more line — Upper material: leather or Best before: 2026-11-30 — and a branch dies. The EU Sizes: 42 line does not change by one character; it resolves to size in one message and servingSize in the other, because of a line that never mentions size. That is meaning derived from structure rather than vocabulary, and it is the thing no dictionary can do.</p><p><strong>Value evidence resolves within-branch ambiguity.</strong> gtin versus hasSerialNumber, and colourCode versus colourDescription, share a domain, so no amount of extra context in this message will separate them. The check digit already proves gtin; a value-shape rule could separate a code from a description. Different mechanism, different layer.</p><p><strong>Constraints make contradictions rejectable.</strong> Asserting owl:disjointWith between WearableProduct and FoodBeverageTobaccoProduct does not resolve this message — it stays ambiguous, correctly — but it turns a record claiming both into an error rather than a shrug. GS1 ships zero disjointness axioms, so this one is on you, which is the point made earlier: the ontology is not the vocabulary you download, it is the constraints you add to it.</p><p>Whatever survives all three is genuinely not in the message. That is where a model gets invited in — to propose, with the contract still holding the veto.</p><h3>References</h3><blockquote><em>Guha, R. V., Brickley, D., Macbeth, S. (2016). </em>Schema.org: Evolution of Structured Data on the Web.<em> Communications of the ACM 59(2). → </em><a href="https://cacm.acm.org/practice/schema-org/"><em>cacm.acm.org/practice/schema-org/</em></a></blockquote><blockquote>schema.org — current version 30.0 (25 March 2026).</blockquote><blockquote>GS1 Web Vocabulary. → <a href="https://ref.gs1.org/voc/">ref.gs1.org/voc/</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f02b74a9a6c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Scalable Graph Processing Pipeline with PySpark and Neo4j]]></title>
            <link>https://medium.com/@abhi1993nair/building-a-scalable-graph-processing-pipeline-with-pyspark-and-neo4j-4f736d0fd917?source=rss-308382bf640e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f736d0fd917</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[cypher]]></category>
            <dc:creator><![CDATA[Abhijith Nair]]></dc:creator>
            <pubDate>Thu, 03 Apr 2025 08:30:33 GMT</pubDate>
            <atom:updated>2025-04-03T08:30:33.686Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rigidity Meets Flexibility</h3><p>In today’s data-driven world, enterprises often need to process massive datasets efficiently while preserving relationships between entities. When dealing with complex, interconnected data graph databases like Neo4j offer a powerful solution.</p><p><strong>Graph databases embrace flexibility</strong>, allowing dynamic, evolving relationships without rigid constraints. Instead of relying on costly joins, they connect data through <strong>nodes and edges</strong>, making them ideal for <strong>social networks, recommendation engines, and fraud detection</strong>.</p><p>In this blog post, we will build a scalable data pipeline using Apache Spark and Neo4j to process ArXiv research paper metadata. The goal is to clean, transform, and load data into Neo4j, allowing us to efficiently query relationships among papers, authors, and categories.</p><figure><img alt="©Neo4j" src="https://cdn-images-1.medium.com/max/678/1*8P6haWq2p2vGFFyPU2ogWQ.png" /><figcaption>©Neo4j</figcaption></figure><h3>Neo4j: A Powerful Graph Database</h3><p>Neo4j is a <strong>graph database management system</strong> designed to handle highly interconnected data efficiently. Unlike traditional relational databases that rely on tables and joins, Neo4j represents data as <strong>nodes</strong> (entities) and <strong>relationships</strong> (connections), making it ideal for scenarios where relationships are as important as the data itself.</p><h4>Key Features</h4><ol><li><strong>Graph-Based Storage</strong> — Neo4j stores data natively as a graph, ensuring fast traversal and query performance.</li><li><strong>Cypher Query Language (CQL)</strong> — A powerful, SQL-like language optimized for graph operations.</li><li><strong>Schema Flexibility</strong> — Unlike rigid relational schemas, Neo4j allows dynamic and evolving data structures.</li><li><strong>Scalability</strong> — It supports horizontal scaling and high availability, making it enterprise-ready.</li><li><strong>ACID Compliance</strong> — Ensures consistency and reliability in transactions.</li></ol><h4>Use Cases</h4><ul><li><strong>Knowledge Graphs</strong>: Organizes and connects vast amounts of information.</li><li><strong>Recommendation Engines</strong>: Suggests products, movies, or friends based on user behavior.</li><li><strong>Social Networks</strong>: Tracks relationships between users, posts, likes, and comments.</li><li><strong>Fraud Detection</strong>: Identifies fraudulent patterns in financial transactions.</li><li><strong>Supply Chain Management</strong>: Optimizes logistics and supplier relationships.</li></ul><h4>Why Use Neo4j?</h4><p>Traditional databases struggle with complex relationships, requiring expensive joins that slow performance. <em>Neo4j, however, traverses relationships </em><strong><em>in constant time</em></strong><em>, regardless of dataset size.</em> This makes it perfect for applications where <strong>connections drive value</strong> — such as LinkedIn’s professional network or Google’s Knowledge Graph.</p><p>With its speed, flexibility, and relationship-first approach, Neo4j is redefining how we store and query connected data. Whether analyzing social networks, detecting fraud, or powering AI-driven insights, Neo4j offers a <strong>smarter, faster way</strong> to manage graph data.</p><h3>Few Types of Graph Models:</h3><h4>1. Property Graph Model (Core Model in Neo4j)</h4><p><strong>Definition</strong>: A graph with <strong>nodes, relationships, and properties (key-value pairs).</strong></p><p><strong>Structure</strong>:</p><ul><li><strong>Nodes</strong>: Represent entities (e.g., Person, Company).</li><li><strong>Edges (Relationships)</strong>: Define connections between nodes.</li><li><strong>Properties</strong>: Attached to nodes and relationships.</li></ul><p><strong>Use Cases</strong>:</p><ul><li>Social networks, fraud detection, recommendation systems.</li></ul><p><strong>Why Use It?</strong>: <strong>Efficient and flexible</strong> for real-world graph applications.</p><h4>2. Labeled Property Graph Model (LPG)</h4><p><strong>Definition</strong>: An <strong>extension</strong> of the Property Graph Model where nodes and relationships have <strong>labels</strong>.</p><p><strong>Structure</strong>:</p><ul><li>Nodes can have <strong>multiple labels</strong> (e.g., :Person and :Employee).</li></ul><p><strong>Use Cases</strong>:</p><ul><li>Hierarchical or category-based queries (e.g., a :Vehicle might also be a :Car or :Bike).</li></ul><pre>CREATE (a:Person:Employee {name: &quot;Alice&quot;, role: &quot;Manager&quot;})</pre><ul><li>Alice is both a Person and an Employee.</li></ul><p><strong>Why Use It?</strong>: Helps in <strong>categorizing entities</strong> efficiently.</p><h4>3. Knowledge Graph Model</h4><p><strong>Definition</strong>: A <strong>semantic graph</strong> that follows a structured ontology (like RDF triples).</p><p><strong>Structure</strong>:</p><ul><li>Nodes represent <strong>concepts/entities</strong>.</li><li>Relationships are <strong>facts/semantic links</strong>.</li></ul><p><strong>Use Cases</strong>:</p><ul><li>AI, search engines (Google Knowledge Graph), NLP.</li></ul><p><strong>Why Use It?</strong>: Provides <strong>structured, AI-driven insights</strong>.</p><p>Some more graph models are as follows :</p><ul><li><strong>Multigraph Model</strong> : A graph where multiple relationships exist between the same nodes.</li><li><strong>Hierarchical Graph Model</strong> : A graph structured as a tree or hierarchy</li><li><strong>Bipartite Graph Model</strong> : A graph where nodes are divided into two sets, and edges only connect nodes from different sets.</li><li><strong>Temporal Graph Model</strong> : A graph where relationships and nodes have timestamps or versioning.</li><li><strong>Spatial Graph Model</strong> : A graph where nodes and relationships have geographic properties (latitude, longitude).</li><li><strong>Graph Projection Model</strong> : A model where a subset of the graph is extracted for analysis.</li></ul><h3>Tech Stack Overview</h3><ol><li>Apache Spark: Distributed processing engine for handling large-scale data.</li><li>Neo4j: Graph database for modeling relationships.</li><li>PySpark: Python API for Spark.</li><li>Fuzzy Matching (thefuzz library): String similarity scoring for data deduplication.</li></ol><h3>Graph over ArXiv Dataset</h3><p>We will create a Property Graph over ArXiv Dataset in Neo4j. ArXiv dataset is a metadata file.</p><p>The metadata file is provided in JSON format and includes an entry for each paper with the following details:</p><ul><li><strong>id</strong>: The ArXiv ID, which can be used to access the paper.</li><li><strong>submitter</strong>: The individual who submitted the paper.</li><li><strong>authors</strong>: Names of the paper’s authors.</li><li><strong>title</strong>: The title of the paper.</li><li><strong>comments</strong>: Additional details such as the number of pages and figures.</li><li><strong>journal-ref</strong>: Information on the journal where the paper was published.</li><li><strong>doi</strong>: The Digital Object Identifier (<a href="https://www.doi.org/">DOI</a>).</li><li><strong>abstract</strong>: A summary of the paper’s content.</li><li><strong>categories</strong>: Tags or classification categories used in the ArXiv system.</li><li><strong>versions</strong>: A history of the paper’s versions.</li></ul><p>You could read and download details about dataset in here: <a href="https://www.kaggle.com/datasets/Cornell-University/arxiv">ArXiv Dataset</a></p><p>Going to use Spark to load the dataset into the Neo4j. Spark is excellent too to transfrom the the data. Neo4j has neo4j-spark-connector which could act as a bridge to connect from Spark to Neo4j. For more about neo4j Connecter visit here : <a href="https://neo4j.com/docs/spark/current/installation/">neo4j-spark-connector</a></p><h4>Data Pipeline Architecture</h4><p>The pipeline consists of the following stages:</p><ol><li><strong>Data Ingestion</strong>: Load JSON metadata from ArXiv.</li><li><strong>Data Transformation &amp; Cleaning</strong>: Process author names, categories, and versions.</li><li><strong>Data Enrichment</strong>: Use fuzzy matching for submitter validation.</li><li><strong>Graph Data Modeling</strong>: Define a Cypher query for inserting nodes and relationships into Neo4j.</li><li><strong>Batch Insertion</strong>: Use Spark’s Neo4j connector for efficient writes.</li></ol><p>Let’s dive into the implementation.</p><h4>Step 1: Initializing Spark and Neo4j Connection</h4><pre>from pyspark.sql import SparkSession<br><br>spark = (SparkSession.builder<br>         .appName(&quot;spark_Neo4j&quot;)<br>         .getOrCreate())<br><br>spark.conf.set(&quot;neo4j.url&quot;, &quot;&lt;bolt connection string&gt;&quot;)<br>spark.conf.set(&quot;neo4j.authentication.basic.username&quot;, &quot;&lt;username&gt;&quot;)<br>spark.conf.set(&quot;neo4j.authentication.basic.password&quot;, &quot;&lt;password&gt;&quot;)<br>spark.conf.set(&quot;neo4j.database&quot;, &quot;&lt;graphDatabase&gt;&quot;)<br>spark.conf.set(&quot;neo4j.batch.size&quot;, &quot;25000&quot;)</pre><p>By setting the batch size to 25,000, each set of nodes is processed at a time, reducing the load time for Neo4j. Since we are loading 1.7 million metadata entries into Neo4j, this approach optimally distributes the processing load on my machine.</p><h4>Step 2: Data Preprocessing</h4><p>We define UDFs (User Defined Functions) to clean and process the data.</p><p><strong>Processing Author Names</strong></p><pre>from pyspark.sql.functions import udf<br>from pyspark.sql.types import ArrayType, StringType<br><br>@udf(ArrayType(StringType()))<br>def processTheName(nameArray):<br>    return [(namelist[-2] + &#39; &#39; + namelist[-1]).strip() for namelist in nameArray]</pre><p><strong>Handling Categories</strong></p><pre>@udf(ArrayType(StringType()))<br>def process_categories(categories):<br>    return [cat.strip() for cat in categories.split(&quot; &quot;)]</pre><p><strong>Fuzzy Matching for Submitter Validation</strong></p><pre>from thefuzz import fuzz<br>import numpy as np<br><br>@udf(StringType())<br>def names_tools_comparision(authors, submitter):<br>    scores = {fuzz.token_sort_ratio(auth, submitter): auth for auth in authors}<br>    return scores.get(max(scores, default=0), submitter)</pre><p><strong>Structuring Version Information</strong></p><pre>@udf(ArrayType(MapType(StringType(), StringType())))<br>def process_versions(versions):<br>    return [{&quot;created&quot;: ver[&#39;created&#39;], &quot;version&quot;: ver[&#39;version&#39;]} for ver in versions]</pre><h4>Step 3: Applying Transformations</h4><pre>jsonData = spark.read.json(f&quot;{PathToFile}/arxiv-metadata-oai-snapshot.json&quot;)<br><br>jsonData_authors_categories = (jsonData<br>    .withColumn(&#39;authors&#39;, processTheName(&#39;authors_parsed&#39;))<br>    .withColumn(&#39;categories&#39;, process_categories(&#39;categories&#39;))<br>    .withColumn(&#39;versions&#39;, process_versions(&#39;versions&#39;))<br>    .withColumn(&#39;submitter&#39;, names_tools_comparision(&#39;authors&#39;, &#39;submitter&#39;))<br>)<br>selectTargetColumns = (jsonData_authors_categories<br>                                .select(<br>                                        &#39;id&#39;,&#39;title&#39;,&#39;abstract&#39;,<br>                                        &#39;authors&#39;,&#39;categories&#39;,&#39;doi&#39;,<br>                                        &#39;journal-ref&#39;,&#39;license&#39;,&#39;submitter&#39;,<br>                                        &#39;update_date&#39;,&#39;versions&#39;)<br>                              )</pre><h4>Step 4: Writing Data to Neo4j</h4><p>Before writing the data into the Neo4j, we will create a Unique keys in the Neo4j and to create them we run below query in Neo4j directly.</p><pre>// Ensure unique Papers by ID<br>CREATE CONSTRAINT paper_id_unique IF NOT EXISTS <br>FOR (p:Paper) REQUIRE p.id IS UNIQUE;<br><br>// Ensure unique Authors by name<br>CREATE CONSTRAINT author_name_unique IF NOT EXISTS <br>FOR (a:Author) REQUIRE a.name IS UNIQUE;<br><br>// Ensure unique Domains (categories) by name<br>CREATE CONSTRAINT domain_name_unique IF NOT EXISTS <br>FOR (c:Domain) REQUIRE c.domain IS UNIQUE;<br><br>// Ensure unique Versions by version number<br>CREATE CONSTRAINT version_unique IF NOT EXISTS <br>FOR (v:Versions) REQUIRE v.version IS UNIQUE;</pre><p>We construct a Cypher query to model the relationships and run them in Spark:</p><pre>query = &quot;&quot;&quot;<br>        MERGE (p:Paper {id: event.id})<br>        SET p.title = event.title, p.abstract =event.abstract<br>        WITH p, event<br>        UNWIND event.authors AS author<br>        MERGE (a:Author {name: author})<br>        MERGE (a)-[:AUTHORED]-&gt;(p)<br>        with p,event<br>        MATCH (sub:Author{name:event.submitter}) <br>        MERGE (sub)-[:SUBMITTED]-&gt;(p)<br>        WITH p,event<br>        UNWIND event.categories AS cat<br>        MERGE (c:Domain {name: cat})<br>        MERGE (c)&lt;-[:BelongsTo]-(p)<br>        WITH p,event<br>        FOREACH (ver IN event.versions | <br>          MERGE (v:Versions {version: ver.version})<br>          ON CREATE SET v.createdDate = ver.created<br>          MERGE (v)&lt;-[:VERSION]-(p)<br>        )<br>        &quot;&quot;&quot;<br><br>(selectTargetColumns.repartition(1).write<br>      .format(&quot;org.neo4j.spark.DataSource&quot;)<br>      .option(&quot;query&quot;, query)<br>      .mode(&quot;overwrite&quot;)<br>      .save())</pre><p>When writing data using the Neo4j Connector for Apache Spark, it’s strongly advised to <strong>repartition your DataFrame to a single partition</strong> (using coalesce(1) or repartition(1)) to prevent deadlocks and optimize data ingestion efficiency</p><h3>Performance Considerations</h3><ul><li><strong>Batch Size Optimization</strong>: Writing in batches of 25,000 records prevents Neo4j from being overwhelmed. One can adjust the batch size according to capacity of machine.</li><li><strong>Efficient Partitioning</strong>: Redistributing data across partitions ensures parallel processing.</li><li><strong>Preprocessing in Spark</strong>: Transforming data before insertion minimizes Neo4j query complexity.</li><li><strong>Fuzzy Matching for Data Integrity</strong>: Ensuring author consistency prevents duplicate nodes.</li></ul><h3>Sample Query</h3><p>Find papers by a specific author and belongs to which domain:</p><pre>MATCH <br>  (a:Author {name: &quot;Markus Andres&quot;})-[:AUTHORED]-&gt;(p:Paper)-<br>  [:BelongsTo]-&gt;(d:Domain)<br>RETURN p,a,d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*oROVytoj6ajh_J-ewtAsRA.png" /><figcaption>Query : Papers by a <strong>Markus Andres</strong> and belongs to which domain</figcaption></figure><p>Who co-authored those papers and who submitted those ?</p><pre>MATCH <br>  (a:Author {name: &quot;Markus Andres&quot;})-[:AUTHORED]-&gt;<br>  (p:Paper)-[:BelongsTo]-&gt;(d:Domain)<br>with a, p, d<br>MATCH (p) &lt;-[:AUTHORED]-(at:Author)<br>RETURN p,a,d,at</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*7XVZTOQVourG58J0R7dalw.png" /><figcaption>Query : Papers by a <strong>Markus Andres</strong> and who co-authored and Submitted those papers</figcaption></figure><p>Find the Authors in each Domain and count Each papers submitted</p><pre>MATCH <br>  (a:Author)-[:AUTHORED]-&gt;(p:Paper)<br>  -[:BelongsTo]-&gt;(c:Domain {name: &quot;math.OA&quot;})<br>WITH a, c, COUNT(DISTINCT p) AS paperCount<br>CALL {<br>    WITH a, c, paperCount<br>    CREATE (v:CountEachPaper {count: paperCount})<br>    RETURN v<br>}<br>MERGE (a)-[:PAPER_COUNT]-&gt;(v)<br>MERGE (a)-[:AUTHOR_SUBITTED_TO_DOMAIN]-&gt;(c);</pre><pre>MATCH (n:Domain {name: &quot;math.OA&quot;})<br>&lt;-[:AUTHOR_SUBITTED_TO_DOMAIN]<br>-(a:Author)-[:PAPER_COUNT]-&gt;(c:CountEachPaper) RETURN n,a,c LIMIT 10</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*ydgQsJXEnlBNjvNAFPSt-Q.png" /><figcaption>Query : Authors in each Domain and count of each papers submitted`</figcaption></figure><h3>Conclusion</h3><p>This pipeline efficiently ingests and processes large-scale research paper metadata, transforming it into a structured graph format suitable for complex relationship queries. Using PySpark and Neo4j together allows for scalable, high-performance data processing while preserving the richness of relationships within the dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f736d0fd917" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>