Sentence Embeddings
Sap S4hana
Zero Shot Classification
Erp Catalog Mapping
Unsupervised Nlp

Assessing ML Applicability for SAP S/4HANA PM Catalog Mapping

Adien Aspen

15 min read6 days ago

An NLP data-prospection study on legacy maintenance records before a ML modeling phase.

Data prospection, ML applicability and readiness verdicts for industrial asset maintenance.

Abstract

Case. In asset-intensive industries, maintenance is often executed by third parties and recorded outside the enterprise system that will later manage the asset. This study asked whether free-text maintenance records, written by technicians in their own words, carry enough signal to be assigned the catalog codes that SAP S/4HANA Plant Maintenance requires, without a domain expert and without reference labels.

Approach. The study built its evidence instead of assuming it. It separated three source paths of different nature, replaced the missing expert with surrogate ground truth and a blind panel of language-model annotators, and read the target system’s own configuration as part of the problem. An applied NLP engine carried technician text to domain concepts and from there to catalog codes, with the embedding model held constant throughout.

Outcome. The result is a decision instrument, the signal report, showing that most of the problem is not derivable to a possible second ML modeling phase: it is resolved through structure, target-system configuration and data preparation, and modeling has a basis only in a bounded subset and under explicit conditions.

Technical Note — scope at a glance

3 Source Paths: Relational work orders · Daily status reports · Daily status workbooks, never aggregated.
5 Catalog Targets: Object Part, Damage, Cause, Symptom, Activity + 2 structural (activity type, functional location).
Annotation: blind, two model families, agreement as raw, κ and AC1.
Metrics: presence, agreement, concentration, expressibility under the catalog profile.
No supervised classifier was trained; no model benchmarks are reported.

1 · Context — the hand-off problem

Legacy maintenance text, enterprise destination

Gas and oil plants, power plants and solar parks are strategic productive assets. Their owners usually manage them in a world-class ERP, where every intervention becomes a structured record tied to a technical object.

The maintenance itself is often executed by third parties. Those parties record their work in their own systems, in their own language and formats: work orders, daily status reports, spreadsheets, free-text notes.

When that information must reach the asset owner’s ERP, the task is not a classic migration. There is no source schema to map field by field.

In this study, migration means turning information produced outside the target system into records the target (ERP) system accepts. Validity is defined by the target, not by the source.

The reference system is SAP S/4HANA Plant Maintenance (PM). In PM, a maintenance notification is coded against catalogs: object part, damage, cause, symptom and activity. Each code must belong to the catalog subset that the technical object allows.

A proof of concept has no client. The study therefore uses a reference operator: a hypothetical owner of the assets and of the ERP configuration. Decisions that depend on that owner, such as which registration level to use, are stated as decisions for the reference operator, not taken by the study.

The study is positioned as a preliminary phase. Its job is to decide, with evidence, whether a full ML phase for extraction and classification has a basis, and under which conditions.

Figure 1 · From legacy maintenance text to a signal report.

2 · The origin — three paths and a time axis

Not one corpus, three

The catalog targets were known from the start. The source paths were found in the work, and they turned out to be three data natures, not one corpus.

Relational work orders: one row per order, on the order of 1⁰⁴ orders, with structured fields and a free-text description.
Daily status reports: one document per day, listing the jobs open on each asset.
Daily status workbooks: one spreadsheet per day, with a similar listing in tabular form.

Each path has its own counting unit. The study never aggregates figures across paths, because a percentage over orders and a percentage over daily rows answer different questions.

The two daily paths share a property that changes everything downstream: they are snapshots. The same job appears again every day until it is closed.

A provisional grouping measured how much repetition this introduces. Compression ranged from about 1.07×, when rows carried a usable date, to about 4.32×, when they did not. One daily path compressed 1.66× from rows to estimated events.

These ratios show the problem; they do not solve it. An estimated event is not yet a maintenance episode, and only an episode is a unit the target system (ERP) can receive.

The study therefore declared episode reconstruction outside its scope and derived it to data engineering. The technique has established names:

Event correlation and Case identification in process mining,
Sessionization, and
Temporal entity resolution.

Signal measured on the snapshot paths is still reported, always as contingent on that reconstruction.

Technical Note — recommended procedure

Event log: one row per mention and day (date · asset · concept · text · status signals).
Candidate episodes: correlate by asset and concept, text similarity across days, maximum gap in days.
Episode boundaries: start, explicit closure or disappearance, episodes still open at the end of the period.
Consolidation: one record per episode, validated on a blind sample with pairwise precision and coverage, and cross-checked against the relational orders.
Chain: daily rows → event log → candidate episodes → bounded episodes → one record per episode.

Lesson for the reader. Before measuring signal, confirm that your unit of analysis is the unit you will migrate.

3 · Working without an expert — evidence engineering

When there is no one to ask

Every project of this kind needs a subject-matter expert to confirm codes, validate labels and settle business rules. This one did not have one. The study calls that condition an SME-free setting.

The condition shapes everything else. It explains the assumptions the study makes, the points it leaves open, and why it built alternative routes to ground truth instead of waiting for one.

A substantial part of the work went into researching codes, building dictionaries and learning the domain lexicon. That work is the base on which every later measurement stands.

Surrogate ground truth

The study’s answer is surrogate ground truth: evidence triangulated from independent sources and graded by tier :

gold · silver-high · silver-medium · unresolved

One rule governs every figure: never hide what is observed, validated, proxy, synthetic or inferred. A proxy figure is never presented as verified accuracy.

A blind rater panel built from language models

To give human-like review to assignments that could not be reviewed otherwise, the study used language models (LLMs) as a surrogate rater panel: independent readers judging assignments under a fixed rubric.

The idea is established practice. The design is where the contribution lies:

Blind: annotators never saw the pipeline’s proposals or scores.
Rules first: operational rules were written before annotating.
Restricted scope: no scripts and no automated rules inside the annotation.
Two model families, so agreement is not one model agreeing with itself.
Verified batches: every batch was checked and rejected whole on failure.

The labels are used for evaluation, not training. This is not synthetic augmentation or oversampling.

Independence is declared precisely. At instance level, the annotators had no part in the project. At family level, only one annotator shares a model family with the assistant used in protocol design.

Agreement measures consistency, not accuracy. The panel is a controlled surrogate for the expert, not an equivalent.

The protocol mattered more than the annotators. Rewriting one operational rule moved κ (kappa index) from 0.086 to 0.893, with the same annotators and the same sample.

Evidence and execution discipline

Seven practices kept the figures honest:

Provenance per figure (every number carries its source and evidence level).
Explicit counting unit (orders, rows or events, stated every time)
Pre-flight against the code (a mechanism is verified in the code before it is assumed).
Replay with a reproduction gate (a figure counts only if a rerun reproduces it).
Sum assertion (parts must add up to the declared total).
Structural vs lexical absence (a missing cataloging code is not the same as a missing word).
Documented artifacts of the mechanism (large numbers produced by the method itself are labeled as such).

Technical Note — annotation setup

Overlap sample annotated by both LLMs annotator families, used for agreement only.
Metrics: raw agreement, Cohen’s κ, Gwet’s AC1, per-class breakdown; prevalence effects declared when κ and AC1 diverge.
Each κ states its question: presence of the target in the text, or the code itself.
Diagnostic role: the samples are not a final held-out test set.

Lesson for the reader. When agreement is low, rewrite the protocol before replacing the annotators.

4 · The NLP engine — from technician text to catalog code

Open lexicon, controlled vocabulary

The core problem is carrying an open lexicon into a controlled vocabulary. Technicians write freely; the catalog accepts only its own cataloging codes.

No supervised classifier was trained. Machine learning was used to represent, detect, link and evaluate, which is what a feasibility phase calls for.

Four terms are kept apart.

The technician’s lexicon is what people write.
A vocabulary is a curated set of concepts used to detect meaning in text.
A dictionary only translates cataloging codes into labels.
The ERP catalog receives the result. Detection and destination are kept as separate roles.

The lexical layer also keeps six namespaces separate:

the the asset-standard designation,
the plant designation code,
the ISO 14224 taxonomy level,
the SAP functional-location code,
the SAP equipment number and
the technician’s own term.

Merging any two of them silently changes what a match means. The engine runs in a strict order:

text → domain concept → hierarchy node → catalog profile → code

4.1 Representation — corpus-native vocabulary

The first step listens to the corpus before reading the catalog. Terms of one to three words were extracted from eligible free text across the source paths, normalized and filtered.

Similar expressions were then grouped. Sentence embeddings from a multilingual MiniLM model represented each term, and MiniBatchKMeans clustered the normalized vectors.

This is a hybrid neural–classical pipeline, not an ensemble !

The result compressed on the order of 1⁰⁴ surface forms into a few hundred concepts. Only after the concepts existed were they contrasted with the catalog, as a separate step.

4.2 Detection — semantic prototypes

Detection is zero-shot classification by similarity to class prototypes. A text is assigned a concept when its embedding is close enough to the concept’s prototype, with thresholds calibrated against annotated samples.

The gain over string matching is large. On the same records, exact lexical matching anchored about 5% of the signal; semantic detection anchored about 63%.

Who writes the prototypes matters. Short phrases written by the analyst tended to abstain. Terms taken from normative published vocabularies recovered signal: read that way, damage appeared in 33.2% of eligible orders. Activity, read through the corpus-native vocabulary, appeared in 70.8%.

4.3 Linking — concept to hierarchy to code

A detected concept is not yet a code. It must be linked to a node of the functional-location hierarchy, and that node decides which catalog subset applies.

Linking uses a similarity threshold, a margin over the second candidate and consensus on ties…

Get Adien Aspen’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It is the most delicate piece of the engine: two chained links, each with its own failure mode. Either no concept is detected, or a concept is detected and not linked to a node.

Neither failure is discarded. Signal without a code is characterized and kept, because it may describe a gap in the configuration rather than an absence of meaning.

For consumed materials during assets maintenance work, the same idea applies as profile-constrained matching: candidate codes are restricted to those the asset’s profile enables before any similarity is computed.

4.4 The model as a constant

The same embedding model was used three ways: anchoring text against catalog descriptions, zero-shot detection against prototypes, and reference-free clustering of the corpus.

The model was the constant; the comparison target was the variable. Each time it was used established the limit of the previous one.

Catalog anchoring showed that the catalog works as a destination, not as a detector. Prototypes showed that their source decides their reach. Clustering gave the corpus its own vocabulary.

Figure 2 · The NLP engine: one embedding model, three uses, two chained links.

Technical Note — representation and linking

Model: paraphrase-multilingual-MiniLM-L12-v2, 384-dimensional sentence embeddings, 128-token input limit declared where it applies.
Clustering: MiniBatchKMeans on L2-normalized vectors; k ∈ {256, 384, 512, 768}.
Selection: k = 256 by cosine silhouette on a deterministic sample of 3,000 extracted terms (not records, not codes), subject to concentration controls.
Later sanitization pruned cluster members without retraining the centroids.
Linking: similarity threshold 0.60, margin 0.05, consensus on ties.
For L2-normalized vectors, Euclidean distance and cosine similarity rank neighbors identically:

Lesson for the reader #1. Hold the model constant and vary what it is compared against; the comparison target often matters more than the model.
Lesson for the reader #2. Listen to the corpus before reading the catalog; map to the destination only after the concepts exist.

5 · The destination — the answer space

The answer was in the target system

The target system’s configuration defines the answer space. In SAP/S4Hana PM module, each technical object carries a catalog profile, and the profile decides which group and code pairs a notification may use.

The study names this the Registration-Level Principle: the level at which a notification is registered chooses the profile, and the profile chooses the catalog subset. The answer space narrows from thousands of codes to tens per record before any model reads a word.

Critical Insight — Destination vs Detector

The catalog works as the mapping destination, not as a text detector. Its descriptions are written for coding, not for recognizing how technicians speak, and when used as detection anchors they tend to over-generate.

Detection belongs to the vocabulary built from the corpus. The catalog is consulted afterwards, to decide which codes can receive what was detected.

Asset level and component level are different sets

Registering at asset or at component level is not a choice between a coarse and a fine version of the same list. The two profiles are different sets, and the meaning they share is a minority that varies by target.

*This figure is still provisional.
Shared = component-profile descriptions also present in the asset-level profile.
Expressible = share of signal units with at least one code enabled under the asset-level profile.
Object part: one enabled pair, uninformative about the part.

Two kinds of gap follow.

In a profile gap, the code exists in the catalog but the profile does not enable it.

In a catalog gap, the catalog has no code for a signal that is clearly present. The target Activity is the clearest case: strong, consistent signal that the catalog barely expresses.

Signal without a code is characterized, not discarded. It becomes evidence about the configuration that the reference operator can act on.

Constraining before matching

Restricting candidates to the codes enabled by the profile under each asset found a valid code for 95.2% of comparable records.

An earlier, lower figure had come from code selection, not from the configuration. Two signals were then compared: the work narrative and the consumed material. They agree on 12.8% of records against roughly 3–6% expected by chance. They are related, and neither validates the other.

Technical Note — answer space

Chain: full catalog (thousands of codes) → profile of the technical object → registration level → tens of valid codes per record.
Shared-description shares are computed over the component-level profiles in use, per destination target.
Expressibility = share of signal units with at least one code enabled by the profile.
Cross-signal agreement measured at equal similarity score; agreement on the exact chosen code is lower.

Lesson for the reader #1. Read the target system’s configuration before designing the detector; it may already contain the answer space.
Lesson for the reader #2. Constrain candidate codes by the profile of the technical object before matching.

6 · The verdict — the signal report

A decision instrument, not a benchmark

The formal deliverable of the study is a signal report. For each decision unit it answers three questions:

is there signal?,
of what quality?, and
what is missing to use it?

The report decides whether a later ML phase has a basis. It does not estimate how well a model would perform.

A decision unit is a target × source path × registration level. Each unit is scored on five dimensions:

Presence: is there signal in the eligible universe?
Consistency: do independent readers agree on it?
Discrimination: does it separate classes, or concentrate on one?
Expressibility: can the target system receive it under the profile?
Dependency: what is missing to use it: nothing, a reference-operator decision, or data preparation?

Cells carry Pass · Partial · Fail · Not measured · Not applicable. Thresholds come from the literature, from flags already used in the project, or from the analyst; the last ones are declared as such.

The verdict rule is fixed.

Ready: every applicable dimension passes and nothing is missing.
Conditional: nothing fails and dependencies become stated conditions.
Not ready: presence, consistency or expressibility fails.
Not assessable: the unit of analysis is not yet valid.

One cap applies. No precision has been validated, so no ML-addressable unit can be Ready; only deterministic units can.

Relational path; free-text targets scored under the asset-level profile. Full template and thresholds in Annex A.

Reading by verdict class

Ready units are deterministic. Activity type and asset-level location are migrated without a model, from a field or a structural join.

Conditional units have signal the destination can receive, once the reference operator fixes the registration level and precision is measured on a human-annotated sample.

Not ready units have signal, but the current profile enables few codes for it, or the signal is scarce. Activity is the instructive case: the clearest, most consistent signal in the study, and almost no code to receive it. That is resolved in the destination’s configuration, not with a model.

Not assessable units wait for episode reconstruction. That is data engineering.

Each verdict states what would change it: a registration-level decision, an annotated sample, a configuration change, or reconstructed episodes.

Lesson for the reader. Separate “is there signal” from “can the destination receive it”; they fail for different reasons.

7 · What comes next

Conditions before modeling

The signal report turns into a short sequence of conditions for the reference operator:

Decide the registration level. Asset and component profiles are different answer spaces; the choice sets what any later model can be asked to do.
Annotate a blind human sample if registration moves to component level, so precision can be measured rather than assumed.
Reconstruct episodes in the snapshot paths before scoring them.
Then, if warranted, open the ML phase, limited to the units the report marks as Conditional.

The limitations are stated once. No precision has been validated. The annotators are language models, not domain experts. Some steps operate under a 128-token input limit. The snapshot paths were scored before reconstruction. The study read one reference configuration. One verdict, damage, is provisional.

8 · Closing — what made this study distinct

Feasibility as a result

Four features shaped the study.

There was no expert and no labels.
The destination held much of the answer.
The source was three data natures, one with a time axis, and
The outcome was an honest feasibility verdict, not a model.

The value of a study like this is to establish, with evidence, which part of a problem is derivable to machine learning and which part belongs to structure, configuration or data preparation.

Here, most of it belonged to the latter, and the part that remains has explicit conditions.

For practitioners in similar niches, the reusable piece is the way of working:

separate the paths,
build evidence with declared provenance,
read the destination early, and
let a signal report decide whether the model comes next.

“Read the destination first; then decide what the model should learn.”

Annex A · Signal report template

The template scores one decision unit per row (target × source path × registration level) on five dimensions, then applies the verdict rule in chapter 6.

It can be reused for any mapping of free text into a controlled vocabulary whose destination is configured per object.

Rules attached to the dimensions:

When κ and AC1 diverge because of prevalence, rate by κ and flag it.
State which question each κ measured: presence of the target, or the code.
Ties above 50% lower discrimination by one level.
Discrimination failing alone leaves the unit Conditional, since profile restriction can correct it.