A helicopter engine measurement needs context.

A temperature that looks unusual in one situation might be expected in another. A torque reading means more when we know the conditions under which it was produced. Before asking an algorithm whether an engine is healthy, we need to understand the physical system it is observing.

That was the starting point for this project: use the PHM Society’s 2024 helicopter turbine engine data challenge to build a complete modelling and review workflow. The work grew from exploratory analysis into a reproducible harness covering feature engineering, model selection, uncertainty, operating-condition diagnostics and a local review dashboard.

The examples below use invented numbers to explain the ideas. The results section identifies the actual experiment results separately.

A short physics primer

The engine supplies the turning effort; the rotor produces lift

Many turbine-powered helicopters use a turboshaft engine. Air enters a compressor, fuel burns in a combustion chamber, and expanding gases drive turbines. Through a shaft and transmission, the engine supplies mechanical power to the rotor system. The spinning rotor blades produce aerodynamic lift. [1]

Two useful concepts are torque and power. Torque is twisting effort — like the effort you apply when turning a wrench. Power describes how quickly work is being done. For a rotating shaft, power equals torque multiplied by angular speed. A torque measurement is therefore informative, but without a correctly interpreted shaft speed it is not a complete power measurement.

Operating conditions change the picture

Helicopter performance depends on the engine and rotor together. Air density, aircraft weight and wind matter. Hot conditions and higher altitude can reduce air density; a heavier helicopter requires more power to hover. Airflow through the rotor also changes the power needed, so hovering and forward flight cannot be treated as equivalent situations. [2]

Imagine comparing an engine on a cool day near sea level with an engine working in hot, less dense air. Different readings need not mean one engine is damaged. Some differences may reflect different operating demands and environmental conditions.

That creates the central modelling challenge: distinguish changes associated with operating conditions from patterns associated with the labelled engine health state.

Our dataset does not contain every quantity needed for a complete physical explanation. We do not have fuel flow, aircraft weight, wind, a full engine map or all the internal pressures and temperatures. Physics can guide useful comparisons, but it cannot fill in missing measurements by assumption.

Torque margin compares output with a reference

The challenge defines torque margin by comparing measured torque with a design target for the operating conditions: [3]

Torque margin (%) = 100 × (measured torque − target torque) / target torque

Suppose the target is 100 units and the engine produces 95. The margin is −5%. If it produces 103, the margin is +3%.

This is a reference-relative performance measure. It is not automatically an aircraft’s available reserve power, and it is not a direct permission to continue operating.

It also does not completely determine the fault label. Our data contains healthy observations with negative margins and faulty observations with nonnegative margins. We therefore predict the margin and the fault label as two related outcomes.

The business problem

The challenge asks for a health classification and a probability distribution for torque margin. It supplies observations from seven engines of the same make and model: four contribute training data and three contribute the external datasets. Engine identities are removed and the observations are shuffled. [3]

We translated that task into a practical engineering question:

Can we turn engine measurements into a useful review signal, while showing when the model’s evidence is weak?

A useful result should tell a reviewer the estimated fault probability, the predicted margin, the uncertainty around that estimate and whether the measurements resemble the model’s reference experience.

The potential users are maintenance engineers, engine health monitoring teams, reliability analysts and fleet planners. Earlier attention to concerning observations could support inspection planning and focused engineering review. But those are intended benefits, not measured savings from this project. We have not established avoided failures, reduced downtime or a financial return.

Missing a fault and creating an unnecessary alert have different consequences. A high accuracy number does not settle that tradeoff. The eventual review action and acceptable alert burden need agreement with the people who will use the system.

Step 1 — Build a harness around the experiment

A harness is the machinery that makes an experiment repeatable.

Instead of relying on a notebook’s current state, our Python harness reads the data, checks its structure, creates partitions, trains candidates, compares them, saves the selected model and writes evaluation outputs.

It also records the configuration, data hashes and software environment. A data hash is a fingerprint of a file’s contents. It helps reveal whether the input changed between experiments.

For example, if an experiment improves after someone replaces a CSV, we need to distinguish a better method from different data. Keeping those records makes the comparison reviewable.

Cursor is the development workspace for editing and running the code. The harness itself is an ordinary Python project; it does not depend on Cursor to make predictions.

Step 2 — Check what the data actually contains

Exploratory data analysis, or EDA, is the first investigation of the dataset.

We inspect missing values, distributions, extreme values, repeated measurements, target relationships and differences between the training and external data. We also check IDs and join features to labels by ID rather than assuming row order matches.

The original labelled dataset contains 742,625 observations and seven predictors: measured torque, outside air temperature, mean gas temperature, PA, indicated airspeed, NP and compressor speed.

An important finding is that the data has no usable timestamps or engine identifiers. We cannot treat adjacent rows as adjacent moments in a flight.

For example, subtracting row 101’s temperature from row 100’s temperature would not measure a temperature change over time. Those observations may come from unrelated situations. That rules out rolling averages, degradation trajectories and meaningful frequency analysis in this harness.

Another finding is that fault labels are not simply determined by whether torque margin is negative. A sign-based rule would discard information that the model needs to learn.

Step 3 — Resolve meanings before writing physics equations

Column names can be misleading.

The organizer describes PA as power available. A published participant paper interprets PA as pressure altitude. Those meanings are incompatible. [3, 4]

Our default configuration therefore treats PA as opaque: it uses the numbers as predictors without assuming an altitude interpretation. NP is also retained, but its physical interpretation requires further confirmation.

Suppose a PA value is 500. If it is altitude in metres, it might support an atmospheric calculation. If it is a power-related quantity, that calculation would be meaningless.

The harness allows optional altitude and air-density features only when the meaning and units are explicitly configured. Celsius temperature interpretation is also an explicit assumption, rather than an invisible convention.

This is where a data owner or domain engineer becomes essential. A plausible numerical range does not establish what a sensor measures.

Step 4 — Separate learning from examination

Models need an examination they cannot study for.

We create four partitions: approximately 60% for initial fitting, 15% for selection, 10% for calibration and 15% for final evaluation. The proportions are based on groups, so row counts do not match those percentages exactly.

The fitting partition teaches candidate models. The selection partition compares approaches. The calibration partition adjusts probabilities, interval scale and the conventional fault threshold. The final holdout assesses the chosen approach.

We keep exact repeated measurements together. Otherwise, a model might train on one copy and be examined on an identical copy.

That is like giving a student a question in a practice paper and repeating it word for word in the exam. The result overstates how well the student handles unfamiliar questions.

This grouping reduces duplicate leakage. It does not prove engine independence: different measurements from the same unknown engine can still appear in different partitions. Our internal holdout cannot establish performance on new engines.

Step 5 — Give the model physics-informed clues

Feature engineering creates useful attributes from the original measurements.

One example is the difference between mean gas temperature and outside air temperature. If both temperatures use the same units, their difference can describe a thermal relationship more clearly than either value alone.

Another example is a temperature-adjusted compressor-speed proxy. With outside temperature expressed in Kelvin, we compare NG after a simple temperature normalization. It is an approximate engineering clue, not a complete correction using all engine inlet conditions.

We also construct torque ratios, temperature ratios and interactions. An interaction lets the model consider two measurements together — for example, whether the relationship between torque and compressor speed changes across operating conditions.

Imagine two observations with the same torque but very different thermal readings. A derived relationship may help the model distinguish them.

These attributes are physics-informed proxies. We do not call a torque-to-temperature ratio an efficiency measurement: the required energy-flow information is absent. Likewise, zero indicated airspeed is an operating-state clue, not proof that the helicopter is hovering.

Step 6 — Add statistical context

Physics suggests relationships. Statistics describes what is typical in the observed data.

We calculate robust deviations from typical values, empirical percentile positions and tail indicators. A percentile tells us where a reading sits relative to the reference observations.

For example, a temperature near the 98th percentile is high relative to the fitting data. That does not make it faulty; it tells the model and analyst that the reading is unusual in that population.

We also group observations into six operating regimes using OAT, PA and IAS. A regime is a cluster of similar conditions, not a recovered engine identity.

This makes it possible to ask whether a gas-temperature reading is unusual for its operating neighbourhood, instead of comparing it only with one global average.

The statistical reference values are learned from fitting data. During validation, the held-out observations do not get to redefine what counts as typical.

Step 7 — Compare approaches instead of assuming complexity wins

We test candidates using raw inputs, physics-informed features, statistical features and combinations of them.

A simpler Ridge margin model provides a control. The main nonlinear models use histogram gradient boosting: successive small decision trees improve predictions by learning patterns left unexplained by earlier trees.

We also test a design-torque reconstruction approach. On labelled fitting data, the margin equation can be rearranged to recover a supervised design-torque target. A model predicts that target from environmental inputs, and the margin is then reconstructed.

Crucially, the true margin is used to construct a training target, never as an input at prediction time.

The reconstruction approach did not win the full experiment. That is useful evidence. An elegant physical hypothesis can still be too restrictive or incomplete for the available data.

The winning approach was a reduced model combining raw and derived features with direct margin prediction. We did not establish that every physics feature helps, or that a fully physical model is superior.

Step 8 — Let predicted margin help classify faults

The classifier uses the original and engineered attributes plus a predicted torque margin. This is called stacking: one model’s output becomes another model’s input.

There is a trap. If we predict margins on the same observations used to train the margin model, those predictions can be unrealistically good. The classifier may learn to trust that clue too much.

We use out-of-fold predictions instead.

Divide fitting data into five group-aware folds. Train the margin model on four folds and predict the fifth. Repeat until every observation has a prediction from a model that did not train on its group. Learned features are also refitted inside each fold.

It is similar to asking a tutor to evaluate students using questions the tutor has not already rehearsed with them.

The same held-out errors help train the uncertainty model. At inference, the margin model has been fitted on all allowed fitting rows, so its behaviour may differ somewhat from the fold models. Operational evaluation needs to check that this difference remains acceptable.

Step 9 — Test whether fewer features are enough

Derived features often overlap. Several ratios may carry much of the same information.

We measure classifier permutation importance on selection data: shuffle one feature, then observe how much AUC drops. AUC describes how well probabilities rank faulty observations above healthy ones.

If shuffling a feature damages that ranking, the classifier was using it helpfully. We retain the raw measurements, select up to 22 positively ranked features, remove duplicates and train a reduced candidate. The predicted margin remains a separate classifier input.

The reduced model must compete with its larger parent. We compare them using a joint development utility that considers classification ranking, probability quality and margin-distribution quality.

For example, dropping redundant features might preserve predictions while making the model smaller. But correlated features can share credit, so a low individual importance is not proof that a feature is useless.

The same selection partition ranks features and compares candidates. That can favour choices suited to that particular sample. The separate final holdout provides another check; repeated development splits would be useful if feature choices prove unstable.

Step 10 — Calibrate probabilities and uncertainty

A model can rank observations well while expressing too much confidence.

Suppose it assigns 80% fault probability to a collection of observations. If only about half are actually faulty, the ranking might still be useful, but the probabilities need correction.

We fit a smooth probability calibration on the separate calibration partition. We also adjust the predicted margin scale using calibration errors.

The margin output is a Gaussian distribution with a predicted centre and a positive scale. An illustrative output might be a mean margin of −3 percentage points and a 90% interval from −5 to −1. The interval communicates the spread of the empirical predictive distribution.

A 90% interval is not a 90% fault probability, and it is not a guaranteed bound for an individual engine. We check how often intervals contain true margins across evaluation observations, especially under changed conditions.

The conventional fault threshold is chosen using F2, which places more emphasis on recall than precision. It is a development choice, not an approved maintenance cost tradeoff. The challenge submission uses a separate score-optimized class and confidence; that confidence must not be confused with calibrated fault probability.

Step 11 — Examine more than accuracy

After choosing the configuration, we refit the final model on fitting plus selection rows, calibrate it separately and assess it on the final holdout.

The full experiment supplied the following internal results:

MeasureFull run internal holdoutObservations111,381Classification accuracy99.78%Fault recall99.88%Missed faulty observations52False positive observations192Torque margin MAE0.535 percentage pointsCoverage of the 90% margin intervals90.63%

Recall tells us how many labelled faulty observations were detected. Precision tells us how many flags were correct. MAE describes the typical absolute margin error. Interval coverage checks the uncertainty estimates.

An MAE of 0.535 percentage points means, for example, that predicting a margin of 4% when it is actually 4.5% produces an error of 0.5 percentage points. It does not mean the aircraft’s power is known within 0.535%.

The uploaded full-run outputs support this results table; its executable model remains in the user’s local workspace. Separately, the bundled sample model has reproducible verification evidence. Neither population supplies labelled external challenge accuracy or operational fleet accuracy.

Step 12 — Ask where the model loses support

An impressive average can hide a weak operating region.

We deliberately remove an entire operating regime from fitting and calibration, then evaluate observations from that regime. This asks what happens when the chosen approach loses coverage of particular joint conditions.

Regime 5 was particularly revealing. In the original exclusion experiment, fault recall fell to about 89.7%, and coverage of the nominal 90% intervals fell to about 64.4%.

A subsequent reproduction compared models on the same 24,456 regime observations. The main model missed 11 faults. The regime-excluded model missed 617. A model with a similar amount of training data removed randomly by duplicate group missed 4.

The reproduction’s membership differs slightly from the original experiment, so it is not an exact recovery. The comparison nevertheless suggests that losing a particular part of operating-condition coverage mattered much more than simply losing the same amount of data.

Nearly all those observations remained inside individual measurement ranges. Their combinations were poorly covered.

Think of someone who has driven in heavy rain and someone who has driven on steep roads. Neither fact proves experience driving on a steep road in heavy rain. Individual conditions can be familiar while their combination is unfamiliar.

The investigation is exploratory, with hidden engine effects and only one random-removal control seed. It identifies a coverage concern rather than proving a failure mechanism. It also does not mean every regime 5 observation should be rejected: the normal full model had that region in training.

Step 13 — Freeze the baseline and build reliability checks

Once we move into a pilot, the model must stop changing underneath the evaluation.

Freezing preserves the predictive artifact, calibration and conventional threshold. A separate monitoring sidecar describes reference support. It checks marginal ranges, distance from similar measurements, unusual interval width and calibration evidence for operating regimes.

For example, two rows may receive the same fault probability, but one resembles many reference observations while the other has an unfamiliar joint combination. The latter deserves additional attention to the prediction’s support.

These are provisional reliability checks. They flag concerns; they do not estimate a certified probability that the model is wrong.

We also check the software environment before loading the artifact. A model trained under one scikit-learn version should not silently run under a different version. The full artifact requires its compatible environment; the sample has its own pinned environment.

Model identity is protected with hashes. Those hashes reveal changes relative to the manifest, but they are not a security signature. Serialized model files must come from a trusted source.

Step 14 — Replay existing data through the complete system

Local replay runs the frozen prediction workflow on CSVs already available on the computer.

It produces predictions, reliability reasons and dashboard records. Where labels are available, it also measures errors. The purpose is to demonstrate that ingestion, inference, monitoring, evaluation and review fit together.

An observation might produce an estimated fault probability, a margin interval and a warning that its environmental combination is far from the reference measurements. That complete record is more useful for review than a bare label of 0 or 1.

The bundled milestone includes verified labelled and unlabelled replays and 34 passing software tests in the sample environment. The tests check contracts such as invalid input rejection, frozen prediction reproduction and label isolation. They do not establish operational suitability.

Step 15 — Move to operational shadow mode and delayed labels

Operational shadow means making predictions on real operational observations while the established engineering and maintenance process continues.

Suppose measurements arrive on Monday and inspection confirms their labelled outcome on Friday. The system saves Monday’s predictions and later evaluates those exact predictions against Friday’s labels. It does not rerun a newer model and rewrite what the system previously believed.

Each label evaluation creates a new snapshot. Partial labels remain partial; cumulative coverage requires a cumulative export. Invalid rows and exact fitting or calibration measurement matches are counted, with those matches excluded from reported evaluation metrics. This prevents direct reuse of known measurements from inflating the assessment, but it does not establish engine independence.

Shadow mode requires confirmed units, real engine and flight identifiers, timestamps, quality indicators and outcome provenance. It can start with batches; a live streaming connection is not required.

We have not yet completed a labelled operational study. The current delayed-label demonstration and temporary test fixtures exercise the software workflow, not real maintenance evidence.

Step 16 — Give engineers a review queue

The dashboard presents predictions alongside their support information and review history.

Its provisional ordering puts a fault flag with a reliability warning first. Supported fault flags and rejected measurements follow, then warnings without fault flags, then routine observations.

For example, an unsupported fault flag needs someone to examine both the possible fault and whether the model has enough relevant experience. A rejected row instead needs a data-quality correction.

These priorities describe workflow, not physical fault severity. True labels and prediction errors do not determine the operational ordering, because they would often be unavailable when review begins.

Reviewers can record follow-up or completion. Completing a review removes it from the open queue without changing the saved prediction. Review history is scoped to the model and prediction cohort so reused observation IDs cannot mix unrelated batches.

The web app runs locally with bounded requests and session checks. Shared deployment would require authenticated users, protected transport, roles, backups and operational ownership. It is currently an engineering pilot, not a fleet-wide production service.

What would make this useful in practice?

The next milestone is independent operational evidence.

We need an agreed data contract, representative engines and flights, trustworthy delayed labels and a clearly defined review action. Before inspecting outcomes, stakeholders should agree acceptable missed-fault rates, false-alert burden, interval coverage, data rejection and minimum evidence counts.

Evaluation should report both overall performance and performance in difficult or poorly supported conditions. Row counts alone are insufficient: thousands of correlated observations from one engine do not replace experience across independent engines.

We also need to measure whether the queue fits reviewer capacity and whether warnings actually concentrate errors. Earlier warning, persistence and remaining useful life need event and time information that the challenge data does not provide.

Retraining remains possible, but it creates a new candidate. The old baseline and its prediction history stay intact. A candidate should earn promotion through independent comparison and an explicit release decision.

The Statistical Design and Decision Record and Architecture Decision Record preserve why these choices were made, their alternatives and the evidence that should trigger reconsideration.

The project now connects a sensor row to a reproducible prediction, an uncertainty estimate, a support check and a review record. The next test is whether that chain helps engineers make better decisions on real observations, with enough independent evidence to trust its limits.

Sources and project evidence

1. FAA, Helicopter Flying Handbook, Chapter 4: Helicopter Components, Sections, and Systems. General engine and rotor-system context.
2. FAA, Helicopter Flying Handbook, Chapter 7: Helicopter Performance. Environmental and operating performance context.
3. PHM Society, 2024 Conference Data Challenge and organizer discussion of training columns. Challenge definition and field descriptions.
4. Ozeki and colleagues, Estimating the health of turbine engine based on domain knowledge and machine learning. Participant interpretation of PA. Its reported performance is not attributed to this project.

Project-specific behaviour is grounded in the complete harness implementation, feature catalogue and decision records. Numerical results come from the uploaded full-run metrics and regime investigation outputs. Software verification claims come from the bundled sample milestone and test log. Reference filenames include final_metrics.json, metric_comparison.csv, FEATURE_CATALOG.md and MILESTONE_REPORT.md.

Project status: local replay and engineering shadow-pilot workflow. Labelled operational validation and stakeholder acceptance remain pending.

Explore the code and join the discussion

The complete project is available on GitHub: phm2024-complete-harness.

Questions, feedback and collaboration ideas are welcome. If you are working on engine health monitoring, physics-informed machine learning or operational model validation, I would be interested in discussing your experience.

Contact Anshuman: zicodasanshuman@gmail.com

Predictive maintenance often gets described in one sentence:

Use sensor data to predict when a machine will need maintenance.

That sounds straightforward.

The reality is much messier.

When I started working on the PHM North America 2025 Data Challenge, I quickly realized that the hardest part was not choosing between LightGBM, CatBoost, or an LSTM. The Full implementation is in the my github repo.

The harder questions were:

• What information would actually be available at prediction time?
• How do I make the training data look like the test data?
• How do I validate a model when I only have a handful of engines?
• What does “good” even mean when late predictions are more expensive than early ones?
• And how do I know whether a strong validation score is genuinely trustworthy?

This article walks through the first version of the pipeline I built, why it worked, and the changes I would make before calling it production-grade.

The Problem: Predict Three Different Maintenance Events

The challenge is based on aircraft engine health data.

The goal is to predict how many cycles remain before three different maintenance events:

• High Pressure Turbine shop visit
• High Pressure Compressor shop visit
• HPC water wash

At first glance, that looks like a standard regression problem.

You have sensor values.

You have targets.

Train a model.

Predict remaining cycles.

But there is an important twist.

The training data contains long engine histories.

The test data does not.

Instead, each test file contains only a relatively short strip of engine behaviour — around 1,500 cycles.

Even more importantly, the cycle count inside those files is reset.

So the model might see:

0 → 1500

even if the actual engine has already operated for several thousand cycles.

That immediately removes one of the easiest shortcuts:

"Just use engine age."

The model has to infer health from behaviour, not from a hidden odometer.

That is where the problem became interesting.

The First Big Decision: Make Training Look Like Inference

One of the most useful decisions I made was also one of the simplest.

The test data comes as 1,500-cycle windows.

So I decided the training data should look the same.

Instead of training directly on the complete 20,000-cycle history of each engine, I cut the training engines into overlapping 1,500-cycle windows.

For example:

Window 1
0 ---------------- 1500

Window 2
   30 ---------------- 1530

Window 3
      60 ---------------- 1560

For every window, the model predicts the remaining maintenance cycles at the end of that window.

This turned four long engine histories into roughly 2,468 training examples.

More importantly, every training example now resembled the shape of the actual test problem.

That was a useful reminder:

Matching the training setup to the inference setup can matter more than choosing a more sophisticated algorithm.

Before Modeling: Stop the Model From Seeing the Future

Time-series leakage can make a model look brilliant very quickly.

Imagine I want to make a prediction at cycle 1,000.

A safe feature can use:

950 → 1000

A leaking feature might accidentally use:

950 → 1050

The model now knows something about the future.

That is especially easy to do with rolling statistics, normalization, or maintenance metadata.

So I added a simple rule to the pipeline:

For a prediction at cycle t, every feature must be computable using only information available at or before t.

That meant excluding things like:

• centered rolling windows
• future maintenance information
• target-derived features
• sensors unavailable in test data
• absolute engine age
• full-engine normalization using future observations

I also removed some cumulative maintenance counters because I could not guarantee that they would have the same meaning at inference time.

This made the model slightly harder to build — but much easier to trust.

Raw Sensors Were Not Enough

Aircraft-engine sensors do not operate in isolation.

A temperature reading can change because of:

• altitude
• Mach number
• outside air temperature
• engine load
• flight phase

So a high temperature does not automatically mean:

"the engine is degrading"

It may simply mean:

"the operating condition changed"

This is where feature engineering became the core of the solution.

For each sensor, I started creating features that answered three different questions:

1. What is happening right now?

latest sensor value

2. How has it changed recently?

difference from previous cycle
rolling mean
rolling standard deviation
rolling slope

3. How different is it from the start of the observed window?

current value - early-window baseline

For example, instead of only using:

T45 = 527

I might also use:

T45 increased by 27
T45 rolling slope = positive
T45 variability is increasing

Those features are much closer to the language of degradation.

I Kept the Different Operating Snapshots Separate

Each engine cycle could contain multiple snapshots.

A tempting option would be to average them.

I decided not to do that initially.

If different snapshots correspond to different engine operating states, averaging can destroy useful information.

So I created features such as:

T45_snapshot_1
T45_snapshot_4
T45_snapshot_7

and also differences such as:

T45_snapshot_1 - T45_snapshot_4

The intuition is simple.

An engine can look normal under one operating condition and abnormal under another.

Those cross-condition differences may reveal deterioration earlier than a simple average.

The Most Interesting Feature Family: Normal Behaviour Residuals

One part of the pipeline became particularly interesting.

Instead of directly asking:

What is the sensor value?

I tried asking:

What should the sensor value have been under these operating conditions?

Suppose altitude, Mach, and ambient temperature suggest that T45 should be around:

520

but the actual measured value is:

545

Then:

Residual = 545 - 520 = +25

That +25 may be a better health indicator than the raw temperature itself.

Conceptually:

Operating conditions
        ↓
Normal Behaviour Model
        ↓
Expected sensor response
        ↓
Actual - Expected
        ↓
Health residual

I built residual features for variables such as:

• T45
• T3
• core speed
• Ps3

Then I created additional features from those residuals:

current residual
rolling residual mean
residual slope
residual change from start of window

This moved the model closer to condition-adjusted health monitoring rather than simple sensor regression.

Validation: Hold Out Entire Engines

With only four labelled engines, validation becomes tricky.

A random train-test split would be misleading.

Why?

Because two overlapping windows from the same engine can look almost identical.

If one lands in training and another in validation, the model may effectively be tested on something it has already seen.

So I used leave-one-engine-out validation.

For example:

Train:
ESN101
ESN102
ESN103

Validate:
ESN104

Then rotate.

This asks a much more meaningful question:

Can the model generalize to an entirely unseen engine?

That is much closer to what the challenge actually requires.

The Baseline Was Intentionally Boring

Before using a machine learning model, I built a simple median baseline.

Its job was not to be clever.

Its job was to answer:

Can the real model beat something extremely simple?

The median baseline produced an official score of:

117.15

That gave me a clean reference point.

Why I Used Three Separate Models

The three maintenance targets are not really the same problem.

Water wash is relatively frequent.

HPC and HPT shop visits happen over much longer time horizons.

So I trained separate models for:

Cycles_to_WW
Cycles_to_HPC_SV
Cycles_to_HPT_SV

The first model family I used was LightGBM.

But instead of ordinary regression, I used quantile regression.

That decision came directly from the competition metric.

The Metric Changed the Entire Modeling Strategy

This was probably the most important insight from the challenge.

The competition does not penalize early and late predictions equally.

Suppose the true remaining life is:

100 cycles

Prediction A:

80

Prediction B:

120

Both are wrong by 20 cycles.

But prediction B is late.

It says maintenance is further away than it really is.

The challenge penalizes that more heavily.

So the best model is not necessarily the one with the lowest MAE.

It may be better to be slightly conservative.

That is why I tested quantiles:

0.30
0.35
0.40
0.45
0.50

The lower the quantile, the more conservative the prediction.

Interestingly, 0.30 performed best for all three targets.

I Then Added Another Conservative Adjustment

After choosing the 0.30 quantile, I tested whether a small negative offset could improve the official score further.

For example:

Raw HPC prediction = 4500
Offset = -560

Final prediction = 3940

The selected offsets were:

This intentionally makes the model predict maintenance earlier.

That would be undesirable if the goal were unbiased RUL estimation.

But under this specific scoring function, it can be rational.

The First Result Was Promising

The median baseline scored:

117.15

The LightGBM quantile model with calibration scored:

26.44

That is roughly a 4.4× improvement on the metric that matters.

The target-level results looked like this:

HPC was clearly the hardest target.

And that also made intuitive sense.

There are fewer HPC maintenance events in the training data, so the model has fewer examples of what an approaching HPC shop visit actually looks like.

The Most Counterintuitive Result: Worse MAE, Better Competition Score

Water Wash gave me a result that initially looked strange.

The median baseline had an MAE of roughly:

250 cycles

The LightGBM model had an MAE of roughly:

309 cycles

So by MAE, the model was worse.

But the official scores were:

Median baseline: 42.48
LightGBM:       19.95

Why?

Because the median is relatively unbiased.

It predicts late about half the time.

The LightGBM model is intentionally early most of the time.

Roughly:

76% early predictions

Since late mistakes are more expensive, the conservative model wins under the challenge metric.

That was one of the biggest lessons from the experiment:

A model is only “better” relative to the decision objective you actually care about.

Where the Current Pipeline Is Still Weak

The first result is strong enough to be interesting.

But I would not yet call 26.44 a fully trustworthy estimate of generalization.

There are a few important reasons.

Learning 1: The Whole Pipeline Must Live Inside Cross-Validation

The Normal Behavior Model was fitted using early-life training data.

That sounds safe.

But there is a subtle problem.

If I fit that residual model using early-life data from all four engines, and then hold out one engine for validation, the held-out engine has still influenced the feature transformation.

No labels leaked.

But information did.

The correct approach is:

Outer validation engine:
ESN104

Fit residual model only on:
ESN101
ESN102
ESN103

Then apply it to:
ESN104

This is a broader machine learning lesson:

Cross-validation should wrap the entire learned preprocessing pipeline, not just the final estimator.

Learning 2: My 26.44 Score Is Partly an Optimization Score

I used out-of-fold predictions to:

• choose the best quantile
• choose the additive offset
• report the final score

That means the same validation predictions influenced model decisions and performance reporting.

The estimate is useful.

But it is not fully untouched.

The cleaner solution is nested cross-validation.

What Nested Validation Looks Like

Imagine I want to evaluate Engine 104.

I keep it completely untouched.

Then I use:

101
102
103

to determine:

• best quantile
• best features
• model hyperparameters
• calibration offset

Only after those decisions are frozen do I make predictions for Engine 104.

So the structure becomes:

Outer holdout: 104

Inner development:
101 / 102 / 103
        ↓
choose model
        ↓
retrain on 101 + 102 + 103
        ↓
predict 104

Then repeat for every engine.

At the end, concatenate all untouched outer predictions and calculate one final score.

That score will be much more meaningful than the current development score.

Learning 3: 2,468 Windows Does Not Mean 2,468 Independent Examples

My training windows overlap heavily.

For example:

Window A
0 ---------------- 1500

Window B
   30 ---------------- 1530

Those two windows share almost all of their observations.

So while the model technically sees thousands of windows, the true amount of independent information is much smaller.

This matters because it can create a false sense of sample size.

The next experiment should test different strides:

30
100
250
500 cycles

If the model remains strong even when windows overlap much less, that is a much better sign that it is learning genuine degradation behaviour.

Learning 4: Quantile and Offset Are Really the Same Decision Layer

Right now the pipeline does this:

choose quantile
      ↓
choose offset

But both mechanisms push the prediction earlier.

So:

q = 0.30
offset = -560

might be similar to:

q = 0.22
offset = 0

A better formulation would be:

Final Prediction
=
a × QuantilePrediction + b

Then optimize:

quantile
a
b

together.

That makes the decision layer more explicit and may generalize better.

Learning 5: WW, HPC, and HPT Probably Need Different Feature Strategies

The first pipeline uses broadly similar features for all three targets.

That is convenient.

But probably not optimal.

For Water Wash, the model may benefit most from recent changes:

short rolling slope
temperature reset
recent residual trend

For HPC, slow degradation may matter much more:

full-window slope
first-quarter vs last-quarter shift
long-term health-index change

For HPT, another combination may be better again.

So the next pipeline should ask:

Which feature family is best for each maintenance mechanism?

rather than:

Which one feature matrix should I use for everything?

Learning 6: HPC Is the Problem I Would Focus on Next

HPC currently has the weakest score:

35.18

That makes it the highest-value target for improvement.

I would test features such as:

median(last 20% of window)
-
median(first 20% of window)

along with:

• long-window slope
• residual slope
• trend curvature
• PCA health-index movement
• cross-snapshot deterioration
• time spent above abnormal residual thresholds

These features may capture slow degradation better than short rolling statistics.

Learning 7: The Normal Behavior Model May Be More Valuable Than the Final Regressor

The residual approach deserves much more attention.

Raw sensor prediction asks:

What is T45?

Condition-adjusted monitoring asks:

How abnormal is T45 under these operating conditions?

That second question is much closer to how real equipment-health monitoring is often approached.

A future version could turn residuals into their own degradation system:

Residual now
Residual rolling mean
Residual slope
Residual volatility
Residual change from window start

This may become especially useful for HPC and HPT.

Learning 8: Try Cat-Boost Before Reaching for Deep Learning

It is very tempting to say:

This is a sequence problem. Let’s use an LSTM.

But there are only a few independent engines.

Thousands of rows do not magically create thousands of independent assets.

So before moving to LSTM, GRU, or Transformers, I would first try Cat-Boost on the exact same validation folds.

Then compare:

LightGBM
vs
CatBoost

If they make different mistakes, there may be value in blending them.

For example:

Final prediction
=
w × LightGBM
+
(1 - w) × CatBoost

The weight w should be learned from out-of-fold predictions rather than chosen manually.

Learning 9: 700 Features Need to Earn Their Place

The current feature matrix has roughly 700 columns.

That is not automatically a good thing.

The next step should include feature-family ablation.

For example:

Model A
Raw sensors only

Model B
Raw + rolling features

Model C
Raw + rolling + residuals

Model D
Raw + rolling + residuals + maintenance features

If one feature family can be removed with no loss in score, the model becomes:

• simpler
• faster
• easier to explain
• less likely to overfit

That is usually a win.

What the Next Version Should Look Like

The next iteration of the pipeline would be more rigorous:

Raw data
   ↓
Profile
   ↓
Leakage audit
   ↓
Outer engine holdout
   ↓
Fit all learned preprocessing
on outer-training engines only
   ↓
Create 1500-cycle windows
   ↓
Build causal features
   ↓
Inner validation
   ↓
Optimize:
features
LightGBM/CatBoost
quantile
calibration
   ↓
Freeze choices
   ↓
Retrain on outer-training engines
   ↓
Predict untouched outer engine
   ↓
Repeat
   ↓
Calculate nested validation score
   ↓
Retrain final pipeline on all labelled engines
   ↓
Generate test predictions

That pipeline is slower.

But the final score means much more.

The Bigger Lesson

The most useful lesson from this challenge has nothing to do with Light-GBM.

It is that predictive maintenance is often less about choosing a fancy model and more about getting five things right:

How the data is represented
+
What information exists at prediction time
+
How degradation is encoded
+
How validation mimics reality
+
What the business metric actually rewards

The first version of the pipeline moved the internal score from:

117.15

to:

26.44

That is encouraging.

But the next goal is not simply:

Get below 26.

The next goal is:

Make sure 26 means what I think it means.

That means stronger validation, cleaner fold-specific preprocessing, less correlated training windows, target-specific features, model comparison, and better out-of-fold calibration.

If the score remains strong after those changes, then the result becomes much more interesting.

It stops being just a competition model.

It starts looking like a reusable blueprint for real predictive-maintenance systems where equipment age is hidden, operating conditions change constantly, and only a short recent history is available.