EN
A public aircraft dataset exposes limits in early fault detection
Pneuma: A Large-Scale Real-World Time Series Benchmark for Commercial Aircraft Predictive Maintenance · arXiv:2610.37563
In plain words
Pneuma is a public collection of flight measurements from commercial aircraft, paired with maintenance records that identify fault events. It also gives researchers shared tasks for predicting system behavior, spotting warning flights, and finding unusual flights.
Across nine bleed-air fault groups, models trained only on normal flights were much less effective at ranking warning flights than the best classifier trained with labeled warning examples. This result applies to the aircraft and faults in the collection.
The research
Can researchers tell whether a model recognizes early aircraft faults, or are apparent gains tied to private data and incompatible tests? Commercial-fleet studies have often used proprietary records and inconsistent tasks, so performance across real aircraft configurations remains hard to compare. A public, shared evaluation can make that question measurable.
Pneuma pairs commercial-aircraft flight records with maintenance-event labels and shared tests for forecasting, fault classification and anomaly detection.
Using maintenance-event labels, the authors define warning flights before a fault and compare classifiers trained on labeled warning examples with detectors trained only on normal flights. Both are judged by how many flagged flights are warnings once at least 10% of warning flights are captured.
Across nine aircraft-fault groups, the best normal-only detector scored far below the best classifier. This supports the limited conclusion that learning normal behavior alone was insufficient for strong warning detection in this benchmark.
The comparison covers nine bleed-air fault groups on four aircraft types, and the authors' explanation for the gap is an interpretation rather than a separate causal test. The findings do not establish performance on other aircraft systems or fault scenarios.
What the evidence shows
- A shared test for aircraft faults
Pneuma is a public dataset and standardized set of tests for predictive maintenance using measurements from in-service commercial aircraft.
The paper addresses a comparison problem: commercial-fleet research often relies on private records and inconsistent tasks or scoring, making it difficult to reproduce findings or compare methods across real aircraft and fault scenarios.
- Labels tied to flight events
The authors connect fault labels to maintenance-log events and use the associated historical flight records. They define warning flights as the 100 flights before a documented fault, while the normal pool excludes flights close to fault events.
Models are evaluated on chronological splits within each event, helping keep the order of flights intact.
- A common warning measure
For warning detection, supervised classifiers learn from labeled normal and warning flights; anomaly detectors learn from normal flights only. Both are scored by the precision among the highest-ranked flights at the point where at least 10% of warning flights have been captured. The score averages that precision over nine aircraft-fault groups and scales it to 0–100.
- Normal-only detection lagged
The best supervised classifier scored 60.22 on the aggregate measure, compared with 23.11 for the best anomaly detector trained only on normal flights. Because the approaches differ in training examples, objective and architecture, this comparison does not identify a single cause for the gap.
The authors suggest that normal-flight variation and subtle, overlapping fault symptoms can make normal-only detection difficult.
Deep
- Original paper authors
- Jiafeng Lin, Yutian Zhang, Huan Zhang, Yuxuan Wang, Jinyan Feng
- Submitted to arXiv
- License
- CC BY 4.0