The evaluation process for a dynamic threshold adjustment mechanism is not a one-time report, but a continuously running, automatically triggered, and auditable closed-loop system. Its core is to strongly correlate "early warning behavior" with "operational results," forming a data-driven self-optimization cycle. The following is a seven-step evaluation process validated in industry:
1. Define Evaluation Objectives and Scope
Objectives: Quantify the improvement effect of dynamic thresholds on false alarm control, missed alarm suppression, operational efficiency, and cost savings.
Scope: Limit the evaluation to the equipment group (e.g., "Welding Robot Production Line A"), time window (≥30 days), and data source (sensors + MES + work order system).
Benchmark: Use the static threshold system's operating period (first 30 days) as the baseline.
2. Establish Evaluation Data Acquisition Links
Table Data Type Source System Field Example Acquisition Frequency
Early Warning Event Prediction Platform Timestamp, Equipment ID, Threshold, Confidence, Trigger Characteristics Real-time
Maintenance Confirmation MES/CMMS True Fault, Fault Type, Maintenance Action, Time Consumption, Spare Parts Consumption After each work order closure
Equipment Status PLC/SCADA Operating Parameters, Downtime Records, Anomaly Codes Per Second/Minute
Operation Feedback Mobile APP Operator Marking: "False Alarm", "Missing Alarm", "Confirmed Fault" Real-time
Key Requirements: All data must be linked via a unique device ID and timestamp to ensure traceability.
3. Trigger Assessment Analysis and Trend Monitoring
Daily: Generate trend charts for F1-score, false alarm rate, and alert lead time.
Weekly: Perform sliding window performance degradation analysis (detect model drift).
Monthly: Run A/B comparison analysis (dynamic vs. static thresholds).
Anomaly Trigger: When the F1-score decreases by more than 5% for three consecutive days or the false alarm rate exceeds 7%, automatically initiate root cause analysis.
Output: Automatically generate a "Weekly Assessment Briefing," including trend charts and anomaly alerts.
4. Initiate Feedback Loop and Model Update
Triggering Conditions:
False alarm rate exceeds the threshold for 3 consecutive times
False negatives are manually confirmed
New fault modes are identified (cluster analysis reveals new feature combinations)
Execution Actions:
Automatically extract relevant data samples (including labels)
Trigger incremental learning task (LightGBM/XGBoost online update)
Generate a new threshold candidate set, and release it after confidence verification
Record adjustment log (see table below)
|
Timestamp |
Original Threshold |
New Threshold |
Reason for Adjustment |
Confidence Change |
Verification Result |
|
2026-04-10 03:12 |
85% |
92% |
3 consecutive false alarms (no maintenance) |
0.81 → 0.93 |
Confirmed as operating condition fluctuation |
|
2026-04-15 14:05 |
92% |
88% |
New fault mode detected (oil contamination) |
0.93 → 0.87 |
Maintenance confirmed as seal aging |
5. Output Standardized Assessment Reports
A "Dynamic Threshold Assessment Report" is generated quarterly, including:
Core Indicator Comparison Table (Dynamic vs. Static)
Cost Savings Analysis (Maintenance Costs, Downtime Losses, Spare Parts Consumption)
OEE Improvement Contribution Breakdown
Threshold Adjustment Log Summary
System Compliance Statement (Compliant with ISO 13374-1, IEC 60038)
Deliverables: PDF report + interactive dashboard (supports filtering by equipment, shift, and month)
6. Auditing and Continuous Optimization
Internal Audit: Semi-annual joint review and assessment of process integrity by O&M and IT
External Certification: Introducing third-party organizations to verify data links and indicator calculation logic
Optimization Directions:
Introducing causal inference models (e.g., DoWhy) to identify the root causes of false alarms
Exploring federated learning, sharing models across plants but not data
Ultimate Goal: To build an assessment process system that is unmanned, continuously evolving, auditable, and reusable.
