The core of designing a comparative experiment is controlling a single variable and quantifying the comparison results. You can implement this by following the process of "experimental group - control group → controlling variables → designing test scenarios → statistical indicators → result judgment." The specific steps are as follows:
1. Experimental Grouping (Clearly Define the Control Relationship) Set up 2-3 comparison groups to clearly demonstrate the differences between algorithms:
Control Group 1: Traditional fixed threshold comparison algorithm (baseline, existing solution)
Experimental Group: The optimized algorithm to be verified (such as the "staged dynamic + 3σ + EWMA combined algorithm" you want to test)
Optional Control Group 2: A single optimized algorithm (such as only staged dynamic threshold, to verify the adaptive update gain)
2. Controlling irrelevant variables to ensure fairness. The following conditions must be fixed throughout the experiment to eliminate interference from non-algorithm factors:
Consistent hardware: Use the same pair of sensors, the same mounting position, and the same set of acquisition PLCs; do not change the hardware.
Consistent operating conditions: Test on the same production line, with the same product and the same process parameters; do not change the product or adjust the process.
Consistent testing cycle: Both groups are tested continuously for the same duration (e.g., each runs for 2 weeks, covering a sufficient number of production cycles).
3. Design three types of test scenarios to cover all performance dimensions. Design scenarios specifically to comprehensively verify the algorithm's capabilities:
(1) Conventional operating condition anti-interference test (verifying false alarm rate)
Test method: Normal production, continuously record all alarms, manually verify the authenticity of each one.
Statistical indicators: Calculate the false alarm rate of the two groups and observe the percentage decrease in the false alarm rate of the optimized algorithm.
Expected conclusion: The false alarm rate of the optimized algorithm should decrease by at least 60% compared to the fixed threshold.
(2) Fault simulation test (verifying missed alarm rate and response speed) Simulate three common industrial sensor faults manually; repeat the test 3 times for each group:
Table: Simulated Fault Type Simulation Method Verification Indicators Instantaneous Zero-Point Drift Add 0.1%FS instantaneous offset to the output of one of the sensors, and recover after 1 second. Verify whether there is a false alarm (normally there should be no alarm). Continuous Slow Aging Add 0.01%FS offset to the output of one sensor every day, accumulating to 0.1%FS. Verify whether there is a timely alarm and whether there is a missed alarm. Step Drift Add 0.2%FS continuous offset to the output of one sensor. Verify the alarm response delay and whether it can be detected in time. Acceptance Criteria: The optimized algorithm detects all continuous faults 100%, with no missed alarms, response delay ≤10 seconds, and no false alarms for instantaneous interference.
(3) Long-Term Aging Adaptability Test (Verify Long-Term Stability) Test Method: Run continuously for 3 months without manual adjustment of parameters, and calculate the false alarm rate once a month. Statistical Indicator: Calculate the change rate of the false alarm rate from the initial value to 3 months later. Acceptance Criteria: For the optimized algorithm with adaptive updates, the change rate of the false alarm rate is ≤2%, and the change rate of the fixed threshold algorithm usually exceeds 5%.
4. Data Statistics and Result Determination
After the experiment, the data was compiled, and performance was determined according to the following rules:
Core Indicators Prioritized: If the optimized algorithm's false alarm rate is ≤5%, and it is ≥60% lower than the control group, while the false alarm rate is ≤1%, then its performance is considered superior.
Long-Term Stability Verification: If the false alarm rate fluctuation is ≤2% after 3 months, it indicates that the adaptive mechanism is effective and its long-term performance is better than the control group.
Engineering Practicality Supplement: Statistical analysis of debugging time and maintenance frequency; a reduction of ≥30% in maintenance frequency after optimization indicates better practicality.
Experimental Precautions
Do not modify both hardware and algorithm simultaneously. Only the algorithm should be the sole variable; otherwise, the results will be inconclusive.
The testing period should not be too short; it should run for at least 1-2 weeks to cover sufficient production cycles for the data to be statistically meaningful.
All alarms must be manually verified. Alarms output by the algorithm should not be directly treated as real data to avoid statistical bias.
