What are the specific methods for statistical analysis of experimental data?

Four steps are involved in the specialized statistical analysis of experimental data: "Complete Recording of Raw Data → Manual Labeling and Classification → Calculation of Core Indicators → Significance Verification." Every stage is useful and guaranties exacting and trustworthy statistical outcomes:

 

Step 1: Complete Raw Data Recording (No Information Lost)

Each data point is recorded in the following way once the experiment has begun. Exporting it as a CSV or Excel table is advised:

Sensor A Raw Value Sensor B Raw Value Algorithm Output Deviation Alarm Triggered Remarks (Manual Labeling) Table Recording Time Process Stage

Injection 12.352 12.358 0.006 True Fault/False Alarm 2026-xx-xx xx:xx

Rules for Recording:

Every time an alarm is set off by the algorithm, it must be completely logged;

The deviation data only needs to be recorded once every hour during regular production without alerts (the whole data does not need to be stored for later 3σ statistics);

In order to calculate reaction delay, the fault injection time should be recorded independently during fault simulation testing.

 

Step 2: Classification by Hand (Differentiating Between True and False Alarms)

Once every alarm record has been finished, manually categorize each one individually:

Type of Alarm

Criteria for Judgment

Real Alarm (TP)

Either a real defect is manually injected, or there is a real sensor fault, installation deviation, or drift that exceeds tolerance (verified after swapping and verification).

False Alarm (FP)

Even when the sensor and installation are normal, an alarm was set off only because of interference or fluctuations.

Alarm Missed (FN)

There is a real problem (either manually introduced or verified), but the algorithm failed to sound the alert.

Note: Only "known genuine faults" in missed alarms must be counted; situations without faults are exempt. Just tally the "number of times a known fault did not trigger an alarm."

 

Step 3: Core Indicator Quantitative Calculation

Use the following formulas to calculate the four basic indicators, then compare the outcomes directly:

1. The rate of false alarms (most core)

Total Alarms (TP + FP) divided by False Alarms (FP) times 100% is the False Alarm Rate.

For instance, if there were 20 alerts in total and 3 of them were false alarms, the false alarm rate would be 3/20, or 15%.

2. Rate of False Alarms

Total Actual Faults (TP + FN) divided by False Alarms (FN) times 100% is the False Alarm Rate.

For instance, there were 10 real defects injected, 0 false alarms, and a false alarm rate of 0%.

3. Mean Reaction Time

To determine the single-attempt delay for each genuine alarm, compute the alarm time minus the fault occurrence time. The average response latency is then calculated by averaging all single-attempt delays.

4. Stability over time (false alarm rate change rate)

False alarm rate change rate = |False alarm rate after three months - initial false alarm rate|

For instance, the initial false alarm rate of 2%, the false alarm rate after three months of 2.8%, and the change rate of 0.8% satisfy the good criteria.

 

Step 4: Verifying the relevance of the results (avoiding random errors)

Use the chi-square test to determine whether the difference in false alarm rates between the two groups is random if it is not statistically significant:

Determine the chi-square value. If p is less than 0.05, it indicates that the optimized algorithm's reduction in false alarm rate is genuine and effective rather than a random fluctuation; if p is greater than 0.05, it indicates that the difference is not significant and that more testing time is necessary before conducting additional statistics.

If you are unfamiliar with the chi-square test, you can simply look at the "false alarm rate reduction percentage": It can be concluded that the performance is better and adequate for engineering scenarios if the optimized algorithm's false alarm rate is ≥ 60% lower than that of the original algorithm.