I. Rapid Recovery from Physical Network Failures
If the network cable or RJ45 connector fails, replace it with a spare shielded industrial network cable, re-crimp the RJ45 connector, and verify that the link indicator light on the network port is functioning properly.
Switch Port Failure: In an effort to re-establish connectivity, promptly transfer the communication network cable to an unused, spare port, reconfigure the port's VLAN, and bypass the faulty port.
Packet Loss Due to Electromagnetic Interference: Replace the existing cable with a high-quality shielded industrial network cable, modify the cable routing to prevent it from passing through power lines, and promptly decrease the link packet loss rate.
Network Loop/Broadcast Storm: Disable the switch loop detection function, disconnect redundant links, and release the saturated network bandwidth.
II. Rapid Recovery from Communication Protocol Failures
Protocol Version Incompatibility: Restore the handshake connection by upgrading or downgrading the OPC UA/MQTT driver version on one end to match the firmware version on the other end.
Firewall-Blocked Port: Implement a rule in the industrial firewall to permit the dedicated port for RC calibration data. This will enable communication to be re-established without the need to restart the firewall.
Security Policy Incompatibility: To prevent the rejection of the handshake connection, it is necessary to unify the security policies and encryption levels of the OPC UA on both endpoints. Modify the conflicting IP address to an available network segment, reconfigure routing rules, and restore network connectivity at both endpoints in the event of an IP address conflict.
III. Rapid Recovery from Data Mapping Failures
Point Address Misalignment: To eliminate the necessity for manual register address modification, import a pre-backed-up RC parameter point mapping configuration file to overwrite incorrect configurations with a single click.
Data Format Mismatch: In order to resolve distorted characters in the data, invoke a preset RC calibration data format conversion script to automatically align the numerical types and decimal places on both sides.
Deviation from the Benchmark File Version: Complete version alignment within one minute by executing a one-click synchronisation procedure to transfer the most recent RC calibration benchmark parameters from the MES end to the SCADA end.
Data Semantic Missing: Ensure that synchronised data can be directly used for process judgement by unifying the binding of data quality identifiers to RC calibration data.
IV. Rapid Recovery from Service Operation Failures
Synchronisation Service Process Crash: Restart the SCADA data acquisition service and MES synchronisation scheduling service with a single click, without restarting the entire machine, thereby restoring service operation within 30 seconds.
Message Queue Backlog: Temporarily clear non-core historical production data messages to prioritise the consumption channel for RC calibration data and alleviate queue congestion swiftly.
Exhausted database connection pool: Restart the database connection pool service, release idle connections, and immediately recommence write operations for synchronised data.
Insufficient system resources: Temporarily suspend non-core background statistics tasks to free up CPU and memory resources, thereby ensuring priority scheduling for RC calibration data synchronisation tasks.
V. Rapid Recovery from Concurrent Timing Faults
Synchronisation request congestion: To prevent the timing pandemonium that can result from concurrent conflicts, it is recommended that the upload times of RC calibration data from multiple devices be staggered.
Excessive timestamp deviation: Enable the PTP/NTP unified time synchronisation mechanism to regulate the time deviation between MES and SCADA within 1ms, thereby guaranteeing the precise matching of interrupted data transmission.
Synchronisation tasks are of low priority. To prevent the preemption of system resources by ordinary production data acquisition tasks, the priority of RC calibration data synchronisation tasks should be set to the highest level.
High polling load: Reduce system load by enabling the data change notification mechanism to initiate synchronisation only when RC calibration data changes.
VI. Automatic recovery mechanism for backups
Immediately activate audible and visual alarms when the synchronisation link is anomalous, thereby identifying potential issues in advance, by deploying a real-time monitoring dashboard for synchronisation status.
Establish an automatic resume mechanism that will automatically retransmit any missing data following a network recovery, without the need for manual intervention.
Regularly conduct comprehensive RC calibration data reconciliation to identify data discrepancies early and prevent the fault from escalating and affecting production.
This comprehensive recovery solution has the potential to reduce the average repair time for RC calibration data synchronisation failures to less than 15 minutes, thereby reducing the impact of synchronisation interruptions on production.
