ENGINEERING NOTE / RECOVERY
Recovery is a sequence, not a switch.
Equipment restoration, service recovery and verification are different events. A synthetic experiment makes the gap visible.
A green component is only one observation.
When equipment becomes available again, it is tempting to describe the system as recovered. That description can hide the state left by the interruption: accumulated heat, a depleted reserve, unavailable applications or an unfinished restart sequence. Restoration changes the available equipment. Recovery requires evidence about the service.
In a small synthetic dependency model, a supply failure removes one cooling unit while the other electrical path maintains the load. Restoring the supply returns its cooling unit immediately. This is deliberately optimistic: the model contains no equipment start delay or interlock. Even with that assumption, accumulated heat takes time to remove.
- 30 sSupply A lost
- 150 sEquipment restored · 26.20°C
- 240 sWindow ends · 24.10°C
The temperature remaining above the initial 24°C makes the gap between restoration and recovery visible.
Make the gap visible.
Restore the supply at 150 seconds. At that point, the 360 kW load has accumulated a 2.2°C rise while only 250 kW of cooling was available. Both units together can now remove 500 kW, leaving 140 kW of spare capacity. Removing that accumulated heat takes about 94 seconds, slightly beyond the four-minute window of the model.
Restore it at 60 seconds instead. The fault lasts 30 seconds, so the accumulated thermal rise is only 0.55°C. The equipment restored is identical. What changes is the state the recovery must work through.
Try it in One Plant, where a repair can arrive before or after the unit trips: let the duty pump fail with nobody starting the standby, then repair it after three minutes, and again after six. The equipment returns both times. Only one of them is a recovery.
Repair after the trip in One Plant →Define what “recovered” means.
This model calls the thermal state recovered when it returns within 0.05°C of the starting temperature. The threshold is explicit and illustrative. It is not a service-level target, a measured RTO or proof that a physical site is ready to operate.
For a real recovery exercise, the acceptance conditions depend on the service: supporting systems available, applications responding, required data accessible, and the operating team able to resume the intended activity. A component-level check and an end-to-end check answer different questions.
Record the sequence, not just the duration.
A useful exercise captures what became unavailable, the dependencies restored, the order in which they returned, and how each acceptance condition was demonstrated. The event sequence makes an unexpected delay understandable and gives the next exercise a testable improvement.
The model's restoration is idealized and its thermal state is synthetic. Read its source on the model code page before interpreting the result. The verification notebook shows what the numerical checks actually establish.
Terms used on this page
- RTO (recovery time objective)
- The longest a service may stay unavailable after a failure before the interruption becomes unacceptable.