Mode switches and Age Watchdog

The Age Watchdog application monitors the ownership of the monitored point in order to prepare for any potential mode-switch.

When a point that is monitored by an ageWatchdog record changes ownership upon mode-switch, there are implications for the ageWatchdog record. In steady state conditions, the timestamps of monitored points on the non-owning system can become extremely out-of-date in comparison to the timestamps on the owning system. This is because the owning system actively polls and krunches the data; however, the default telemetry does not force replication on the non-owning system unless a state or value change is drastic enough to trigger it. Timestamps on the non-owning system can be as old as the last time that system had ownership. If monitored points continued to be sampled immediately after a mode-switch, it would result in points being falsely declared as old, and potentially false alarms, events, and staled remotes.

In order to prevent this, the Age Watchdog application monitors which system owns the monitored point, and when it takes ownership. It also uses a registry variable to configure a delay in sampling after a mode-switch occurs.

The figure above shows a point for which ownership changes upon mode-switch. During the day, meanalog (a point monitored by Age Watchdog) is owned by the ES system. During the night, a mode switch occurs. Immediately after the mode-switch Main takes ownership of meanalog, but its timestamp on the Main system is from the early morning. To prevent Age Watchdog sampling from occurring until the Main system has polled and krunched all points in its dataset, the registry variable is used to pause processing long enough that all timestamps should be up-to-date.

The RealTime settings key that tells Age Watchdog how long it should pause is called AGEWATCHDOG_SAMPLEDELAY_SECONDS. This overriden key is located in %DNADataRoot%\RealTime\config\Settings\RealTime.json and is defaulted to 100 seconds:

"AGEWATCHDOG_SAMPLEDELAY_SECONDS": 100

Notes: For information on overriding default configuration settings, refer to Modify configuration settings.

The minimum allowable value for this variable is 40 seconds. A RealTime service startup is required at the server (where changes to this variable are made), in order for the new setting to come into effect.

AGEWATCHDOG_SAMPLEDELAY_SECONDS is configurable on a per server basis to accommodate differing system load requirements and differing physical polling capabilities across geographic locations. For instance, imagine that in the example above the ES system is a central control system that is located in a city with good internet access, and Main is located in the mountains away from advanced technology. If Main has to use slower radio or satellite communications to poll remotes, it will take longer for meanalog to be krunched; therefore the AGEWATCHDOG_SAMPLEDELAY_SECONDS key should be set long enough to ensure that the point will be updated before Age Watchdog resumes sampling.