Arbitration States of Multi-Threaded Application Calculation Engine

The Application Calculation Engine (ACE) is part of the RealTime Service and will therefore change how it operates based on the arbitration state of RealTime. As a result, when we refer to the ACE as running in Hot mode and Standby mode, this is actually referring to the state of RealTime. Since the ACE is part of an arbitrated service, it must handle running in these two modes and switching between them during failovers, cold starts and shutdowns. This section discusses how the ACE handles arbitration states.

When RealTime starts up, it will run the ACE Process. During the ACE startup, PubSub subscriptions are obtained for the ACE code (acecode) and ACE Configuration (aceconfig) databases as well as for several other RealTime database (RTDB) tables. These subscriptions are used to inform the ACE of configuration changes to the ACE records and to keep the input data consistent with the database. A Checkin Thread runs continuously and monitors the arbitration state of RealTime. When RealTime has fully started up and has become Hot, the ACE detects this and proceeds with starting up three groups of threads. One of these threads is a scan thread that checks for the ACE procedures that can be run and queues them up. Another is a group of execution threads that read from this queue and execute the procedures. Monitoring threads are also started to check for timeouts and deadlocks. While RealTime is Hot, the ACE continuously executes the ACE procedures.

In general, enterprise systems have a dual-hosted service that contains a second server running in Standby mode. As such, the ACE must recognize when RealTime is running in Standby mode and act accordingly. While operating in Standby mode, the ACE has most of its threads shut down. ACE does not execute any ACE procedures while in Standby mode. While RealTime is in Standby mode, only the Checkin Thread runs, which also monitors the arbitration state of RealTime. The ACE sits idle, signaling to the process monitor that it is waiting for the system to become Hot. The PubSub subscriptions are still active however, so the Standby ACE still remains up-to-date with changes to the RealTime aceconfig and acecode databases.

When standing down, ACE checks the arbitration state periodically, at a frequency controlled by the IDLE_SLEEP timeout. When it detects that the RealTime service is no longer Hot, the ACE will request a shutdown of all threads and stops queuing procedures for execution. The shutdown waits for a period of time (configured in the SHUTDOWN_TIMEOUT variable) for any running ACE procedures to complete. Any procedures that are not completed are aborted and the following error message is logged in the oasErrLog, where <name> is the name of the interrupted record:

aceconfig <name> interrupted during execution as a result of a shutdown timeout