Cloverleaf auto-healing toolset
The auto-healing toolset for the auto-start environment facilitates Cloverleaf in transitioning runtime when failure happens on the primary node. The transition is made from the primary auto-start/Cluster node to the secondary auto-start/Cluster node.
Auto-Start scripts
The auto-start scripts in this topic are what a cluster management system or an OS service management calls to start CIS runtime instances; for example, engine, monitor daemon, host server, and others.
The scripts can also recover the runtimes from the last failover. The scripts also clean up PID files, MSI shared files, and unlock the Lock Manager and Raima databases.
By default, if these files are not explicitly assigned a separate path during installation, they are installed under HCIROOT/auto-start. The installer has updated the scripts to point them to its installation. With this, you can copy the entire directory for use.
These files are under the auto-start directory:
- as.hci.profile is under the auto-start directory. This file contains all properties required to configure the auto-start logic. Every property is explained inside the file.
- Cloverleaf also looks for auto-start.profile under
HCIROOT/server as an alternative to
as.hci.profile.
You can copy as.hci.profile and rename it to take effect. Properties defined inside HCIROOT/server/auto-start.profile receiver a higher priority, except for AS_HCI_ROOT, as the scripts use it to first find HCIROOT. The Cloverleaf installer updates it in as.hci.profile to point to its installation (without any additional configuration).
- as.hci.start.pl is under the auto-start directory. This is the entry point to invocation for the runtime startup. It is expected to be launched by the root user with the Perl installation inside CIS.
as.hci.start.pl usage:
as.hci.start.pl [-f profile] [-p] [-h] [-s] [-c]
Where:
-f profile
assigns the profile path for the start script. By default, the script attempts to locate as.hci.profile in the same directory.-h
starts the host server.-s
starts the security server.-c
starts the scheduler.-p
starts configured sites.
Notes:
By default, when not configured, -h
and -p
are "on".
This script works as a compound autostart script for integration
service.
When any one of -h
/-s
/-p
is explicitly
configured, the rest are "off" unless that one is also explicitly configured.
Service installation
Some HA systems reuse their OS service management when working for failover. The Cloverleaf installer on all supported OS platforms installs one or more CIS services. In this way, the CIS engine, monitor daemon, host server, and other runtime instances are brought up during OS startup.
- On Windows, a Windows NT Service is installed. Both the host server and auto-start script can be brought up by it after they are properly configured.
- On Linux and AIX, the CIS services can be installed by the
hciunixservice script under HCIROOT/sbin. The
script requires root privileges to fully work.
Usage:
hciunixservice -i h|s|e [auto-start script directory] [-c]
or
hciunixservice -r h|s|e
-i
installs the specified services. Additional auto-start scripts' install paths can be assigned, under which the Cloverleaf auto-start service is referred:-r
removes the specified services.-
h
is the host server service. -
s
is the security server service. e
is the cloverleaf scheduler service.c
is a special option for CentOS 7 and later. This tunes Systemd unit files.By default, assertions are made on HCIROOT for services and an error is reported to the system to mark the service as failure. Under cluster environment, HCIROOT is only available to the backup node when the major one is down. This option can be used to mute the alert. Systemd skips the service when HCIROOT is absent.
System Services
For AIX and Linux distribution versions that have not introduced Systemd as the first process, the hciunixservice script installs the services under SysV style. Initial scripts are linked to hostserverservice or securityserverservice under HCIROOT/sbin, according to how Cloverleaf is installed.
Linux comes with Systemd.
On Linux, hciunixservice installs these services:
- Cloverleaf Integration Server:
-
- cloverleaf-integration.target
- cloverleaf-autostart.service
- cloverleaf-hostserver.service
- Cloverleaf Security Server: cloverleaf-securityserver.service
For Cloverleaf Integration Server, cloverleaf-integration.target brings up the host server and auto-start services. These services bring up all expected monitor daemons and processes in configured sites.
The host server and auto-start are split into two services under Systemd. This utilizes the OS feature.
Host server is a stand-alone process in the system. A fork style service unit assists the system to ensure better control and to know the exact running status of the system.
Conversely, auto-start can create numerous processes. One service cannot monitor the status of all processes. The auto-start service is a "oneshot"-style service.