The message, Warning: Unable to determine local domain name for .., is a warning and will not cause Serviceguard commands to fail. However, it will generate similar messages in syslog.log repeatedly whenever a Serviceguard command is performed, to the consternation of the system administrators. It is better to configure the system to avoid these messages.

The message:

ERROR: Unable to determine a unique identifier for physical volume /dev/disk/disk1_p2 on node vihp35. Use pvcreate to give the disk an identifier.

_p2 indicates a boot partition. Verify lvlnboot produces reasonable output and that the device is properly partitioned.

Example:

# lvlnboot -v vg00
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/disk/disk3_p2 -- Boot Disk
Boot: lvol1 on: /dev/disk/disk3_p2
Root: lvol3 on: /dev/disk/disk3_p2
Swap: lvol2 on: /dev/disk/disk3_p2
Dump: lvol2 on: /dev/disk/disk3_p2, 0

Validate the primary partition.

Example:

# idisk -v /dev/rdisk/disk3

(produces no output)

The following is a comprehensive series of actions that separately may result in restoring Serviceguard command operations.

Treating the central message:

Unable to receive device query message from vihp35: Software caused connection abort

Serviceguard communicates with other nodes in the cluster using hacl and identd/auth ports (identified in /etc/services and /etc/inetd.conf). It is critical that these ports be operational in order to process Serviceguard commands. At a minimum, the following must be active:

$ netstat -a grep hacl
tcp 0 0 *.hacl-probe *.* LISTEN
tcp 0 0 *.hacl-cfg *.* LISTEN
udp 0 0 *.hacl-cfg *.*

If they are not, inspect and correct the above files and restart inetd. Also verify that this line is one of the first declared in /etc/hosts:

127.0.0.1 localhost loopback

If /var/adm/inetd.sec exists, ensure it does not deny these services for any node.

Edit /etc/hosts, and add _any_ IP that is permanently assigned to _any_ NIC on _any_ node, and then alias that IP to the simple hostname of the node.

Serviceguard uses networking ports to operate configuration commands so it must be able to associate the IP embedded in the request with the simple hostname, so it may use any cluster-common network to communicate with other nodes in the cluster. Example:

10.5.5.1 vihp35hb
10.5.5.2 vihp36hb
10.5.6.1 vihp35h1
10.5.6.2 vihp36h1
Change to:
10.5.5.1 vihp35hb.corp.<yourcorp>.com vihp35
10.5.5.2 vihp36hb.corp.<yourcorp>.com vihp36
10.5.6.1 vihp35h1.corp.<yourcorp>.com vihp35
10.5.6.2 vihp36h1.corp.<yourcorp>.com vihp36

This action is required per the Managing Serviceguard manual at the Name resolution section of the below-mentioned article:

Click here to refer to the Managing Serviceguard manual at http://www.hp.com/go/hpux-SG-docs.

Verify that LAN NIC failure has been recovered, on each cluster node.

Example:

$ grep failed /var/adm/syslog/syslog.log grep lan
Jun 12 02:12:44 vihp35 cmnetd[2878]: lan3 failed.
Jun 12 02:12:52 vihp35 cmnetd[2878]: lan1 failed.

If the user does not see a lanX recovered message, investigate whether the cable is detached, the network switch port is down, or the NIC is hardware down.

Example:

root@rxg16u09:PKGS/# lanscan
Hardware Station Crd Hdw Net-Interface NM MAC HP-DLPI DLPI
Path Address In# State NamePPA ID Type Support Mjr#
----<snipped>----
0/4/1/0/6/0 0x00237DA97FE5 3 UP lan3 snap3 3 ETHER Yes 119
0/1/2/1 0x00226493AF08 1 UP lan1 snap1 4 ETHER Yes 119
----<snipped>----

(Verify Hdw State = UP). If not up, consider resetting the NIC:

$ landiag
Enter command: lan
Enter command: ppa
Enter PPA Number. Currently 0: 1
Enter command: reset
Resetting LAN Interface to run selftest.

Edit /etc/nsswitch.conf to point to files before external name resolution resources so that Serviceguard commands are not delayed by problems with external sources.

Example: Change the following:

ipnodes: dns [NOTFOUND=return] files
services: nis [NOTFOUND=return] files

T:

ipnodes: files [NOTFOUND=return] dns
services: files [NOTFOUND=return] nis

This is also documented in the manual Managing Serviceguard.

Since inetd is involved in Serviceguard command processing, kill inetd and restart it (inetd -k ; inetd) to insure a fresh incarnation is running.

Serviceguard A.11.19 and A.11.20 use the same cluster manager engine and may start up to 6 cmclconfd -c processes when the node joins the cluster. Kill any additional cmclconfd processes that are started later. It will not affect Serviceguard or business applications. For that matter, killing all cmclconfd processes will not impair Serviceguard or the packages it is running because inetd will restart them as needed.

Example:

$ ps -ef grep cmclconfd grep -v grep sort -k 5,6
----<snipped>----
root 2767 1560 0 Jul 16 ? 0:00 cmclconfd -c
root 2955 1560 0 Jul 16 ? 0:00 cmclconfd -c
root 12591 1560 0 Jul 16 ? 0:00 cmclconfd -c
root 20554 1560 0 Jul 16 ? 0:00 cmclconfd -c
root 25164 1560 0 Jul 16 ? 0:00 cmclconfd -c
root 3665 1560 0 Nov 25 ? 0:00 cmclconfd -c
root 3679 1560 0 Nov 25 ? 0:00 cmclconfd -c
root 3744 1560 0 Nov 25 ? 0:00 cmclconfd -c
root 3745 1560 0 Nov 25 ? 0:00 cmclconfd -c
root 3749 1560 0 Nov 25 ? 0:00 cmclconfd -c
root 7256 1560 0 01:22:41 ? 0:00 cmclconfd -c
root 10866 1560 0 01:42:36 ? 0:00 cmclconfd -c
root 10973 1560 0 01:43:13 ? 0:00 cmclconfd -c
----<snipped>----

The example issue occurred in July, and the system joined the cluster in November, so all cmclconfd processes that were not started in November should be killed. Repeat on the sister nodes that experience Serviceguard configuration command troubles.

Use cksum to verify the checksum of the cluster binary file is the same across nodes in the cluster.

Example:

# cksum /etc/cmcluster/cmclconfig
1780696186 34528 /etc/cmcluster/cmclconfig

compare this number across nodes

Determine if cmquerycl runs without options. If it fails, fundamental hacl port activities are not completing. Test whether the identd port if working.

Example:

# netstat -an grep .5302 head -1
tcp 0 0 127.0.0.1.58401 127.0.0.1.5302 ESTABLISHED
----- ----
/
note

Port 58401 is a random port that Serviceguard picks and port 5302 is the port that awakens cmclconfd.

# telnet localhost 113
Trying..
Connected to localhost.
Escape character is '^]'.
58401, 5302 <--- enter the two ports
58401 , 5302 : USERID : UNIX :root <--- this is a successful connection
Connection closed by foreign host.

If the USERID is printed, the auth/identd daemon is functioning as it should.

If the problem has not been solved by the above actions, determine whether disabling Serviceguard's reliance on identd will restore Serviceguard configuration command operation. Edit /etc/inetd.conf on each node that fails to process Serviceguard commands and add the -i option as follows:

hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c -i
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -i -f /var/opt/cmom/cmomd.log -r /var/opt/cmom

Then restart inetd and test Serviceguard commands.

If the previous steps fail to alleviate the problem, it may be necessary to reboot the system(s) to clear a potential problem with the kernel. Perhaps now would be a good time to install needed patches.

Run swverify on each system, looking for permission problems with configuration files in /etc/ or /usr/lbin.

# swverify *

Report if any of the following files are missing on either of the cluster nodes:

# ll /var/adm/cmcluster/.*socket
srw------- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmcld_local_socket
srw-rw-rw- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmgmsd_local_socket
s------rw- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmnetd.ANY_USER.socket
srw------- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmnetd.root.socket
s------rw- 1 root root 0 Jun 12 09:01 /var/adm/cmcluster/.cmproxyd.ANY_USER.socket
srw------- 1 root root 0 Jun 12 09:01 /var/adm/cmcluster/.cmproxyd.root.socket
srw------- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmresourced.root.socket
srw------- 1 root root 0 Jul 9 13:06 /var/adm/cmcluster/.cmserviced.root.socket

If some are missing, consider restarting the cluster or rebooting.

Locate core files spawned by Serviceguard:

# ll /core /etc/cmcluster/core /etc/lvmconf/core /var/adm/cmcluster/core
# file <filepath> grep 'from 'cmclconfd'

If the user has the expertise, run a debugger to identify the stack trace:

# /opt/langtools/bin/gdb /usr/lbin/cmclconfd <core file>

If all of the above and rebooting do not remedy the condition, get the HP Customer Support Center involved.

Issues with cmclconfd may require debug logging. Add the following lines to /etc/cmcluster.conf:

CMCLCONFD_LOG_LEVEL=6
CMCLCONFD_LOG_FILE=/var/adm/cmcluster/cmclconfd.log

Kill inetd and all cmclconfd processes and immediately restart inetd (inetd -k ; inetd). This can be done while the node is running Serviceguard packages.

Add a test statement to syslog.

Example:

$ logger TESTING cmclconfd debug

Run the Serviceguard configuration commands that fail to get cmclconfd debug messages.

Tag the syslog file:

$ logger END test

Comment out the newly added lines in /etc/cmcluster.conf and restart inetd again and kill cmclconfd processes. Be prepared to send the debug logs to HP Support.

So this is working on all type of windows with one click.