Monitoring and Troubleshooting a Cluster

This chapter presents general information for monitoring and troubleshooting an HACMP for Linux configuration.
This chapter contains the following sections:
•Problem Determination Tools
•Viewing Cluster Information (clstat) in WebSMIT
•Useful Commands
•Logging Messages
•Solving Common Problems with Networks and Applications.
Problem Determination Tools
WebSMIT Problem Determination Tools menu has a set of tools for troubleshooting and recovering from problems that may arise in a cluster environment.
The Problem Determination Tools panel in WebSMIT includes:
•View Current State. WebSMIT displays cluster information using a slightly different layout and organization. Cluster components are displayed along their status. Expanding the item reveals additional information about it, including network, interfaces and active resource groups.
•HACMP Log Viewing and Management. Contains utilities that display or manage logs maintained by HACMP. These include the log file named hacmp.out, which keeps a record of all of the local cluster events as performed by the HACMP event scripts. These HACMP event scripts automate many common system administration tasks, and, in the event of a failure, will manage HACMP and system resource to provide recovery.
•Recover From HACMP Script Failure. Contains a command that HACMP will run to recover from a script failure. This is useful if the Cluster Manager is in reconfiguration due to a failed event script. Use this option after having manually fixed the error condition.
•Restore HACMP Configuration Database from Active Configuration.
Viewing Cluster Information (clstat) in WebSMIT
With HACMP 5.4.1, you can use WebSMIT to:
•Display detailed cluster information
•Navigate and view the status of the running cluster
•Configure and manage the cluster
•View graphical displays of sites, networks, nodes and resource group dependencies.
Useful Commands
You have these additional utilities:
•To view the resource group location and status, use the clRGinfo command.
•To view the service IP label information, run the ifconfig command on the node that currently owns the resource group.
For a list of commands supported in HACMP for Linux, see Command Reference in Appendix A: Command Reference and the clinfo Utility.
Logging Messages
HACMP for Linux uses the standard logging facilities for HACMP. For information about logging in HACMP, see the HACMP for AIX Troubleshooting Guide.
To troubleshoot the HACMP operations in your cluster, use the event summaries in the hacmp.out file and syslog.
The system logs messages into the following files:
•/tmp/clstrmgr.debug
•/tmp/cspoc.log
•/tmp/clappmond
•/tmp/hacmp.out
•/usr/es/adm/cluster.log
•/var/hacmp/clcomd/clcomd.log
•/var/hacmp/clcomd/clcomddiag.log
•/var/hacmp/log/clutils.log
•/usr/es/sbin/cluster/wsm/logs/wsm_smit.*
•/websmit/logs/wsm_smit.*
•/usr/es/sbin/cluster/snapshots/*
Collecting Cluster Log Files for Problem Reporting
To view the system files and log files as they are collected in an archive file:
1.In WebSMIT, go to the Collect Cluster log files for Problem Reporting menu.
2.Type or select values in entry fields.
3.Use an appropriate Linux tool to extract or view the archive file. The archive file contains the log and system files.

Resetting Cluster Tunables

You can change the settings for a list of tunable values that were altered during cluster maintenance and reset them to their default settings, or installation-time cluster settings. The installation-time cluster settings are equal to the values that appear in the cluster after installing HACMP from scratch.
Note:Resetting the tunable values does not change any other aspects of the configuration, while installing HACMP removes all user-configured configuration information including nodes, networks, and resources.
To reset the cluster tunable values:
1.Stop the cluster services.
2.Log in to a URL where WebSMIT is installed. The browser window displays the top-level WebSMIT screen.
3.In WebSMIT, select Extended Configuration > Extended Topology Configuration > Configure an HACMP Cluster > Reset Cluster Tunables and press Continue.
Use this option to reset all the tunables (customizations) made to the cluster. For a list of the tunable values that will change, see the section Listing Tunable Values. Using this option returns all tunable values to their default values but does not change the cluster configuration. HACMP takes a snapshot file before resetting. You can choose to have HACMP synchronize the cluster when this operation is complete.
4.Select the options as follows and press Continue:
Synchronize Cluster Configuration
If you set this option to yes, HACMP synchronizes the cluster after resetting the cluster tunables.
5.HACMP asks: “Are you sure?”
6.Press Continue.
HACMP resets all the tunable values to their original settings and removes those that should be removed (such as the nodes’ knowledge about customized pre- and post-event scripts).
Resetting HACMP Tunable Values using the Command Line
We recommend that you use the SMIT interface to reset the cluster tunable values. The clsnapshot -t command also resets the cluster tunables. This command is intended for use by IBM support. See the man page for more information.
Listing Tunable Values
You can change and reset the following list of tunable values:
•User-supplied information.
•Network module tuning parameters, such as, failure detection rate, grace period and heartbeat rate. HACMP resets these parameters to their installation-time default values.
•Cluster event customizations, such as, all changes to cluster events. Note that resetting changes to cluster events does not remove any files or scripts that the customization use; it only removes the knowledge HACMP has of pre- and post-event scripts.
•Cluster event rule changes made to the event rules database are reset to the installation-time default values.
•HACMP command customizations made to the default set of HACMP commands are reset to the installation-time defaults.
•Automatically generated and discovered information.
Generally users cannot see this information. HACMP rediscovers or regenerates this information when the cluster services are restarted or during the next cluster synchronization.
HACMP resets the following:
•Local node names stored in the cluster definition database
•Netmasks for all cluster networks
•Netmasks, interface names and aliases for disk heartbeating (if configured) for all cluster interfaces
•SP switch information generated during the latest node_up event (this information is regenerated at the next node_up event)
•Instance numbers and default log sizes for the RSCT subsystem.
Understanding How HACMP Resets Cluster Tunables
HACMP resets tunable values to their default values under the following conditions:
•Before resetting HACMP tunable values, HACMP takes a cluster snapshot. After the values have been reset to defaults, if you want to go back to your customized cluster settings, you can restore them with the cluster snapshot. HACMP saves snapshots of the last ten configurations in the default cluster snapshot directory, /usr/es/sbin/cluster/snapshots, with the name active.x.odm, where x is a digit between 0 and 9, with 0 being the most recent.
•Stop cluster services on all nodes before resetting tunable values. HACMP prevents you from resetting tunable values in a running cluster.
In some cases, HACMP cannot differentiate between user-configured information and discovered information, and does not reset such values. For example, you may enter a service label and HACMP automatically discovers the IP address that corresponds to that label. In this case, HACMP does not reset the service label or the IP address. The cluster verification utility detects if these values do not match.
The clsnapshot.log file in the snapshot directory contains log messages for this utility. If any of the following scenarios are run, then HACMP cannot revert to the previous configuration:
•cl_convert is run automatically
•cl_convert is run manually

System Management (C-SPOC) Tasks

Use the System Management (C-SPOC) panel in WebSMIT to configure from one node the resources that are shared among nodes. System Management utility of HACMP lets you administer many aspects of the cluster and its components from one Cluster Single Point of Control (C-SPOC). By automating repetitive tasks, C-SPOC eliminates a potential source of errors, and speeds up the cluster maintenance process.
In WebSMIT, you access C-SPOC using the System Management (C-SPOC) menu.
In this panel, you can do the following tasks from one node:
•Manage HACMP services, or start and stop cluster services: Cluster Manager (clstrmgr) and Cluster Information (clinfo).
•HACMP Communication Interface Management. Manage the communication interfaces of existing cluster nodes using C-SPOC.
•HACMP Resource Group and Application Management Provides menus to manage cluster resource groups and analyze cluster applications.
•HACMP Log Viewing and Management. Manage, view, and collect HACMP log files and event summaries.
Starting HACMP Cluster Services
To start HACMP cluster services:
1.Log in to a URL where WebSMIT is installed. The browser window displays the top-level WebSMIT screen.
2.In WebSMIT, select System Management (C-SPOC) > Manage HACMP Services > Start HACMP Services and press Continue.
For detailed instructions, see the HACMP on AIX Administration Guide.
Stopping HACMP Cluster Services
To stop HACMP cluster services:
1.Log in to a URL where WebSMIT is installed. The browser window displays the top-level WebSMIT screen.
2.In WebSMIT, select System Management (C-SPOC) > Manage HACMP Services > Start HACMP Services and press Continue.
For detailed instructions, see the HACMP on AIX Administration Guide.
Managing Resource Groups and Applications
To manage resource groups and applications:
1.Log in to a URL where WebSMIT is installed. The browser window displays the top-level WebSMIT screen.
2.In WebSMIT, select System Management (C-SPOC) > HACMP Resource Group and Application Management and press Continue.
Viewing and Managing Logs
To view and manage logs:
1.Log in to a URL where WebSMIT is installed. The browser window displays the top-level WebSMIT screen.
2.In WebSMIT, select System Management (C-SPOC) > HACMP Log Viewing and Management and press Continue.
For detailed instructions, see the HACMP on AIX Administration Guide.