HACMP v3.1

ITEM: RTA000065101

According to the announcement of HACMP V3.1, there is a new mechanism           
that enables to control the trigger time at which a heart-beat                  
connection is considered dead (Fast Fallover). It's settable via SMIT.          
How does this work ?                                                            
I know that in HACMP V2.1, a node is considered dead after 12 missed            
Keep-Alive packets, and then a further 9 Keep-alive packets are sent.           
Does this mechanism change those numbers and in what way ??                     
With HACMP 3.1, you will be able, through SMIT, to set a Failure                
Detection Rate of Slow, Normal, or Fast independently for each network          
type that you have configured in your cluster. HACMP 3.1 will distinguish       
between "network types", such as Ethernet/IP, Token Ring, Target Mode           
SCSI or RS232, FDDI, etc. Each of these network types that you have in         
your cluster has its own Network Interface Module (or NIM) in each node,        
responsible for sending and receiving KAs, and detecting network                
failures. The effect of the Slow, Normal or Fast settings for the various       
network types are shown in the table below:                                     
                                    Slow       Normal        Fast               
                                    ----       ------        ----               
Ethernet/IP       KA Rate (secs)     1.0         0.5          0.5               
                   Missed KAs        12          12            9                
Token Ring        KA Rate (secs)     1.0         0.5          0.5               
                   Missed KAs        24          24           12                
TMSCSI/RS232      KA Rate (secs)     3.0         1.5          0.5               
                   Missed KAs         8           6            5                
SOCC              KA Rate (secs)     1.0         0.5          0.5               
                   Missed KAs        12          12            9               
FDDI              KA Rate (secs)     1.0         0.5          0.5               
                   Missed KAs        12          12            9                
SLIP              KA Rate (secs)     2.5         1.0          0.5               
                   Missed KAs        18          12           12                
Normal and Slow are the recommended settings. Slow is useful in configur-       
ations where temporary network or CPU load situations might otherwise           
cause false takeovers to occur. A Fast setting exposes you more to the          
danger of false takeovers. Node failure processing starts with the              
fastest network error detection.                                                
In HACMP 3.1, after a node detects an error, the nodes will send Message        
packets to each other to agree on the correct event to process next,            
similar to the "stabilization" period of HACMP 2.1. I do not know exactly       
at this point how long this stage takes, but I would expect it is              
similar to the stabilization period in HACMP 2.1.                               
Thank you for your detailed answer.                                             
I would like to ask some more questions, please.                                
First, the environment is made of two RS/6000 - 380, without                    
any shared disks. Only IP, and maybe HW address takeover.                       
The most important consideration is the time it takes to do                     
So, the first question would be:                                                
1. Taking into account that there are only two machines in the                  
cluster, is there any way to avoid sending the 9 KA for stabilization,          
or, at least changing the number to a smaller one ?                             
2. In HACMP V2.1 there are some "command line parameters" that one              
may change ( see lpp.doc from PTF U431606), like:                              
 - normal_ka_cycles  dealing with the time between normal KA's                  
 - speed_ka_cycles   dealing with the time between stabilizing KA's             
It seems that those two should be set to the same value.                        
Is this true, or may I set the speed_ka_cycles so that the                      
stabilizing KA would be sent faster then the normal KA's?                       
Are those parameters still available in V3.1?                                   
Thanks and Regards... Liviu                                                     
Actually the behavior of the cluster manager in HACMP 3.1 is different          
from that of HACMP 2.1 in how it detects and processes events. In HACMP         
2.1, you have two major stages of event detection and reaction:                 
  1. Event detection time - composed of the KA cycle time multiplied by         
      the "cyles to fail" parameter. These two parameters (cycle time           
      and cycles to fail) became configurable as of PTF U432018.               
  2. Synchronization time - 9 KA packets, which is not adjustable. This         
      is time to allow all cluster managers to agree before any cluster         
      events are generated.                                                     
In HACMP 3.1, you also have 2 stages of event detection and reaction, but       
they are slightly different.                                                    
  1. Event detection time - Also defined by KA cycle time multiplied by         
      cycles to fail. This is now adjustable by network type as described       
      in my previous response.                                                  
  2. Message time - I don't think there is really an official word for          
      this time, so I will call it "message time" for lack of a better          
      name. Instead of sending synchronization packets, each cluster            
      manager now sends "message packets" to every other node on all            
      network connections when it detects an event. This works as               
      as follows:                                                              
        a) New event message sent to all other nodes.                           
        b) Wait 2 seconds or until all nodes acknowledge.                       
            - Node down event is added for all nodes that do not                
        c) Each node votes on event to process. If there is disagreement,       
           the highest priority event is triggered, as follows:                 
             1. node failure                                                    
             2. network failure                                                 
             3. network join                                                    
             4. node join                                                       
        d) All nodes run their event scripts, with message packets sent         
           to synchronize execution (for instance in node joins).               
      This procedure is not adjustable by the user. The time taken will         
      vary depending on how many nodes there are, and how many events are       
      being detected. The shortest time it could take would be 2 seconds,      
      since if every other node does not acknowledge, the remaining node        
      will begin to process the highest priority event in its queue. The        
      time it takes for this though is for every node to vote on the next       
      event. Typically there will only be one event happening at a time,        
      so this should not be too long. In the case of a 2 node cluster, as       
      in your example, this time would really be 2 seconds at the most in       
      a node down. If the other node did not acknowledge, after 2 seconds       
      the remaining node starts processing the event. This new design           
      should speed up event processing in 2 node clusters.                      
There is another enhancement in 3.1's event processing that should speed        
up processing of local failures. Each cluster manager will check for and        
process local failures at 1/2 the error detection time. For instance, if        
a local network adapter fails, the local cluster manager will detect and        
begin to process that event in 1/2 the "cycles to fail". In this way, a        
local adapter swap can be processed and done before the other nodes             
officially detect the problem.                                                  
I hope this description helps to answer your two questions. The cluster         
manager event processing mechanism has changed with HACMP 3.1, so we            
really can't use exactly the same terms to describe it as in 2.1. I             
should just clarify something else from your second question though. As         
of HACMP 2.1, there was no longer a "speed_ka_cycles" parameter. If you         
read the U431606 memo_to_users more carefully, you should not see any           
mention of this. As of HACMP 2.1, there is only a normal KA cycle, used         
for both event detection and synchronization or stabilization. As for           
whether these parameters are still available in V3.1, the preceding will        
tell you that the KA cycle is adjustable by network type to either normal       
(recommended), slow (recommended), or fast (use at your own risk, and           
Thanks again. Very helpful¢¢                                                    
Still another question:                                                         
- As I understand, the Failure Detection Rate parameter settable                
  via SMIT determines the number of missed KA and ALSO the KA rate.             
  Can the KA rate (normal_ka_cycles in V2.1) be adjusted by a                   
  "command line option" of clstrmgr ?                                           
  What I mean is:                                                               
   Say I set the Failure Detection Rate to Fast (for ethernet).                 
   That means I have 9 KA at 0.5 seconds interval.                              
   Can I now further adjust this 0.5 sec interval ? Like to 0.3 sec?            
Thanks and Regards... Liviu                                                     
Thanks again¢¢¢                                                                 
One more question.                                                             
It appears that Target Mode SCSI is NOT supported over SCSI2 F/W                
in AIX 3.2, only over SCSI2 Fast.                                               
In AIX 4.1.1, Target Mode SCSI IS supported over SCSI2 F/W,                     
but HACMP is not there yet.                                                     
For my project is important to have TMSCSI over F/W.                            
Do you know if AIX 3.2.5 will have this capability ?                            
Thanks and Regards... Liviu                                                     
I have not gotten HACMP 3.1 yet to try this myself, but apparently this         
is possible also, by using the command:                                         
   chssys -a'-n ######'                                                         
where ###### represents the number of microseconds desired in each KA           
cycle. A stop and restart of the cluster manager activates the change.          
Again, I must warn you, though. Speeding up the cycle exposes you much          
more to the possibility of false takeover, and undesired timeouts of the        
deadman switch causing a node to crash. It is provided by development as        
requested by some customers, but not recommended for most users. To read        
about the relationship of KA cycle and "cycles to fail" on the timeout          
of the deadman switch, read the "memo to users" of HACMP 2.1 PTF U432018        
or U431606.                                                                     
Yes, target mode support for SCSI-2 differential Fast/Wide has just been        
made available for AIX 3.2.5 in PTF U434679. You must be aware that it          
has just been made available for AIX, but has not yet been tested and           
certified with HACMP 2.1 by HACMP development as of December 1994. You          
can expect to get news of this when it is done through an update to the         
HAMATRIX PACKAGE on the MKTTOOLS disk, if you subscribe to it.                  
Meanwhile, the base support is there, and it should work with HACMP, but        
it hasn't officially been tested.                                               
S e a r c h - k e y w o r d s:                                                  

WWQA: ITEM: RTA000065101 ITEM: RTA000065101
Dated: 08/1995 Category: ITSAIHA6000
This HTML file was generated 99/06/24~12:43:24
Comments or suggestions? Contact us