[ Bottom of Page | Previous Page | Next Page | Contents | Index | Library Home | Legal | Search ]

System Management Concepts:
Operating System and Devices

Dynamic Processor Deallocation

Starting with machine type 7044 model 270, the hardware of all systems with more than two processors is able to detect correctable errors, which are gathered by the firmware. These errors are not fatal and, as long as they remain rare occurrences, can be safely ignored. However, when a pattern of failures seems to be developing on a specific processor, this pattern might indicate that this component is likely to exhibit a fatal failure in the near future. This prediction is made by the firmware based on the failure rates and threshold analysis.

On these systems, AIX implements continuous hardware surveillance and regularly polls the firmware for hardware errors. When the number of processor errors hits a threshold and the firmware recognizes that there is a distinct probability that this system component will fail, the firmware returns an error report. In all cases, the error is logged in the system error log. In addition, on multiprocessor systems, depending on the type of failure, AIX attempts to stop using the untrustworthy processor and deallocate it. This feature is called Dynamic Processor Deallocation.

At this point, the processor is also flagged by the firmware for persistent deallocation for subsequent reboots, until maintenance personnel replaces the processor.

Potential Impact to Applications

Processor deallocation is transparent for the vast majority of applications, including drivers and kernel extensions. However, you can use the published interfaces to determine whether an application or kernel extension is running on a multiprocessor machine, find out how many processors there are, and bind threads to specific processors.

The interface for binding processes or threads to processors uses logical CPU numbers. The logical CPU numbers are in the range [0..N-1] where N is the total number of CPUs. To avoid breaking applications or kernel extensions that assume no "holes" in the CPU numbering, AIX always makes it appear to applications as if it is the "last" (highest numbered) logical CPU to be deallocated. For instance, on an 8-way SMP, the logical CPU numbers are [0..7]. If one processor is deallocated, the total number of available CPUs becomes 7, and they are numbered [0..6]. Externally, it seems as if CPU 7 has disappeared, regardless of which physical processor failed.

Note
In the rest of this description, the term CPU is used for the logical entity and the term processor for the physical entity.

Potentially, applications or kernel extensions that are binding processes or threads could be broken if AIX silently terminated their bound threads or forcefully moved them to another CPU when one of the processors needs to be deallocated. Dynamic Processor Deallocation provides programming interfaces so that such applications and kernel extensions can be notified that a processor deallocation is about to happen. When these applications and kernel extensions receive notification, they are responsible for moving their bound threads and associated resources (such as timer request blocks) away from the last logical CPU and for adapting themselves to the new CPU configuration. After notification, if some threads remain bound to the last logical CPU, the deallocation is aborted, the aborted deallocation is logged in the error log, and AIX continues using the ailing processor. When the processor ultimately fails, it causes a total system failure. Therefore, it is important that applications or kernel extensions receive notification of an impending processor deallocation and act on this notice.

Even in the rare cases that the deallocation cannot go through, Dynamic Processor Deallocation still gives advanced warning to system administrators. By recording the error in the error log, it gives them a chance to schedule a maintenance operation on the system to replace the ailing component before a global system failure occurs.

Processor Deallocation

The typical flow of events for processor deallocation is as follows:

  1. The firmware detects that a recoverable error threshold has been reached by one of the processors.
  2. The firmware error report is logged in the system error log, and, when AIX is executing on a machine that supports processor deallocation, AIX starts the deallocation process.
  3. AIX notifies non-kernel processes and threads bound to the last logical CPU.
  4. AIX waits up to ten minutes for all the bound threads to move away from the last logical CPU. If threads remain bound, AIX aborts the deallocation.
  5. If all processes or threads are unbound from the ailing processor, the previously registered High Availability Event Handlers (HAEHs) are invoked. An HAEH might return an error that aborts the deallocation.
  6. Unless aborted, the deallocation process ultimately stops the failing processor.

If there is a failure at any point of the deallocation, the failure and its cause are logged. The system administrator can look at the error log, take corrective action (when possible) and restart the deallocation. For instance, if the deallocation was aborted because an application did not unbind its bound threads, the system administrator can stop the application, restart the deallocation, and then restart the application.

Turning Processor Deallocation On and Off

Dynamic Processor Deallocation can be enabled or disabled by changing the value of the cpuguard attribute of the ODM object sys0. The possible values for the attribute are enable and disable.

Beginning with AIX 5.2, the default is enabled (the attribute cpuguard has a value of enable). System administrators who want to disable this feature must use either the Web-based System Manager system menus, the SMIT System Environments menu, or the chdev command. (In previous AIX versions, the default was disabled.)

Note
If processor deallocation is turned off (disabled), the errors are still logged. The error log will contain an error such as CPU_FAILURE_PREDICTED, indicating that AIX was notified of a problem with a CPU.

Restarting an Aborted Processor Deallocation

Sometimes the processor deallocation fails because an application did not move its bound threads away from the last logical CPU. Once this problem has been fixed, either by unbinding (when it is safe to do so) or by stopping the application, the system administrator can restart the processor deallocation process using the ha_star command.

The syntax for this command is:

	ha_star -C

where -C is for a CPU predictive failure event.

Processor State Considerations

Physical processors are represented in the ODM database by objects named procn where n is a decimal number that represents the physical processor number. Like any other device represented in the ODM database, processor objects have a state, such as Defined/Available, and attributes.

The state of a proc object is always Available as long as the corresponding processor is present, regardless of whether it is usable. The state attribute of a proc object indicates if the processor is used and, if not, the reason. This attribute can have three values:

enable The processor is used.
disable The processor has been dynamically deallocated.
faulty The processor was declared defective by the firmware at startup time.

If an ailing processor is successfully deallocated, its state goes from enable to disable. Independently of AIX, this processor is also flagged in the firmware as defective. Upon reboot, the deallocated processor will not be available and will have its state set to faulty. The ODM proc object, however, is still marked Available. You must physically remove the defective CPU from the system board or remove the CPU board (if possible) for the proc object to change to Defined.

Example

In the following scenario, processor proc4 is working correctly and is being used by the operating system, as shown in the following output:

	# lsattr -EH -l proc4
	attribute	value			description		user_settable

	state		enable			Processor state		False
	type		PowerPC_RS64-III	Processor type		False
		#

When processor proc4 gets a predictive failure, it gets deallocated by the operating system, as shown in the following:

	# lsattr -EH -l proc4
	attribute	value			description		user_settable

	state		disable			Processor state		False
	type		PowerPC_RS64-III	Processor type		False
		#

At the next system restart, processor proc4 is reported by firmware as defective, as shown in the following:

	# lsattr -EH -l proc4
	attribute	value			description		user_settable

	state		faulty			Processor state		False
	type		PowerPC_RS64-III	Processor type		False
		#

But the status of processor proc4 remains Available, as shown in the following:

	# lsdev -CH -l proc4
	name		status			location		description
	
	proc4		Available		00-04			Processor
	#

Error Log Entries

Three different error log messages are associated with CPU deallocation. The following are examples.

errpt short format - summary
The following is an example of entries displayed by the errpt command (without options): 

# errpt
IDENTIFIER      TIMESTAMP       T    C    RESOURCE_NAME    DESCRIPTION
804E987A        1008161399      I    O    proc4            CPU DEALLOCATED
8470267F        1008161299      T    S    proc4            CPU DEALLOCATION ABORTED
1B963892        1008160299      P    H    proc4            CPU FAILURE PREDICTED
#
errpt long format - detailed description
The following is the form of output obtained with errpt -a:

[ Top of Page | Previous Page | Next Page | Contents | Index | Library Home | Legal | Search ]