Since Per Node Key Management depends on a variety of configuration events and customer configuration choices, it can be difficult to isolate a problem.
These are general steps that apply to Per Node Key Management. The steps are presented in decreasing order of likelihood that they will resolve the problem.
The spnkeyman daemon is started on the nodes on a reboot after a node has been configured with DCE. The spnkeyman daemon must be started manually on the control workstation. Verify whether the daemon was started, and if it is sleeping or running, using the AIX SRC command:
lssrc -s spnkeyman
Check the AIX error log for errors from spnkeyman that would prevent the daemon from running. If the daemon is not running, start it using either the startsrc command or the spnkeyman_start command.
Key files for the SP services are located in /spdata/sys1/keyfiles/product/dcehostname/service. These key files are used to login to DCE to obtain the expiration of the keys and to update the keys when necessary. The key files are created when DCE is configured for SP use. For information on the naming of DCE entities used on the SP system, refer to PSSP: Planning, Volume 2.
The spnkeyman daemon cannot use a key file if the key file is corrupted. This can occur if a key file has a zero length, contains corrupted data, or contains an entry for the service principal's account that does not match the DCE registry. If the key file contains the maximum number of entries, new entries cannot be added, and keys cannot be updated. You may need to delete entries if this situation occurs. For more information, see IBM Distributed Computing Environment 3.1 for AIX: Administration Guide and Reference.
The AIX error log has a limitation on the amount of data that can be issued, and some error messages may be truncated. If this happens for a specific error, you may be able to gather more information by attempting to add a key to the failing key file by issuing the dcecp keytab add command.
To correct the key files, see Action 5 - Correct key files.
If no changes have been made to the password expiration in the SP DCE organization, spsec-services, then keys for SP services will never expire. If you have updated the password expiration for the SP services, it is recommended that only the password lifetime value be used. This value will trigger updates to the SP services keys through spnkeyman. If you have changed this value and wish to have spnkeyman check it immediately, stop and restart the spnkeyman daemon on each host. Otherwise, the value will be checked within 24 hours.
You can check using the DCE dcecp command. If no password expiration information is displayed, then the password expiration information has never been changed.
dcecp -c organization show spsec-services
Neither an SP service nor the spnkeyman daemon will be able to login if the service's key has expired. You can check using the dsrvtgt command. If you receive output similar to the following, the key has not expired. Be sure to delete the credentials once verification is complete.
dsrvtgt ssp/css FILE:/opt/dcelocal/var/security/creds/dcecred_7aa04600 kdestroy -c FILE:/opt/dcelocal/var/security/creds/dcecred_7aa04600
Otherwise, you may have received a "keys have expired" error message. Other messages are generated for other types of errors.
If the keys have expired, refer to the chapter "Managing and Using SP Security Services", heading "Managing Server Keys for DCE" in PSSP: Administration Guide. See also Action 5 - Correct key files. Per Node Key Management will generate an error message for each SP service, if the daemon cannot login as that service.
The spnkeyman daemon checks the expiration time set in the SP DCE organization, spsec-services, every 24 hours. If you have set the expiration time to less than 24 hours, the daemon will not have checked, nor updated, the keys and the keys will expire.
Once the keys have expired, neither the SP services nor the spnkeyman daemon will be able to login to DCE, and they will fail. The key for each SP service must then be manually updated in the DCE registry and in the key file on the host using the DCE command, dcecp.
Check that the spsec-services file exists, and that its contents are correct, by using the DCE dcecp command. If no password expiration information is displayed, then the password expiration information has never been changed.
dcecp -c organization show spsec-services
The spnkeyman daemon uses the SP DCE organization, spsec-services, to obtain data on the key expiration. This organization is normally created when DCE is configured for SP system use. If the spsec-services file has not been created, refer to the DCE section in PSSP: Installation and Migration Guide.
You can check using the DCE dcecp command.
dcecp -c organization show spsec-services
Before spnkeyman can manage the DCE keys for SP services, the DCE registry must be populated with the principals and accounts for the SP services. This is normally done when DCE is configured for SP system use. For more information, refer to PSSP: Installation and Migration Guide. You may also wish to verify that the principal name is correct if you caused the default to be overwritten by using the spsec_overrides file.
Use the DCE dcecp command to list the principals and accounts in the DCE registry. If there is a particular service you are interested in, you can use the show attribute rather than the list attribute.
dcecp -c principal cat dcecp -c principal show /.../c55_cell/ssp/c55s/sdr
DCE is required for spnkeyman to function. DCE must be installed and configured as well as configured for SP use before spnkeyman can provide its function.
The DCE client daemons must be running in order for the SP system to function in DCE mode for this host. The daemons must be running in order for spnkeyman to function regardless of which authentication is enabled.
The required DCE daemons are: dced, cdsadv, secd, and cdsd. See Verify DCE Security Services configuration.
The DCE client daemons must have access over the network to the DCE server daemons whether those server daemons are on the control workstation or elsewhere on your site. Name resolution must also be working properly.
The secd daemon is the DCE security daemon. If the daemon is not running, PNKM and many other SP services daemons may hang. You may also see this situation if secd is running, but the registry has been disabled via a DCE registry disable command. You may see errors like:
2502-606 DCE error in sec_login_valid_from_keytable: Cannot find KDC for requested realm (dce/krb) spsec_keyman_get_expiration error for LoadL/Schedd. err=1"
Since the PNKM daemon restarts on error, you may need to forcibly stop the daemon until the situation is corrected. Do this using the stopsrc command with the -c flag or the -f flag. Issue the command with the -c flag first, and if this fails to stop the daemon, issue the command with the -f flag.