IBM Books

Diagnosis Guide


Error symptoms, responses, and recoveries

A user of SP trusted services interacts with SP security services in the following ways:

  1. The user identifies himself to the underlying security mechanisms, DCE or Kerberos V4. This process obtains a ticket-granting ticket for the client principal based on either the user's password or a service's private key. The former is achieved interactively using the dce_login and k4init commands. Background processes running as root and invoking shell scripts use the dsrvtgt command for DCE and the rcmdtgt or ksrvtgt commands for Kerberos V4, to get tickets as service principals.
  2. The user invokes a command that directly or indirectly invokes an SP trusted service. The client command uses security services to obtain a DCE authentication token to pass to a server. Clients that use Kerberos V4 in compatibility mode similarly obtain Kerberos V4 authentication tokens.
  3. The user issues the kdestroy command for DCE or the k4destroy command for Kerberos V4 to remove the authentication tickets following their use.

Errors detected by SP Security Services are reported in three different ways:

  1. Errors that occur while running security services commands (for example, chauthpts, spgrpname), are reported by error messages written to stderr. Most of the messages have the 2502, 2503, and 2504 SP Security Services prefix. Some of the messages have the 0016 installation and configuration prefix, or the 0025 SDR prefix.
  2. Other errors are reported by either client or server daemon programs that comprise the SP trusted services. The security services runtime support used by those programs reports error conditions it detects, as well as errors detected by the DCE library subroutines it invokes.
  3. The SP trusted services incorporate error message text describing those errors in their own output to stderr, or to daemon log files, or the AIX error log, as appropriate to the runtime environment.

Many of the errors, such as those caused by the failure of a user to login to DCE or Kerberos V4, or a user with expired credentials, are straightforward, and the recovery actions are clear. Other errors are reported by DCE-supplied error text, and the explanations and recovery actions for these error are located in IBM DCE for AIX, Version 3.1: Problem Determination Guide.

Other less common error conditions include those in Table 47.

Table 47. SP Security Services symptoms and recovery actions

Symptom Recovery
The service principal for a target server is unknown.

  • DCE messages similar to "registry object unknown"
  • Kerberos V4 messages similar to "Kerberos principal unknown"
See Action 1 - Check the service principal.
Authentication server is not available.

  • Users unable to login to DCE or Kerberos V4.
  • Servers do not initialize properly when using DCE.
  • Servers hang using DCE.
  • When DCE authentication fails due to an unavailable server, error messages similar to "registry server unavailable".
  • When a client cannot access the Kerberos V4 authentication server, error messages similar to "Kerberos error: retry count exceeded".
See Action 2 - Check the authentication server.
A user cannot login to Kerberos V4.

The login fails with a message similar to:

  • "Bad Kerberos name format'
  • "Kerberos principal unknown"
  • "Incorrect Kerberos password"
See Action 3 - Propagate Kerberos V4 database.
Authorization to use a trusted service is denied.

Messages vary. A message similar to "Cannot obtain service ticket for ..." may appear.

See Action 4 - Check for authorization or authentication problems.
A background client cannot login as a service principal.

Messages vary. A message similar to "Incorrect Kerberos Password" may appear.

See Action 5 - Correct key files.
PSSP configuration error: error message from install_cw, the SDR, and System Monitor indicate that all user requests are unauthorized.

Corrupted or destroyed authentication method settings.

Messages 2502-604, 2515-031, 0026-894, 0031-737, and 2539-498 or other similar messages.

See Action 7 - Run the chauthts command.
Problems with Sysctl using multiple authentication methods.
See Action 8 - Diagnosing Sysctl security problems.
Incorrect information in the SDR, after migrating to PSSP 3.2 from a level of PSSP earlier than PSSP 3.1.
See Action 6 - Reset authentication values.

Actions

Action 1 - Check the service principal

If a trusted services client program attempts to obtain a service ticket in order to pass an authentication token to a server, the service principal used by the server program as its identity for purposes of authentication must exist in the authentication database. When using DCE, this is the security registry. When using Kerberos V4, this is either the PSSP Kerberos V4 authentication database or an AFS authentication database.

This error is reported by DCE as "registry object unknown" error returned by the subroutine gss_accept_sec_context(). When the error occurs using Kerberos V4, the error is reported as "Kerberos principal unknown".

Use the diagnostic procedures in Verify DCE Security Services configuration to verify that the proper SP Security Services configuration steps were performed. If necessary, refer to diagnostic procedures for SP Installation and Configuration. If those procedures indicate that the required service principal does not exist, identify and perform the required configuration task to create the entries in the authentication database. This may require that a DCE cell administrator run the config_spsec command, and the root user perform other configuration tasks as well. For a missing Kerberos V4 service principal, the Kerberos V4 administrator (usually root.admin) must run setup_server to customize the target server node.

Another possible cause of this error is incomplete propagation of an spsec_overrides file that contains an override name for one or more services to RS/6000 workstations. PSSP software automatically propagates the file from the control workstation to SP nodes, when they are installed. However, if you want to use security services client programs on another workstation, you must make sure the file is copied from the control workstation before running config_spsec on the workstation.

Action 2 - Check the authentication server

This error condition can manifest itself in several different ways and have many different causes. Users may be unable to login to DCE or Kerberos V4. Servers may not initialize properly when using DCE authentication. When DCE authentication cannot complete due to an unavailable server, the most common error message reported by DCE is "registry server unavailable".

When a client cannot get to the Kerberos V4 authentication server, the most common error message is "Kerberos error: retry count exceeded".

  1. Follow the diagnostic procedures described previously to check server status, including the log files. See Check that the Kerberos V4 daemons are running.
  2. Check the state of the server daemons.

    For DCE, issue the command lsdce -r on both application client and application server hosts, as well as on the DCE server hosts. If the servers are down, continue to the next step.

  3. Check the DCE log files for server termination errors.

    If servers terminated, a full file system is a possible cause. If the server terminated, determine the reason from the log file, take corrective action, and restart the server.

  4. Make sure that you have an appropriate number of backup (replica) servers for your system.
  5. Also make sure that your pe_site file is kept current on all hosts in your DCE cell.

Errors may result from the inability to communicate with the authentication servers. If this is the case, check for network problems or interface and routing problems.

For Kerberos V4, the primary server system should have the kerberos and kadmind daemons running under the AIX SRC. A secondary server system should have the kerberos and kpropd daemons running (from init).

Check the Kerberos V4 daemon log - each Kerberos daemon program records errors and some status in a log file in the /var/adm/SPlogs/kerberos directory. Check these files if you suspect that one or more of the daemons has terminated. This is an example of the log file created by the kerberos daemon:

03-May-1999 14:06:45 Kerberos started, PID=22408
07-May-1999 07:53:07 Kerberos started, PID=8642 
07-May-1999 07:53:07 kerberos: 2503-604 Cannot verify master key.
07-May-1999 07:55:11 Kerberos started, PID=8648 

This is an example of the log file created by the kadmind daemon:

21-May-1999 16:40:12 The Kerberos administration server has started,
                                           PID=10188
21-May-1999 16:41:40 kadmind: 2503-101 Error: 2504-318 Could not verify
                                                        master key
21-May-1999 16:41:40 Shutting down the Kerberos administration server

This is an example of the log file created by the kpropd daemon:

13-Oct-1999 09:27:32 Established socket
13-Oct-1999 10:19:09 Connection from cwksta.xyz.abc.com, 129.49.100.41
13-Oct-1999 10:19:09 kpropd: Connection from rcmd.cwksta@XYZ.ABC.COM
13-Oct-1999 10:19:09 File received.
13-Oct-1999 10:19:09 Temp file renamed to /var/kerberos/database/slavedb
14-Oct-1999 07:36:41 Connection from cwksta.xyz.abc.com, 129.49.100.41
14-Oct-1999 07:36:41 kpropd: Connection from rcmd.cwksta@XYZ.ABC.COM
14-Oct-1999 07:36:41 File received.
14-Oct-1999 07:36:41 Temp file renamed to /var/kerberos/database/slavedb

Action 3 - Propagate Kerberos V4 database

Logging into Kerberos V4, these are the most common error messages encountered:

  1. Bad Kerberos name format

    You probably entered name.admin in response to the prompt "Kerberos name:". You are not allowed to specify an instance in addition to the name. If you want to enter the name and instance together, enter them as the command-line argument when you invoke k4init. To request k4init to prompt you separately for the instance, invoke k4init with the -i flag.

  2. Kerberos principal unknown

    You did not enter a principal name that is defined in the database. Perhaps you misspelled it, or the administrator did so when entering it into the database.

  3. Incorrect Kerberos password

    You entered the wrong password, or your password was recently changed and has not yet been propagated to the secondary authentication server that you are using. If you incorrectly entered your password, just try again. Otherwise, if you suspect an out-of-date database, contact the administrator of your authentication service.

    This error can also occur when using a secondary authentication server, if the primary database has not been propagated since your principal was added. Check with the administrator responsible for maintaining the authentication service to determine if this is the case. The recovery action in this situation is to force the propagation of database changes without waiting for the normal cron process.

To correct the preceding problems 2 and 3, force the propagation of Kerberos V4 database:

  1. Issue the k4init command, specifying as your principal name any user principal listed in the root user's .klogin file on the primary server. The administrative principal name that was used to set up authentication on the primary server can be used. Any other user principal that the administrator has subsequently added to the file can be used.
  2. Issue the following command to remotely perform the database propagation from the primary server to all secondary servers:
    dsh -w primary /usr/kerberos/etc/push-kprop
    
    where primary is the hostname of the primary Kerberos V4 authentication server.
  3. Successful propagation is reported by a message for each secondary server hostname. If unsuccessful, review the kpropd.log file. See Log files.

Action 4 - Check for authorization or authentication problems

This is by far the most common problem encountered when using security services. Most security errors ultimately result in a failure to authorize a task that a client requested of a server. First, determine whether the problem is an authorization problem or an authentication problem. If the error resulted from an authentication problem, you should have at least one error message from the trusted service describing the type of failure, in addition to the authorization failure. When the trusted service uses DCE and Kerberos V4 for authentication (Sysctl and Hardware Monitor), the client could report authentication error messages regarding one authentication method or the other, or both.

A possible cause of authentication failures is the lack of a common authentication method on the client and server hosts. Client programs on a node in a system partition that is not configured to use DCE cannot access trusted services on nodes in another partition that is configured to use only DCE.

Similarly, a client host must use Kerberos V4 authentication, in order to run a Sysctl procedure that requires authentication on a target server that uses only Kerberos V4 for authentication.

Issue the commands from SP trusted services authentication errors, on the client and server hosts to look for error messages. If the authorization failure was accompanied by no message regarding authentication failure, consult the security administrator to determine whether the requisite permission had been granted. For most trusted services, the requisite permission requires membership in a DCE group, or an entry in a DCE ACL. For Sysctl and Hardware Monitor failures using Kerberos V4, the requisite permission usually requires an entry in an ACL file.

When using DCE authentication, an authorization failure may mean that the user failed to re-login to DCE after the administrator added his principal to one of the SP trusted services access groups.

When you use any of the authenticated client/server applications to administer or control the SP system, the error messages you receive on authentication failures will vary according to the application. For example, if you are using the command-line interface, you might see error messages such as the following, indicating that the System Monitor is unable to obtain Kerberos V4 credentials:

0026-706 Cannot obtain service ticket for hardmon.cwksta 
Kerberos error code is 76, Kerberos error message is: 
2504-076 Kerberos ticket file was not found.
spmon: 0026-001 Opening session failed.
 

The message states that you have no tickets, expired or unexpired, or your KRBTKFILE environment variable specifies a nonexistent file.

The following message states that you have tickets that have expired in the ticket cache file (specified by your KRBTKFILE environment variable or defaulted to /tmp/tktuid.

0026-706 Cannot obtain service ticket for hardmon.cwksta
Kerberos error code is 76, Kerberos error message is:
2504-032 Kerberos ticket expired.
spmon: 0026-001 Opening session failed.

If application error messages indicate probable authentication failure, use the klist or k4list command to check your authentication status. These commands always displays the current active ticket cache file.

Action 5 - Correct key files

SP Security Services include commands that can be used by a background process to obtain authenticated tickets, when no user is logged in to the client AIX system. SP installation and system management scripts use the dsrvtgt, rcmdtgt, or ksrvtgt command to login.

For DCE, the dsrvtgt command logs in as a service principal or a user, using a DCE key file. For a service principal that cannot be logged into, check the spelling of the service name with the default name contained in the /usr/lpp/ssp/config/spsec_defaults file. Use Check SP-unique principals, accounts, and keytabs to verify that the required DCE registry information exists for the principal and that the key file exists.

If the configuration information and key file exist, check on the operation of the spnkeymand daemon that keeps the server keys refreshed. Check the password expiration policy for the spsec-services organization (or override organization name, if applicable). If a node or workstation is out of service for an extended period of time, it is possible for the keys to have not been refreshed before their expiration. See Diagnosing Per Node Key Management (PNKM) problems.

If this is the case, you must use the recovery procedures provided by SP Installation and Configuration, using the rm_spsec and config_spsec. Distribute new key files to a node by customizing it or by running create_keyfiles on the node. If the error requires re-creation of key files for Topology Services, customize the nodes in the partition or run get_keyfiles on each node.

To recover from a single missing or corrupted DCE key file, see Recover from a missing or corrupt DCE key file.

Errors logging into Kerberos V4 could be reported as:

Incorrect Kerberos password

If this error occurs during installation, or when performing administrative tasks requiring remote invocation on SP nodes, it can indicate one of several error conditions:

Compare server key versions. An administrator running as root can compare the versions of the server keys in the server key file and the database, when AFS is not being used. Use the ksrvutil command to show the version numbers of server keys in /etc/krb-srvtab. See Check the Kerberos V4 srvtab files. The actual value of the keys cannot be compared directly, because the copies in the database are encrypted in the master key. Compare the key versions.

When the key file, /etc/krb-srvtab, does not exist on the server system, or has the wrong key version, you must re-create the file. When the control workstation is an SP authentication server, customizing an SP node will automatically create a new server key file for the node. If customizing the node for this purpose is too disruptive, or if the system whose key file must be replaced is not an SP node, follow the following procedures.

Re-create Kerberos V4 server key files

You can use authentication administration commands to re-create an erroneous or missing server key file. Each system with an SP authentication service installed has its own unique server key file, containing the encrypted keys for the service instances that are available on the system.

On the control workstation and other IBM RS/6000 workstations that have client services installed and initialized, the file contains entries for services rcmd and hardmon. Separate instances of these principals are defined for each network interface on the system, where the instance-name is the short form of the network name.

There is also a special principal, root.SPbgAdm, that is used by root processes that run in the background and need access to services that use authentication. For example, on a client system with token ring and FDDI interfaces named wksta5t.xyz.abc.com and wksta5f.def.abc.com, the following principals' keys are kept in the server key file:

On the SP nodes, the hardmon entries are not included. The hardmon files are created by the setup_server command.

The server key files on these systems are created by the setup_authent command. During installation, they are kept in the /spdata/sys1/k4srvtabs directory on the control workstation, with file names of the form hostname-new-srvtab, where hostname is the short form of the hostname for each node.

When the control workstation is an SP authentication server, these files are retained there only until they are copied to the node during network boot. If the node boots from another node, these files are retained until they are copied to the node's boot/install server. A new server key file is generated any time the node is set up for a network boot.

When the control workstation is not configured as an authentication server, or when AFS authentication is used, the server key files for the SP nodes are not removed from the /spdata/sys1/k4srvtabs directory on the control workstation, once they are created.

If they are deleted or corrupted, or if you choose to change keys for any reason, follow the rest of the procedures to create new key files. In these procedures, instance1 ... instancen are the network names (short form) of all the system's interfaces, and hostname is the short form of the system's hostname.

Replace a Kerberos V4 authentication server's key file

To re-create the file for a workstation (control workstation or other) that is configured as an authentication server, the root user follows these steps:

  1. Create new key files in the /tmp directory for each instance:
    cd /tmp
    /usr/kerberos/etc/ext_srvtab -n instance1...instancen
    
  2. Combine the key files into a single file:
    /bin/cat instance1-new-srvtab...instancen-new-srvtab\
                    /etc/spa-srvtab >/etc/krb-srvtab
    /bin/rm -f instance1-new-srvtab...instancen-new-srvtab
     
    
  3. Make sure that the key file is readable by root only:
    /bin/chmod 400 /etc/krb-srvtab
    

Replace a client workstation's Kerberos V4 key file

When a workstation is not an authentication server, the root user can use the remote commands to perform the same function on a server system, and move the file to the local system. The principal name specified on the k4init command must be in the root user's .klogin file on the server:

  1. Get a ticket-granting ticket to allow use of rsh and rcp.
    k4init principal
    
  2. Create the new key files in /tmp on the server.
    rsh server cd /tmp\;
    /usr/kerberos/etc/ext_srvtab -n instance1...instancen SPbgAdm
    
  3. Copy the files we created to the local /tmp directory.
    cd /tmp
    rcp server:/tmp/instance1-new-srvtab...instancen-new-srvtab\
           SPbgAdm-new-srvtab
    
  4. Delete the files on the server.
    rsh server /bin/rm -f /tmp/SPbgAdm-new-srvtab \
             /tmp/instance1-new-srvtab...instancen-new-srvtab
    
  5. Combine the local files into a single file.
    /bin/cat instance1-new-srvtab...instancen-new-srvtab SPbgAdm-new-srvtab\
                 >/etc/krb-srvtab
    /bin/rm -f instance1-new-srvtab...instancen-new-srvtab SPbgAdm-new-srvtab 
    
  6. Make sure that the key file is readable only by root.
    /bin/chmod 400 /etc/krb-srvtab
    

Replace an SP node's Kerberos V4 key file

The most straightforward way to replace a node's key file is to customize the node, using the spbootins command or SMIT. If you prefer, you can use procedures similar to the preceding example. The root user on the node can use the procedure if the control workstation is a Kerberos V4 authentication server. Specify the control workstation hostname as the server.

When the control workstation is not an authentication server, the server key file must be re-created at the server and then moved securely into the /spdata/sys1/k4srvtabs directory on the control workstation. For this, the root user should be logged into the control workstation.

When the authentication server is another workstation running the SP server or another MIT Kerberos Version 4 implementation, use the following procedure:

  1. Get a ticket-granting ticket to allow use of rsh and rcp.
    k4init principal
    
  2. Create the new key files in /tmp on the server.
    rsh server cd /tmp\;
    /usr/kerberos/etc/ext_srvtab -n instance1...instancen SPbgAdm
    
  3. Copy the files we created to the local /tmp directory.
    cd /tmp
    rcp server:/tmp/instance1-new-srvtab...instancen-new-srvtab\
                  SPbgAdm-new-srvtab
    
  4. Delete the files on the server.
    rsh server /bin/rm -f /tmp/SPbgAdm-new-srvtab \
                   /tmp/instance1-new-srvtab...instancen-new-srvtab
    
  5. Combine the local files into a single file.
    bin/cat instance1-new-srvtab...instancen-new-srvtab\
     SPbgAdm-new-srvtab >/etc/krb-srvtab
    /bin/rm -f instance1-new-srvtab...instancen-new-srvtab SPbgAdm-new-srvtab 
    
  6. Make sure that the key file is readable only by root.
    /bin/chmod 400 /etc/krb-srvtab
    

The new key file can then be installed on the node by either:

Replace a server key file using AFS servers

When an AFS authentication server is being used, follow this procedure and be sure you are logged in to the control workstation as root.

  1. Change the service password to a new, known value, but not one that is obvious.
  2. Repeat this step for each instance (each short network name).
    kas setpassword -name rcmd.instance -new_password new-password 
     -kvno 1 -admin_username afs-admin-name
     -password_for_admin afs-admin-passwd
    
  3. Use the same principals and passwords to create a srvtab file.
    /usr/kerberos/bin/ksrvutil -afs 
     -f /spdata/sys1/k4srvtabs/hostname-new-srvtab add 
    
  4. Use ksrvutil, an interactive program whose sequence of prompts and messages is as follows:
    Name: rcmd
    Instance: instance
    Realm: <Enter> 
    Version number: 0 
    New principal: rcmd.instance@realm Version 0 
    Is this correct? y
    Password: 
    Key successfully added.
    Would you like to add another key?
    (reply y until all instances have been entered)
     
    
  5. Make sure that the key file is readable only by root.
    /bin/chmod 400 /spdata/sys1/k4srvtabs/hostname-new-srvtab
    

Recover from a missing or corrupt DCE key file

SP trusted services key files are under /spdata/sys1/keyfiles (main path). Under normal conditions, this directory exists and contains the following directories:

     drwxr-xr-x   3 root     system       512 Oct 26 13:04 LoadL/
     drwxr-xr-x   3 root     system       512 Oct 26 13:04 mmfs/
     drwxr-xr-x   3 root     system       512 Oct 26 13:04 ppe/
     drwxr-xr-x   4 root     system       512 Oct 26 13:06 rsct/
     drwxr-xr-x   3 root     system       512 Oct 26 13:04 ssp/

Under each of these directories is another directory. The name of this directory is the DCE hostname of the control workstation or node. For example, for node sp3en5:

     drwxr-xr-x   2 root     system       512 Nov 17 12:29 sp3en5

The DCE hostname directory contains the DCE key files for a given service or services:

     -rw-------   1 root     system       298 Oct 27 23:43 css
     -rw-------   1 root     system       382 Oct 27 23:43 pmand
     -rw-------   1 root     system       407 Oct 27 23:44 sp_configd
     -rw-------   1 root     system       397 Oct 27 23:44 spbgroot
     -rw-------   1 root     system       382 Oct 27 23:43 spmgr
     -rw-------   1 root     system       407 Oct 27 23:43 switchtbld
     -rw-------   1 root     system       152 Oct 26 12:22 sysctl

The key files on the control workstation and on the nodes differ in two ways:

  1. On the control workstation, the /ssp directory contains at least two subdirectories. One subdirectory is named after the DCE hostname directory. Another subdirectory is named after the default partition of the system. If more than two subdirectories exist, their names correspond to the names of the remaining PSSP system partitions. This example relates to a system with three partitions, named secsys1, secsys2, and secsys3.
     ls -l
         total 32
         drwxr-xr-x   2 root     system       512 Nov 17 12:29 sp3en0/
         drwxr-xr-x   2 root     system       512 Oct 26 12:21 secsys1/
         drwxr-xr-x   2 root     system       512 Oct 26 12:22 secsys2/
         drwxr-xr-x   2 root     system       512 Oct 26 12:22 secsys3/      
    

    The ssp/partition_name subdirectories contain a DCE key file for each sdrd. The commands:

    cd secsys1
    ls -l
     
    

    produce output similar to:

    total 8
    -rw-------   1 root     system       112 Oct 26 12:22 sdr
    
  2. The ssp/dce_hostname directory has these files:
    ls -l
         total 64
         -rw-------   1 root     system       298 Oct 27 23:43 css
         -rw-------   1 root     system       154 Oct 26 12:21 hardmon
         -rw-------   1 root     system       382 Oct 27 23:43 pmand
         -rw-------   1 root     system       407 Oct 27 23:44 sp_configd
         -rw-------   1 root     system       397 Oct 27 23:44 spbgroot
         -rw-------   1 root     system       382 Oct 27 23:43 spmgr
         -rw-------   1 root     system       407 Oct 27 23:43 switchtbld
         -rw-------   1 root     system       152 Oct 26 12:22 sysctl
     
    
Note:
Even if the SP system is not configured for LoadLeveler, PE, GPFS, or an SP switch, PSSP DCE key files are created for these services. This includes a css key file for use with PSSP switch services and commands. This is not a security exposure because the key files can be read and used only by a user with root authority.

In the event that a PSSP DCE key file does not exist in its expected location, determine if the key file was removed (deleted) from the host, or was not created during the installation and configuration process.

On the host where the DCE PSSP key file is missing, run the create_keyfiles command in verbose mode. If the command is run on the control workstation, ensure that the -c flag is specified on the create_keyfiles command. If the key file did exist, but was deleted, an attempt to create a new key file will fail, because the local DCE service will already contain a keytab object for the missing key file.

In this sequence, sysctl is the missing key file.

  1. The key file catalog contains an entry for sysctl. Issue this command:
    dcecp -c keytab cat
    

    Output is similar to:

         /.../sp_cell/hosts/sp3en0/config/keytab/self
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/GSmonitor
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Kbdd
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Master
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Negotiator
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Schedd
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Startd
         /.../sp_cell/hosts/sp3en0/config/keytab/LoadL/sp3en0/Starter
         /.../sp_cell/hosts/sp3en0/config/keytab/mmfs/sp3en0/mmfsd
         /.../sp_cell/hosts/sp3en0/config/keytab/ppe/sp3en0/dpcl
         /.../sp_cell/hosts/sp3en0/config/keytab/ppe/sp3en0/pmdv3
         /.../sp_cell/hosts/sp3en0/config/keytab/rsct/sp3en0/rsct
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/css
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/pmand
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/sp_configd
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/spbgroot
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/switchtbld
         /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/sysctl
     
    
  2. The physical sysctl key file is missing. Issue this command:
    ls -l
    

    Output is similar to:

              total 48
         -rw-------   1 root     system       223 Oct 27 23:38 css
         -rw-------   1 root     system       229 Oct 27 23:38 pmand
         -rw-------   1 root     system       244 Oct 27 23:38 sp_configd
         -rw-------   1 root     system       238 Oct 27 23:38 spbgroot
         -rw-------   1 root     system       244 Oct 27 23:38 switchtbld
     
    
  3. Attempt to recreate the missing key file. Issue this command:
    create_keyfiles -v
    

    At the end of this output, you will see the expected error: Cannot create object; already exists. Output is similar to:

         Running "check_prereqs" subroutine ... 
         Checking state of DCE ... 
         Running "parse_defaults" subroutine ... 
         Parsing spsec_defaults file ... 
         Running "parse_overrides" subroutine ... 
         Running "create_keys" subroutine ... 
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/GSmonitor already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Kbdd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Master already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Negotiator already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Schedd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Startd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Starter already exists.
         Keyfile /spdata/sys1/keyfiles/mmfs/sp3en0/mmfsd already exists.
         Keyfile /spdata/sys1/keyfiles/ppe/sp3en0/dpcl already exists.
         Keyfile /spdata/sys1/keyfiles/ppe/sp3en0/pmdv3 already exists.
         Keyfile /spdata/sys1/keyfiles/rsct/sp3en0/rsct already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/css already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/pmand already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/sp_configd already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/spbgroot already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/switchtbld already exists.
         Running "do_keytab_work" subroutine ... 
         Creating keytab object, "ssp/sp3en0/sysctl" and randomizing keys. 
         ***********************************************************
         /usr/lpp/ssp/bin/create_keyfiles: 0016-511 The dcecp command 
                         below returned non-zero return code.
         /opt/dcelocal/bin/dcecp -c keytab create ssp/sp3en0/sysctl\ 
                    -storage /spdata/sys1/keyfiles/ssp/sp3en0/sysctl -data 
                    { ssp/sp3en0/sysctl plain 1 "svc_pwd_was_here" }
         Command output is:
         Error: msgID=0x113DB0CE  Cannot create object; already exists 
    

To re-create the missing key file with a random password, perform these steps:

  1. As root, (which has access to the DCE self-host principal credentials), on the host with the missing key, remove the sysctl keytab object from the local catalog. Issue these commands:
    1. dcecp -c keytab delete /.:/hosts/sp3en0/config/keytab/ssp/sp3en0/sysctl
    2. dcecp -c keytab cat | grep sysctl
  2. After the keytab object is deleted, delete and re-create the principal and account. On the control workstation, login as the cell administrator and delete the principal and account.
    dcecp -c principal delete ssp/sp3en0/sysctl
    

    Verify that the entries are gone:

    1. dcecp -c principal show ssp/sp3en0/sysctl

      The output is similar to:

      Error: msgID=0x1712207A  Registry object not found 
      
    2. dcecp -c account show ssp/sp3en0/sysctl

      The output is similar to:

      Error: msgID=0x1712207A  Registry object not found 
      

    Re-create the principal and account. If running on the control workstation, ensure that the -c flag is specified.

    config_spsec        
    

    Answer the prompts that follow:

    This command requires cell administrator authority. Continue? (y/n) y
     
     
     Please enter cell administrator id to be added to ACL admin group: cell_admin
     
     
         Your cell administrator password is required to create accounts.
         Please enter your cell administrator password: 
     
    

    Verify that the principal and account were re-created. Issue these commands:

    1. dcecp -c prin show ssp/sp3en0/sysctl

      Output is similar to:

      {fullname {}}
           {uid 1883}
           {uuid 0000075b-9dcd-21d3-9c00-0004ac493bac}
           {alias no}
           {quota unlimited}
           {groups spsec-services}
      
    2. dcecp -c account show ssp/sp3en0/sysctl -all

      Output is similar to:

           {acctvalid yes}
           {client yes}
           {created /.../sp_cell/cell_admin 1999-11-18-10:30:41.000-05:00I-----}
           {description {}}
           {dupkey no}
           {expdate none}
           {forwardabletkt yes}
           {goodsince 1999-11-18-10:30:41.000-05:00I-----}
           {group spsec-services}
           {home /}
           {lastchange /.../sp_cell/cell_admin 1999-11-18-10:30:41.000-05:00I-----}
           {organization spsec-services}
           {postdatedtkt no}
           {proxiabletkt no}
           {pwdvalid yes}
           {renewabletkt yes}
           {server yes}
           {shell {}}
           {stdtgtauth yes}
           {usertouser no}
           nopolicy
      
    3. kdestroy
    4. exit
  3. On the host where the key file is missing, create a new key file. Run the PSSP script create_keyfiles in verbose mode, as the self-host principal. Messages about already existing key files are expected. If the create_keyfiles command is issued on the control workstation, ensure that the -c flag is specified. Issue this command:
    create_keyfiles -v
    

    Output is similar to:

         Running "check_prereqs" subroutine ... 
         Checking state of DCE ... 
         Running "parse_defaults" subroutine ... 
         Parsing spsec_defaults file ... 
         Running "parse_overrides" subroutine ... 
         Running "create_keys" subroutine ... 
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/GSmonitor already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Kbdd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Master already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Negotiator already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Schedd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Startd already exists.
         Keyfile /spdata/sys1/keyfiles/LoadL/sp3en0/Starter already exists.
         Keyfile /spdata/sys1/keyfiles/mmfs/sp3en0/mmfsd already exists.
         Keyfile /spdata/sys1/keyfiles/ppe/sp3en0/dpcl already exists.
         Keyfile /spdata/sys1/keyfiles/ppe/sp3en0/pmdv3 already exists.
         Keyfile /spdata/sys1/keyfiles/rsct/sp3en0/rsct already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/css already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/pmand already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/sp_configd already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/spbgroot already exists.
         Keyfile /spdata/sys1/keyfiles/ssp/sp3en0/switchtbld already exists.
         Running "do_keytab_work" subroutine ... 
         Creating keytab object, "ssp/sp3en0/sysctl" and randomizing keys.
     
    
  4. Verify that the key file physically exists and is in the keytab catalog. Issue these commands:
    1. ls -ltr

      Output is similar to:

       total 48
           -rw-------   1 root     system       148 Oct 30 13:02 css
           -rw-------   1 root     system       152 Oct 30 13:02 pmand
           -rw-------   1 root     system       162 Oct 30 13:02 sp_configd
           -rw-------   1 root     system       158 Oct 30 13:02 spbgroot
           -rw-------   1 root     system       162 Oct 30 13:02 switchtbld
           -rw-------   1 root     system       154 Nov 18 11:25 sysctl
       
      
    2. dcecp -c keytab cat | grep sysctl /.../sp_cell/hosts/sp3en0/config/keytab/ssp/sp3en0/sysctl
  5. Verify that key file can be used to login to DCE. As root, issue these commands:
    1. dsrvtgt ssp/sysctl

      Output is similar to:

      FILE:/opt/dcelocal/var/security/creds/dcecred_68408200
      
    2. export KRB5CCNAME=FILE:/opt/dcelocal/var/security/creds/dcecred_6840820
    3. klist | grep lob Global Principal: /.../sp_cell/ssp/sp3en0/sysctl
    4. kdestroy -c FILE:/opt/dcelocal/var/security/creds/dcecred_68408200
    5. klist

      Output is similar to:

      No DCE identity available: No currently established network identity for this
                       context exists (dce / sec)
       Kerberos Ticket Information:
           klist: No credentials cache file found (dce / krb)
                         (ticket cache /opt/dcelocal/var/security/creds/dcecred_68408200)
      
    6. unset KRB5CCNAME
    7. klist | grep lob Global Principal: /.../sp_cell/hosts/sp3en0/self

    The dsrvtgt command can be used with other PSSP service/principal name pairs. See the entry for dsrvtgt in PSSP: Command and Technical Reference.

  6. In this example, the sysctl key file was recreated. To ensure that the sysctld daemon can use the new key file, stop sysctld (if not already stopped), and then start sysctld. The sysctld daemon should not have any problem using the new key file, because the dsrvtgt validation in the previous step was successful.
    1. startsrc -s sysctld -a '-d'

      Output is similar to:

      0513-059 The sysctld Subsystem has been started. Subsystem PID is 21636.
      
    2. lssrc -s sysctld

      Output is similar to:

      Subsystem         Group            PID     Status 
        sysctld                          21636   active
      

    Now verify that sysctld is working with the DCE method. In this example, the command was issued in a DCE-only environment.

    sysctl whoami -v
    

    The output is similar to:

    DCE: /.../sp_cell/hosts/sp3en0/self
         K4:  
         AIX: root
    

    For other PSSP services whose key files were re-created, run appropriate client commands against those services. Review the output of the commands, and their corresponding server logs, for normal processing indications (responses and entries) in a DCE mode.


Action 6 - Reset authentication values

If you have migrated to PSSP 3.2 from a level of PSSP earlier than 3.1, there are extra steps for security that are required. If those steps are omitted, the information in the SDR will be incorrect.

If you have a situation where the SDR shows the following authentication states after a migration:

splstdata -p
.
.
. auth_install   K4
 auth_root_rcmd  std
 ts_auth_methods ''
 auth_methods    std
.
.
.

You must run the following steps on the control workstation in order to reset the authentication values for migration to complete properly.

  1. chauthent -k4 -std
  2. spsetauth -p default partition name k4 std

    If you have more than one system partition, issue this command for each of them.

  3. chauthpts -c compat
  4. chauthpar -c k4 std

    If you have more than one system partition, issue this command for each of them. Specify the partition with the -p flag.

  5. install_cw

For more information, see the chapter "Migrating to the Latest Level of PSSP" of PSSP: Installation and Migration Guide. In this chapter, see the section "Migrating the Control Workstation to PSSP 3.2", steps "Verify the Authentication Values in the SDR" and "Verify the Authentication Value for AIX Remote Commands".

Action 7 - Run the chauthts command

During configuration of the Control Workstation, the chauthts command must be run in order to inform the PSSP configuration code which authentication method will be used to complete the configuration. When this step is omitted, there are several symptoms.

To recover from this situation:

  1. Run the chauthts command to indicate your chosen authentication method.
  2. Rerun install_cw or SDR_init to populate the SDR with appropriate values.
  3. Continue with the configuration steps.

Action 8 - Diagnosing Sysctl security problems

Use this procedure to diagnose Sysctl problems that deal with multiple authentication methods.

  1. If the node is running the current release of PSSP, on the client and server nodes, issue the lsauthts command to determine the authentication methods supported. If the node is running an earlier release, assume compat mode. Make sure that the client and server hosts have environments which will operate together. A Sysctl client on a host that uses neither DCE nor Kerberos V4 (compat mode) authentication is permitted to access Sysctl servers only as an unauthenticated user. A Sysctl client on a host using only DCE for authentication cannot access a Sysctl server on a host using only Kerberos V4. A Sysctl client on a host using only Kerberos V4 for authentication cannot access a Sysctl server on a host using only DCE.
  2. Use the getauth Sysctl procedure to determine the authorization callback for an object you cannot access. If you are not authorized to use getauth, ask an administrator to do so. If the ACL callback is specified, use the acllist procedure to verify that you have permission to access the object, through an entry in the ACL file that protects the object. Make sure that the entry is appropriate for the authentication method that Sysctl is using. See Step 1.
  3. If site security policy requires the user to be authenticated, use the DCE klist command to verify that the user has DCE credentials. If the system administrator has not configured AIX login to obtain DCE credentials, the user may need to login to DCE separately using the dce_login command with the user's principal name. If the user just needs to refresh credentials, use the dce_login or DCE kinit command.
  4. Use the dcecp command to verify that the client user has a DCE principal and account. If you are using ACLs to authorize access to the user, use dcecp also to list the ACLs that control access to the desired objects and make any necessary changes.
  5. If previously obtained credentials appear to be no longer valid, check whether an administrator has recently changed the server's keys. If so, you must use dce_login or kinit to obtain new credentials before retrying the client program.
  6. Check the sysctld daemon log file for error information. The default name is /var/adm/SPlogs/sysctl/sysctld.log)


[ Top of Page | Previous Page | Next Page | Table of Contents | Index ]