Diagnosis Guide

AIX remote command diagnostics

To see all error messages, ensure that the AIX environment variable, K5MUTE, is set to 0.

Since the remote commands depend on a variety of configuration events, customer configuration choices and user actions, isolating a problem can be confusing. Knowing the authentication methods and states of enablement on the SP system is the first step to problem isolation. Also, user education on the proper actions for gaining tickets or credentials is crucial.

One event to be aware of is that the remote command may be working properly, but the user may not have the proper permissions. The user may need permissions to run a command issued through a remote command, or permission to write to a directory accessed through remote copy on the target machine. This can happen, for example, if the user tries to remote copy a file into an AFS file system to which permission has not been granted. The remote copy command simply returns the error message it received. The actual error is outside of the realm of the remote copy command.

In general, Kerberos V5 (through DCE) messages have a prefix of: Kerberos or sometimes rshd. Kerberos V4 messages usually have a prefix of spk4rsh or spk4rcp.

Another helpful problem isolation action is to try to duplicate the problem by running the remote command outside of the script. Run the command at the command line, with as much of the remote command's environment duplicated as possible.

This is a list of general steps that apply to the remote commands under each authentication mechanism. They are listed in decreasing order of probability that the step will find the source of the problem. Answer these questions and follow these steps:

1. For rsh and rcp, is there a link from /usr/lpp/ssp/rcmd/bin/rsh or /usr/lpp/ssp/rcmd/bin/rcp to /usr/bin/rsh or /usr/bin/rcp respectively?

   As of AIX 4.3.1, /usr/bin/rsh supports Kerberos V5 (through DCE), Kerberos V4 (through libspk4rcmd.a), and Standard AIX. The executables formerly in /usr/lpp/ssp/rcmd/bin are no longer shipped.

2. Is the user using the correct flags for the function being used?

   For example, the -n flag is necessary when running rsh in the background.

3. Is the user principal located in the authorization file on the target host?
   • For Kerberos V5, this is .k5login. The principal format is:

     dce_principal_name@cell_name
     

     Using a DCE self_host principal in a cell named "dcecell" for example: hosts/trw.pok.ibm.com/self@dcecell.

   • For Kerberos V4, this is .klogin. The principal format for an SP service principal is:

     rcmd.name@domain
     

     where name is the short hostname returned by the hostname command, and domain is the domain name. An example is:rcmd.trw@POK.IBM.COM.
4. Did the user obtain tickets or credentials?
   • For Kerberos V5, the user must issue the command: dce_login
   • For Kerberos V4, the command is k4init or /usr/lpp/ssp/kerberos/bin/kinit.
5. Does the user have a principal in the database or registry?

   To verify that the user has a valid principal:

   • For Kerberos V5, use the dcecp command under DCE.
   • For Kerberos V4, use the lskp command.
6. If the principal is a "server" principal and an SP system daemon or background function does not work, are the credentials valid? (that is, have the keys expired in Kerberos V5, or can the principal get a service ticket in Kerberos V4)?

   To verify that tickets or credentials are in place:

   • For Kerberos V5, use the klist command or /usr/lpp/dce/bin/klist command.
   • To verify if SP Services keys (passwords) are still valid under Kerberos V5, use the following dsrvtgt command. If you receive output similar to the following, then the keys have not expired. If you receive an error message, the keys have expired. Be sure to destroy the credentials once verification is complete. Use the SP Service principal you are checking in place of ssp/css.
     1. dsrvtgt ssp/css

        Output is similar to:

        FILE:/opt/dcelocal/var/security/creds/dcecred_7aa04600
         
        

     2. kdestroy -c FILE:/opt/dcelocal/var/security/creds/dcecred_7aa04600
   • For Kerberos V4, use the k4list command or /usr/lpp/ssp/kerberos/bin/klist command.
7. What are the authentication mechanisms enabled and should the user be authenticating via that method?

   If you have DCE (Kerberos V5) and Kerberos V4 enabled, but the user has a principal only for Kerberos V4, you will see messages coming from Kerberos V5. This is because Kerberos V5 tries and fails to authenticate the user before trying Kerberos V4. If you do not want these messages, set the AIX environment variable K5MUTE.

8. Are there errors in processing in the krshd server on the target host?

   Initiate the logging through syslogd to verify that krshd is working or generating errors. krshd is the server for both Kerberos V5 and Kerberos V4 authentication mechanisms. If it generates error messages, save the log for analysis by the IBM Support Center.

9. Is the configuration correct to use krshd?

   Check the /etc/services and /etc/inetd.conf files for the kshell service. The kshell service in the /etc/inetd.conf file should point to the /usr/sbin/krshd file. See Unknown service: kshell or TCP.

10. Are the authentication methods you wish enabled for the remote commands actually enabled on the local source and target hosts and in the system partition if applicable?

    Check the local host settings using the lsauthent command. Check the partition settings using the lsauthpar command. The settings should have at least one authentication method enabled in common with the proper authorization files in place. Of course, the user should have obtained tickets or credentials for that particular method as well.

11. Are the required servers and clients for the authentication mechanism up and running properly?

    If servers are located off of the SP system, also be sure to check for network connections and proper routing for the particular authentication mechanism in use. If using Kerberos V5 under DCE, check whether permissions have been changed in ACL files for specific groups.

12. Has the SP system been configured to use that particular authentication mechanism?

    Enabling the method does not necessary mean the authentication mechanism has been properly installed and configured for SP system use. This can happen when methods are being enabled locally using the AIX chauthent command.

    ATTENTION - READ THIS FIRST !
    On the SP system, the chauthent command should be used with caution. The SP Installation and Configuration code will set the proper settings. The chauthent command has the unfortunate side effect that all of the current methods must be on the command line, along with the ones you are adding. It is easy to unintentionally turn off enablement of other methods using the chauthent command.

13. Has the proper service been applied to pick up any fixes for the SP system and AIX?

    Occasionally, fixes are required from both AIX and the SP system. AIX fixes usually pertain to the remote commands using Kerberos V5 or Standard AIX. SP system fixes usually pertain to rsh and rsh client routines using Kerberos V4 in the libspk4rcmd.a library. Note that the AIX krshd daemon processes requests from both Kerberos V5 and Kerberos V4 rsh and rcp commands.

Diagnostics specific to Kerberos V4

1. Is a file or link present in the /usr/kerberos/bin directory for rcp?

   A link should be present if the Kerberos V4 shipped with the SP system is installed and it should point to the file /usr/lpp/ssp/rcmd/bin/rcp. A file should be present only if an MIT version of Kerberos V4 is installed.

2. Has the system administrator run a ksrvutil change command?

   All current tickets become no longer valid, and the k4init command must be run by each user to obtain a current ticket. It is likely that the ksrvutil change command will not be run very often.

Diagnostics specific to Kerberos V5

Has the user principal been set properly in the DCE Registry to allow credential forwarding? Has the user invoked rsh with the proper flag to indicate forwarding of credentials?

DCE requires that the forwarding-allowed attribute be set to true when creating the user account. Also, the user must use dce_login -f when logging into DCE. These actions will allow credential forwarding when the proper flag is present on the rsh command invocation. There are two flags associated with forwarding credentials through rsh. See the AIX rsh man page for more information. Also, consult IBM Distributed Computing Environment 3.1 for AIX: Administration Guide and Reference for information about principals, accounts, and forwarding of credentials.