Kernel Extensions and Device Support Programming Concepts

FCP and iSCSI Adapter and Device Interface

The adapter device driver does not contain the ddread and ddwrite entry points, but does contain the ddconfig, ddopen, ddclose, dddump, and ddioctl entry points.

Therefore, the adapter device driver's entry in the kernel devsw table contains only those entries plus an additional ddstrategy entry point. This ddstrategy routine is the path that the device driver uses to pass commands to the device driver. Access to these entry points is possible through the following kernel services:

• fp_open
• fp_close
• devdump
• fp_ioctl
• devstrat

The adapter is accessed by the device driver through the /dev/fscsi# special files, where # indicates ascending numbers 0,1, 2, and so on. The adapter is designed so that multiple devices on the same adapter can be accessed at the same time.

The iSCSI adapter is accessed by the device driver through the /dev/iscsin special files, where n indicates ascending numbers 0, 1, 2, and so on. The adapter is designed so that multiple devices on the same adapter can be accessed at the same time.

For additional information on spanned and gathered write commands, see Understanding the Execution of FCP and iSCSI Initiator I/O Requests.

scsi_buf Structure

The I/O requests made from the device driver to the adapter device driver are completed through the use of the scsi_buf structure, which is defined in the /usr/include/sys/scsi_buf.h header file. This structure, which is similar to the buf structure in other drivers, is passed between the two subsystem drivers through the strategy routine. The following is a brief description of the fields contained in the scsi_buf structure:

• Reserved fields should be set to a value of 0, except where noted.
• The bufstruct field contains a copy of the standard buf buffer structure that documents the I/O request. Included in this structure, for example, are the buffer address, byte count, and transfer direction. The b_work field in the buf structure is reserved for use by the adapter device driver. The current definition of the buf structure is in the /usr/include/sys/buf.h include file.
• The bp field points to the original buffer structure received by the Device Driver from the caller, if any. This can be a chain of entries in the case of spanned transfers (commands that transfer data from or to more than one system-memory buffer). A null pointer indicates a nonspanned transfer. The null value specifically tells the adapter device driver that all the information needed to perform the DMA data transfer is contained in the bufstruct fields of the scsi_buf structure.
• The scsi_command field, defined as a scsi_cmd structure, contains, for example, the SCSI command length, SCSI command, and a flag variable:
  • The scsi_length field is the number of bytes in the actual SCSI command. This is normally 6,10,12, or 16 (decimal).
  • The FCP_flags field contains the following bit flags:

    SC_NODISC

    Do not allow the target to disconnect during this command.

    SC_ASYNC

    Do not allow the adapter to negotiate for synchronous transfer to the device.

    During normal use, the SC_NODISC bit should not be set. Setting this bit allows a device executing commands to monopolize the transport layer. Sometimes it is desirable for a particular device to maintain control of the transport layer once it has successfully arbitrated for it; for instance, when this is the only device on the transport layer or the only device that will be in use. For performance reasons, it might not be desirable to go through selections again to save transport layer overhead on each command.

    Also during normal use, the SC_ASYNC bit must not be set. It should be set only in cases where a previous command to the device ended in an unexpected transport free condition. This condition is noted as SCSI_TRANSPORT_FAULT in the adapter_status field of the scsi_cmd structure. Because other errors might also result in the SCSI_TRANSPORT_FAULT flag being set, the SC_ASYNC bit should only be set on the last retry of the failed command.

  • The scsi_cdb structure contains the physical SCSI command block. The 6 to 16 bytes of a single SCSI command are stored in consecutive bytes, with the op code identified individually. The scsi_cdb structure contains the following fields:
    1. The scsi_op_code field specifies the standard op code for this command.
    2. The scsi_bytes field contains the remaining command-unique bytes of the command block. The actual number of bytes depends on the value in the scsi_op_code field.
• The timeout_value field specifies the time-out limit (in seconds) to be used for completion of this command. A time-out value of 0 means no time-out is applied to this I/O request.
• The status_validity field contains an output parameter that can have one of the following bit flags as a value:

  SC_SCSI_ERROR

  The scsi_status field is valid.

  SC_ADAPTER_ERROR

  The adapter_status field is valid.

• The scsi_status field in the scsi_buf structure is an output parameter that provides valid command completion status when its status_validity bit is nonzero. The scsi_buf.bufstruct.b_error field should be set to EIO anytime the scsi_status field is valid. Typical status values include:

  SC_GOOD_STATUS

  The target successfully completed the command.

  SC_CHECK_CONDITION

  The target is reporting an error, exception, or other conditions.

  SC_BUSY_STATUS

  The target is currently transporting and cannot accept a command now.

  SC_RESERVATION_CONFLICT

  The target is reserved by another initiator and cannot be accessed.

  SC_COMMAND_TERMINATED

  The target terminated this command after receiving a terminate I/O process message from the adapter.

  SC_QUEUE_FULL

  The target's command queue is full, so this command is returned.

  SC_ACA_ACTIVE

  The device has an ACA (auto contingent allegiance) condition that requires a Clear ACA to request to clear it.

• The adapter_status field is an output parameter that is valid when its status_validity bit is nonzero. The scsi_buf.bufstruct.b_erro field should be set to EIO anytime the adapter_status field is valid. This field contains generic adapter card status. It is intentionally general in coverage so that it can report error status from any typical adapter.

  If an error is detected during execution of a command, and the error prevented the command from actually being sent to the transport layer by the adapter, then the error should be processed or recovered, or both, by the adapter device driver.

  If it is recovered successfully by the adapter device driver, the error is logged, as appropriate, but is not reflected in the adapter_status byte. If the error cannot be recovered by the adapter device driver, the appropriate adapter_status bit is set and the scsi_buf structure is returned to the device driver for further processing.

  If an error is detected after the command was actually sent to the device, then it should be processed or recovered, or both, by the device driver.

  For error logging, the adapter device driver logs transport layer and adapter-related conditions, andl the device driver logs device-related errors. In the following description, a capital letter (A) after the error name indicates that the adapter device driver handles error logging. A capital letter (H) indicates that the device driver handles error logging.

  Some of the following error conditions indicate a device failure. Others are transport layer or adapter-related.

  SCSI_HOST_IO_BUS_ERR (A)

  The system I/O transport layer generated or detected an error during a DMA or Programmed I/O (PIO) transfer.

  SCSI_TRANSPORT_FAULT (H)

  The transport protocol or hardware was unsuccessful.

  SCSI_CMD_TIMEOUT (H)

  The command timed out before completion.

  SCSI_NO_DEVICE_RESPONSE (H)

  The target device did not respond to selection phase.

  SCSI_ADAPTER_HDW_FAILURE (A)

  The adapter indicated an onboard hardware failure.

  SCSI_ADAPTER_SFW_FAILURE (A)

  The adapter indicated microcode failure.

  SCSI_FUSE_OR_TERMINAL_PWR (A)

  The adapter indicated a blown terminator fuse or bad termination.

  SCSI_TRANSPORT_RESET (A)

  The adapter indicated the transport layer has been reset.

  SCSI_WW_NAME_CHANGE (A)

  The adapter indicated the device at this SCSI ID has a new world wide name.

  SCSI_TRANSPORT_BUSY (A)

  The adapter indicated the transport layer is busy.

  SCSI_TRANSPORT_DEAD (A)

  The adapter indicated the transport layer currently inoperative and is likely to remain this way for an extended time.

• The add_status field contains additional device status. For devices, this field contains the Response code returned.
• When the FCP device driver queues multiple transactions to a device, the adap_q_status field indicates whether or not the FCP adapter driver has cleared its queue for this device after an error has occurred. The flag of SC_DID_NOT CLEAR_Q indicates that the adapter driver has not cleared its queue for this device and that it is in a halted state (so none of the pending queued transactions are sent to the device).
• The q_tag_msg field indicates if the adapter can attempt to queue this transaction to the device. This information causes the adapter to fill in the Queue Tag Message Code of the queue tag message for a command. The following values are valid for this field:

  SC_NO_Q

  Specifies that the adapter does not send a queue tag message for this command, and so the device does not allow more than one command on its command queue. This value must be used for all commands sent to devices that do not support command tag queuing.

  SC_SIMPLE_Q

  Specifies placing this command in the device's command queue. The device determines the order that it executes commands in its queue. The SCSI-2 specification calls this value the Simple Queue Tag Message.

  SC_HEAD_OF_Q

  Specifies placing this command first in the device's command queue. This command does not preempt an active command at the device, but it is executed before all other commands in the command queue. The SCSI-2 specification calls this value the Head of Queue Tag Message.

  SC_ORDERED_Q

  Specifies placing this command in the device's command queue. The device processes these commands in the order that they are received. The SCSI-2 specification calls this value the Ordered Queue Tag Message.

  SC_ACA_Q

  Specifies placing this command in the device's command queue, when the device has an ACA (auto contingent allegiance) condition. The SCSI-3 Architecture Model calls this value the ACA task attribute.

  Note
  Commands with the value of SC_NO_Q for the q_tag_msg field (except for request sense commands) should not be queued to a device whose queue contains a command with another value for q_tag_msg. If commands with the SC_NO_Q value (except for request sense) are sent to the device, then the device driver must make sure that no active commands are using different values for q_tag_ms. Similarly, the device driver must also make sure that a command with a q_tag_msg value of SC_ORDERED_Q, SC_HEAD_Q, or SC_SIMPLE_Q is not sent to a device that has a command with the q_tag_msg field of SC_NO_Q.
• The flags field contains bit flags sent from the device driver to the adapter device driver. The following flags are defined:

  SC_RESUME

  When set, means the adapter device driver should resume transaction queuing for this ID/LUN. Error recovery is complete after a SCIOLHALT operation, check condition, or severe transport error. This flag is used to restart the adapter device driver following a reported error.

  SC_DELAY_CMD

  When set, means the adapter device driver should delay sending this command (following a reset or BDR to this device) by at least the number of seconds specified to the adapter device driver in its configuration information. For devices that do not require this function, this flag should not be set.

  SC_Q_CLR

  When set, means the adapter driver should clear its transaction queue for this ID/LUN. The transaction containing this flag setting does not require an actual command in the scsi_buf because it is flushed back to the device driver with the rest of the transactions for this ID/LUN. However, this transaction must have the SCSI ID field (scsi_buf.scsi_id) and the LUN field (scsi_buf.lun_id) filled in with the device's SCSI ID and LUN. This flag is valid only during error recovery of a check condition or command terminated at a command tag queuing device when the SC_DID_NOT_CLR_Q flag is set in the scsi_buf.adap_q_status field.

  SC_Q_RESUME

  When set, means that the adapter driver should resume its halted transaction queue for this ID/LUN. The transaction containing this flag setting does not require an actual command to be sent to the adapter driver. However, this transaction must have the SCSI ID field (scsi_buf.scsi_id) and the LUN field (scsi_buf.lun_id) filled in with the device's SCSI ID and logical unit number (LUN). If the transaction containing this flag setting is the first issued by the device driver after it receives an error (indicating that the adapter driver's queue is halted), then the SC_RESUME flag must be set also.

  SC_CLEAR_ACA

  When set, means the SCSI adapter driver should issue a Clear ACA task management request for this ID/LUN. This flag should be used in conjunction with either the SC_Q_CLR or SC_Q_RESUME flags to clear or resume the SCSI adapter driver's queue for this device. If neither of these flags is used, then this transaction is treated as if the SC_Q_RESUME flag is also set. The transaction containing the SC_CLEAR_ACA flag setting does not require an actual SCSI command in the sc_buf. If this transaction contains a SCSI command then it will be processed depending on whether SC_Q_CLR or SC_Q_RESUME is set. This transaction must have the SCSI ID field (scsi_buf.scsi_id) and the LUN field (scsi_buf.lun_id) filled in with the device's SCSI ID and LUN. This flag is valid only during error recovery of a check condition or command terminated at a command tag queuing.

  SC_TARGET_RESET

  When set, means the SCSI adapter driver should issue a Target Reset task management request for this ID/LUN. This flag should be used in conjunction with ethe SC_Q_CLR flag flag.The transaction containing this flag setting does allow an actual command to be sent to the adapter driver. However, this transaction must have the SCSI ID field (scsi_buf.scsi_id) filled in with the device's SCSI ID. If the transaction containing this flag setting is the first issued by the device driver after it receives an error (indicating that the adapter driver's queue is halted), then the SC_RESUME flag must be set also.

  SC_LUN_RESET

  When set, means the SCSI adapter driver should issue a Lun Reset task management request for this ID/LUN. This flag should be used in conjunction with ethe SC_Q_CLR flag flag.The transaction containing this flag setting does allow an actual command to be sent to the adapter driver. However, this transaction must have the the SCSI ID field (scsi_buf.scsi_id) and the LUN field (scsi_buf.lun_id) filled in with the device's SCSI ID and logical unit number (LUN). If the transaction containing this flag setting is the first issued by the device driver after it receives an error (indicating that the adapter driver's queue is halted), then the SC_RESUME flag must be set also.

• The dev_flags field contains additional values sent from the FCP device driver to the FCP adapter device driver. This field is not used for iSCSI device drivers. The following values are defined:

  FC_CLASS1

  When set, this tells the SCSI adapter driver that it should issue this request as a Fibre Channel Class 1 request. If the SCSI adapter driver does not support this class, then it will fail the scsi_buf with an error of EINVAL. If no Fibre Channel Class is specified in the scsi_buf then the SCSI adapter will default to a Fibre Channel Class.

  FC_CLASS2

  When set, this tells the SCSI adapter driver that it should issue this request as a Fibre Channel Class 2 request. If the SCSI adapter driver does not support this class, then it will fail the scsi_buf with an error of EINVAL. If no Fibre Channel Class is specified in the scsi_buf then the SCSI adapter will default to a Fibre Channel Class.

  FC_CLASS3

  When set, this tells the SCSI adapter driver that it should issue this request as a Fibre Channel Class 3 request. If the SCSI adapter driver does not support this class, then it will fail the scsi_buf with an error of EINVAL. If no Fibre Channel Class is specified in the scsi_buf then the SCSI adapter will default to a Fibre Channel Class.

  FC_CLASS4

  When set, this tells the SCSI adapter driver that it should issue this request as a Fibre Channel Class 4 request. If the SCSI adapter driver does not support this class, then it will fail the scsi_buf with an error of EINVAL. If no Fibre Channel Class is specified in the scsi_buf then the SCSI adapter will default to a Fibre Channel Class.

• The add_work field is reserved for use by the adapter device driver.
• The adap_set_flags field contains an output parameter that can have one of the following bit flags as a value:

  SC_AUTOSENSE_DATA_VALID

  Autosense data was placed in the autosense buffer referenced by the autosense_buffer_ptr field.

• The autosense_length field contains the length in bytes of the SCSI device driver's sense buffer, which is referenced via the autosense_buffer_ptr field. For devices this field must be non-zero, otherwise the autosense data will be lost.
• The autosense_buffer_ptr field contains the address of the SCSI devices driver's autosense buffer for this command. For devices this field must be non-NULL, otherwise the autosense data will be lost.
• The dev_burst_len field contains the burst size if this write operation in bytes. This should only be set by the device driver if it has negotiated with the device and it allows burst of write data without transfer readys. For most operations, this should be set to 0.
• The scsi_id field contains the 64-bit SCSI ID for this device. This field must be set for FCP devices.
• The lun_id field contains the 64-bit lun ID for this device. This field must be set for devices.
• The kernext_handle field contains the pointer returned from the kernext_handle field of the scsi_sciolst argument for the SCIOLSTART ioctl.

Adapter and Device Driver Intercommunication

In a typical request to the device driver, a call is first made to the device driver's strategy routine, which takes care of any necessary queuing. The device driver's strategy routine then calls the device driver's start routine, which fills in the scsi_buf structure and calls the adapter driver's strategy routine through the devstrat kernel service.

The adapter driver's strategy routine validates all of the information contained in the scsi_buf structure and also performs any necessary queuing of the transaction request. If no queuing is necessary, the adapter driver's start subroutine is called.

When an interrupt occurs, adapter driver interrupt routine fills in the status_validity field and the appropriate scsi_status or adapter_status field of the scsi_buf structure. The bufstruct.b_resid field is also filled in with the value of nontransferred bytes. The adapter driver's interrupt routine then passes this newly filled in scsi_buf structure to the iodone kernel service, which then signals the device driver's iodone subroutine. The adapter driver's start routine is also called from the interrupt routine to process any additional transactions on the queue.

The device driver's iodone routine should then process all of the applicable fields in the queued scsi_buf structure for any errors and attempt error recovery if necessary. The device driver should then dequeue the scsi_buf structure and then pass a pointer to the structure back to the iodone kernel service so that it can notify the originator of the request.