View figure.
Thin, wide, and high nodes can coexist in the same frame and in the same SP system partition. Whether or not you use nodes with varied physical node sizes, you must very carefully consider the set of supported frame configurations. Extension nodes (like an SP Switch Router), SP-attached servers, and SP Expansion I/O Units must also be accommodated. Use the information in this section for:
| Hardware planning is described in Volume 1. |
|---|
|
This book covers switch planning only in the context of system configuration. For physical planning regarding switch wiring, cabling, and allowing for future hardware expansion, see the book IBM RS/6000 SP: Planning Volume 1, Hardware and Physical Environment. |
Each tall SP frame contains eight drawers which have two slots each for a total of 16 slots. The short SP frame has only four drawers and eight slots. When viewing a tall SP frame from the front, the 16 slots are numbered sequentially from bottom left to top right.
The position of a node in an SP system is sensed by the hardware. That position is the slot to which it is wired. That slot is the slot number of the node.
Figure 13. Node slot assignment
View figure.
|The administrator establishes the frame numbers when the system is |installed. To allow for growth you can skip frame numbers, but the |highest number you can use for frames with nodes is 128. You can use |higher frame numbers for additional frames that contain only switches or SP |Expansion I/O Units. Each frame is referenced by the tty port to which |the frame supervisor is attached and is assigned a numeric identifier. |The order in which the frames are numbered determines the sequence in which |they are examined by the PSSP software during the configuration |process. This order is used to assign global identifiers to the switch |ports, nodes, and SP Expansion I/O Units. This is also the order used |to determine which frames share a switch.
For this discussion the types of frames are the following:
|If you have multiple NSB, switch-only, or non-node frames, configure |them as the last frames in your SP system with numbers above 128. |Assign high frame numbers to allow maximum configuration of nodes and for |future expansion.
|If you are planning a system of clustered enterprise servers, you |can assign frame numbers in any order within the noted ranges. See Considering a clustered enterprise server configuration.
A node number is a global ID assigned to a node. It is the primary means by which an administrator can reference a specific node in the system. Node numbers are assigned for all nodes regardless of node or frame type. Replace node number with expansion number for calculating the global ID of an SP Expansion I/O Unit. Global IDs are automatically assigned using the following formula:
where slot_number is the lowest slot number occupied by the node or unit. |Each type is considered to occupy one slot or a consecutive sequence |of slots. For each type, there is an integer n such that a |thin node, a p690 LPAR, or an I/O expansion unit occupies slot n, a |wide node occupies slots n, n+1, and a high node occupies n, |n+1, n+2, n+3. An SP-attached server is considered to be one node |in one frame, unless it is a p690 with multiple LPARs, in which case each |LPAR is a node in that frame. For single thin nodes (not in a pair), |wide nodes, and high nodes, n must be odd. For an SP-attached server, |n is 1. Use n in place of slot_number |in the formula.
Node numbers are assigned independent of whether the frame is fully populated. Figure 14 demonstrates node numbering. Frame 4 represents an SP-attached server in a position where it does not interrupt the switched frame and companion non-switched expansion frame, in other words the switch capsule configuration. It can use a switch port on frame 2 which is left available by the high nodes in frame 3. Its node number is determined by using the formula.
Figure 14. Node numbering
View figure.
In order to properly plan an SP system with the SP Switch, you must understand the supported frame and switch configurations and the distribution of the switch port assignments in each of the supported configurations.
The PSSP system with the SP Switch supports the four possible frame and switch configurations shown in Figure 15. Think of each configuration as a switch capsule which is comprised of a switched frame and its possible companion non-switched expansion frames. A non-switched expansion frame is a successor frame within the switch capsule that has SP nodes using switch ports of the switched frame.
Figure 15 has four switch capsules, illustrating the four frame and switch configurations that are supported and the switch port number assignments in each. In the figure, no shading indicates a valid slot in which a node can be placed, the number in the slot represents that node's switch port assignment, and shading indicates that a node cannot be placed in that slot.
The four switch capsules can be repeated and mixed throughout your SP system. For example, consider an SP system with a switched frame followed by two non-switched expansion frames. They in turn might be followed by another switched frame and one more non-switched expansion frame. That SP system is therefore comprised of one switch capsule matching configuration 2 followed by another switch capsule matching configuration 1.
In every configuration that follows the first, the switch port numbers assigned increment to allow for the switches that reside in numerically lower frames. In the previous example, the frames in the first switch capsule (matching configuration 2) are assigned switch port numbers 0 through 15 and the frames in the second switch capsule (matching configuration 1) are assigned switch port numbers 16 through 31.
Figure 15. Supported SP Switch configurations showing switch port assignments
View figure.
Keep in mind that any non-switched expansion frames must have frame numbers that immediately follow their associated switched frame without any gaps. For instance, if a system has a switched frame numbered 1, and two non-switched expansion frames attached to the switch on frame 1, the non-switched expansion frames must be numbered frame 2 and frame 3.
Frame numbers can be skipped between switched frames. It is a good idea to skip numbers to allow for future expansion. For example, consider a system that has a switched frame with four high nodes and another switched frame with 16 thin nodes. At this point the switched frame with four high nodes is a switch capsule that is not fully populated. To accommodate future expansion, you would be wise to assign frame number 1 to the high node frame and number 5 to the thin node frame. That allows for the future addition of up to three non-switched expansion frames to the high node frame without disrupting the system. If the thin node frame had been numbered frame 2, the addition of a non-switched expansion frame would require you to reconfigure the thin node frame and all of its nodes.
An SP-attached server is managed by the PSSP components as though it is in a frame of its own. Servers with multiple LPARs are managed as a separate node for each LPAR within that frame. However, a server does not enter into the determination of the frame and switch capsule configuration of your SP system. It has the following additional characteristics:
SP Expansion I/O Units also do not enter into the determination of the frame and switch capsule configuration. They do not connect to a switch so the switch port corresponding to the slot occupied by an SP Expansion I/O Unit is available to be otherwise used, for instance by an SP-attached server or SP Switch Router. The nodes to which SP Expansion I/O Units are connected must honor the placement convention. These expansion units however can be placed in any unoccupied slot of a frame with nodes or in a non-node frame that is used specifically for them. In a system with the SP Switch, give a non-node frame a number that fits between switch capsules.
|An SP Switch2 system is more flexible than an SP Switch system in regard to |node placement. Actually the SP Switch2 has no restrictions on node |placement. A node can be placed anywhere it is allowed by the physical |constraints. The only SP Switch2 rules have to do with the system |configuration and with nodes connecting to the SP Switch2, which are the |following: |
|Figure 16 shows an example of a system with a two-plane SP |Switch2 configuration.
|Figure 16. Nodes in a two-plane SP Switch2 configuration
|
View figure.
|Every node in the system has two switch adapters, css0 |connected to sw1 and css1 connected to sw2. The |css0 network is plane 0 and the css1 network |is plane 1. Additionally the system is comprised of the |following: |
|There is still room for upgrades in the future. For instance, by |replacing the wide nodes in Frame 1 with two high nodes, another SP expansion |frame as Frame 4 can house two more high nodes and SP Expansion I/O |Units. With the SP Switch2, node numbers and switch port numbers are |automatically generated.
In a switched system, the switch boards are attached to each other to form a larger communication fabric. Each switch provides some number of ports to which a node can connect (16 ports for an SP Switch, and 8 ports for the SP Switch-8.) In larger systems, additional switch boards (intermediate switch boards) must be introduced to provide for switch board connectivity; such boards do not provide node switch ports.
Switch boards are numbered sequentially starting with 1 from the frame with the lowest frame number to that with the highest frame number. Each full switch board contains a range of 16 switch port numbers (also known as switch node numbers) that can be assigned. These ranges are also in sequential order with their switch board number. For example, switch board 1 contains switch port numbers 0 through 15.
Switch port numbers are used internally in PSSP software as a direct index into the switch topology and to determine routes between switch nodes.
|The switch port numbers for an SP Switch2 system are automatically |assigned sequentially by the PSSP 3.4 switch management component |(CSS). As a node is assigned a CSS adapter it is given the lowest |available switch node number from 0 through 511. There is no |correlation between the switch port number and any hardware |connections.
The SP Switch has 16 ports. Whether a node is connected to a switch within its frame or to a switch outside of its frame, you can evaluate the following formula to determine the switch port number to which a node is attached:
where switch_number is the number of the switch board to which the node is connected and switch_port_assigned is the number assigned to the port on the switch board (0 to 15) to which the node is connected. This is demonstrated in Figure 18.
For additional explanation with switch port numbers, see Chapter 7, Planning SP system partitions, particularly Example 3 - An SP with 3 frames, 2 SP Switches, and various node sizes.
Node numbers for short and tall frames are assigned using the same algorithm. See Node numbering.
However, for the SP Switch-8, a different algorithm is used for assigning nodes their switch port numbers. A system with this switch contains only switch port numbers 0 through 7.
The following algorithm is used to assign nodes their switch port numbers in systems with eight port switches:
Repeat until you reach the last slot in the frame or switch port number 7, whichever comes first.
Figure 17 and Table 41 contain sample switch port numbers for a system with a short frame and an eight port switch.
Figure 17. Switch port numbering for an SP Switch-8 in a short frame
View figure.
Table 41. Sample switch port numbers for the SP Switch-8
| Slot Number | Populated? | Node Number | Switch Port Number |
|---|---|---|---|
| 1 | Yes | 1 | 0 |
| 2 | No |
|
|
| 3 | Yes | 3 | 1 |
| 4 | No |
|
|
| 5 | Yes | 5 | 2 |
| 6 | Yes | 6 | 3 |
| 7 | Yes | 7 | 4 |
| 8 | Yes | 8 | 5 |
| 9 - 16* | No |
|
|
* Slot numbers 9-16 are used only for tall models.
You need to plan a switch network, even if you do not plan to use an SP Switch, whenever you plan to use any of the following:
In any switchless system, a logical switch port number can be evaluated using frame numbers and slot numbers with the following formula:
where frame_number is the number of the frame in which the node is located and slot_number is the lowest numbered slot the node occupies from 1 to 16. For SP-attached servers, slot number is that of the associated slot in an SP frame where the switch port will not be otherwise used.
Even in a switchless system, you need to fill in the switch worksheet to set Switch Port Number when you plan to use SP-attached servers. You can do this for a clustered enterprise server as if it were an SP-attached server |if you might add an SP Switch or expand to a standard SP system in the future. During the system installation and configuration process, you will be asked to enter the value. This is because PSSP cannot dynamically determine the value as it can for SP nodes. See Completing the switch configuration worksheet.
Switch port numbering is used to determine the IP address of the nodes on the switch. If your system is not ARP-enabled on the css0 adapter, and the css1 adapter in a two-plane SP Switch2 system, choose the IP address of the first node on the first frame. The switch port number is used as an offset added to that address to calculate all other switch IP addresses.
|If ARP is enabled for the css0 and css1 |adapters, the IP addresses can be assigned like any other adapter. That |is, they can be assigned beginning and ending at any node. They do not |have to be contiguous addresses for all the css0 and |css1 adapters in the system. Switch port numbers are |automatically generated by the SP Switch2.
| Figure 18 illustrates the switch port numbers for an SP |system that uses the SP Switch. It also illustrates how the switch port numbers are set for a non-switched expansion frame and for an SP-attached server. In Figure 18, Switch 2 connects to the nodes in Frame 3. Specifically, the nodes of Frame 3 use respective ports of Switch 2 which are not used by nodes in Frame 2. To determine the switch port number for nodes that are not in a switched frame, use the following formula:
Based on the formula, the first slot of Frame 3 has switch port number 17. The formula also results in switch port number 27 for the SP-attached server as frame 4 using switch port 11 in switch number 2.
Figure 18. Switch port numbering sequence
View figure.