9037-002 Maintenance Introduction.

Bernard TANGUY
Dept. 1590
TANGUYB at IBMFR
Tel. (0)4-67-34-44-90
S/390 P.E. Montpellier
IBM FRANCE

22nd/09/2000.

Document Number 9037-002

September 2000.

Last version updates: see abstracts below.

Documentation issued by: P.E. MOP Dpt 1590.

Preface

Its intent is to complete the course and to give hands-on about the installation, maintenance, documentation and education.

It will be maintained from our common experience.

Any comments or suggestions are welcome.

Do not copy this document, request the last version from the web MOP PE server (http://moppe.mop.ibm.com) or VM (contact EPSG or myself).

• 1st version dated 25/11/96.
• last version -13th- dated 22nd/Sept/2000.

• Document review
• No leap second to occur on dec 31th (update 5.5)
• Interconnect Products Alert site link: 2029 - 9032 - 9037 (new 1.4 )

Previous version updates:

• Edited on December 24th to update the 2029 redbook URL (upd 1.4)
• Included the 2029 in the link extension RPQ 8P1955. (new 2.2/2.9)
• Sysplex Timer links are supported by the 2029 as "base channels", ("high availability" channel configuration not supported ). Sysplex Timer links are not supported in hubbed ring configurations. (new 2.9).
• 2029 intersites maximum distances. (new 2.9).
• 2029 redpiece URL (new 1.4).

  ---

  Table of Contents

  Preface

  Chapter 1. 9037 Model 2 Documentation - Education

  1.1 Customer Documentation
  1.2 C.E. Documentation
  1.3 Tools
  1.4 Other Documentation/redbook/PE links:
  Chapter 2. Installation
  2.1 Configurations:
  2.2 Features:
  2.3 Recommendations:
  2.4 LAN:
  2.5 Initialization / Offset changes:
  2.6 Distant Sites Installation Sequence (E.A. Configuration):
  2.7 Upgrade an Expanded Basic Configuration to Expanded Availability:
  2.8 Distant Sites Upgrade Sequence (E.B. to E.A. Configuration):
  2.9 Extended distance (2029/9729):
  2.10 Timer Network display:
  2.11 Console configuration:
  2.12 Consoles switching:
  2.13 DCAF control (Remote Access Console):
  2.14 Adding a Port card:
  2.15 TIPS (EXTRACT of 9037 model 2 related tips):
  Chapter 3. Maintenance Information
  3.1 Maintenance guidelines
  3.2 Problem determination - "Maintenance Information" documentation:
  3.3 Support request:
  3.4 Timer LIC and Console Application levels:
  3.5 Console Restore tapes - optical disk -diskettes P/Ns:
  3.6 Timer LIC install:
  3.7 MVS Commands:
  3.8 WAIT CODES 0A2:
  3.9 Token ring failure:
  3.10 Loss of communication -RC 040/041- with the Active (Standby) console:
  Chapter 4. E.C.s (purpose and contents):
  4.1 LIC HISTORY:
  4.2 ECA009 - Version 02.00.00 information:
  4.3 ECA010 - Selective PSA replacement:
  4.4 Selected CTP cards replacement:
  Chapter 5. External Time Source
  5.1 Sysplex Timer accuracy and E.T.S.:
  5.2 External Time Source tests:
  5.3 Basic Configuration:
  5.4 Time tracking:
  5.5 Leap seconds:
  5.6 Modem Access - US and EMEA telephone numbers:
  5.7 ETS connector pin numbering:
  5.8 GPS/Low freq receiver Protocol 1 documentation:
  Chapter 6. Sysplex disaster recovery procedure
  6.1 9037 model 2 disaster scenario:
  6.2 9037 model 2 disaster prevention:
  6.3 9037 model 2 disaster recovery:
  Chapter 7. Hardware Management Console Support for the 9037 Model 2 Sysplex Timer
  7.1 Feature / Documentation
  7.2 Installation Pre-reqs.
  7.3 Installation Summary
  7.4 Console Application backup:
  7.5 Keyboard customization:
  7.6 "Undefined Timer/Director Consoles" Icone (Group view):
  7.7 Miscellaneous
  Chapter 8. ESCON Director Console Support for the 9037 Model 2 Sysplex Timer
  8.1 Feature / Documentation
  8.2 Installation Pre-reqs (minimum):
  8.3 Installation Summary
  8.4 Miscellaneous

Chapter 1. 9037 Model 2 Documentation - Education

1.1 Customer Documentation

• Planning Guide SA22-7233 (overview and planning for the Sysplex Timer).
• 9037-002 User Guide SA22-7230 shipped with the Unit(s).
• User Guide Reference Summary SA22-7231 shipped with the Unit(s).
• Operator Panel Reference Summary SA22-7232 shipped with the Unit(s).
• Customer Exception Letter DocNum. T200CU01 shipped with the Unit(s), dated October 30th, 1998 for the current level (02.00.00) of new build or recently upgraded machines. A copy can be obtained from the server with the following VM command: REQUEST T200CU01 PACKAGE FROM FOCUSPE AT PKEDVM9
• Customer Exception Letter DocNum. T130CU03 shipped with the Unit(s), dated July 17, 1998 for the previous level (01.03.00). A copy can be obtained from the server with the following VM command: REQUEST T130CU03 PACKAGE FROM FOCUSPE AT PKEDVM9

1.2 C.E. Documentation

• Maintenance Information SY27-2641 shipped with the Unit(s)
  The current MIM version (SY27-2641-02) is completed by the Service Exception Letter - T200SR01 October 30th, 1998 - A copy can be obtained from the server with the following VM command: REQUEST T200SR01 PACKAGE FROM FOCUSPE AT PKEDVM9
• "S/390 Console Installation and User's Guide - 9032 Models 3 and 5 Escon Director - 9033 Model 4 Escon Director - 9037 Model 2 Sysplex Timer" GA22-7291 shipped with the Unit(s) to install and use the PC-330, the PC 365,the HMC, the ESCD consoles) etc. (delivered from 01.03.00 LIC level).
• Hardware Management Console Support for the 9037 Model 2 Sysplex Timer PN 620-900012-000 shipped with FC 4910
• Escon Director Console Support for the 9037 Model 2 Sysplex Timer PN 620-900013-000 shipped with FC 4910
• 9037-001 to 9037-002 Migration Planning (MKTTOOLS)
• 9037-001 to 9037-002 Migration Procedures obtained from FOCUSPE. See the above planning doc and "TIPS" in the next Chapter ("Installation").

1.3 Tools

Fiber optic protective plug (must be present as often as a fiber is not connected including to return a card for repair).

Service key.

Torque tool for logic cards (unique torque specifications).

CBT Course (WIN/xx Sysplex Timer training course: 3 diskettes). This 2 hours Computer Based Terminal Course runs under WIN-OS/2, WINDOWS 3.1, WIN/NT. The following documentation is necessary:

• Maintenance Information SY27-2641-02.
• 9037-002 User Guide SA22-7230.

The last version of the CBT course can be requested with the command "TOOLS SENDTO LEXVMIC1 TOOLS SERVTOOL GET 9037CBT PACKAGE". Follow the instructions in the file "9037CBT PACKAGE" to create and install the diskettes. The timestamp for the 9037CBT* DSKFBIN files is 24 Oct 1996.

Useful "helps" from the Console Application (or CBT Standalone C.A.)

• PF1 to obtain information on any components of the timer itself or function of the console application.
• Click on HELP then HOW DO I to obtain the procedures to be followed for proper timer network operations.

To clear check marks of completed modules and lessons use "CRTL+C".

Report any error observed during the CBT course use.

1.4 Other Documentation/redbook/PE links:

9037 Redbook:
"S/390 Time Management and IBM 9037 Sysplex Timer redbook link"
 

9729 Wavelength Division Multiplexer in Large Systems Environment:
"9729 redbook Link"
 

2029 Redbook:
"IBM 2029 Fiber Saver Implementation Guide, SG24-5608"
 

FocusPE link for ETS protocols description, Interconnect products year 2000 information, Sysplex Timer Console Application on the S/390 HMC II, Model 1 to Model 2 migration procedure, Exception Letters, LIC, etc.
"FocusPE link"
 

The Montpellier P.E. "9037-002 Maintenance Introduction" document is on the MOP PE server:
"MOP PE server"
 

Interconnect Product Alert site (2029 / 9032 / 9037) :
"ICP alerts"

Chapter 2. Installation

2.1 Configurations:

A standby console is recommended, it is optional for every configuration.

A Basic configuration is not upgradable concurrently, as the CLO card needs to be installed.

Two ETR networks can be monitored from a console.

2.2 Features:

The M.A.U. is not automatically delivered with the 9037-002.

Here are the feature codes for 9037-002:

• FC4040 Port Expansion Feature ( 4 additional ports ).
• FC4048 Expanded Availability.
• FC4400 Multistation Access Unit.
• FC4900 Console (Plant Merge).
• FC4910 HMC and ESCD Console Support (See the proper chapters).
• FC6150 ETR master card.
• FC6151 ETR slave card.
• FC6152 ETR dual port card.
• FC6153 ETR External Cable Feature.
• RPQ 8K1919 Repeater - 9036-003.
• RPQ 8K1903 Used with above (pointer from 9036).
• RPQ 8Q1488 Wavelength division multiplexor (9729).
• RPQ 8P1955 CF/Sysplex Timer Link extension with 9729/2029.

2.3 Recommendations:

The Planning Guide lists recommendations for a continuous availability .

The total cable distance between both boxes (CLO links) cannot exceed 40 kms (RPQ 8P1955).

The CPC Transmit and Receive fibers must have matched lengths within 10 meters (using trunks and distribution panels).

Example of single point of failure consequences: Both CLO fibers paths (A and B) must be different as the loss of both signals will get the secondary timer to the "Dormant" status i.e. not emitting to the CPCs until the resynchronisation with the primary is successful via one CLO link. This could lead to system(s) down if the alternate ETR port of one or more CECs does not receive the ETR signal from the primary timer as a consequence of the current repeater failure (same paths used), or for any other reason. Also read the "Sysplex disaster recovery" chapter in this document.

Prior to the timers installation you need to determine:

• The Network Address (Keep the same as the one already in use in case of concurrent migration from a model 001 to a model 002, refered to as "Limited Outage Migration procedure" ). Once initialized the only way to modify it is disruptive (The Network ID is checked by MVS). After a customer outage is planned, this could be done by an initialization ( from the Initialize Timer Network dialog box).
  At level 01.02.00 an Initialization with a new Network Address will result in a sequence of RCs 77/93/06 and S.E.L. on without impact as the CLO A error is recovered (see TIP H057415 - corrected by LIC 01.03.00).
• The Unit Adresses or Box Ids (ETR IDs for MVS). Use different IDs from the existing ones in case of concurrent migration - Limited customer operations outage - when the procedure is available (See "TIPS" in this chapter).
• The IP Addresses if the LAN is not dedicated or if the console application is not installed on a dedicated PC (HMC or ESCON Director console). Though the TCP/IP address can be modified online you could plan for the future if dedicated PCs are to be installed now and planned to be connected later to an existing LAN (for instance to have a HMC as spare console to replace a defective dedicated one): have matching subnet addresses now (3 first octets with a subnet mask of 255.255.255), remember the default subnet mask is to be modified on a dedicated PC, then insure the TCP/IP address is unique (last octet). Do not use a TCP/IP address of 0 which is the network address, or 255 which is the broadcast address (e.g. 9.130.1.0 or 9.130.1.255 are not to be used using the above subnet mask). Depending upon the subnet mask length, these reserved addresses for the resulting sub-network may be extended to more than one digit.

2.4 LAN:

The default data rate for the timer and the console is 16 Mbps.

A dedicated LAN is recommended. A congested LAN isn't.

If the timers are connected through a router, it must employ bridging functions. Routing functions are not supported.

2.5 Initialization / Offset changes:

During normal operations, a planned offset change (DST + Leap seconds) will be applied on date, even without Console Application connexion. In case of power outage at this very moment, offsets will be applied after the next IML following this planned date. On 967X refer to the TIP H015665 to update manually the S.E. clock not automatically updated for DST modifications (no UTC change) if necessary.

If a DST offset change is not seen by MVS put the timers maintenance mode ON then OFF to force MVS to analyze the offsets. This problem is corrected by the following MCL: Driver 12 stream F11130 MCL003 - Driver A2 stream F10985 MCL006.-

Here are the next DST dates:

• 28.03.1999: start of daylight saving time
• 31.10.1999: end of daylight saving time
• 26.03.2000: start of daylight saving time
• 29.10.2000: end of daylight saving time
• 25.03.2001: start of daylight saving time
• 28.10.2001: end of daylight saving time

2.6 Distant Sites Installation Sequence (E.A. Configuration):

Until it is clearly stated in the Installation Sequence checklist it may be profitable to install distant sites with the following sequence if time constraints are to be considered or 2 C.E.s not available:

Follow the Installation instructions modifying the sequence as follows:

1. On one site ( the secondary site ):
   • Install one timer (the remote one).
   • Install the standby console if any.
   • When you have performed the CLO Ports Tests (Wrap tests) you could run the CPC ports tests now to avoid to come back on this site:
   • Ensure the unit is able to transmit ( "Offline" state ): you need to define the console as "Active" then initialize the network in Expanded Basic configuration.
   • Run the CPC Ports Tests (Wrap tests) - Refer to the Installation Sequence number 4 Task 5 (test CPC ports).
   • Re-initialize in Expanded Availability configuration, then define this console as "Standby".
   • The timer will stay 'Dormant' (RC 77 + 78 + 91) until the other timer is installed provided you connect the CLO fibers.
2. On the other site ( the main site ):
   • Install one timer (the local one).
   • Install the active console.
   • Initialize the Network, Verify Operations and Load LIC.
   • Test CPC ports. At this point it is possible to enable the ports from the active console and ask a CE on the remote site to perform the wrap tests if not already done as per above. If, for any reason, the wrap tests are initiated from the remote site on the standby console (Ports enabling) the Timer must be put in maintenance mode so that the Port Control (On/Off) function is available. You may also choose to redefine the local console as standby and the remote console as active, refer to "Consoles switching". Refer to the "Console Functions Available in Maintenance Mode" table in the Chapter 2 of the Maintenance Information book (page 2.167).
   • Installation Cleanup.
   • Post Installation (Final definitions, initialization and connexion with the customer).

2.7 Upgrade an Expanded Basic Configuration to Expanded Availability:

2.8 Distant Sites Upgrade Sequence (E.B. to E.A. Configuration):

If the new timer installed in the remote site is NOT to be LAN attached:

First install it on the main site near the existing E.B. one to have it LAN attached in order to perform the upgrade procedure with the console.

When the new timer is inserted in the existing ETR network and just before connecting the CPC fibers you'll need to power off the new OFFLINE timer, disconnect the CLO cables (do not disconnect them before powering off), and move it to its destination site.

Note that moving the timer after the CPC fibers attachement would require a "SCP connectivity" verification in order to ensure that two OPERATIONAL/ENABLED ports are available for each SCP image and no disruption will occur to the customer operations. Follow the "Service Impact Analysis" procedure as if you were performing a non hot-plug part replacement.

2.9 Extended distance (2029/9729):

Note that 2029 Sysplex Timer links cannot be configured as "high availability" channel ( not supported ), configure them as a "base channel". Sysplex Timer links are not supported in hubbed ring configurations.

RPQ 8P1955 (CF/Sysplex Timer Link extension with 9729 /2029): This RPQ is a pricing vehicle that addresses deviations from the allowed link distance, link budget or other physical layer specification for the Sysplex Timer and Coupling Facility Links when used in conjunction with the IBM 9729 Wavelength Division Multiplexor or 2029 (DWDM). With approval of this RPQ, distances can be increased to 40 km for both timer-to-timer and timer-to-CPC connections. It is extremely important to note that the timer-to-timer CLO link distance cannot exceed 40 km without obtaining explicit IBM approval. Extending the CLO link distance beyond 40 km could result in the two timers being out of synchronization with each other, thereby resulting in a potential data integrity exposure. The out-of-synchronization condition is not detectable during normal system operation. Support of this RPQ may require modifications of the standard 9729 product; these modifications may include, but are not limited to, hardware changes such as removal of pluggable optical attenuator and custom card rework on the 9729 laser and receiver cards. Approval is based on modelling and experimental measurements made in the S/390 Poughkeepsie Hardware Development Laboratory.

9729: The ETR support is provided with the ETR/ESCON card PN 85H7277.

Pls read the chapter 3 named "IBM 9729 in a Large System Environment" in the 9729 redbook SG242138 below):

"9729 Redbook"

2.10 Timer Network display:

The Unit Operational State ("Dormant"-"Starting"-"Offline"-"Online"- "Service") is described in the appendix B of the Maintenance Information manual or in the Online Help (PF1).

2.11 Console configuration:

Initial passwords are ADMIN, MAINT, OPERATOR. If a password is deleted for a level, it is automatically deleted for subsequent levels. The highest authority password (Administrative pw) can be used for any level.

2.12 Consoles switching:

When you switch the local active as standby you have a warning that the link will be momentarily dropped. Then if you try to get the sysplex network console window the Link status indicator stays red with a message "Duplicate console" in the Console type indicator (bottom line) This is a normal situation as 2 consoles of the same type cannot co-exist. (Active or Standby). When, from the remote site, you switch the previously standby as active there is an automatic recovery on the local console which becomes standby without any other intervention. Some 2/20 seconds may be necessary depending on the installation (public busy LAN, HMC, etc.).

2.13 DCAF control (Remote Access Console):

A DCAF tcp/ip problem exists when using a 6887KAG HMC as a controler to a 6562 HMC target. This is due to the MATROX video driver v2.14 of the 6562. Fix: V2.21 driver which will be in the next HMC driver CD ROM (DR14 or above when available).

Important points:

• The PC 330 M/T 6576 cannot act as a DCAF TARGET.
• When accessing the C.A. from a remote console it must be set in window mode. If you do run in window mode: when minimizing the 9037 Console Application, click the right mouse button on the desk-top then select "Refresh now" to restore the screen. Refer to the Service and Customer Exception Letters.
• To DCAF into the HMC, the remote accessing console must have the same DCAF level. Verify on DCAF TARGET+HELP+Product Info if the session is started, or "SYSLEVEL" from an OS/2 window. Refer to the TIP H136366 for details about the HMC levels.
• For the console "auto discovery" HMC function refer to the chapter 7 "Undefined Timer/Director Consoles Icone (Groups view)" item.

2.14 Adding a Port card:

2.15 TIPS (EXTRACT of 9037 model 2 related tips):

TDR H161444 "ALLOW 9037-2 (ETR) MIGRATION ON 9021 711 BASE MODELS". 9021 patches required for the above migration (G status on 10/15/1997)

TDR H094436 "RESERVED FOR SYSPLEX TIMER CEC PATCHES". 3090/9021/9121 hardware patches and MVS APAR/PTFs

TDR H163842 "9037 MODEL 1 CONSOLE YEAR 2000 SUPPORT" (Jan 1st 2000 is a saturday - March 1st 2000 is a wednesday due to 02/29)

TDR H107416 "ADDITIONAL BOC/TOD INFORMATION". Guidelines for time changes in a Sysplex environment.

TDR H015665 "PROCEDURE TO SET CPC TOD ON 9672'S OR 2003"

TDR H126950 "9672 9673 9674 BOC TOD CLOCK INFO"

TDR H13617 "SETTING THE DATE AND TIME IN MVS"

TDR H057867 "TELEPHONE DATE CODE PROVIDERS URL + PHONE (MODEM2 PROTOCOL"

TDR H022920 "967X MVS/ETR REFCODE DEFINITIONS (753A01XX)"

TDR H131142 "967X MVS/ETR REFCODE DEFINITIONS (753A01XX)"

TDR H123826 "IEA261I DURING MVS IPL BUT ETR PORTS OPERATIONAL".

TDR H012151 "967X CALLS HOME WITH 753A0101"

TDR H126171 "MVS MESSAGE IEA262I ON IPL / SE REFCODE 753A0102"

TDR H134542 "RC 753A0100 ON 967X PROCESSORS"

TDR H00892 "IEA272I RC 042 MVS MESSAGE INFORMATION"

TDR H16445 "MVS MESSAGE IEA272I RC40 GENERATED ON 9037 CONSOLE SHUTDOWN"

TDR H131232 (Potential Sysplex down sit. with 9037-001) "9037 MODEL 001 RC040 COMMUNICATIONS LOSS WITH CONSOLE"

TDR H045772 "ETR PORT NOT OPER: MVS IEA262I MSG+NO CALL HOME RC 753A0102"

TDR H057928 "RESTORE TIMER AND ESCD CONSOLE FILES AFTER UPGRADE TO DRV 22"

TDR H166191 "TRAP 000D AFTER 02.00.00 CODE ECA009 INSTALL"

TDR H123826 "IEA261I DURING MVS IPL BUT ETR PORTS OPERATIONAL"

TDR H057587 "9032/9037 C.A. HMC BACKUP FILE OVERWRITTEN - NOT RESTORED".

TDR H057553 "HMC UPGRADE DR98 TO DRA2 FAILURE".

TDR H123632 "9037-002 ALL PORTS FAIL RC32. WRAP TESTS FAIL ALL PORT"

TDR H057415 "9037002 RC 077/093/006 DURING AN INIT WITH A NEW NETWORK ID"

H057418 "WSAASYNCSELECT() FUNCTION ERR. USING THE 9037002 C.A. on HMC"

H057419 "9037002 C.A. FRENCH KEYBOARD CUSTOMIZATION ON HMC AT D92W"

TDR H057441 "77320118 / 77320100 RESULTING IN A 100% MCM REP. CALL HOME"

TDR H16656 "AFTER HMC CODE UPGRADE 9032 AND/OR 9037 CONSOLE MISSING"

TDR H04112 "HMC GETS ANYNET ERROR : ANY0352I WHILE LOADING NWWAIT-SECS"

TDR H161369 "REPLACING A MOD 002 CTP CARD, POSSIBLE LICC VERSION CHANGE" (RC 014)

TDR H055797 "9037 MODEL 002 USING EXTERNAL TIME SOURCE (ETS) FROM HOPF"

TDR H054682 "9037 MODEL 002 USING EXTERNAL TIME SOURCE (ETS) FROM HOPF"

TDR H135103 "9037 MODEL 002 FEATURE CODE 4910"

TDR H107214 "9037 USA TIME PHONE NUMBERS"(VOICE-MODEM)."

TDR H136366 "DCAF LEVEL ON THE HMC CAN BE DISPLAYED"

TDR H132707 "SEAMLESS APPLICATIONS HANG UNDER OS/2 2.11"

TDR H131749 "NEW MAU'S(8228) MUST BE INITIALIZED BEFORE USE"

TDR H135769 "PC 330 MODEL 6576 DCAF RESTRICTION FOR 9032 / 9033"

TDR H135763 "WARNING - TWO UNIQUE RESTORE TAPES FOR 9037"

TDR H134179 "WORDING IN 9037 MODEL 1 MIM CAN CAUSE CUSTOMER OUTAGE".

TDR H057790 "INSTALL LATEST MICROCODE.. VERSION 02.00.00.(MODEL 002 ONLY)"

TDR H057794 "REPLACE CURRENT POWER SUPPLY IN SELECTED 9037 MODEL 002."

TDR H161516 "CRITICAL EC UPDATES LIC TO VERSION 01.02.01 (MODEL 2 ONLY)". See ECA008 in the EC chapter.

TDR H057483 "9037-002 TIMER LIC VERSION 01.02.01 IMMEDIATE AVAILABILITY"

TDR H161550 "CORRECT A POTENTIAL PROBLEM WITH CTP CARD 9037 MOD 2 ONLY". See ECA007 in the EC chapter.

TDR H162807 "UPDATE CONSOLE/SYSTEM LIC TO VERSION 01.02.00 (MODEL 2 ONLY)". This TIP provides the console application improvements.

TDR H16482 "9037 MODEL 2 INTERIM FIX LIC" (LIC 01.01.04)" (RC044 + shutdown // Two bridges connexions // RC032 all ports + POSTS RC089 problems).

TDR H135763 "INSTALLATION NOTES FOR ECA004 9037 MODEL 2"

TDR H136497 "ECA004 - GA LEVEL CLO CARD, LIC & DOCUMENTATION"

TDR H136779 "INSTALLATION NOTES FOR ECA005 9037 MODEL 2"

TDR H136498 "ECA005 - ECE46347 REPLACE LOGIC BOARD" (see ECA004).

TDR H132964 "MISSING STEP FOR INSTALL/REMOVE 9037 FEATURE PORTS" (MODEL 1 only).

Chapter 3. Maintenance Information

3.1 Maintenance guidelines

The current MIM version (SY27-2641-02) is completed by the Service Exception Letter - T200SR01 October 30th, 1998 - A copy can be obtained from the server with the following VM command: REQUEST T200SR01 PACKAGE FROM FOCUSPE AT PKEDVM9

The objective of the MIM is to maintain a continuous availability, with NO outage if a unit power off is necessary (E.A. configuration) during an intervention:

1. A "Service Impact Analysis procedure" must be performed before a 9037-002 is taken out of service to asset the impact on the Sysplex and avoid operations outage.
2. A procedure to "Prepare the 9037-002 For Service" must be performed to isolate one timer from the sysplex configuration when it is necessary to service it.

The console is required to:

1. Install the timer.
2. Define or change the ETR network definitions (active/standby console - Modify the IP address to communicate with the other LAN attached box if necessary).
3. Manage and monitor the ETR networks ( Two networks maximum).
4. Display/set/adjust the time and date.
5. Access the logs.

1. Configure Time Offsets.
2. Initialize Timer Network.
3. Adjust Time (calculate or adjust).

More recommendations:

1. Do not close the Console Application, when not needed minimize the window.
2. Do not leave the Maintenance mode set, critical error reporting could be lost, i.e. not reported to attached CPCs.
3. The console System Error Indicator should be reset as soon as a System Error event is analyzed/repaired to avoid masking the next one. Refer to the Event Codes in appendix A of the Maintenance Information manual to identify such System Error Light events (SEL). This is not necessarily linked to the severity (2 and above).
4. Use an ESD wrist strap to remove FRUs.
5. The handle must only be used to replace the logic board.

3.2 Problem determination - "Maintenance Information" documentation:

The MVS Msg IEA100I is generated by informational only reason codes. Any reason code of a higher severity, when reported to the CPC, is logged with the MVS Msg IEA272I and results in a service call. So does an event detected by MVS, e.g. a non-operational port (IEA262I). Every 967X call home is performed under RC 753A010* (See Tip H131142). On 967X, the "A1_record" S.E. log entry reports the reason code when available to MVS along with the MVS message.

Diagnostic Information: Chapter 2 (MAPS and Utilities).

"Service Impact Analysis procedure" and "Prepare the 9037-002 For Service" : Chapter 3.

Offline/Online verification procedure used after the replacement of a FRU: Chapter 3 (Repair Information).

Operator Panel - Console Messages (Unit status line messages, Error messages, LIC install messages): Appendix B.

MVS/ESA messages: Appendix C.

TPF messages: Appendix D.

3.3 Support request:

1. Use the Data Collection Procedure, MIM page 2-187, send the COLLECT.* files to the support VM below, and record it in the PMH. If possible, perform it on both consoles. When the first console COLLECT.* files are saved, rename them to avoid overwritting.
2. Before uploading both set of files to VM in binary, rename them with a file name related to the pmh number and type of console, then send them to "MOPLOGS AT MOPVMA". (e.g. 06060ACT.TRM - and 06060SBY.TRM).

3.4 Timer LIC and Console Application levels:

The levels are identified as "XX.YY.ZZ" for "Version.Release.Patch".

To check both levels:

• Console Application level: Click on the "Help" menu then "Product Information" for the Console Application code level: 02.00.00 for the current field EC level
• Timer LIC level: Either read it from the Operator Panel or click on the File menu then on Open Timer Network Window then select one timer network if two are in control and click on Timer Network Info: the LIC Version of the box(es) is displayed.

  
  For safe operations every timer must be at least at the LIC level 01.02.01 (the critical EC level, available since feb 98)
   
   

3.5 Console Restore tapes - optical disk -diskettes P/Ns:

Level (01.03.00) and 0S/2 Warp 4.0 (6589 PC):

• Optical Disk "Base Console restore version 01.03.00" P/N 07H5977
• Diskette "Base Console Restore" P/N 07H5978
• Console restore tapes p/ns are 07H6005 (d/t 6571) and 07H6006 (d/t 6576)

• Diskette System Version 07H6008
• Diskette Console version 07H6009

3.6 Timer LIC install:

LIC Install Messages are explained in the appendix page B-9.

Follow the appropriate procedure to install a new version of the LIC on the timers: "Installing LIC on a Timer Network" in the Maintenance Information (An existing version can't be re-installed).

Do not perform a code download in a Basic or EB configuration while the timer is ONLINE: this would cause a Sysplex down situation. Instead, plan for a system maintenance slot (timer network outage) and follow the "Service Impact Analysis" procedure (MIM).

The LIC install is concurrent with customer operations in E.A. configuration provided some requirements are met: Follow the procedure in the Maintenance Information Manual to insure that a LIC install will not cause a customer outage in an E.A. configuration. During the sequenced timer restart following the code load, it may NOT BE CONCURRENT in some situations:

• If one port is not available for a MVS image (NON OPERATIONAL or DISABLED).
• For ES/9021 if only one port is connected on a H2 (drop to local).

Do not rely on the Port State indicator (An arrow is up when the port is transmitting normally). This is not a guarantie of the receiver status. Refer to the Service Impact Procedure and verify with the MVS command "D ETR, DATA" that both ports are OPERATIONAL and ENABLED for each MVS image before the LIC install is attempted. The route command hp2."RO *ALL,D ETR,DATA" can be issued to see all partitions ports in a row. On MPs only one side status can be displayed ( 0 or 1, not both).

Of course, during the Timer code load operation the boxes are restarted (stop transmitting to the CPCs) one after the other to avoid a disruption:

• The LIC is downloaded from the console to the 'inactive' RAM partition of the local unit, then the remote.
• In the remote unit, the boostrap loader pointer is set to the new code partition which becomes the 'active' partition, then a restart occurs. (The unit is IMLed automatically with the new code, stopping ETR data Xmission to CECs momentarily - "Restarting" message displayed at the console)
• This unit will then send a 'Restart after delay' command to the local unit.
• The local unit sits idle for 5 minutes then, unless a failure occurs in the remote, activates its own code (a restart occurs).
• This means that if a failure occurs, one unit will be kept Xmitting to ensure no disruption occurs to the sysplex.

3.7 MVS Commands:

Enter "D ETR, DATA" at the SCP console to display:

IEA282I 03.50.05 ETR STATUS
 

SYNCHRONISATION MODE = ETR       CPC SIDE = 0
CPC PORT 0 <=== ACTIVE            CPC PORT 1
 

OPERATIONAL                          OPERATIONAL
ENABLED                                    ENABLED
ETR NET ID = 10                         ETR NET ID = 10
ETR PORT = 02                           ETR PORT = 02
ETR ID = 02                                 ETR ID = 03
 
 

Verify the results with the Service Impact Procedure, the last line displayed is the Unit Address of each box. (Details of MVS msg IEA282I in Appendix C of the Maintenance Information manual).

Another useful command, to put ENABLED a port DISABLED by MVS (After analyze and repair to avoid a further re-occurence of this DISABLED -not stepping- state):

Enter "SETETR PORT=n" at the SCP console (n=0 or 1).

3.8 WAIT CODES 0A2:

3.9 Token ring failure:

Console >---tkrg---> timer "A" >---CLO Links A/B---> timer "B"

Follow the related MAPs and TIPS. Token ring errors must be investigated before the occurence of a RC 138 "Token ring permanently disabled" that halts the recursive recovery and disables tkrg communications. This situation is recoverable only with an IML of the unit (DISRUPTIVE for non E.A.). To perform an IML in a non-disruptive way (E.A. config. only), follow the maps to perform a dummy CTP replacement. ('Service Impact Analysis' and 'Preparing the Sysplex Timer for service' procedures). If the other box can be accessed from the console, restart the failing unit using the 'utility' menu (CLO links communications must be available). If not, force the IML from the box: power off/on the failing unit. Then continue with the MIM procedure to restore the configuration.

The CTP card replacement should be attempted in the last, as it is disruptive except for EA configurations: If possible create a 'private' LAN with only a MAU, the console and the timer in the configuration, this will prevent interferences with other equipments existing on a customer 'public' LAN and should facilitate the investigations.

Isolating parts from the current LAN will show up if the problem is coming from the console or timer or from another component on the current LAN.

• In order to isolate the probable defective external component (vs timer) setup a specific lan (console-MAU-timer alone) and ping continuously the timer IP address from its console (from an OS/2 reduced window) Put the timer in maintenance mode to avoid MVS msgs and calls home. You may observe both ping results and 9037 Console Application link status by opening the 9037 C.A. in a window and reducing its size so that you do not need to switch between windows.
• When performing investigations ensure for each action:
  1. that the link is reset (double click on the link status indicator then on the reset link push button) and some seconds elapsed (30s - 1 min);
  2. that the green led (back of the console) of the tkrg card is on; (if not a shutdown/reeboot is necessary).
  3. that the last action performed didn't cause the timer to generate an incident (RC on the operator panel). (e.g. a RC138 indicates that the retry capability is exceeded and the LAN communication disabled, requiring a power off/on following the Service Impact Analysis procedure to ensure that no disruption occurs to the customer operations if in E.A. mode (disruptive if basic or E.B., and E.A. for images with only one ETR port available).
• If it still fails ensure that no external application is running on the timer console that could impact the LAN communication. Then replace parts (MAU, Console card, cables, CTP card in the last). If no failure, the previous LAN configuration is defective (MAU, cables, IP or MAC addresses, etc.)
• Before dropping the power for any reason follow the Service Impact Analysis proc to ensure that there will be no impact to the customer operations. If you plan to power off/on a number of times proceed in the MIM as if you were going to replace a CTP card (i.e. disconnect all fibers - CLOs and CPCs - AFTER power off).
• Reset the timer maintenance mode as soon as debug ops are finished.

3.10 Loss of communication -RC 040/041- with the Active (Standby) console:

A loss of communication with the console(s) has no impact on the timer operations.

Do not close the Console Application, when not needed minimize the window. From LIC 01.03.00 a dialog box appears to warn the user not to close the application.

A RC 40(41) will appear in the Event log only after reconnexion.

Possible causes of RC 40 or 41 when they are recorded alone:

• See TIP H022920 if MVS IEA272I RC40 AFTER 9037-2 ECA009 (the backup operation hanging the console is under investigation).
• A Network definition modification in order to modify the name of the network. A record "Timer renamed" will be registered in the Console Audit Log that you can relate to the RC timestamp (if the P.C. has the same time as the timers, refer to the "PC Time" column).
• A Network definition modification in order to modify the IP address in order to communicate with the other box (or same if modified).
• A Network definition modification in order to switch from active to backup console.
• Closing the Console Application. A record "Console terminated" will be registered in the Console Audit Log. When the C.A. is not needed do not close it, minimize the window.
• Shuting down the Operating System of the PC
• Powering off the PC.
• A failure in the PC or in the application.
• A LAN failure.

Chapter 4. E.C.s (purpose and contents):

4.1 LIC HISTORY:

4.2 ECA009 - Version 02.00.00 information:

4.3 ECA010 - Selective PSA replacement:

4.4 Selected CTP cards replacement:

The EC E46348 package is released (ECA007):

• Installation package

To avoid a RC 014 due to different LIC levels in both boxes read the appropriate TIP (ref TIP list).

Chapter 5. External Time Source

5.1 Sysplex Timer accuracy and E.T.S.:

The Sysplex Timer is able to steer its oscillator to maintain an accuracy of 100 us vs an ETS (5ms for the model 1) if a CLO card is installed (E.B. or E.A. configuration), with the 1 PPS option. Without, the accuracy is maintained to within 0.001s of the ETS.

The ACTS (Boulder, CO) is accurate to within 0.01s to U.T.C. with a propagation delay through telephone lines less than 0.1s for the U.S.

The european Telephone Date Code support is provided with the LIC 02.00.00, allowing an alternate modem source.

The Radio Receivers and Time Code Generators are typically within 0.1 to 0.5 S of UTC.

A GPS Satellite is typically within 100 nanoseconds of UTC.

5.2 External Time Source tests:

5.3 Basic Configuration:

Then, the ETS is only accessed for information, no steering is possible. Time difference can be seen through the ETS results dialog box.

5.4 Time tracking:

For the Expanded Basic or Expanded Availability configurations the Automatic Adjustment option must be set (ETS definition dialog box). If not set, the ETS will only be accessed for the Calculate option (Adjust Time dialog box) or initialization.

To guarantee tracking two E.T.S. must be used, one of which can be designated as the prefered one.

How to display the difference between the Timer time and the ETS time ?

• Double click on the ETS port.
• From the menu: "Time" then "External Time Source results".
• From the menu: "Time" then "Adjust Time" then "Calculate".(without OK)

5.5 Leap seconds:

No positive leap second will be introduced at the end of December 2000

For announcements of next leap second occurence consult the "Bulletin C" of the International Earth Rotation Service (IERS) below:

"Observatoire de Paris - IERS link"

5.6 Modem Access - US and EMEA telephone numbers:

• Modem1 protocol (A.C.T.S - U.S.A.): National Institute of Standards and Technology: "Automated Computer Time Service" 1- 30 34 94 47 74
• Modem2 protocol (Telephone Date Code - E.M.E.A.): TIP H057867
  1. Domestic only:
     • >> Italy: Istituto Elettrotecnico Nazionale Galileo Ferraris "IEN Telephone Date Code" : 01 66 11 46 15
     • >> Sweden: Precision Timing "Computer Time Service" 09 00 100 2000 07 19 31 27 19
     • >> United Kingdom: National Physical Laboratory "NPL Truetime" : 08 91 51 63 33
  2. International access:
     • >> Austria: Technische Universitaet Graz "Telephone Time Service" (43) - 3 16 47 23 66
     • >> Belgium: Royal Observatory of Belgium "Computer Time Code Service" (32) - 23 73 03 20
     • >> Germany: Physikalisch Technische Bundesanstalt "Telephone Time Service" (49) - 5 31 51 20 38
     • >> Netherlands: Nederlands Meetinstituut "Time Service" (31) - 1 52 61 71 81
     • >> Switzerland: "Time Service" (41) - 3 13 23 32 25

5.7 ETS connector pin numbering:

• 1/ free
• 2/ TxD (transmit data RS232c)
• 3/ RxD (receive data RS232c)
• 4/ RTS (ready to send RS232c)
• 5/ CTS (clear to send RS232c)
• 6/ free
• 7/ 0V GND
• 8-10/ free
• 11/ TxD (transmit data RS422)
• 12/ /TxD (transmit data RS422)
• 13-21/ free
• 22/ RxD (receive data RS422)
• 23/ /RxD (receive data RS422)
• 24-25/ free

URLs of manufacturers are in the M.I.M. page F-8

5.8 GPS/Low freq receiver Protocol 1 documentation:

As this protocol is common to both sysplex timer models you can also find the protocol 1 description under the 9037 model 1 chapter on the focuspe web site:

"ETS Protocol 1 Information link"

For protocols 2 and 3 use the MIM as referenced above.

Chapter 6. Sysplex disaster recovery procedure

6.1 9037 model 2 disaster scenario:

Internal errors are usually detected (e.g. hardware error or power loss) and an OLS sent through the CLO link to the other unit. In this situation the alive timer will stay online (emitting) without disruption, either because it is already acting as primary or switched to primary on receipt of the OLS.

A sum of incidents or a major problem may led to both units not emitting when the OLS is not received by a secondary unit.

6.2 9037 model 2 disaster prevention:

The Planning Guide "Recommendations for Continuous Availability" chapter must be followed for the installation of the S.T. network.

To avoid the accumulation of incidents, do not leave unattended a CLO Reason Code indicating that a single link failure occured (RC "77" or "78" alone). In this situation, from the version 01.02.00, a message will be issued to the host every 8 hours to warn the operator.

When any intervention is required, follow the M.I.M. procedures to avoid and create such a scenario. (e.g. if CLO cables are disconnected before powering off one box, the secondary will go dormant - not emitting - and you may then power off the primary - emitting - box and cause the disaster.) Following the M.I.M. procedures "Service Impact Analysis" then "Preparing the Timer for Service" will guide you to avoid this. Ensure you have the last version of the M.I.M./Service Exception Letter.

Have a backup tape of the latest configuration.

Check from the following procedure which item would cause a delay and take the proper actions before the 'panic'.

6.3 9037 model 2 disaster recovery:

The following manual recovery list ONLY APPLIES TO ONE SPECIFIC SITUATION when, in an E.A. configuration, one box is definitely out of order and time is needed for a repair action. The other box is alive but in the status 'DORMANT' that keeps it from emitting on the CPC ports. In this specific situation it will not be possible to recover in E.A. (Expanded Availability). The purpose of this manual recovery is to recover the Sysplex down situation asap, using a restart of the S.T. network in an Expanded Basic (E.B.) configuration until the other box is repaired.

Recovery sequence:

• Isolate the defective unit, disconnect CPC and CLO fibers.
• Recover the Console to Timer link.
  1. You recorded the timer IP adresses and offsets previously.
  2. Use the current active console to link the alive timer box -or reconfigure the standby console as active- Modify the Network definition (IP Address) if necessary to access a remote linkable box (same LAN or existing bridge).
  3. You may have to install the Console Application on a HMC (console crash).
  4. You may have to connect the boxes to the HMC LAN and modify the IP addresses of the timer units to match the new subnet.
• Set a Log Marker.
• Check the status of the alive unit (DORMANT) and CPC ports. If the status is different exit from this procedure (other scenario).
• Define the E.T.S. configuration if necessary. (This could be delayed if no E.T.S. device already available).
• Configure the unit in E.B. configuration using the "Recall" push button to get the last saved init. parameters, then modify the configuration to Expanded Basic and use the "Modify by delta" radio button with 00:00:00 as delta value if you do not need to modify the time. If the configuration was not previously saved fill every field, even unused offsets (use 00:00 for T.Z. and D.S.T., 0 for Leap seconds in this case). Initialize in E.B. configuration.
• Ensure of no pending abnormal RCs, check the CPC port status then clear the S.E.L.
• Once re-initialized, if the CPC links are useable, it is possible to restart the sysplex and plan for the other box repair.
• Save the actual configuration.
• Set a Log Marker.
• If details are needed about the following sequence, you may refer to the "Upgrade an E.B. configuration to E.A." page 6.110 of the M.I.M. (6-93 in the latest version SY27-2641-02).
• When the other box is repaired and available, it can be inserted in the configuration in the following non disruptive way:
  1. Do not connect the CLOs cables for now.
  2. You need to be aware of the following steps before performing the "Offline Verification Testing" (MIM) to restore the ETR network.
     During this Offline Verification procedure, after Power On, first ensure the new unit is configured for Expanded Availability: link it with an active console, verify the configuration from the Timer Network Info panel. If Expanded Basic is displayed, initialize in E.A. Verify the unit is now in the "DORMANT" status when in E.A. CLO ports tests are part of the above procedure (MIM).
  3. Another point: after the CLO fibers are connected, the configuration will be automatically updated to Expanded Availability with the new timer insertion if its LIC version was the correct one (no RC014 - see below). Be patient and wait for the console to retrieve/display the new configuration. If you have any concern about this timer insertion and transition of the ETR network from the E.B. configuration to the E.A. one, perform the following step before connecting the CPC fibers Re-establish the Console link with the existing old "E.B." unit. Verify the repaired unit switched to the status "OFFLINE", the Timer Network Info panel displays Expanded Availability and no pending abnormal RCs on both units. If you get the RC014 when connecting one CLO cable the new timer will go to "Service" mode (the existing one stays ONLINE of course...) Follow the MIM or the TIP to load the same LIC level as the ONLINE timer. If, at that time, you get a warning msg ""Unit ? is processing a restart" disconnect the CLO cable(s), close the Console Application then reopen it. You will now be able to proceed with the LIC install on the timer in "Service".
  4. When the Offline Testing procedure is completed you just verified the SCP connectivity for 2 OPERATIONAL/ENABLED ports per image. You must now Restart the existing old E.B. timer - i.e. the one that was alive for the recovery - to fully insert it in the E.A. configuration and avoid further MVS problems. This step may be disruptive to the sysplex if the SCP connectivity for 2 OPERATIONAL/ENABLED ports per image was not just verified carefully. Check the status at the end of the restart. (CPC and CLO ports). Verify again the SCP connectivity for 2 OPERATIONAL/ENABLED ports per image.
  5. Restore the previously dedicated LAN if necessary.
  6. Ensure of no pending abnormal RCs, then clear the S.E.L.
  7. Deselect "Maintenance" mode for both timers.
  8. Save the actual configuration.
  9. Set a Log Marker.

Chapter 7. Hardware Management Console Support for the 9037 Model 2 Sysplex Timer

7.1 Feature / Documentation

Documentation PN 620-900012-000

Documentation sent with the FC4910:

• ESCD Console support for the 9037 Model 2 Sysplex Timer "Installation and maintenance Information"
• HMC Console support for the 9037 Model 2 Sysplex Timer "Installation and maintenance Information"
• Maintenance Information for the 9037 Model 2 Sysplex Timer
• Using the 9037 Model 2 Sysplex Timer.
• Users's Guide Reference Summary for the 9037 Model 2 Sysplex Timer.

7.2 Installation Pre-reqs.

Installation Time: 30 min

Pre-req PC:

• Pentium - 90 Mhz
• Memory 24 MB
• Code EC mini level: E45552 MCL 015 activated (Driver 66).
• Backup ROC available.

The timer can communicate with two consoles. If you are installing the Console Application on a HMC for testing purposes while 2 dedicated PCs are already installed, as active and backup consoles, shutdown one of these ones before starting the Console Application.

7.3 Installation Summary

• Setup the Console and Timers with same TCP/IP subnet addresses.
• Install the Console Application on the HMC, in the proper directory: use the TCCONVRG.CMD. Do not use INSTALL.CMD which is only for dedicated consoles.
• Verify the operations of both applications (Sysplex Timer and HMC).
• Installation Completion Report.
• Backup Critical Data.

7.4 Console Application backup:

7.5 Keyboard customization:

Refer to TDR H057419, you can get the file by entering the following command from any VM system command line: TOOLS SENDTO MOPVMA CETOOLS RSCP GET KBDFR DLLBIN Copy the missing file in C:\OS2\MDOS\WINOS2\SYSTEM\

7.6 "Undefined Timer/Director Consoles" Icone (Group view):

For appropriate procedures refer to the "S/390 Console Installation and User's Guide - 9032 Models 3 and 5 Escon Director - 9033 Model 4 Escon Director - 9037 Model 2 Sysplex Timer" - Ref GA22-7291 - shipped with the Unit(s) to install and use the PC-330, the PC 365,the HMC, the ESCD consoles) etc. (delivered from 01.03.00 LIC level).

Operations flow:

• Set the IP addresses to have a common subnet address.
• Set the 9037-002 Console Application session in Win-OS/2 Window mode.
• Start the DCAF TCP/IP Session on the target PC.
• On the HMC wait 2 minutes till the "Console" icone underlined with the TCP/IP address appears in the "Undefined Director/Timer" console work area.
• Drag this icone to the "Add object" one to get it under the "Defined Director/Timer Console" menu.
• From the "Defined Director/Timer Console" area drag the console icone to the "Single object" one to established the session (NO password to be given).

7.7 Miscellaneous

If the HMC hard drive was rebuild the programs associated with the "Extra Applications Folder" are restored but the folder isn't. Follow directions to recreate this folder.

When installing the HMC do not configure TCP/IP outside of the HWMCA to avoid interferences between both applications.

When installing the Sysplex Timer Console Application do not place its icon in the startup folder. (The HWMCA starts TCP/IP during its initialization).

Chapter 8. ESCON Director Console Support for the 9037 Model 2 Sysplex Timer

8.1 Feature / Documentation

Documentation PN 620-900013-000

Documentation sent with the FC4910:

• ESCD Console support for the 9037 Model 2 Sysplex Timer "Installation and maintenance Information"
• HMC Console support for the 9037 Model 2 Sysplex Timer "Installation and maintenance Information"
• Maintenance Information for the 9037 Model 2 Sysplex Timer
• Using the 9037 Model 2 Sysplex Timer.
• Users's Guide Reference Summary for the 9037 Model 2 Sysplex Timer.

8.2 Installation Pre-reqs (minimum):

The ESCON Director Console code level must be at 2.1 or higher.

Familiarize with the documentation.

Installation Time: 20 min

Pre-req PC:

• Processor 486DX2 - 66Mhz
• Memory 16 MB
• Hard disk 250 MB
• Backup tape 250 MB mini.
• Token Ring Adapter 4/16 Mbps

Do not install the Console Application on a Director Console controlling 16 Directors. The console is able to communicate with up to 14 directors and two timers networks.

A remote LAN containing a Director Console can connect to the Timers LAN through a bridge.

The timer can communicate with two consoles. If you are installing the Console Application on a ESCD Console for testing purposes while 2 dedicated PCs are already installed, as active and backup consoles, shutdown one of these ones before starting the Console Application.

8.3 Installation Summary

• Setup the Console and Timers with same TCP/IP subnet addresses.
• Install the Console Application on the Director Console.
• Verify the operations of both applications (Sysplex Timer and Director Console).
• Backup Critical Data.

8.4 Miscellaneous