Source
RETAIN tip: H202210
Symptom
- Unexpected Link Negotiation Errors when booting from Storage Area Network (SAN). The blade may fail to boot or the fiber connection may negotiate to a speed less than 8 Gigabits.
- Excessive Cyclical Redundancy Check (CRC) or Encode errors in switch logs are seen during run-time. This may cause link drops, failover in Multipath environment, or poor performance.
See RETAIN tip H202550 for help with G-Port login symptoms at the following URL:
Affected configurations
The system may be any of the following IBM servers:
- BladeCenter H, type 7989, any model
- BladeCenter H, type 8852, any model
The system is configured with one or more of the following IBM Options:
- Brocade 10-Port 8 Gigabit SAN Switch Module for IBM BladeCenter, Option part number 44X1921, replacement part number 44X1927
- Brocade 20-Port 8 Gigabit SAN Switch Module for IBM BladeCenter, Option part number 44X1920, replacement part number 44X1926
- Emulex 8 Gigabit Fibre Channel Expansion Card (CIOv) for IBM BladeCenter, Option part number 46M6140, replacement part number 46M6138
- QLogic 20-Port 8 Gigabit SAN Switch Module for IBM BladeCenter, Option part number 44X1905, replacement part number 44X1914
- QLogic 8 Gigabit Fibre Channel Expansion Card (CIOv) for IBM BladeCenter, Option part number 44X1945, replacement part number 44X1948
- QLogic 8 Gigabit Intelligent Pass-thru Module for IBM BladeCenter, Option part number 44X1907, replacement part number 44X1916
- QLogic Ethernet and 8 Gigabit Fibre Channel Expansion Card (CFFh) for IBM BladeCenter, Option part number 44X1940, replacement part number 44X1943
This tip is not software specific.
Solution
Configure the Fibre Channel Host Bus Adapter (HBA) and switch module as indicated in "Additionla information" below.
Additional Information
Select one of the following five (5) supported configurations for 8 Gigabit Fibre:
Configuration 1:
8 Gigabit Emulex Combination Input Output Vertical (CIOv) with 8 Gigabit Brocade FCSM
8 Gigabit Emulex CIOv Host Bus Adapter (HBA) configuration:
| |
Firmware 2.01a3
Boot Code 5.12a5
EDC 3.22 (see note 1)
Speed Setting Auto:
- Local Boot Auto
- Boot from SAN Auto
|
8 Gigabit Brocade FCSM switch configuration:
| |
Firmware 6.4.2b
FPGA 97 (see note 7)
Internal Port Settings:
Speed Setting: (see note 8)
- Local Boot Auto (AN)
- Boot from SAN Auto (AN)
Fill Word 1 (see note 6)
|
Configuration 2:
8 Gigabit Emulex CIOv with 8 Gigabit QLogic IPTM or FCSM
8 Gigabit Emulex CIOv HBA configuration:
| |
Firmware 2.01a3
Boot Code 5.12a5
EDC 3.22 (see note 1)
Speed Setting:
- Local Boot Auto
- Boot from SAN Auto
|
8 Gigabit QLogic IPTM or FCSM switch configuration:
| |
Firmware 7.10.1.26
Internal Port Speed Setting: (see note 8)
- Local Boot Auto
- Boot from SAN Fixed
|
Configuration 3:
8 Gigabit QLogic CIOv or Compact Form Factor Horizontal (CFFh) with 8 Gigabit Brocade FCSM
8 Gigabit QLogic CIOv or CFFh HBA configuration:
| |
Update Driver or Bundled Firmware (see note 2)
Run EDC update Kit (see notes 3,4)
Speed Setting:
- Local Boot Auto
- Boot from SAN Fixed
|
8 Gigabit Brocade FCSM switch configuration:
| |
Firmware 6.4.2b
FPGA 97 (see note 7)
Internal Port Settings:
Speed Setting: (see note 8)
- Local Boot Auto (AN)
- Boot from SAN Fixed
Fill Word 1 (see note 6)
|
Configuration 4:
8 Gigabit QLogic CIOv or CFFh with 8 Gigabit QLogic IPTM or FCSM
8 Gigabit QLogic CIOv or CFFh HBA configuration:
| |
Update Driver and/or Bundled Firmware (see note 2)
Run EDC update Kit (see notes 3,4)
Speed Setting:
- Local Boot Auto
- Boot from SAN Fixed
|
8 Gigabit QLogic IPTM or FCSM switch configuration:
| |
Firmware 7.10.1.26
Internal Port Speed Setting: (see note 8)
- Local Boot Fixed
- Boot from SAN Fixed
|
Configuration 5:
With any configuration that includes XIV storage models prior to 8 Gigabit Gen3 (4 Gigabit machine types 2810-A14 or 2812-A14).
Speed Setting Fixed 4 Gigabit on HBA and switch (see note 5)
In all configurations, ensure user is using the latest driver for the operating system.
If the problems continues after using one of the configurations above, set speed on both the HBA and the IPTM or FCSM to 4 Gigabit fixed.
If issue seen with QLogic IPTM or FCSM in bays 3 or 4 using best practices configuration, try running QLogic IPTM or FCSM in slot 8 or 10.
Notes:
- Emulex EDC (3.22) can be found on Emulex web site. Electronic Dispersion Compensation (EDC) marks the changes required for Signal Integrity. The EDC code is included in the Vital Product Data (VPD) applied during manufacturing. For the Emulex card EDC is updated using the Vitesse Flash Utility.
The utility can be found under the 'Patches' section of the Emulex download page:
http://www.emulex.com/downloads/IBM/FW-and-bootcode/lpe1205-ciov-for-IBM-BladeCenter.html
After flashing the Emulex EDC (Vitesse) code, the adapter will need to be flashed to 2.01a3 firmware, even if this step has been previously performed.
- Depending on the operating system running on the blade, the driver and bundled firmware will need to be updated, or for operating systems where the inbox driver is the supported driver, only the bundled firmware will need to be updated.
To determine the appropriate update for your operating system, reference table 3-1 in the Application Note for the update:
http://filedownloads.qlogic.com/files/Driver/82210/AppNote_QLogic_8Gb_FC_ExpCard_EDC_FW_Update_CU0251401-00A.pdf
If the operating system requires both a driver and bundled firmware update, download the specified driver package and install it according to the instructions provided. The driver install package will update both the driver and the bundled firmware. If the operating system specifies the inbox driver, download the specified RPM file to update the bundled firmware and run the RPM. The update files can be downloaded at:
http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Product_detail_new.aspx?oemid=224&companyid=6
- The EDC code level will be determined by the switch type being used. The EDC update kit is run from the operating system, it will ask whether the the connected switch is a Brocade or QLogic switch and automatically load the correct EDC code. The blade must be rebooted after the EDC update kit has been run. It is critical that the system not be interrupted or powered off during this reboot. This reboot will take slightly longer than a normal reboot. The EDC update kits are operating system specific and can be downloaded at:
http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Product_detail_new.aspx?oemid=224&companyid=6
- EDC code is only applicable to the CIOv card.
- A small number of CRC errors can be seen when running at 8 Gigabit. Current XIV storage firmware is sensitive to small numbers of CRC errors, resulting in loss of storage access. XIV is currently working on firmware that will address this issue. This occurs when the frequency of CRCs is well under the industry standard. Since XIV storage is currently 4 Gigabit max, setting speed to 4 Gigabit does not impact end to end SAN performance.
- Fill word settings can be verified using the portcfgshow command (also shows the port speed settings). The fill word for internal blade facing ports should always be set to 1 (which is the switch default). Fill word for external ports may need to be modified based on the requirements of your upstream device.
- For the Brocade 8 Gigabit switch module, the FPGA version # can be verified using the Command Line Interface (CLI), command fpga_update.
- The speed setting for internal blade facing ports should be set as specified in this document. Speed settings for external ports may need to be modified based on the requirements of your upstream device.