QLogic OFED+ Host Software User Guide, Rev. B

QLogic OFED+ Host Software 

User Guide 

QLogic OFED+ Version 1.5 

D000046-005 B

QLogic OFED+ Host Software User Guide 


Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no 

responsibility for its use, nor for any infringements of patents or other rights of third parties, which may result from its 

use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications 

described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no 

representation nor warranty that such applications are suitable for the specified use without further testing or 

modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document. 

No part of this document may be copied nor reproduced by any means, nor translated nor transmitted to any magnetic 

medium without the express written consent of QLogic Corporation. In accordance with the terms of their valid QLogic 

agreements, customers are permitted to make electronic and paper copies of this document for their own exclusive 

use. 

Rev. B, November 2010 

Document Revision History 

Changes 

Sections Affected 

Added IB Bonding sub-section Section 3, page 3-6 

Added IPATH_HCA_SELECTION_ALG to Table 

4-7. Environment Variables 

“Environment Variables” on page 4-20 

ii 

D000046-005 B

Table of Contents 

1 Introduction 

How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 

Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3 

2 Step-by-Step Cluster Setup and MPI Usage Checklists 

Cluster Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1 

Using MPI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2 

3 TrueScale Cluster Setup and Administration 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1 

Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2 

TrueScale and OpenFabrics Driver Overview . . . . . . . . . . . . . . . . . . . . . . . 3-3 

IPoIB Network Interface Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4 

IPoIB Administration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 

Administering IPoIB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 

Stopping, Starting and Restarting the IPoIB Driver. . . . . . . . . . . 3-5 

Configuring IPoIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6 

Editing the IPoIB Configuration File . . . . . . . . . . . . . . . . . . . . . . 3-6 

IB Bonding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-6 

Interface Configuration Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7 

Red Hat EL4 Update 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-7 

Red Hat EL5, All Updates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8 

SuSE Linux Enterprise Server (SLES) 10 and 11. . . . . . . . . . . . 3-9 

Verify IB Bonding is Configured. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10 

Subnet Manager Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12 

QLogic Distributed Subnet Administration . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13 

Applications that use Distributed SA . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14 

Virtual Fabrics and the Distributed SA. . . . . . . . . . . . . . . . . . . . . . . . . 3-14 

Configuring the Distributed SA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14 

Default Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14 

Multiple Virtual Fabrics Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-15 

Virtual Fabrics with Overlapping Definitions . . . . . . . . . . . . . . . . . . . . 3-16 

D000046-005 iii



Distributed SA Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-19 

SID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-19 

ScanFrequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 

LogFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 

Dbg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 

Other Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 

MPI over uDAPL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 

Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 

Managing the TrueScale Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-22 

Configure the TrueScale Driver State . . . . . . . . . . . . . . . . . . . . . . . . . 3-23 

Start, Stop, or Restart TrueScale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-23 

Unload the Driver/Modules Manually. . . . . . . . . . . . . . . . . . . . . . . . . . 3-24 

TrueScale Driver Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-24 

More Information on Configuring and Loading Drivers. . . . . . . . . . . . . . . . . 3-25 

Performance Settings and Management Tips . . . . . . . . . . . . . . . . . . . . . . . 3-25 

Homogeneous Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26 

Adapter and Other Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26 

Remove Unneeded Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28 

Disable Powersaving Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 

Hyper-Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 

Host Environment Setup for MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 

Configuring for ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-30 

Configuring ssh and sshd Using shosts.equiv . . . . . . . . . . 3-30 

Configuring for ssh Using ssh-agent . . . . . . . . . . . . . . . . . . . 3-32 

Process Limitation with ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33 

Checking Cluster and Software Status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34 

ipath_control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34 

iba_opp_query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-35 

ibstatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36 

ibv_devinfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36 

ipath_checkout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-37 

4 Running QLogic MPI on QLogic Adapters 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1 

QLogic MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1 

PSM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 

Other MPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 

Linux File I/O in MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 

MPI-IO with ROMIO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 

Getting Started with MPI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 

iv D000046-005



Copy Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 

Create the mpihosts File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 

Compile and Run an Example C Program . . . . . . . . . . . . . . . . . . . . . 4-4 

Examples Using Other Programming Languages . . . . . . . . . . . . . . . . 4-5 

QLogic MPI Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-6 

Use Wrapper Scripts for Compiling and Linking . . . . . . . . . . . . . . . . . 4-7 

Configuring MPI Programs for QLogic MPI . . . . . . . . . . . . . . . . . . . . . 4-8 

To Use Another Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 

Compiler and Linker Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 

Process Allocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11 

TrueScale Hardware Contexts on the DDR and QDR 

InfiniBand Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12 

Enabling and Disabling Software Context Sharing . . . . . . . . . . . 4-13 

Restricting TrueScale Hardware Contexts 

in a Batch Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13 

Context Sharing Error Messages . . . . . . . . . . . . . . . . . . . . . . . . 4-14 

Running in Shared Memory Mode . . . . . . . . . . . . . . . . . . . . . . . 4-14 

mpihosts File Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-15 

Using mpirun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-16 

Console I/O in MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18 

Environment for Node Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-19 

Environment Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20 

Running Multiple Versions of TrueScale or MPI . . . . . . . . . . . . . . . . . 4-22 

Job Blocking in Case of Temporary InfiniBand Link Failures. . . . . . . . 4-23 

Performance Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23 

CPU Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23 

mpirun Tunable Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 

MPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 

MPD Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 

Using MPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-25 

QLogic MPI and Hybrid MPI/OpenMP Applications . . . . . . . . . . . . . . . . . . . 4-25 

Debugging MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26 

MPI Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26 

Using Debuggers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-27 

QLogic MPI Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-28 

5 Using Other MPIs 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1 

Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2 

Open MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 

D000046-005 v



Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 

Compiling Open MPI Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 

Running Open MPI Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-4 

Further Information on Open MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

MVAPICH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

Compiling MVAPICH Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

Running MVAPICH Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-6 

Further Information on MVAPICH . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-6 

Managing Open MPI, MVAPICH, and QLogic MPI 

with the mpi-selector Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-6 

HP-MPI and Platform MPI 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8 

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8 

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8 

Compiling Platform MPI 7 Applications . . . . . . . . . . . . . . . . . . . . . . . . 5-8 

Running Platform MPI 7 Applications . . . . . . . . . . . . . . . . . . . . . . . . . 5-9 

More Information on Platform MPI 7 . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9 

Platform (Scali) MPI 5.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9 

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9 

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-9 

Compiling Platform MPI 5.6 Applications . . . . . . . . . . . . . . . . . . . . . . 5-9 

Running Platform MPI 5.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . 5-10 

Further Information on Platform MPI 5.6 . . . . . . . . . . . . . . . . . . . . . . . 5-10 

Intel MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-11 

Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-11 

Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-11 

Compiling Intel MPI Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13 

Running Intel MPI Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13 

Further Information on Intel MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-14 

Improving Performance of Other MPIs Over InfiniBand Verbs. . . . . . . . . . . 5-14 

6 Performance Scaled Messaging 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-1 

Virtual Fabric Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2 

Using SL and PKeys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-2 

Using Service ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-3 

SL2VL mapping from the Fabric Manager . . . . . . . . . . . . . . . . . . . . . . . . . . 6-3 

Verifying SL2VL tables on QLogic 7300 Series Adapters . . . . . . . . . . . . . . 6-4 

vi D000046-005



7 Dispersive Routing 

8 gPXE 

gPXE Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-1 

Required Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2 

Preparing the DHCP Server in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2 

Installing DHCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-2 

Configuring DHCP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3 

Netbooting Over InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-4 

Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5 

Boot Server Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-5 

Steps on the gPXE Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-14 

HTTP Boot Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-14 

A 

B 

C 

mpirun Options Summary 

Job Start Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 

Essential Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 

Spawn Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2 

Quiescence Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3 

Verbosity Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3 

Startup Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3 

Stats Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4 

Tuning Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 

Shell Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-6 

Debug Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-6 

Format Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-7 

Other Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-7 

Benchmark Programs 

Benchmark 1: Measuring MPI Latency Between Two Nodes . . . . . . . . . . . B-1 

Benchmark 2: Measuring MPI Bandwidth Between Two Nodes . . . . . . . . . B-3 

Benchmark 3: Messaging Rate Microbenchmarks. . . . . . . . . . . . . . . . . . . . B-4 

Benchmark 4: Measuring MPI Latency in Host Rings . . . . . . . . . . . . . . . . . B-5 

VirtualNIC Interface Configuration and Administration 

VirtualNIC Interface Configuration and Administration. . . . . . . . . . . . . . . . . C-1 

Getting Information about Ethernet IOCs on the Fabric . . . . . . . . . . . C-1 

Editing the VirtualNIC Configuration file . . . . . . . . . . . . . . . . . . . . . . . C-4 

Format 1: Defining an IOC using the IOCGUID . . . . . . . . . . . . . C-5 

Format 2: Defining an IOC using the IOCSTRING . . . . . . . . . . . C-6 

Format 3: Starting VNIC using DGID . . . . . . . . . . . . . . . . . . . . . C-6 

D000046-005 vii



VirtualNIC Failover Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-7 

Failover to a different Adapter port on the same Adapter. . . . . . C-7 

Failover to a different Ethernet port on the same 

Ethernet gateway . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-7 

Failover to a port on a different Ethernet gateway . . . . . . . . . . . C-8 

Combination method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-8 

Creating VirtualNIC Ethernet Interface Configuration Files . . . . . . . . . C-8 

VirtualNIC Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-9 

Starting, Stopping and Restarting the VirtualNIC Driver . . . . . . . . . . . C-12 

Link Aggregation Configuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-14 

Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-14 

VirtualNIC Configuration Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . C-14 

D 

SRP Configuration 

SRP Configuration Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1 

Important Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-1 

QLogic SRP Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2 

Stopping, Starting and Restarting the SRP Driver . . . . . . . . . . . . . . . . D-3 

Specifying a Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3 

Determining the values to use for the configuration . . . . . . . . . . D-6 

Specifying an SRP Initiator Port of a Session by Card and 

Port Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-8 

Specifying an SRP Initiator Port of Session by Port GUID . . . . . D-8 

Specifying a SRP Target Port . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9 

Specifying a SRP Target Port of a Session by IOCGUID . . . . . . D-10 

Specifying a SRP Target Port of a Session by Profile String . . . D-10 

Specifying an Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10 

Restarting the SRP Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10 

Configuring an Adapter with Multiple Sessions . . . . . . . . . . . . . . . . . . D-11 

Configuring Fibre Channel Failover. . . . . . . . . . . . . . . . . . . . . . . . . . . D-13 

Failover Configuration File 1: Failing over from one 

SRP Initiator port to another. . . . . . . . . . . . . . . . . . . . . . . . . . . D-14 

Failover Configuration File 2: Failing over from a port on the 

VIO hardware card to another port on the VIO hardware card. D-15 

Failover Configuration File 3: Failing over from a port on a 

VIO hardware card to a port on a different VIO hardware card 

within the same Virtual I/O chassis . . . . . . . . . . . . . . . . . . . . . D-16 


VIO hardware card to a port on a different VIO hardware 

card in a different Virtual I/O chassis . . . . . . . . . . . . . . . . . . . . D-17 

viii D000046-005



Configuring Fibre Channel Load Balancing. . . . . . . . . . . . . . . . . . . . . D-18 

1 Adapter Port and 2 Ports on a Single VIO. . . . . . . . . . . . . . . . D-18 

2 Adapter Ports and 2 Ports on a Single VIO Module . . . . . . . . D-19 

Using the roundrobinmode Parameter . . . . . . . . . . . . . . . . . . . . D-20 

Configuring SRP for Native InfiniBand Storage. . . . . . . . . . . . . . . . . . D-21 

Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-23 

Additional Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-23 

Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-23 

OFED SRP Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-24 

E 

F 

Integration with a Batch Queuing System 

Using mpiexec with PBS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-1 

Using SLURM for Batch Queuing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-2 

Allocating Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-3 

Generating the mpihosts File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-3 

Simple Process Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-4 

Clean Termination of MPI Processes . . . . . . . . . . . . . . . . . . . . . . . . . E-4 

Lock Enough Memory on Nodes when Using SLURM. . . . . . . . . . . . . . . . . E-5 

Troubleshooting 

Using LEDs to Check the State of the Adapter . . . . . . . . . . . . . . . . . . . . . . F-1 

BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-2 

Kernel and Initialization Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-2 

Driver Load Fails Due to Unsupported Kernel. . . . . . . . . . . . . . . . . . . F-3 

Rebuild or Reinstall Drivers if Different Kernel Installed . . . . . . . . . . . F-3 

InfiniPath Interrupts Not Working. . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-3 

OpenFabrics Load Errors if ib_qib Driver Load Fails . . . . . . . . . . . . F-4 

InfiniPath ib_qib Initialization Failure. . . . . . . . . . . . . . . . . . . . . . . . F-5 

MPI Job Failures Due to Initialization Problems . . . . . . . . . . . . . . . . . F-6 

OpenFabrics and InfiniPath Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-6 

Stop Infinipath Services Before Stopping/Restarting InfiniPath . . . . . . F-6 

Manual Shutdown or Restart May Hang if NFS in Use . . . . . . . . . . . . F-7 

Load and Configure IPoIB Before Loading SDP . . . . . . . . . . . . . . . . . F-7 

Set $IBPATH for OpenFabrics Scripts . . . . . . . . . . . . . . . . . . . . . . . . F-7 

SDP Module Not Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-7 

ibsrpdm Command Hangs when Two Host Channel 

Adapters are Installed but Only Unit 1 is Connected 

to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-8 

Outdated ipath_ether Configuration Setup Generates Error . . . . . . . . F-8 

System Administration Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . F-8 

D000046-005 ix



Broken Intermediate Link. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-8 

Performance Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-9 

Large Message Receive Side Bandwidth Varies with 

Socket Affinity on Opteron Systems . . . . . . . . . . . . . . . . . . . . . . . . . F-9 

Erratic Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-9 

Method 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-10 

Method 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-10 

Performance Warning if ib_qib Shares Interrupts with eth0 . . . . . F-11 

QLogic MPI Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-11 

Mixed Releases of MPI RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-12 

Missing mpirun Executable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-12 

Resolving Hostname with Multi-Homed Head Node . . . . . . . . . . . . . . F-13 

Cross-Compilation Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-13 

Compiler/Linker Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-14 

Compiler Cannot Find Include, Module, or Library Files . . . . . . . . . . . F-14 

Compiling on Development Nodes . . . . . . . . . . . . . . . . . . . . . . . F-15 

Specifying the Run-time Library Path . . . . . . . . . . . . . . . . . . . . . F-15 

Problem with Shell Special Characters and Wrapper Scripts . . . . . . . F-16 

Run Time Errors with Different MPI Implementations . . . . . . . . . . . . . F-17 

Process Limitation with ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-19 

Number of Processes Exceeds ulimit for Number of Open Files . . F-19 

Using MPI.mod Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-20 

Extending MPI Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-20 

Lock Enough Memory on Nodes When Using a Batch 

Queuing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-22 

Error Creating Shared Memory Object . . . . . . . . . . . . . . . . . . . . . . . . F-23 

gdb Gets SIG32 Signal Under mpirun -debug with the 

PSM Receive Progress Thread Enabled . . . . . . . . . . . . . . . . . . . . . F-24 

General Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-25 

Error Messages Generated by mpirun . . . . . . . . . . . . . . . . . . . . . . . F-25 

Messages from the QLogic MPI (InfiniPath) Library. . . . . . . . . . F-25 

MPI Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-27 

Driver and Link Error Messages Reported by MPI Programs. . . F-29 

MPI Stats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-30 

G 

ULP Troubleshooting 

Troubleshooting VirtualNIC and VIO Hardware Issues . . . . . . . . . . . . . . . . G-1 

x D000046-005



Checking the logical connection between the InfiniBand Host 

and the VIO hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G-1 

Verify that the proper VirtualNIC driver is running . . . . . . . . . . . G-2 

Verifying that the qlgc_vnic.cfg file contains the correct 

information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G-2 

Verifying that the host can communicate with the I/O 

Controllers (IOCs) of the VIO hardware . . . . . . . . . . . . . . . . . . G-3 

Checking the interface definitions on the host. . . . . . . . . . . . . . . . . . . G-6 

Interface does not show up in output of 'ifconfig' . . . . . . . . . . . . G-6 

Verify the physical connection between the VIO hardware and 

the Ethernet network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G-7 

Troubleshooting SRP Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G-8 

ib_qlgc_srp_stats showing session in disconnected state . . . . . G-8 

Session in 'Connection Rejected' state . . . . . . . . . . . . . . . . . . . . . . . . G-9 

Attempts to read or write to disk are unsuccessful . . . . . . . . . . . . . . . G-11 

Four sessions in a round-robin configuration are active . . . . . . . . . . . G-12 

Which port does a port GUID refer to . . . . . . . . . . . . . . . . . . . . . . . . G-12 

How does the user find a Host Channel Adapter port GUID . . . . . . . G-13 

Need to determine the SRP driver version.. . . . . . . . . . . . . . . . . . . . . G-15 

H 

I 

Write Combining 

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-1 

Verify Write Combining is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-1 

PAT and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-2 

MTRR Mapping and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H-2 

Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . . . . . . H-2 

Use the ipath_mtrr Script to Fix MTRR Issues. . . . . . . . . . . . . . . . H-3 

Useful Programs and Files 

Check Cluster Homogeneity with ipath_checkout . . . . . . . . . . . . . . . . . I-1 

Restarting InfiniPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-1 

Summary and Descriptions of Useful Programs . . . . . . . . . . . . . . . . . . . . . I-2 

dmesg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-3 

iba_opp_query. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-4 

ibhosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-7 

ibstatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-7 

ibtracert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-8 

ibv_devinfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-9 

ident . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-9 

ipathbug-helper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-10 

ipath_checkout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-10 

D000046-005 xi



ipath_control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-12 

ipath_mtrr. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-13 

ipath_pkt_test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-14 

ipathstats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-15 

lsmod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-15 

modprobe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-15 

mpirun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-15 

mpi_stress. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-16 

rpm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-16 

strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-17 

Common Tasks and Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-17 

Summary and Descriptions of Useful Files . . . . . . . . . . . . . . . . . . . . . . . . . I-18 

boardversion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-19 

status_str. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-19 

version . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-21 

Summary of Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-21 

J 

Recommended Reading 

References for MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-1 

Books for Learning MPI Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-1 

Reference and Source for SLURM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-1 

InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-1 

OpenFabrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-1 

Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-2 

Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-2 

Rocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-2 

Other Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J-2 

List of Figures 

Figure 

Page 

3-1 QLogic OFED+ Software Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1 

3-2 Distributed SA Default Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16 

3-3 Distributed SA Multiple Virtual Fabrics Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17 

3-4 Distributed SA Multiple Virtual Fabrics Configured Example . . . . . . . . . . . . . . . . . . 3-17 

3-5 Virtual Fabrics with Overlapping Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-18 

3-6 Virtual Fabrics with PSM_MPI Virtual Fabric Enabled . . . . . . . . . . . . . . . . . . . . . . . 3-18 

3-7 Virtual Fabrics with all SIDs assigned to PSM_MPI Virtual Fabric. . . . . . . . . . . . . . 3-19 

3-8 Virtual Fabrics with Unique Numeric Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-19 

C-1 Without IB_Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-10 

C-2 With IB_Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-11 

xii D000046-005



List of Tables 

Table 

Page 

4-1 QLogic MPI Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 

4-2 Command Line Options for Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 

4-3 Intel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 

4-4 Portland Group (PGI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 

4-5 PathScale Compiler Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 

4-6 Available Hardware and Software Contexts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11 

4-7 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20 

5-1 Other Supported MPI Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1 

5-2 Open MPI Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 

5-3 MVAPICH Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-5 

5-4 Platform MPI 7 Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8 

5-5 Platform MPI Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-10 

5-6 Intel MPI Wrapper Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-13 

F-1 LED Link and Data Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-1 

I-1 Useful Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-2 

I-2 ipath_checkout Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-11 

I-3 Common Tasks and Commands Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-17 

I-4 Useful Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-19 

I-5 status_str File Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-19 

I-6 Status—Other Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-20 

I-7 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-21 

D000046-005 xiii



xiv D000046-005

Preface 

The QLogic OFED+ Host Software User Guide shows end users how to use the 

installed software to setup the fabric. End users include both the cluster 

administrator and the Message-Passing Interface (MPI) application programmers, 

who have different but overlapping interests in the details of the technology. 

For specific instructions about installing the QLogic QLE7140, QLE7240, 

QLE7280, QLE7340, QLE7342, QMH7342, and QME7342 PCI Express ® (PCIe ® ) 

adapters see the QLogic InfiniBand Adapter Hardware Installation Guide, and the 

initial installation of the Fabric Software, see the QLogic Fabric Software 

Installation Guide. 

Intended Audience 

This guide is intended for end users responsible for administration of a cluster 

network as well as for end users who want to use that cluster. 

This guide assumes that all users are familiar with cluster computing, that the 

cluster administrator is familiar with Linux ® administration, and that the application 

programmer is familiar with MPI, vFabrics, VNIC, SRP, and Distributed SA. 

Related Materials 

• QLogic InfiniBand Adapter Hardware Installation Guide 

• QLogic Fabric Software Installation Guide 

• Release Notes 

Documentation Conventions 

This guide uses the following documentation conventions: 

• NOTE: provides additional information. 

• CAUTION! indicates the presence of a hazard that has the potential of 

causing damage to data or equipment. 

• WARNING!! indicates the presence of a hazard that has the potential of 

causing personal injury. 

D000046-005 B xv

Preface 

License Agreements 

• Text in blue font indicates a hyperlink (jump) to a figure, table, or section in 

this guide, and links to Web sites are shown in underlined blue. For 

example: 

Table 9-2 lists problems related to the user interface and remote agent. 

See “Installation Checklist” on page 3-6. 

For more information, visit www.qlogic.com. 

• Text in bold font indicates user interface elements such as a menu items, 

buttons, check boxes, or column headings. For example: 

Click the Start button, point to Programs, point to Accessories, and 

then click Command Prompt. 

Under Notification Options, select the Warning Alarms check box. 

• Text in Courier font indicates a file name, directory path, or command line 

text. For example: 

To return to the root directory from anywhere in the file structure: 

Type cd /root and press ENTER. 

Enter the following command: sh ./install.bin 

• Key names and key strokes are indicated with UPPERCASE: 

Press CTRL+P. 

Press the UP ARROW key. 

• Text in italics indicates terms, emphasis, variables, or document titles. For 

example: 

For a complete listing of license agreements, refer to the QLogic 

Software End User License Agreement. 

What are shortcut keys 

 

To enter the date type mm/dd/yyyy (where mm is the month, dd is the 

day, and yyyy is the year). 

• Topic titles between quotation marks identify related topics either within this 

manual, or in the online help that is referred to as the help system 

throughout this document. 

License Agreements 

Refer to the QLogic Software End User License Agreement for a complete listing 

of all license agreements affecting this product. 

xvi 

D000046-005 B

Preface 

Technical Support 


Availability 

Customers should contact their authorized maintenance provider for technical 

support of their QLogic InfiniBand products. QLogic-direct customers may contact 

QLogic Technical Support; others will be redirected to their authorized 

maintenance provider. 

Visit the QLogic support Web site listed in Contact Information for the latest 

firmware and software updates. 

QLogic Technical Support for products under warranty is available during local 

standard working hours excluding QLogic Observed Holidays. 

Training 

QLogic offers training for technical professionals for all iSCSI, InfiniBand, and 

Fibre Channel products. From the main QLogic web page at www.qlogic.com, 

click the Education and Resources tab at the top, then click the Education & 

Training tab on the left. The QLogic Global Training Portal offers online courses, 

certification exams, and scheduling of in-person training. 

Technical Certification courses include installation, maintenance and 

troubleshooting QLogic SAN products. Upon demonstrating knowledge using live 

equipment, QLogic awards a certificate identifying the student as a Certified 

Professional. The training professionals at QLogic may be reached by e-mail at 

training@qlogic.com. 

Contact Information 

Please feel free to contact your QLogic approved reseller or QLogic Technical 

Support at any phase of integration for assistance. QLogic Technical Support can 

be reached by the following methods: 

Web 

Email 

http://support.qlogic.com 

support@qlogic.com 

Knowledge Database 

The QLogic knowledge database is an extensive collection of QLogic product 

information that you can search for specific solutions. We are constantly adding to 

the collection of information in our database to provide answers to your most 

urgent questions. Access the database from the QLogic Support Center: 

http://support.qlogic.com. 

D000046-005 B xvii

Preface 


xviii 

D000046-005 B

1 Introduction 

How this Guide is Organized 

The QLogic OFED+ Host Software User Guide is organized into these sections: 

• Section 1, provides an overview and describes interoperability. 

• Section 2, describes how to setup your cluster to run high-performance MPI 

jobs. 

• Section 3, describes the lower levels of the supplied QLogic OFED+ Host 

software. This section is of interest to a TrueScale cluster administrator. 

• Section 4, helps the Message Passing Interface (MPI) programmer make the 

best use of the QLogic MPI implementation. Examples are provided for 

compiling and running MPI programs. 

• Section 5, gives examples for compiling and running MPI programs with 

other MPI implementations. 

• Section 6, describes QLogic Performance Scaled Messaging (PSM) that 

provides support for full Virtual Fabric (vFabric) integration, allowing users to 

specify InfiniBand Service Level (SL) and Partition Key (PKey), or to provide 

a configured Service ID (SID) to target a vFabric. 

• Section 7, describes dispersive routing in the InfiniBand ® fabric to avoid 

congestion hotspots by “sraying” messages across the multiple potential 

paths. 

• Section 8, describes open-source Preboot Execution Environment (gPXE) 

boot including installation and setup. 

• Appendix A, describes the most commonly used options to mpirun. 

• Appendix B, describes how to run QLogic’s performance measurement 

programs. 

• Appendix C, describes the VirtualNIC interface configuration and 

administration, providing virtual Ethernet connectivity. 

• Appendix D, describes SCSI RDMA Protocol (SRP) configuration that allows 

the SCSI protocol to run over InfiniBand for Storage Area Network (SAN) 

usage. 

D000046-005 B 1-1

1–Introduction 

Overview 

Overview 

• Appendix E, describes two methods the administrator can use to allow users 

to submit MPI jobs through batch queuing systems. 

• Appendix F, provides information for troubleshooting installation, cluster 

administration, and MPI. 

• Appendix G, provides information for troubleshooting the upper layer 

protocol utilities in the fabric. 

• Appendix H, provides instructions for checking write combining and for using 

the Page Attribute Table (PAT) and Memory Type Range Registers (MTRR). 

• Appendix I, contains useful programs and files for debugging, as well as 

commands for common tasks. 

• Appendix J, contains a list of useful web sites and documents for a further 

understanding of the InfiniBand fabric, and related information. 

In addition, the QLogic InfiniBand Adapter Hardware Installation Guide contains 

information on QLogic hardware installation and the QLogic Fabric Software 

Installation Guide contains information on QLogic software installation. 

The material in this documentation pertains to a QLogic OFED cluster. A cluster is 

defined as a collection of nodes, each attached to an InfiniBand-based fabric 

through the QLogic interconnect. 

The QLogic InfiniBand adapters are InfiniBand 4X adapters. The quad data rate 

(QDR) adapters (QLE7340, QLE7342, QMH7342, and QME7342) have a raw 

data rate of 40Gbps (data rate of 32Gbps). The double data rate (DDR) adapters 

(QLE7240 and QLE7280) have a raw data rate of 20Gbps (data rate of 16Gbps). 

The single data rate (SDR) adapters (QLE7140) have a raw data rate of 10Gbps 

(data rate of 8Gbps). The QLE7340, QLE7342, QMH7342, and QME7342 

adapters can also run in DDR or SDR mode, and the QLE7240 and QLE7280 can 

also run in SDR mode. 

The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and 

cabling. The QLogic interconnect is designed to work with all InfiniBand-compliant 

switches. 

NOTE: 

If you are using the QLE7240 or QLE7280, and want to use DDR mode, 

then DDR-capable switches must be used. Likewise, when using the 

QLE7300 series adapters in QDR mode, a QDR switch must be used. 

1-2 D000046-005 B


Interoperability 

QLogic OFED+ software is interoperable with other vendors’ IBTA compliant 

InfiniBand adapters running compatible OFED releases. There are several 

options for subnet management in your cluster: 

• An embedded subnet manager can be used in one or more managed 

switches. QLogic offers the QLogic Embedded Fabric Manager (FM) for 

both DDR and QDR switch product lines supplied by your InfiniBand switch 

vendor. 

• A host-based subnet manager can be used. QLogic provides the QLogic 

Fabric Manager (FM), as a part of the QLogic InfiniBand Fabric Suite. 


QLogic OFED+ participates in the standard InfiniBand subnet management 

protocols for configuration and monitoring. Note that: 

• QLogic OFED+ (including Internet Protocol over InfiniBand (IPoIB)) is 

interoperable with other vendors’ InfiniBand adapters running compatible 

OFED releases. 

• The QLogic MPI stack is not interoperable with other InfiniBand adapters 

and target channel adapters. Instead, it uses an InfiniBand-compliant, 

vendor-specific protocol that is highly optimized for QLogic MPI and the 

QLogic PSM API. 

• In addition to supporting running MPI over verbs, QLogic provides a 

high-performance InfiniBand-Compliant vendor-specific protocol for running 

over verbs, known as PSM. MPIs run over PSM will not interoperate with 

other adapters. 

NOTE: 

See the OpenFabrics web site at www.openfabrics.org for more information 

on the OpenFabrics Alliance. 

D000046-005 B 1-3



1-4 D000046-005 B

2 Step-by-Step Cluster Setup 

and MPI Usage Checklists 

Cluster Setup 

This section describes how to set up your cluster to run high-performance 

Message Passing Interface (MPI) jobs. 

Perform the following tasks when setting up the cluster. These include BIOS, 

adapter, and system settings. 

1. Make sure that hardware installation has been completed according to the 

instructions in the QLogic InfiniBand Adapter Hardware Installation Guide 

and software installation and driver configuration has been completed 

according to the instructions in the QLogic Fabric Software Installation 

Guide. To minimize management problems, the compute nodes of the 

cluster must have very similar hardware configurations and identical 

software installations. See “Homogeneous Nodes” on page 3-26 for more 

information. 

2. Check that the BIOS is set properly according to the instructions in the 

QLogic InfiniBand Adapter Hardware Installation Guide. 

3. Set up the Distributed Subnet Administration (SA) to correctly synchronize 

your virtual fabrics. See “QLogic Distributed Subnet Administration” on 

page 3-13 

4. Adjust settings, including setting the appropriate MTU size. See “Adapter 

and Other Settings” on page 3-26. 

5. Remove unneeded services. QLogic recommends turning irqbalance off. 

See “Remove Unneeded Services” on page 3-28. 

6. Disable powersaving features. See “Disable Powersaving Features” on 

page 3-29. 

7. Check other performance tuning settings. See “Performance Settings and 

Management Tips” on page 3-25. 

8. If using Intel ® processors, turn off Hyper-Threading. See “Hyper-Threading” 

on page 3-29. 

D000046-005 B 2-1

2–Step-by-Step Cluster Setup and MPI Usage Checklists 

Using MPI 

Using MPI 

9. Set up the host environment to use ssh. Two methods are discussed in 

“Host Environment Setup for MPI” on page 3-29. 

10. Verify the cluster setup. See “Checking Cluster and Software Status” on 

page 3-34. 

1. Verify that the QLogic hardware and software has been installed on all the 

nodes you will be using, and that ssh is set up on your cluster (see all the 

steps in the Cluster Setup checklist). 

2. Copy the examples to your working directory. See “Copy Examples” on 

page 4-3. 

3. Make an mpihosts file that lists the nodes where your programs will run. 

See “Create the mpihosts File” on page 4-3. 

4. Compile the example C program using the default wrapper script mpicc. 

Use mpirun to run it. See “Compile and Run an Example C Program” on 

page 4-4. 

5. Try the examples with other programming languages, C++, Fortran 77, and 

Fortran 90 in “Examples Using Other Programming Languages” on 

page 4-5. 

6. To test using other MPIs that run over PSM, such as MVAPICH, Open MPI, 

HP ® -MPI, Platform MPI, and Intel MPI, see Section 5 Using Other MPIs. 

7. To switch between multiple versions of Open MPI, MVAPICH, and QLogic 

MPI, use the mpi-selector. See “Managing Open MPI, MVAPICH, and 

QLogic MPI with the mpi-selector Utility” on page 5-6. 

8. Refer to “QLogic MPI Details” on page 4-6 for more information about 

QLogic MPI, and to “Performance Tuning” on page 4-23 to read more about 

runtime performance tuning. 

9. Refer to Section 5 Using Other MPIs to learn about using other MPI 

implementations. 

2-2 D000046-005 B

3 TrueScale Cluster Setup 

and Administration 

Introduction 

This section describes what the cluster administrator needs to know about the 

QLogic OFED+ software and system administration. 

The TrueScale driver ib_qib, QLogic Performance Scaled Messaging (PSM), 

accelerated Message-Passing Interface (MPI) stack, the protocol and MPI support 

libraries, and other modules are components of the QLogic OFED+ software. This 

software provides the foundation that supports the MPI implementation. 

Figure 3-1 illustrates these relationships. Note that HP-MPI, Scali, MVAPICH, 

MVAPICH2, and Open MPI can run either over PSM or OpenFabrics ® User Verbs. 

The QLogic Virtual Network Interface Controller (VNIC) driver module is also 

illustrated in the figure. 

MPI Applications 

User Space 

Commo 

n 

InfiniBand/OpenFabri 

QLogic OFED+ 

Hardware 

QLogic MPI 

HP-MPI 

Scali 

MVAPICH 

Open MPI 

QLogic OFED+ 

Communication 

Library (PSM) 

HP-MPI 

Scali 

MVAPICH 

MVAPICH2 

Open MPI 

User Verbs 

HP-MPI 

Intel MPI 

uDAPL 

QLogic FM 

uMAD API 

Kernel Space 

TCP/IP 

IPoIB 

VNIC 

QLogic OFED+ Driver 

QLogic infiniband adapter 

Figure 3-1. QLogic OFED+ Software Structure 

D000046-005 B 3-1

3–TrueScale Cluster Setup and Administration 

Installed Layout 


This section describes the default installed layout for the QLogic OFED+ software 

and QLogic-supplied MPIs. 

The QLogic MPI is installed in: 

/usr/mpi/qlogic 

The shared libraries are installed in: 

/usr/mpi/qlogic/lib for 32-bit applications 

/usr/mpi/qlogic/lib64 for 64-bit applications 

MPI include files are in: 

/usr/mpi/qlogic/cdinclude 

MPI programming examples and the source for several MPI benchmarks are in: 

/usr/mpi/qlogic/share/mpich/examples 

NOTE: 

If QLogic MPI is installed in an alternate location, the argument passed to 

--prefix (Your location) replaces the default /usr/mpi/qlogic 

prefix. QLogic MPI binaries, documentation, and libraries are installed under 

that prefix. However, a few configuration files are installed in /etc 

regardless of the desired --prefix. 

If you have installed the software into an alternate location, the 

$MPICH_ROOT environment variable needs to match --prefix. 

QLogic OFED+ utility programs, are installed in: 

/usr/bin 

Documentation is found in: 

/usr/share/man 

/usr/share/doc/infinipath 

/usr/share/doc/mpich-infinipath 

License information is found only in usr/share/doc/infinipath. QLogic 

OFED+ Host Software user documentation can be found on the QLogic web site 

on the software download page for your distribution. 

Configuration files are found in: 

/etc/sysconfig 

Init scripts are found in: 

/etc/init.d 

3-2 D000046-005 B


TrueScale and OpenFabrics Driver Overview 

The TrueScale driver modules in this release are installed in: 

/lib/modules/$(uname -r)/ 

updates/kernel/drivers/infiniband/hw/qib 

Most of the other OFED modules are installed under the infiniband 

subdirectory. Other modules are installed under: 

/lib/modules/$(uname -r)/updates/kernel/drivers/net 

The RDS modules are installed under: 

/lib/modules/$(uname -r)/updates/kernel/net/rds 

QLogic-supplied OpenMPI and MVAPICH RPMs with PSM support and compiled 

with GCC, PathScale, PGI, and the Intel compilers are now installed in directories 

using this format: 

/usr/mpi//--qlc 

For example: 

/usr/mpi/gcc/openmpi-1.4-qlc 

TrueScale and OpenFabrics Driver Overview 

The TrueScale ib_qib module provides low-level QLogic hardware support, and 

is the base driver for both MPI/PSM programs and general OpenFabrics protocols 

such as IPoIB and service data point (SDP). The driver also supplies the Subnet 

Management Agent (SMA) component. 

Optional configurable OpenFabrics components and their default settings at 

startup are: 

• IPoIB network interface. This component is required for TCP/IP networking 

for running Ethernet traffic over the TrueScale link. It is not running until it is 

configured. 

• VNIC. It is not running until it is configured. 

• OpenSM. This component is disabled at startup. It can be installed on one 

node as a master, with another node being a standby, or disable it on all 

nodes except where it will be used as an SM. 

• SRP (OFED and QLogic modules). SRP is not running until the module is 

loaded and the SRP devices on the fabric have been discovered. 

• MPI over uDAPL (can be used by Intel MPI or HP ® -MPI). IPoIB must be 

configured before MPI over uDAPL can be set up. 

Other optional drivers can now be configured and enabled, as described in “IPoIB 

Network Interface Configuration” on page 3-4. 

D000046-005 B 3-3


IPoIB Network Interface Configuration 

Complete information about starting, stopping, and restarting the QLogic OFED+ 

services are in “Managing the TrueScale Driver” on page 3-22. 

IPoIB Network Interface Configuration 

The following instructions show you how to manually configure your OpenFabrics 

IPoIB network interface. QLogic recommends using the QLogic OFED+ Host 

Software Installation package that automatically installs the IPoIB Network 

Interface configuration. This example assumes that you are using sh or bash as 

your shell, all required QLogic OFED+ and OpenFabric’s RPMs are installed, and 

your startup scripts have been ran (either manually or at system boot). 

For this example, the IPoIB network is 10.1.17.0 (one of the networks reserved for 

private use, and thus not routable on the Internet), with a /8 host portion. In this 

case, the netmask must be specified. 

This example assumes that no hosts files exist, the host being configured has the 

IP address 10.1.17.3, and DHCP is not used. 

NOTE: 

Instructions are only for this static IP address case. Configuration methods 

for using DHCP will be supplied in a later release. 

1. Type the following command (as a root user): 

ifconfig ib0 10.1.17.3 netmask 0xffffff00 

2. To verify the configuration, type: 

ifconfig ib0 

ifconfig ib1 

The output from this command will be similar to: 

ib0 Link encap:InfiniBand HWaddr 

00:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00: 

00 

inet addr:10.1.17.3 Bcast:10.1.17.255 Mask:255.255.255.0 

UP BROADCAST RUNNING MULTICAST MTU:4096 Metric:1 

RX packets:0 errors:0 dropped:0 overruns:0 frame:0 

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 

collisions:0 txqueuelen:128 

RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) 

3-4 D000046-005 B


IPoIB Administration 

3. Type: 

ping -c 2 -b 10.1.17.255 

The output of the ping command will be similar to the following, with a line 

for each host already configured and connected: 

WARNING: pinging broadcast address 

PING 10.1.17.255 (10.1.17.255) 517(84) bytes of data. 

174 bytes from 10.1.17.3: icmp_seq=0 ttl=174 time=0.022 

ms 

64 bytes from 10.1.17.1: icmp_seq=0 ttl=64 time=0.070 ms 

(DUP!) 

64 bytes from 10.1.17.7: icmp_seq=0 ttl=64 time=0.073 ms 

(DUP!) 

The IPoIB network interface is now configured. 

4. Restart (as a root user) by typing: 

/etc/init.d/openibd restart 

NOTE: 

• The configuration must be repeated each time the system is rebooted. 

• IPoIB-CM (Connected Mode) is enabled by default. The setting in 

/etc/infiniband/openib.conf is SET_IPOIB_CM=yes. To use 

datagram mode, use change the setting to SET_IPOIB_CM=no. 

IPoIB Administration 

Administering IPoIB 

Stopping, Starting and Restarting the IPoIB Driver 

QLogic recommends using the QLogic IFS Installer TUI to stop, stat and restart 

the IPoIB driver. Refer to the QLogic Fabric Software Installation Guide for more 

information. For using the command line to stop, start, and restart the IPoIB driver 

use the following commands. 

To stop the IPoIB driver, use the following command: 

/etc/init.d/openibd stop 

To start the IPoIB driver, use the following command: 

/etc/init.d/openibd start 

D000046-005 B 3-5


IB Bonding 

To restart the IPoIB driver, use the following command: 


Configuring IPoIB 

QLogic recommends using the QLogic IFS Installer TUI to configure the IPoIB 

driver. Refer to the QLogic Fabric Software Installation Guide for more 

information. For using the command line to configure the IPoIB driver use the 

following commands. 

Editing the IPoIB Configuration File 

1. For each IP Link Layer interface, create an interface configuration file, 

/etc/sysconfig/network-scripts/ifcfg-NAME, where NAME is the 

value of the NAME field specified in the CREATE block. The following is a 

sample: /etc/sysconfig/network-scripts/ifcfg-ib1 file: 

DEVICE=ib1 

BOOTPROTO=static 

BROADCAST=192.168.18.255 

IPADDR=192.168.18.120 

NETMASK=255.255.255.0 

ONBOOT=yes 

NOTE: 

For IPoIB, the INSTALL script for the adapter now helps the user 

create the ifcfg files. 

IB Bonding 

2. After modifying the /etc/sysconfig/ipoib.cfg file, restart the IPoIB driver 

with the following: 


IB bonding is a high availability solution for IPoIB interfaces. It is based on the 

Linux Ethernet Bonding Driver and was adopted to work with IPoIB. The support 

for IPoIB interfaces is only for the active-backup mode, other modes should not be 

used. QLogic only supports bonding across Host Channel Adapter ports at this 

time. Bonding port 1 and port 2 on the same Host Channel Adapter is not 

supported in this release. 

3-6 D000046-005 B


IB Bonding 

Interface Configuration Scripts 

Create interface configuration scripts for the ibX and bondX interfaces. Once the 

configurations are in place, perform a server reboot, or a service network restart. 

For SLES operating systems (OS), a server reboot is required. Refer to the 

following standard syntax for bonding configuration by the OS. 

NOTE: 

For all of the following OS configuration script examples that set MTU, 

MTU=65520 is valid only if all IPoIB slaves operate in connected mode and 

are configured with the same value. For IPoIB slaves that work in datagram 

mode, use MTU=2044. If the MTU is not set correctly or the MTU is not set 

at all (set to the default value), performance of the interface may be lower. 

Red Hat EL4 Update 8 

The following is an example for bond0 (master). The file is named 

/etc/sysconfig/network-scripts/ifcfg-bond0: 

DEVICE=bond0 

IPADDR=192.168.1.1 

NETMASK=255.255.255.0 

NETWORK=192.168.1.0 

BROADCAST=192.168.1.255 

ONBOOT=yes 

BOOTPROTO=none 

USERCTL=no 

TYPE=Bonding 

MTU=65520 

BONDING_OPTS="primary=ib0 updelay=0 downdelay=0 

The following is an example for ib0 (slave). The file is named 

/etc/sysconfig/network-scripts/ifcfg-ib0: 

DEVICE=ib0 

USERCTL=no 

ONBOOT=yes 

MASTER=bond0 

SLAVE=yes 


TYPE=InfiniBand 

PRIMARY=yes 

D000046-005 B 3-7


IB Bonding 

The following is an example for ib1 (slave 2). The file is named 


DEVICE=ib1 

USERCTL=no 

ONBOOT=yes 

MASTER=bond0 

SLAVE=yes 



Add the following lines to the file /etc/modprobe.conf: 

alias bond0 bonding 

options bond0 miimon=100 mode=1 max_bonds=1 

Red Hat EL5, All Updates 



DEVICE=bond0 

IPADDR=192.168.1.1 

NETMASK=255.255.255.0 

NETWORK=192.168.1.0 

BROADCAST=192.168.1.255 

ONBOOT=yes 


USERCTL=no 

MTU=65520 

BONDING_OPTS="primary=ib0 updelay=0 downdelay=0" 



DEVICE=ib0 

USERCTL=no 

ONBOOT=yes 

MASTER=bond0 

SLAVE=yes 



PRIMARY=yes 

3-8 D000046-005 B


IB Bonding 



DEVICE=ib1 

USERCTL=no 

ONBOOT=yes 

MASTER=bond0 

SLAVE=yes 



Add the following lines to the file /etc/modprobe.conf: 

alias bond0 bonding 

options bond0 miimon=100 mode=1 max_bonds=1 

SuSE Linux Enterprise Server (SLES) 10 and 11 



DEVICE="bond0" 

TYPE="Bonding" 

IPADDR="192.168.1.1" 

NETMASK="255.255.255.0" 

NETWORK="192.168.1.0" 

BROADCAST="192.168.1.255" 

BOOTPROTO="static" 

USERCTL="no" 

STARTMODE="onboot" 

BONDING_MASTER="yes" 

BONDING_MODULE_OPTS="mode=active-backup miimon=100 

primary=ib0 updelay=0 downdelay=0" 

BONDING_SLAVE0=ib0 

BONDING_SLAVE1=ib1 

MTU=65520 

D000046-005 B 3-9


IB Bonding 



DEVICE='ib0' 

BOOTPROTO='none' 

STARTMODE='off' 

WIRELESS='no' 

ETHTOOL_OPTIONS='' 

NAME='' 

USERCONTROL='no' 

IPOIB_MODE='connected' 



DEVICE='ib1' 

BOOTPROTO='none' 

STARTMODE='off' 

WIRELESS='no' 

ETHTOOL_OPTIONS='' 

NAME='' 

USERCONTROL='no' 

IPOIB_MODE='connected' 

Verify the following line is set to the value of yes in /etc/sysconfig/boot: 

RUN_PARALLEL="yes" 

Verify IB Bonding is Configured 

After the configuration scripts are updated, and the service network is restarted or 

a server reboot is accomplished, use the following CLI commands to verify that IB 

bonding is configured. 

• cat /proc/net/bonding/bond0 

• # ifconfig 

3-10 D000046-005 B


IB Bonding 

Example of cat /proc/net/bonding/bond0 output: 

# cat /proc/net/bonding/bond0 

Ethernet Channel Bonding Driver: v3.2.3 (December 6, 2007) 

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac) 

Primary Slave: ib0 

Currently Active Slave: ib0 

MII Status: up 

MII Polling Interval (ms): 100 

Up Delay (ms): 0 

Down Delay (ms): 0 

Slave Interface: ib0 


Link Failure Count: 0 

Permanent HW addr: 80:00:04:04:fe:80 

Slave Interface: ib1 


Link Failure Count: 0 

Permanent HW addr: 80:00:04:05:fe:80 

D000046-005 B 3-11


Subnet Manager Configuration 

Example of Ifconfig output: 

st2169:/etc/sysconfig # ifconfig 

bond0 Link encap:InfiniBand HWaddr 

80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 

inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 

inet6 addr: fe80::211:7500:ff:909b/64 Scope:Link 

UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1 




RX bytes:10132014352 (9662.6 Mb) TX bytes:10614493096 (10122.7 Mb) 


80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 

UP BROADCAST RUNNING SLAVE MULTICAST MTU:65520 Metric:1 






80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 

UP BROADCAST RUNNING SLAVE MULTICAST MTU:65520 Metric:1 





Subnet Manager Configuration 

QLogic recommends using the QLogic Fabric Manager to manage your fabric. 

Refer to the QLogic Fabric Manager User Guide for information on configuring the 

QLogic Fabric Manager. 

OpenSM is an optional component of the OpenFabrics project that provides a 

Subnet Manager (SM) for InfiniBand networks. This package can be installed on 

all machines, but only needs to be enabled on the machine in the cluster that will 

act as a subnet manager. You can’t use OpenSM if any of your InfiniBand 

switches provide a subnet manager, or if you are running a host-based SM. 

WARNING!! 

Don’t run OpenSM with QLogic FM in the same fabric. 

3-12 D000046-005 B


QLogic Distributed Subnet Administration 

If you are using the Installer tool, you can set the OpenSM default behavior at the 

time of installation. 

OpenSM only needs to be enabled on the node that acts as the subnet manager, 

so use the chkconfig command (as a root user) to enable it on the node where it 

will be run: 

chkconfig opensmd on 

The command to disable it on reboot is: 

chkconfig opensmd off 

You can start opensmd without rebooting your machine by typing: 

/etc/init.d/opensmd start 

You can stop opensmd again by typing: 

/etc/init.d/opensmd stop 

If you want to pass any arguments to the OpenSM program, modify the following 

file, and add the arguments to the OPTIONS variable: 

/etc/init.d/opensmd 

For example: 

Use the UPDN algorithm instead of the Min Hop algorithm. 

OPTIONS="-R updn" 

For more information on OpenSM, see the OpenSM man pages, or look on the 

OpenFabrics web site. 


As InfiniBand clusters are scaled into the Petaflop range and beyond, a more 

efficient method for handling queries to the Fabric Manager is required. One of 

these issues is that while the Fabric Manager can configure and operate that 

many nodes, under certain conditions it can become overloaded with queries from 

those same nodes. 

For example, consider an InfiniBand fabric consisting of 1,000 nodes, each with 4 

processors. When a large MPI job is started across the entire fabric, each process 

needs to collect InfiniBand path records for every other node in the fabric - and 

every single process is going to be querying the subnet manager for these path 

records at roughly the same time. This amounts to a total of 3.9 million path 

queries just to start the job! 

In the past, MPI implementations have side-stepped this problem by hand crafting 

path records themselves, but this solution cannot be used if advanced fabric 

management techniques such as virtual fabrics and mesh/torus configurations are 

being used. In such cases, only the subnet manager itself has enough information 

to correctly build a path record between two nodes. 

D000046-005 B 3-13



The Distributed Subnet Administration (SA) solves this problem by allowing each 

node to locally replicate the path records needed to reach the other nodes on the 

fabric. At boot time, each Distributed SA queries the subnet manager for 

information about the relevant parts of the fabric, backing off whenever the subnet 

manager indicates that it is busy. Once this information is in the Distributed SA's 

database, it is ready to answer local path queries from MPI or other InfiniBand 

applications. If the fabric changes (due to a switch failure or a node being added 

or removed from the fabric) the Distributed SA updates the affected portions of the 

database. The Distributed SA is installed and runs on every node in the fabric, 

except the management node running QLogic FM. 

Applications that use Distributed SA 

The QLogic PSM Library has been extended to take advantage of Distributed SA. 

Therefore, all MPIs that use the QLogic PSM library can take advantage of the 

Distributed SA. Other applications must be modified specifically to take advantage 

of it. For developers writing applications that use the Distributed SA, please refer 

to the header file /usr/include/Infiniband/ofedplus_path.h for information on 

modifying applications to use the Distributed SA. This file can be found on any 

node where the Distributed SA is installed. For further assistance please contact 

QLogic Support. 

Virtual Fabrics and the Distributed SA 

The Distributed SA is designed to be aware of Virtual Fabrics, but to only store 

records for those Virtual Fabrics that match Service ID records in the Distributed 

SA's configuration file. In addition, the Distributed SA recognizes when multiple 

SIDs match the same Virtual Fabric and will only store one copy of each path 

record within a Virtual Fabric. Next, SIDs that match more than one Virtual Fabric 

will be assigned to a single Virtual Fabrics. Finally, Virtual Fabrics that do not 

match SIDs in the Distributed SA's database will be ignored. 

Configuring the Distributed SA 

In order to absolutely minimize the number of queries made by the Distributed SA, 

it is important to configure it correctly, both to match the configuration of the Fabric 

Manager and to exclude those portions of the fabric that will not be used by 

applications using the Distributed SA. The configuration file for the Distributed SA 

is named /etc/sysconfig/iba/qlogic_sa.conf. 

Default Configuration 

As shipped, the QLogic Fabric Manager creates a single virtual fabric, called 

“Default” and maps all nodes and Service IDs to it, and the Distributed SA ships 

with a configuration that lists a set of thirty-one Service IDs (SID 

0x1000117500000000 through 0x100011750000000f and 0x1 through 0xf). This 

results in an arrangement like the one shown in Figure 3-2 

3-14 D000046-005 B



Virtual Fabric “Default” 

Pkey: 0xffff 

SID Range: 0 x0-0 xffffffffffffffff 


Pkey: 0xffff 

SID Range: 0x1-0xf 

SID Range: 0x 1000117500000000-0x 100011750000000f 

Infiniband Fabric 

Distributed SA 

Figure 3-2. Distributed SA Default Configuration 

If you are using the QLogic FM in its default configuration, and you are using the 

standard QLogic PSM Service IDs, this arrangement will work fine and you will not 

need to modify the Distributed SA's configuration file - but notice that the 

Distributed SA has restricted the range of Service IDs it cares about to those that 

were defined in its configuration file. Attempts to get path records using other SIDs 

will not work, even if those other SIDs are valid for the fabric. 

Multiple Virtual Fabrics Example 

A person configuring the physical InfiniBand fabric may want to limit how much 

InfiniBand bandwidth MPI applications are permitted to consume. In that case, 

they may re-configure the QLogic Fabric Manager, turning off the "Default" Virtual 

Fabric and replacing it with several other Virtual Fabrics. 

In Figure 3-3, the administrator has divided the physical fabric into four virtual 

fabrics: "Admin" (used to communicate with the Fabric Manager), "Storage" (used 

by SRP), "PSM_MPI" (used by regular MPI jobs) and a special "Reserved" fabric 

for special high-priority jobs. 

D000046-005 B 3-15



Virtual Fabric “Admin” 

Pkey: 0x7fff 

Virtual Fabric 

“Reserved” 

Pkey: 0x8002 

SID Range: 0x10-0x1f 

Virtual Fabric “Storage” 

Pkey: 0x8001 

SID: 

0x0000494353535250 


“PSM_MPI” 

Pkey: 0x8003 


SID Range: 

0x1000117500000000- 

0x100011750000000f 

Virtual Fabric “PSM_MPI” 

Pkey: 0x8003 


SID Range: 0x1000117500000000- 

0x100011750000000f 



Figure 3-3. Distributed SA Multiple Virtual Fabrics Example 

Due to the fact that the Distributed SA was not configured to include the SID 

Range 0x10 through 0x1f, it has simply ignored the "Reserved" VF. Adding those 

SIDs to the qlogic_sa.conf file solves the problem as shown in Figure 3-4. 

Virtual Fabric “Admin” 

Pkey: 0x7fff 


“Reserved” 

Pkey: 0x8002 


Virtual Fabric “Reserved” 

Pkey: 0x8002 


Virtual Fabric “Storage” 

Pkey: 0x8001 

SID: 

0x0000494353535250 


“PSM_MPI” 

Pkey: 0x8003 


SID Range: 

0x1000117500000000- 

0x100011750000000f 


Pkey: 0x8003 


SID Range: 0x1000117500000000- 

0x100011750000000f 



Figure 3-4. Distributed SA Multiple Virtual Fabrics Configured Example 

Virtual Fabrics with Overlapping Definitions 

As defined, SIDs should never be shared between Virtual Fabrics. Unfortunately, 

it is very easy to accidentally create such overlaps. Figure 3-5 shows an example 

with overlapping definitions. 

3-16 D000046-005 B




Virtual 

Pkey: 

Fabric 

0xffff 

“Default” 

SID Range: 

Pkey: 

0x0-0xffffffffffffffff 

0xffff 

SID Range: Virtual 0x0-0xffffffffffffffff 

Fabric “PSM_MPI” 

Pkey: 0x8002 


SID Range: 

0x1000117500000000- 

0x100011750000000f 

 

Looking for for SID SID Ranges Range 0x1-0xf 0x1-0xf and 

and 0x1000117500000000- 

0x100011750000000f 



Figure 3-5. Virtual Fabrics with Overlapping Definitions 

In Figure 3-5, the fabric administrator enabled the "PSM_MPI" Virtual Fabric 

without modifying the "Default" Virtual Fabric. As a result, the Distributed SA sees 

two different virtual fabrics that match its configuration file. 

In Figure 3-6, the person administering the fabric has created two different Virtual 

Fabrics without turning off the Default - and two of the new fabrics have 

overlapping SID ranges. 


ID: 2 

Pkey: 0x8003 


Virtual Virtual Fabric Fabric “Default” “Default” Pkey: 0xffff 

SID Range: Pkey: 0x0-0xffffffffffffffff 

0xffff 

SID Range: 

Virtual 



ID: 1 Pkey: 0x8002 


SID Range: 

0x1000117500000000- 

0x100011750000000f 

 


Pkey: 0xffff 


SID 

0x1000117500000000- 

Range: 0x1000117500000000- 

0x100011750000000f 

Looking for SID Ranges 0x1-0xf and 



Figure 3-6. Virtual Fabrics with PSM_MPI Virtual Fabric Enabled 

In Figure 3-6, the administrator enabled the "PSM_MPI" fabric, and then added a 

new "Reserved" fabric that uses one of the SID ranges that "PSM_MPI" uses. 

When a path query has been received, the Distributed SA deals with these 

conflicts as follows: 

D000046-005 B 3-17



First, any virtual fabric with a pkey of 0xffff is declared to be the "Default". The 

"Default" Virtual Fabric is treated as a special case by the Distributed SA. The 

"Default" Virtual Fabric is used only as a last resort. Stored SIDs are only mapped 

to the default if they do not match any other Virtual Fabrics. Thus, in the first 

example, Figure 3-6, the Distributed SA will assign all the SIDs in its configuration 

file to the "PSM_MPI" Virtual Fabric as shown in Figure 3-7. 


Pkey: 0xffff 

SID Range: 0x0-0xffffffffffffffff 


Pkey: 0x8002 


SID Range: 

0x1000117500000000- 

0x100011750000000f 


Pkey: 0x8002 


SID Range: 0x1000117500000000- 

0x100011750000000f 



Figure 3-7. Virtual Fabrics with all SIDs assigned to PSM_MPI Virtual Fabric 

Second, the Distributed SA handles overlaps by taking advantage of the fact that 

Virtual Fabrics have unique numeric indexes. (These IDs can be seen by using 

the command "iba_saquery -o vfinfo".) The Distributed SA will always 

assign a SID to the Virtual Fabric with the lowest ID number, as shown in 

Figure 3-8. This ensures that all copies of the Distributed SA in the InfiniBand 

fabric will make the same decisions about assigning SIDs. However, it also means 

that the behavior of your fabric can be affected by the order you configured the 

virtual fabrics. 


ID: 2 Pkey: 0x8003 


Virtual Fabric Virtual “Default” Fabric “Default” Pkey: 0xffff 

SID Range: Pkey: 0x0-0xffffffffffffffff 

0xffff 

SID Range: 

Virtual 



ID: 1 Pkey: 0x8002 


SID Range: 

0x1000117500000000- 

0x100011750000000f 


Virtual Fabric “PSM_MPI” “Default” 

Pkey: 0x8002 0xffff 


SID Range: 0x1000117500000000- 

0x100011750000000f 


Figure 3-8. Virtual Fabrics with Unique Numeric Indexes 

3-18 D000046-005 B



In Figure 3-8, the Distributed SA assigns all overlapping SIDs to the "PSM_MPI" 

fabric because it has the lowest Index 

NOTE: 

The Distributed SA makes these assignments not because they are right, 

but because they allow the fabric to work even though there are errors. The 

correct solution in these cases is to redefine the fabric so that no node will 

ever be a member of two Virtual Fabrics that service the same SID. 

Distributed SA Configuration File 

The Distributed SA configuration file is 

/etc/sysconfig/iba/qlogic_sa.conf. It has several settings, but normally 

administrators will only need to deal with two or three of them. 

SID 

The SID is the primary configuration setting for the Distributed SA, and it can be 

specified multiple times. When the Distributed SA starts, it loads information about 

the Virtual Fabrics specified by the Fabric Manager and each SID is mapped to a 

single virtual fabric. Multiple SIDs can be mapped to the same virtual fabric. The 

default configuration for the Distributed SA includes all the SIDs defined in the 

default Qlogic FM configuration for use by MPI. 

The SID arguments have a very particular logic that must be understood for 

correct operation. A SID= argument defines one Service ID that is associated with 

a single virtual fabric. In addition, multiple SID= arguments can point to a single 

virtual fabric. For example, a virtual fabric has three sets of SIDs associated with 

it: 0x0a1 through 0x0a3, 0x1a1 through 0x1a3 and 0x2a1 through 0x2a3. You 

would define this as 

SID=0x0a1 

SID=0x0a2 

SID=0x0a3 

SID=0x1a1 

SID=0x1a2 

SID=0x1a3 

SID=0x2a1 

SID=0x2a2 

SID=0x2a3 

NOTE: 

A SID of zero is not supported at this time. Instead, the OPP libraries treat 

zero values as "unspecified". 

D000046-005 B 3-19



ScanFrequency 

Periodically, the Distributed SA will completely re synchronize its database. This 

also occurs if the Fabric Manager is restarted. ScanFrequency defines the 

minimum number of seconds between complete re synchronizations. It defaults to 

600 seconds, or 10 minutes. On very large fabrics, increasing this value can help 

reduce the total amount of SM traffic. For example, to set the interval to 15 

minutes, add this line to the bottom of the qlogic_sa.conf file: 

ScanFrequency=900 

LogFile 

Normally, the Distributed SA logs special events to /var/log/messages. This 

parameter allows you to specify a different destination for the log messages. For 

example, to direct Distributed SA messages to their own log, add this line to the 

bottom of the qlogic_sa.conf file: 

LogFile=/var/log/SAReplica.log 

Dbg 

This parameter controls how much logging the Distributed SA will do. It can be set 

to a number between one and seven, where one indicates no logging and seven 

includes informational and debugging messages. To change the Dbg setting for 

Distributed SA, find the line in qlogic_sa.conf that reads Dbg=5 and change it to a 

different value, between 1 and 7. The value of Dbg changes the amount of logging 

that the Distributed SA generates as follows: 

• Dbg=1 or Dbg=2: Alerts and Critical Errors 

Only errors that will cause the Distributed SA to terminate will be 

reported. 

• Dbg=3: Errors 

Errors will be reported, but nothing else. (Includes Dbg=1 and Dbg=2) 

• Dbg=4: Warnings 

Errors and warnings will be reported. (Includes Dbg=3) 

• Dbg=5: Normal 

Some normal events will be reported along with errors and warnings. 

(Includes Dbg=4) 

• Dbg=6: Informational Messages 

In addition to the normal logging, Distributed SA will report detailed 

information about its status and operation. Generally, this will produce 

too much information for normal use. (Includes Dbg=5) 

3-20 D000046-005 B


MPI over uDAPL 

• Dbg=7: Debugging 

This should only be turned on at the request of QLogic Support. This 

will generate so much information that system operation will be 

impacted. (Includes Dbg=6) 

Other Settings 

The remaining configuration settings for the Distributed SA are generally only 

useful in special circumstances and are not needed in normal operation. The 

sample qlogic_sa.conf configuration file contains a brief description of each. 

MPI over uDAPL 

Intel MPI can be run over uDAPL, which uses InfiniBand Verbs. uDAPL is the user 

mode version of the Direct Access Provider Library (DAPL), and is provided as a 

part of the OFED packages. You will also have to have IPoIB configured. 

The setup for Intel MPI is described in the following steps: 

1. Make sure that DAPL 1.2 (not version 2.0) is installed on every node. In this 

release they are called compat-dapl. They can be installed either with the 

QLogic OFED Host Software package. 

2. Verify that there is a /etc/dat.conf file. The file dat.conf contains a list of 

interface adapters supported by uDAPL service providers. In particular, it 

must contain mapping entries for OpenIB-cma for dapl 1.2.x, in a form 

similar to this (all on one line): 

OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 

"ib0 0" "" 

3. On every node, type the following command (as a root user): 

# modprobe rdma_ucm 

To ensure that the module is loaded when the driver is loaded, add 

RDMA_UCM_LOAD=yes to the /etc/infiniband/openib.conf file. (Note that 

rdma_cm is also used, but it is loaded automatically.) 

4. Bring up an IPoIB interface on every node, for example, ib0. See the 

instructions for configuring IPoIB for more details. 

For more information on using Intel MPI, see Section 5. 

Changing the MTU Size 

The Maximum Transfer Unit (MTU) is set to 4K and enabled in the driver by 

default. To see the current MTU size, and the maximum supported by the adapter, 

type the command: 

$ ibv_devinfo 

D000046-005 B 3-21


Managing the TrueScale Driver 

To change the driver default back to 2K MTU, add this line (as a root user) into 

/etc/modprobe.conf (or /etc/modprobe.conf.local): 

options ib_qib ibmtu=4 

Restart the driver as described in “Managing the TrueScale Driver” on page 3-22. 

NOTE: 

To use 4K MTU, set the switch to have the same 4K default. If you are using 

QLogic switches, the following applies: 

• For the Externally Managed 9024, use 4.2.2.0.3 firmware 

(9024DDR4KMTU_firmware.emfw) for the 9024 EM. This has the 4K MTU 

default, for use on fabrics where 4K MTU is required. If 4K MTU support 

is not required, then use the 4.2.2.0.2 DDR *.emfw file for DDR 

externally-managed switches. Use FastFabric (FF) to load the firmware 

on all the 9024s on the fabric. 

• For the 9000 chassis, use the most recent 9000 code 4.2.4.0.1. The 4K 

MTU support is in 9000 chassis version 4.2.1.0.2 and later. For the 9000 

chassis, when the FastFabric 4.3 (or later) chassis setup tool is used, 

the user is asked to select an MTU. FastFabric can then set that MTU in 

all the 9000 internally managed switches. The change will take effect on 

the next reboot. Alternatively, for the internally managed 9000s, the 

ismChassisSetMtu Command Line Interface (CLI) command can be 

used. This should be executed on every switch and both hemispheres of 

the 9240s. 

• For the 12000 switches, refer to the QLogic FastFabric User Guide for 

externally managed switches, and to the QLogic FastFabric CLI 

Reference Guide for the internally managed switches. 

For reference, see the QLogic FastFabric User Guide and the QLogic 9000 

CLI Reference Guide. Both are available from the QLogic web site. 

For other switches, see the vendors’ documentation. 


The startup script for ib_qib is installed automatically as part of the software 

installation, and normally does not need to be changed. It runs as a system 

service. 

The primary configuration file for the TrueScale driver ib_qib and other modules 

and associated daemons is /etc/infiniband/openib.conf. 

3-22 D000046-005 B



Normally, this configuration file is set up correctly at installation and the drivers are 

loaded automatically during system boot once the RPMs have been installed. 

However, the ib_qib driver has several configuration variables that set reserved 

buffers for the software, define events to create trace records, and set the debug 

level. 

If you are upgrading, your existing configuration files will not be overwritten. 

See the ib_qib man page for more details. 

Configure the TrueScale Driver State 

Use the following commands to check or configure the state. These methods will 

not reboot the system. 

To check the configuration state, use this command. You do not need to be a root 

user: 

$ chkconfig --list openibd 

To enable the driver, use the following command (as a root user): 

# chkconfig openibd on 2345 

To disable the driver on the next system boot, use the following command (as a 

root user): 

# chkconfig openibd off 

NOTE: 

This command does not stop and unload the driver if the driver is already 

loaded. 

Start, Stop, or Restart TrueScale 

Restart the software if you install a new QLogic OFED+ Host Software release, 

change driver options, or do manual testing. 

QLogic recommends using the QLogic IFS Installer TUI to stop, stat and restart 

the IPoIB driver. Refer to the QLogic Fabric Software Installation Guide for more 

information. For using the command line to stop, start, and restart (as a root user) 

the TrueScale support use the following syntex: 

# /etc/init.d/openibd [start | stop | restart] 

WARNING!! 

If QLogic Fabric Manager, or OpenSM is configured and running on the 

node, it must be stopped before using the openibd stop command, and 

may be started after using the openibd start command. 

D000046-005 B 3-23



WARNING!! 

Stopping or restarting openibd terminates any QLogic MPI, VNIC, and SRP 

processes, as well as any OpenFabrics processes that are running at the 

time. QLogic recommends stopping all processes prior to using the openibd 

command. 

This method will not reboot the system. The following set of commands shows 

how to use this script. 

When you need to determine which TrueScale and OpenFabrics modules are 

running, use the following command. You do not need to be a root user. 

$ lsmod | egrep ’ipath_|ib_|rdma_|findex’ 

You can check to see if opensmd is running by using the following command (as a 

root user); if there is no output, opensmd is not configured to run: 

# /sbin/chkconfig --list opensmd | grep -w on 

Unload the Driver/Modules Manually 

You can also unload the driver/modules manually without using 

/etc/init.d/openibd. Use the following series of commands (as a root user): 

# umount /ipathfs 

# fuser -k /dev/ipath* /dev/infiniband/* 

# lsmod | egrep ’îb_|^rdma_|îw_’ | xargs modprobe -r 

TrueScale Driver Filesystem 

The TrueScale driver supplies a filesystem for exporting certain binary statistics to 

user applications. By default, this filesystem is mounted in the /ipathfs directory 

when the TrueScale script is invoked with the start option (e.g. at system 

startup). The filesystem is unmounted when the TrueScale script is invoked with 

the stop option (for example, at system shutdown). 

Here is a sample layout of a system with two cards: 

/ipathfs/0/flash 

/ipathfs/0/port2counters 


/ipathfs/0/portcounter_names 

/ipathfs/0/counter_names 

/ipathfs/0/counters 

/ipathfs/driver_stats_names 

3-24 D000046-005 B


More Information on Configuring and Loading Drivers 

/ipathfs/driver_stats 

/ipathfs/1/flash 



/ipathfs/1/portcounter_names 

/ipathfs/1/counter_names 

/ipathfs/1/counters 

The driver_stats file contains general driver statistics. There is one numbered 

subdirectory per TrueScale device on the system. Each numbered subdirectory 

contains the following per-device files: 

• port1counters 

• port2counters 

• flash 

The driver1counters and driver2counters files contain counters for the 

device, for example, interrupts received, bytes and packets in and out, etc. The 

flash file is an interface for internal diagnostic commands. 

The file counter_names provides the names associated with each of the counters 

in the binary port#counters files, and the file driver_stats_names provides the 

names for the stats in the binary driver_stats files. 

More Information on Configuring and Loading 

Drivers 

See the modprobe(8), modprobe.conf(5), and lsmod(8) man pages for more 

information. Also see the file /usr/share/doc/initscripts-*/sysconfig.txt 

for more general information on configuration files. 

Performance Settings and Management Tips 

The following sections provide suggestions for improving performance and 

simplifying cluster management. Many of these settings will be done by the 

system administrator. User level runtime performance settings are shown in 

“Performance Tuning” on page 4-23. 

D000046-005 B 3-25



Homogeneous Nodes 

To minimize management problems, the compute nodes of the cluster should 

have very similar hardware configurations and identical software installations. A 

mismatch between the TrueScale software versions can also cause problems. Old 

and new libraries must not be run within the same job. It may also be useful to 

distinguish between the TrueScale-specific drivers and those that are associated 

with kernel.org, OpenFabrics, or are distribution-built. The most useful tools are: 

• ident (see “ident” on page I-9) 

• ipathbug-helper (see “ipathbug-helper” on page I-10) 

• ipath_checkout (see “ipath_checkout” on page I-10) 

• ipath_control (see “ipath_control” on page I-12) 

• mpirun (see “mpirun” on page I-15) 

• rpm (see “rpm” on page I-16) 

• strings (see “strings” on page I-17) 

NOTE: 

Run these tools to gather information before reporting problems and 

requesting support. 

Adapter and Other Settings 

The following adapter and other settings can be adjusted for better performance. 

• Use an InfiniBand MTU of 4096 bytes instead of 2048 bytes, if available, 

with the QLE7340, QLE7342, QLE7240, QLE7280, and QLE7140. 4K 

MTU is enabled in the TrueScale driver by default. To change this setting for 

the driver, see “Changing the MTU Size” on page 3-21. 

• Use a PCIe Max Read Request size of at least 512 bytes with the 

QLE7340, QLE7342, QLE7240 and QLE7280. QLE7240 and QLE7280 

adapters can support sizes from 128 bytes to 4096 byte in powers of two. 

This value is typically set by the BIOS. 

For QLogic 7300 and 7200 Series Host Channel Adapters, to improve peak 

IB bandwidth on Nehalem and Harpertown CPU systems, set PCIe 

parameters as follows: 

 

 

Set PCIe Max Read Request to 4096 bytes 

Set PCIe Max Payload to 256 bytes by, as root, adding the following 

line: 

options ib_qib pcie_caps=0x51 

to the /etc/modprobe.conf file: 

The above should be sufficient on Intel Nehalem CPUs or newer. 

3-26 D000046-005 B



On Intel Harpertown CPUs, it may be beneficial to add a 

pcie_coalesce=1 parameter to this line. 

On AMD CPUs (PCIe Gen1) no ib_qib parameter changes are 

recommended. 

Alternatively, these PCIe parameters can also be set in the BIOS on some 

systems. 

• Use PCIe Max Payload size of 256, where available, with the QLE7340, 

QLE7342, QLE7240 and QLE7280. The QLE7240 and QLE7280 adapters 

can support 128, 256, or 512 bytes. This value is typically set by the BIOS 

as the minimum value supported both by the PCIe card and the PCIe root 

complex. 

• Make sure that write combining is enabled. The x86 Page Attribute Table 

(PAT) mechanism that allocates Write Combining (WC) mappings for the 

PIO buffers has been added and is now the default. If PAT is unavailable or 

PAT initialization fails for some reason, the code will generate a message in 

the log and fall back to the MTRR mechanism. See Appendix H Write 

Combining for more information. 

• Check the PCIe bus width. If slots have a smaller electrical width than 

mechanical width, lower than expected performance may occur. Use this 

command to check PCIe Bus width: 

$ ipath_control -iv 

This command also shows the link speed. 

• Experiment with non-default CPU affinity while running 

single-process-per-node latency or bandwidth benchmarks. Latency 

may be slightly lower when using different CPUs (cores) from the default. On 

some chipsets, bandwidth may be higher when run from a non-default CPU 

or core. See “Performance Tuning” on page 4-23 for more information on 

using taskset with QLogic MPI. With another MPI, look at its documentation 

to see how to force a benchmark to run with a different CPU affinity than the 

default. With OFED micro benchmarks such as from the qperf or perftest 

suites, taskset will work for setting CPU affinity. 

Turn C-state Off to improve MPI latency (ping-pong) benchmarks on 

Nehalem systems. In the BIOS, look for advanced CPU settings, and set the 

C-State parameter to "disable." 

D000046-005 B 3-27



• Allocate all chip resources to a single port on a dual port card. The 

singleport parameter, when set to a non-zero value at driver load, will cause 

dual port Truescale cards to act as single port cards, with only infiniband port 

1 enabled. The board identification string is not affected (that is, a QLE7342 

with singleport set will still be identified as a QLE7342, not a QLE7340). The 

default value of this parameter is 0, however it may be set to 1 during the 

IFS installation process 

By default, in the QLogic Installation TUI will configure a dual-port card so all 

chip resources are directed to port1 to maximize performance. This also 

saves power by not enabling the second port. If you want to utilize both ports 

specify when asked about single port to use dual ports. 

Remove Unneeded Services 

The cluster administrator can enhance application performance by minimizing the 

set of system services running on the compute nodes. Since these are presumed 

to be specialized computing appliances, they do not need many of the service 

daemons normally running on a general Linux computer. 

Following are several groups constituting a minimal necessary set of services. 

These are all services controlled by chkconfig. To see the list of services that are 

enabled, use the command: 

$ /sbin/chkconfig --list | grep -w on 

Basic network services are: 

• network 

• ntpd 

• syslog 

• xinetd 

• sshd 

For system housekeeping, use: 

• anacron 

• atd 

• crond 

If you are using Network File System (NFS) or yellow pages (yp) passwords: 

• rpcidmapd 

• ypbind 

• portmap 

• nfs 

• nfslock 

• autofs 

To watch for disk problems, use: 

3-28 D000046-005 B


Host Environment Setup for MPI 

• smartd 

• readahead 

The service comprising the TrueScale driver and SMA is: 

• openibd 

Other services may be required by your batch queuing system or user community. 

If your system is running the daemon irqbalance, QLogic recommends turning it 

off. Disabling irqbalance will enable more consistent performance with programs 

that use interrupts. Use this command: 

# /sbin/chkconfig irqbalance off 

See “Erratic Performance” on page F-9 for more information. 

Disable Powersaving Features 

Hyper-Threading 

If you are running benchmarks or large numbers of short jobs, it is beneficial to 

disable the powersaving features, since these features may be slow to respond to 

changes in system load. 

For RHEL4 and RHEL5, run this command as a root user: 

# /sbin/chkconfig --level 12345 cpuspeed off 

For SLES 10 and SLES 11, run this command as a root user: 

# /sbin/chkconfig --level 12345 powersaved off 

After running either of these commands, reboot the system for the changes to 

take effect. 

If you are using Intel NetBurst ® Processors that support Hyper-Threading, QLogic 

recommends turning off Hyper-Threading in the BIOS, which will provide more 

consistent performance. You can check and adjust this setting using the BIOS 

Setup utility. For specific instructions, follow the hardware documentation that 

came with your system. 


After the QLogic OFED+ Host software and the GNU (GCC) compilers have been 

installed on all the nodes, the host environment can be set up for running MPI 

programs. 

D000046-005 B 3-29



Configuring for ssh 

Running MPI programs with the command mpirun on an TrueScale cluster 

depends, by default, on secure shell ssh to launch node programs on the nodes. 

In QLogic MPI, mpirun uses the secure shell command ssh to start instances of 

the given MPI program on the remote compute nodes without the need for 

interactive password entry on every node. 

To use ssh, you must have generated Rivest, Shamir, Adleman (RSA) or Digital 

Signal Algorithm (DSA) keys, public and private. The public keys must be 

distributed and stored on all the compute nodes so that connections to the remote 

machines can be established without supplying a password. 

You or your administrator must set up the ssh keys and associated files on the 

cluster. There are two methods for setting up ssh on your cluster. The first 

method, the shosts.equiv mechanism, is typically set up by the cluster 

administrator. The second method, using ssh-agent, is more easily 

accomplished by an individual user. 

NOTE: 

• rsh can be used instead of ssh. To use rsh, set the environment 

variable MPI_SHELL=rsh. See “Environment Variables” on page 4-20 for 

information on setting environment variables. Also see “Shell Options” 

on page A-6 for information on setting shell options in mpirun. 

• rsh has a limit on the number of concurrent connections it can have, 

typically 255, which may limit its use on larger clusters. 

Configuring ssh and sshd Using shosts.equiv 

This section describes how the cluster administrator can set up ssh and sshd 

through the shosts.equiv mechanism. This method is recommended, provided 

that your cluster is behind a firewall and accessible only to trusted users. 

“Configuring for ssh Using ssh-agent” on page 3-32 shows how an individual user 

can accomplish the same thing using ssh-agent. 

The example in this section assumes the following: 

• Both the cluster nodes and the front end system are running the openssh 

package as distributed in current Linux systems. 

• All cluster end users have accounts with the same account name on the 

front end and on each node, by using Network Information Service (NIS) or 

another means of distributing the password file. 

• The front end used in this example is called ip-fe. 

3-30 D000046-005 B



• Root or superuser access is required on ip-fe and on each node to 

configure ssh. 

• ssh, including the host’s key, has already been configured on the system 

ip-fe. See the sshd and ssh-keygen man pages for more information. 

To use shosts.equiv to configure ssg and sshd: 

1. On the system ip-fe (the front end node), change the 

/etc/ssh/ssh_config file to allow host-based authentication. Specifically, 

this file must contain the following four lines, all set to yes. If the lines are 

already there but commented out (with an initial #), remove the #. 

RhostsAuthentication yes 

RhostsRSAAuthentication yes 

HostbasedAuthentication yes 

EnableSSHKeysign yes 

2. On each of the TrueScale node systems, create or edit the file 

/etc/ssh/shosts.equiv, adding the name of the front end system. Add the 

line: 

ip-fe 

Change the file to mode 600 when you are finished editing. 

3. On each of the TrueScale node systems, create or edit the file 

/etc/ssh/ssh_known_hosts. You will need to copy the contents of the file 

/etc/ssh/ssh_host_dsa_key.pub from ip-fe to this file (as a single line), 

and then edit that line to insert ip-fe ssh-dss at the beginning of the line. 

This is very similar to the standard known_hosts file for ssh. An example 

line might look like this (displayed as multiple lines, but a single line in the 

file): 

ip-fe ssh-dss 

AAzAB3NzaC1kc3MAAACBAPoyES6+Akk+z3RfCkEHCkmYuYzqL2+1nwo4LeTVW 

pCD1QsvrYRmpsfwpzYLXiSJdZSA8hfePWmMfrkvAAk4ueN8L3ZT4QfCTwqvHV 

vSctpibf8n 

aUmzloovBndOX9TIHyP/Ljfzzep4wL17+5hr1AHXldzrmgeEKp6ect1wxAAAA 

FQDR56dAKFA4WgAiRmUJailtLFp8swAAAIBB1yrhF5P0jO+vpSnZrvrHa0Ok+ 

Y9apeJp3sessee30NlqKbJqWj5DOoRejr2VfTxZROf8LKuOY8tD6I59I0vlcQ 

812E5iw1GCZfNefBmWbegWVKFwGlNbqBnZK7kDRLSOKQtuhYbGPcrVlSjuVps 

fWEju64FTqKEetA8l8QEgAAAIBNtPDDwdmXRvDyc0gvAm6lPOIsRLmgmdgKXT 

GOZUZ0zwxSL7GP1nEyFk9wAxCrXv3xPKxQaezQKs+KL95FouJvJ4qrSxxHdd1 

NYNR0DavEBVQgCaspgWvWQ8cL 

0aUQmTbggLrtD9zETVU5PCgRlQL6I3Y5sCCHuO7/UvTH9nneCg== 

Change the file to mode 600 when you are finished editing. 

D000046-005 B 3-31



4. On each node, the system file /etc/ssh/sshd_config must be edited, so 

that the following four lines are uncommented (no # at the start of the line) 

and set to yes. (These lines are usually there, but are commented out and 

set to no by default.) 

RhostsAuthentication yes 

RhostsRSAAuthentication yes 

HostbasedAuthentication yes 

PAMAuthenticationViaKbdInt yes 

5. After creating or editing the three files in Steps 2, 3, and 4, sshd must be 

restarted on each system. If you are already logged in via ssh (or any other 

user is logged in via ssh), their sessions or programs will be terminated, so 

restart only on idle nodes. Type the following (as root) to notify sshd to use 

the new configuration files: 

# killall -HUP sshd 

NOTE: 

This command terminates all ssh sessions into that system. Run from 

the console, or have a way to log into the console in case of any 

problem. 

At this point, any end user should be able to login to the ip-fe front end system 

and use ssh to login to any TrueScale node without being prompted for a 

password or pass phrase. 

Configuring for ssh Using ssh-agent 

The ssh-agent, a daemon that caches decrypted private keys, can be used to 

store the keys. Use ssh-add to add your private keys to ssh-agent’s cache. 

When ssh establishes a new connection, it communicates with ssh-agent to 

acquire these keys, rather than prompting you for a passphrase. 

The process is described in the following steps: 

1. Create a key pair. Use the default file name, and be sure to enter a 

passphrase. 

$ ssh-keygen -t rsa 

2. Enter a passphrase for your key pair when prompted. Note that the key 

agent does not survive X11 logout or system reboot: 

$ ssh-add 

3. The following command tells ssh that your key pair should let you in: 

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 

3-32 D000046-005 B



Edit the ~/.ssh/config file so that it reads like this: 

Host* 

ForwardAgent yes 

ForwardX11 yes 

CheckHostIP no 

StrictHostKeyChecking no 

This file forwards the key agent requests back to your desktop. When you 

log into a front end node, you can use ssh to compute nodes without 

passwords. 

4. Follow your administrator’s cluster policy for setting up ssh-agent on the 

machine where you will be running ssh commands. Alternatively, you can 

start the ssh-agent by adding the following line to your ~/.bash_profile 

(or equivalent in another shell): 

eval ‘ssh-agent‘ 

Use back quotes rather than single quotes. Programs started in your login 

shell can then locate the ssh-agent and query it for keys. 

5. Finally, test by logging into the front end node, and from the front end node 

to a compute node, as follows: 

$ ssh frontend_node_name 

$ ssh compute_node_name 

For more information, see the man pages for ssh(1), ssh-keygen(1), 

ssh-add(1), and ssh-agent(1). 

Process Limitation with ssh 

Process limitation with ssh is primarily an issue when using the mpirun option 

-distributed=off. The default setting is now -distributed=on; therefore, in 

most cases, ssh process limitations will not be encountered. This limitation for the 

-distributed=off case is described in the following paragraph. See “Process 

Limitation with ssh” on page F-19 for an example of an error message associated 

with this limitation. 

MPI jobs that use more than 10 processes per node may encounter an ssh 

throttling mechanism that limits the amount of concurrent per-node connections 

to 10. If you need to use more processes, you or your system administrator must 

increase the value of MaxStartups in your /etc/ssh/sshd_config file. 

D000046-005 B 3-33


Checking Cluster and Software Status 


ipath_control 

InfiniBand status, link speed, and PCIe bus width can be checked by running the 

program ipath_control. Sample usage and output are as follows: 

$ ipath_control -iv 

$Id: QLogic OFED Release 1.4.2 $ $Date: 2009-03-10-10:15 $ 

0: Version: ChipABI 2.0, InfiniPath_QLE7280, InfiniPath1 5.2, 

PCI 2, SW Compat 2 

0: Status: 0xe1 Initted Present IB_link_up IB_configured 

0: LID=0x1f MLID=0xc042 GUID=00:11:75:00:00:ff:89:a6 Serial: 

AIB0810A30297 

0: HRTBT:Auto RX_polarity_invert:Auto RX_lane_reversal: Auto 

0: LinkWidth:4X of 1X|4X Speed:DDR of SDR|DDR 

0: LocalBus: PCIe,2500MHz,x16 

3-34 D000046-005 B



iba_opp_query 

iba_opp_query is used to check the operation of the Distributed SA. You can run it 

from any node to verify that the replica on that node is working correctly. See 

“iba_opp_query” on page I-4 for detailed usage information. 

# iba_opp_query --slid 0x31 --dlid 0x75 --sid 0x107 

Query Parameters: 

resv1 

0x0000000000000107 

dgid :: 

sgid :: 

dlid 

0x75 

slid 

0x31 

hop 

0x0 

flow 

0x0 

tclass 

0x0 

num_path 

0x0 

pkey 

0x0 

qos_class 

0x0 

sl 

0x0 

mtu 

0x0 

rate 

0x0 

pkt_life 

0x0 

preference 

0x0 

resv2 

0x0 

resv3 

0x0 

Using HCA qib0 

Result: 

resv1 

0x0000000000000107 

dgid 

fe80::11:7500:79:e54a 

sgid 

fe80::11:7500:79:e416 

dlid 

0x75 

slid 

0x31 

hop 

0x0 

flow 

0x0 

tclass 

0x0 

num_path 

0x0 

pkey 

0xffff 

qos_class 

0x0 

sl 

0x1 

mtu 

0x4 

D000046-005 B 3-35



rate 

pkt_life 

preference 

resv2 

resv3 

0x6 

0x10 

0x0 

0x0 

0x0 

ibstatus 

Another useful program is ibstatus. Sample usage and output are as follows: 

$ ibstatus 

Infiniband device ’qib0’ port 1 status: 

default gid: fe80:0000:0000:0000:0011:7500:00ff:89a6 

base lid: 0x1f 

sm lid: 

0x1 

state: 

4: ACTIVE 

phys state: 5: LinkUp 

rate: 

20 Gb/sec (4X DDR) 

ibv_devinfo 

ibv_devinfo queries RDMA devices. Use the -v option to see more information. 

Sample usage: 

$ ibv_devinfo 

hca_id: qib0 

fw_ver: 0.0.0 

node_guid: 

0011:7500:00ff:89a6 

sys_image_guid: 

0011:7500:00ff:89a6 

vendor_id: 

0x1175 

vendor_part_id: 29216 

hw_ver: 

0x2 

board_id: 

InfiniPath_QLE7280 

phys_port_cnt: 1 

port: 1 

state: PORT_ACTIVE (4) 

max_mtu: 4096 (5) 

active_mtu: 4096 (5) 

sm_lid: 1 

port_lid: 31 

port_lmc: 

0x00 

3-36 D000046-005 B



ipath_checkout 

ipath_checkout is a bash script that verifies that the installation is correct and 

that all the nodes of the network are functioning and mutually connected by the 

TrueScale fabric. It must be run on a front end node, and requires specification of 

a nodefile. For example: 

$ ipath_checkout [options] nodefile 

The nodefile lists the hostnames of the nodes of the cluster, one hostname per 

line. The format of nodefile is as follows: 

hostname1 

hostname2 

... 

For more information on these programs, see “ipath_control” on page I-12, 

“ibstatus” on page I-7, and “ipath_checkout” on page I-10. 

D000046-005 B 3-37



3-38 D000046-005 B

4 Running QLogic MPI on 

QLogic Adapters 

Introduction 

QLogic MPI 

This section provides information on using the QLogic Message-Passing Interface 

(MPI). Examples are provided for setting up the user environment, and for 

compiling and running MPI programs. 

The MPI standard is a message-passing library or collection of routines used in 

distributed-memory parallel programming. It is used in data exchange and task 

synchronization between processes. The goal of MPI is to provide portability and 

efficient implementation across different platforms and architectures. 

QLogic’s implementation of the MPI standard is derived from the MPICH 

reference implementation version 1.2.7. The QLogic MPI (TrueScale) libraries 

have been highly tuned for the QLogic interconnect, and will not run over other 

interconnects. 

QLogic MPI is an implementation of the original MPI 1.2 standard. The MPI-2 

standard provides several enhancements of the original standard. Of the MPI-2 

features, QLogic MPI includes only the MPI-IO features implemented in ROMIO 

version 126 and the generalized MPI_All to allow communication exchange. 

The QLogic MPI implementation in this release supports hybrid MPI/OpenMP and 

other multi-threaded programs, as long as only one thread uses MPI. For more 

information, see “QLogic MPI and Hybrid MPI/OpenMP Applications” on 

page 4-25. 

D000046-005 B 4-1

4–Running QLogic MPI on QLogic Adapters 

Introduction 

PSM 

The PSM TrueScale Messaging API, or PSM API, is QLogic's low-level user-level 

communications interface for the TrueScale family of products. Other than using 

some environment variables with the PSM prefix, MPI users typically need not 

interact directly with PSM. The PSM environment variables apply to other MPI 

implementations as long as the environment with the PSM variables is correctly 

forwarded. See “Environment Variables” on page 4-20 for a summary of the 

commonly used environment variables. 

For more information on PSM, email QLogic at support@qlogic.com. 

Other MPIs 

In addition to QLogic MPI, other high-performance MPIs such as HP-MPI version 

2.3, Open MPI version 1.4, Ohio State University MVAPICH version 1.2, 

MVAPICH2 version 1.4, and Scali (Platform) MPI, have been ported to the PSM 

interface. 

Open MPI, MVAPICH, HP-MPI, and Scali also run over InfiniBand Verbs (the 

Open Fabrics Alliance API that provides support for user-level upper-layer 

protocols like MPI). Intel MPI, although not ported to the PSM interface, is 

supported over uDAPL, which uses InfiniBand Verbs. For more information, see 

Section 5 Using Other MPIs. 

Linux File I/O in MPI Programs 

MPI node programs are Linux programs that can execute file I/O operations to 

local or remote files in the usual ways through APIs of the language in use. 

Remote files are accessed via a network file system, typically NFS. Parallel 

programs usually need to have some data in files to be shared by all of the 

processes of an MPI job. Node programs can also use non-shared, node-specific 

files, such as for scratch storage for intermediate results or for a node’s share of a 

distributed database. 

There are different ways of handling file I/O of shared data in parallel 

programming. You may have one process, typically on the front end node or on a 

file server that is the only process to touch the shared files, and passes data to 

and from the other processes via MPI messages. Alternately, the shared data files 

can be accessed directly by each node program. In this case, the shared files are 

available through some network file support, such as NFS. Also, in this case, the 

application programmer is responsible for ensuring file consistency, either through 

proper use of file locking mechanisms offered by the operating system and the 

programming language, such as fcntl in C, or by using MPI synchronization 

operations. 

4-2 D000046-005 B


Getting Started with MPI 

MPI-IO with ROMIO 

MPI-IO is the part of the MPI-2 standard, supporting collective and parallel file I/O 

operations. One advantage of using MPI-IO is that it can take care of managing 

file locks when file data is shared among nodes. 

QLogic MPI includes ROMIO version 1.2.6. ROMIO is a high-performance, 

portable implementation of MPI-IO from Argonne National Laboratory. ROMIO 

includes everything defined in the MPI-2 I/O chapter of the MPI-2 standard except 

support for file interoperability and user-defined error handlers for files. Of the 

MPI-2 features, QLogic MPI includes only the MPI-IO features implemented in 

ROMIO version 126 and the generalized MPI_All to allow communication 

exchange. See the ROMIO documentation at http://www.mcs.anl.gov/romio for 

details. 

NFS, PanFS, and local (UFS) support is enabled. 


Copy Examples 

This section shows how to compile and run some simple example programs that 

are included in the QLogic OFED+ Host software product. Compiling and running 

these examples enables you to verify that QLogic MPI and its components have 

been properly installed on the cluster. See “QLogic MPI Troubleshooting” on 

page F-11 if you have problems compiling or running these examples. 

These examples assume that your cluster’s policy allows you to use the mpirun 

script directly, without having to submit the job to a batch queuing system. 

Start by copying the examples to your working directory: 

$ cp /usr/mpi/qlogic/share/mpich/examples/basic/* . 

or 

$ cp /usr/share/mpich/examples/basic/* . 

Create the mpihosts File 

Next, create an MPI hosts file in the same working directory. It contains the host 

names of the nodes in your cluster that run the examples, with one host name per 

line. Name this file mpihosts. The contents can be in the following format: 

hostname1 

hostname2 

... 

More details on the mpihosts file can be found in “mpihosts File Details” on 

page 4-15. 

D000046-005 B 4-3



Compile and Run an Example C Program 

In this step you will compile and run your MPI program. 

QLogic MPI uses some shell scripts to find the appropriate include files and 

libraries for each supported language. Use the script mpicc to compile an MPI 

program in C and the script mpirun to execute the file. 

The supplied example program cpi.c computes an approximation to pi. First, 

compile it to an executable named cpi. For example: 

$ mpicc -o cpi cpi.c 

By default, mpicc runs the GNU gcc compiler, and is used for both compiling and 

linking, the same function as the gcc command. 

NOTE: 

For information on using other compilers, see “To Use Another Compiler” on 

page 4-9. 

Then, run the program with several different specifications for the number of 

processes: 

$ mpirun -np 2 -m mpihosts ./cpi 

Process 0 on hostname1 


pi is approximately 3.1416009869231241, 

Error is 0.0000083333333309 

wall clock time = 0.000149 

In this example, ./cpi designates the executable of the example program in the 

working directory. The -np parameter to mpirun defines the number of 

processes to be used in the parallel computation. Here is an example with four 

processes, using the same two hosts in the mpihosts file: 

$ mpirun -np 4 -m mpihosts ./cpi 





pi is approximately 3.1416009869231249, 

Error is 0.0000083333333318 

wall clock time = 0.000603 

4-4 D000046-005 B



Generally, mpirun tries to distribute the specified number of processes evenly 

among the nodes listed in the mpihosts file. However, if the number of 

processes exceeds the number of nodes listed in the mpihosts file, then some 

nodes will be assigned more than one instance of the program. 

When you run the program several times with the same value of the -np 

parameter, the output lines may display in different orders. This is because they 

are issued by independent asynchronous processes, so their order is 

non-deterministic. 

Details on other ways of specifying the mpihosts file are provided in “mpihosts 

File Details” on page 4-15. 

More information on the mpirun options are in “Using mpirun” on page 4-16 and 

Appendix A mpirun Options Summary. “Process Allocation” on page 4-11 

explains how processes are allocated by using hardware and software contexts. 

Examples Using Other Programming Languages 

This section gives similar examples for computing pi for Fortran 77 and 

Fortran 90. Fortran 95 usage is similar to Fortran 90. The C++ example uses the 

traditional “Hello, World” program. All programs are located in the same directory. 

fpi.f is a Fortran 77 program that computes pi in a way similar to cpi.c. 

Compile and link, and run it as follows: 

$ mpif77 -o fpi fpi.f 

$ mpirun -np 2 -m mpihosts ./fpi 

pi3f90.f90 is a Fortran 90 program that does the same computation. Compile 

and link, and run it as follows: 

$ mpif90 -o pi3f90 pi3f90.f90 

$ mpirun -np 2 -m mpihosts ./pi3f90 

The C++ program hello++.cc is a parallel processing version of the traditional 

“Hello, World” program. Notice that this version makes use of the external C 

bindings of the MPI functions if the C++ bindings are not present. 

D000046-005 B 4-5


QLogic MPI Details 

Compile and run it as follows: 

$ mpicxx -o hello hello++.cc 

$ mpirun -np 10 -m mpihosts ./hello 

Hello World! I am 9 of 10 










Each of the scripts invokes the GNU compiler for the respective language and the 

linker. See “To Use Another Compiler” on page 4-9 for an example of how to use 

other compilers. The use of mpirun is the same for programs in all languages. 


The following sections provide more details on the use of QLogic MPI. These 

sections assume that you are familiar with standard MPI. For more information, 

see the references in “References for MPI” on page J-1. This implementation 

includes the man pages from the MPICH implementation for the numerous MPI 

functions. 

4-6 D000046-005 B



Use Wrapper Scripts for Compiling and Linking 

The scripts in Table 4-1 invoke the compiler and linker for programs in each of the 

respective languages, and take care of referring to the correct include files and 

libraries in each case. 

Table 4-1. QLogic MPI Wrapper Scripts 

Wrapper Script Name 

Language 

mpicc 

mpicxx 

C 

C++ 

mpif77 Fortran 77 



On x86_64, these scripts (by default) call the GNU compiler and linker. To use 

other compilers, see “To Use Another Compiler” on page 4-9. 

These scripts all provide the command line options listed in Table 4-2. 

Table 4-2. Command Line Options for Scripts 

Command 

Meaning 

-help 

-show 

-echo 

-compile_info 

-link_info 

Provides help 

Lists each of the compiling and linking commands that would be 

called without actually calling them 

Gets verbose output of all the commands in the script 

Shows how to compile a program 

Shows how to link a program 

In addition, each of these scripts allow a command line option for specifying a 

different compiler/linker as an alternative to the GNU Compiler Collection (GCC). 

For more information, see “To Use Another Compiler” on page 4-9. 

Most other command line options are passed on to the invoked compiler and 

linker. The GNU compiler and alternative compilers all accept numerous 

command line options. See the GCC compiler documentation and the man pages 

for gcc and gfortran for complete information on available options. See the 

corresponding documentation for any other compiler/linker you may call for its 

options. Man pages for mpif90(1), mpif77(1), mpicc(1), and mpiCC(1) are 

available. 

D000046-005 B 4-7



Configuring MPI Programs for QLogic MPI 

When configuring an MPI program (generating header files and/or Makefiles) for 

QLogic MPI, you usually need to specify mpicc, mpicxx, and so on as the 

compiler, rather than gcc, g++, etc. 

Specifying the compiler is typically done with commands similar to the following, 

assuming that you are using sh or bash as the shell: 

$ export CC=mpicc 

$ export CXX=mpicxx 

$ export F77=mpif77 



The shell variables will vary with the program being configured, but these 

examples show frequently used variable names. If you use csh, use commands 

similar to the following: 

$ setenv CC mpicc 

You may need to pass arguments to configure directly, for example: 

$ ./configure -cc=mpicc -fc=mpif77 -c++=mpicxx 

-c++linker=mpicxx 

You may also need to edit a Makefile to achieve this result, adding lines similar to: 

CC=mpicc 

F77=mpif77 

F90=mpif90 

F95=mpif95 

CXX=mpicxx 

In some cases, the configuration process may specify the linker. QLogic 

recommends that the linker be specified as mpicc, mpif90, etc. in these cases. 

This specification automatically includes the correct flags and libraries, rather than 

trying to configure to pass the flags and libraries explicitly. For example: 

LD=mpicc 

LD=mpif90 

These scripts pass appropriate options to the various compiler passes to include 

header files, required libraries, etc. While the same effect can be achieved by 

passing the arguments explicitly as flags, the required arguments may vary from 

release to release, so it is good practice to use the provided scripts. 

4-8 D000046-005 B



To Use Another Compiler 

QLogic MPI and all other MPIs that run on TrueScale, support a number of 

compilers, in addition to the default GNU Compiler Collection (GCC, including gcc, 

g++ and gfortran, but g77 is not supported) versions 3.3 and later. These include 

the PathScale Compiler Suite 3.0, 3.1, and 3.2; PGI 5.2, 6.0, 7.1, 8.0, and 9.0; 

and Intel 9.x, 10.1, and 11.x. 

NOTE: 

The PathScale compiler suite is not supported on the RHEL 4 U8 or the 

SLES 11 distribution. 

These compilers can be invoked on the command line by passing options to the 

wrapper scripts. Command line options override environment variables, if set. 

Tables 4-3, 4-4, and 4-5 show the options for each of the compilers. 

In each case, ..... stands for the remaining options to the mpicxx script, the 

options to the compiler in question, and the names of the files that it operates. 

Table 4-3. Intel 

Compiler 

Command 

C $ mpicc -cc=icc ..... 

C++ 

$ mpicc -CC=icpc 

Fortran 77 $ mpif77 -fc=ifort ..... 

Fortran 90/95 $ mpif90 -f90=ifort ..... 

$ mpif95 -f95=ifort ..... 

Table 4-4. Portland Group (PGI) 

Compiler 

Command 

C mpicc -cc=pgcc ..... 

C++ 

mpicc -CC=pgCC 

Fortran 77 mpif77 -fc=pgf77 ..... 

Fortran 90/95 mpif90 -f90=pgf90 ..... 

mpif95 -f95=pgf95 ..... 

D000046-005 B 4-9



Table 4-5. PathScale Compiler Suite 

Compiler 

Command 

C mpicc -cc=pathcc ..... 

C++ mpicc -CC=pathCC ..... 

Fortran 77 mpif77 -fc=pathf95 ..... 

Fortran 90/95 mpif90 -f90=pathf95 ..... 

mpif90 -f95=pathf95 ..... 

NOTE: pathf95 invokes the Fortran 77, 

Fortran 90, and Fortran 95 compilers. 

Also, use mpif77, mpif90, or mpif95 for linking; otherwise, .true. may have 

the wrong value. 

If you are not using the provided scripts for linking, link a sample program using 

the -show option as a test (without the actual build) to see what libraries to add to 

your link line. Some examples of the using the PGI compilers follow. 

For Fortran 90 programs: 

$ mpif90 -f90=pgf90 -show pi3f90.f90 -o pi3f90 

pgf90 -I/usr/include/mpich/pgi5/x86_64 -c -I/usr/include 

pi3f90.f90 -c 

pgf90 pi3f90.o -o pi3f90 -lmpichf90 -lmpich -lmpichabiglue_pgi5 

Fortran 95 programs will be similar to the above. 

For C programs: 

$ mpicc -cc=pgcc -show cpi.c 

pgcc -c cpi.c 

pgcc cpi.o -lmpich -lpgftnrtl -lmpichabiglue_pgi5 

Compiler and Linker Variables 

When you use environment variables (e.g., $MPICH_CC) to select the compiler 

mpicc (and others) will use, the scripts will also set the matching linker variable 

(for example, $MPICH_CLINKER), if it is not already set. When both the 

environment variable and command line options are used (-cc=gcc), the 

command line variable is used. 

When both the compiler and linker variables are set, and they do not match for the 

compiler you are using, the MPI program may fail to link; or, if it links, it may not 

execute correctly. For a sample error message, see “Compiler/Linker Mismatch” 

on page F-14. 

4-10 D000046-005 B



Process Allocation 

Normally MPI jobs are run with each node program (process) being associated 

with a dedicated QLogic infiniband adapter hardware context that is mapped to a 

CPU. 

If the number of node programs is greater than the available number of hardware 

contexts, software context sharing increases the number of node programs that 

can be run. Each adapter supports four software contexts per hardware context, 

so up to four node programs (from the same MPI job) can share that hardware 

context. There is a small additional overhead for each shared context. 

Table 4-6 shows the maximum number of contexts available for each adapter. 

Table 4-6. Available Hardware and Software Contexts 

Adapter 

Available Hardware 

Contexts (same as number 

of supported CPUs) 

Available Contexts when 

Software Context Sharing is 

Enabled 

QLE7140 4 16 

QLE7240/ 

QLE7280 

QLE7342/ 

QLE7340 

16 64 

16 64 

The default hardware context/CPU mappings can be changed on the TrueScale 

DDR and QDR InfiniBand Adapters (QLE72x0 and QLE734x). See “TrueScale 

Hardware Contexts on the DDR and QDR InfiniBand Adapters” on page 4-12 for 

more details. 

Context sharing is enabled by default. How the system behaves when context 

sharing is enabled or disabled is described in “Enabling and Disabling Software 

Context Sharing” on page 4-13. 

When running a job in a batch system environment where multiple jobs may be 

running simultaneously, it is useful to restrict the number of TrueScale contexts 

that are made available on each node of an MPI. See “Restricting TrueScale 

Hardware Contexts in a Batch Environment” on page 4-13. 

Errors that may occur with context sharing are covered in “Context Sharing Error 

Messages” on page 4-14. 

There are multiple ways of specifying how processes are allocated. You can use 

the mpihosts file, the -np and -ppn options with mpirun, and the 

MPI_NPROCS and PSM_SHAREDCONTEXTS_MAX environment variables. How 

these all are set are covered later in this document. 

D000046-005 B 4-11



TrueScale Hardware Contexts on the DDR and QDR 

InfiniBand Adapters 

On the QLE7240 and QLE7280 DDR adapters, adapter receive resources are 

statically partitioned across the TrueScale contexts according to the number of 

TrueScale contexts enabled. The following defaults are automatically set 

according to the number of online CPUs in the node: 

For four or less CPUs: 5 (4 + 1 for kernel) 

For five to eight CPUs: 9 (8 + 1 for kernel) 

For nine or more CPUs: 17 (16 + 1 for kernel) 

On the QLE7340 and QLE7342 QDR adapters, adapter receive resources are 

statically partitioned across the TrueScale contexts according to the number of 

TrueScale contexts enabled. The following defaults are automatically set 

according to the number of online CPUs in the node: 

For four or less CPUs: 6 (4 + 2) 

For five to eight CPUs: 10 (8 + 2) 

For nine or more CPUs: 18 (16 + 2) 

The one additional context on QDR adapters are to support the kernel on each 

port. 

Performance can be improved in some cases by disabling TrueScale hardware 

contexts when they are not required so that the resources can be partitioned more 

effectively. 

To disable this behavior, explicitly configure for the number you want to use with 

the cfgctxts module parameter in the file /etc/modprobe.conf (or 

/etc/modprobe.conf.local on SLES). 

The maximum that can be set is 17 on DDR InfiniBand Adapters and 18 on QDR 

InfiniBand Adapters. 

The driver must be restarted if this default is changed. See “Managing the 

TrueScale Driver” on page 3-22. 

4-12 D000046-005 B



NOTE: 

In rare cases, setting contexts automatically on DDR and QDR InfiniBand 

Adapters can lead to sub-optimal performance where one or more 

TrueScale hardware contexts have been disabled and a job is run that 

requires software context sharing. Since the algorithm ensures that there is 

at least one TrueScale context per online CPU, this case occurs only if the 

CPUs are over-subscribed with processes (which is not normally 

recommended). In this case, it is best to override the default to use as many 

TrueScale contexts as are available, which minimizes the amount of 

software context sharing required. 

Enabling and Disabling Software Context Sharing 

By default, context sharing is enabled; it can also be specifically disabled. 

Context Sharing Enabled: The MPI library provides PSM the local process 

layout so that TrueScale contexts available on each node can be shared if 

necessary; for example, when running more node programs than contexts. All 

PSM jobs assume that they can make use of all available TrueScale contexts to 

satisfy the job requirement and try to give a context to each process. 

When context sharing is enabled on a system with multiple QLogic adapter 

(TrueScale) boards (units) and the IPATH_UNIT environment variable is set, the 

number of TrueScale contexts made available to MPI jobs is restricted to the 

number of contexts available on that unit. When multiple TrueScale devices are 

present, it restricts the use to a specific TrueScale unit. By default, all configured 

units are used in round robin order. 

Context Sharing Disabled: Each node program tries to obtain exclusive access 

to an TrueScale hardware context. If no hardware contexts are available, the job 

aborts. 

To explicitly disable context sharing, set this environment variable in one of the 

two following ways: 

PSM_SHAREDCONTEXTS=0 

PSM_SHAREDCONTEXTS=NO 

The default value of PSM_SHAREDCONTEXTS is 1 (enabled). 

Restricting TrueScale Hardware Contexts 

in a Batch Environment 

If required for resource sharing between multiple jobs in batch systems, you can 

restrict the number of TrueScale hardware contexts that are made available on 

each node of an MPI job by setting that number in the 

PSM_SHAREDCONTEXTS_MAX environment variable. 

D000046-005 B 4-13



For example, if you are running two different jobs on nodes using the QLE7140, 

set PSM_SHAREDCONTEXTS_MAX to 2 instead of the default 4. Each job would 

then have at most two of the four available hardware contexts. Both of the jobs 

that want to share a node would have to set PSM_SHAREDCONTEXTS_MAX=2 on 

that node before sharing begins. 

However, note that setting PSM_SHAREDCONTEXTS_MAX=2 as a clusterwide 

default would unnecessarily penalize nodes that are dedicated to running single 

jobs. So a per-node setting, or some level of coordination with the job scheduler 

with setting the environment variable, is recommended. 

If some nodes have more cores than others, then the setting must be adjusted 

properly for the number of cores on each node. 

Additionally, you can explicitly configure for the number of contexts you want to 

use with the cfgctxts module parameter. This will override the default settings 

(on the QLE7240 and QLE7280) based on the number of CPUs present on each 

node. See “TrueScale Hardware Contexts on the DDR and QDR InfiniBand 

Adapters” on page 4-12. 

Context Sharing Error Messages 

The error message when the context limit is exceeded is: 

No free InfiniPath contexts available on /dev/ipath 

This message appears when the application starts. 

Error messages related to contexts may also be generated by ipath_checkout 

or mpirun. For example: 

PSM found 0 available contexts on InfiniPath device 

The most likely cause is that the cluster has processes using all the available 

PSM contexts. Clean up these processes before restarting the job. 

Running in Shared Memory Mode 

QLogic MPI supports running exclusively in shared memory mode; no QLogic 

adapter is required for this mode of operation. This mode is used for running 

applications on a single node rather than on a cluster of nodes. 

To enable shared memory mode, use either a single node in the mpihosts file or 

use these options with mpirun: 

$ mpirun -np= -ppn= 

needs to be equal in both cases. 

NOTE: 

For this release, must be ≤ 64. 

4-14 D000046-005 B



When you are using a non-QLogic MPI that uses the TrueScale PSM layer, 

ensure that the parallel job is contained on a single node and set the 

PSM_DEVICES environment variable: 

PSM_DEVICES="shm,self" 

If you are using QLogic MPI, you do not need to set this environment variable; it is 

set automatically if np == ppn. 

When running on a single node with QLogic MPI, no infiniband adapter hardware 

is required if -disable-dev-check is passed to mpirun. 

mpihosts File Details 

As noted in “Create the mpihosts File” on page 4-3, an mpihosts file (also called 

a machines file, nodefile, or hostsfile) has been created in your current working 

directory. This file names the nodes that the node programs may run. 

The two supported formats for the mpihosts file are: 

hostname1 

hostname2 

... 

or 

hostname1:process_count 

hostname2:process_count 

... 

In the first format, if the -np count (number of processes to spawn in the mpirun 

command) is greater than the number of lines in the machine file, the hostnames 

will be repeated (in order) as many times as necessary for the requested number 

of node programs. 

In the second format, process_count can be different for each host, and is 

normally the number of available processors on the node. When not specified, the 

default value is one. The value of process_count determines how many node 

programs will be started on that host before using the next entry in the mpihosts 

file. When the full mpihosts file is processed, and there are additional processes 

requested, processing starts again at the start of the file. 

NOTE: 

To create an mpihosts file, use the ibhosts program. It will generate a 

list of available nodes that are already connected to the switch. 

There are several alternative ways of specifying the mpihosts file: 

D000046-005 B 4-15



Using mpirun 

• As noted in “Compile and Run an Example C Program” on page 4-4, you 

can use the command line option -m: 

$mpirun -np n -m mpihosts [other options] program-name 

In this case, if the named file cannot be opened, the MPI job fails. 

An alternate mechanism to -m for specifying hosts is the -H or -hosts 

followed by a host list. The host list can follow one of the following examples: 

host-[01-02,04,06-08] , or 

host-01,host-02,host-04,host-06,host-07,host-08 

• When neither the -m or the -H options are used, mpirun checks the 

environment variable MPIHOSTS for the name of the MPI hosts file. If this 

variable is defined and the file it names cannot be opened, the MPI job fails. 

• In the absence of the -m option, the -H option, and the MPIHOSTS 

environment variable, mpirun uses the file ./mpihosts, if it exists. 

• If none of these four methods of specifying the hosts file are used, mpirun 

looks for the file ~/.mpihosts. 

If you are working in the context of a batch queuing system, it may provide a job 

submission script that generates an appropriate mpihosts file. 

The script mpirun is a front end program that starts a parallel MPI job on a set of 

nodes in an TrueScale cluster. mpirun may be run on any i386 or x86_64 

machine inside or outside the cluster, as long as it is on a supported Linux 

distribution, and has TCP connectivity to all TrueScale cluster machines to be 

used in a job. 

The script starts, monitors, and terminates the node programs. mpirun uses ssh 

(secure shell) to log in to individual cluster machines and prints any messages 

that the node program prints on stdout or stderr, on the terminal where 

mpirun is invoked. 

4-16 D000046-005 B



NOTE: 

The mpi-frontend-* RPM needs to be installed on all nodes that will be 

using mpirun. Alternatively, you can use the mpirun option 

-distributed=off, which requires that only the mpi-frontend RPM is 

installed on the node where mpirun is invoked. Using -distributed=off 

can have a negative impact on mpirun’s performance when running 

large-scale jobs. More specifically, this option increases the memory usage 

on the host where mpirun is started and will slow down the job startup, 

since it will spawn MPI processes serially. 

The general syntax is: 

$ mpirun [mpirun_options...] program-name [program options] 

program-name is usually the pathname to the executable MPI program. When 

the MPI program resides in the current directory and the current directory is not in 

your search path, then program-name must begin with ‘./’, for example: 

./program-name 

Unless you want to run only one instance of the program, use the -np option, for 

example: 

$ mpirun -np n [other options] program-name 

This option spawns n instances of program-name. These instances are called 

node programs. 

Generally, mpirun tries to distribute the specified number of processes evenly 

among the nodes listed in the mpihosts file. However, if the number of 

processes exceeds the number of nodes listed in the mpihosts file, then some 

nodes will be assigned more than one instance of the program. 

Another command line option, -ppn, instructs mpirun to assign a fixed number p 

of node programs (processes) to each node, as it distributes n instances among 

the nodes: 

$ mpirun -np n -m mpihosts -ppn p program-name 

This option overrides the :process_count specifications, if any, in the lines of 

the mpihosts file. As a general rule, mpirun distributes the n node programs 

among the nodes without exceeding, on any node, the maximum number of 

instances specified by the :process_count option. The value of 

the :process_count option is specified by either the -ppn command line 

option or in the mpihosts file. 

D000046-005 B 4-17



NOTE: 

When the -np value is larger than the number of nodes in the mpihosts file 

times the -ppn value, mpirun cycles back through the hostsfile, assigning 

additional node programs per host. 

Typically, the number of node programs should not be larger than the number of 

processor cores, at least not for compute-bound programs. 

This option specifies the number of processes to spawn. If this option is not set, 

then environment variable MPI_NPROCS is checked. If MPI_NPROCS is not set, 

the default is to determine the number of processes based on the number of hosts 

in the machinefile -M or the list of hosts -H. 

-ppn processes-per-node 

This option creates up to the specified number of processes per node. 

Each node program is started as a process on one node. While a node program 

may fork child processes, the children themselves must not call MPI functions. 

The -distributed=on|off option has been added to mpirun. This option 

reduces overhead by enabling mpirun to start processes in parallel on multiple 

nodes. Initially, mpirun spawns one mpirun child per node from the root node, 

each of which in turn spawns the number of local processes for that particular 

node. Control the use of distributed mpirun job spawning mechanism with this 

option: 

-distributed [=on|off] 

The default is on. To change the default, put this option in the global 

mpirun.defaults file or a user-local file. See “Environment for Node 

Programs” on page 4-19 and “Environment Variables” on page 4-20 for details. 

mpirun monitors the parallel MPI job, terminating when all the node programs in 

that job exit normally, or if any of them terminates abnormally. 

Killing the mpirun program kills all the processes in the job. Use CTRL+C to kill 

mpirun. 

Console I/O in MPI Programs 

mpirun sends any output printed to stdout or stderr by any node program to 

the terminal. This output is line-buffered, so the lines output from the various node 

programs will be non-deterministically interleaved on the terminal. Using the -l 

option to mpirun will label each line with the rank of the node program where it 

was produced. 

Node programs do not normally use interactive input on stdin, and by default, 

stdin is bound to /dev/null. However, for applications that require standard 

input redirection, QLogic MPI supports two mechanisms to redirect stdin: 

4-18 D000046-005 B



• When mpirun is run from the same node as MPI rank 0, all input piped to 

the mpirun command is redirected to rank 0. 

• When mpirun is not run from the same node as MPI rank 0, or if the input 

must be redirected to all or specific MPI processes, the -stdin option can 

redirect a file as standard input to all nodes (or to a particular node) as 

specified by the -stdin-target option. 

Environment for Node Programs 

TrueScale-related environment variables are propagated to node programs. 

These include environment variables that begin with the prefix IPATH_, PSM_, 

MPI_ or LD_. Some other variables (such as HOME) are set or propagated by 

ssh(1). 

NOTE: 

The environment variable LD_BIND_NOW is not supported for QLogic MPI 

programs. Not all symbols referenced in the shared libraries can be resolved 

on all installations. (They provide a variety of compatible behaviors for 

different compilers, etc.) Therefore, the libraries are built to run in lazy 

binding mode; the dynamic linker evaluates and binds to symbols only when 

needed by the application in a given runtime environment. 

mpirun checks for these environment variables in the shell where it is invoked, 

and then propagates them correctly. The environment on each node is whatever it 

would be for the user’s login via ssh, unless you are using a Multi-Purpose 

Daemon (MPD) (see “MPD” on page 4-24). 

Environment variables are specified in descending order, as follows: 

1. Set in the default shell environment on a remote node, e.g., ~/.bashrc or 

equivalents. 

2. Set in -rcfile. 

3. Set the current shell environment for the mpirun command. 

4. If nothing has been set (none of the previous sets have been performed), 

the default value of the environment variable is used. 

As noted in the above list, using an mpirunrc file overrides any environment 

variables already set by the user. You can set environment variables for the node 

programs with the -rcfile option of mpirun with the following command: 

$ mpirun -np n -m mpihosts -rcfile mpirunrc program_name 

In the absence of this option, mpirun checks to see if a file called 

$HOME/.mpirunrc exists in the user's home directory. In either case, the file is 

sourced by the shell on each node when the node program starts. 

D000046-005 B 4-19



The .mpirunrc command line cannot contain any interactive commands. It can 

contain commands that output on stdout or stderr. 

There is a global options file for mpirun arguments. The default location of this 

file is: 

/opt/infinipath/etc/mpirun.defaults 

You can use an alternate file by setting the environment variable 

$PSC_MPIRUN_DEFAULTS_PATH. See the mpirun man page for more 

information. 

Environment Variables 

Table 4-7 contains a summary of the environment variables that are used by 

TrueScale and mpirun. 

Table 4-7. Environment Variables 

Name 

Description 

MPICH_ROOT 

This variable is used by mpirun to find the 

mpirun-ipath-ssh executable, set up 

LD_LIBRARY_PATH, and set up a prefix for all 

InfiniPath pathnames. This variable is used by the 

--prefix argument (or is the same as --prefix), 

if installing TrueScale RPMs in an alternate 

location. 

Default: Unset 

IPATH_PORT Specifies the port to use for the job, 1 or 2. 

Specifying 0 will autoselect IPATH_PORT. 


IPATH_SL 

IPATH_UNIT 

LD_LIBRARY_PATH 

Service Level for QDR Adapters, these are used 

to work with the switch's Vfabric feature. 


This variable is for context sharing. When multiple 

TrueScale devices are present, this variable 

restricts the use to a specific TrueScale unit. By 

default, all configured units are used in round 

robin order. 


This variable specifies the path to the run-time 

library. It is often set in the .mpirunrc file. 


4-20 D000046-005 B



Table 4-7. Environment Variables (Continued) 

Name 

Description 

MPICH_CC 

MPICH_CCC 

MPICH_F90 

MPIHOSTS 

MPI_NPROCS 

MPI_SHELL 

PSM_DEVICES 

PSC_MPIRUN_DEFAULTS_PATH 

PSM_SHAREDCONTEXTS 

PSM_SHAREDCONTEXTS_MAX 

This variable selects the compiler to use for 

mpicc, and others. 


mpicxx, and others. 


mpif90, and others. 

This variable sets the name of the machines 

(mpihosts) file. 


This variable specifies the number of MPI processes 

to spawn. 


Specifies the name of the program to log into 

remote hosts. 

Default: ssh unless MPI_SHELL is defined. 

Non-QLogic MPI users can set this variable to 

enable running in shared memory mode on a single 

node. This variable is automatically set for 

QLogic MPI. 

Default: PSM_DEVICES="self,ipath" 

This variable sets the path to a user-local mpirun 

defaults file. 

Default: 

/opt/infinipath/etc/mpirun.defaults 

This variable overrides automatic context sharing 

behavior. YES is equivalent to 1 (see Default). 

Default: PSM_SHAREDCONTEXTS=1 

This variable restricts the number of TrueScale 

contexts that are made available on each node of 

an MPI job. 

Default: 

PSM_SHAREDCONTEXTS_MAX=4 (QLE7140) 

Up to 16 on (QLE7240 and QLE7280; set automatically 

based on number of CPUs on node) 

D000046-005 B 4-21



Table 4-7. Environment Variables (Continued) 

Name 

IPATH_HCA_SELECTION_ALG 

Description 

This variable provides user-level support to specify 

Host Channel Adapter/port selection algorithm 

through the environment variable. The default 

option is Round Robin that allocates the Host 

Channel Adapters in a round robin fashion. The 

older mechanism option is Packed that fills all 

contexts on a Host Channel Adapter before allocating 

from the next Host Channel Adapter. 

For example: In the case of using two single-port 

Host Channel Adapters, the default or 

IPATH_HCA_SELECTION_ALG="Round Robin" 

setting, will allow 2 or more MPI processes per 

node to use both Host Channel Adapters and to 

achieve performance improvements compared to 

what can be achieved with one Host Channel 

Adapter. 

Running Multiple Versions of TrueScale or MPI 

The variable MPICH_ROOT sets a root prefix for all InfiniPath-related paths. It is 

used by mpirun to try to find the mpirun-ipath-ssh executable, and it also 

sets up the LD_LIBRARY_PATH for new programs. Consequently, multiple 

versions of the TrueScale software releases can be installed on some or all 

nodes, and QLogic MPI and other versions of MPI can be installed at the same 

time. It may be set in the environment, in mpirun.defaults, or in an rcfile (such 

as .mpirunrc, .bashrc, or .cshrc) that will be invoked on remote nodes. 

If you have installed the software into an alternate location using the --prefix 

option with rpm, --prefix would have been set to $MPICH_ROOT. 

If MPICH_ROOT is not set, the normal PATH is used unless mpirun is invoked with 

a full pathname. 

NOTE: 

mpirun-ssh was renamed mpirun-ipath-ssh to avoid name conflicts 

with other MPI implementations. 

4-22 D000046-005 B


Performance Tuning 

Job Blocking in Case of Temporary InfiniBand Link Failures 

By default, as controlled by mpirun’s quiescence parameter -q, an MPI job is 

killed for quiescence in the event of an InfiniBand link failure (or unplugged cable). 

This quiescence timeout occurs under one of the following conditions: 

• A remote rank’s process cannot reply to out-of-band process checks. 

• MPI is inactive on the InfiniBand link for more than 15 minutes. 

To keep remote process checks but disable triggering quiescence for temporary 

InfiniBand link failures, use the -disable-mpi-progress-check option with a 

nonzero -q option. To disable quiescence triggering altogether, use -q 0. No 

matter how these options are used, link failures (temporary or other) are always 

logged to syslog. 

If the link is down when the job starts and you want the job to continue blocking 

until the link comes up, use the -t -1 option. 

Performance Tuning 

CPU Affinity 

These methods may be used at runtime. Performance settings that are typically 

set by the system administrator are listed in “Performance Settings and 

Management Tips” on page 3-25. 

InfiniPath attempts to run each node program with CPU affinity set to a separate 

logical processor, up to the number of available logical processors. If CPU affinity 

is already set (with sched_setaffinity() or with the taskset utility), then 

InfiniPath will not change the setting. 

Use the taskset utility with mpirun to specify the mapping of MPI processes to 

logical processors. This combination makes the best use of available memory 

bandwidth or cache locality when running on dual-core Symmetric 

MultiProcessing (SMP) cluster nodes. 

The following example uses the NASA Advanced Supercomputing (NAS) Parallel 

Benchmark’s Multi-Grid (MG) benchmark and the -c option to taskset. 

$ mpirun -np 4 -ppn 2 -m $hosts taskset -c 0,2 bin/mg.B.4 

$ mpirun -np 4 -ppn 2 -m $hosts taskset -c 1,3 bin/mg.B.4 

The first command forces the programs to run on CPUs (or cores) 0 and 2. The 

second command forces the programs to run on CPUs 1 and 3. See the taskset 

man page for more information on usage. 

To turn off CPU affinity, set the environment variable IPATH_NO_CPUAFFINITY. 

This environment variable is propagated to node programs by mpirun. 

D000046-005 B 4-23


MPD 

mpirun Tunable Options 

MPD 

There are some mpirun options that can be adjusted to optimize communication. 

The most important one is: 

-long-len, -L [default: 64000] 

This option determines the length of the message that the rendezvous protocol 

(instead of the eager protocol) must use. The default value for -L was chosen for 

optimal unidirectional communication. Applications that have this kind of traffic 

pattern benefit from this higher default value. Other values for -L are appropriate 

for different communication patterns and data size. For example, applications that 

have bidirectional traffic patterns may benefit from using a lower value. 

Experimentation is recommended. 

Two other options that are useful are: 

-long-len-shmem, -s [default: 16000] 

This option determines the length of the message within the rendezvous protocol 

(instead of the eager protocol) to be used for intra-node communications. This 

option is for messages going through shared memory. The InfiniPath rendezvous 

messaging protocol uses a two-way handshake (with MPI synchronous send 

semantics) and receive-side DMA. 

-rndv-window-size, -W [default: 262144] 

When sending a large message using the rendezvous protocol, QLogic MPI splits 

it into a number of fragments at the source and recombines them at the 

destination. Each fragment is sent as a single rendezvous stage. This option 

specifies the maximum length of each fragment. The default is 262144 bytes. 

For more information on tunable options, type: 

$ mpirun -h 

MPD Description 

The complete list of options is contained in Appendix A. 

The Multi-Purpose Daemon (MPD) is an alternative to mpirun for launching MPI 

jobs. It is described briefly in the following sections. 

MPD was developed by Argonne National Laboratory (ANL) as part of the 

MPICH-2 system. While the ANL MPD had some advantages over the use of their 

mpirun (faster launching, better cleanup after crashes, better tolerance of node 

failures), the QLogic mpirun offers the same advantages. 

4-24 D000046-005 B


QLogic MPI and Hybrid MPI/OpenMP Applications 

Using MPD 

The disadvantage of MPD is reduced security, since it does not use ssh to launch 

node programs. It is also more complex to use than mpirun because it requires 

starting a ring of MPD daemons on the nodes. Therefore, QLogic recommends 

using the normal mpirun mechanism for starting jobs, as described in the 

previous chapter. However, if you want to use MPD, it is included in the InfiniPath 

software. 

To start an MPD environment, use the mpdboot program. You must provide 

mpdboot with a file that lists the machines that will run the mpd daemon. The 

format of this file is the same as for the mpihosts file in the mpirun command. 

Here is an example of how to run mpdboot: 

$ mpdboot -f hostsfile 

After mpdboot has started the MPD daemons, it will print a status message and 

drop into a new shell. 

To leave the MPD environment, exit from this shell. This will terminate the 

daemons. 

To use rsh instead of ssh with mpdboot, set the environment variable MPD_RSH 

to the pathname of the desired remote shell. For example: 

MPD_RSH=‘which rsh‘ mpdboot -n 16 -f hosts 

To run an MPI program from within the MPD environment, use the mpirun 

command. You do not need to provide an mpihosts file or a count of CPUs; by 

default, mpirun uses all nodes and CPUs available within the MPD environment. 

To check the status of the MPD daemons, use the mpdping command. 

NOTE: 

To use MPD, the software package mpi-frontend-*.rpm and python 

(available with your distribution) must be installed on all nodes. See the 

QLogic Fabric Software Installation Guide for more details on software 

installation. 

QLogic MPI and Hybrid MPI/OpenMP 

Applications 

QLogic MPI supports hybrid MPI/OpenMP applications, provided that MPI 

routines are called only by the master OpenMP thread. This application is called 

the funneled thread model. Instead of MPI_Init/MPI_INIT (for C/C++ and 

Fortran respectively), the program can call 

MPI_Init_thread/MPI_INIT_THREAD to determine the level of thread 

support, and the value MPI_THREAD_FUNNELED will be returned. 

D000046-005 B 4-25


Debugging MPI Programs 

To use this feature, the application must be compiled with both OpenMP and MPI 

code enabled. To do this, use the -mp flag on the mpicc compile line. 

As mentioned previously, MPI routines can be called only by the master OpenMP 

thread. The hybrid executable is executed as usual using mpirun, but typically 

only one MPI process is run per node and the OpenMP library will create 

additional threads to utilize all CPUs on that node. If there are sufficient CPUs on 

a node, you may want to run multiple MPI processes and multiple OpenMP 

threads per node. 

The number of OpenMP threads is typically controlled by the OMP_NUM_THREADS 

environment variable in the .mpirunrc file. (OMP_NUM_THREADS is used by 

other compilers’ OpenMP products, but is not a QLogic MPI environment 

variable.) Use this variable to adjust the split between MPI processes and 

OpenMP threads. Usually, the number of MPI processes (per node) times the 

number of OpenMP threads will be set to match the number of CPUs per node. An 

example case would be a node with four CPUs, running one MPI process and four 

OpenMP threads. In this case, OMP_NUM_THREADS is set to four. 

OMP_NUM_THREADS is on a per-node basis. 

See “Environment for Node Programs” on page 4-19 for information on setting 

environment variables. 

At the time of publication, the MPI_THREAD_SERIALIZED and 

MPI_THREAD_MULTIPLE models are not supported. 


MPI Errors 

NOTE: 

When there are more threads than CPUs, both MPI and OpenMP 

performance can be significantly degraded due to over-subscription of the 

CPUs. 

Debugging parallel programs is substantially more difficult than debugging serial 

programs. Thoroughly debugging the serial parts of your code before parallelizing 

is good programming practice. 

Almost all MPI routines (except MPI_Wtime and MPI_Wtick) return an error 

code; either as the function return value in C functions or as the last argument in a 

Fortran subroutine call. Before the value is returned, the current MPI error handler 

is called. By default, this error handler aborts the MPI job. Therefore, you can get 

information about MPI exceptions in your code by providing your own handler for 

MPI_ERRORS_RETURN. See the man page for the MPI_Errhandler_set for 

details. 

4-26 D000046-005 B



NOTE: 

MPI does not guarantee that an MPI program can continue past an error. 

Using Debuggers 

See the standard MPI documentation referenced in Appendix J for details on the 

MPI error codes. 

The InfiniPath software supports the use of multiple debuggers, including 

pathdb, gdb, and the system call tracing utility strace. These debuggers let you 

set breakpoints in a running program, and examine and set its variables. 

Symbolic debugging is easier than machine language debugging. To enable 

symbolic debugging, you must have compiled with the -g option to mpicc so that 

the compiler will have included symbol tables in the compiled object code. 

To run your MPI program with a debugger, use the -debug or 

-debug-no-pause and -debugger options for mpirun. See the man pages to 

pathdb, gdb, and strace for details. When running under a debugger, you get 

an xterm window on the front end machine for each node process. Therefore, you 

can control the different node processes as desired. 

To use strace with your MPI program, the syntax is: 

$ mpirun -np n -m mpihosts strace program-name 

The following features of QLogic MPI facilitate debugging: 

• Stack backtraces are provided for programs that crash. 

• The -debug and -debug-no-pause options are provided for mpirun. 

These options make each node program start with debugging enabled. The 

-debug option allows you to set breakpoints, and start running programs 

individually. The -debug-no-pause option allows postmortem inspection. 

Be sure to set -q 0 when using -debug. 

• Communication between mpirun and node programs can be printed by 

specifying the mpirun -verbose option. 

• MPI implementation debug messages can be printed by specifying the 

mpirun -psc-debug-level option. This option can substantially impact 

the performance of the node program. 

• Support is provided for progress timeout specifications, deadlock detection, 

and generating information about where a program is stuck. 

D000046-005 B 4-27


QLogic MPI Limitations 

• Several misconfigurations (such as mixed use of 32-bit/64-bit executables) 

are detected by the runtime. 

• A formatted list containing information useful for high-level MPI application 

profiling is provided by using the -print-stats option with mpirun. 

Statistics include minimum, maximum, and median values for message 

transmission protocols as well as a more detailed information for expected 

and unexpected message reception. See “MPI Stats” on page F-30 for more 

information and a sample output listing. 

NOTE: 

The TotalView ® debugger can be used with the Open MPI supplied in this 

release. Consult the TotalView documentation for more information. 

QLogic MPI Limitations 

The current version of QLogic MPI has the following limitations: 

• There are no C++ bindings to MPI; use the extern C MPI function calls. 

• In MPI-IO file I/O calls in the Fortran binding, offset, or displacement 

arguments are limited to 32 bits. Thus, for example, the second argument of 

MPI_File_seek must be between -2 31 and 2 31 -1, and the argument to 

MPI_File_read_at must be between 0 and 2 32 -1. 

4-28 D000046-005 B

5 Using Other MPIs 

Introduction 

This section provides information on using other MPI implementations. 

Support for multiple high-performance MPI implementations has been added. 

Most implementations run over both PSM and OpenFabrics Verbs (see 

Table 5-1). 

Table 5-1. Other Supported MPI Implementations 

MPI 

Implementation 

Runs Over 

Compiled 

With 

Comments 

Open MPI 1.5 

PSM 

Verbs 

GCC, Intel, PGI, 

PathScale 

Provides some MPI-2 functionality 

(one-sided operations 

and dynamic 

processes). 

Available as part of the 

QLogic download. 

Can be managed by 

mpi-selector. 

MVAPICH version 

1.2 

PSM 

Verbs 


PathScale 

Provides MPI-1 functionality. 

Available as part of the 

QLogic download. 


mpi-selector. 

MVAPICH2 version 

1.4 

PSM 

Verbs 


PathScale 

Provides MPI-2 Functionality. 


MPI-Selector. 

Platform MPI 7 and 

HP-MPI 2.3 

PSM 

Verbs 

GCC (default) 

Provides some MPI-2 functionality 

(one-sided operations). 

Available for purchase from 

HP. 

D000046-005 B 5-1

5–Using Other MPIs 


Table 5-1. Other Supported MPI Implementations (Continued) 

MPI 

Implementation 

Runs Over 

Compiled 

With 

Comments 

Platform (Scali) 5.6 

PSM 

GCC (default) 

Provides MPI-1 functionality. 

Verbs 


Platform. 

Intel MPI version 4.0 

TMI/PSM, 

uDAPL 

GCC (default) 

Provides MPI-1 and MPI-2 

functionality. 


Intel. 

Table Notes 

MVAPICH and Open MPI have been have been compiled for PSM to support the following versions 

of the compilers: 

• (GNU) gcc 4.1.0 

• (PathScale) pathcc 3.2 

• (PGI) pgcc 9.0 

• (Intel) icc 11.1 

These MPI implementations run on multiple interconnects, and have their own 

mechanisms for selecting the interconnect that runs on. Basic information about 

using these MPIs is provided in this section. However, for more detailed 

information, see the documentation provided with the version of MPI that you want 

to use. 


By default, the MVAPICH and Open MPI MPIs are installed in this directory tree: 

/usr/mpi//- 

The QLogic-supplied MPIs precompiled with the GCC, PathScale, PGI, and the 

Intel compilers will also have -qlc appended after . 

For example: 

/usr/mpi/gcc/openmpi-1.5-qlc 

If a prefixed installation location is used, /usr is replaced by $prefix. 

The following examples assume that the default path for each MPI implementation 

to mpirun is: 

/usr/mpi///bin/mpirun 

Again, /usr may be replaced by $prefix. This path is sometimes referred to as 

$mpi_home/bin/mpirun in the following sections. 

5-2 D000046-005 B


Open MPI 

Open MPI 

Installation 

See the documentation for HP-MPI, Intel MPI, and Platform MPI for their default 

installation directories. 

Open MPI is an open source MPI-2 implementation from the Open MPI Project. 

Pre-compiled versions of Open MPI version 1.5 that run over PSM and are built 

with the GCC, PGI, PathScale, and Intel compilers are available with the QLogic 

download. 

Open MPI that runs over Verbs and is pre-compiled with the GNU compiler is also 

available. 

Open MPI can be managed with the mpi-selector utility, as described in 

“Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility” on 

page 5-6. 

Follow the instructions in the QLogic Fabric Software Installation Guide for 

installing Open MPI. 

Newer versions than the one supplied with this release can be installed after 

QLogic OFED 1.4.2 has already been installed; these may be downloaded from 

the Open MPI web site. Note that versions that are released after the QLogic 

OFED 1.4.2 release will not be supported. 

Setup 

If you use the mpi-selector tool, the necessary setup is done for you. If you do 

not use this tool, you can put your Open MPI installation directory in the PATH: 

add /bin to PATH 

The is the directory path where the desired MPI was installed. 

Compiling Open MPI Applications 

As with QLogic MPI, QLogic recommends that you use the included wrapper 

scripts that invoke the underlying compiler (see Table 5-2). 

Table 5-2. Open MPI Wrapper Scripts 


Language 

mpicc 

mpiCC, mpicxx, or mpic++ 

C 

C++ 


D000046-005 B 5-3


Open MPI 

Table 5-2. Open MPI Wrapper Scripts 


Language 


To compile your program in C, type: 

$ mpicc mpi_app_name.c -o mpi_app_name 

Running Open MPI Applications 

By default, Open MPI shipped with the InfiniPath software stack will run over PSM 

once it is installed. 

Here is an example of a simple mpirun command running with four processes: 

$ mpirun -np 4 -machinefile mpihosts mpi_app_name 

To specify the PSM transport explicitly, add --mca mtl psm to the above 

command line. 

To run over InfiniBand Verbs instead, use this mpirun command line: 

$ mpirun -np 4 -machinefile mpihosts --mca btl sm --mca btl 

openib,self --mca mtl ^psm mpi_app_name 

The following command enables shared memory: 

--mca btl sm 

The following command enables openib transport and communication to self: 

--mca btl openib, self 

The following command disables PSM transport: 

--mca mtl ^psm 

In these commands, btl stands for byte transport layer and mtl for matching 

transport layer. 

PSM transport works in terms of MPI messages. OpenIB transport works in terms 

of byte streams. 

Alternatively, you can use Open MPI with a sockets transport running over IPoIB, 

for example: 

$ mpirun -np 4 -machinefile mpihosts --mca btl sm --mca btl 

tcp,self --mca btl_tcp_if_exclude eth0 --mca btl_tcp_if_include 

ib0 --mca mtl ^psm mpi_app_name 

Note that eth0 and psm are excluded, while ib0 is included. These instructions 

may need to be adjusted for your interface names. 

Note that in Open MPI, machinefile is also known as the hostfile. 

5-4 D000046-005 B


MVAPICH 

Further Information on Open MPI 

MVAPICH 

Installation 

For more information about Open MPI, see: 

http://www.open-mpi.org/ 

http://www.open-mpi.org/faq 

Pre-compiled versions of MVAPICH 1.2 built with the GNU, PGI, PathScale, and 

Intel compilers, and that run over PSM, are available with the QLogic download. 

MVAPICH that runs over Verbs and is pre-compiled with the GNU compiler is also 

available. 

MVAPICH can be managed with the mpi-selector utility, as described in 

“Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility” on 

page 5-6. 

To install MVAPICH, follow the instructions in the appropriate installation guide. 

Newer versions than the one supplied with this release can be installed after 

QLogic OFED 1.4.2 has already been installed; these may be downloaded from 

the MVAPICH web site. Note that versions that are released after the QLogic 

OFED 1.4.2 release will not be supported. 

Setup 

To launch MPI jobs, the MVAPICH installation directory must be included in PATH 

and LD_LIBRARY_PATH. 

When using sh for launching MPI jobs, run the command: 

$ source /usr/mpi///bin/mpivars.sh 

When using csh for launching MPI jobs, run the command: 

$ source /usr/mpi///bin/mpivars.csh 

Compiling MVAPICH Applications 



Table 5-3. MVAPICH Wrapper Scripts 


Language 

mpicc 

C 

D000046-005 B 5-5


Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility 

Table 5-3. MVAPICH Wrapper Scripts 


Language 

mpiCC, mpicxx 

C++ 



To compile your program in C, type: 


To check the default configuration for the installation, check the following file: 

/usr/mpi///etc/mvapich.conf 

Running MVAPICH Applications 

By default, the MVAPICH shipped with the InfiniPath software stack runs over 

PSM once it is installed. 


$ mpirun -np 4 -hostfile mpihosts mpi_app_name 

Password-less ssh is used unless the -rsh option is added to the command line 

above. 

Further Information on MVAPICH 

For more information about MVAPICH, see: 

http://mvapich.cse.ohio-state.edu/ 

Managing Open MPI, MVAPICH, and QLogic MPI 

with the mpi-selector Utility 

When multiple MPI implementations have been installed on the cluster, you can 

use the mpi-selector to switch between them. The MPIs that can be managed 

with the mpi-selector are: 

• Open MPI 

• MVAPICH 

• MVAPICH2 

• QLogic MPI 

The mpi-selector is an OFED utility that is installed as a part of QLogic OFED 

1.4.2. Its basic functions include: 

5-6 D000046-005 B


Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility 

• Listing available MPI implementations 

• Setting a default MPI to use (per user or site wide) 

• Unsetting a default MPI to use (per user or site wide) 

• Querying the current default MPI in use 

Following is an example for listing and selecting an MPI: 

$ mpi-selector --list 

mpi-1.2.3 

mpi-3.4.5 

$ mpi-selector --set mpi-3.4.5 

The new default take effect in the next shell that is started. See the 

mpi-selector man page for more information. 

For QLogic MPI inter-operation with the mpi-selector utility, you must install all 

QLogic MPI RPMs using a prefixed installation. Once the $prefix for QLogic 

MPI has been determined, install the qlogic-mpi-register with the same 

$prefix, this registers QLogic MPI with the mpi-selector utility and shows 

QLogic MPI as an available MPI implementation with the four different compilers. 

See the QLogic Fabric Software Installation Guide for information on prefixed 

installations. 

The example shell scripts mpivars.sh and mpivars.csh, for registering with 

mpi-selector, are provided as part of the mpi-devel RPM in 

$prefix/share/mpich/mpi-selector-{intel,gnu,pathscale,pgi} 

directories. 

For all non-GNU compilers that are installed outside standard Linux search paths, 

set up the paths so that compiler binaries and runtime libraries can be resolved. 

For example, set LD_LIBRARY_PATH, both in your local environment and in an rc 

file (such as .mpirunrc, .bashrc, or .cshrc), are invoked on remote nodes. 

See “Environment for Node Programs” on page 4-19 and “Compiler and Linker 

Variables” on page 4-10 for information on setting up the environment and 

“Specifying the Run-time Library Path” on page F-15 for information on setting the 

run-time library path. Also see “Run Time Errors with Different MPI 

Implementations” on page F-17 for information on run time errors that may occur if 

there are MPI version mismatches. 

NOTE: 

The Intel-compiled versions require that the Intel compiler be installed and 

that paths to the Intel compiler runtime libraries be resolvable from the user’s 

environment. The version used is Intel 10.1.012. 

D000046-005 B 5-7


HP-MPI and Platform MPI 7 

HP-MPI and Platform MPI 7 

Installation 

Platform Computing acquired HP-MPI from HP. Platform MPI 7 (formerly HP–MPI) 

is a high performance, production–quality implementation of the Message Passing 

Interface (MPI), with full MPI-2 funcionality. HP-MPI / Platform MPI 7 is distributed 

by over 30 commercial software vendors, so you may need to use it if you use 

certain HPC applications, even if you don't purchase the MPI separately. 

Follow the instructions for downloading and installing Platform MPI 7 from the 

Platform Computing web site. 

Setup 

Edit two lines in the hpmpi.conf file as follows: 

Change, 

MPI_ICMOD_PSM__PSM_MAIN = "îb_ipath" 

to, 

MPI_ICMOD_PSM__PSM_MAIN = "^" 

Change, 

to, 

MPI_ICMOD_PSM__PSM_PATH = "îb_ipath" 

MPI_ICMOD_PSM__PSM_PATH = "^" 

Compiling Platform MPI 7 Applications 



Table 5-4. Platform MPI 7 Wrapper Scripts 


Language 

mpicc 

mpiCC 

C 

C 

mpi77 Fortran 77 


5-8 D000046-005 B


Platform (Scali) MPI 5.6 

To compile your program in C using the default compiler, type: 


Running Platform MPI 7 Applications 

Here is an example of a simple mpirun command running with four processes, 

over PSM: 

$ mpirun -np 4 -hostfile mpihosts -PSM mpi_app_name 

To run over InfiniBand Verbs, type: 

$ mpirun -np 4 -hostfile mpihosts -IBV mpi_app_name 

To run over TCP (which could be IPoIB if the hostfile is setup for IPoIB interfaces), 

type: 

$ mpirun -np 4 -hostfile mpihosts -TCP mpi_app_name 

More Information on Platform MPI 7 

For more information on Platform MPI 7, see the Platform Computing web site 


Installation 

Platform MPI 5.6 was formerly known as Scali MPI Connect. The version tested 

with this release is 5.6.4. 

Follow the instructions for downloading and installing Platform MPI 5.6 from the 

Platform (Scali) web site. 

Setup 

To run over PSM, by default, add the line networks=infinpath to the file 

opt/scali/etc/ScaMPI.conf. 

If running over InfiniBand Verbs, Platform MPI needs to know which InfiniBand 

adapter to use. This is achieved by creating the file 

/opt/scali/etc/iba_params.conf using a line such as: 

hcadevice=qib0 

For a second InfiniPath card, ipath1 would be used, and so on. 

Compiling Platform MPI 5.6 Applications 


scripts that invoke the underlying compiler (see Table 5-5). The scripts default to 

using gcc/g++/g77. 

D000046-005 B 5-9



Table 5-5. Platform MPI Wrapper Scripts 


Language 

mpicc 

mpic++ 

C 

C++ 





To invoke another compiler, in this case PathScale, use the -cc1 option, for 

example: 

$ mpicc -cc1 pathcc mpi_app_name.c -o mpi_app_name 

Running Platform MPI 5.6 Applications 

Here is an example of a simple mpirun command running with four processes, 

over PSM: 

$ mpirun -np 4 -machinefile mpihosts mpi_app_name 

or if you have not set /opt/scali/etc/ScaMPI.conf to use PSM by default, 

use: 

$ mpirun -np 4 -machinefile mpihosts -networks=infinipath 

mpi_app_name 

Once installed, Platform MPI uses the PSM transport by default. To specify PSM 

explicitly, add -networks infinipath to the above command. 

To run Scali MPI over InfiniBand Verbs, type: 

$ mpirun -np 4 -machinefile mpihosts -networks ib,smp mpi_app_name 

This command indicates that ib is used for inter-node communications, and smp 

is used for intra-node communications. 

To run over TCP (or IPoIB), type: 

$ mpirun -np 4 -machinefile mpihosts -networks tcp,smp mpi_app_name 

Further Information on Platform MPI 5.6 

For more information on using Platform MPI 5.6, see: 

http://www.platform.com/cluster-computing/platform-mpi. 

5-10 D000046-005 B


Intel MPI 

Intel MPI 

Installation 

Intel MPI version 4.0 is the version tested with this release. 

Follow the instructions for download and installation of Intel MPI from the Intel web 

site. 

Setup 

Intel MPI can be run over Tag Matching Interface (TMI) 


1. Make sure that the TMI psm provider is installed on every node and all 

nodes have the same version installed. In this release it is called tmi-1.0 and 

is supplied with the QLogic OFED+ host software package. It can be 

installed either with the QLogic OFED+ Host Software installation or using 

the rpm files after the QLogic OFED+ Host Software tar file has been 

unpacked. For example: 

$ rpm -qa | grep tmi 

tmi-1.0-1 

2. Verify that there is a /etc/tmi.conf file. It should be installed by the 

tmi-1.0-1 RPM. The file tmi.conf contains a list of TMI psm providers. In 

particular it must contain an entry for the PSM provider in a form similar to: 

psm 1.0 libtmip_psm.so " " # Comments OK 

Intel MPI can also be run over uDAPL, which uses InfiniBand Verbs. uDAPL is the 

user mode version of the Direct Access Provider Library (DAPL), and is provided 

as a part of the OFED packages. You will also have to have IPoIB configured. 


1. Make sure that DAPL 1.2 or 2.0 is installed on every node and all nodes 

have the same version installed. In this release they are called 

compat-dapl. Both versions are supplied with the OpenFabrics RPMs and 

are included in the QLogic OFED+ Host Software package. They can be 

installed either with the QLogic OFED+ Host Software installation or using 

the rpm files after the QLogic OFED+ Host Software tar file has been 

unpacked. For example: 

Using DAPL 1.2. 

$ rpm -qa | grep compat-dapl 

compat-dapl-1.2.12-1.x86_64.rpm 

compat-dapl-debuginfo-1.2.12-1.x86_64.rpm 

compat-dapl-devel-1.2.12-1.x86_64.rpm 

D000046-005 B 5-11


Intel MPI 

compat-dapl-devel-static-1.2.12-1.x86_64.rpm 

compat-dapl-utils-1.2.12-1.x86_64.rpm 

Using DAPL 2.0. 

$ rpm -qa | grep dapl 

dapl-devel-static-2.0.19-1 

compat-dapl-1.2.14-1 

dapl-2.0.19-1 

dapl-debuginfo-2.0.19-1 

compat-dapl-devel-static-1.2.14-1 

dapl-utils-2.0.19-1 

compat-dapl-devel-1.2.14-1 

dapl-devel-2.0.19-1 

2. Verify that there is a /etc/dat.conf file. It should be installed by the 

dapl- RPM. The file dat.conf contains a list of interface adapters 

supported by uDAPL service providers. In particular, it must contain 

mapping entries for OpenIB-cma for dapl 1.2.x and ofa-v2-ib for 

dapl 2.0.x, in a form similar to this (each on one line): 

OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 

"ib0 0" "" 

and 

ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 

"ib0 0" "" 

3. On every node, type the following command (as a root user): 

# modprobe rdma_ucm 

To ensure that the module is loaded when the driver is loaded, add 

RDMA_UCM_LOAD=yes to the /etc/infiniband/openib.conf file. 

(Note that rdma_cm is also used, but it is loaded automatically.) 

4. Bring up an IPoIB interface on every node, for example, ib0. See the 

instructions for configuring IPoIB for more details. 

Intel MPI has different bin directories for 32-bit (bin) and 64-bit (bin64); 64-bit is 

the most commonly used. 

To launch MPI jobs, the Intel installation directory must be included in PATH and 

LD_LIBRARY_PATH. 

When using sh for launching MPI jobs, run the following command: 

$ source /bin64/mpivars.sh 

5-12 D000046-005 B


Intel MPI 

When using csh for launching MPI jobs, run the following command: 

$ source /bin64/mpivars.csh 

Substitute bin if using 32-bit. 

Compiling Intel MPI Applications 

As with QLogic MPI, QLogic recommended that you use the included wrapper 

scripts that invoke the underlying compiler. The default underlying compiler is 

GCC, including gfortran. Note that there are more compiler drivers (wrapper 

scripts) with Intel MPI than are listed here (see Table 5-6); check the Intel 

documentation for more information. 

Table 5-6. Intel MPI Wrapper Scripts 


Language 

mpicc 

mpiCC 

C 

C++ 



mpiicc 

mpiicpc 

mpiifort 

C (uses Intel C compiler) 

C++ (uses Intel C++ compiler) 

Fortran 77/90 (uses Intel Fortran compiler) 



To use the Intel compiler wrappers (mpiicc, mpiicpc, mpiifort), the Intel 

compilers must be installed and resolvable from the user’s environment. 

Running Intel MPI Applications 


$ mpirun -np 4 -f mpihosts mpi_app_name 

For more information, follow the Intel MPI instructions for usage of mpirun, 

mpdboot, and mpiexec (mpirun is a wrapper script that invoked both mpdboot 

and mpiexec). Remember to use -r ssh with mpdboot if you use ssh. 

Pass the following option to mpirun to select TMI: 

-genv I_MPI_FABRICS tmi 

Pass the following option to mpirun to select uDAPL: 

D000046-005 B 5-13


Improving Performance of Other MPIs Over InfiniBand Verbs 

uDAPL 1.2: 

-genv I_MPI_DEVICE rdma:OpenIB-cma 

uDAPL 2.0: 

-genv I_MPI_DEVICE rdma:ofa-v2-ib 

To help with debugging, you can add this option to the Intel mpirun command: 

TMI: 

-genv TMI_DEBUG 1 

uDAPL: 

-genv I_MPI_DEBUG 2 

Further Information on Intel MPI 

For more information on using Intel MPI, see: http://www.intel.com/ 

Improving Performance of Other MPIs Over 

InfiniBand Verbs 

Performance of MPI applications when using an MPI implementation over 

InfiniBand Verbs can be improved by tuning the InfiniBand MTU size. 

NOTE: 

No manual tuning is necessary for PSM-based MPIs, since the PSM layer 

determines the largest possible InfiniBand MTU for each source/destination 

path. 

The maximum supported MTU size of InfiniBand adapter cards is 4K. 

Support for 4K InfiniBand MTU requires switch support for 4K MTU. The method 

to set the InfiniBand MTU size varies by MPI implementation: 

• Open MPI defaults to the lower of either the InfiniBand MTU size or switch 

MTU size. 

• MVAPICH defaults to an InfiniBand MTU size of 1024 bytes. This can be 

over-ridden by setting an environment variable: 

$ export VIADEV_DEFAULT_MTU=MTU4096 

Valid values are MTU256, MTU512, MTU1024, MTU2048 and MTU4096. This 

environment variable must be set for all processes in the MPI job. To do so, 

use ~/.bashrc or use of /usr/bin/env. 

5-14 D000046-005 B



• HP-MPI over InfiniBand Verbs automatically determines the InfiniBand MTU 

size. 

• Platform (Scali) MPI defaults to an InfiniBand MTU of 1KB. This can be 

changed by adding a line to /opt/scali/etc/iba_params.conf, for 

example: 

mtu=2048 

A value of 4096 is not allowed by the Scali software (as of Scali 

Connect 5.6.0); in this case, a default value of 1024 bytes is used. This 

problem has been reported to support at Platform Inc. The largest value that 

can currently be used is 2048 bytes. 

• Intel MPI over uDAPL (which uses InfiniBand Verbs) automatically 

determines the InfiniBand MTU size. 

D000046-005 B 5-15



5-16 D000046-005 B

6 Performance Scaled 

Messaging 

Introduction 

Performance Scaled Messaging (PSM) provides support for full Virtual Fabric 

(vFabric) integration, allowing users to specify InfiniBand Service Level (SL) and 

Partition Key (PKey), or to provide a configured Service ID (SID) to target a 

vFabric. Support for using InfiniBand path record queries to the QLogic Fabric 

Manager during connection setup is also available, enabling alternative switch 

topologies such as Mesh/Torus. Note that this relies on the Distributed SA cache 

from FastFabric. 

All PSM enabled MPIs can leverage these capabilities transparently, but only two 

MPIs (QLogic MPI and OpenMPI) are configured to support it natively. Native 

support here means that MPI specific mpirun switches are available to 

activate/deactivate these features. Other MPIs will require use of environment 

variables to leverage these capabilities. With MPI applications, the environment 

variables need to be propagated across all nodes/processes and not just the node 

from where the job is submitted/run. The mechanisms to do this are MPI specific, 

but for two common MPIs the following may be helpful: 

• OpenMPI: Use –x ENV_VAR=ENV_VAL in the mpirun command line. 

Example: 

mpirun –np 2 –machinefile machinefile -x 

PSM_ENV_VAR=PSM_ENV_VAL prog prog_args 

• MVAPICH2: Use mpirun_rsh to perform job launch. Do not use mpiexec 

or mpirun. Specify the environment variable and value in the mpirun 

command line before the program argument. 

Example: 

mpirun_rsh –np 2 –hostfile machinefile PSM_ENV_VAR=PSM_ENV_VAL 

prog prog_args 

Some of the features available require appropriate versions of associated 

software and firmware for correct operation. These requirements are listed in the 

relevant sections. 

D000046-005 B 6-1

6–Performance Scaled Messaging 

Virtual Fabric Support 

Virtual Fabric Support 

Virtual Fabric (vFabric) in PSM is supported with the QLogic Fabric Manager. The 

latest version of the QLogic Fabric Manager contains a sample qlogic_fm.xml 

file with pre-configured vFabrics for PSM. Sixteen unique Service IDs have been 

allocated for PSM enabled MPI vFabrics to ease their testing however any 

Service ID can be used. Refer to the QLogic Fabric Manager User Guide on how 

to configure vFabrics. 

There are two ways to use vFabric with PSM. The “legacy” method requires the 

user to specify the appropriate SL and Pkey for the vFabric in question. For 

complete integration with vFabrics, users can now specify a Service ID (SID) that 

identifies the vFabric to be used. PSM will automatically obtain the SL and Pkey to 

use for the vFabric from the QLogic Fabric Manager via path record queries. 

Using SL and PKeys 

SL and Pkeys can be specified natively for OpenMPI and QLogic MPI. For other 

MPIs use the following list of environment variables to specify the SL and Pkey. 

The environment variables need to be propagated across all processes for correct 

operation. 

NOTE: 

This is available with OpenMPI v1.3.4rc4 and above only! 

• OpenMPI: Use mca parameters (mtl_psm_ib_service_level and 

mtl_psm_ib_pkey) to specify the pkey on the mpirun command line. 

Example: 

mpirun –np 2 –machinefile machinefile -mca 

mtl_psm_ib_service_level SL -mca mtl_psm_ib_pkey Pkey prog 

prog_args. 

• QLogic MPI: Requires use of IPATH_SL environment variable to specify 

the SL and the –p switch to mpirun for the Pkey. 

Example: 

IPATH_SL=SL mpirun –np 2 –m machinefile -p Pkey prog prog_args 

• Other MPIs can use the following environment variables that are propagated 

across all processes. This process is MPI library specific but samples on 

how to do this for OpenMPI and MVAPICH2 are listed in the “Introduction” 

on page 6-1. 

IPATH_SL=SL # Service Level to Use 0-15 

 

PSM_PKEY=Pkey # Pkey to use 

6-2 D000046-005 B


Using Service ID 

Using Service ID 

Full vFabric integration with PSM is available, allowing the user to specify a SID. 

For correct operation, PSM requires the following components to be available and 

configured correctly. 

• QLogic host Fabric Manager Configuration – PSM MPI vFabrics need to be 

configured and enabled correctly in the qlogic_fm.xml file. 16 unique 

SIDs have been allocated in the sample file. 

• OFED+ library needs to be installed on all nodes. This is available as part of 

Fast Fabrics tools. 

• QLogic Distributed SA needs to be installed, configured and activated on all 

the nodes. This is part of FastFabrics tools. Please refer to QLogic Fast 

Fabric User Guide on how to configure and activate the Distributed SA. The 

SIDs configured in the QLogic Fabric Manager configuration file should also 

be provided to the Distributed SA for correct operation. 

Service ID can be specified natively for OpenMPI and QLogic MPI. For other MPIs 

use the following list of environment variables. The environment variables need to 

be propagated across all processes for correct operation. 

• OpenMPI: Use mca parameters (mtl_psm_ib_service_id and 

mtl_psm_path_query) to specify the service id on the mpirun command 

line. Example: 

mpirun –np 2 –machinefile machinefile -mca mtl_psm_path_query 

opp -mca mtl_psm_ib_service_id SID prog prog_args 

• QLogic MPI: Use the –P and –S switch to mpirun command line to specify 

the Path record query library (always opp for OFED Plus Path in this 

release) and Service ID to use. Example: 

mpirun –np 2 –m machinefile -P opp –S SID prog prog_args 

• Other MPIs can use the following environment variables: 

PSM_PATH_REC=opp # Path record query mechanism to 

use. Always specify opp 

PSM_IB_SERVICE_ID=SID # Service ID to use 

SL2VL mapping from the Fabric Manager 

PSM is able to use the SL2VL table as programmed by the QLogic Fabric 

Manager. Prior releases required manual specification of the SL2VL mapping via 

an environment variable. 

D000046-005 B 6-3


Verifying SL2VL tables on QLogic 7300 Series Adapters 

Verifying SL2VL tables on QLogic 7300 Series 

Adapters 

iba_saquery can be used to get the SL2VL mapping for any given port 

however, QLogic 7300 series adapters exports the SL2VL mapping via sysfs files. 

These files are used by PSM to implement the SL2VL tables automatically. The 

SL2VL tables are per port and available under /sys/class/infiniband/hca 

name/ports/port #/sl2vl. The directory contains 16 files numbered 0-15 

that specify the SL. Listing the SL files returns the VL as programmed by the SL. 

6-4 D000046-005 B

7 Dispersive Routing 

Infiniband uses deterministic routing that is keyed from the Destination LID (DLID) 

of a port. The Fabric Manager programs the forwarding tables in a switch to 

determine the egress port a packet takes based on the DLID. 

Deterministic routing can create hotspots even in full bisection bandwidth (FBB) 

fabrics for certain communication patterns if the communicating node pairs map 

onto a common upstream link, based on the forwarding tables. Since routing is 

based on DLIDs, the InfiniBand fabric provides the ability to assign multiple LIDs 

to a physical port using a feature called Lid Mask Control (LMC). The total number 

of DLIDs assigned to a physical port is 2^LMC with the LIDS being assigned in a 

sequential manner. The common InfiniBand fabric uses a LMC of 0, meaning 

each port has 1 LID assigned to it. With non-zero LMC fabrics, this results in 

multiple potential paths through the fabric to reach the same physical port. For 

example, multiple DLID entries in the port forwarding table that could map to 

different egress ports. 

Dispersive routing, as implemented in the PSM, attempts to avoid congestion 

hotspots described above by “spraying” messages across these paths. A 

congested path will not bottleneck messages flowing down the alternate paths 

that are not congested. The current implementation of PSM supports fabrics with 

a maximum LMC of 3 (8 LIDs assigned per port). This can result in a maximum of 

64 possible paths between a SLID, DLID pair ([SLID, DLID],[SLID, DLID+1], 

[SLID,DLID+2]…..[SLID,DLID+8],[SLID+1, DLID],[SLID+1, DLID+1]…..[SLID+7, 

DLID+8]). Keeping state associated with these many paths requires large amount 

of memory resources, with empirical data showing not much gain in performance 

beyond utilizing a small set of multiple paths. Therefore PSM reduces the number 

of paths actually used in the above case to 8 where the following paths are the 

only ones considered for transmission — [SLID, DLID], [SLID + 1, DLID + 1], 

[SLID + 2, DLID + 2] ….. [SLID + N, DLID + N]. This makes the resource 

requirements manageable while providing most of the benefits of dispersive 

routing (congestion avoidance by utilizing multiple paths). 

D000046-005 B 7-1

7–Dispersive Routing 

Internally, PSM utilizes dispersive routing differently for small and large 

messages. Large messages are any messages greater-than or equal-to 64K. For 

large messages, the message is split into message fragments of 128K by default 

(called a window). Each of these message windows is sprayed across a distinct 

path between ports. All packets belonging to a window utilize the same path 

however the windows themselves can take a different path through the fabric. 

PSM assembles the windows that make up an MPI message before delivering it to 

the application. This allows limited out of order semantics through the fabrics to be 

maintain with little overhead. Small messages on the other hand always utilize a 

single path when communicating to a remote node however different processes 

executing on a node can utilize different paths for their communication between 

the nodes. For example, two nodes A and B each with 8 processors per node. 

Assuming the fabric is configured for a LMC of 3, PSM constructs 8 paths through 

the fabric as described above and a 16 process MPI application that spans these 

nodes (8 process per node). Then: 

• Each MPI process is automatically bound to a given CPU core numbered 

between 0-7. PSM does this at startup to get improved cache hit rates and 

other benefits. 

• Small Messages sent from a process on core N will use path N. 

NOTE: 

Only path N will be used by this process for all communications to any 

process on the remote node. 

• For a large message, each process will utilize all of the 8 paths and spray 

the windowed messages across it. 

The above highlights the default path selection policy that is active in PSM when 

running on non-zero LMC configured fabrics. There are 3 other path selection 

policies that determine how to select the path (or path index from the set of 

available paths) used by a process when communicating with a remote node. The 

above path policy is called adaptive. The 3 remaining path policies are static 

policies that assign a static path on job startup for both small and large message 

transfers. 

• Static_Src: Only one path per process is used for all remote 

communications. The path index is based on the CPU number the process 

is running. 

NOTE: 

Multiple paths are still used in the fabric if multiple processes (each on 

a different CPU) are communicating. 

7-2 D000046-005 B


• Static_Dest: The path selection is based on the CPU index of the 

destination process. Multiple paths can be used if data transfer is to different 

remote processes within a node. If multiple processes from Node A send a 

message to a single process on Node B only one path will be used across all 

processes. 

• Static_Base: The only path that is used is the base path [SLID,DLID] 

between nodes regardless of the LMC of the fabric or the number of paths 

available. This is similar to how PSM operated till the IFS 5.1 release. 

NOTE: 

A fabric configured with LMC of 0 even with the default adaptive policy 

enabled operates as the Static_Base policy as there only exists a 

single path between any pairs of port. 

D000046-005 B 7-3


7-4 D000046-005 B

8 gPXE 

gPXE Setup 

gPXE is an open source (GPL) network bootloader. It provides a direct 

replacement for proprietary PXE ROMs. See http://etherboot.org/wiki/index.php 

for documentation and general information. 

At least two machines and a switch are needed (or connect the two machines 

back-to-back and run QLogic Fabric Manager on the server). 

• A DHCP server 

• A boot server or http server (can be the same as the DHCP server) 

• A node to be booted 

Use a QLE7340 or QLE7342 adapter for the node. 

The following software is included with the QLogic OFED+ installation software 

package: 

• gPXE boot image 

• patch for DHCP server 

• tool to install gPXE boot image in EPROM of card 

• sample gPXE script 

Everything that can be done with the proprietary PXE loader over Ethernet, can be 

done with the gPXE loader over IB. The gPXE boot code is only a mechanism to 

load an initial boot image onto the system. It is up to the downloaded boot image 

to do the rest. 

For example, the boot image could be: 

• A stand-alone memory test program 

• A diskless kernel image that mounts its file systems via NFS 

Refer to http://www.faqs.org/docs/Linux-HOWTO/Diskless-HOWTO.html 

• A Linux install image like kickstart, which then installs software to the local 

hard drive(s). Refer to 

http://www.faqs.org/docs/Linux-HOWTO/KickStart-HOWTO.html 

D000046-005 B 8-1

8–gPXE 

Preparing the DHCP Server in Linux 

Required Steps 

• A second stage boot loader 

• A live CD Linux image 

• A gPXE script 

1. Download a copy of the gPXE image. 

Located at: 

• The executable to flash the EXPROM on the TrueScale InfiniBand 

adapters is located at: /usr/sbin/ipath_exprom 

• The gPXE driver for QLE7200 series InfiniBand adapters (the 

EXPROM image) is located at: 

/usr/share/infinipath/gPXE/iba7220.rom 

• The gPXE driver for QLE7300 series InfiniBand adapters (the 

EXPROM image) is located at: 

/usr/share/infinipath/gPXE/iba7322.rom 

2. In order for dhcpd to correctly load, assign IP addresses to the InfiniBand 

adapter GUID. The dhcpd on the existing DHCP server may need to be 

patched. This patch will be provided via the gPXE rpm installation. 

3. Write the ROM image to the InfiniBand adapter. 

This only needs to be done once per InfiniBand adapter. 

ipath_exprom -e -w iba7xxx.rom 

In some cases, executing the above command results in a hang. If you 

experience a hang, type CTRL+C to quit, then execute one flag at a time: 

ipath_exprom -e iba7xxx.rom 

ipath_exprom -w iba7xxx.rom 

4. Enable booting from the InfiniBand adapter (gPXE device) in the BIOS 


Installing DHCP 

When the boot session starts, the gPXE firmware attempts to bring up an adapter 

network link. If it succeeds to bring up a connected link, the gPXE firmware 

communicates with the DHCP server. The DHCP server assigns an IP address to 

the gPXE client and provides it with the location of the boot program. 

gPXE requires that the DHCP server runs on a machine that supports IP over IB. 

8-2 D000046-005 B

8–gPXE 


NOTE: 

Prior to installing DHCP, make sure that QLogic OFED+ is already installed 

on your DHCP server. 

1. Download and install the latest DHCP server from www.isc.org. 

Standard DHCP fields holding MAC address are not large enough to contain 

an IPoIB hardware address. To overcome this problem, DHCP over 

InfiniBand messages convey a client identifier field used to identify the 

DHCP session. This client identifier field can be used to associate an IP 

address with a client identifier value, such that the DHCP server will grant 

the same IP address to any client that conveys this client identifier. 

2. Unpack the latest downloaded DHCP server. 

tar zxf dhcp-release.tar.gz 

3. Uncomment the line /* #define USE_SOCKETS */ in 

dhcp-release/includes/site.h 

4. Change to the main directory. 

cd dhcp-release 

NOTE: 

If there is an older version of DHCP installed, save it before continuing 

with the following steps. 

5. Configure the source. 

./configure 

6. When the configuration of DHCP is finished, build the DHCP server. 

make 

Configuring DHCP 

7. When the DHCP has successfully finished building, install DHCP. 

make install 

1. From the client host, find the GUID of the Host Channel Adapter by using 

p1info or look at the GUID label on the Infiniband adapter. 

2. Turn the GUID into a MAC address and specify the port of the InfiniBand 

adapter that is going to be used at the end, using b0 for port0 or b1 for 

port1. 

D000046-005 B 8-3

8–gPXE 

Netbooting Over InfiniBand 

For example for a GUID that reads 0x00117500005a6eec, the MAC 

address would read: 00:11:75:00:00:5a:6e:ec:b0 

3. Add the MAC address to the DHCP server. 

The following is the sample /etc/dhcpd.conf file that specifies the Host 

Channel Adapter GUID for the hardware address: 

# 

# DHCP Server Configuration file. 

# see /usr/share/doc/dhcp*/dhcpd.conf.sample 

# 

ddns-update-style none; 

subnet 10.252.252.0 netmask 255.255.255.0 { 

option subnet-mask 255.255.255.0; 

range dynamic-bootp 10.252.252.100 10.252.252.109; 

host hl5-0 { 

hardware unknown-32 00:11:75:00:00:7e:c1:b0; 

option host-name "hl5"; 

} 

host hl5-1 { 

hardware unknown-32 00:11:75:00:00:7e:c1:b1; 

option host-name "hl5"; 

} 

} 

filename 

"http://10.252.252.1/images/uniboot/uniboot.php"; 

In this example, host hl5 has a dual port InfiniBand adapter. hl5-0 

corresponds to port 0, and hl5-1 corresponds to port 1 on the adapter. 

4. Restart the DHCP server 


The following procedures are an example of netbooting over InfinBand, using an 

HTTP boot server. 

8-4 D000046-005 B

8–gPXE 


Prerequisites 

• Required steps from above have been executed. 

• The BIOS has been configured to enable booting from the InfiniBand 

adapter. The gPXE InfiniBand device should be listed as the first boot 

device. 

• Apache server has been configured with PHP on your network, and is 

configured to serve pages out of /vault. 

• It is understood in this example that users would have their own tools and 

files for diskless booting with an http boot server. 

Boot Server Setup 

NOTE: 

The dhcpd and apache configuration files referenced in this example 

are included as examples, and are not part of the QLogic OFED+ 

installed software. Your site boot servers may be different, see their 

documentation for equivalent information. 

Instructions on installing and configuring a dhcp server or a boot server 

are beyond the scope of this document. 

Configure the boot server for your site. 

NOTE: 

gPXE supports several file transfer methods such as TFTP, HTTP, iSCSI. 

This example uses HTTP since it generally scales better and is the preferred 

choice. 

NOTE: 

This step involves setting up a http server and needs to be done by a user 

that understands server setup on the http server is being used 

1. Install Apache. 

2. Create an images.conf file and a kernels.conf file and place them in 

the /etc/httpd/conf.d directory. This sets up aliases for and tells 

apache where to find them: 

/images — http://10.252.252.1/images/ 

/kernels — http://10.252.252.1/kernels/ 

D000046-005 B 8-5

8–gPXE 


The following is an example of the images.conf file 

Alias /images /vault/images 

 

AllowOverride All 

Options Indexes FollowSymLinks 

Order allow,deny 

Allow from all 

 

The following is an example of the kernels.conf file 

Alias /kernels /boot 

 

AllowOverride None 

Order allow,deny 

Allow from all 

 

3. Make a uniboot directory: 

mkdir -p /vault/images/uniboot 

4. Create a initrd.img file 

Prerequisites 

• “gPXE Setup” on page 8-1 has been completed. 

• “Preparing the DHCP Server in Linux” on page 8-2 has been 

completed 

To add an InfiniBand driver into the initrd file, The InfiniBand modules 

need to be copied to the diskless image. The host machine needs to be 

pre-installed with the QLogic OFED+ Host Software that is appropriate for 

the kernel version the diskless image will run. The QLogic OFED+ Host 

Software is available for download from 

http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/default.aspx 

NOTE: 

The remainder of this section assumes that QLogic OFED+ has been 

installed on the Host machine. 

8-6 D000046-005 B

8–gPXE 


WARNING!! 

The following procedure modifies critical files used in the boot 

procedure. It must be executed by users with expertise in the boot 

process. Improper application of this procedure may prevent the 

diskless machine from booting. 

a. If /vault/images/initrd.img file is already present on the server 

machine, back it up. For example: 

cp -a /vault/images/initrd.img /vault/images/ 

initrd.img.bak 

D000046-005 B 8-7

8–gPXE 


b. The infinipath rpm will install the file 

/usr/share/infinipath/gPXE/gpxe-qib-modify-initrd 

with contents similar to the following example. You can either run the 

script to generate a new initrd image, or use it as an example, and 

customize as appropriate for your site. 

# This assumes you will use the currently running version 

of linux, and 

# that you are starting from a fully configured machine of 

the same type 

# (hardware configuration), and BIOS settings. 

# 

# start with a known path, to get the system commands 

PATH=/sbin:/usr/sbin:/bin:/usr/bin:$PATH 

# start from a copy of the current initd image 

mkdir -p /var/tmp/initrd-ib 

cd /var/tmp/initrd-ib 

kern=$(uname -r) 

if [ -e /boot/initrd-${kern}.img ]; then 

initrd=/boot/initrd-${kern}.img 

elif [ -e /boot/initrd ]; then 

initrd=/boot/initrd 

else 

echo Unable to locate correct initrd, fix script and 

re-run 

exit 1 

fi 

cp ${initrd} initrd-ib-${kern}.img 

# Get full original listing 

gunzip -dc initrd-ib-${kern}.img | cpio -it --quiet | 

grep -v '^\.$' | sort -o Orig-listing 

8-8 D000046-005 B

8–gPXE 


# start building modified image 

rm -rf new # for retries 

mkdir new 

cd new 

# extract previous contents 

gunzip -dc ../initrd-ib-${kern}.img | cpio --quiet -id 

# add infiniband modules 

mkdir -p lib/ib 

find /lib/modules/${kern}/updates -type f | \ 

egrep 

'(iw_cm|ib_(mad|addr|core|sa|cm|uverbs|ucm|umad|ipoib|qib 

).ko|rdma_|ipoib_helper)' | \ 

xargs -I '{}' cp -a '{}' lib/ib 

# Some distros have ipoib_helper, others don't require it 

if [ -e lib/ib/ipoib_helper ]; then 

helper_cmd='/sbin/insmod /lib/ib/ipoib_helper.ko' 

fi 

# On some kernels, the qib driver will require the dca 

module 

if modinfo -F depends ib_qib | grep -q dca; then 

cp $(find /lib/modules/$(uname -r) -name dca.ko) lib/ib 

dcacmd='/sbin/insmod /lib/ib/dca.ko' 

else 

dcacmd= 

fi 

# IB requires loading an IPv6 module. If you do not have 

it in your initrd, add it 

if grep -q ipv6 ../Orig-listing; then 

# already added, and presumably insmod'ed, along with any 

dependencies 

v6cmd= 

else 

echo -e 'Adding IPv6 and related modules\n' 

cp /lib/modules/${kern}/kernel/net/ipv6/ipv6.ko lib 

IFS=' ' v6cmd='echo "Loading IPV6" 

D000046-005 B 8-9

8–gPXE 


/sbin/insmod /lib/ipv6.ko' 

# Some versions of IPv6 have dependencies, add them. 

xfrm=$(modinfo -F depends ipv6) 

if [ ${xfrm} ]; then 

cp $(find /lib/modules/$(uname -r) -name ${xfrm}.ko) 

lib 

IFS=' ' v6cmd='/sbin/insmod /lib/'${xfrm}'.ko 

'"$v6cmd" 

crypto=$(modinfo -F depends $xfrm) 

if [ ${crypto} ]; then 

cp $(find /lib/modules/$(uname -r) -name 

${crypto}.ko) lib 

IFS=' ' v6cmd='/sbin/insmod /lib/'${crypto}'.ko 

'"$v6cmd" 

fi 

fi 

fi 

# we need insmod to load the modules; if not present it, 

copy it 

mkdir -p sbin 

grep -q insmod ../Orig-listing || cp /sbin/insmod sbin 

echo -e 'NOTE: you will need to config ib0 in the normal 

way in your booted root 

filesystem, in order to use it for NFS, etc.\n' 

# Now build the commands to load the additional modules. 

We add them just after 

# the last existing insmod command, so all other 

dependences will be resolved 

# You can change the location if desired or necessary. 

# loading order is important. You can verify the order 

works ahead of time 

# by running "/etc/init.d/openibd stop", and then running 

these commands 

# manually by cut and paste 

# This will work on SLES, although different than the 

standard mechanism 

cat > ../init-cmds

8–gPXE 


# Start of IB module block 

$v6cmd 

echo "loading IB modules" 

/sbin/insmod /lib/ib/ib_addr.ko 

/sbin/insmod /lib/ib/ib_core.ko 

/sbin/insmod /lib/ib/ib_mad.ko 

/sbin/insmod /lib/ib/ib_sa.ko 

/sbin/insmod /lib/ib/ib_cm.ko 

/sbin/insmod /lib/ib/ib_uverbs.ko 

/sbin/insmod /lib/ib/ib_ucm.ko 

/sbin/insmod /lib/ib/ib_umad.ko 

/sbin/insmod /lib/ib/iw_cm.ko 

/sbin/insmod /lib/ib/rdma_cm.ko 

/sbin/insmod /lib/ib/rdma_ucm.ko 

$dcacmd 

/sbin/insmod /lib/ib/ib_qib.ko 

$helper_cmd 

/sbin/insmod /lib/ib/ib_ipoib.ko 

echo "finished loading IB modules" 

# End of IB module block 

EOF 

# first get line number where we append (after last insmod 

if any, otherwse 

# at start 

line=$(egrep -n insmod init | sed -n '$s/:.*//p') 

if [ ! "${line}" ]; then line=1; fi 

sed -e "${line}r ../init-cmds" init > init.new 

# show the difference, then rename 

echo -e 'Differences between original and new init 

command script\n' 

diff init init.new 

mv init.new init 

chmod 700 init 

# now rebuilt the initrd image 

find . | cpio --quiet -H newc -o | gzip > 

../initrd-${kern}.img 

D000046-005 B 8-11

8–gPXE 


cd .. 

# get the file list in the new image 

gunzip -dc initrd-${kern}.img | cpio --quiet -it | grep 

-v '^\.$' | sort -o New-listing 

# and show the differences. 

echo -e '\nChanges in files in initrd image\n' 

diff Orig-listing New-listing 

# copy the new initrd to wherever you have configure the 

dhcp server to look 

# for it (here we assume it's /images) 

mkdir -p /images 

cp initrd-${kern}.img /images 

echo -e '\nCompleted initrd for IB' 

ls -l /images/initrd-${kern}.img 

c. Run the 

usr/share/infinipath/gPXE/gpxe-qib-modify-initrd 

script to create the initrd.img file. 

At this stage, the initrd.img file is ready and located at the location 

where the DHCP server was configured to look for it. 

5. Create auniboot.php file and save it to /vault/images/uniboot. 

NOTE: 

The uniboot.php generates a gPXE script that will attempt to 

boot from the /boot/vmlinuz-2.6.18-128.el5 kernel. If 

you want to boot from a different kernel, edit uniboot.php with 

the appropriate kernel string in the $kver variable. 

8-12 D000046-005 B

8–gPXE 


The following is an example of a uniboot.php file: 

< 

header ( 'Content-type: text/plain' ); 

function strleft ( $s1, $s2 ) { 

return substr ( $s1, 0, strpos ( $s1, $s2 ) ); 

} 

function baseURL() { 

$s = empty ( $_SERVER["HTTPS"] ) '' : 

( $_SERVER["HTTPS"] == "on" ) "s" : ""; 

$protocol = strleft ( strtolower ( 

$_SERVER["SERVER_PROTOCOL"] ), "/" ).$s; 

$port = ( $_SERVER["SERVER_PORT"] == "80" ) "" : 

( ":".$_SERVER["SERVER_PORT"] ); 

return $protocol."://".$_SERVER['SERVER_NAME'].$port; 

} 

$baseurl = baseURL(); 

$selfurl = $baseurl.$_SERVER['REQUEST_URI']; 

$dirurl = $baseurl.( dirname ( $_SERVER['SCRIPT_NAME'] ) 

); 

$kver = "2.6.18-164.11.1.el5"; 

echo

8–gPXE 

HTTP Boot Setup 

This is the kernel that will boot. 

This file can be copied from any machine that has RHEL5.3 installed. 

2. Start httpd 

Steps on the gPXE Client 

1. Ensure that the Host Channel Adapter is listed as the first bootable device in 

the BIOS. 

2. Reboot the test node(s) and enter the BIOS boot setup. 

This is highly dependent on the BIOS for the system but you should see a 

menu for boot options and a submenu for boot devices. 

Select gPXE IB as the first boot device. 

When you power on the system or press the reset button, the system will 

execute the boot code on the Host Channel Adapter that will query the 

DHCP server for the IP address and boot image to download. 

Once the boot image is downloaded, the BIOS/Host Channel Adapter is 

finished and the boot image is ready. 

3. Verify system boots off of the kernel image on the boot server. The best way 

to do this is to boot into a different kernel from the one installed on the hard 

drive on the client, or to un-plug the hard drive on the client and verify that on 

boot up, a kernel and file system exist. 


gPXE supports booting diskless machines. To enable using an IB driver, the 

(remote) kernel or initrd image must include and be configured to load that driver. 

This can be achieved either by compiling the Host Channel Adapter driver into the 

kernel, or by adding the device driver module into the initrd image and loading it. 

1. Make a new directory 

mdir /vault/images/uniboot 

2. Change directories 

cd /vault/images/uniboot 

3. Create a initrd.img file using the information and example in Step 4 

of Boot Server Setup. 

4. Create a uniboot.php file using the example in Step 5 of Boot Server 

Setup. 

8-14 D000046-005 B

8–gPXE 


5. Create an images.conf file and a kernels.conf file using the 

examples in Step 2 of Boot Server Setup and place them in the 

/etc/httpd/conf.d directory. 

6. Edit /etc/dhcpd.conf file to boot the clients using HTTP 

filename "http://172.26.32.9/images/uniboot/uniboot.php"; 

7. Restart the DHCP server 

8. Start HTTP if it is not already running: 

/etc/init.d/httpd start 

D000046-005 B 8-15

8–gPXE 


8-16 D000046-005 B

A 

mpirun Options Summary 

This section summarizes the most commonly used options to mpirun. See the 

mpirun (1) man page for a complete listing. 

Job Start Options 

-mpd 

This option is used after running mpdboot to start a daemon, rather than using the 

default ssh protocol to start jobs. See the mpdboot(1) man page for more 

information. None of the other mpirun options (with the exception of -h) are valid 

when using this option. 

-ssh 

This option uses the ssh program to start jobs, either directly or through 

distributed startup. This is the default. 

Essential Options 

-H, -hosts hostlist 

When this option is used, the list of possible hosts to run on is taken from the 

specified hostlist, which has precedence over the -machinefile option. The 

hostlist can be comma-delimited or quoted as a space-delimited list. The 

hostlist specification allows compressed representation of the form: 

host-[01-02,04,06-08], is equivalent to: 

host-01,host-02,host-04,host-06,host-07,host-08 

If the -np count is unspecified, it is adjusted to the number of hosts in the 

hostlist. If the -ppn count is specified, each host will receive as many 

processes. 

-machinefile filename, -m filename 

This option specifies the machines (mpihosts) file that contains the list of hosts to 

be used for this job. The default is $MPIHOSTS, then ./mpihosts, and finally 

~/.mpihosts. 

-nonmpi 

This option runs a non-MPI program, and is required if the node program makes 

no MPI calls. This option allows non-QLogic MPI applications to use mpirun’s 

parallel spawning mechanism. 

D000046-005 B A-1

A–mpirun Options Summary 

Spawn Options 

-np np 

This option specifies the number of processes to spawn. If this option is not set, 

then the environment variable MPI_NPROCS is checked. If MPI_NPROCS is not 

set, the default is to determine the number of processes based on the number of 

hosts in the machinefile -M or the list of hosts -H. 

-ppn processes-per-node 

This option creates up to the specified number of processes per node. 

By default, a limit is enforced that depends on how many InfiniPath contexts are 

supported by the node (depends on the hardware type and the number of 

InfiniPath cards present). 

InfiniPath context (port) sharing is supported, beginning with the InfiniPath 2.0 

release. This feature allows running up to four times as many processes per node 

as was previously possible, with a small additional overhead for each shared 

context. 

Context sharing is enabled automatically if needed. Use of the full number of 

available contexts is assumed. To restrict the number of contexts, use the 

environment variable PSM_SHAREDCONTEXTS_MAX to divide the available 

number of contexts. 

Context sharing behavior can be overriden by using the environment variable 

PSM_SHAREDCONTEXTS. Setting this variable to zero disables context sharing, 

and jobs that require more than the available number of contexts cannot be run. 

Setting this variable it to one (the default) causes context sharing to be enabled if 

needed. 

-rcfile node-shell-script 

This is the startup script for setting the environment on nodes. Before starting 

node programs, mpirun checks to see if a file called .mpirunrc exists in the 

user’s home directory. If the file exists, it is sourced into the running remote 

shell. Use -rcfile node-shell-script or .mpirunrc to set paths and other 

environment variables such as LD_LIBRARY_PATH. 

Default: $HOME/.mpirunrc 

Spawn Options 

-distributed [=on|off] 

This option controls use of the distributed mpirun job spawning mechanism. The 

default is on. To change the default, put this option in the global 

mpirun.defaults file or a user-local file (see the environment variable 

PSC_MPIRUN_DEFAULTS_PATH for details). When the option appears more than 

once on the command line, the last setting controls the behavior. 

Default: on. 

A-2 D000046-005 B


Quiescence Options 

Quiescence Options 

-disable-mpi-progress-check 

This option disables the MPI communication progress check without disabling the 

ping reply check. 

If quiescence or a lack of ping reply is detected, the job and all compute 

processes are terminated. 

-i, -ping-interval,seconds 

This options specifies the seconds to wait between ping packets to mpirun 

(if -q > 0). 

Default: 60 

-q, -quiescence-timeout,seconds 

This option specifies the wait time (in seconds) for quiescence (absence of MPI 

communication or lack of ping reply) on the nodes. It is useful for detecting 

deadlocks. A value of zero disables quiescence detection. 

Default: 900 

Verbosity Options 

-job-info 

This option prints brief job startup and shutdown timing information. 

-no-syslog 

When this option is specified, critical errors are not sent through syslog. By 

default, critical errors are sent to the console and through syslog. 

-V, -verbose 

This option prints diagnostic messages from mpirun itself. The verbose option is 

useful in troubleshooting. 

Verbosity will also list the IPATH_* and PSM_* environment variable settings that 

affect MPI operation. 

Startup Options 

-I, -open-timeout seconds 

This option tries for the number of seconds to open the InfiniPath device. If 

seconds is -1 (negative one), the node program waits indefinitely. Use this option 

to avoid having all queued jobs in a batch queue fail when a node fails for some 

reason, or is taken down for administrative purposes. The -t option is also 

normally set to -1. 

D000046-005 B A-3


Stats Options 

Stats Options 

-k, -kill-timeout seconds 

This option indicates the time to wait for other ranks after the first rank exits. 

Default: 60 

-listen-addr | 

This option specifies the hostname (or IPv4 address) to listen on for incoming 

socket connections. It is useful for an mpirun front-end multihomed host. By 

default, mpirun assumes that ranks can independently resolve the hostname 

obtained on the head node with gethostname(2). To change the default, put 

this option in the global mpirun.defaults file or a user-local file. 

-runscript 

This is the script used to run the node program. 

-t, -timeout seconds 

This option waits for specified time (in seconds) for each node to establish 

connection back to mpirun. If seconds is -1 (negative one), mpirun will wait 

indefinitely. 

Default: 60 

-M [=stats_types], -print-stats [=stats_types] 

Statistics include minimum, maximum, and median values for message 

transmission protocols as well as more detailed information for expected and 

unexpected message reception. If the option is provided without an argument, 

stats_types is assumed to be mpi. 

The following stats_types can be specified: 

mpi 

ipath 

p2p 

counters 

devstats. 

all 

Shows an MPI-level summary (expected, unexpected message) 

Shows a summary of InfiniPath interconnect communication 

Shows detailed per-MPI rank communication information 

Shows low-level InfiniPath device counters 

Shows InfiniPath driver statistics 

Shows statistics for all stats_types 

One or more statistics types can be specified by separating them with a comma. 

For example, -print-stats=ipath,counters displays InfiniPath 

communication protocol as well as low-level device counter statistics. For details, 

see “MPI Stats” on page F-30. 

A-4 D000046-005 B


Tuning Options 

-statsfile file-prefix 

This option specifies an alternate file to receive the output from the 

-print-stats option. 

Default: stderr 

-statsmode absolute|diffs 

When printing process statistics with the -print-stats option, this option 

specifies if the printed statistics have the absolute values of the QLogic adapter 

chip counters and registers or if there are differences between those values at the 

start and end of the process. 

Default mode: diffs 

Tuning Options 

-L, -long-len length 

This option determines the length of the message used by the rendezvous 

protocol. The InfiniPath rendezvous messaging protocol uses two-way handshake 

(with MPI synchronous send semantics) and receive-side DMA. 

Default: 64000 

-N, -num-send-bufs buffer-count 

QLogic MPI uses the specified number as the number of packets that can be sent 

without having to wait from an acknowledgement from the receiver. Each packet 

contains approximately 2048 bytes of user data. 

Default: 512 

-s,-long-len-shmem length 

This option specifies the length of the message used by the rendezvous protocol 

for intra-node communications. The InfiniPath rendezvous messaging protocol 

uses two-way handshake (with MPI synchronous send semantics) and 

receive-side DMA. 

Default: 16000 

-W, -rndv-window-size length 

When sending a large message using the rendezvous protocol, QLogic MPI splits 

the message into a number of fragments at the source and recombines them at 

the destination. Each fragment is sent as a single rendezvous stage. This option 

specifies the maximum length of each fragment. 

Default: 262144 bytes 

D000046-005 B A-5


Shell Options 

Shell Options 

-shell shell-name 

This option specifies the name of the program to use to log into remote hosts. 

Default: ssh, unless $MPI_SHELL is defined. 

-shellx shell-name 

This option specifies the name of program to use to log into remote hosts with X11 

forwarding. This option is useful when running with -debug or in xterm. 

Default: ssh, unless $MPI_SHELL_X is defined. 

Debug Options 

-debug 

This option starts all the processes under debugger, and waits for the user to set 

breakpoints and run the program. The gdb option is used by default, but can be 

overridden using the -debugger argument. Other supported debuggers are 

strace and the QLogic debugger pathdb. 

-debug-no-pause 

This option is similar to -debug, except that it does not pause at the beginning. 

The gdb option is used by default. 

-debugger gdb|pathdb|strace 

This option uses the specified debugger instead of the default gdb. 

-display X-server 

This option uses the specified X server for invoking remote xterms. (-debug, 

-debug-no-pause, and -in-xterm options use this value.) 

Default: whatever is set in $DISPLAY 

-in-xterm 

This option runs each process in an xterm window. This is implied when -debug 

or -debug-no-pause is used. 

Default: write to stdout with no stdin 

-psc-debug-level mask 

This option controls the verbosity of messages printed by the MPI and InfiniPath 

protocol layer. The default is 1, which displays error messages. A value of 3 displays 

short messaging details such as source, destination, size, etc. A value of FFh prints 

detailed information in a messaging layer for each message. Use this option with care, 

since too much verbosity will negatively affect application performance. 

Default: 1 

-xterm xterm 

This option specifies the xterm to use. 

Default: xterm 

A-6 D000046-005 B


Format Options 

Format Options 

-l, -label-output 

This option labels each line of output on stdout and stderr with the rank of the 

MPI process that produces the output. 

-y, -labelstyle string 

This option specifies the label that is prefixed to error messages and statistics. 

Process rank is the default prefix. The label that is prefixed to each message can 

be specified as one of the following: 

Other Options 

%n Hostname that the node process executes 

%r Rank of the node process 

%p Process ID of the node process 

%L LID (InfiniPath local identifier (LID) adapter identifier) of the node 

%P InfiniPath port of the node process 

%l Local rank of the node process within a node 

%% Percent sign 

-h -help 

This option prints a summary of mpirun options, then exits. 

-stdin filename 

This option specifies the filename that must be fed as stdin to the node program. 

Default: /dev/null 

-stdin-target 0..np-1 | -1 

This option specifies the process rank that must receive the file specified with the 

-stdin option. Negative one (-1) means all ranks. 

Default: -1 

-v, -version 

This option prints the mpirun version, then exits. 

-wdir path-to-working_dir 

This option sets the working directory for the node program. 

Default: -wdir current-working-dir 

D000046-005 B A-7


Other Options 

Notes 

A-8 D000046-005 B

B 

Benchmark Programs 

Several MPI performance measurement programs are installed from the 

mpi-benchmark RPM. This appendix describes these benchmarks and how to 

run them. These programs are based on code from the group of Dr. Dhabaleswar 

K. Panda at the Network-Based Computing Laboratory at the Ohio State 

University. For more information, see: http://mvapich.cse.ohio-state.edu/ 

These programs allow you to measure the MPI latency and bandwidth between 

two or more nodes in your cluster. Both the executables, and the source for those 

executables, are shipped. The executables are installed by default under 

/usr/mpi/qlogic/bin (though /usr/bin where they are under a 

non-default-install). The remainder of this chapter will assume that QLogic MPI 

was installed in the default location of /usr/mpi/qlogic and that mpi-selector 

is used to choose the MPI to be used. The source is installed under 

/usr/mpi/qlogic/share/mpich/examples/performance. 

The following examples are intended to show only the syntax for invoking these 

programs and the meaning of the output. They are not representations of actual 

TrueScale performance characteristics. 

Benchmark 1: Measuring MPI Latency Between 

Two Nodes 

In the MPI community, latency for a message of given size is the time difference 

between a node program’s calling MPI_Send and the time that the corresponding 

MPI_Recv in the receiving node program returns. The term latency, alone without 

a qualifying message size, indicates the latency for a message of size zero. This 

latency represents the minimum overhead for sending messages, due to both 

software overhead and delays in the electronics of the fabric. To simplify the 

timing measurement, latencies are usually measured with a ping-pong method, 

timing a round-trip and dividing by two. 

The program osu_latency, from Ohio State University, measures the latency for 

a range of messages sizes from 0 to 4 megabytes. It uses a ping-pong method, 

where the rank zero process initiates a series of sends and the rank one process 

echoes them back, using the blocking MPI send and receive calls for all 

operations. Half the time interval observed by the rank zero process for each 

exchange is a measure of the latency for messages of that size, as previously 

D000046-005 B B-1

B–Benchmark Programs 

Benchmark 1: Measuring MPI Latency Between Two Nodes 

defined. The program uses a loop, executing many such exchanges for each 

message size, to get an average. The program defers the timing until the 

message has been sent and received a number of times, to be sure that all the 

caches in the pipeline have been filled. 

This benchmark always involves two node programs. It can be run with the 

command: 

$ mpirun -H host1,host2 osu_latency 

-H (or --hosts) allows the specification of the host list on the command line 

instead of using a host file (with the -m or -machinefile option). Since only two 

hosts are listed, this implies that two host programs will be started (as if -np 2 

were specified). The output of the program looks like: 

# OSU MPI Latency Test (Version 2.0) 

# Size Latency (us) 

0 1.06 

1 1.06 

2 1.06 

4 1.05 

8 1.05 

16 1.30 

32 1.33 

64 1.30 

128 1.36 

256 1.51 

512 1.84 

1024 2.47 

2048 3.79 

4096 4.99 

8192 7.28 

16384 11.75 

32768 20.57 

65536 58.28 

131072 98.59 

262144 164.68 

524288 299.08 

1048576 567.60 

2097152 1104.50 

4194304 2178.66 

The first column displays the message size in bytes. The second column displays 

the average (one-way) latency in microseconds. This example shows the syntax of 

the command and the format of the output, and is not meant to represent actual 

values that might be obtained on any particular TrueScale installation. 

B-2 D000046-005 B


Benchmark 2: Measuring MPI Bandwidth Between Two Nodes 

Benchmark 2: Measuring MPI Bandwidth 

Between Two Nodes 

The osu_bw benchmark measures the maximum rate that you can pump data 

between two nodes. This benchmark also uses a ping-pong mechanism, similar to 

the osu_latency code, except in this case, the originator of the messages 

pumps a number of them (64 in the installed version) in succession using the 

non-blocking MPI_I send function, while the receiving node consumes them as 

quickly as it can using the non-blocking MPI_Irecv function, and then returns a 

zero-length acknowledgement when all of the sent data has been received. 

You can run this program by typing: 

$ mpirun -H host1,host2 osu_bw 

Typical output might look like: 

# OSU MPI Bandwidth Test (Version 2.0) 

# Size Bandwidth (MB/s) 

1 3.549325 

2 7.110873 

4 14.253841 

8 28.537989 

16 42.613030 

32 81.144290 

64 177.331433 

128 348.122982 

256 643.742171 

512 1055.355552 

1024 1566.702234 

2048 1807.872057 

4096 1865.128035 

8192 1891.649180 

16384 1898.205188 

32768 1888.039542 

65536 1931.339589 

131072 1942.417733 

262144 1950.374843 

524288 1954.286981 

1048576 1956.301287 

2097152 1957.351171 

4194304 1957.810999 

The increase in measured bandwidth with the messages’ size is because the 

latency’s contribution to the measured time interval becomes relatively smaller. 

D000046-005 B B-3


Benchmark 3: Messaging Rate Microbenchmarks 

Benchmark 3: Messaging Rate Microbenchmarks 

mpi_multibw is the microbenchmark that highlights QLogic’s messaging rate 

results. This benchmark is a modified form of the OSU Network-Based Computing 

Lab’s osu_bw benchmark (as shown in the previous example). It has been 

enhanced with the following additional functionality: 

• The messaging rate and the bandwidth are reported. 

• N/2 is dynamically calculated at the end of the run. 

• You can run multiple processes per node and see aggregate bandwidth and 

messaging rates. 

The benchmark has been updated with code to dynamically determine what 

processes are on which host. Here is an example output when running 

mpi_multibw: 

$ mpirun -np 16 -ppn 8 -H host1,host2 ./mpi_multibw 

This will run on eight processes per node. Typical output might look like: 

# PathScale Modified OSU MPI Bandwidth Test 

(OSU Version 2.2, PathScale $Revision$) 

# Running on 8 procs per node (uni-directional traffic for each 

process pair) 

# Size Aggregate Bandwidth (MB/s) Messages/s 

1 26.890668 26890667.530474 

2 53.692685 26846342.327320 

4 107.662814 26915703.518342 

8 214.526573 26815821.579971 

16 88.356173 5522260.840754 

32 168.514373 5266074.141949 

64 503.086611 7860728.303972 

128 921.257051 7197320.710406 

256 1588.793989 6206226.519112 

512 1716.731626 3352991.457783 

1024 1872.073401 1828196.680564 

2048 1928.774223 941784.288727 

4096 1928.763048 470889.416123 

8192 1921.127830 234512.674597 

16384 1919.122008 117133.911629 

32768 1898.415975 57935.057817 

65536 1953.063214 29801.379615 

131072 1956.731895 14928.679615 

262144 1957.544289 7467.438845 

524288 1957.952782 3734.498562 

1048576 1958.235791 1867.519179 

2097152 1958.333161 933.806019 

4194304 1958.400649 466.919100 

B-4 D000046-005 B


Benchmark 4: Measuring MPI Latency in Host Rings 

Searching for N/2 bandwidth. Maximum Bandwidth of 1958.400649 

MB/s... 

Found N/2 bandwidth of 992.943275 MB/s at size 153 bytes 

Benchmark 4: Measuring MPI Latency in Host 

Rings 

The program mpi_latency measures latency in a ring of hosts. Its syntax is 

different from Benchmark 1 in that it takes command line arguments that let you 

specify the message size and the number of messages to average the results. For 

example, running on the nodes, host1, host2, host3 and host4, the command: 

$ mpirun -np 4 -H host[1-4] mpi_latency 100 0 

Might produce output like this: 

0 1.760125 

This output indicates that it took an average of 1.76 microseconds per hop to send 

a zero-length message from the first host, to the second, to the third, to the fourth, 

and then receive replies back in the other direction. 

D000046-005 B B-5


Benchmark 4: Measuring MPI Latency in Host Rings 

Notes 

B-6 D000046-005 B

C 

VirtualNIC Interface 

Configuration and 

Administration 

VirtualNIC Interface Configuration and 

Administration 

The VirtualNIC (VNIC) Upper Layer Protocol (ULP) works in conjunction with 

firmware running on Virtual Input/Output (VIO) hardware such as the QLogic 

Ethernet Virtual I/O Controller (EVIC) or the InfiniBand/Ethernet Bridge Module 

for IBM ® BladeCenter ® , providing virtual Ethernet connectivity. 

The VNIC driver, along with QLogic EVIC’s two 10 Gigabit ethernet ports, enables 

Infiniband clusters to connect to Ethernet networks. This driver also works with the 

earlier version of the I/O controller, the VEx. 

The QLogic VNIC driver creates virtual Ethernet interfaces and tunnels the 

Ethernet data to/from the EVIC over InfiniBand using an InfiniBand reliable 

connection. 

The virtual Ethernet interface supports any Ethernet protocol. It operates like any 

other interface: ping, ssh, scp, netperf, etc. 

The VNIC interface must be configured before it can be used. Perform the steps in 

the following sub-sections to set up and configure the VNIC interface: 

Getting Information about Ethernet IOCs on the Fabric 

When ib_qlgc_vnic_query is executed without any options, it displays detailed 

information about all Virtual I/O IOCs present on the fabric including the EVIC/VEx 

Input/Output Controllers (IOCs) present on the fabric. 

For writing the configuration file, you will need information about the EVIC/VEx 

IOCs present on the fabric, such as their IOCGUID, IOCSTRING, etc. 

D000046-005 B C-1

C–VirtualNIC Interface Configuration and Administration 


NOTE: 

An EVIC has 2 IOCs; one for each Ethernet port. Each EVIC contains a 

unique set of IOCGUIDs: (e.g., IOC 1 maps to Ethernet Port 1 and IOC 2 

maps to Ethernet Port 2). 

1. Ensure you are logged in as a root user. 

2. Type ib_qlgc_vnic_query 

This displays detailed information about all the EVIC/VEx IOCs present on 

the fabric. For example: 

# ib_qlgc_vnic_query 

HCA No = 0, HCA = mlx4_0, Port = 1, Port GUID = 0x0002c903000010f9, 

State = Active 

IO Unit Info: 

port LID: 0009 

port GID: fe8000000000000000066a11de000070 

change ID: 0003 

max controllers: 0x02 

controller[ 1] 

GUID: 00066a01de000070 

vendor ID: 00066a 

device ID: 000030 

IO class : 2000 

ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1 

service entries: 2 

service[ 0]: 1000066a00000001 / 

InfiniNIC.InfiniConSys.Control:01 

service[ 1]: 1000066a00000101 / 

InfiniNIC.InfiniConSys.Data:01 

IO Unit Info: 

port LID: 000b 

port GID: fe8000000000000000066a21de000070 



C-2 D000046-005 B




GUID: 00066a02de000070 






service[ 0]: 1000066a00000002 / 


service[ 1]: 1000066a00000102 / 


HCA No = 0, HCA = mlx4_0, Port = 2, Port GUID = 0x0002c903000010fa, 

State = Active 

IO Unit Info: 


port GID: fe8000000000000000066a11de000070 




GUID: 00066a01de000070 






service[ 0]: 1000066a00000001 / 


service[ 1]: 1000066a00000101 / 


IO Unit Info: 

port LID: 000b 

port GID: fe8000000000000000066a21de000070 



D000046-005 B C-3




GUID: 00066a02de000070 






service[ 0]: 1000066a00000002 / 


service[ 1]: 1000066a00000102 / 


Editing the VirtualNIC Configuration file 

Look at the qlgc_vnic.cfg.sample file to see how VNIC configuration files 

are written. This file can be found with the OFED documentation, or in the 

qlgc_vnictools subdirectory of the QLogic OFED+ Host Software 

download. You can use this configuration file as the basis for creating a 

configuration file by replacing the destination global identifier (DGID), 

IOCGUID, and IOCSTRING values with those of the EVIC/VEx IOCs 

present on your fabric. 

QLogic recommends using the DGID of the EVIC/VEx IOC, as it ensures the 

quickest startup of the VNIC service. When DGID is specified, the IOCGUID 

must also be specified. For more details, see the qlgc_vnic.cfg sample 

file. 

1. Edit the VirtualNIC configuration file, /etc/infiniband/qlgc_vnic.cfg. 

For each IOC connection, add a CREATE block to the file using the following 

format: 

{CREATE; NAME="eioc2"; 

PRIMARY={IOCGUID=0x66A0130000105; INSTANCE=0; PORT=1; } 

SECONDARY={IOCGUID=0x66A013000010C; INSTANCE=0; PORT=2;} 

} 

NOTE: 

The qlgc_vnic.cfg file is case and format sensitive. 

C-4 D000046-005 B



NOTE: 

For the following sections, determine the necessary connection type. To 

always have a host connect to the same IOC on the same VIO card 

regardless of where that card is in the fabric, use format 1. To always have a 

host to connect to the same IOC in the same chassis and/or slot, use format 

2. 

Format 1: Defining an IOC using the IOCGUID 

NOTE: 

Determine the needed connection type. If a host is to connect to the same 

IOC on the same card, no matter where that card is in the fabric, then use 

this format. 

Use the following format to cause the host to connect to a specific VIO hardware 

card, regardless of which chassis and/or slot the VIO hardware card resides: 

{CREATE; 

NAME="eioc1"; 

IOCGUID=0x66A0137FFFFE7;} 

} 

The following is an example of VIO hardware failover: 


PRIMARY={IOCGUID=0x66a01de000003; INSTANCE=1; PORT=1; } 

SECONDARY={IOCGUID=0x66a02de000003; INSTANCE=1; PORT=1;} 

} 

NOTE: 

Do not create EIOC names with similar character strings (for example, 

eioc3 and eioc30). There is a limitation with certain Linux operating 

systems that cannot recognize the subtle differences. The result is that 

the user will be unable to ping across the network. 

D000046-005 B C-5



Format 2: Defining an IOC using the IOCSTRING 

Defining the IOC using the IOCSTRING allows VIO hardware to be hot-swapped 

in and out of a specific slot. The host will attempt to connect to the specified IOC 

(1 or 2) on the VIO hardware that currently resides in the specified slot of the 

specified Chassis. Use the following format to allow the host to connect to a VIO 

hardware that resides in a specific slot of a specific chassis: 

{CREATE; 

NAME="eioc1"; 

IOCSTRING="Chassis 0x00066A0005000001, Slot 1, IOC 1"; 

RX_CSUM=TRUE; 

HEARTBEAT=100; } 

NOTE: 

The IOCSTRING field is a literal, case-sensitive string, whose syntax must 

be exactly in the format shown in the example above, including the 

placement of commas. To reduce the likelihood of syntax error, the user 

should execute ib_qlgc_vnic_query -es. Note that the chassis serial 

number must match the chassis 0x (Hex) value. The slot serial number is 

specific to the line card as well. 

Each CREATE block must specify a unique NAME. The NAME represents the Ethernet 

interface name that will be registered with the Linux Operating System. 

Format 3: Starting VNIC using DGID 

NOTE: 

It is not always necessary to use DGID. It is recommended to use DGID if 

the user has a large cluster. 

Following is an example of a DGID and IOCGUID VNIC configuration. This 

configuration allows for the quickest start up of VNIC service: 


DGID=0xfe8000000000000000066a0258000001;IOCGUID=0x66a01300000 

01; 

} 

This example uses DGID, IOCGUID and IOCSTRING: 


DGID=0xfe8000000000000000066a0258000001; 

IOCGUID=0x66a0130000001; 

IOCSTRING="Chassis 0x00066A00010003F2, Slot 1, IOC 1"; 

} 

C-6 D000046-005 B



VirtualNIC Failover Definition 

The VirtualNIC configuration file allows the user to define virtual NIC failover as: 

1. Failover to a different adapter port on the same adapter. 

2. Failover to a port on a different adapter. 

3. Failover to a different Ethernet port on the same Ethernet gateway. 

4. Failover to a port on a different Ethernet gateway. 

5. A combination of scenarios 1 or 2 and 3 or 4. 

Failover to a different Adapter port on the same Adapter 

In this example, if adapter Port 1 fails (i.e., the InfiniBand connection between the 

adapter and the InfiniBand switch is lost) then traffic going over eioc1 will begin 

using adapter port 2. In this example the same Ethernet gateway port is being 

used for both the primary and the secondary connection. 


PRIMARY={ DGID=fe8000000000000000066a11de00003c; 

IOCGUID=00066a01de00003c; INSTANCE=0; PORT=1; } 

SECONDARY={ DGID=fe8000000000000000066a11de00003c; 


} 

Failover to a different Ethernet port on the same 

Ethernet gateway 

In this example, if the Ethernet port associated with iocguid 0x66a01de00003c 

(i.e., Ethernet port 1 on gateway 03c) fails, then traffic going over eioc1 will begin 

using Ethernet port 2 on the same Ethernet gateway. 

NOTE: 

In this example the same adapter port is being used for both the primary and 

the secondary connection. 


PRIMARY={ DGID=fe8000000000000000066a11de00003c; 




} 

D000046-005 B C-7



Failover to a port on a different Ethernet gateway 

In this example, if the Ethernet port associated with iocguid 0x66a02de000048 

(i.e., Ethernet port 2 on gateway 48) fails, then traffic going over eioc1 will begin 

using Ethernet port 2 on gateway 03c. This type of failover allows traffic to 

continue even if the entire gateway card fails or is rebooted. This type of failure 

requires a second gateway card to be in the fabric. 

NOTE: 

In this example the same adapter port is being used for both the primary and 

the secondary connection. 


PRIMARY={ DGID=fe8000000000000000066a21de000048; 

IOCGUID=00066a02de000048; INSTANCE=0; PORT=1; } 



} 

Combination method 

In this example, if either the Ethernet port, the Ethernet gateway or the adapter 

port fails, traffic using eioc1 will begin using Ethernet port 2 on gateway 03c via 

adapter port 1. 


PRIMARY={ DGID=fe8000000000000000066a21de000048; 

IOCGUID=00066a02de000048; INSTANCE=0; PORT=2; } 



} 

Creating VirtualNIC Ethernet Interface Configuration Files 

For each Ethernet interface defined in the /etc/infiniband/qlgc_vnic.cfg 

file, create an interface configuration file: 

For SuSE and SLES OS: 

/etc/sysconfig/network/ifcfg-NAME 

For RHEL OS: 

/etc/sysconfig/network-scripts/ifcfg-NAME 

where NAME is the value of the NAME field specified in the CREATE block. 

C-8 D000046-005 B



Example of ifcfg-eiocx setup for RedHat systems: 

DEVICE=eioc1 


IPADDR=172.26.48.132 

BROADCAST=172.26.63.130 

NETMASK=255.255.240.0 

NETWORK=172.26.48.0 

ONBOOT=yes 

TYPE=Ethernet 

Example of ifcfg-eiocx setup for SuSE and SLES systems: 

BOOTPROTO='static' 

IPADDR='172.26.48.130' 

BROADCAST='172.26.63.255' 

NETMASK='255.255.240.0' 

NETWORK='172.26.48.0' 

STARTMODE='hotplug' 

TYPE='Ethernet' 

After modifying the /etc/infiniband/qlgc_vnic.cfg file, restart the 

VirtualNIC driver with the following: 

/etc/init.d/qlgc_vnic restart 

VirtualNIC Multicast 

The primary goal of VNIC Multicast is to reduce the replication of multicast packet 

transmission from the EVIC to InfiniBand hosts. Figure C-1 demonstrates the 

transmission of multicast traffic without using IB_MULTICAST: 

D000046-005 B C-9



IB Host 1 


IB Switch 

EVIC 


Multicast 

Data (RC) 

Figure C-1. Without IB_Multicast 

Figure C-1 is showing that each multicast packet is sent by the EVIC via the 

Reliable Connect (RC) Queue Pair to each and every host. In the case when 

multicast applications on multiple hosts are sharing the same multicast address, 

the result is large amounts of traffic, one per host, between the EVIC and the 

switch. This also creates additional work load for the EVIC. 

With multicast enabled at the host using IB_MULTICAST (located in the VNIC 

configuration file) and also enabled at the EVIC, multicast traffic is handled as 

follows: 

C-10 D000046-005 B





IB Switch 

EVIC 


Single packet to 

IB_multicast group 

traffic 

Packet forwarded to 

host (UD) 

Figure C-2. With IB_Multicast 

Figure C-2 is showing that the EVIC sends a multicast packet ONCE to the 

InfiniBand switch while posting to a specific InfiniBand multicast group. The switch 

then forwards the packet to all hosts who have joined the same multicast group 

that the host has joined. Since the EVIC only has to post the packet once instead 

of once for each host, there is a substantial savings. Additionally, the packets are 

delivered by an unreliable datagram (UD) queue pair, which means there is less 

overhead per packet, per host. 

For each IOC, the EVIC creates a unique multicast group. When the VNIC driver 

provides multicast addresses for a virtual port (i.e., viport) to the EVIC, the EVIC 

returns the multicast group for the IOC used by the viport. The VNIC driver then 

joins the specified multicast group. When a viport connection is taken down, the 

host leaves the multicast group. If an EVIC has two IOCs and the VNIC driver on a 

host is configured to create viports using both IOCs, a join is issued by each 

viport to join the corresponding IOC it is using. 

D000046-005 B C-11



NOTE: 

When a viport is using two different IOCs for primary and secondary paths, 

then the host will join the multicast group for the IOC on the primary path as 

well as the multicast group for the IOC on the secondary path. The 

secondary path and the corresponding multicast group will not be used until 

a failover occurs. 

The create multicast group issued by the EVIC, along with the join multicast 

group and leave multicast group requests issued by the host are handled by the 

subnet manager. For details, refer to the Fabric Manager Users Guide. 

On the EVIC, IB_MULTICAST can be enabled or disabled manually using the CLI. 

For details, refer to the Ethernet section of the Hardware CLI Reference Guide. A 

reboot of the EVIC is required after IB_MULTICAST has been enabled or 

disabled. 

On the host, IB_MULTICAST is enabled by default. The user must add 

IB_MULTICAST=FALSE to the VNIC configuration file to disable the feature. To view 

the current status of the feature do the following: 

cat 

/sys/class/infiniband_qlgc_vnic/interfaces//* 

path/multicast_state 

For each path, there is a primary_path and a secondary_path directory present 

and the contents of the multicast_state file contain: 

feature not enabled - feature is disabled at either host or 

EVIC or both ends 

state=Joined & Attached MGID: MLID: 

To disable IB_MULTICAST at the host edit the /etc/infiniband/qlgc_vnic.cfg 

file to add IB_MULTICAST=FALSE for the viport inside both Primary and 

Secondary. 

Starting, Stopping and Restarting the VirtualNIC Driver 

Once you have created a configuration file, you can start the VNIC driver and 

create the VNIC interfaces specified in the configuration file. 

NOTE: 

Ensure you are logged in as a root user to start, stop, or restart the VNIC 

driver. 

C-12 D000046-005 B



To start the qlgc_vnic driver and the QLogic VNIC interfaces, use the following 

command: 

/etc/init.d/qlgc_vnic start 

To stop the qlgc_vnic driver and bring down the VNIC interfaces, use the following 

command: 

/etc/init.d/qlgc_vnic stop 

To restart the qlgc_vnic driver, use the following command: 


If you have not started the InfiniBand network stack (QLogic OFED+ or OFED), 

then running the /etc/init.d/qlgc_vnic start command also starts the 

InfiniBand network stack, since the QLogic VNIC service requires the InfiniBand 

stack. 

If you start the InfiniBand network stack separately, then the correct starting order 

is: 

• Start the InfiniBand stack. 

• Start QLogic VNIC service. 

For example, if you use QLogic OFED+ Host Software, the correct starting order 

is: 

/etc/init.d/openibd start 

/etc/init.d/qlgc_vnic start 

If you want to restart the QLogic VNIC interfaces, run the following command: 


You can get information about the QLogic VNIC interfaces by using the following 

script (as a root user): 

ib_qlgc_vnic_info 

This information is collected from the 

/sys/class/infiniband_qlgc_vnic/interfaces/ directory, where there is a 

separate directory corresponding to each VNIC interface. 

VNIC interfaces can be deleted by writing the name of the interface to the 

/sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic file. For 

example, to delete interface veth0, run the following command (as a root user): 

echo -n veth0 > 

/sys/class/infiniband_qlgc_vnic/interfaces/delete_vnic 

D000046-005 B C-13



Link Aggregation Configuring 


To configure link aggregation for all ports of a VIO hardware card, do the following: 

1. Modify the chassis GUID and slot number. 

2. Edit /etc/infiniband/qlgc_vnic.cfg file with information similar to 

the following: 


PRIMARY={IOCSTRING="Chassis 0x00066A0050000018, Slot 

2, IOC 1"; } 

SECONDARY={IOCSTRING="Chassis 0x00066A0050000018, Slot 

2, IOC 2"; } 

} 

3. Create ifcfg-eioc1 file in directory 

/etc/sysconfig/network-scripts. 

4. Physically connect 2 or 3 ports of a VEx to an Ethernet switch. 

5. Using the management interface for the Ethernet switch, configure the 

switch to aggregate all 3 ports that connect to the VEx. 

6. Restart the qlgc_vnic module with the following command: 


Refer to Appendix G for information about troubleshooting. 

VirtualNIC Configuration Variables 

NAME 

The device name for the interface. 

IOCGUID 

Defines the port controller GUID to be used to identify the specific EVIC leaf 

and port used by the VNIC. All of the IOC GUIDS detected on the fabric can 

be found by typing ib_qlgc_vnic_query -e and noting the value of the 

reported IOC GUIDs. This can be used instead of IOCSTRING. 

IOCSTRING 

Defines the IOC Profile ID String of the IOC to be used. All of the IOC Profile 

ID Strings detected on the fabric can be found by typing 

ib_qlgc_vnic_query -s and noting the value of the reported string. This 

can be used instead of IOCGUID. 

C-14 D000046-005 B



DGID 

Defines the Destination Global Identifier (DGID) of the IOC to be used. This 

parameter should be used in all CREATE blocks. The ib_qlgc_vnic_query 

command will show all the DGIDs that the host can see. The DGID 

parameter is identified as dgid in the output of the ib_qlgc_vnic_query 

-e command. 

INSTANCE 

Defaults to 0. The range is 0-255. If a host connects to the same IOC more 

than once, each connection must be assigned a unique instance. 

RX_CSUM 

Defaults to TRUE. When true, RX_CSUM indicates that the receive 

checksum should be verified by the VIO hardware. 

HEARTBEAT 

Defaults to 100. Specifies the time in 1/100 of a second between the 

heartbeats that occur between a host and an EVIC. 

PORT 

Alternative specification for a local adapter port. The first port is 1. 

NOTE: 

If there is only one adapter in the system, then use PORT/HCA to 

specify the adapter port to be used. If there is more than one adapter in 

the system then it is recommended to use PortGuid to specify the 

adapter port to be used. 

PORTGUID 

The PORTGUID of the InfiniBand port to be used. Use of the PORTGUID 

parameter for configuring the VNIC interface has an advantage on hosts 

having more than 1 Host Channel Adapter. PORTGUID is persistent for a 

given InfiniBand port, meaning VNIC configurations should be consistent 

and reliable - unaffected by restarts of the OFED InfiniBand stack on host 

having more than 1 adapter. 

HCA 

An optional Host Channel Adapter specification for use with the PORT 

specification. The first adapter found by the sysem upon boot is HCA 0. If 

there is only one adapter in the system, then the default value of this 

parameter (0) can be used, so "HCA" does not need to be specified in the 

qlgc_vnic.cfg file. If there is more than one adapter in the system, then 

QLogic recommends using the PORTGUID parameter to specify an adapter 

port (instead of using HCA/PORT) since there is no guarantee that a system 

booting up always finds the same adapter first. 

D000046-005 B C-15



IB_MULTICAST 

Defaults to TRUE. The InfiniBand multicast parameter can be set to either 

TRUE or FALSE. When the parameter is set to true, it will take effect only if 

the corresponding CLI command (ethVirtVnic2Mcastset) is executed on 

the EVIC. This command is supported on EVIC FW 4.3 and above. 

C-16 D000046-005 B

D 

SRP Configuration 

SRP Configuration Overview 

SRP stands for SCSI RDMA Protocol. It allows the SCSI protocol to run over 

InfiniBand for Storage Area Network (SAN) usage. SRP interfaces directly to the 

Linux file system through the SRP Upper Layer Protocol (ULP). SRP storage can 

be treated as another device. 

In this release, two versions of SRP are available: QLogic SRP and OFED SRP. 

QLogic SRP is available as part of the QLogic OFED Host Software, QLogic 

InfiniBand Fabric Suite, Rocks Roll, and Platform PCM downloads. 

SRP has been tested on targets from DataDirect Networks and Engenio (now 

LSI Logic ® ). 

NOTE: 

Before using SRP, the SRP targets must already be set up by your system 

administrator. 

Important Concepts 

• A SRP Initiator Port is a adapter port through which the host communicates 

with a SRP target device (e.g., a Fibre Channel disk array) via a SRP target 

port. 

• A SRP Target Port is an IOC of the VIO hardware. In the context of VIO 

hardware, an IOC can be thought of as a SRP target. An FVIC contains 2 

IOCs. IOC1 maps to the first adapter on the FVIC, and IOC2 maps to the 

2nd adapter on the FVIC. On an FCBM, there are also 2 IOCs, and IOC1 

maps to port 1 of the adapter of the FC BM and IOC2 maps to port 2 of the 

adapter of the FC BM. 

• A Fibre Channel Target Device is a device containing storage resources that 

is located remotely from a Fibre Channel host. In the context of SRP/VIO 

hardware, this is typically an array of disks connected via Fibre Channel to 

the VIO hardware. 

D000046-005 B D-1

D–SRP Configuration 

QLogic SRP Configuration 

• A SRP Initiator Extension is a 64-bit numeric value that is appended to the 

port GUID of the SRP initiator port, which allows an SRP initiator port to 

have multiple SRP maps associated with it. Maps are for FVIC only. 

InfiniBand attached storage will use their own mechanism as maps are not 

necessary. 

• A SRP Initiator is the combination of an SRP initiator port and an SRP 

initiator extension. 

• A SRP Target is identified by the combination of an SRP target IOC and a 

SRP target extension. 

• A SRP Session defines a connection between an SRP initiator and a SRP 

target. 

• A SRP Map associates an SRP session with a Fibre Channel Target Device. 

This mapping is configured on the VIO hardware. Maps are for FVIC only. 

InfiniBand attached storage will use their own mechanism as maps are not 

necessary. 

NOTE: 

• If a device connected to a map is changed, the SRP driver must 

be restarted. 

• If the connected device is unreachable for a period of time, the 

Linux kernel may set the device offline. If this occurs the SRP 

driver must be restarted. 

• A SRP Adapter is a collection of SRP sessions. This collection is then 

presented to the Linux kernel as if those sessions were all from a single 

adapter. All sessions configured for an adapter must ultimately connect to 

the same target device. 

NOTE: 

• The SRP driver must be stopped before OFED (i.e., openibd) is 

stopped or restarted. This is due to SRP having references on 

OFED modules. The Linux kernel will not allow those OFED 

modules to be unloaded. 


The QLogic SRP is installed as part of the QLogic OFED+ Host Software or the 

QLogic InfiniBand Fabric Suite. The following section provide procedures to set up 

and configure the QLogic SRP. 

D-2 D000046-005 B



Stopping, Starting and Restarting the SRP Driver 

To stop the qlgc_srp driver, use the following command: 

/etc/init.d/qlgc_srp stop 

To start the qlgc_srp driver, use the following command: 

/etc/init.d/qlgc_srp start 

To restart the qlgc_srp driver, use the following command: 

Specifying a Session 

/etc/init.d/qlgc_srp restart 

In the SRP configuration file, a session command is a block of configuration 

commands, surrounded by begin and end statements. Sessions can be specified 

in several different ways, but all consist of specifying an SRP initiator and an SRP 

target port. For example: 

session 

begin 

card: 0 

port: 1 

targetIOCGuid: 0x00066AXXXXXXXXXX 

initiatorExtension: 2 

end 

The session command has two parts; the part that specifies the SRP initiator and 

the part that specifies the SRP target port. The SRP initiator contains two parts, 

the SRP initiator port and the SRP initiator extension. The SRP initiator extension 

portion of the SRP initiator is optional, and defaults to a value of 1. However, if a 

SRP initiator extension is not specified, each port on the adapter can use only one 

SRP map per VIO device. In addition a targetExtension can be specified (the 

default is 1). 

The SRP Initiator Port may be specified in two different ways: 

1. By using the port GUID of the adapter port used for the connection, or 

2. Specify the index of the adapter card being used (this is zero-based, so if 

there is only one adapter card in the system use a value of 0) and the index 

of the port number (1 or 2) of the adapter card being used. 

The SRP target port may be specified in two different ways: 

D000046-005 B D-3



1. By the port GUID of the IOC, or 

2. By the IOC profile string that is created by the VIO device (i.e., a string 

containing the chassis GUID, the slot number and the IOC number). FVIC 

creates the device in this manner, other devices have their own naming 

method. 

To specify the host InfiniBand port to use, the user can either specify the port 

GUID of the local InfiniBand port, or simply use the index numbers of the cards 

and the ports on the cards. Cards are numbered from 0 on up, based on the order 

they occur in the PCI bus. Ports are numbered in the same way, from first to last. 

To see which cards and ports are available for use, type the following command: 

ib_qlgc_srp_query 

The system returns input similar to the following: 

D-4 D000046-005 B



st187:~/qlgc-srp-1_3_0_0_1 # ib_qlgc_srp_query 

QLogic Corporation. Virtual HBA (SRP) SCSI Query Application, version 1.3.0.0.1 

1 IB Host Channel Adapter present in system. 

HCA Card 0 : 0x0002c9020026041c 

Port 1 GUID : 0x0002c9020026041d 

Port 2 GUID : 0x0002c9020026041e 

SRP Targets : 

SRP IOC Profile : FVIC in Chassis 0x00066a000300012a, Slot 17, Ioc 1 

SRP IOC GUID : 0x00066a01dd000021 

SRP IU SIZE : 320 

SRP IU SG SIZE: 15 

SRP IO CLASS : 0xff00 

service 0 : name SRP.T10:0000000000000001 id 0x0000494353535250 




Target Path(s): 

HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a11dd000021 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a11dd000021 

SRP IOC Profile : FVIC in Chassis 0x00066a000300012a, Slot 17, Ioc 2 










HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a21dd000021 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a21dd000021 

SRP IOC Profile : Chassis 0x00066A0050000135, Slot 5, IOC 1 

SRP IOC GUID : 0x00066a013800016c 









HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a026000016c 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a026000016c 

D000046-005 B D-5




SRP IOC GUID : 0x00066a023800016c 









HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a026000016c 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a026000016c 


SRP IOC GUID : 0x00066a0138000174 






HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a0260000174 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a0260000174 


SRP IOC GUID : 0x00066a0238000174 






HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 0xfe8000000000000000066a0260000174 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 0xfe8000000000000000066a0260000174 

st187:~/qlgc-srp-1_3_0_0_1 # 

Determining the values to use for the configuration 

In order to build the configuration file, use the command 

ib_qlgc_srp_build_cfg script as follows: 

Enter ib_qlgc_srp_build_cfg. The system provides output similar to the 

following: 

D-6 D000046-005 B



# qlgc_srp.cfg file generated by /usr/sbin/ib_qlgc_srp_build_cfg, version 1.3.0.0.17, on 

Mon Aug 25 13:42:16 EDT 2008 

#Found QLogic OFED SRP 

registerAdaptersInOrder: ON 

# ============================================================= 

# IOC Name: BC2FC in Chassis 0x0000000000000000, Slot 6, Ioc 1 

# IOC GUID: 0x00066a01e0000149 SRP IU SIZE : 320 

# service 0 : name SRP.T10:0000000000000001 id 0x0000494353535250 

session 

begin 

card: 0 

port: 1 

#portGuid: 0x0002c9030000110d 


targetIOCGuid: 0x00066a01e0000149 

targetIOCProfileIdString: "FVIC in Chassis 0x0000000000000000, Slot 6, Ioc 1" 

targetPortGid: 0xfe8000000000000000066a01e0000149 

targetExtension: 0x0000000000000001 

SID: 0x0000494353535250 

IOClass: 0xff00 

end 

adapter 

begin 

adapterIODepth: 1000 

lunIODepth: 16 

adapterMaxIO: 128 

adapterMaxLUNs: 512 

adapterNoConnectTimeout: 60 

adapterDeviceRequestTimeout: 2 

# set to 1 if you want round robin load balancing 

roundrobinmode: 0 

# set to 1 if you do not want target connectivity verification 

noverify: 0 

description: "SRP Virtual HBA 0" 

end 

The ib_qlgc_srp_build_cfg command creates a configuration file based on 

discovered target devices. By default, the information is sent to stdout. In order 

to create a configuration file, output should be redirected to a disk file. Enter 

ib_qlgc_srp_build_cfg -h for a list and description of the option flags. 

D000046-005 B D-7



NOTE: 

The default configuration generated by ib_qlgc_srp_build_cfg for OFED 

is similar to the one generated for the QuickSilver host stack with the 

following differences: 

• For OFED, the configuration automatically includes IOClass 

• For OFED, the configuration automatically includes SID 

• For OFED, the configuration provides information on targetPortGid 

instead of targetPortGuid 

• For OFED, the configuration automatically includes 

targetIOCProfileIdString. 

Specifying an SRP Initiator Port of a Session by Card and 

Port Indexes 

The following example specifies a session by card and port indexes. If the system 

contains only one adapter, use this method. 

session 

begin 

#Specifies the near side by card index 

card: 0 #Specifies first HCA 

port: 1 #Specifies first port 

targetIOCGuid: 0x00066013800016C 

end 

Specifying an SRP Initiator Port of Session by Port GUID 

The following example specifies a session by port GUID. If the system contains 

more than one adapter, use this method. 

session 

begin 

portGuid: 0x00066A00a00001a2 #Specifies port by its GUID 

targetIOCGuid: 0x00066A013800016C 

end 

NOTE: 

When using this method, if the port GUIDs are changed, they must also be 

changed in the configuration file. 

D-8 D000046-005 B



Specifying a SRP Target Port 

The SRP target can be specified in two different ways. To connect to a particular 

SRP target no matter where it is in the fabric, use the first method (By IOCGUID). 

To connect to a SRP target that is in a certain chassis/slot, no matter which card it 

is on (For FVIC, the user does not want to change the configuration, if cards are 

switched in a slot) then use the second method. 

1. By IOCGUID. For example: 

targetIOCGuid: 0x00066A013800016c 

2. By target IOC Profile String. For example: 

targetIOCProfileIdString: "FVIC in Chassis 

0x00066A005000010E, Slot 1, IOC 1" 

NOTE: 

When specifying the targetIOCProfileIdString, the string is case 

and format sensitive. The easiest way to get the correct format is to cut 

and paste it from the output of the /usr/sbin/ib_qlgc_srp_query 

program. 

NOTE: 

For FVIC, by specifying the SRP Target Port by IOCGUID, this ensures 

that the session will always be mapped to the specific port on this 

specific VIO hardware card, even if the card is moved to a different slot 

in the same chassis or even if it is moved to a different chassis. 

NOTE: 

For FVIC, by specifying the SRP Target Port by Profile String, this 

ensures that the session will always be mapped to the VIO hardware 

card in the specific slot of a chassis, even if the VIO hardware card 

currently in that slot is replaced by a different VIO hardware card. 

D000046-005 B D-9



Specifying a SRP Target Port of a Session by IOCGUID 

The following example specifies a target by IOC GUID: 

session 

begin 

card: 0 

port: 1 

targetIOCGuid: 0x00066A013800016c #IOC GUID of the InfiniFibre port 

end 

• 0x00066a10dd000046 

• 0x00066a20dd000046 

Specifying a SRP Target Port of a Session by Profile String 

The following example specifies a target by Profile String: 

Specifying an Adapter 

session 

begin 

card: 0 

port: 1 

# FVIC in Chassis 0x00066A005000010E, 

# Slot number 1, port 1 

targetIOCProfileIdString: “FVIC in Chassis 0x00066A005000010E, Slot 1, IOC 1” 

end 

An adapter is a collection of sessions. This collection is presented to the Linux 

kernel as if the collection was a single Fibre Channel adapter. The host system 

has no information regarding session connectivity. It only sees the end target fibre 

channel devices. The adapter section of the qlgc_srp configuration file contains 

multiple parameters. These parameters are listed in the adapter section of the 

ib_qlgc_srp_build_cfg script system output shown in “Determining the values 

to use for the configuration” on page D-6 The following example specifies an 

adapter: 

adapter 

begin 

description: “Oracle RAID Array” 

end 

Restarting the SRP Module 

For changes to take effect, including changes to the SRP map on the VIO card, 

SRP will need to be restarted. To restart the qlgc_srp driver, use the following 

command: 

/etc/init.d/qlgc_srp restart 

D-10 D000046-005 B



Configuring an Adapter with Multiple Sessions 

Each adapter can have an unlimited number of sessions attached to it. Unless 

round robin is specified, SRP will only use one session at a time. However, 

there is still an advantage to configuring an adapter with multiple sessions. For 

example, if an adapter is configured with only one session and that session fails, 

all SCSI I/Os on that session will fail and access to SCSI target devices will be 

lost. While the qlgc_srp module will attempt to recover the broken session, this 

may take some time (e.g., if a cable was pulled, the FC port has failed, or an 

adapter has failed). However, if the host is using an adapter configured with 

multiple sessions and the current session fails, the host will automatically switch 

to an alternate session. The result is that the host can quickly recover and 

continue to access the SCSI target devices. 

WARNING!! 

When using two VIO hardware cards within one Adapter, the cards must 

have identical Fibre Channel configurations and maps. Data corruption can 

result from using different configurations and/or maps. 

D000046-005 B D-11



When the qlgc_srp module encounters an adapter command, that adapter is 

assigned all previously defined sessions (that have not been assigned to other 

adapters). This makes it easy to configure a system for multiple SRP adapters. 

The following is an example configuration that uses multiple sessions and 

adapters: 

session 

begin 

card: 0 

port: 2 


0x00066A005000011D, Slot 1, IOC 1" 


end 

adapter 

begin 

description: "Test Device" 

end 

session 

begin 

card: 0 

port: 1 


0x00066A005000011D, Slot 2, IOC 1" 


end 

adapter 

begin 

description: "Test Device 1" 

end 

D-12 D000046-005 B



session 

begin 

card: 0 

port: 1 


0x00066A005000011D, Slot 1, IOC 2" 


end 

adapter 

begin 

description: "Test Device 1" 

end 

Configuring Fibre Channel Failover 

Fibre Channel failover is essentially failing over from one session in an adapter to 

another session in the same adapter. 

Following is a list of the different type of failover scenarios: 

• Failing over from one SRP initiator port to another. 

• Failing over from a port on the VIO hardware card to another port on the VIO 

hardware card. 

• Failing over from a port on a VIO hardware card to a port on a different VIO 

hardware card within the same virtual I/O chassis. 

• Failing over from a port on a VIO hardware card to a port on a different VIO 

hardware card in a different virtual I/O chassis. 

D000046-005 B D-13



Failover Configuration File 1: Failing over from one 

SRP Initiator port to another 

In this failover configuration file, the first session (using adapter Port 1) is used to 

reach the SRP Target Port. If a problem is detected in this session (e.g., the 

InfiniBand cable on port 1 of the adapter is pulled) then the 2nd session (using 

adapter Port 2) will be used. 

# service 0: name SRP.T10:0000000000000001 id 

0x0000494353535250 

session 

begin 

card: 0 

port: 1 

#portGuid: 0x0002c903000010f1 



targetIOCProfileIdString: "BC2FC in Chassis 

0x0000000000000000, Slot 6, Ioc 1" 



SID: 0x0000494353535250 


end 

session 

begin 

card: 0 

port: 2 

#portGuid: 0x0002c903000010f2 



targetIOCProfileIdString: "BC2FC in Chassis 

0x0000000000000000, Slot 6, Ioc 1" 



SID: 0x0000494353535250 


end 

adapter 

begin 






D-14 D000046-005 B






# set to 1 if you do not want target connectivity 

verification 

noverify: 0 


end 

Failover Configuration File 2: Failing over from a port on the 

VIO hardware card to another port on the VIO hardware card 

session 

begin 

card: 0 (InfiniServ HCA card number) 

port: 1 (InfiniServ HCA port number) 

targetIOCProfileIdString: "FVIC in Chassis , 

, " 


end 

session 

begin 




, " 

initiatorExtension: 1 (Here the extension should be different 

if using the same IOC in this adapter for FVIC, so that 

separate maps can be created for each session). 

end 

adapter 

begin 

description: “FC port Failover” 

end 

On the VIO hardware side, the following needs to be ensured: 

• The target device is discovered and configured for each of the ports that is 

involved in the failover. 

• The SRP Initiator is discovered and configured once for each different 

initiatorExtension. 

• Each map should use a different Configured Device, e.g Configured Device 

1 has the Target being discovered over FC Port 1, and Configured Device 2 

has the Target being discovered over FC Port 2.) 

D000046-005 B D-15




VIO hardware card to a port on a different VIO hardware card 

within the same Virtual I/O chassis 

session 

begin 




, " 


end 

session 

begin 




, " (Slot number differs to indicate a different 

VIO card) 

initiatorExtension: 1 (Here the initiator extension can be the 

same as in the previous definition, because the SRP map is 

being defined on a different FC gateway card) 

end 

adapter 

begin 

description: “FC Port Failover” 

end 

On the VIO hardware side, the following need to be ensured on each FVIC 

involved in the failover: 

• The target device is discovered and configured through the appropriate FC 

port 

• The SRP Initiator is discovered and configured once for the proper 


• The SRP map created for the initiator connects to the same target 

D-16 D000046-005 B




VIO hardware card to a port on a different VIO hardware 

card in a different Virtual I/O chassis 

session 

begin 




, " 


end 

session 

begin 




, " (Chassis GUID differs to indicate a card in a 

different chassis) 

initiatorExtension:1 (Here the initiator extension can be the 

same as in the previous definition, because the SRP map is 

being defined on a different FC gateway card) 

end 

adapter 

begin 

description: “FC Port Failover” 

end 

On the VIO hardware side, the following need to be ensured on each FVIC 

involved in the failover: 

• The target device is discovered and configured through the appropriate FC 

port 

• The SRP Initiator is discovered and configured once for the proper 


• The SRP map created for the initiator connects to the same target 

D000046-005 B D-17



Configuring Fibre Channel Load Balancing 

The following examples display typical scenarios for how to configure Fibre 

Channel load balancing. 

In the first example, traffic going to any Fibre Channel Target Device where both 

ports of the VIO hardware card have a valid map, are split between the two ports 

of the VIO hardware card. If one of the VIO hardware ports goes down, then all of 

the traffic will go over the remaining port that is up. 

1 Adapter Port and 2 Ports on a Single VIO 

session 

begin 

card: 0 

port: 1 


0x00066A0050000123, Slot 1, IOC 1" 


end 

session 

begin 

card: 0 

port: 1 


0x00066A0050000123, Slot 1, IOC 2" 


end 

adapter 

begin 



end 

D-18 D000046-005 B



2 Adapter Ports and 2 Ports on a Single VIO Module 

In this example, traffic is load balanced between adapter Port 2/VIO hardware 

Port 1 and adapter Port1/VIO hardware Port 1. If one of the sessions goes down 

(due to an InfiniBand cable failure or an FC cable failure), all traffic will begin using 

the other session. 

session 

begin 

card: 0 

port: 2 


0x00066A0050000123, Slot 1, IOC 1" 


end 

session 

begin 

card: 0 

port: 1 


0x00066A0050000123, Slot 1, IOC 2" 


end 

adapter 

begin 



end 

D000046-005 B D-19



Using the roundrobinmode Parameter 

In this example, the two sessions use different VIO hardware cards as well as 

different adapter ports. Traffic will be load-balanced between the two sessions. If 

there is a failure in one of the sessions (e.g., one of the VIO hardware cards is 

rebooted) traffic will begin using the other session. 

session 

begin 

card: 0 

port: 2 


0x00066A005000011D, Slot 1, IOC 1" 


end 

session 

begin 

card: 0 

port: 1 


0x00066A005000011D, Slot 2, IOC 1" 


end 

adapter 

begin 



end 

D-20 D000046-005 B



Configuring SRP for Native InfiniBand Storage 

1. Review ib_qlgc_srp_query. 

QLogic Corporation. Virtual HBA (SRP) SCSI Query Application, 

version 1.3.0.0.1 

1 IB Host Channel Adapter present in system. 

HCA Card 1 : 0x0002c9020026041c 

Port 1 GUID : 0x0002c9020026041d 

Port 2 GUID : 0x0002c9020026041e 

SRP Targets : 

SRP IOC Profile : Native IB Storage SRP Driver 





service 0 : name SRP.T10:0000000000000001 id 

0x0000494353535250 


0x0000494353535250 


0x0000494353535250 


0x0000494353535250 


HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 

0xfe8000000000000000066a11dd000021 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 

0xfe8000000000000000066a11dd000021 

2. Edit /etc/sysconfig/qlgc_srp.cfg to add this information. 

# service : name SRP.T10:0000000000000001 id 

0x0000494353535250 

session 

begin 

card: 0 

port: 1 

#portGuid: 0x0002c903000010f1 



D000046-005 B D-21



targetIOCProfileIdString: “Native IB Storage SRP Driver” 



SID: 0x0000494353535250 

IOClass: 0x0100 

end 

session 

begin 

card: 0 

port: 2 

#portGuid: 0x0002c903000010f2 



targetIOCProfileIdString: Native IB Storage SRP Driver 



SID: 0x0000494353535250 

IOClass: 0x0100 

end 

adapter 

begin 









# set to 1 if you do not want target connectivity 

verification 

noverify: 0 


end 

D-22 D000046-005 B



3. Note the correlation between the output of ib_qlgc_srp_query and 

qlgc_srp.cfg 


HCA 0 Port 1 0x0002c9020026041d -> Target Port GID 

0xfe8000000000000000066a11dd000021 

HCA 0 Port 2 0x0002c9020026041e -> Target Port GID 

0xfe8000000000000000066a11dd000021 

qlgc_srp.cfg: 

session 

begin 

. . . . 

targetIOCGuid: 0x0002C90200400098 

targetExtension: 0x0002C90200400098 

end 

adapter 

begin 

description: "Native IB storage" 

end 

Notes 

• There is a sample configuration in qlgc_srp.cfg. 

 

 

The correct TargetExtension must be added to session. 

It is important to use the IOC ID method since most Profile ID strings 

are not guaranteed to be unique. 

• Other possible parameters: 

 

initiatorExtension may be used by the storage device to identify 

the host. 

Additional Details 

• All LUNs found are reported to the Linux SCSI mid-layer. 

• Linux may need the max_scsi_luns (2.4 kernels) or max_luns (2.6 kernels) 

parameter configured in scsi_mod. 


For troubleshooting information, refer to “Troubleshooting SRP Issues” on 

page G-8. 

D000046-005 B D-23


OFED SRP Configuration 


To use OFED SRP, follow these steps: 

1. Add the line SRP_LOAD=yes to the module list in 

/etc/infiniband/openib.conf to have it automatically loaded. 

2. Discover the SRP devices on your fabric by running this command (as a root 

user): 

ibsrpdm 

In the output, look for lines similar to these: 

GUID: 0002c90200402c04 

ID: LSI Storage Systems SRP Driver 200400a0b8114527 


service[ 0]: 200400a0b8114527 / SRP.T10:200400A0B8114527 

GUID: 0002c90200402c0c 

ID: LSI Storage Systems SRP Driver 200500a0b8114527 


service[ 0]: 200500a0b8114527 / SRP.T10:200500A0B8114527 

GUID: 21000001ff040bf6 

ID: Data Direct Networks SRP Target System 


service[ 0]: f60b04ff01000021 / SRP.T10:21000001ff040bf6 

Note that not all the output is shown here; key elements are expected to 

show the match in Step 3. 

3. Choose the device you want to use, and run the command again with the -c 

option (as a root user): 

# ibsrpdm -c 

id_ext=200400A0B8114527,ioc_guid=0002c90200402c04,dgid=fe8000 

00000000000002c90200402c05,pkey=ffff,service_id=200400a0b8114 

527 

id_ext=200500A0B8114527,ioc_guid=0002c90200402c0c,dgid=fe8000 

00000000000002c90200402c0d,pkey=ffff,service_id=200500a0b8114 

527 

id_ext=21000001ff040bf6,ioc_guid=21000001ff040bf6,dgid=fe8000 

000000000021000001ff040bf6,pkey=ffff,service_id=f60b04ff01000 

021 

D-24 D000046-005 B



4. Find the result that corresponds to the target you want, and echo it into the 

add_target file: 

echo 

"id_ext=21000001ff040bf6,ioc_guid=21000001ff040bf6,dgid=fe800 

0000000000021000001ff040bf6,pkey=ffff,service_id=f60b04ff0100 

0021,initiator_ext=0000000000000001" > 

/sys/class/infiniband_srp/srp-ipath0-1/add_target 

5. Look for the newly created devices in the /proc/partitions file. The file 

will look similar to this example (the partition names may vary): 

# cat /proc/partitions 

major minor #blocks name 

8 64 142325760 sde 

8 65 142319834 sde1 

8 80 71162880 sdf 

8 81 71159917 sdf1 

8 96 20480 sdg 

8 97 20479 sdg1 

6. Create a mount point (as root) where you will mount the SRP device. For 

example: 

mkdir /mnt/targetname 

mount /dev/sde1 /mnt/targetname 

NOTE: 

Use sde1 rather than sde. See the mount(8) man page for more 

information on creating mount points. 

D000046-005 B D-25



Notes 

D-26 D000046-005 B

E 

Integration with a Batch 

Queuing System 

Most cluster systems use some kind of batch queuing system as an orderly way to 

provide users with access to the resources they need to meet their job’s 

performance requirements. One task of the cluster administrator is to allow users 

to submit MPI jobs through these batch queuing systems. Two methods are 

described in this document: 

• Use mpiexec within the Portable Batch System (PBS) environment. 

• Invoke a script, similar to mpirun, within the SLURM context to submit MPI 

jobs. A sample is provided in “Using SLURM for Batch Queuing” on 

page E-2. 

Using mpiexec with PBS 

mpiexec can be used as a replacement for mpirun within a PBS cluster 

environment. The PBS software performs job scheduling. 

For PBS-based batch systems, QLogic MPI processes can be spawned using the 

mpiexec utility distributed and maintained by the Ohio Supercomputer Center 

(OSC). 

Starting with mpiexec version 0.84, MPI applications compiled and linked with 

QLogic MPI can use mpiexec and PBS’s Task Manager (TM) interface to spawn 

and correctly terminate parallel jobs. 

To download the latest version of mpiexec, go to: 

http://www.osc.edu/~pw/mpiexec/ 

To build mpiexec for QLogic MPI and install it in /usr/local, type: 

$ tar zxvf mpiexec-0.84.tgz 

$ cd mpiexec-0.84 

$ ./configure --enable-default-comm=mpich-psm && gmake all install 

D000046-005 B E-1

E–Integration with a Batch Queuing System 

Using SLURM for Batch Queuing 

NOTE: 

This level of support is specific to QLogic MPI, and not to other MPIs that 

currently support InfiniPath. 

For more usage information, see the OSC mpiexec documentation. 

For more information on PBS, go to: http://www.pbsgridworks.com/ 


The following is an example of the some of the functions that a batch queuing 

script might perform. The example is in the context of the Simple Linux Utility 

Resource Manager (SLURM) developed at Lawrence Livermore National 

Laboratory. These functions assume the use of the bash shell. The following 

script is called batch_mpirun: 

#! /bin/sh 

# Very simple example batch script for QLogic MPI, using slurm 

# (http://www.llnl.gov/linux/slurm/) 

# Invoked as: 

#batch_mpirun #cpus mpi_program_name mpi_program_args ... 

# 

np=$1 mpi_prog="$2" # assume arguments to script are correct 

shift 2 # program args are now $@ 

eval ‘srun --allocate --ntasks=$np --no-shell‘ 

mpihosts_file=‘mktemp -p /tmp mpihosts_file.XXXXXX‘ 

srun --jobid=${SLURM_JOBID} hostname -s | sort | uniq -c \ 

| awk ’{printf "%s:%s\n", $2, $1}’ > $mpihosts_file 

mpirun -np $np -m $mpihosts_file "$mpi_prog" $@ 

exit_code=$ 

scancel ${SLURM_JOBID} 

rm -f $mpihosts_file 

exit $exit_code 

In the following sections, the setup and the various script functions are discussed 

in more detail. 

E-2 D000046-005 B



Allocating Resources 

When the mpirun command starts, it requires specification of the number of node 

programs it must spawn (via the -np option) and specification of an mpihosts 

file listing the nodes that the node programs run on. (See “Environment for Node 

Programs” on page 4-19 for more information.) Since performance is usually 

important, a user might require that his node program be the only application 

running on each node CPU. In a typical batch environment, the MPI user would 

still specify the number of node programs, but would depend on the batch system 

to allocate specific nodes when the required number of CPUs become available. 

Thus, batch_mpirun would take at least an argument specifying the number of 

node programs and an argument specifying the MPI program to be executed. For 

example: 

$ batch_mpirun -np n my_mpi_program 

After parsing the command line arguments, the next step of batch_mpirun is to 

request an allocation of n processors from the batch system. In SLURM, this uses 

the command: 

eval ‘srun --allocate --ntasks=$np --no-shell‘ 

Make sure to use back quotes rather than normal single quotes. $np is the shell 

variable that your script has set from the parsing of its command line options. The 

--no-shell option to srun prevents SLURM from starting a subshell. The srun 

command is run with eval to set the SLURM_JOBID shell variable from the output 

of the srun command. 

With these specified arguments, the SLURM function srun blocks until there are 

$np processors available to commit to the caller. When the requested resources 

are available, this command opens a new shell and allocates the number of 

processors to the requestor. 

Generating the mpihosts File 

Once the batch system has allocated the required resources, your script must 

generate an mpihosts file that contains a list of nodes that are used. To do this, 

the script must determine which nodes the batch system has allocated, and how 

many processes can be started on each node. This is the part of the script 

batch_mpirun that performs these tasks, for example: 

mpihosts_file=‘mktemp -p /tmp mpihosts_file.XXXXXX‘ 

srun --jobid=${SLURM_JOBID} hostname -s | sort | uniq -c \ 

| awk ’{printf "%s:%s\n", $2, $1}’ > $mpihosts_file 

The first command creates a temporary hosts file with a random name, and 

assigns the name to the variable mpihosts_file it has generated. 

The next instance of the SLURM srun command runs hostname -s once for 

each process slot that SLURM has allocated. If SLURM has allocated two slots on 

one node, hostname -s is output twice for that node. 

D000046-005 B E-3



The sort | uniq -c component determines the number of times each unique 

line was printed. The awk command converts the result into the mpihosts file 

format used by mpirun. Each line consists of a node name, a colon, and the 

number of processes to start on that node. 

NOTE: 

This is one of two formats that the file can use. See “Console I/O in MPI 

Programs” on page 4-18 for more information. 

Simple Process Management 

At this point, the script has enough information to be able to run an MPI program. 

The next step is to start the program when the batch system is ready, and notify 

the batch system when the job completes. This is done in the final part of 

batch_mpirun, for example: 

mpirun -np $np -m $mpihosts_file "$mpi_prog" $@ 

exit_code=$ 

scancel ${SLURM_JOBID} 

rm -f $mpihosts_file 

exit $exit_code 

Clean Termination of MPI Processes 

The InfiniPath software normally ensures clean termination of all MPI programs 

when a job ends, but in some rare circumstances an MPI process may remain 

alive, and potentially interfere with future MPI jobs. To avoid this problem, run a 

script before and after each batch job that kills all unwanted processes. QLogic 

does not provide such a script, but it is useful to know how to find out which 

processes on a node are using the QLogic interconnect. The easiest way to do 

this is with the fuser command, which is normally installed in /sbin. 

Run these commands as a root user to ensure that all processes are reported. 

# /sbin/fuser -v /dev/ipath 

/dev/ipath: 22648m 22651m 

In this example, processes 22648 and 22651 are using the QLogic interconnect. It 

is also possible to use this command (as a root user): 

# lsof /dev/ipath 

This command displays a list of processes using InfiniPath. Additionally, to get all 

processes, including stats programs, ipath_sma, diags, and others, run the 

program in this way: 

# /sbin/fuser -v /dev/ipath* 

lsof can also take the same form: 

# lsof /dev/ipath* 

E-4 D000046-005 B


Lock Enough Memory on Nodes when Using SLURM 

The following command terminates all processes using the QLogic interconnect: 

# /sbin/fuser -k /dev/ipath 

For more information, see the man pages for fuser(1) and lsof(8). 

Note that hard and explicit program termination, such as kill -9 on the mpirun 

Process ID (PID), may result in QLogic MPI being unable to guarantee that the 

/dev/shm shared memory file is properly removed. As many stale files 

accumulate on each node, an error message can appear at startup: 

node023:6.Error creating shared memory object in shm_open(/dev/shm 

may have stale shm files that need to be removed): 

If this occurs, administrators should clean up all stale files by using this command: 

# rm -rf /dev/shm/psm_shm.* 

See “Error Creating Shared Memory Object” on page F-23 for more information. 

Lock Enough Memory on Nodes when Using 

SLURM 

This section is identical to information provided in “Lock Enough Memory on 

Nodes When Using a Batch Queuing System” on page F-22. It is repeated here 

for your convenience. 

QLogic MPI requires the ability to lock (pin) memory during data transfers on each 

compute node. This is normally done via /etc/initscript, which is created or 

modified during the installation of the infinipath RPM (setting a limit of 

128 MB, with the command ulimit -l 131072). 

Some batch systems, such as SLURM, propagate the user’s environment from 

the node where you start the job to all the other nodes. For these batch systems, 

you may need to make the same change on the node from which you start your 

batch jobs. 

If this file is not present or the node has not been rebooted after the infinipath 

RPM has been installed, a failure message similar to one of the following will be 

generated. 

The following message displays during installation: 

$ mpirun -np 2 -m ~/tmp/sm mpi_latency 1000 1000000 

iqa-19:0.ipath_userinit: mmap of pio buffers at 100000 failed: 

Resource temporarily unavailable 

iqa-19:0.Driver initialization failure on /dev/ipath 




D000046-005 B E-5


Lock Enough Memory on Nodes when Using SLURM 

The following message displays after installation: 

$ mpirun -m ~/tmp/sm -np 2 -mpi_latency 1000 1000000 

node-00:1.ipath_update_tid_err: failed: Cannot allocate memory 

mpi_latency: 

/fs2/scratch/infinipath-build-1.3/mpi-1.3/mpich/psm/src 

mq_ips.c:691: 

mq_ipath_sendcts: Assertion ‘rc == 0’ failed. MPIRUN: Node program 

unexpectedly quit. Exiting. 

You can check the ulimit -l on all the nodes by running ipath_checkout. A 

warning similar to this displays if ulimit -l is less than 4096: 

!!!ERROR!!! Lockable memory less than 4096KB on x nodes 

To fix this error, install the infinipath RPM on the node, and reboot it to ensure 

that /etc/initscript is run. 

Alternately, you can create your own /etc/initscript and set the ulimit 

there. 

E-6 D000046-005 B

F 


This appendix describes some of the tools you can use to diagnose and fix 

problems. The following topics are discussed: 

• Using LEDs to Check the State of the Adapter 

• BIOS Settings 

• Kernel and Initialization Issues 

• OpenFabrics and InfiniPath Issues 

• System Administration Troubleshooting 

• Performance Issues 

• QLogic MPI Troubleshooting 

Troubleshooting information for hardware installation is found in the QLogic 

InfiniBand Adapter Hardware Installation Guide and software installation is found 

in the QLogic Fabric Software Installation Guide. 

Using LEDs to Check the State of the Adapter 

The LEDs function as link and data indicators once the InfiniPath software has 

been installed, the driver has been loaded, and the fabric is being actively 

managed by a subnet manager. 

Table F-1 describes the LED states. The green LED indicates the physical link 

signal; the amber LED indicates the link. The green LED normally illuminates first. 

The normal state is Green On, Amber On. The QLE7240 and QLE7280 have an 

additional state, as shown in Table F-1. 

Table F-1. LED Link and Data Indicators 

Green OFF 

Amber OFF 

LED States 

Indication 

The switch is not powered up. The software is neither 

installed nor started. Loss of signal. 

Verify that the software is installed and configured with 

ipath_control -i. If correct, check both cable connectors. 

D000046-005 B F-1

F–Troubleshooting 

BIOS Settings 

Table F-1. LED Link and Data Indicators (Continued) 

LED States 

Green ON 

Amber OFF 

Green ON 

Amber ON 

Green BLINKING (quickly) 

Amber ON 

Green BLINKING a 

Amber BLINKING 

Table Notes 

Signal detected and the physical link is up. Ready to talk 

to SM to bring the link fully up. 

If this state persists, the SM may be missing or the link 

may not be configured. 

Use ipath_control -i to verify the software state. If 

all infiniband adapters are in this state, then the SM is 

not running. Check the SM configuration, or install and 

run opensmd. 

The link is configured, properly connected, and ready. 

Signal detected. Ready to talk to an SM to bring the link 

fully up. 

The link is configured. Properly connected and ready to 

receive data and link packets. 

Indicates traffic 

Indication 

Locates the adapter 

This feature is controlled by ipath_control -b [On 

| Off] 

a 

This feature is available only on the QLE7340, QLE7342, QLE7240 and QLE7280 adapters 

BIOS Settings 

This section covers issues related to BIOS settings.The most important setting is 

Advanced Configuration and Power Interface (ACPI). This setting must be 

enabled. If ACPI has been disabled, it may result in initialization problems, as 

described in “InfiniPath Interrupts Not Working” on page F-3. 

You can check and adjust the BIOS settings using the BIOS Setup utility. Check 

the hardware documentation that came with your system for more information. 

Kernel and Initialization Issues 

Issues that may prevent the system from coming up properly are described in the 

following sections. 

F-2 D000046-005 B



Driver Load Fails Due to Unsupported Kernel 

If you try to load the InfiniPath driver on a kernel that InfiniPath software does not 

support, the load fails. Error messages similar to this display: 

modprobe: error inserting 

’/lib/modules/2.6.3-1.1659-smp/updates/kernel/drivers/infiniband/h 

w/qib/ib_qib.ko’: -1 Invalid module format 

To correct this problem, install one of the appropriate supported Linux kernel 

versions, then reload the driver. 

Rebuild or Reinstall Drivers if Different Kernel Installed 

If you upgrade the kernel, then you must reboot and then rebuild or reinstall the 

InfiniPath kernel modules (drivers). QLogic recommends using the InfiniBand 

Fabric Suite Software Installation TUI to preform this rebuild or reinstall. Refer to 

the QLogic Fabric Software Installation Guide for more information. 

InfiniPath Interrupts Not Working 

The InfiniPath driver cannot configure the InfiniPath link to a usable state unless 

interrupts are working. Check for this problem with the command: 

$ grep ib_qib /proc/interrupts 

Normal output is similar to this: 

CPU0 CPU1 

185: 364263 0 IO-APIC-level ib_qib 

NOTE: 

The output you see may vary depending on board type, distribution, or 

update level. 

If there is no output at all, the driver initialization failed. For more information on 

driver problems, see “Driver Load Fails Due to Unsupported Kernel” on page F-3 

or “InfiniPath ib_qib Initialization Failure” on page F-5. 

If the output is similar to one of these lines, then interrupts are not being delivered 

to the driver. 

66: 0 0 PCI-MSI ib_qib 

185: 0 0 IO-APIC-level ib_qib 

The following message appears when driver has initialized successfully, but no 

interrupts are seen within 5 seconds. 

ib_qib 0000:82:00.0: No interrupts detected. 

D000046-005 B F-3



A zero count in all CPU columns means that no InfiniPath interrupts have been 

delivered to the processor. 

The possible causes of this problem are: 

• Booting the Linux kernel with ACPI disabled on either the boot command 

line or in the BIOS configuration 

• Other infinipath initialization failures 

To check if the kernel was booted with the noacpi or pci=noacpi option, use 

this command: 

$ grep -i acpi /proc/cmdline 

If output is displayed, fix the kernel boot command line so that ACPI is enabled. 

This command line can be set in various ways, depending on your distribution. If 

no output is displayed, check that ACPI is enabled in your BIOS settings. 

To track down other initialization failures, see “InfiniPath ib_qib Initialization 

Failure” on page F-5. 

The program ipath_checkout can also help flag these kinds of problems. See 

“ipath_checkout” on page I-10 for more information. 

OpenFabrics Load Errors if ib_qib Driver Load Fails 

When the ib_qib driver fails to load, the other OpenFabrics drivers/modules will 

load and be shown by lsmod, but commands like ibstatus, ibv_devinfo, 

and ipath_control -i will fail as follows: 

# ibstatus 

Fatal error: device ’*’: sys files not found 

(/sys/class/infiniband/*/ports) 

# ibv_devinfo 

libibverbs: Fatal: couldn’t read uverbs ABI version. 

No IB devices found 

# ipath_control -i 

InfiniPath driver not loaded 

No InfiniPath info available 

F-4 D000046-005 B



InfiniPath ib_qib Initialization Failure 

There may be cases where ib_qib was not properly initialized. Symptoms of this 

may show up in error messages from an MPI job or another program. Here is a 

sample command and error message: 

$ mpirun -np 2 -m ~/tmp/mbu13 osu_latency 

:ipath_userinit: assign_port command failed: Network is 

down 

:can’t open /dev/ipath, network down 

This will be followed by messages of this type after 60 seconds: 

MPIRUN: 1 rank has not yet exited 60 seconds 

after rank 0 (node ) exited without reaching 

MPI_Finalize(). 

MPIRUN:Waiting at most another 60 seconds for 

the remaining ranks to do a clean shutdown before terminating 1 

node processes. 

If this error appears, check to see if the InfiniPath driver is loaded by typing: 

$ lsmod | grep ib_qib 

If no output is displayed, the driver did not load for some reason. In this case, try 

the following commands (as root): 

# modprobe -v ib_qib 

# lsmod | grep ib_qib 

# dmesg | grep -i ib_qib | tail -25 

The output will indicate whether the driver has loaded. Printing out messages 

using dmesg may help to locate any problems with ib_qib. 

If the driver loaded, but MPI or other programs are not working, check to see if 

problems were detected during the driver and QLogic hardware initialization with 

the command: 

$ dmesg | grep -i ib_qib 

This command may generate more than one screen of output. 

Also, check the link status with the commands: 

$ cat /sys/class/infiniband/ipath*/device/status_str 

These commands are normally executed by the ipathbug-helper script, but 

running them separately may help locate the problem. 

See also “status_str” on page I-19 and “ipath_checkout” on page I-10. 

D000046-005 B F-5


OpenFabrics and InfiniPath Issues 

MPI Job Failures Due to Initialization Problems 

If one or more nodes do not have the interconnect in a usable state, messages 

similar to the following appear when the MPI program is started: 

userinit: userinit ioctl failed: Network is down [1]: device init 

failed 

userinit: userinit ioctl failed: Fatal Error in keypriv.c(520): 

device init failed 

These messages may indicate that a cable is not connected, the switch is down, 

SM is not running, or that a hardware error occurred. 


The following sections cover issues related to OpenFabrics (including Subnet 

Managers) and InfiniPath. 

Stop Infinipath Services Before Stopping/Restarting 

InfiniPath 

The following Infinipath services must be stopped before 

stopping/starting/restarting InfiniPath: 

• QLogic Fabric Manager 

• OpenSM 

• QLogic MPI 

• VNIC 

• SRP 

Here is a sample command and the corresponding error messages: 

# /etc/init.d/openibd stop 

Unloading infiniband modules: sdp cm umad uverbs ipoib sa ipath mad 

coreFATAL:Module ib_umad is in use. 

Unloading infinipath modules FATAL: Module ib_qib is in use. 

[FAILED] 

F-6 D000046-005 B



Manual Shutdown or Restart May Hang if NFS in Use 

If you are using NFS over IPoIB and use the manual /etc/init.d/openibd 

stop (or restart) command, the shutdown process may silently hang on the 

fuser command contained within the script. This is because fuser cannot 

traverse down the tree from the mount point once the mount point has 

disappeared. To remedy this problem, the fuser process itself needs to be killed. 

Run the following command either as a root user or as the user who is running the 

fuser process: 

# kill -9 fuser 

The shutdown will continue. 

This problem is not seen if the system is rebooted or if the filesystem has already 

been unmounted before stopping infinipath. 

Load and Configure IPoIB Before Loading SDP 

SDP generates Connection Refused errors if it is loaded before IPoIB has been 

loaded and configured. To solve the problem, load and configure IPoIB first. 

Set $IBPATH for OpenFabrics Scripts 

The environment variable $IBPATH must be set to /usr/bin. If this has not been 

set, or if you have it set to a location other than the installed location, you may see 

error messages similar to the following when running some OpenFabrics scripts: 

/usr/bin/ibhosts: line 30: /usr/local/bin/ibnetdiscover: No such 

file or directory 

For the OpenFabrics commands supplied with this InfiniPath release, set the 

variable (if it has not been set already) to /usr/bin, as follows: 

$ export IBPATH=/usr/bin 

SDP Module Not Loading 

If the settings for debug level and the zero copy threshold from InfiniPath 

release 2.0 are present in the release 2.2 /etc/modprobe.conf file (RHEL) or 

/etc/modprobe.conf.local (SLES) file, the SDP module may not load: 

options ib_sdp sdp_debug_level=4 

sdp_zcopy_thrsh_src_default=10000000 

To solve the problem, remove this line. 

D000046-005 B F-7


System Administration Troubleshooting 

ibsrpdm Command Hangs when Two Host Channel 

Adapters are Installed but Only Unit 1 is Connected 

to the Switch 

If multiple infiniband adapters (unit 0 and unit 1) are installed and only unit 1 is 

connected to the switch, the ibsrpdm command (to set up an SRP target) can 

hang. If unit 0 is connected and unit 1 is disconnected, the problem does not 

occur. 

When only unit 1 is connected to the switch, use the -d option with ibsrpdm. Then, 

using the output from the ibsrpdm command, echo the new target information into 

/sys/class/infiniband_srp/srp-ipath1-1/add_target. 

For example: 

# ibsrpdm -d /dev/infiniband/umad1 -c 

# echo \ 

id_ext=21000001ff040bf6,ioc_guid=21000001ff040bf6,dgid=fe800000000 

0000021000001ff040bf6,pkey=ffff,service_id=f60b04ff01000021 > 

/sys/class/infiniband_srp/srp-ipath0-1/add_target 

Outdated ipath_ether Configuration Setup Generates Error 

Ethernet emulation (ipath_ether) has been removed in this release, and, as a 

result, an error may be seen if the user still has an alias set previously by 

modprobe.conf (for example, alias eth2 ipath_ether). 

When ifconfig or ifup are run, the error will look similar to this (assuming 

ipath_ether was used for eth2): 

eth2: error fetching interface information: Device not found 

To prevent the error message, remove the following files (assuming 

ipath_ether was used for eth2): 

/etc/sysconfig/network-scripts/ifcfg-eth2 (for RHEL) 

/etc/sysconfig/network/ifcfg-eth2 (for SLES) 

QLogic recommends using the IP over InfiniBand protocol (IPoIB-CM), included in 

the standard OpenFabrics software releases, as a replacement for 

ipath_ether. 

System Administration Troubleshooting 

The following sections provide details on locating problems related to system 

administration. 

Broken Intermediate Link 

Sometimes message traffic passes through the fabric while other traffic appears 

to be blocked. In this case, MPI jobs fail to run. 

F-8 D000046-005 B


Performance Issues 

In large cluster configurations, switches may be attached to other switches to 

supply the necessary inter-node connectivity. Problems with these inter-switch (or 

intermediate) links are sometimes more difficult to diagnose than failure of the 

final link between a switch and a node. The failure of an intermediate link may 

allow some traffic to pass through the fabric while other traffic is blocked or 

degraded. 

If you notice this behavior in a multi-layer fabric, check that all switch cable 

connections are correct. Statistics for managed switches are available on a 

per-port basis, and may help with debugging. See your switch vendor for more 

information. 

QLogic recommends using FastFabric to help diagnose this problem. If 

FastFabric is not installed in the fabric, there are two diagnostic tools, ibhosts 

and ibtracert, that may also be helpful. The tool ibhosts lists all the 

InfiniBand nodes that the subnet manager recognizes. To check the InfiniBand 

path between two nodes, use the ibtracert command. 


The following sections discuss known performance issues. 

Large Message Receive Side Bandwidth Varies with 

Socket Affinity on Opteron Systems 

On Opteron systems, when using the QLE7240 or QLE7280 in DDR mode, there 

is a receive side bandwidth bottleneck for CPUs that are not adjacent to the PCI 

Express root complex. This may cause performance to vary. The bottleneck is 

most obvious when using SendDMA with large messages on the farthest sockets. 

The best case for SendDMA is when both sender and receiver are on the closest 

sockets. Overall performance for PIO (and smaller messages) is better than with 

SendDMA. 

Erratic Performance 

Sometimes erratic performance is seen on applications that use interrupts. An 

example is inconsistent SDP latency when running a program such as netperf. 

This may be seen on AMD-based systems using the QLE7240 or QLE7280 

adapters. If this happens, check to see if the program irqbalance is running. 

This program is a Linux daemon that distributes interrupts across processors. 

However, it may interfere with prior interrupt request (IRQ) affinity settings, 

introducing timing anomalies. After stopping this process (as a root user), bind 

IRQ to a CPU for more consistent performance. First, stop irqbalance: 

# /sbin/chkconfig irqbalance off 

# /etc/init.d/irqbalance stop 

D000046-005 B F-9



Next, find the IRQ number and bind it to a CPU. The IRQ number can be found in 

one of two ways, depending on the system used. Both methods are described in 

the following paragraphs. 

NOTE: 

Take care when cutting and pasting commands from PDF documents, as 

quotes are special characters and may not be translated correctly. 

Method 1 

Check to see if the IRQ number is found in /proc/irq/xxx, where xxx is the 

IRQ number in /sys/class/infiniband/ipath*/device/irq. Do this as a 

root user. For example: 

# my_irq=‘cat /sys/class/infiniband/ipath*/device/irq‘ 

# ls /proc/irq 

If $my_irq can be found under /proc/irq/, then type: 

# echo 01 > /proc/irq/$my_irq/smp_affinity 

Method 2 

If command from Method 1, ls /proc/irq, cannot find $my_irq, then use the 

following commands instead: 

# my_irq=‘cat /proc/interrupts|grep ib_qib|awk \ 

’{print $1}’|sed -e ’s/://’‘ 

# echo 01 > /proc/irq/$my_irq/smp_affinity 

This method is not the first choice because, on some systems, there may be two 

rows of ib_qib output, and you will not know which one of the two numbers to 

choose. However, if you cannot find $my_irq listed under /proc/irq 

(Method 1), this type of system most likely has only one line for ib_qib listed in 

/proc/interrupts, so you can use Method 2. 

Here is an example: 

# cat /sys/class/infiniband/ipath*/device/irq 

98 

# ls /proc/irq 

0 10 11 13 15 233 4 50 7 8 90 

1 106 12 14 2 3 5 58 66 74 9 

(Note that you cannot find 98.) 

# cat /proc/interrupts|grep ib_qib|awk \ 

’{print $1}’|sed -e ’s/://’ 

106 

# echo 01 > /proc/irq/106/smp_affinity 

Using the echo command immediately changes the processor affinity of an IRQ. 

F-10 D000046-005 B


QLogic MPI Troubleshooting 

NOTE: 

• The contents of the smp_affinity file may not reflect the expected 

values, even though the affinity change has taken place. 

• If the driver is reloaded, the affinity assignment will revert to the default, 

so you will need to reset it to the desired value. 

You can look at the stats in /proc/interrupts while the adapter is active to 

observe which CPU is fielding ib_qib interrupts. 

Performance Warning if ib_qib Shares Interrupts with eth0 

When ib_qib shares interrupts with eth0, performance may be affected the 

OFED ULPs, such as IPoIB. A warning message appears in syslog, and also on 

the console or tty session where /etc/init.d/openibd start is run (if 

messages are set up to be displayed). Messages are in this form: 

Nov 5 14:25:43 infinipath: Shared interrupt will 

affect performance: vector 169: devices eth0, ib_qib 

Check /proc/interrupts: "169" is in the first column, and "devices" are shown 

in the last column. 

You can also contact your system vendor to see if the BIOS settings can be 

changed to avoid the problem. 


Problems specific to compiling and running MPI programs are described in the 

following sections. 

D000046-005 B F-11



Mixed Releases of MPI RPMs 

Make sure that all of the MPI RPMs are from the same release. When using 

mpirun, an error message will occur if different components of the MPI RPMs are 

from different releases. In the following example, mpirun from release 2.1 is 

being used with a 2.2 library. 

$ mpirun -np 2 -m ~/tmp/x2 osu_latency 

MPI_runscript-xqa-14.0: ssh -x> Cannot detect InfiniPath 

interconnect. 

MPI_runscript-xqa-14.0: ssh -x> Seek help on loading InfiniPath 

interconnect driver. 

MPI_runscript-xqa-15.1: ssh -x> Cannot detect InfiniPath 

interconnect. 

MPI_runscript-xqa-15.1: ssh -x> Seek help on loading InfiniPath 

interconnect driver. 

MPIRUN: Node program(s) exitted during connection setup 

$ mpirun -v 

MPIRUN:Infinipath Release2.3: Built on Wed Nov 6 17:28:58 PDT 2008 

by mee 

The following example is the error that occurs when mpirun from the 2.2 release 

is being used with the 2.1 libraries. 

$ mpirun-ipath-ssh -np 2 -ppn 1 -m ~/tmp/idev osu_latency 

MPIRUN: mpirun from the 2.3 software distribution requires all 

node processes to be running 2.3 software. At least node 

uses non-2.3 MPI libraries 

The following string means that either an incompatible non-QLogic mpirun binary 

has been found or that the binary is from an InfiniPath release prior to 2.3. 

Found incompatible non-InfiniPath or pre-2.3 

InfiniPath mpirun-ipath-ssh (exec=/usr/bin/mpirun-ipath-ssh) 

Missing mpirun Executable 

When the mpirun executable is missing, the following error appears: 

Please install mpirun on or provide a path to 

mpirun-ipath-ssh 

(not found in $MPICH_ROOT/bin, $PATH 

or path/to/mpirun-ipath-ssh/on/the/head/node) or run with 

mpirun -distributed=off 

This error string means that an mpirun executable (mpirun-ipath-ssh) was 

not found on the computation nodes. Make sure that the mpi-frontend-* RPM 

is installed on all nodes that will use mpirun. 

F-12 D000046-005 B



Resolving Hostname with Multi-Homed Head Node 

By default, mpirun assumes that ranks can independently resolve the hostname 

obtained on the head node with gethostname. However, the hostname of a 

multi-homed head node may not resolve on the compute nodes. To address this 

problem, the following new option has been added to mpirun: 

-listen-addr 

This address will be forwarded to the ranks. To change the default, put this option 

in the global mpirun.defaults file or in a user-local file. 

If the address on the frontend cannot be resolved, then a warning is sent to the 

console and to syslog. If you use the following command line, you may see 

messages similar to this: 

% mpirun-ipath-ssh -np 2 -listen-addr foo -m ~/tmp/hostfile-idev 

osu_bcast 

MPIRUN.: Warning: Couldn’t resolve listen address ’foo’ 

on head node 

(Unknown host), using it anyway... 

MPIRUN.: No node programs have connected within 60 

seconds. 

This message occurs if none of the ranks can connect back to the head node. 

The following message may appear if some ranks cannot connect back: 

MPIRUN.: Not all node programs have connected within 

60 seconds. 

MPIRUN.: No connection received from 1 node process on 

node 

Cross-Compilation Issues 

The GNU 4.x environment is supported in the PathScale Compiler Suite 3.x 

release. 

However, the 2.x QLogic PathScale compilers are not currently supported on 

SLES 10 systems that use the GNU 4.x compilers and compiler environment 

(header files and libraries). 

QLogic recommends installing the PathScale 3.1 release. 

D000046-005 B F-13



Compiler/Linker Mismatch 

If the compiler and linker do not match in C and C++ programs, the following error 

message appears: 

$ export MPICH_CC=gcc 

$ mpicc mpiworld.c 

/usr/bin/ld: cannot find -lmpichabiglue_gcc3 

collect2: ld returned 1 exit status 

Compiler Cannot Find Include, Module, or Library Files 

RPMs can be installed in any location by using the --prefix option. This can 

introduce errors when compiling, if the compiler cannot find the include files (and 

module files for Fortran 90 and Fortran 95) from mpi-devel*, and the libraries 

from mpi-libs*, in the new locations. Compiler errors similar to the following 

appear: 

$ mpicc myprogram.c 

/usr/bin/ld: cannot find -lmpich 

collect2: ld returned 1 exit status 

NOTE: 

As noted in the QLogic Fabric Software Installation Guide, all development 

files now reside in specific *-Devel subdirectories. 

On development nodes, programs must be compiled with the appropriate options 

so that the include files and the libraries can be found in the new locations. In 

addition, when running programs on compute nodes, you need to ensure that the 

run-time library path is the same as the path that was used to compile the 

program. 

The following examples show what compiler options to use for include files and 

libraries on the development nodes, and how to specify the new library path on 

the compute nodes for the runtime linker. The affected RPMs are: 

• mpi-devel* (on the development nodes) 

• mpi-libs* (on the development or compute nodes) 

For the examples in “Compiling on Development Nodes” on page F-15, it is 

assumed that the new locations are: 

/path/to/devel (for mpi-devel-*) 

/path/to/libs (for mpi-libs-*) 

F-14 D000046-005 B



Compiling on Development Nodes 

If the mpi-devel-* RPM is installed with the --prefix /path/to/devel 

option, then mpicc, etc. must be passed in -I/path/to/devel/include for 

the compiler to find the MPI include files, as in this example: 

$ mpicc myprogram.c -I/path/to/devel/include 

If you are using Fortran 90 or Fortran 95, a similar option is needed for the 

compiler to find the module files: 

$ mpif90 myprogramf90.f90 -I/path/to/devel/include 

If the mpi-lib-* RPM is installed on these development nodes with the 

--prefix /path/to/libs option, then the compiler needs the 

-L/path/to/libs option so it can find the libraries. Here is the example for 

mpicc: 

$ mpicc myprogram.c -L/path/to/libs/lib (for 32 bit) 

$ mpicc myprogram.c -L/path/to/libs/lib64 (for 64 bit) 

To find both the include files and the libraries with these non-standard locations, 

type: 

$ mpicc myprogram.c -I/path/to/devel/include -L/path/to/libs/lib 

Specifying the Run-time Library Path 

There are several ways to specify the run-time library path so that when the 

programs are run, the appropriate libraries are found in the new location. There 

are three different ways to do this: 

• Use the -Wl,-rpath, option when compiling on the development node. 

• Update the /etc/ld.so.conf file on the compute nodes to include the path. 

• Export the path in the .mpirunrc file. 

These methods are explained in more detail in the following paragraphs. 

An additional linker option, -Wl,-rpath, supplies the run-time library path when 

compiling on the development node. The compiler options now look like this: 

$ mpicc myprogram.c -I/path/to/devel/include -L/path/to/libs/lib 

-Wl,-rpath,/path/to/libs/lib 

The above compiler command ensures that the program will run using this path on 

any machine. 

D000046-005 B F-15



For the second option, change the file /etc/ld.so.conf on the compute 

nodes rather than using the -Wl,-rpath, option when compiling on the 

development node. It is assumed that the mpi-lib-* RPM is installed on the 

compute nodes with the same --prefix /path/to/libs option as on the 

development nodes. Then, on the computer nodes, add the following lines to the 

file /etc/ld.so.conf: 

/path/to/libs/lib 

/path/to/libs/lib64 

To make sure that the changes take effect, run (as a root user): 

# /etc/ldconfig 

The libraries can now be found by the runtime linker on the compute nodes. The 

advantage to this method is that it works for all InfiniPath programs, without 

having to remember to change the compile/link lines. 

Instead of either of the two previous mechanisms, you can also put the following 

line in the ~/.mpirunrc file: 

export LD_LIBRARY_PATH=/path/to/libs/{lib,lib64} 

See “Environment for Node Programs” on page 4-19 for more information on 

using the -rcfile option with mpirun. 

Choices between these options are left up to the cluster administrator and the MPI 

developer. See the documentation for your compiler for more information on the 

compiler options. 

Problem with Shell Special Characters and Wrapper Scripts 

Be careful when dealing with shell special characters, especially when using the 

mpicc, etc. wrapper scripts. These characters must be escaped to avoid the shell 

interpreting them. 

For example, when compiling code using the -D compiler flag, mpicc (and other 

wrapper scripts) will fail if the defined variable contains a space, even when 

surrounded by double quotes. In the following example, the result of the -show 

option reveals what happens to the variable: 

$ mpicc -show -DMYDEFINE="some value" test.c 

gcc -c -DMYDEFINE=some value test.c 

gcc -Wl,--export-dynamic,--allow-shlib-undefined test.o -lmpich 

F-16 D000046-005 B



The shell strips off the double quotes before handing the arguments to the mpicc 

script, thus causing the problem. The workaround is to escape the double quotes 

and white space by using backslashes, so that the shell does not process them. 

(Also note the single quote (‘) around the -D, since the scripts do an eval rather 

than directly invoking the underlying compiler.) Use this command instead: 

$ mpicc -show -DMYDEFINE=\"some\ value\" test.c 

gcc -c ‘-DMYDEFINE="some value"‘ test.c 

gcc -Wl,--export-dynamic,--allow-shlib-undefined test.o -lmpich 

Run Time Errors with Different MPI Implementations 

It is now possible to run different implementations of MPI, such as HP-MPI, over 

InfiniPath. Many of these implementations share command (such as mpirun) and 

library names, so it is important to distinguish which MPI version is in use. This is 

done primarily through careful programming practices. 

Examples are provided in the following paragraphs. 

In the following command, the HP-MPI version of mpirun is invoked by the full 

path name. However, the program mpi_nxnlatbw was compiled with the QLogic 

version of mpicc. The mismatch produces errors similar this: 

$ /opt/hpmpi/bin/mpirun -hostlist "bbb-01,bbb-02,bbb-03,bbb-04" 

-np 4 /usr/bin/mpi_nxnlatbw 

bbb-02: Not running from mpirun. 

MPI Application rank 1 exited before MPI_Init() with status 1 







D000046-005 B F-17



In the next case, mpi_nxnlatbw.c is compiled with the HP-MPI version of 

mpicc, and given the name hpmpi-mpi_nxnlatbw, so that it is easy to see 

which version was used. However, it is run with the QLogic mpirun, which 

produces errors similar to this: 

$ /opt/hpmpi/bin/mpicc \ 

/usr/share/mpich/examples/performance/mpi_nxnlatbw.c -o 

hpmpi-mpi_nxnlatbw 

$ mpirun -m ~/host-bbb -np 4 ./hpmpi-mpi_nxnlatbw 

./hpmpi-mpi_nxnlatbw: error while loading shared libraries: 

libmpio.so.1: cannot open shared object file: No such file or 

directory 



directory 



directory 



directory 


The following two commands will work properly. 

QLogic mpirun and executable used together: 

$ mpirun -m ~/host-bbb -np 4 /usr/bin/mpi_nxnlatbw 

The HP-MPI mpirun and executable used together: 

$ /opt/hpmpi/bin/mpirun -hostlist \ 

"bbb-01,bbb-02,bbb-03,bbb-04" -np 4 ./hpmpi-mpi_nxnlatbw 

Hints 

• Use the rpm command to find out which RPM is installed in the standard 

installed layout. For example: 

# rpm -qf /usr/bin/mpirun 

mpi-frontend-2.3-5314.919_sles10_qlc 

• Check all rcfiles and /opt/infinipath/etc/mpirun.defaults to 

make sure that the paths for binaries and libraries ($PATH and 

$LD_LIBRARY _PATH) are consistent. 

• When compiling, use descriptive names for the object files. 

F-18 D000046-005 B



See “Compiler Cannot Find Include, Module, or Library Files” on page F-14, 

“Compiling on Development Nodes” on page F-15, and “Specifying the Run-time 

Library Path” on page F-15 for additional information. 

Process Limitation with ssh 

MPI jobs that use more than eight processes per node may encounter an ssh 

throttling mechanism that limits the amount of concurrent per-node connections 

to 10. If you have this problem, a message similar to this appears when using 

mpirun: 

$ mpirun -m tmp -np 11 ~/mpi/mpiworld/mpiworld 

ssh_exchange_identification: Connection closed by remote host 


If you encounter a message like this, you or your system administrator should 

increase the value of MaxStartups in your sshd configurations. 

NOTE: 

This limitation applies only if -distributed=off is specified. By default, 

with -distributed=on, you will not normally have this problem. 

Number of Processes Exceeds ulimit for Number of Open 

Files 

When users scale up the number of processes beyond the number of open files 

allowed by ulimit, mpirun will print an error message. The ulimit for the 

number of open files is typically 1024 on both Red Hat and SLES systems. The 

message will look similar to: 

MPIRUN.up001: Warning: ulimit for the number of open files is only 

1024, but this mpirun request requires at least 

open files (sockets). The shell ulimit for open files needs to be 

increased. 

This is due to limit: 

descriptors 1024 

The ulimit can be increased; QLogic recommends an increase of 

approximately 20 percent over the number of CPUs. For example, in the case of 

2048 CPUs, ulimit can be increased to 2500: 

ulimit -n 2500 

The ulimit needs to be increased only on the host where mpirun was started, 

unless the mode of operation allows mpirun from any node. 

D000046-005 B F-19



Using MPI.mod Files 

MPI.mod (or mpi.mod) are the Fortran 90/Fortran 95 mpi modules files. These 

files contain the Fortran 90/Fortran 95 interface to the platform-specific MPI 

library. The module file is invoked by ‘USE MPI’ or ‘use mpi’ in your application. If 

the application has an argument list that does not match what mpi.mod expects, 

errors such as this can occur: 

$ mpif90 -O3 -OPT:fast_math -c communicate.F 

call mpi_recv(nrecv,1,mpi_integer,rpart(nswap),0, 

^ 

pathf95-389 pathf90: ERROR BORDERS, File = communicate.F, Line = 

407, Column = 18 

No specific match can be found for the generic subprogram call 

"MPI_RECV". 

If it is necessary to use a non-standard argument list, create your own MPI 

module file and compile the application with it, rather than using the standard MPI 

module file that is shipped in the mpi-devel-* RPM. 

The default search path for the module file is: 

/usr/include 

To include your own MPI.mod rather than the standard version, use 

-I/your/search/directory, which causes /your/search/directory to 

be checked before /usr/include. For example: 

$ mpif90 -I/your/search/directory myprogram.f90 

Usage for Fortran 95 will be similar to the example for Fortran 90. 

Extending MPI Modules 

MPI implementations provide procedures that accept an argument having any 

data type, any precision, and any rank. However, it is not practical for an MPI 

module to enumerate every possible combination of type, kind, and rank. 

Therefore, the strict type checking required by Fortran 90 may generate errors. 

For example, if the MPI module tells the compiler that mpi_bcast can operate on 

an integer but does not also say that it can operate on a character string, you may 

see a message similar to the following: 

pathf95: ERROR INPUT, File = input.F, Line = 32, Column = 14 

No specific match can be found for the generic subprogram call 

"MPI_BCAST". 

F-20 D000046-005 B



If you know that an argument can accept a data type that the MPI module does 

not explicitly allow, you can extend the interface for yourself. For example, the 

following program shows how to extend the interface for mpi_bcast so that it 

accepts a character type as its first argument, without losing the ability to accept 

an integer type as well: 

module additional_bcast 

use mpi 

implicit none 

interface mpi_bcast 

module procedure additional_mpi_bcast_for_character 

end interface mpi_bcast 

contains 

subroutine additional_mpi_bcast_for_character(buffer, count, 

datatype, & root, comm, ierror) 

character*(*) buffer 

integer count, datatype, root, comm, ierror 

! Call the Fortran 77 style implicit interface to "mpi_bcast" 

external mpi_bcast 

call mpi_bcast(buffer, count, datatype, root, comm, ierror) 

end subroutine additional_mpi_bcast_for_character 

end module additional_bcast 

program myprogram 

use mpi 

use additional_bcast 

implicit none 

character*4 c 

integer master, ierr, i 

! Explicit integer version obtained from module "mpi" 

call mpi_bcast(i, 1, MPI_INTEGER, master, MPI_COMM_WORLD, ierr) 

! Explicit character version obtained from module 

"additional_bcast" 

call mpi_bcast(c, 4, MPI_CHARACTER, master, MPI_COMM_WORLD, 

ierr) 

end program myprogram 

D000046-005 B F-21



This is equally applicable if the module mpi provides only a lower-rank interface 

and you want to add a higher-rank interface, for example, when the module 

explicitly provides for 1-D and 2-D integer arrays, but you need to pass a 3-D 

integer array. Add a higher-rank interface only under the following conditions: 

• The module mpi provides an explicit Fortran 90 style interface for 

mpi_bcast. If the module mpi does not have this interface, the program 

uses an implicit Fortran 77 style interface, which does not perform any type 

checking. Adding an interface will cause type-checking error messages 

where there previously were none. 

• The underlying function accepts any data type. It is appropriate for the first 

argument of mpi_bcast because the function operates on the underlying 

bits, without attempting to interpret them as integer or character data. 

Lock Enough Memory on Nodes When Using a Batch 

Queuing System 

QLogic MPI requires the ability to lock (pin) memory during data transfers on each 

compute node. This is normally done via /etc/initscript, which is created or 

modified during the installation of the infinipath RPM (setting a limit of 

128 MB, with the command ulimit -l 131072). 

Some batch systems, such as SLURM, propagate the user’s environment from 

the node where you start the job to all the other nodes. For these batch systems, 

you may need to make the same change on the node from which you start your 

batch jobs. 

If this file is not present or the node has not been rebooted after the infinipath 

RPM has been installed, a failure message similar to one of the following will be 

generated. 

The following message displays during installation: 

$ mpirun -np 2 -m ~/tmp/sm mpi_latency 1000 1000000 







F-22 D000046-005 B



The following message displays after installation: 

$ mpirun -m ~/tmp/sm -np 2 -mpi_latency 1000 1000000 

node-00:1.ipath_update_tid_err: failed: Cannot allocate memory 

mpi_latency: 

/fs2/scratch/infinipath-build-1.3/mpi-1.3/mpich/psm/src 

mq_ips.c:691: 

mq_ipath_sendcts: Assertion ‘rc == 0’ failed. MPIRUN: Node program 

unexpectedly quit. Exiting. 

You can check the ulimit -l on all the nodes by running ipath_checkout. A 

warning similar to this displays if ulimit -l is less than 4096: 

!!!ERROR!!! Lockable memory less than 4096KB on x nodes 

To fix this error, install the infinipath RPM on the node, and reboot it to ensure 

that /etc/initscript is run. 

Alternately, you can create your own /etc/initscript and set the ulimit 

there. 

Error Creating Shared Memory Object 

QLogic MPI (and PSM) use Linux’s shared memory mapped files to share 

memory within a node. When an MPI job is started, a shared memory file is 

created on each node for all MPI ranks sharing memory on that one node. During 

job execution, the shared memory file remains in /dev/shm. At program exit, the 

file is removed automatically by the operating system when the QLogic MPI 

(InfiniPath) library properly exits. Also, as an additional backup in the sequence of 

commands invoked by mpirun during every MPI job launch, the file is explicitly 

removed at program termination. 

However, under circumstances such as hard and explicit program termination (i.e. 

kill -9 on the mpirun process PID), QLogic MPI cannot guarantee that the 

/dev/shm file is properly removed. As many stale files accumulate on each node, 

an error message like the following can appear at startup: 

node023:6.Error creating shared memory object in shm_open(/dev/shm 

may have stale shm files that need to be removed): 

If this occurs, administrators should clean up all stale files by running this 

command (as a root user): 

# rm -rf /dev/shm/psm_shm.* 

You can also selectively identify stale files by using a combination of the fuser, 

ps, and rm commands (all files start with the psm_shm prefix). Once identified, 

you can issue rm commands on the stale files that you own. 

D000046-005 B F-23



NOTE: 

It is important that /dev/shm be writable by all users, or else error 

messages like the ones in this section can be expected. Also, non-QLogic 

MPIs that use PSM may be more prone to stale shared memory files when 

processes are abnormally terminated. 

gdb Gets SIG32 Signal Under mpirun -debug with the 

PSM Receive Progress Thread Enabled 

When you run mpirun -debug and the PSM receive progress thread is enabled, 

gdb (the GNU debugger) reports the following error: 

(gdb) run 

Starting program: /usr/bin/osu_bcast < /dev/null [Thread debugging 

using libthread_db enabled] [New Thread 46912501386816 (LWP 

13100)] [New Thread 1084229984 (LWP 13103)] [New Thread 1094719840 

(LWP 13104)] 

Program received signal SIG32, Real-time event 32. 

[Switching to Thread 1084229984 (LWP 22106)] 0x00000033807c0930 in 

poll () from /lib64/libc.so.6 

This signal is generated when the main thread cancels the progress thread. To fix 

this problem, disable the receive progress thread when debugging an MPI 

program. Add the following line to $HOME/.mpirunrc: 

export PSM_RCVTHREAD=0 

NOTE: 

Remove the above line from $HOME/.mpirunrc after you debug an MPI 

program. If this line is not removed, the PSM receive progress thread will be 

permanently disabled. To check if the receive progress thread is enabled, 

look for output similar to the following when using the mpirun -verbose 

flag: 

idev-17:0.env PSM_RCVTHREAD Recv thread flags 

0 disables thread) => 0x1 

The value 0x1 indicates that the receive thread is currently enabled. A value 

of 0x0 indicates that the receive thread is disabled. 

F-24 D000046-005 B



General Error Messages 

The following message may be generated by ipath_checkout or mpirun: 

PSM found 0 available contexts on InfiniPath device 

The most likely cause is that the cluster has processes using all the available 

PSM contexts. 

Error Messages Generated by mpirun 

The following sections describe the mpirun error messages. These messages 

are in one of these categories: 

• Messages from the QLogic MPI (InfiniPath) library 

• MPI messages 

• Messages relating to the InfiniPath driver and InfiniBand links 

Messages generated by mpirun follow this format: 

program_name: message 

function_name: message 

Messages can also have different prefixes, such as ipath_ or psm_, which 

indicate where in the software the errors are occurring. 

Messages from the QLogic MPI (InfiniPath) Library 

Messages from the QLogic MPI (InfiniPath) library appear in the mpirun output. 

The following example contains rank values received during connection setup that 

were higher than the number of ranks (as indicated in the mpirun startup code): 

sender rank rank is out of range (notification) 

sender rank rank is out of range (ack) 

The following are error messages that indicate internal problems and must be 

reported to Technical Support. 

unknown frame type type 

[n] Src lid error: sender: x, exp send: y 

Frame receive from unknown sender. exp. sender = x, came from y 

Failed to allocate memory for eager buffer addresses: str 

The following error messages usually indicate a hardware or connectivity 

problem: 

Failed to get IB Unit LID for any unit 

Failed to get our IB LID 

Failed to get number of Infinipath units 

In these cases, try to reboot. If that does not work, call Technical Support. 

D000046-005 B F-25



The following message indicates a mismatch between the QLogic interconnect 

hardware in use and the version where the software was compiled: 

Number of buffer avail registers is wrong; have n, expected m 

build mismatch, tidmap has n bits, ts_map m 

These messages indicate a mismatch between the InfiniPath software and 

hardware versions. Consult Technical Support after verifying that current drivers 

and libraries are installed. 

The following examples are all informative messages about driver initialization 

problems. They are not necessarily fatal themselves, but may indicate problems 

that interfere with the application. In the actual printed output, all of the messages 

are prefixed with the name of the function that produced them. 

assign_port command failed: str 

Failed to get LID for unit u: str 

Failed to get number of units: str 

GETPORT ioctl failed: str 

can't allocate memory for ipath_ctrl: str 

can't stat infinipath device to determine type: str 

file descriptor is not for a real device, failing 

get info ioctl failed: str 

ipath_get_num_units called before init 

ipath_get_unit_lid called before init 

mmap of egr bufs from h failed: str 

mmap of pio buffers at %llx failed: str 

mmap of pioavail registers (%llx) failed: str 

mmap of rcvhdr q failed: str 

mmap of user registers at %llx failed: str 

userinit command failed: str 

Failed to set close on exec for device: str 

NOTE: 

These messages should never occur. If they do, notify Technical Support. 

The following message indicates that a node program may not be processing 

incoming packets, perhaps due to a very high system load: 

eager array full after overflow, flushing (head h, tail t) 

The following error messages should rarely occur; they indicate internal software 

problems: 

ExpSend opcode h tid=j, rhf_error k: str 

Asked to set timeout w/delay l, gives time in past (t2 < t1) 

Error in sending packet: str 

In this case, str can give additional information about why the failure occurred. 

F-26 D000046-005 B



The following message usually indicates a node failure or malfunctioning link in 

the fabric: 

Couldn’t connect to (LID=::). Time 

elapsed 00:00:30. Still trying... 

IP is the MPI rank’s IP address, and are the rank’s lid, 

port, and subport. 

If messages similar to the following display, it may mean that the program is trying 

to receive to an invalid (unallocated) memory address, perhaps due to a logic 

error in the program, usually related to malloc/free: 

ipath_update_tid_err: Failed TID update for rendezvous, allocation 

problem 

kernel: infinipath: get_user_pages (0x41 pages starting at 

0x2aaaaeb50000 

kernel: infinipath: Failed to lock addr 0002aaaaeb50000, 65 pages: 

errno 12 

TID is short for Token ID, and is part of the QLogic hardware. This error indicates 

a failure of the program, not the hardware or driver. 

MPI Messages 

Some MPI error messages are issued from the parts of the code inherited from 

the MPICH implementation. See the MPICH documentation for message 

descriptions. This section discusses the error messages specific to the QLogic 

MPI implementation. 

These messages appear in the mpirun output. Most are followed by an abort, 

and possibly a backtrace. Each is preceded by the name of the function where the 

exception occurred. 

The following message is always followed by an abort. The processlabel is 

usually in the form of the host name followed by process rank: 

processlabel Fatal Error in filename line_no: error_string 

At the time of publication, the possible error_strings are: 

Illegal label format character. 

Memory allocation failed. 

Error creating shared memory object. 

Error setting size of shared memory object. 

Error mmapping shared memory. 

Error opening shared memory object. 

Error attaching to shared memory. 

Node table has inconsistent len! Hdr claims %d not %d 

Timeout waiting %d seconds to receive peer node table from mpirun 

D000046-005 B F-27



The following indicates an unknown host: 

$ mpirun -np 2 -m ~/tmp/q mpi_latency 100 100 

MPIRUN: Cannot obtain IP address of : Unknown host 

15:35_~.1019 

The following indicates that there is no route to a valid host: 


ssh: connect to host port 22: No route to host 

MPIRUN: Some node programs ended prematurely without connecting to 

mpirun. 

MPIRUN: No connection received from 1 node process on node 

 

The following indicates that there is no route to any host: 




MPIRUN: All node programs ended prematurely without connecting to 

mpirun. 

The following indicates that node jobs have started, but one host could not 

connect back to mpirun: 


9139.psc_skt_connect: Error connecting to socket: No route to host 

. Cannot connect to spawner on host %s port %d 

within 60 seconds. 

MPIRUN: Some node programs ended prematurely without connecting to 

mpirun. 

MPIRUN: No connection received from 1 node process on node 

 

The following indicates that node jobs have started, but both hosts could not 

connect back to mpirun: 








MPIRUN: All node programs ended prematurely without connecting to 

mpirun. 


MPIRUN: node program unexpectedly quit: Exiting. 

F-28 D000046-005 B



The following indicates that one program on one node died: 


MPIRUN: node program unexpectedly quit: Exiting. 

The quiescence detected message is printed when an MPI job is not making 

progress. The default timeout is 900 seconds. After this length of time, all the 

node processes are terminated. This timeout can be extended or disabled with the 

-quiescence-timeout option in mpirun. 

$ mpirun -np 2 -m ~/tmp/q -q 60 mpi_latency 1000000 1000000 

MPIRUN: MPI progress Quiescence Detected after 9000 seconds. 

MPIRUN: 2 out of 2 ranks showed no MPI send or receive progress. 

MPIRUN: Per-rank details are the following: 

MPIRUN: Rank 0 ( ) caused MPI progress Quiescence. 

MPIRUN: Rank 1 ( ) caused MPI progress Quiescence. 

MPIRUN: both MPI progress and Ping Quiescence Detected after 120 

seconds. 

Occasionally, a stray process will continue to exist out of its context. mpirun 

checks for stray processes; they are killed after detection. The following code is 

an example of the type of message that displays in this case: 

$ mpirun -np 2 -ppn 1 -m ~/tmp/mfast mpi_latency 500000 2000 

iqa-38: Received 1 out-of-context eager message(s) from stray 

process PID=29745 

running on host 192.168.9.218 

iqa-35: PSM pid 10513 on host IP 192.168.9.221 has detected that I 

am a stray process, exiting. 

2000 5.222116 

iqa-38:1.ips_ptl_report_strays: Process PID=29745 on host 

IP=192.168.9.218 sent 

1 stray message(s) and was told so 1 time(s) (first stray message 

at 0.7s (13%),last at 0.7s (13%) into application run) 

The following message should never occur. If it does, notify Technical Support: 

Internal Error: NULL function/argument found:func_ptr(arg_ptr) 

Driver and Link Error Messages Reported by MPI Programs 

The following driver and link error messages are reported by MPI programs. 

When the InfiniBand link fails during a job, a message is reported once per 

occurrence. The message will be similar to: 

ipath_check_unit_status: IB Link is down 

D000046-005 B F-29



MPI Stats 

This message occurs when a cable is disconnected, a switch is rebooted, or when 

there are other problems with the link. The job continues retrying until the 

quiescence interval expires. See the mpirun -q option for information on 

quiescence. 

If a hardware problem occurs, an error similar to this displays: 

infinipath: [error strings ] Hardware error 

In this case, the MPI program terminates. The error string may provide additional 

information about the problem. To further determine the source of the problem, 

examine syslog on the node reporting the problem. 

Using the -print-stats option to mpirun provides a listing to stderr of 

various MPI statistics. Here is example output for the -print-stats option 

when used with an eight-rank run of the HPCC benchmark, using the following 

command: 

$ mpirun -np 8 -ppn 1 -m machinefile -M ./hpcc 

STATS: MPI Statistics Summary (max,min @ rank) 

STATS: Eager count sent (max=171.94K @ 0, min=170.10K @ 3, med=170.20K @ 5) 

STATS: Eager bytes sent (max=492.56M @ 5, min=491.35M @ 0, med=491.87M @ 1) 

STATS: Rendezvous count sent (max= 5735 @ 0, min= 5729 @ 3, med= 5731 @ 7) 

STATS: Rendezvous bytes sent (max= 1.21G @ 4, min= 1.20G @ 2, med= 1.21G @ 0) 

STATS: Expected count received(max=173.18K @ 4, min=169.46K @ 1, med=172.71K @ 7) 

STATS: Expected bytes received(max= 1.70G @ 1, min= 1.69G @ 2, med= 1.70G @ 7) 

STATS: Unexpect count received(max= 6758 @ 0, min= 2996 @ 4, med= 3407 @ 2) 

STATS: Unexpect bytes received(max= 1.48M @ 0, min=226.79K @ 5, med=899.08K @ 2) 

By default, -M assumes -M=mpi and that the user wants only mpi level statistics. 

The man page shows various other low-level categories of statistics that are 

provided. Here is another example: 

$ mpirun -np 8 -ppn 1 -m machinefile -M=mpi,ipath hpcc 

STATS: MPI Statistics Summary (max,min @ rank) 

STATS: Eager count sent (max=171.94K @ 0, min=170.10K @ 3, med=170.22K @ 1) 

STATS: Eager bytes sent (max=492.56M @ 5, min=491.35M @ 0, med=491.87M @ 1) 

STATS: Rendezvous count sent (max= 5735 @ 0, min= 5729 @ 3, med= 5731 @ 7) 

STATS: Rendezvous bytes sent (max= 1.21G @ 4, min= 1.20G @ 2, med= 1.21G @ 0) 

STATS: Expected count received(max=173.18K @ 4, min=169.46K @ 1, med=172.71K @ 7) 

STATS: Expected bytes received(max= 1.70G @ 1, min= 1.69G @ 2, med= 1.70G @ 7) 

STATS: Unexpect count received(max= 6758 @ 0, min= 2996 @ 4, med= 3407 @ 2) 

STATS: Unexpect bytes received(max= 1.48M @ 0, min=226.79K @ 5, med=899.08K @ 2) 

STATS: InfiniPath low-level protocol stats 

STATS: pio busy count (max=190.01K @ 0, min=155.60K @ 1, med=160.76K @ 5) 

STATS: scb unavail exp count (max= 9217 @ 0, min= 7437 @ 7, med= 7727 @ 4) 

STATS: tid update count (max=292.82K @ 6, min=290.59K @ 2, med=292.55K @ 4) 

STATS: interrupt thread count (max= 941 @ 0, min= 335 @ 7, med= 439 @ 2) 

STATS: interrupt thread success(max= 0.00 @ 3, min= 0.00 @ 1, med= 0.00 @ 0) 

F-30 D000046-005 B



Statistics other than MPI-level statistics are fairly low level; most users will not 

understand them. Contact QLogic Technical Support for more information. 

Message statistics are available for transmitted and received messages. In all 

cases, the MPI rank number responsible for a minimum or maximum value is 

reported with the relevant value. For application runs of at least three ranks, a 

median is also available. 

Since transmitted messages employ either an Eager or a Rendezvous protocol, 

results are available relative to both message count and aggregated bytes. 

Message count represents the amount of messages transmitted by each protocol 

on a per-rank basis. Aggregated amounts of message bytes indicate the total 

amount of data that was moved on each rank by a particular protocol. 

On the receive side, messages are split into expected or unexpected messages. 

Unexpected messages cause the MPI implementation to buffer the transmitted 

data until the receiver can produce a matching MPI receive buffer. Expected 

messages refer to the inverse case, which is the common case in most MPI 

applications. An additional metric, Unexpected count %, representing the 

proportion of unexpected messages in relation to the total number of messages 

received, is also shown because of the notable effect unexpected messages have 

on performance. 

For more detailed information, use MPI profilers such as mpiP. For more 

information on mpiP, see: http://mpip.sourceforge.net/ 

For information about the HPCC benchmark, see: http://icl.cs.utk.edu/hpcc/ 

D000046-005 B F-31



F-32 D000046-005 B

G ULP Troubleshooting 

Troubleshooting VirtualNIC and VIO Hardware 

Issues 

To verify that an InfiniBand host can access an Ethernet system through the EVIC, 

issue a ping command to the Ethernet system from the InfiniBand host. Make 

certain that the route to the Ethernet system is using the VIO hardware by using 

the Linux route command on the InfiniBand host, then verify that the route to the 

subnet is using one of the virtual Ethernet interfaces (i.e., an EIOC). 

NOTE: 

If the ping command fails, check the following: 

• The logical connection between the InfiniBand host and the EVIC 

Checking the logical connection between the InfiniBand Host and the 

VIO hardware. 

• The interface definitions on the host Checking the interface definitions 

on the host. 

• The physical connection between the VIO hardware and the Ethernet 

network Verify the physical connection between the VIO hardware and 

the Ethernet network. 

Checking the logical connection between the InfiniBand Host 

and the VIO hardware 

To determine if the logical connection between the InfiniBand host and the VIO 

hardware is correct, check the following: 

• The correct VirtualNIC driver is running. 

• The /etc/infiniband/qlgc_vnic.cfg file contains the desired 

information. 

• The host can communicate with the I/O Controllers (IOCs) of the VIO 

hardware. 

D000046-005 B G-1

G–ULP Troubleshooting 

Troubleshooting VirtualNIC and VIO Hardware Issues 

Verify that the proper VirtualNIC driver is running 

Check that a VirtualNIC driver is running by issuing an lsmod command on the 

InfiniBand host. Make sure that the qlgc_vnic is displayed on the list of 

modules. Following is an example: 

st186:~ # lsmod 

Module 

Size Used by 

cpufreq_ondemand 25232 1 

cpufreq_userspace 23552 0 

cpufreq_powersave 18432 0 

powernow_k8 30720 2 

freq_table 

22400 1 powernow_k8 

qlgc_srp 93876 0 

qlgc_vnic 116300 0# 

Verifying that the qlgc_vnic.cfg file contains the correct 

information 

Use the following scenarios to verify that the qlgc_vnic.cfg file contains a 

definition for the applicable virtual interface: 

Issue the command ib_qlgc_vnic_query to get the list of IOCs the host 

can see. 

If the list is empty, there may be a syntax error in the qlgc_vnic.cfg file (e.g., a 

missing semicolon). Look in /var/log/messages at the time qlgc_vnic was 

last started to see if any error messages were put in the log at that time. 

If the qlgc_vnic.cfg file has been edited since the last time the VirtualNIC 

driver was started, the driver needs restarted. To restart the driver, so that it uses 

the current qlgc_vnic.cfg file, issue a /etc/init.d/qlgc_vnic restart. 

G-2 D000046-005 B



Verifying that the host can communicate with the I/O 

Controllers (IOCs) of the VIO hardware 

To display the Ethernet VIO cards that the host can see and communicate with, 

issue the command ib_qlgc_vnic_query. The system returns information 

similar to the following: 

IO Unit Info: 


port GID: fe8000000000000000066a0258000001 




GUID: 00066a0130000001 




ID: Chassis 0x00066A00010003F2, Slot 1, IOC 1 


service[ 0]: 1000066a00000001 / 


service[ 1]: 1000066a00000101 / 



GUID: 00066a0230000001 






service[ 0]: 1000066a00000002 / 


service[ 1]: 1000066a00000102 / 



GUID: 00066a0330000001 




D000046-005 B G-3





service[ 0]: 1000066a00000003 / 


service[ 1]: 1000066a00000103 / 


When ib_qlgc_vnic_query is run with -e option, it reports the IOCGUID 

information. With the -s option it reports the IOCSTRING information for the 

Virtual I/O hardware IOCs present on the fabric. Following is an example: 

# ib_qlgc_vnic_query -e 

ioc_guid=00066a0130000001,dgid=fe8000000000000000066a02580000 

01,pkey=ffff 


01,pkey=ffff 


01,pkey=ffff 

#ib_qlgc_vnic_query -s 

"Chassis 0x00066A00010003F2, Slot 1, IOC 1" 



#ib_qlgc_vnic_query -es 


01,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 1" 





G-4 D000046-005 B



If the host can not see applicable IOCs, there are two things to check. First, verify 

that the adapter port specified in the eioc definition of the 

/etc/infiniband/qlgc_vnic.cfg file is active. This is done using the 

ibv_devinfo commands on the host, then checking the value of state. If the 

state is not Port_Active, the adapter port is not logically connected to the 

fabric. It is possible that one of the adapter ports is not physically connected to an 

InfiniBand switch. For example: 

st139:~ # ibv_devinfo 

hca_id: mlx4_0 

fw_ver: 2.2.000 

node_guid: 

0002:c903:0000:0f80 


0002:c903:0000:0f83 

vendor_id: 

0x02c9 


hw_ver: 

0xA0 

board_id: 

MT_04A0110002 


port: 1 

state: 

PORT_ACTIVE 

(4) 

max_mtu: 2048 (4) 


sm_lid: 1 

port_lid: 8 

port_lmc: 

0x00 

(4) 

port: 2 

state: 

PORT_ACTIVE 

max_mtu: 2048 (4) 


sm_lid: 1 

port_lid: 9 

port_lmc: 0x00# 

Second, verify that the adapter port specified in the EIOC definition is the correct 

port. The host sees the IOCs, but not over the adapter port in the definition of the 

IOC. For example, the host may see the IOCs over adapter Port 1, but the eioc 

definition in the /etc/infiniband/qlgc_vnic.cfg file specifies PORT=2. 

D000046-005 B G-5



Another reason why the host might not be able to see the necessary IOCs is that 

the subnet manager has gone down. Issue an iba_saquery command to make 

certain that the response shows all of the nodes in the fabric. If an error is 

returned and the adapter is physically connected to the fabric, then the subnet 

manager has gone down, and this situation needs to be corrected. 

Checking the interface definitions on the host 

If it is not possible to ping from an InfiniBand host to the Ethernet host, and the 

ViPort State of the interface is VIPORT_CONNECTED, then issue an 

ifconfig command. The interfaces defined in the configuration files listed in 

/etc/sysconfig/network directory for SLES hosts or the 

/etc/sysconfig/network-scripts for Red Hat hosts should be displayed in 

the list of interfaces in the ifconfig output. For example, the ifconfig file 

should show an interface for each EIOC configuration file in the following list: 

# ls /etc/sysconfig/network-scripts 

ifcfg-eioc1 

ifcfg-eioc2 

ifcfg-eioc3 

ifcfg-eioc4 

ifcfg-eioc5 

ifcfg-eioc6 

Interface does not show up in output of 'ifconfig' 

If an interface is not displayed in the output of an ifconfig command, there is 

most likely a problem in the definition of that interface in the 

/etc/sysconfig/network-scripts/ifcfg- (for RedHat systems) 

or /etc/sysconfig/network/ifcfg- (for SuSE systems) file, where 

is the name of the virtual interface (e.g., eioc1). 

NOTE: 

For the remainder of this section, ifcfg directory refers to 

/etc/sysconfig/network-scripts/ on RedHat systems, and 

/etc/sysconfig/network on SuSE systems. 

Issue an ifup command. If the interface is displayed when issuing an 

ifconfig command, there may be a problem with the way the interface startup 

is defined in the ifcfg directory'/ifcfg- file that is preventing the 

interface from coming up automatically. 

If the interface does not come up, check the interface definitions in the ifcfg 

directory. Make certain that there are no misspellings in the ifcfg- file. 

G-6 D000046-005 B



Example of ifcfg-eiocx setup for RedHat systems: 

DEVICE=eioc1 


IPADDR=172.26.48.132 

BROADCAST=172.26.63.130 

NETMASK=255.255.240.0 

NETWORK=172.26.48.0 

ONBOOT=yes 

TYPE=Ethernet 

Example of ifcfg-eiocx setup for SuSE and SLES systems: 

BOOTPROTO='static' 

IPADDR='172.26.48.130' 

BROADCAST='172.26.63.255' 

NETMASK='255.255.240.0' 

NETWORK='172.26.48.0' 

STARTMODE='hotplug' 

TYPE='Ethernet' 

Verify the physical connection between the VIO hardware and 

the Ethernet network 

If the interface is displayed in an ifconfig and a ping between the InfiniBand 

host and the Ethernet host is still unsuccessful, verify that the VIO hardware 

Ethernet ports are physically connected to the correct Ethernet network. Verify 

that the Ethernet port corresponding to the IOCGUID for the interface to be used 

is connected to the expected Ethernet network. 

There are up to 6 IOC GUIDs on each VIO hardware module (6 for the IB/Ethernet 

Bridge Module, 2 for the EVIC), one for each Ethernet port. If a VIO hardware 

module can be seen from a host, the ib_qlgc_vnic_query -s file displays 

information similar to: 

EVIC in Chassis 0x00066a000300012a, Slot 19, Ioc 1 




EVIC in Chassis 0x00066a00da000100, Slot 2, Ioc 1 

EVIC in Chassis 0x00066a00da000100, Slot 2, Ioc 2 

D000046-005 B G-7


Troubleshooting SRP Issues 


ib_qlgc_srp_stats showing session in disconnected state 

Problem: 

If the session is part of a multi-session adapter, ib_qlgc_srp_stats will show it 

to be in the disconnected state. For example: 

SCSI Host # : 17 | Mode : ROUNDROBIN 

Trgt Adapter Depth : 1000 | Verify Target : Yes 

Rqst Adapter Depth : 1000 | Rqst LUN Depth : 16 

Tot Adapter Depth : 1000 | Tot LUN Depth : 16 

Act Adapter Depth : 998 | Act LUN Depth : 16 

Max LUN Scan : 512 | Max IO : 131072 (128 KB) 

Max Sectors : 256 | Max SG Depth : 33 

Session Count : 2 | No Connect T/O : 60 Second(s) 

Register In Order : ON | Dev Reqst T/O : 2 Second(s) 

Description : SRP Virtual HBA 1 

Session : Session 1 | State : Disconnected 

Source GID 

: 0xfe8000000000000000066a000100d051 

Destination GID : 0xfe8000000000000000066a0260000165 


SRP Target IOClass : 0xFF00 

| SRP Target SID : 0x0000494353535250 

SRP IPI Guid : 0x00066a000100d051 | SRP IPI Extnsn : 0x0000000000000001 

SRP TPI Guid : 0x00066a0138000165 | SRP TPI Extnsn : 0x0000000000000001 

Source LID : 0x000b | Dest LID : 0x0004 

Completed Sends : 0x00000000000002c0 | Send Errors : 0x0000000000000000 

Completed Receives : 0x00000000000002c0 | Receive Errors : 0x0000000000000000 

Connect Attempts : 0x0000000000000000 | Test Attempts : 0x0000000000000000 

Total SWUs 

: 0x00000000000003e8 | Available SWUs : 0x00000000000003e8 

Busy SWUs : 0x0000000000000000 | SRP Req Limit : 0x00000000000003e8 

SRP Max ITIU : 0x0000000000000140 | SRP Max TIIU : 0x0000000000000140 

Host Busys 

: 0x0000000000000000 | SRP Max SG Used : 0x000000000000000f 


Source GID 

: 0xfe8000000000000000066a000100d052 







Source LID : 0x000c | Dest LID : 0x0004 




Total SWUs 






G-8 D000046-005 B



Solution: 

Perhaps an interswitch cable has been disconnected, or the VIO hardware is 

offline, or the Chassis/Slot does not contain a VIO hardware card. Instead of 

looking at this file, use the ib_qlgc_srp_query command to verify that the 

desired adapter port is in the active state. 

NOTE: 

It is normal to see the "Can not find a path" message when the system 

first boots up. Sometimes SRP comes up before the subnet manager has 

brought the port state of the adapter port to active. If the adapter port is not 

active, SRP will not be able to find the VIO hardware card. Use the 

appropriate OFED command to show the port state. 

Session in 'Connection Rejected' state 

Problem: 

The session is in the 'Connection Rejected' state according to 

/var/log/messages. If the session is part of a multi-session adapter, 

ib_qlgc_srp_stats shows it in the "Connection Rejected" state. 

A host displays 

"Connection Failed for Session X: IBT Code = 0x0 

"Connection Failed for Session X: SRP Code = 0x1003 

"Connection Rejected" 

D000046-005 B G-9



Following is an example: 

SCSI Host # : 17 | Mode : ROUNDROBIN 

Trgt Adapter Depth : 1000 | Verify Target : Yes 

Rqst Adapter Depth : 1000 | Rqst LUN Depth : 16 

Tot Adapter Depth : 1000 | Tot LUN Depth : 16 

Act Adapter Depth : 998 | Act LUN Depth : 16 

Max LUN Scan : 512 | Max IO : 131072 (128 KB) 

Max Sectors : 256 | Max SG Depth : 33 

Session Count : 2 | No Connect T/O : 60 Second(s) 

Register In Order : ON | Dev Reqst T/O : 2 Second(s) 

Description : SRP Virtual HBA 1 


Source GID 

: 0xfe8000000000000000066a000100d051 







Source LID : 0x000b | Dest LID : 0x0004 




Total SWUs 







Source GID 

: 0xfe8000000000000000066a000100d052 







Source LID : 0x000c | Dest LID : 0x0004 




Total SWUs 






AND 

The VIO hardware displays "Initiator Not Configured within IOU: 

initiator 

Initiatorport 

identifier ( is 

invalid/not allowed to use this FCIOU”. 

G-10 D000046-005 B



Solution 1: 

The host initiator has not been configured as an SRP initiator on the VIO 

hardware SRP Initiator Discovery screen. Via Chassis Viewer, bring up the SRP 

Initiator Discovery screen and either 

Click on 'Add New' to add a wildcarded entry with the initiator extension to 

match what is in the session entry in the qlgc_srp.cfg file, or 

Click on the Start button to discover the adapter port GUID, and then click 

'Configure' on the row containing the adapter port GUID and give the entry 

a name. 

Solution 2: 

Check the SRP map on the VIO hardware specified in the failing Session block of 

the qlgc_srp.cfg file. Make certain there is a map defined for the row specified 

by either the initiatorExtension in the failing Session block of the 

qlgc_srp.cfg file or the adapter port GUID specified in the failing Session block 

of the qlgc_srp.cfg file. Additionally, make certain that the map in that row is in 

the column of the IOC specified in the failing Session block of the qlgc_srp.cfg 

file. 

Attempts to read or write to disk are unsuccessful 

Problem: 

Attempts to read or write to the disk are unsuccessful when SRP comes up. About 

every five seconds the VIO hardware displays 

"fcIOStart Failed", 

"CMDisconnect called for Port: xxxxxxxx Initiator: Target: ", 

and 

"Target Port Deleted for Port: xxxxxxxx Initiator: Target: " 

The host log shows a session transitioning between Connected and Down. The 

host log also displays "Test Unit Ready has FAILED", "Abort Task has 

SUCCEEDED", and "Clear Task has FAILED". 

Solution: 

This indicates a problem in the path between the VIO hardware and the target 

storage device. After an SRP host has connected to the VIO hardware 

successfully, the host sends a "Test Unit Ready" command to the storage 

device. After five seconds, if that command is not responded to, the SRP host 

brings down the session and retries in five seconds. Verify that the status of the 

connection between the appropriate VIO hardware port and the target device is 

UP on the FCP Device Discovery screen. 

D000046-005 B G-11



Problem: 

Attempts to read or write to the disk are unsuccessful, when they were previously 

successful. The host displays 'Sense Data indicates recovery is 

necessary on Session' and the "Test Unit Ready has FAILED", "Abort 

Task has SUCCEEDED", "Clear Task has FAILED" messages. 

Solution: 

If there is a problem with communication between the VIO hardware and the 

storage device (e.g., the cable between the storage device and the Fibre Channel 

switch was pulled) the VIO hardware log will display a "Connection Lost to 

NPort Id" message. The next time the host tries to do an input/output (I/O), the 

'Sense Data indicates recovery is necessary' appears. Then SRP will 

recycle the session. As part of trying to move the session from 'Connected' to 

'Active', SRP will issue the 'Test Unit Ready' command. 

Verify that the status of the connection between the appropriate VIO hardware 

port and the target device is UP on the FCP Device Discovery screen. 

Additionally, there may occasionally be messages in the log such as: 

Connection Failed for Session X: IBT Code = 0x0 

Connection Failed for Session X: SRP Code = 0x0 

That may indicate a problem in the path between the VIO hardware and the target 

storage device. 

Four sessions in a round-robin configuration are active 

Problem: 

Four sessions in a round-robin configuration are active according to 

ib_qlgc_srp_stats. However, only one disk can be seen, although five should 

be seen. 

Solution 1: 

Make certain that Max LUNs Scanned is reporting the same value as 

adapterMaxLUNs is set to in qlgc_srp.cfg. 

Solution 2: 

Make certain that all sessions have a map to the same disk defined. The fact that 

the session is active means that the session can see a disk. However, if one of the 

sessions is using a map with the 'wrong' disk, then the round-robin method could 

lead to a disk or disks not being seen. 

Which port does a port GUID refer to 

Solution: 

A QLogic Host Channel Adapter Port GUID is of the form 00066appa0iiiiii 

G-12 D000046-005 B



where pp gives the port number (0 relative) 

and iiiiiii gives the individual id number of the adapter 

so 00066a00a0iiiiiii is the port guid of the 1st port of the adapter 

and 00066a01a0iiiiiii is the port guid of the 2nd port of the adapter. 

Similarly, a VFx Port GUID is of the form 00066app38iiiiii 

where pp gives the IOC number (1 or 2) 

and iiiiiii gives the individual ID number of the VIO hardware 

so 00066a0138iiiiiii is the port guid of IOC 1 of VIO hardware iiiiiii 

and 00066a0238iiiiiii is the port guid of IOC 2 of VIO hardware iiiiiii 

NOTE: 

After a virtual adapter has been successfully added (meaning at least 1 

session where part of the adapter has gone to the Active state) the SRP 

module will indicates what type of session was created in the mode variable 

(ib_qlgc_srp_stats) file, depending on whether "roundrobinmode: 

1" is set in the qlgc_srp.cfg file. In this case "X" is the virtual adapter 

number, with number 0 being the first one created. 

If no sessions were successfully brought to the Active state, then the 

roundrobin_X or failover_X file will not be created. 

In a round robin configuration, if everything is configured correctly, all sessions will 

be Active. 

In a failover configuration, if everything is configured correctly, one session will be 

Active and the rest will be Connected. The transition of a session from Connected 

to Active will not be attempted until that session needs to become Active, due to 

the failure of the previously Active session. 

How does the user find a Host Channel Adapter port GUID 

Solution: 

A Host Channel Adapter Port GUID is displayed by entering the following at any 

host prompt: 

ibv_devinfo -i 1 for port 1 

ibv_devinfo -i 2 for port 2 

D000046-005 B G-13



The system displays information similar to the following: 

st106:~ # ibv_devinfo -i 1 

hca_id: mthca0 

fw_ver: 5.1.9301 

node_guid: 

0006:6a00:9800:6c9f 


0006:6a00:9800:6c9f 

vendor_id: 

0x066a 


hw_ver: 

0xA0 

board_id: 

SS_0000000005 


port: 1 


max_mtu: 2048 (4) 


sm_lid: 71 

port_lid: 60 

port_lmc: 

0x00 

st106:~ # ibv_devinfo -i 2 

hca_id: mthca0 

fw_ver: 5.1.9301 

node_guid: 

0006:6a00:9800:6c9f 


0006:6a00:9800:6c9f 

vendor_id: 

0x066a 


hw_ver: 

0xA0 

board_id: 

SS_0000000005 


port: 2 


max_mtu: 2048 (4) 


sm_lid: 71 

port_lid: 64 

port_lmc: 

0x00 

G-14 D000046-005 B



Need to determine the SRP driver version. 

Solution: 

To determine the SRP driver version number, enter the command modinfo -d 

qlgc-srp, which returns information similar to the following: 

st159:~ # modinfo -d qlgc-srp 

QLogic Corp. Virtual HBA (SRP) SCSI Driver, version 1.0.0.0.3 

D000046-005 B G-15



G-16 D000046-005 B

H 

Write Combining 

Introduction 

Write combining improves write bandwidth to the QLogic chip by writing multiple 

words in a single bus transaction (typically 64 bytes). Write combining applies only 

to x86_64 systems. 

The x86 Page Attribute Table (PAT) mechanism that allocates Write Combining 

(WC) mappings for the PIO buffers has been added and is now the default. 

If PAT is unavailable or PAT initialization fails, the code will generate a message in 

the log and fall back to the Memory Type Range Registers (MTRR) mechanism. 

If write combining is not working properly, lower than expected bandwidth may 

occur. 

The following sections provide instructions for checking write combining and for 

using PAT and MTRR. 

Verify Write Combining is Working 

To see if write combining is working correctly and to check the bandwidth, run the 

following command: 

$ ipath_pkt_test -B 

With write combining enabled, the QLE7140 and QLE7240 report in the range 

of 1150–1500 MBps. The QLE7280 reports in the range of 1950–3000 MBps. 

You can also use ipath_checkout (use option 5) to check bandwidth. 

Although the PAT mechanism should work correctly by default, increased latency 

and low bandwidth may indicate a problem. If so, the interconnect operates, but in 

a degraded performance mode, with latency increasing to several microseconds, 

and bandwidth decreasing to as little as 200 MBps. 

Upon driver startup, you may see these errors: 

ib_qib 0000:04:01.0: infinqib0: Performance problem: bandwidth to 

PIO buffers is only 273 MiB/sec 

. 

. 

. 

D000046-005 B H-1

H–Write Combining 

PAT and Write Combining 

If you do not see any of these messages on your console, but suspect this 

problem, check the /var/log/messages file. Some systems suppress driver 

load messages but still output them to the log file. 

Methods for enabling and disabling the two write combining mechanisms are 

described in the following sections. There are no conflicts between the two 

methods. 

PAT and Write Combining 

This is the default mechanism for allocating Write Combining (WC) mappings for 

the PIO buffers. It is set as a parameter in /etc/modprobe.conf (on Red Hat 

systems) or /etc/modprobe.conf.local (on SLES systems). The default is: 

option ib_qib wc_pat=1 

If PAT is unavailable or PAT initialization fails, the code generates a message in 

the log and falls back to the Memory Type Range Registers (MTRR) mechanism. 

To use MTRR, disable PAT by setting this module parameter to 0 (as a root user): 

option ib_qib wc_pat=0 

Then, revert to using MTRR-only behavior by following one of the two suggestions 

in “MTRR Mapping and Write Combining” on page H-2. 

The driver must be restarted after the changes have been made. 

NOTE: 

There will be no WC entry in /proc/mtrr when using PAT. 

MTRR Mapping and Write Combining 

. 

Two suggestions for properly enabling MTRR mapping for write combining are 

described in the following sections. 

See “Performance Issues” on page F-9 for more details on a related performance 

issue. 

Edit BIOS Settings to Fix MTRR Issues 

You can edit the BIOS setting for MTRR mapping. The BIOS setting looks similar 

to: 

MTRR Mapping 

[Discrete] 

For systems with very large amounts of memory (32GB or more), it may also be 

necessary to adjust the BIOS setting for the PCI hole granularity to 2GB. This 

setting allows the memory to be mapped with fewer MTRRs, so that there will be 

one or more unused MTRRs for the InfiniPath driver. 

H-2 D000046-005 B



Some BIOS’ do not have the MTRR mapping option. It may have a different 

name, depending on the chipset, vendor, BIOS, or other factors. For example, it is 

sometimes referred to as 32 bit memory hole. This setting must be enabled. 

If there is no setting for MTRR mapping or 32 bit memory hole, and you have 

problems with degraded performance, contact your system or motherboard 

vendor and ask how to enable write combining. 

Use the ipath_mtrr Script to Fix MTRR Issues 

QLogic also provides a script, ipath_mtrr, which sets the MTRR registers, 

enabling maximum performance from the InfiniPath driver. This Python script is 

available as a part of the InfiniPath software download, and is contained in the 

infinipath* RPM. It is installed in /bin. 

To diagnose the machine, run it with no arguments (as a root user): 

# ipath_mtrr 

The test results will list any problems, if they exist, and provide suggestions on 

what to do. 

To fix the MTRR registers, use: 

# ipath_mtrr -w 

Restart the driver after fixing the registers. 

This script needs to be run after each system reboot. It can be set to run 

automatically upon restart by adding this line in /etc/sysconfig/infinipath: 

IPATH_MTRR_ACTIVE=1 

See the ipath_mtrr(8) man page for more information on other options. 

D000046-005 B H-3



Notes 

H-4 D000046-005 B

I 

Useful Programs and Files 

The most useful programs and files for debugging, and commands for common 

tasks, are presented in the following sections. Many of these programs and files 

have been discussed elsewhere in the documentation. This information is 

summarized and repeated here for your convenience. 

Check Cluster Homogeneity with ipath_checkout 

Many problems can be attributed to the lack of homogeneity in the cluster 

environment. Use the following items as a checklist for verifying homogeneity. A 

difference in any one of these items in your cluster may cause problems: 

• Kernels 

• Distributions 

• Versions of the QLogic boards 

• Runtime and build environments 

• .o files from different compilers 

• Libraries 

• Processor/link speeds 

• PIO bandwidth 

• MTUs 

With the exception of finding any differences between the runtime and build 

environments, ipath_checkout will pick up information on all the above items. 

Other programs useful for verifying homogeneity are listed in Table I-1. More 

details on ipath_checkout are in “ipath_checkout” on page I-10. 

Restarting InfiniPath 

When the driver status appears abnormal on any node, you can try restarting (as 

a root user). Type: 

# /etc/init.d/openibd restart 

These two commands perform the same function as restart: 

# /etc/init.d/openibd stop 

# /etc/init.d/openibd start 

Also check the /var/log/messages file for any abnormal activity. 

D000046-005 B I-1

I–Useful Programs and Files 

Summary and Descriptions of Useful Programs 


Useful programs are summarized in Table I-1. Names in blue text are linked to a 

corresponding section that provides further details. Check the man pages for 

more information on the programs. 

Table I-1. Useful Programs 

Program Name 

chkconfig 

dmesg 

Function 

Checks the configuration state and enables/disables services, 

including drivers. Can be useful for checking homogeneity. 

Prints out bootup messages. Useful for checking for initialization 

problems. 

ibhosts a 

ibstatus a 

ibtracert a 

ibv_devinfo a 

ident b 

ipathbug-helper c 

ipath_checkout c 

ipath_control c 

ipath_mtrr c 

ipath_pkt_test c 

Checks that all hosts in the fabric are up and visible to the 

subnet manager and to each other 

Checks the status of InfiniBand devices when OpenFabrics is 

installed 

Determines the path that InfiniBand packets travel between 

two nodes 

Lists information about InfiniBand devices in use. Use when 

OpenFabrics is enabled. 

Identifies RCS keyword strings in files. Can check for dates, 

release versions, and other identifying information. 

A shell script that gathers status and history information for 

use in analyzing InfiniPath problems 

A bash shell script that performs sanity testing on a cluster 

using QLogic hardware and InfiniPath software. When the 

program runs without errors, the node is properly configured. 

A shell script that manipulates various parameters for the 

InfiniPath driver. 

This script gathers the same information contained in boardversion, 

status_str, and version. 

A Python script that sets the MTRR registers. 

Tests the InfiniBand link and bandwidth between two QLogic 

infiniband adapters, or, using an InfiniBand loopback connector, 

tests within a single QLogic infiniband adapter 

I-2 D000046-005 B



Table I-1. Useful Programs (Continued) 

Program Name 

ipathstats c 

lsmod 

modprobe 

mpi_stress 

mpirun d 

ps 

rpm 

strings e 

Table Notes 

Function 

Displays driver statistics and hardware counters, including 

performance and "error" (including status) counters 

Shows status of modules in the Linux kernel. Use to check 

whether drivers are loaded. 

Adds or removes modules from the Linux kernel. 

An MPI stress test program designed to load up an MPI interconnect 

with point-to-point messages while optionally checking 

for data integrity. 

A front end program that starts an MPI job on an InfiniPath 

cluster. Use to check the origin of the drivers. 

Displays information on current active processes. Use to 

check whether all necessary processes have been started. 

Package manager to install, query, verify, update, or erase 

software packages. Use to check the contents of a package. 

Prints the strings of printable characters in a file. Useful for 

determining contents of non-text files such as date and version. 

a 

These programs are contained in the OpenFabrics openib-diags RPM. 

b 

These programs are contained within the rcs RPM for your distribution. 

c 

These programs are contained in the infinipath RPM. To use these programs, install the 

infinipath RPM on the nodes where you install the mpi-frontend RPM. 

d 

These programs are contained in the QLogic mpi-frontend RPM. 

e 

These programs are contained within the binutils RPM for your distribution. 

dmesg 

dmesg prints out bootup messages. It is useful for checking for initialization 

problems. You can check to see if problems were detected during the driver and 

QLogic hardware initialization with the command: 

$ dmesg|egrep -i infinipath|qib 

This command may generate more than one screen of output. 

D000046-005 B I-3



iba_opp_query 

This command retrieves path records from the Distributed SA and is somewhat 

similar to iba_saquery. It is intended for testing the Distributed SA 

(qlogic_sa) and for verifying connectivity between nodes in the fabric. For 

information on configuring and using the Distributed SA, refer to “QLogic 

Distributed Subnet Administration” on page 3-13. 

iba_opp_query does not access the SM when doing queries, it only accesses 

the local Distributed SA database. For that reason, the kinds of queries that can 

be done are much more limited than with iba_saquery. In particular, it can only 

find paths that start on the machine where the command is run. (In other words, 

the source LID or source GID must be on the local node.) In addition, queries 

must supply either a source and destination LID, or a source and destination GID. 

They cannot be mixed. In addition, you will usually need to provide either a SID 

that was specified in Distributed SA configuration file, or a pkey that matches such 

a SID. 

Usage 

iba_opp_query [-v level] [-hca hca] [-p port] [-s LID] [-d 

LID] [-S GID] [-D GID] [-k pkey] [-i sid] [-H] 

Options 

-v/--verbose level — Debug level. Should be a number between 1 

and 7. Default is 5. 

-s/--slid LID — Source LID. Can be in decimal, hex (0x##) or octal 

(0##) 

-d/--dlid LID — Destination LID. Can be in decimal, hex (0x##) or octal 

(0##) 

-S/--sgid GID — Source GID. (Can be in GID 

("0x########:0x########") or inet6 format ("##:##:##:##:##:##:##:##")) 

-D/--dgid GID — Destination GID. (Can be in GID 

("0x########:0x########") or inet6 format ("##:##:##:##:##:##:##:##")) 

-k/--pkey pkey — Partition Key 

-i/--sid sid — Service ID 

-h/--hca hca — The Host Channel Adapter to use. (Defaults to the first 

Host Channel Adapter.) The Host Channel Adapter can be identified by 

name ("mthca0", “qib1”, et cetera) or by number (1, 2, 3, et cetera). 

-p/--port port — The port to use. (Defaults to the first port) 

-H/--help — Provides this help text. 

All arguments are optional, but ill-formed queries can be expected to fail. You 

must provide at least a pair of LIDs, or a pair of GIDs. 

I-4 D000046-005 B



Sample output: 



resv1 

0x0000000000000107 

dgid :: 

sgid :: 

dlid 

0x75 

slid 

0x31 

hop 

0x0 

flow 

0x0 

tclass 

0x0 

num_path 

0x0 

pkey 

0x0 

qos_class 

0x0 

sl 

0x0 

mtu 

0x0 

rate 

0x0 

pkt_life 

0x0 

preference 

0x0 

resv2 

0x0 

resv3 

0x0 


Result: 

resv1 

0x0000000000000107 

dgid 

fe80::11:7500:79:e54a 

sgid 

fe80::11:7500:79:e416 

dlid 

0x75 

slid 

0x31 

hop 

0x0 

flow 

0x0 

tclass 

0x0 

num_path 

0x0 

pkey 

0xffff 

qos_class 

0x0 

sl 

0x1 

mtu 

0x4 

rate 

0x6 

pkt_life 

0x10 

preference 

0x0 

D000046-005 B I-5



resv2 

resv3 

0x0 

0x0 

Explanation of Sample Output: 

This is a simple query, specifying the source and destination LIDs and the 

desired SID. The first half of the output shows the full “query” that will be 

sent to the Distributed SA. Unused fields are set to zero or are blank. 

In the center, the line “Using HCA qib0” tells us that, because we did not 

specify which Host Channel Adapter to query against, the tool chose one for 

us. (Normally, the user will never have to specify which Host Channel 

Adapter to use. This is only relevant in the case where a single node is 

connected to multiple physical IB fabrics.) 

Finally, the bottom half of the output shows the result of the query. Note that, 

if the query had failed (because the destination does not exist or because 

the SID is not found in the Distributed SA) you will receive and error instead: 



resv1 

0x0000000000000108 

dgid :: 

sgid :: 

dlid 

0x75 

slid 

0x31 

hop 

0x0 

flow 

0x0 

tclass 

0x0 

num_path 

0x0 

pkey 

0x0 

qos_class 

0x0 

sl 

0x0 

mtu 

0x0 

rate 

0x0 

pkt_life 

0x0 

preference 

0x0 

resv2 

0x0 

resv3 

0x0 


****** 

Error: Get Path returned 22 for query: Invalid argument 

****** 

I-6 D000046-005 B



ibhosts 

ibstatus 

Examples: 

Query by LID and SID: 

iba_opp_query -s 0x31 -d 0x75 -i 0x107 

iba_opp_query --slid 0x31 --dlid 0x75 --sid 0x107 

Queries using octal or decimal numbers: 

iba_opp_query --slid 061 --dlid 0165 --sid 0407 (using octal 

numbers) 

iba_opp_query –slid 49 –dlid 113 –sid 263 (using decimal 

numbers) 

Note that these queries are the same as the first two, only the base of the 

numbers has changed. 

Query by LID and PKEY: 

iba_opp_query --slid 0x31 --dlid 0x75 –pkey 0x8002 

Query by GID: 

iba_opp_query -S fe80::11:7500:79:e416 -D 

fe80::11:7500:79:e54a --sid 0x107 

iba_opp_query -S 0xfe80000000000000:0x001175000079e416 -D 

0xfe80000000000000:0x001175000079e394 --sid 0x107 

As before, these queries are identical to the first two queries – they are just 

using the GIDs instead of the LIDs to specify the ports involved. 

This tool determines if all the hosts in your InfiniBand fabric are up and visible to 

the subnet manager and to each other. It is installed from the openib-diag 

RPM. Running ibhosts (as a root user) produces output similar to this when run 

from a node on the InfiniBand fabric: 

# ibhosts 

Ca : 0x0008f10001280000 ports 2 "Voltaire InfiniBand 

Fiber-Channel Router" 

Ca : 0x0011750000ff9869 ports 1 "idev-11" 


Ca : 0x0011750000ff985c ports 1 "idev-06" 


This program displays basic information on the status of InfiniBand devices that 

are currently in use when OpenFabrics RPMs are installed. It is installed from the 

openib-diag RPM. 

D000046-005 B I-7



ibtracert 

Following is a sample output for the SDR adapters: 

$ ibstatus 


default gid: fe80:0000:0000:0000:0011:7500:0005:602f 

base lid: 0x35 

sm lid: 

0x2 

state: 

4: ACTIVE 


rate: 

10 Gb/sec (4X) 

Following is a sample output for the DDR adapters; note the difference in rate: 

$ ibstatus 


default gid: fe80:0000:0000:0000:0011:7500:00ff:9608 

base lid: 0xb 

sm lid: 

0x1 

state: 

4: ACTIVE 


rate: 

20 Gb/sec (4X DDR) 

The tool ibtracert determines the path that InfiniBand packets travel between 

two nodes. It is installed from the openib-diag RPM. The InfiniBand LIDs of the 

two nodes in this example are determined by using the ipath_control -i 

command on each node. The ibtracert tool produces output similar to the 

following when run (as a root user) from a node on the InfiniBand fabric: 

# ibtracert 0xb9 0x9a 

From ca {0x0011750000ff9886} portnum 1 lid 0xb9-0xb9 "iqa-37" 

[1] -> switch port {0x0002c9010a19bea0}[1] lid 0x14-0x14 

"MT47396 Infiniscale-III" 

[24] -> switch port {0x00066a0007000333}[8] lid 0xc-0xc 

"SilverStorm 9120 GUID=0x00066a000200016c Leaf 6, Chip A" 

[6] -> switch port {0x0002c90000000000}[15] lid 0x9-0x9 

"MT47396 Infiniscale-III" 

[7] -> ca port {0x0011750000ff9878}[1] lid 0x9a-0x9a "idev-05" 

To ca {0x0011750000ff9878} portnum 1 lid 0x9a-0x9a "idev-05" 

I-8 D000046-005 B



ibv_devinfo 

This program displays information about InfiniBand devices, including various 

kinds of identification and status data. It is installed from the openib-diag RPM. 

Use this program when OpenFabrics is enabled. ibv_devinfo queries RDMA 

devices. Use the -v option to see more information. For example: 

$ ibv_devinfo 

hca_id: qib0 

fw_ver: 0.0.0 

node_guid: 

0011:7500:00ff:89a6 


0011:7500:00ff:89a6 

vendor_id: 

0x1175 


hw_ver: 

0x2 

board_id: 

InfiniPath_QLE7280 


port: 1 


max_mtu: 4096 (5) 


sm_lid: 1 

port_lid: 31 

port_lmc: 

0x00 

ident 

The ident strings are available in ib_qib.ko. Running ident provides driver 

information similar to the following. For QLogic RPMs, it will look like the following 

example: 

$ ident /lib/modules/$(uname-r)/updates/kernel/drivers/ 

infiniband/hw/ib_qib.ko 

/lib/modules/$(uname-r)/updates/kernel/drivers/infiniband/hw/ib_qi 

b.ko: 

$Id: QLogic OFED Release 1.5 $ 

$Date: 2010-02-17-18:51 $ 

If the /lib/modules/$(uname -r)/updates directory is not present, 

then the driver in use is the one that comes with the core kernel. 

In this case, either the kernel-ib RPM is not installed or it is 

not configured for the current running kernel. 

D000046-005 B I-9



If the updates directory is present, but empty except for the subdirectory 

kernel, then an OFED install is probably being used, and the ident string will 

be empty. For example: 

$ cd /lib/modules/$(uname -r)/updates 

$ ls 

kernel 

$ cd kernel/drivers/infiniband/hw/qib/ 

lib/modules/2.6.18-8.el5/updates/kernel/drivers/infiniband/hw/qib 

$ ident ib_qib.ko 

ib_qib.ko: 

ident warning: no id keywords in ib_qib.ko 

ipathbug-helper 

ipath_checkout 

NOTE: 

ident is in the optional rcs RPM, and is not always installed. 

The tool ipathbug-helper is useful for verifying homogeneity. It is installed 

from the infinipath RPM. Before contacting QLogic Technical Support, run this 

script on the head node of your cluster and the compute nodes that you suspect 

are having problems. Looking at the output often helps you find the problem. Run 

ipathbug-helper on several nodes and examine the output for differences. 

It is best to run ipathbug-helper with root privilege, since some of the queries 

it makes require this level of privilege. There is also a --verbose parameter that 

increases the amount of gathered information. 

If you cannot see the problem, send the stdout output to your reseller, along with 

information on the version of the InfiniPath software you are using. 

The ipath_checkout tool is a bash script that verifies that the installation is 

correct and that all the nodes of the network are functioning and mutually 

connected by the InfiniPath fabric. It is installed from the infinipath RPM. It 

must be run on a front end node, and requires specification of a nodefile. For 

example: 

$ ipath_checkout [options] nodefile 

The nodefile lists the hostnames of the nodes of the cluster, one hostname per 

line. The format of nodefile is as follows: 

hostname1 

hostname2 

... 

I-10 D000046-005 B



NOTE: 

• The hostnames in the nodefile are Ethernet hostnames, not IPv4 

addresses. 

• To create a nodefile, use the ibhosts program. It will generate a list 

of available nodes that are already connected to the switch. 

ipath_checkout performs the following seven tests on the cluster: 

1. Executes the ping command to all nodes to verify that they all are 

reachable from the front end. 

2. Executes the ssh command to each node to verify correct configuration of 

ssh. 

3. Gathers and analyzes system configuration from the nodes. 

4. Gathers and analyzes RPMs installed on the nodes. 

5. Verifies InfiniPath hardware and software status and configuration, including 

tests for link speed, PIO bandwidth (incorrect MTRR settings), and MTU 

size. 

6. Verifies the ability to mpirun jobs on the nodes. 

7. Runs a bandwidth and latency test on every pair of nodes and analyzes the 

results. 

The options available with ipath_checkout are shown in Table I-2. 

Table I-2. ipath_checkout Options 

Command 

-h, --help 

-v, --verbose 

-vv, --vverbose 

-vvv, --vvverbose 

-c, --continue 

Meaning 

These options display help messages describing how a command 

is used. 

These options specify three successively higher levels of 

detail in reporting test results. There are four levels of detail 

in all, including the case where none of these options are 

given. 

When this option is not specified, the test terminates when 

any test fails. When specified, the tests continue after a failure, 

with failing nodes excluded from subsequent tests. 

D000046-005 B I-11



Table I-2. ipath_checkout Options (Continued) 

Command 

-k, --keep 

--workdir=DIR 

--run=LIST 

--skip=LIST 

-d, --debug 

Meaning 

This option keeps intermediate files that were created while 

performing tests and compiling reports. Results are saved in 

a directory created by mktemp and named 

infinipath_XXXXXX or in the directory name given to 

--workdir. 

Use DIR to hold intermediate files created while running 

tests. DIR must not already exist. 

This option runs only the tests in LIST. See the seven tests 

listed previously. For example, --run=123 will run only 

tests 1, 2, and 3. 

This option skips the tests in LIST. See the seven tests listed 

previously. For example, --skip=2457 will skip tests 2, 4, 

5, and 7. 

This option turns on the -x and -v flags in bash(1). 

ipath_control 

In most cases of failure, the script suggests recommended actions. Also refer to 

the ipath_checkout man page. 

The ipath_control tool is a shell script that manipulates various parameters 

for the InfiniPath driver. It is installed from the infinipath RPM. Many of the 

parameters are used only when diagnosing problems, and may require special 

system configurations. Using these options may require restarting the driver or 

utility programs to recover from incorrect parameters. 

Most of the functionality is accessed via the /sys filesystem. This shell script 

gathers the same information contained in these files: 

/sys/class/infiniband/qib0/device/boardversion 

/sys/class/infiniband/qib0/ports/1/linkcontrol/status_str 

/sys/class/infiniband/qib0/device/driver/version 

These files are also documented in Table I-4 and Table I-5. 

Other than the -i option, this script must be run with root permissions. See the 

man pages for ipath_control for more details. 

I-12 D000046-005 B



Here is sample usage and output: 

% ipath_control -i 

$Id: QLogic OFED Release 1.5 $ $Date: 2010-03-01-23:28 $ 

0: Version: ChipABI 2.0, InfiniPath_QLE7342, InfiniPath1 6.1, SW 

Compat 2 

0: Serial: RIB0941C00005 LocalBus: PCIe,5000MHz,x8 

0,1: Status: 0xe1 Initted Present IB_link_up IB_configured 

0,1: LID=0x1 GUID=0011:7500:0079:e574 

0,1: HRTBT:Auto LINK:40 Gb/sec (4X QDR) 

0,2: Status: 0x21 Initted Present [IB link not Active] 

0,2: LID=0xffff GUID=0011:7500:0079:e575 

The -i option combined with the -v option is very useful for looking at the 

InfiniBand width/rate and PCIe lanes/rate. For example: 

% ipath_control -iv 

$Id: QLogic OFED Release 1.5 $ $Date: 2010-03-01-23:28 $ 

0: Version: ChipABI 2.0, InfiniPath_QLE7342, InfiniPath1 6.1, SW 

Compat 2 

0: Serial: RIB0941C00005 LocalBus: PCIe,5000MHz,x8 

0,1: Status: 0xe1 Initted Present IB_link_up IB_configured 

0,1: LID=0x1 GUID=0011:7500:0079:e574 

0,1: HRTBT:Auto LINK:40 Gb/sec (4X QDR) 

0,2: Status: 0x21 Initted Present [IB link not Active] 

0,2: LID=0xffff GUID=0011:7500:0079:e575 

0,2: HRTBT:Auto LINK:10 Gb/sec (4X) 

NOTE: 

On the first line, Release version refers to the current software release. 

The second line contains chip architecture version information. 

ipath_mtrr 

Another useful option blinks the LED on the InfiniPath adapter (QLE7240 and 

QLE7280 adapters). This is useful for finding an adapter within a cluster. Run the 

following as a root user: 

# ipath_control -b [On|Off] 

NOTE: 

Use ipath_mtrr if you are not using the default PAT mechanism to enable 

write combining. 

D000046-005 B I-13



MTRR is used by the InfiniPath driver to enable write combining to the QLogic 

on-chip transmit buffers. This option improves write bandwidth to the QLogic chip 

by writing multiple words in a single bus transaction (typically 64 bytes). This 

option applies only to x86_64 systems. It can often be set in the BIOS. 

However, some BIOS’ do not have the MTRR mapping option. It may have a 

different name, depending on the chipset, vendor, BIOS, or other factors. For 

example, it is sometimes referred to as 32 bit memory hole. This setting must be 

enabled. 

If there is no setting for MTRR mapping or 32 bit memory hole, contact your 

system or motherboard vendor and ask how to enable write combining. 

You can check and adjust these BIOS settings using the BIOS Setup utility. For 

specific instructions, follow the hardware documentation that came with your 

system. 

QLogic also provides a script, ipath_mtrr, which sets the MTRR registers, 

enabling maximum performance from the InfiniPath driver. This Python script is 

available as a part of the InfiniPath software download, and is contained in the 

infinipath* RPM. It is installed in /bin. 

To diagnose the machine, run it with no arguments (as a root user): 

# ipath_mtrr 

The test results will list any problems, if they exist, and provide suggestions on 

what to do. 

To fix the MTRR registers, use: 

# ipath_mtrr -w 

Restart the driver after fixing the registers. 

This script needs to be run after each system reboot. It can be set to run 

automatically upon restart by adding this line in /etc/sysconfig/infinipath: 

IPATH_MTRR_ACTIVE=1 

See the ipath_mtrr(8) man page for more information on other options. 

ipath_pkt_test 

This program is installed from the infinipath RPM. Use ipath_pkt_test to 

do one of the following: 

• Test the InfiniBand link and bandwidth between two InfiniPath infiniband 

adapters. 

• Using an InfiniBand loopback connector, test the link and bandwidth within a 

single InfiniPath infiniband adapter. 

I-14 D000046-005 B



ipathstats 

The ipath_pkt_test program runs in either ping-pong mode (send a packet, 

wait for a reply, repeat) or in stream mode (send packets as quickly as possible, 

receive responses as they come back). 

Upon completion, the sending side prints statistics on the packet bandwidth, 

showing both the payload bandwidth and the total bandwidth (including InfiniBand 

and InfiniPath headers). See the man page for more information. 

The ipathstats program is useful for diagnosing InfiniPath problems, 

particularly those that are performance related. It is installed from the 

infinipath RPM. It displays both driver statistics and hardware counters, 

including both performance and "error" (including status) counters. 

Running ipathstats -c 10, for example, displays the number of packets and 

32-bit words of data being transferred on a node in each 10-second interval. This 

output may show differences in traffic patterns on different nodes, or at different 

stages of execution. See the man page for more information. 

lsmod 

When you need to find which InfiniPath and OpenFabrics modules are running, 

type the following command: 

# lsmod | egrep ’ipath_|ib_|rdma_|findex’ 

modprobe 

Use this program to load/unload the drivers. You can check to see if the driver has 

loaded by using this command: 

# modprobe -v ib_qib 

The -v option typically only prints messages if there are problems. 

The configuration file that modprobe uses is /etc/modprobe.conf 

(/etc/modprobe.conf.local on SLES). In this file, various options and 

naming aliases can be set. 

mpirun 

mpirun determines whether the program is being run against a QLogic or 

non-QLogic driver. It is installed from the mpi-frontend RPM. Sample 

commands and results are shown in the following paragraphs. 

QLogic-built: 

$ mpirun -np 2 -m /tmp/id1 -d0x101 mpi_latency 1 0 

asus-01:0.ipath_setaffinity: Set CPU affinity to 1, port 0:2:0 

(1 active chips) 

asus-01:0.ipath_userinit: Driver is QLogic-built 

D000046-005 B I-15



mpi_stress 

Non-QLogic built: 

$ mpirun -np 2 -m /tmp/id1 -d0x101 mpi_latency 1 0 

asus-01:0.ipath_setaffinity: Set CPU affinity to 1, port 0:2:0 

(1 active chips) 

asus-01:0.ipath_userinit: Driver is not QLogic-built 

This is an MPI stress test program designed to load up an MPI interconnect with 

point-to-point messages while optionally checking for data integrity. By default, it 

runs with all-to-all traffic patterns, optionally including oneself and one’s local 

shared memory (shm) peers. It can also be set up with multi-dimensional grid 

traffic patterns; this can be parameterized to run rings, open 2D grids, closed 

2D grids, cubic lattices, hypercubes, and so on. 

Optionally, the message data can be randomized and checked using CRC 

checksums (strong but slow) or XOR checksums (weak but fast). The 

communication kernel is built out of non-blocking point-to-point calls to load up the 

interconnect. The program is not designed to exhaustively test out different MPI 

primitives. Performance metrics are displayed, but should be carefully interpreted 

in terms of the features enabled. 

This is an MPI application and should be run under mpirun or its equivalent. 

The following example runs 16 processes and a specified hosts file using the 

default options (all-to-all connectivity, 64 to 4MB messages in powers of two, one 

iteration, no data integrity checking): 

$ mpirun -np 16 -m hosts mpi_stress 

There are a number of options for mpi_stress; this one may be particularly useful: 

-P 

This option poisons receive buffers at initialization and after each receive; 

pre-initialize with random data so that any parts that are not being correctly 

updated with received data can be observed later. 

See the mpi_stress(1) man page for more information. 

rpm 

To check the contents of an installed RPM, use these commands: 

$ rpm -qa infinipath\* mpi-\* 

$ rpm -q --info infinipath # (etc) 

The option-q queries. The option --qa queries all. To query a package that has 

not yet been installed, use the -qpl option. 

I-16 D000046-005 B


Common Tasks and Commands 

strings 

Use the strings command to determine the content of and extract text from a 

binary file. 

The command strings can also be used. For example, the command: 

$ strings -a /usr/lib/libinfinipath.so.4.0 | grep Date: 

produces this output: 

$Date: 2009-02-26 12:05 Release2.3 InfiniPath $ 

NOTE: 

The strings command is part of binutils (a development RPM), and 

may not be available on all machines. 

Common Tasks and Commands 

Table I-3 lists some common commands that help with administration and 

troubleshooting. Note that mpirun in nonmpi mode can perform a number of 

checks. 

Table I-3. Common Tasks and Commands Summary 

Function 

Check the system state 

Verify hosts via an Ethernet 

ping 

Verify ssh 

Command 

ipath_checkout [options] hostsfile 

ipathbug-helper -m hostsfile \ 

> ipath-info-allhosts 

mpirun -m hostsfile -ppn 1 \ 

-np numhosts -nonmpi ipath_control -i 

Also see the file: 

/sys/class/infiniband/ipath*/device/status_str 

where * is the unit number. This file provides information 

about the link state, possible cable/switch problems, 

and hardware errors. 

ipath_checkout --run=1 hostsfile 


Show uname -a for all hosts mpirun -m hostsfile -ppn 1 \ 

-np numhosts -nonmpi uname -a 

D000046-005 B I-17


Summary and Descriptions of Useful Files 

Table I-3. Common Tasks and Commands Summary (Continued) 

Reboot hosts 

Function 

Command 

As a root user: 


-np numhosts -nonmpi reboot 

Run a command on all hosts mpirun -m hostsfile -ppn 1 \ 

-np numhosts -nonmpi 

Examples: 


-np numhosts -nonmpi hostname 


-np numhosts -nonmpi date 

Copy a file to all hosts 

Summarize the fabric components 

Show the status of host Infini- 

Band ports 

Verify that the hosts see each 

other 

Check MPI performance 

Using bash: 

$ for i in $( cat hostsfile ) 

do 

scp $i: 

done 






-np numhosts -nonmpi ipath_control -i 



Generate all hosts problem 

report information 



Table Notes 

The " \ " indicates commands that are broken across multiple lines. 


Useful files are summarized in Table I-4. Names in blue text are linked to a 

corresponding section that provides further details. 

I-18 D000046-005 B



Table I-4. Useful Files 

File Name 

boardversion 

status_str 

/var/log/messages 

version 

Function 

File that shows the version of the chip architecture. 

File that verifies that the InfiniPath software is loaded and 

functioning 

Logfile where various programs write messages. Tracks 

activity on your system 

File that provides version information of installed software/drivers 

boardversion 

It is useful to keep track of the current version of the chip architecture. You can 

check the version by looking in this file: 

/sys/class/infiniband/qib0/device/boardversion 

Example contents are: 

ChipABI 2.0,InfiniPath_QLE7280,InfiniPath1 5.2,PCI 2,SW Compat 2 

This information is useful for reporting problems to Technical Support. 

status_str 

NOTE: 

This file returns information of where the form factor adapter is installed. The 

PCIe half-height, short form factor is referred to as the QLE7140, QLE7240, 

QLE7280, QLE7340, or QLE7342. 

Check the file status_str to verify that the InfiniPath software is loaded and 

functioning. The file is located here: 

/sys/class/infiniband/qib/device/status_str 

Table I-5 shows the possible contents of the file, with brief explanations of the 

entries. 

Table I-5. status_str File Contents 

Initted 

File Contents 

Description 

The driver has loaded and successfully initialized 

the IBA6110 or IBA7220 ASIC. 

D000046-005 B I-19



Table I-5. status_str File Contents (Continued) 

File Contents 

Present 

IB_link_up 

IB_configured 

NOIBcable 

Fatal_Hardware_Error 

Description 

The IBA6110 or IBA7220 ASIC has been detected 

(but not initialized unless Initted is also present). 

The InfiniBand link has been configured and is in 

the active state; packets can be sent and received. 

The InfiniBand link has been configured. It may or 

may not be up and usable. 

Unable to detect link present. This problem can be 

caused by one of the following problems with the 

QLE7140, QLE7240, or QLE7280 adapters: 

• No cable is plugged into the adapter. 

• The adapter is connected to something other 

than another InfiniBand device, or the connector 

is not fully seated. 

• The switch where the adapter is connected is 

down. 

Check the system log (default is /var/log/messages) 

for more information, then call Technical 

Support. 

This same directory contains other files with information related to status. These 

files are summarized in Table I-6. 

Table I-6. Status—Other Files 

File Name 

lid 

mlid 

guid 

nguid 

serial 

Contents 

InfiniBand LID. The address on the InfiniBand fabric, similar conceptually 

to an IP address for TCP/IP. Local refers to it being unique 

only within a single InfiniBand fabric. 

The Multicast Local ID (MLID), for InfiniBand multicast. Used for 

InfiniPath ether broadcasts, since InfiniBand has no concept of 

broadcast. 

The GUID for the InfiniPath chip, it is equivalent to a MAC address. 

The number of GUIDs that are used. If nguids == 2 and two chips 

are discovered, the first chip is assigned the requested GUID (from 

eeprom, or ipath_sma), and the second chip is assigned GUID+1. 

The serial number of the QLE7140, QLE7240, or QLE7280 adapter. 

I-20 D000046-005 B


Summary of Configuration Files 

Table I-6. Status—Other Files (Continued) 

File Name 

unit 

status 

Contents 

A unique number for each card or chip in a system. 

The numeric version of the status_str file, described in Table I-5. 

version 

You can check the version of the installed InfiniPath software by looking in: 

/sys/class/infiniband/qib0/device/driver/version 

QLogic-built drivers have contents similar to: 

$Id: QLogic OFED Release 1.4.2$ $Date: Fri Feb 27 16:14:31 PST 2009 

$ 

Non-QLogic-built drivers (in this case kernel.org) have contents similar to: 

$Id: QLogic kernel.org driver $ 


Table I-7 contains descriptions of the configuration and configuration template 

files used by the InfiniPath and OpenFabrics software. 

Table I-7. Configuration Files 

Configuration File Name 

/etc/infiniband/qlgc_vnic.cfg 

/etc/modprobe.conf 

Description 

VirtualNIC configuration file. Create this file 

after running ib_qlgc_vnic_query to 

get the information you need. This file was 

named /etc/infiniband/qlogic_vnic.cfg 

or 

/etc/sysconfig/ics_inic.cfg in 

previous releases. See the sample file 

qlgc_vnic.cfg.sample (described 

later) to see how it should be set up. 

Specifies options for modules when added 

or removed by the modprobe command. 

Also used for creating aliases. The PAT 

write-combing option is set here. 

For Red Hat systems. 

D000046-005 B I-21



Table I-7. Configuration Files (Continued) 

Configuration File Name 

/etc/modprobe.conf.local 

/etc/infiniband/openib.conf 

/etc/sysconfig/infinipath 

/etc/sysconfig/network/ifcfg- 

 

Description 

Specifies options for modules when added 

or removed by the modprobe command. 

Also used for creating aliases. The PAT 

write-combing option is set here. 

For SLES systems. 

The primary configuration file for Infini- 

Path, OFED modules, and other modules 

and associated daemons. Automatically 

loads additional modules or changes IPoIB 

transport type. 

Contains settings, including the one that 

sets the ipath_mtrr script to run on 

reboot. 

Network configuration file for network interfaces 

When used for VNIC configuration, 

is in the form eiocX, where X is 

the device number. There will be one 

interface configuration file for each interface 

defined in /etc/infiniband/qlgc_vnic.cfg. 

For SLES systems. 

/etc/sysconfig/network-scripts/ifcfg- 

Network configuration file for network interfaces 

When used for VNIC configuration, 

is in the form eiocX, where X is 

the device number. There will be one 

interface configuration file for each interface 

defined in /etc/infiniband/qlgc_vnic.cfg. 


I-22 D000046-005 B



Table I-7. Configuration Files (Continued) 

Sample and Template Files 

qlgc_vnic.cfg.sample 

/usr/share/doc/initscripts-*/ 

sysconfig.txt 

Description 

Sample VNIC config file. It can be found 

with the OFED documentation, or in the 

qlgc_vnictools subdirectory of the 

QLogic OFED Host Software download. It 

is also installed in /etc/infiniband. 

File that explains many of the entries in the 

configuration files 


D000046-005 B I-23



Notes 

I-24 D000046-005 B

J 

Recommended Reading 

Reference material for further reading is provided in this appendix. 

References for MPI 

The MPI Standard specification documents are located at: 

http://www.mpi-forum.org/docs 

The MPICH implementation of MPI and its documentation are located at: 

http://www-unix.mcs.anl.gov/mpi/mpich/ 

The ROMIO distribution and its documentation are located at: 

http://www.mcs.anl.gov/romio 

Books for Learning MPI Programming 

Gropp, William, Ewing Lusk, and Anthony Skjellum, Using MPI, Second Edition, 

1999, MIT Press, ISBN 0-262-57134-X 

Gropp, William, Ewing Lusk, and Anthony Skjellum, Using MPI-2, Second Edition, 

1999, MIT Press, ISBN 0-262-57133-1 

Pacheco, Parallel Programming with MPI, 1997, Morgan Kaufman Publishers, 

ISBN 1-55860 

Reference and Source for SLURM 

InfiniBand 

OpenFabrics 

The open-source resource manager designed for Linux clusters is located at: 

http://www.llnl.gov/linux/slurm/ 

The InfiniBand specification can be found at the InfiniBand Trade Association site: 

http://www.infinibandta.org/ 

Information about the Open InfiniBand Alliance is located at: 

http://www.openfabrics.org 

D000046-005 B J-1

J–Recommended Reading 

Clusters 

Clusters 

Networking 

Rocks 

Gropp, William, Ewing Lusk, and Thomas Sterling, Beowulf Cluster Computing 

with Linux, Second Edition, 2003, MIT Press, ISBN 0-262-69292-9 

The Internet Frequently Asked Questions (FAQ) archives contain an extensive 

Request for Command (RFC) section. Numerous documents on networking and 

configuration can be found at: 

http://www.faqs.org/rfcs/index.html 

Extensive documentation on installing Rocks and custom Rolls can be found at: 

http://www.rocksclusters.org/ 

Other Software Packages 

Environment Modules is a popular package to maintain multiple concurrent 

versions of software packages and is available from: 

http://modules.sourceforge.net/ 

J-2 D000046-005 B

Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 www.qlogic.com 

International Offices UK | Ireland | Germany | India | Japan | China | Hong Kong | Singapore | Taiwan 

© 2010 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo are 

registered trademarks of QLogic Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. 

Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic Corporation assumes no responsibility for any errors in 

this brochure. QLogic Corporation reserves the right, without notice, to make changes in product design or specifications.

QLogic OFED+ Host Software User Guide, Rev. B

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?