Solaris 11: Configure IP Over Infiniband Devices

In this article we will be going over the configuration of an infiniband interface with the IPoIB protocol on Solaris 11, specifically Solaris 11.2 (previous versions of Solaris 11 should work the same, however, there have been changes in the ipadm and dladm commands).

Identify Infiniband Datalinks

First we need to identify the underlying interfaces of the infiniband interfaces.  In my case net2 and net3.

# dladm show-phys<br />

LINK MEDIA STATE SPEED DUPLEX DEVICE<br />

net1 Ethernet unknown 0 unknown ixgbe1<br />

net0 Ethernet up 1000 full ixgbe0<br />

net2 Infiniband up 32000 unknown ibp0<br />

net3 Infiniband up 32000 unknown ibp1<br />

net5 Ethernet up 1000 full vsw0

Another way to confirm the infiniband interfaces is to use the show-ib command.

# dladm show-ib<br />

LINK HCAGUID PORTGUID PORT STATE GWNAME GWPORT PKEYS<br />

net2 10E0000128EBC8 10E0000128EBC9 1 up kel01-gw01 0a-eth-1 7FFF,FFFF<br />

kel01-gw02 0a-eth-1<br />

net3 10E0000128EBC8 10E0000128EBCA 2 up kel01-gw01 0a-eth-1 7FFF,FFFF<br />

kel01-gw02 0a-eth-1

Rename Infiniband Datalinks

I like to rename the datalinks to ib0 and ib1, it makes it easier to keep everything nice and tidy.

# dladm rename-link net2 ib0<br />

# dladm rename-link net3 ib1

Now to show the updated datalinks.

# dladm show-phys<br />

LINK MEDIA STATE SPEED DUPLEX DEVICE<br />

net1 Ethernet unknown 0 unknown ixgbe1<br />

net0 Ethernet up 1000 full ixgbe0<br />

ib0 Infiniband up 32000 unknown ibp0<br />

ib1 Infiniband up 32000 unknown ibp1<br />

net5 Ethernet up 1000 full vsw0

Now in subsequent actions we will use ib0 and ib1 as our datalinks.

Create Infiniband Partition

First, let’s talk about partitions, partitions are most closely related to VLANs.  However the purpose of partitions is to provide isolated segments, so there is no concept of a “router” on IB.  So your use case might be for isolating storage or database services or even isolating customers from one another (which you definitely should do if you have a multitenant environment where customers have access to the operating system.  So what we want to do is identify the partition to be created, if you do not use IB partitioning, then you will need to use the “default” partition of ffff.

# dladm create-part -l ib0 -P 0xffff pffff.ib0

If you do use partitioning, then you will need to define the partition that you wish to use, for this example 7fff.  Which partition to use is determined by the dladm show-ib output, it lists the PKEY that are available, these are partitions.

# dladm create-part -l ib0 -P 0x7fff p7fff.ib0

Now lets review the partitions.

# dladm show-part<br />

LINK PKEY OVER STATE FLAGS<br />

pffff.ib0 FFFF ib0 unknown —-<br />

p7fff.ib0 7FFF ib0 unknown —-

We now have our two partitions defined.

Create IP Interfaces

Now that we have the Infiniband pieces configured, we simply create the IP interfaces, so that we can subsequently assign an IP address, the IP interfaces are named as follows (ibpartition.interfacename).  Below is for the “default” partition.

# ipadm create-ip pffff.ib0

And for our named partition for 7fff we create an interface as well.

# ipadm create-ip p7fff.ib0

Now we have our interfaces configured correctly.

Create IP Address

Now the easy part this is exactly the same as we would do with a standard ethernet interface.  Assign a static IP address for the default partition.

# ipadm create-addr -T static -a 10.1.10.11/24 pffff.ib0/v4

Also for our named partition.

# ipadm create-addr -T static -a 10.2.10.11/24 p7fff.ib0/v4

Now a few ping tests and we are in business.  Remember you will not be able to ping from one partition to another, so you will need to identify a few endpoints on your existing Infiniband networks to test your configuration.