Add lpf IPoIB support

[TimoRoth]

I'd like to replace good old dhcpd with something more modern, but we have an InfiniBand network, and thus need IPoIB support for DHCP4.

This is my current work in progress for adding support to the linux lpf side of things.
It's not quite complete, since I'd like to gather some feedback on the interest and way of implementation first, so here is what I got so far.

I couldn't find any documentation on the IPoIB raw header format, but by looking at it, it seems to be just the 20 byte long hwaddr followed by the same 2 byte ethernet protocol type, and two bytes which are always zero.
Based on that, I ended up with this patch.

There is still one issue, which is that in case of a broadcast, InfiniBand does not neccesarily have a generic broadcast address. Instead it has to be queried from the interface, which will need to be added to the generic interface code still.

[tomaszmrugalski]

Very interesting. There's rfc4390 that documents the IB flavor of DHCP.

Couple potential problems ahead:

• ISC doesn't have any IB hardware (or experience with it), so ongoing testing would be an issue
• my understanding is that all devices report MAC address of 0 and client-id is used to contain the actual identifier. So Kea's assumption that the MAC address is unique would not hold. I suspect many things could break in subtle and not so subtle ways.

None of the above are show-stoppers, just something to be cautious about.

[TimoRoth]

I think they have something like a MAC, it's just obscenely (20 bytes) long.
A device looks like this:

6: ibp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc mq state UP mode DEFAULT group default qlen 256
    link/infiniband 00:00:01:9f:fe:80:00:00:00:00:00:00:e0:07:1b:ff:ff:70:d3:e0 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

With old ISC DHCPd, the entry for that node would look like this:

host nodeExample {
        hardware ethernet E0:07:1B:70:D3:E0;
        option dhcp-client-identifier=20:E0:07:1B:FF:FF:70:D3:E0;
        fixed-address 10.10.10.10;
        option host-name "nodeExample";
}

I'm not quite sure who came up with the translation back to the 6 byte MAC in dhcpd (it's a Mellanox Patch-Set to make it support Infiniband), but all the values are directly derived from one another.
The leading 0x20 in the client-id seems to be just the HTYPE_INFINIBAND.

I have access to a bunch of hardware to test with, and specially their PXE boot environment is rather particular about doing things correctly.

I just read the RFC, and it states that the server does not know the clients hardware address.
While that's true at the pure DHCP protocol level, at least on Linux via raw packet sockets, the Kernel happily tells one the source hardware address. So a unicast-response is in fact possible.
But I'd imagine Kea would nontheless respect the BROADCAST-Flag and broadcast back the reply, which works fine enough.

[sempervictus]

Thank you, neat.
Might make sense to qualify that the kernel's behavior on this is consistent w/ upstream implementation & ofed along with it being a defined behavior that won't be patched-away when its discovered to be related to a security or performance concern.

[TimoRoth]

I haven't had a chance to give this a test yet, but in theory the code should be complete.

The Kernel behaviour in regards to the response address doesn't really matter at all, since the RFC states that the clients must always request a broadcast response.

[TimoRoth]

I have now finally had the chance to test this, and after fixing two small kinks (confused IFLA_BROADCAST and IFA_BROADCAST and Pkt4 refusing the long CHADDR), it's working just fine.

Here's a log of a node booting up via HTTP boot. First instance is the UEFI HTTP boot tool reqeusting an IP, and the second is dhclient in the initrd:

May 22 00:20:38 login01 kea-dhcp4[421812]: INFO  DHCP4_QUERY_LABEL received query: [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448
May 22 00:20:38 login01 kea-dhcp4[421812]: INFO  EVAL_RESULT [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: Expression httpclients evaluated to true
May 22 00:20:38 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_RECEIVED [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface ibp1s0
May 22 00:20:38 login01 kea-dhcp4[421812]: INFO  DHCP4_LEASE_OFFER [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: lease 10.110.10.150 will be offered
May 22 00:20:38 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_SEND [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: trying to send packet DHCPOFFER (type 2) from 10.110.10.250:67 to 255.255.255.255:68 on interface ibp1s0
May 22 00:20:41 login01 kea-dhcp4[421812]: INFO  DHCP4_QUERY_LABEL received query: [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448
May 22 00:20:41 login01 kea-dhcp4[421812]: INFO  EVAL_RESULT [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: Expression httpclients evaluated to true
May 22 00:20:41 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_RECEIVED [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: DHCPREQUEST (type 3) received from 0.0.0.0 to 255.255.255.255 on interface ibp1s0
May 22 00:20:41 login01 kea-dhcp4[421812]: INFO  DHCP4_LEASE_ALLOC [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: lease 10.110.10.150 has been allocated for 3600 seconds
May 22 00:20:41 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_SEND [hwtype=1 e0:07:1b:70:14:d0], cid=[no info], tid=0x8dfd9448: trying to send packet DHCPACK (type 5) from 10.110.10.250:67 to 255.255.255.255:68 on interface ibp1s0


May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_QUERY_LABEL received query: [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_RECEIVED [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface ibp1s0
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_LEASE_OFFER [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: lease 10.110.10.2 will be offered
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_SEND [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: trying to send packet DHCPOFFER (type 2) from 10.110.10.250:67 to 255.255.255.255:68 on interface ibp1s0
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_QUERY_LABEL received query: [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_RECEIVED [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: DHCPREQUEST (type 3) received from 0.0.0.0 to 255.255.255.255 on interface ibp1s0
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_LEASE_ALLOC [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: lease 10.110.10.2 has been allocated for 3600 seconds
May 22 00:22:10 login01 kea-dhcp4[421812]: INFO  DHCP4_PACKET_SEND [hwtype=32 e0:07:1b:ff:ff:70:14:d0], cid=[20:e0:07:1b:ff:ff:70:14:d0], tid=0xc4181770: trying to send packet DHCPACK (type 5) from 10.110.10.250:67 to 255.255.255.255:68 on interface ibp1s0

Writing reservations for this is super annoying, since the two clients can't agree on neither hwaddr nor cid.
But luckily the pre-boot environment does not need to get the correct IP + hostname assigned.

[TimoRoth]

I've been using this on our network/cluster since a while now, and it's been running without issues.
So if there is anything further I should do to move forward with the PR, please let me know.

[gamucf]

Thanks for authoring this. I've really been waiting for this too. I run an HPC center and we use Infiniband throughout all the nodes. Hope it see it in a full release soon!

[zeronewb]

Would be really nice to have, since rhel10, switched to Kea.

[vrisk]

Hey, I just wanted to say, we see this contribution and we know it has been outstanding for a while. Right now the team is short-handed but someone will take a look as soon as we can.

[TimoRoth]

This is also at https://gitlab.isc.org/isc-projects/kea/-/merge_requests/2385 and was adopted into https://gitlab.isc.org/isc-projects/kea/-/merge_requests/2445 from there, but seems to have stalled again.

[vrisk]

I asked about the abandoned merge request, and the problem is, we can't merge it because we don't have any Infiniband hw to test it with. Is there any software-based emulator that is protocol compliant, that you know of??

[BtbN]

I looked into that before, and unfortunately, there is no way to just attach a virtual IB NIC to a VM.
You will at the very least need two true IB NICs.
I'd recommend anything that runs with the mlx5 kernel driver, since the older one has a few odd quirks.
The ConntectX-4 series of IB cards are going EOL right about now, so you can pick them up for cheap on eBay.
All you need is two cards and a cable. No switch. And then plug two machines together, or plug both cards into one server, and PCIe passthrough them into two VMs.
That's roughly a cost of 200-300$, looking at eBay.
All you need to look out for is that the ConnectX-4 cards you buy are the VPI variant, not the EN one.
The latter is just normal Ethernet with RDMA hardware offloads.

[gamucf]

Would be really nice to have, since rhel10, switched to Kea.

+1 from me. We also run IB and can't upgrade to kea without support. Not sure what the future will be for us since rhel10 only has kea. Scary.

[TimoRoth]

I'm using this PRs branch since almost 2 years on our system now, and it works flawlessly.

[gamucf]

I may have to do the same. The issue I have is old ISC dhcp crashes on my Rocky 9 installs, so i have moved it to a sole remaining Rocky 8 install. I'd really like to modernize but many HPC clusters use Infiniband. Wish they'd keep ISC updated until Kea can have all features.

[tomaszmrugalski]

Folks, apologies it took so long to get to this. There is a chance we'll merge it this summer. I've acquired IB hardware to test it and found a person who is able to dedicate time for it. I've added a note on gitlab.

If anyone experienced with Infiniband could tell us if the hardware we got would be sufficient to have a minimal IB network, that would be much appreciated:

• 2 MELLANOX MHQH29B-XTR INFINIBAND Cards
• Mellanox Voltaire VLT-30111 4036 Infiniband 36x40G switch
• cable 1: AOC SFP+ 10G 850nm 2m OM2 50/125 UL OFNP
• cable 2: AOC QSFP to QSFP 40G 850nm 2m DELTA QAOC-10G4F1A02 40 Gb/s

[BtbN]

That should definitely be enough. You shouldn't even need the Switch, just plug the two machines together directly.
I think you can also do this all with a singular IB adapter, one one host, by making use of SR-IOV virtual functions, splitting it up into multiple virtual adapters, and attaching them to VMs on the host.
Though I never tested that myself.

The problem with the card you picked is primarily that it's really old, ConnectX-2. The driver in the kernel for that is pretty "crusty".
It should still work for your purposes though.

[tomaszmrugalski]

Yeah, we don't exactly have a budget for something modern and the MHQH29B-XTR cards were cheap, available and had Infiniband written on the box 😉 We already have the hardware, so we'll experiment and see what works.

[rahulc07]

Yeah, we don't exactly have a budget for something modern and the MHQH29B-XTR cards were cheap, available and had Infiniband written on the box 😉 We already have the hardware, so we'll experiment and see what works.

This is probably enough just be careful with drivers. You'll probably want to run Debian or something similar cause mlx4 was removed in a lot of other distros (mainly rocky and rhel)

I would avoid sriov in this setup, the open source subnet manager and early switch firmware doesn't support it so you would need to run some ancient versions of mlnx ofed. Honestly I wouldn't recommend using the switch sm either but it should be fine for a simple test.

If it's any help I can test anything/help with any fabric issues as I've worked extensively with infiniband, and just for the record this PR works amazingly in my setup (8 nodes connect x5 + sb7890) where I'm even doing some fairly fancy stuff with Infiniband Partitions + Proxy ARP + pxe + dhcp 121. I'm sharing the same server with Ethernet devices (with and without vlan devices). The only oddity that I've run into (which makes sense) is that when doing host reservations you have to put 20: before the client-id reported by the system. This is visible in leases search in stork.

Build instructions can extracted from the Dockerfile here
https://github.com/rahulc07/stork-and-kea/tree/main

If you need remote access to a modern dev cluster I can get you that if you need. I have a 3 node cluster with semi modern Connect X4s I can wipe.