Adventures in fibre channel over ethernet. This is a draft so can be said to have errors for now.
Here we assume target device has a MAC Address of ac:de:48:23:45:67 on int0 and the initiator of 00:00:5e:00:53:00 on eth0.
AS a first step, have unlocked the max MTU on the involved interfaces
It does seem to be legitimate to run FCoE on a brctl bridge as it is just another set of ethertypes
on target, install the targetcli
may not be needed for vn2vn
on target, we will run targetcli and export a LVM2 logical volume called fcoetest
creates entry in /sys/class/fc_host/
now can do:
We now can set up access to the LUN effectively to the other mac address
To redo this, we can /tcm_fc delete 20:00:ac:de:48:23:45:67
Complete a step not done by fcoe-utils tools dpkg ☺
If all is satisfactory, use /saveconfig to keep the settings
The default wanted a so-called fcf device, which we will call a special kind of ethernet forwarder. Not got one of those.
We want vn2vn mode which means we can generally do FCoE between 2 ordinary computers with ethernet interfaces, the only need here is zero frame loss.
Now modprobe fcoe on initiator and targets and tell them where to do “create_vn2vn” or stop doing “destroy” FCoE, the same step is done on both ends, though the target also had the access control rules setup as above with targetcli.
Might also run into, problem: Unknown value type 'fc_wwn'
Temporarily edited config.py and utils.py to suppress warning and treat as valid.
Set permission on the block devices to access from the userspace, and then read and write octets with hexedit. The long term intention is to put a distributed cluster filesystem on it, the block should be proven stable first though.
But its slow, was it because fcoeadm -l says "MaxFrameSize: 2112", first thought was to make this bigger but it turns out this is by FC design. It is time to set up iSCSI to compare performance
Again, in targetcli, we setup a comparable iSCSI setup with the same LUN as for FCoE
For iSCSI the lun acces is set target to wwn in/etc/iscsi/initiatorname.iscsi of initiator
At this point it was expected that FCoE would outpace iSCSI but it went very wrong. iSCSI was in lead by far, and the cause must be thought to be dropped frames, somewhere… Time to find out why.
Here is a tcpdump string for wireshark ether proto 0x8906 or tcp port 3260 we can watch iSCSI and FCoE together
Using dropwatch there were signs of trouble already:
A packet injector is used to create a flood of frames as fast as we can.
There are published adjustments to reduce the possibility that core drops frames:
What we still drop frames?!
It turns out that frames are dropped in rtl8169 driver, though this was not declared in the ifconfig output as actually the NIC drops them as “rx_missed” due to the driver not processing frames efficiently enough.
rx_missed ?!
Make the interrupt handler do more work maybe, 64 is not big enough?
It was said that realteks can hold up to 1024 frames before they drop so lets empty up to that many per RX interrupt
It turns out to really improve rtl8169.ko we must hack source, users can recompile just the modules to be patched.
The secret is that _rx_interrupt does rather lot of work via memcpy of whole frames and we remove that, instead rotating the RX ring pointers as frames are received.
If we remove that then it should be faster, we can preallocate frames to mtu during driver initialisation, and alter pointers in the RX ring buffer as frames are moved from the ring into linux networking via skb_put etc.
One side effect is we do use more precious kernel memory, and use too much and system crash, so we compromise on enough to go drop-free.
With adjusted rtl8169 driver, FC perfomance approaches that of iSCSI and other applications like NFS seem to benefit from less droppiness.