Introduction

The VAST NFS package provides a modified version of the Linux NFS client and server kernel code stacks designed to be built for a large range of kernel versions.

It contains backported upstream NFS stack code from Linux v5.15.x LTS kernel branch with. This allows older kernels to receive the full functionality of newer NFS stack code. Only kernels from 4.15.x and above receive the backports and the fully added functionality. The package applies also to older kernels but with less functionality (legacy VAST NFS).

Features

  • NFS stack improvements and fixes
  • Multipath support for NFSv3
  • Multipath support for NFSv4.1 (not for legacy kernels)
  • Nvidia GDS integration

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Package download

For some distribution kernels it is possible to obtain pre-built binary packages from the internet:

curl -sSf https://vast-nfs.s3.amazonaws.com/download.sh | bash -s -- --version <version>

The script tries to find a pre-built package for the currently running system. If a package is obtained, then it can be installed and used right away, see Installation.

If no pre-built package exists, it is possible to obtain the source package and built instead, via:

curl -sSf https://vast-nfs.s3.amazonaws.com/download.sh | bash -s -- --version <version> --source

See Source build.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Source build

There are various ways possible to build this package from source.

  • Packaged build output (RPM or Deb package)
  • Build without packaging, for the purpose loading directly from a development tree

This section provides information about how to accomplish it.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Supported kernel versions

This package is supported on the following Linux distribution kernels:

  • Red Hat Enterprise Linux 8.1 to 8.6 and derivatives (CentOS, Rocky, Alma)
  • SUSE Enterprise Linux 15
  • Ubuntu 20 LTS
  • Ubuntu 18 LTS

Generic upstream kernel support ranges are the following:

  • Linux v4.15 to v5.15.

NOTE: Some older distribution kernels (such RH 8.1) may require a recent Mellanox OFED being installed, as it provides a more recent IB layer along with newer kernel APIs.

Legacy VAST NFS 3.x

Additionally, there is a support for a lesser extent for the following distributions:

  • Red Hat Enterprise Linux 7.x and derivatives (CentOS)
  • SUSE Enterprise Linux 12

Attempting to build for an unsupported kernel may produce compilation errors.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Mellanox OFED support

All versions of Mellanox OFED that successfully install on a Linux system should be supported by this package. Newer OFED versions are preferred.

If the Mellanox OFED is installed on a system, this package can also be built in such a way that NFS RDMA is functional and working using the Mellanox RDMA stack instead of the inbox kernel RDMA stack, which the installation of the Mellanox OFED supersedes.

The state where the Mellanox OFED is installed is detected automatically by the build scripts, and the resultant package will only work with that particular Mellanox OFED version.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Building an installable package

The following command will build the source package. It detects the currently running kernel, distribution, and optional Mellanox OFED installation.

./build.sh bin

Following a successful build, the resultant package should reside in a dist/ sub-directory, for example:

$ ls -1 dist/vastnfs* | grep -v debug

vastnfs-4.0-kernel_4.18.0_348.20.1.el8_5.x86_64.rpm

For older kernel, the build relies on the legacy VAST NFS 3.x sources that are included in the source package.

See Installation for installation instructions.

Arguments

The build.sh bin command can receive the following optional arguments:

  • --without-rpcrdma: Don't build RDMA support
  • --dkms: For Debian-based system, generate a DKMS-based package even if no DKMS-based Mellanox OFED is installed.
  • --no-ofed: Don't detect Mellanox OFED installation, and instead build a driver targeting the inbox kernel instead.

Regarding DKMS

DKMS is a source-based package for which the binary driver is built on-demand. This is useful for minor kernel updates.

If Mellanox OFED installation is detected, and if that installation is DKMS-based, then the resultant package will also be DKMS-based by default. This is currently supported only on Debian-based system. Otherwise, the built package will install binaries that depend on the current kernel version.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Development build

The following command should build the driver for the current kernel, assuming that a compiler and the kernel headers are installed for the local machine. Further development package of the relevant Linux distro may be needed.

make

When this command is issued, no package is being generated and instead the kernel modules reside within the built source tree.

To load the newly built development version, the following command can be issued (sudo is being used by the script):

./scripts/control.sh dev-reload

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Installation

This section deals with installation of the binary package. The goal is for the installed NFS stack to supersede the inbox kernel NFS stack.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

RHEL 8.x

Install:

sudo yum install ./dist/vastnfs-vast*.x86_64.rpm
sudo dracut -f

Following install, rebooting is required for the newly installed drivers to load.

Verify that the package is indeed installed (versions should match):

rpm -qa | grep vastnfs

To further validate installation, verify that rpcrdma in the right path exists and get loaded from it. All the added kernel modules go under updates/bundle:

rpm -qif /lib/modules/`uname -r`/extra/vastnfs/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

$ modinfo rpcrdma | grep filename:
filename:       /lib/modules/..../extra/vastnfs/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Ubuntu

Install:

sudo apt-get install ./dist/vastnfs-modules_*-generic_amd64.deb
sudo update-initramfs -u -k `uname -r`

Verify that the package is indeed installed (versions should match):

dpkg -l | grep vastnfs-modules

To further validate installation, verify that rpcrdma in the right path exists and get loaded from it. All the added kernel modules go under updates/bundle:

dpkg -S /lib/modules/`uname -r`/updates/bundle/net/sunrpc/xprtrdma/rpcrdma.ko

Verification

Reboot, to ensure that this version is loaded correctly.

Once rebooted, and to verify fully once it is loaded, for each kernel module, loaded version should match installed version. For example:

$ cat /sys/module/sunrpc/srcversion
4CC8389C7889F82F5A59269

$ modinfo sunrpc | grep srcversion
srcversion:     4CC8389C7889F82F5A59269

(This is only an example - srcversion is different for each build and kernel module).

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Configuration

This section deals with configurations for the provided NFS stack in this package.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Multipath

To make an efficient load balanced used of single NFS mount, multipath should be enabled.

Under the multipath scheme used in this package, all participating local and remote ports have distinct network IP addresses. This section deals with the configuration that enable this mode.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

RHEL 8.x/7.x

This package is supported on RHEL, and the RHEL-based distributions such as CentOS Linux, Rocky Linux, and Alma Linux.

NOTE: On RHEL 7.x-based kernels the package only provides multipath on NFSv3 mounts.

Here we demonstrate multipath with two local interfaces on the same subnet.

First we need to install NetworkManager-config-routing-rules.

yum install NetworkManager-config-routing-rules

We configure new source routing tables:

echo '101 101' >> /etc/iproute2/rt_tables
echo '102 102' >> /etc/iproute2/rt_tables

For this example we assume two interfaces are configured via ifcfg- scripts:

$ grep IPADDR /etc/sysconfig/network-scripts/ifcfg-ib0
IPADDR=192.168.40.1
$ grep IPADDR /etc/sysconfig/network-scripts/ifcfg-ib1
IPADDR=192.168.40.2

For each interface we need to add a route- and rule- files:

$ cat /etc/sysconfig/network-scripts/route-ib0
192.168.40.0/24 via 192.168.40.1 table 101
$ cat /etc/sysconfig/network-scripts/rule-ib0
from 192.168.40.1/32 table 101

NOTE: This is only an example. The IP addresses you pick depend on your network configuration.

After reloading with nmcli connection reload, the ip command with regard to routing should look like this (notice the two added lookup lines):

$ ip rule
0:      from all lookup local
32764:  from 192.168.40.1 lookup 101
32765:  from 192.168.40.2 lookup 102
32766:  from all lookup main
32767:  from all lookup default

$ ip route show table 101
192.168.40.0/24 via 192.168.40.1 dev ib0

$ ip route show table 102
192.168.40.0/24 via 192.168.40.2 dev ib1

Verify that the IPs and routing tables appear correctly. Below are examples which will vary based on environment:

IP addresses:

$ ip a s | grep 192.168.40

    inet 192.168.40.1/24 brd 192.168.40.255 scope global ib0
    inet 192.168.40.2/24 brd 192.168.40.255 scope global ib1

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Ubuntu

Ubuntu installations can use netplan to specify the network configuration. For each port, we need to have one IP address in the subnet. For each IP address, source routing needs to be enabled.

For example, in the updated 01-netcfg.yml for local port configuration (ib0, ib1, ib2, ib3), append routes and routing-policy rules:

    ib0:
        dhcp4: no
        addresses: [172.25.1.101/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.101
           table: 101
        routing-policy:
         - from: 172.25.1.101
           table: 101
    ib1:
        dhcp4: no
        addresses: [172.25.1.102/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.102
           table: 102
        routing-policy:
         - from: 172.25.1.102
           table: 102
    ib2:
        dhcp4: no
        addresses: [172.25.1.103/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.103
           table: 103
        routing-policy:
         - from: 172.25.1.103
           table: 103
    ib3:
        dhcp4: no
        addresses: [172.25.1.104/24]
        routes:
         - to: 172.25.1.0/24
           via: 172.25.1.104
           table: 104
        routing-policy:
         - from: 172.25.1.104
           table: 104

NOTE: This is only an example. The IP addresses you pick depend on your network configuration.

The apply the new configuration, run:

sudo netplan apply

Verify that the IPs and routing tables appear correctly. Below are examples which will vary based on environment:

IP addresses:

$ ip a s | grep 172.25.1

    inet 172.25.1.101/20 brd 172.25.1.255 scope global ib0
    inet 172.25.1.102/20 brd 172.25.1.255 scope global ib1
    inet 172.25.1.103/20 brd 172.25.1.255 scope global ib2
    inet 172.25.1.104/20 brd 172.25.1.255 scope global ib3

Routing tables:

$ for i in 101 103 102 104; do ip route show table $i; done;

172.25.1.0/20 via 172.25.1.101 dev ib0 proto static
172.25.1.0/20 via 172.25.1.102 dev ib1 proto static
172.25.1.0/20 via 172.25.1.103 dev ib2 proto static
172.25.1.0/20 via 172.25.1.104 dev ib3 proto static

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Usage

There are extra features that this stack enables and these are described in this section.

If no extra features are used, the usage and behavior of the provided NFS stack matches the upstream or distribution NFS stack versions.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Mount parameters

This package adds on the standard mount parameters that the original Linux NFS stack originally supports.

Based on the IP source routing configuration, extra parameters are needed to utilize the configuration in the mount path.

The code changes in this repository add the following parameters:

  • remoteports=<FIRST-LAST>~<SINGLE-IP-ADDR>... - A list of IPv4 addresses for the remote ports to bind.

IP addresses can be given as an inclusive range, with - as a delimiter, e.g. FIRST-LAST. Multiple ranges or IP addresses can be separated by ~.

  • localports=<FIRST-LAST>~<SINGLE-IP-ADDR>... - A list of IPv4 addresses for the local ports to bind.

The localports parameter is not necessary for multipath to work. However if the local machine has more than one IP address configured for the local subnet, it may be useful to prevent a single local port from trying to surpass its maximum line rate while the other local ports are underused.

Also, for GDS-based setups, it is necessary for making the transport layer to perform GPU-based routing for GPU overload to be utilized.

  • nconnect=<NUM> - Upstream NFS stacks introduced this parameter to allow multiple connections per mount. Legacy VAST NFS 3.x introduces this for Linux kernels for which it was not backported. The recommended value for this parameter is 4 for RDMA mounts and 8 for TCP mounts.

Usage of parameter for support of multipath

Multipath can be used with TCP (proto=tcp) at the cost of higher CPU utilization compared to RDMA. Here is an example of such multipath mount:

mount -o vers=3,nconnect=8,remoteports=172.25.1.1-172.25.1.32 172.25.1.1:/

This mount command will result in 8 TCP connections, going to a psuedo-random sub-range of 8 addresses under the provided 32 address range.

Here is a more advanced example of an RDMA mount with 4 connections, 8 remote ports and 4 local ports:

mount -o proto=rdma,port=20049,vers=3,nconnect=4,localports=172.25.1.101-172.25.1.104,remoteports=172.25.1.1-172.25.1.8 172.25.1.1:/

Advanced parameters

NFS optimizations

forcerdirplus - Instructs the NFS client to always send an NFSv3 READDIRPLUS requests for servers that support it. On some remote systems this can improve directory listing performance significantly.

Additional transport parameters

These parameters can be used for experimentation and fine-tuning.

  • noidlexprt - Do not disconnect idle connections.

  • remoteports_offset - Controls the offset into picking out of remoteports for transports if the number of actual transports via nconnect is smaller than the amount of IPs given here. If not given a pseudo-random number is picked based on source IP address.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

vastnfs-ctl

This helper script is installed with the package, and can assist in various situations.

The usage of this script is optional in most cases.

Status

Observe the status of the loaded NFS services and kernel modules

vastnfs-ctl status

Example output:

version: v4.0-pre4-11-ge520dd8eb9cb
kernel modules: sunrpc rpcrdma compat_nfs_ssc lockd nfs_acl auth_rpcgss nfsd rpcsec_gss_krb5 nfs nfsv3 nfsv4
services: rpcbind.socket rpcbind
rpc_pipefs: /var/lib/nfs/rpc_pipefs

Reload

The following command will try to reload the NFS stack. It will first try to unmount all NFS file systems, and then reload all the necessary kernel modules.

vastnfs-ctl reload

Example output:

vastnfs-ctl: stopping service rpcbind.socket
vastnfs-ctl: umounting fs /var/lib/nfs/rpc_pipefs
vastnfs-ctl: unloading kmod nfsv4
vastnfs-ctl: unloading kmod nfsv3
vastnfs-ctl: unloading kmod nfs
vastnfs-ctl: unloading kmod rpcsec_gss_krb5
vastnfs-ctl: unloading kmod nfsd
vastnfs-ctl: unloading kmod auth_rpcgss
vastnfs-ctl: unloading kmod nfs_acl
vastnfs-ctl: unloading kmod lockd
vastnfs-ctl: unloading kmod compat_nfs_ssc
vastnfs-ctl: unloading kmod rpcrdma
vastnfs-ctl: unloading kmod sunrpc
vastnfs-ctl: loading kmod sunrpc
vastnfs-ctl: loading kmod rpcsec_gss_krb5
vastnfs-ctl: loading kmod rpcrdma
vastnfs-ctl: loading kmod nfsv4
vastnfs-ctl: loading kmod nfsv3
vastnfs-ctl: loading kmod nfsd
vastnfs-ctl: loading kmod nfs_acl
vastnfs-ctl: loading kmod nfs
vastnfs-ctl: loading kmod lockd
vastnfs-ctl: loading kmod compat_nfs_ssc
vastnfs-ctl: loading kmod auth_rpcgss
vastnfs-ctl: mounting fs /var/lib/nfs/rpc_pipefs
vastnfs-ctl: starting service rpcbind.socket

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Monitoring and diagnosis

This section contains resources that may help in diagnosing NFS mount issues.

There are specific VAST NFS extensions such as multipath that we should consider when debugging. Also, there can be issue outside the scope of differences between this implementation and the upstream or distribution-based implementation.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

Mount and transport state

Normally, the mountstats command shows NFS mount status. We have extended the interface it uses, /proc/self/mountstats, with our own extra state reporting via /sys. The extended information contains the IP addresses related to the transport, and a string that indicates its state flags.

The extended form of it is obtainable via /sys/kernel/debug. For the association to an NFS mount, we need to obtain the related sunrpc-id of the mount point.

But first, we need to enable sunrpc-id reporting. This can be done using the following command after boot:

echo 1 | sudo tee /sys/module/sunrpc/parameters/expose_id_in_mountstats

Then, identify the sunrpc-id relevant to the mount point by looking into /proc/self/mountstats:

$ cat /proc/self/mountstats | grep -E 'fstype nfs|sunrpc-id'
device 192.168.40.7:/opt/export mounted on /mnt/export with fstype nfs statvers=1.1
        sunrpc-id:      4

Now we are ready to fetch the full mountstats via the following command:

    sudo cat /sys/kernel/debug/sunrpc/rpc_clnt/4/stats

The added information contains multipath IP address information per xprt (transport) and xprt state in string format. The numbers in the first xprt: line are counters and other numbers related that transports.

For example:

        xprt:   rdma 0 0 1 0 24 3 3 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.101 -> 172.25.1.1, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 24 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.102 -> 172.25.1.2, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 23 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.103 -> 172.25.1.3, state: CONNECTED BOUND
        xprt:   rdma 0 0 1 0 22 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11 0 0 0
                172.25.1.104 -> 172.25.1.4, state: CONNECTED BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.101 -> 172.25.1.5, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.102 -> 172.25.1.6, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.103 -> 172.25.1.7, state: BOUND
        xprt:   rdma 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
                172.25.1.104 -> 172.25.1.8, state: BOUND

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

External resources

Official upstream guide for admins: Linux v5.15 NFS admin guide.

On the linux-nfs.org site: General troubleshooting recommendations - Linux NFS

Linux distribution pages

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev

3.9.15

  • build.sh fixes: add --without-rpcrdma, --dkms, avoid a benign warning

3.9.14

  • Build fixes for SLES15-SP2.
  • Add a forcerdirplus option, disabled by default
  • For upstream-5.15 branch, fix for newer Debian kernels and allow building without rpcrdma via CONFIG_SUNRPC_XPRT_RDMA=n.

3.9.13

  • Add gitversion metadata in /sys/module/sunrpc.

3.9.12

  • Revert Mellanox changes from OFED 5.4 that regressed RDMA memory region scatter-list handling on Ubuntu kernels 5.4.0-68.76 and upstream backports branch.

3.9.11

  • Further OFED 5.5 build fix.
  • Extended support for 5.10.x - 5.15.x kernels based on Linux v5.15.30.

3.9.10

  • Fix for OFED 5.5 build on the rest of the supported kernels.

3.9.9

  • Support up to 2048 VIPs in remoteports=.

3.9.8

  • Fix for OFED 5.5 on RHEL 8.5.
  • Various fixes for upstream-5.14 backports branch, including build fix for DKMS mode.

3.9.7

  • Fixes to multipath request route balancing.

3.9.6

  • RHEL/CentOS 8.5 backporting fix for RDMA mount.

3.9.5

  • RHEL/CentOS 8.5 backport.
  • upstream-5.14: Add compat checks for Ubuntu-5.11.0-41.45.

3.9.4

  • RHEL/CentOS 7.9 kernel sync to fix performance issue (Red Hat bug #1995649).

3.9.3

  • Support all kernels Linux 5.11 to 5.14, using a patched backport of the Linux 5.14.9 tree.

3.9.2

  • RHEL/CentOS 8.4: rebase for NFS client patch.

3.9.1

  • RHEL/CentOS 7.x: fix a bug that prevented proper Kerberos mounts.

3.9

  • Fix module load conflict on several kernels.
  • RDMA fixes from Linux v5.13.
  • Improve build.sh for Rocky Linux.

3.8.9

  • CentOS 8.x rebase for 4.18.0-305.10.2.el8_4.
  • Have kernel branch version detection built inside build.sh so that exact kernel versions need not be specified.

3.8.8

  • Fix package dependencies under Ubuntu/Debian for non-OFED builds.

3.8.7

  • Fix hashing logic for unspecified remoteports_offset - the output to the hash function did not have good nconnect > 2 support.
  • Fix exposing of remoteports_offset value in sysfs - it showed 0 instead of the used value.

3.8.6

  • When remoteports_offset is not specified, use current node name hash in addition to local port IP address detection for randomized offset.

3.8.5

  • On NFSv4, fix GetACL and FS_LOCATIONS requests that got broken in bac-porting.

3.8.4

  • Enable DKMS auto install. Installing the DKMS build of the driver enable the dkms autoinstall feature of attempting to rebuild the drivers for newer kernels.

3.8.3

  • On TCP-based mounts, fix support for localports=.
  • Support additional kernels minors.

3.8.2

  • For Debian-based systems, bump version epoch so that VAST NFS always supersedes mlnx-nfsrdma package from Mellanox.

3.8.1

  • Ensure build for all CentOS 7.7 minors.

3.8

  • Add documentation to the built binary packages.
  • Support CentOS/RHEL kernel 3.10.0-1160.15.2.

3.7.7

  • Fix implementation of handling nconnect smaller than remoteports (implemented in 3.7-rc10), where the actual chosen ports were always the first ones, unless remoteports_offset or localports is specified.
  • For better diagnostics, extend debug interface for easier association of NFS mounts with SunRPC clients under /sys/kernel/debug.
  • Fix handling of EJUKEBOX that was affected by transport selection changes in 3.7-rc7.
  • Improve back-porting to support PowerPC architecture on CentOS 7.x.

3.7.6

  • Rebase CentOS 7.9 kernel support to NFS level of 3.10.0-1160.11.1, and support that version.

3.7.5

  • For CentOS 7.x kernels, fix instability related to TCP mount and requests with sparse pages responses, specifically large NFSv3 ACLs responses.

3.7.4

  • Fix meta-data inside package for DKMS-based builds, which are required on some Mellanox OFED 4.7-1.0.0.1 versions and below on Ubuntu.

3.7.3

  • Fix package filename for non-OFED RPM builds.
  • Support Ubuntu kernel 4.15.0-55.

3.7.2

  • Support added for special LLNL kernel, kernel-3.10.0-1127.18.2.1chaos.ch6.src.rpm, including merging of 16 > groups patch.

3.7.1

  • Fixes for building on PowerPC and other architectures.

3.7.0

  • Fixes to issue found in upstream kernel.
  • Improvement to responsiveness under HA.
  • Improvement to queue handling under TCP soft mounts.
  • Fix for package OFED version to match the one shown in ofed_info -s.
  • Allow non-OFED builds to be properly packaged.

3.7-rc19

  • Trivial fix for nfsd backport in CentOS 8.1.
  • Support for building on native kernels, without Mellanox OFED (package rename pending.)

3.7-rc18

  • Fixes for GDS integration with multipath. This takes into account differences between gdsbeta 0.7 and gdsbeta 0.8, and the behavior regarding idle transports timed with GDS-enabled application execution.

3.7-rc17

  • Fix remaining kernels for backporting bug in nfs4_xdr_dec_access: ubuntu-4.18.0, ubuntu-4.15.0 and sles-4.12.14-122.

3.7-rc16

  • CentOS 7.x kernels: fix a backporting bug in nfs4_xdr_dec_access, happened to NFSv4 clients.

3.7-rc15

  • Fixes to nfsd related issues across the rest of the kernels.
  • Backporting fixes for improved stability on some of the CentOS 7.x kernels.

3.7-rc14

  • In debug monitoring, print the PCI device related to the IB transport.
  • Directory listing improvements across all supported kernels.

3.7-rc13

  • Further fixes for SLES when using nfsd.
  • Improve handling on failures of GDS device priority function. This issue manifested with older versions of GDS on an unsupported OFED version.

3.7-rc12

  • Stability improvement for SLES kernel. Issue fixed occurred during NFS writeback under memory pressure.

3.7-rc11

  • Configure depmod on Ubuntu to prefer loading from our directory, and not from unintended MOFED outputs that happen to also reside in /lib/modules.

3.7-rc10

  • Local port failover disabled by default. localports_failover mount option brings it back if needed.
  • Pseudo-random range pick out of remoteports if nconnect is smaller. Can be controlled using remoteports_offset mount option.

3.7-rc9

  • Better guards against loading version mismatch.

3.7-rc8

  • Support three more kernels.
  • Allow to build RPMs for a kernel that is currently not running.
  • Fix for SUSE's "supported modules" verification, flagging the kernel modules as "externally supported".

3.7-rc7

  • Support for select CentOS 7.x kernels, and latest CentOS 8.2.
  • SLES 12 SP5 support.
  • NFS: Differentiate mounts based on multipath parameters.
  • NFS: Client fixes for directory listing.
  • Fixes related to OFED 5.1.
  • File handle based transport selection (enabled by default only for READ/WRITE).
  • A fix for the mount command in a boot session, not properly waiting for rpcrdma to load.
  • Shared XID between transport. This is a protocol detail to improve caching correctness in fail-over cases.
  • Workaround IB bug related to address resolution.

3.6

  • Fix for missing module rpcsec_gss_krb5.ko in install.

3.5

  • Fix GDS for disconnections during IOs. This is actually an upstream bug that manifests itself more greatly with the GDS integration, which relies on buffers being unmapped.

3.4

  • Workaround for NFS direct IO with GDS. The issue is that a bad interaction with the NVidia GDS base 0.7.1 when the process is killed with the IOs return with ERESTARTSYS, causing an unreleased mutex in the Nvidia propritary driver that causes invocations of other users such as nvidia-smi to get stuck.

3.3

  • Build scripts improvement for supporting multiple kernel versions. Currently supporting two Ubuntu kernels. Source package detects the kernel version currently installed and builds the right version.

3.2

  • Fix for TCP mount. The TCP support does not yet extend to multipath like it does for RDMA.

3.1

  • Support fore more than 8 multipath ports.

3.0

  • Initial versions supporting Multipath and GDS integration.

Document generated on 2022.05.17 09:13 for v4.0-pre6, branch dev