Kerberized NFSv4, local root and autofs

This particular scenario assumes that we want local root user (which is typically used by Linux system services such as Docker) on various kerberized NFSv4 clients to be able to access and create files on a kerberized share. We also want files and directories created by local root user of one client to be operable on by local root user of other clients. The typical use case would be a cluster of Docker or Kubernetes nodes that all need access to the same data.

Following these suggestions REQUIRES that all your Linux machines are joined to Active Directory as described in this article and your NFSv4 configuration is done in accordance with this one. The resulting configuration will allow for local root user on your client machines to automatically and transparently authenticate against AD using machine account credentials stored in your krb5.keytab. DO NOT run kinit manually as local root at any point or you will likely screw up your authentication context.

Put AD machine accounts of clients (rockytest and docker01 in this example) we want to grant access to data into an AD group, let’s call it NFS_docker. Then we do the following server-side:

chmod 770 /nfs-export
chgrp NFS_docker /nfs-export
chmod g+s /nfs-export

At this point, new files and directories created under /nfs-export on the server will be inheriting the NFS_docker group. Note that for a few moments after setup, I noticed that while server-side the group showed up correctly on newly created files immideately, clients that had the NFS share already mounted displayed group nobody (Rocky client) and group 4294967294 (Ubuntu client) for a few minutes. In the few minutes I had spent googling about this, the problem had resolved itself without any intervention, meaning SSSD had done it’s magic in resolving the gids from AD.

Now we need to solve the issue of umasks. Default umasks for both Redhat and Debian family distributions not only differ, neither distro family allow for other group members to modify files created by one another by default. While in theory you could enforce umasks client-side, we are doing this for local root users and enforcing 770 directory and 660 file permissions for all files created by root is a massive security disaster waiting to happen, so we are going to do the enforcement by creating an ACL server-side:

setfacl -d -m u::rwx /nfs-export
setfacl -d -m g::rwx /nfs-export
setfacl -d -m o::- /nfs-export

After the change, new files and directories created by local root of kerberized clients on the share look like this:

drwxrws--- 2 rockytest$ nfs_docker 6 Jul 1 23:30 testdir1
drwxrws--- 2 docker01$ nfs_docker 6 Jul 1 23:31 testdir2
-rw-rw---- 1 rockytest$ nfs_docker 0 Jul 1 23:30 testfile1
-rw-rw---- 1 docker01$ nfs_docker 0 Jul 1 23:31 testfile2

Now we have NFS client machines belonging to a mutual AD group being capable of operating on each others data. At this point, instead of declaring the nfsv4 mount in every clients fstab, we will use autofs to configure on-demand share access. This eliminates the risk of a stalled mount process during boot due to lack of networking or Kerberos ticket as well as reduces unneeded strain on the network by only keeping up an NFS connection that is actually being used for something.

First we install the autofs package for our distribution on the client, then we define a direct automount map file in our autofs master configuration file /etc/auto.master:

/-        /etc/auto.clientmount

Then we create the abovementioned /etc/auto.clientmount map file with the following configuration:

/path/to/local/mountpoint    -fstype=nfs4    nfs.server.fqdn:/nfs-export/

The “clientmount” map name is an arbitrary example and you can name it however you want. Unmount and disable any previously existing mount created manually or via fstab on the clients, then enable and start the service (or restart it if it was already running):

systemctl enable autofs
systemctl start autofs

At this point, you might try running “df” or “ls /path/to/mount” and wonder why nothing is showing up. This is by design. Neither command actually uses the filesystem to the point of autofs actually enabling the defined mount. Try to cd into the mount path or create a new file in it and suddenly the mount appears. Depending on your distribution, the mount will autoremove itself after 5-10 minutes of inactivity.

If you are having trouble, need to debug and journalctl isn’t being too helpful:

Stop the autofs daemon:

systemctl autofs stop

Run automount in the foreground with verbose information:

automount -f -v

Open another terminal and try accessing the mount path and watch the first terminal for errors.


When using Docker or Kubernetes on top of autofs, special consideration must be given to your container volume mount configuration or you risk running into “Too many levels of symbolic links” issue that seems well-documented online. Docker needs “:shared” to be included in volume mount configuration and there are various solutions for Kubernetes as well. You could, obviously, take another approach and skip autofs altogether, keeping your NFS storage permanently mounted on all nodes and script some some sort of delay into the NFS mount process to avoid potential boot stalls.

ACLs in modern (RHEL 8.x, Ubuntu 20.04) distributions seem to mostly “just work”. Contrary to most guides found online, there is apparently no longer any need to deliberately enable acl support via filesystem mount options. Neither on the server nor the clients.

Utilities provided by the nfs4-acl-tools package such as nfs4_getfacl and nfs4_setfacl will ONLY work from client-side. We are using setfacl directly on the NFS server.

You REALLY don’t want to kinit manually as local root on the clients once everything is running / otherwise screw with contents of root’s KEYRING:persistent:%{uid} ccache.

Not really sure why, but after everything is configured and working, running “klist” as local root on RHEL-based clients will show output similar to the following:

Ticket cache: KCM:0:53219
Valid starting       Expires              Service principal
01/01/1970 02:00:00  01/01/1970 02:00:00  Encrypted/Credentials/v1@X-GSSPROXY:

While running “klist” on an Ubuntu client will result in:

klist: Credentials cache keyring 'persistent:0:0' not found

Yet, both clients have access.

NFSv4 with Kerberos against Active Directory

What this guide will help you do: You will be able to access NFSv4 shares when logged onto a Linux client system with an Active Directory user account, your NFS traffic will no longer be clear-text and vulnerable to traffic snooping, both server and client will transparently verify each other’s identity and you will be able to map AD user and group permissions to your NFS shares.

All Linux machines involved must be joined to AD, see here for an example of a valid configuration.
They must have their hostnames in FQDN format
They must have valid A+PTR DNS records
Port 2049 on the server needs to be accessible
ALL of the below has to work:

getent group "Domain Users"
nslookup my.domain
chronyc sources (or another timesync client, should point towards your domain time sources)

NFSv4 server (RHEL/CENTOS 8.x) configuration:

First, we install the server binaries and enable require services:

yum install -y nfs-utils
systemctl enable gssproxy.service
systemctl enable nfs-server

Your /etc/idmapd.conf on the NFS server should have the following:

Domain = my.domain
Local-Realms = MY.DOMAIN

Method = nsswitch,static
GSS-Methods = nsswitch,static

Let’s disable old protocol versions in /etc/nfs.conf:


Now we disable unneeded pre-nfv4 services and start the server:

systemctl mask --now rpc-statd.service rpcbind.service rpcbind.socket
systemctl start nfs-server

Let’s create a directory on the server to be shared:

mkdir /nfs-export
chown aduser /nfs-export
chgrp adgroup /nfs-export

Assuming you did not disable SELinux, we need to fix the security context:

yum install -y policycoreutils-python
semanage fcontext -a -t nfs_t "/nfs-export(/.*)?"
restorecon -Rv /nfs-export

Now we create our share via /etc/exports file on the server and allow clients from access:


And export the newly created share with:

exportfs -rav

You can check for any current exports on the server with:

cat /var/lib/nfs/etab
exportfs -v

We obtain a ticket for our AD admin account and create an NFS SPN for our server AD computer account:

adcli update --service-name=nfs

You can check that a new keytab entry was created for the SPN in /etc/krb5.keytab on the server with:

klist -k

Having multiple new entries show up is normal to support a multitude of cipher suites. If you open the server’s computer account details in AD and look up servicePrincipalName in Attribute Editor, you should now see entries that look like nfs/hostname and nfs/fqdn.

NFSv4 client configuration:

This is the easy part. Assuming your client was joined to AD in the same fashion as the server via the method previously linked, mounting the share is simply a matter of having the tools installed and the correct service enabled on the client. No actual NFS configuration is required.

Redhat/CentOS 8.x (seemingly autoenabled the required service):

yum install -y nfs-utils
mount -vvv -t nfs4 -o vers=4.2,sec=krb5p,rw nfs.server.fqdn:/nfs-export /mnt/whatever

Ubuntu 20.04 LTS:

apt install nfs-common
systemctl enable
systemctl start
mount -vvv -t nfs4 -o vers=4.2,sec=krb5p,rw nfs.server.fqdn:/nfs-export /mnt/whatever

Login via SSH to the Linux NFSv4 client using an AD user account that you granted access while creating the export directory on the server and MAGIC HAPPENED. The AD user logged on to the client has access to the files shared from the server to the client.

If you need to have local root user on multiple kerberized NFS clients to be able to access and manipulate files and directories created by each other, look here.

Ansible: joining Redhat 8 to Active Directory

Now that I’ve been using Ansible for joining Redhat and CentOS 8.x Linux machines to Active Directory domains in production for a while, I’m confident enough to share the playbook I’ve been using. This example uses password-based authentication and sudo elevation for the ansible user, so make sure to edit accordingly in case you are using key-based auth.

When testing, don’t forget to add your AD account to the newly created AD admin group, this has bitten me way more times than I’d like to admit and in case you’re wondering what that OID in ad_access_filter does: it allows for recursive group permissions to work, so while you can add users directly, you do not HAVE to.

join-ad-linux-rhel tree:

admin.account@server join-ad-linux-rhel]$ tree
├── ansible.cfg
├── hosts
├── join-ad-linux-rhel.yaml
└── templates
    ├── ADsudoers.j2
    ├── krb5.j2
    └── sssd.j2

1 directory, 6 files


inventory           = hosts
host_key_checking   = False


examplehost ansible_host= ansible_ssh_user=ansibleuser


- hosts: all
  user: ansibleuser
  become: true

      - sssd
      - sssd-tools
      - realmd
      - oddjob
      - oddjob-mkhomedir
      - adcli
      - samba-common
      - samba-common-tools
      - krb5-workstation
      - openldap-clients
    AD_Domain: MY.DOMAIN
    AD_Domain_alt: my.domain
    Join_OU: OU="member servers",OU=computers,DC=my,DC=domain
    SRV_ADM_GRP_OU: OU=groups,DC=my,DC=domain

    - name: username
      prompt: "What is your AD administrator username?"
      private: no

    - name: password
      prompt: "What is your AD administrator password?"
      private: yes

    - name: adhostname
      prompt: "What is the desired hostname in a simple, non-fqdn format?"
      private: no


  - name: Checking if running RedHat/CentOS
      msg: The system is not running RedHat/CentOS, aborting
    when: ansible_facts['os_family'] != 'RedHat'

  - name: Checking if packages required to join AD realm are present
    yum: name={{ pkgs }} state=present update_cache=yes

  - name: Settings up hostname
    shell: hostnamectl set-hostname {{ adhostname }}.{{ AD_Domain_alt }}

  - name: Joinining the AD realm (creating AD computer account and updating /etc/krb5.keytab)
    shell: echo '{{ password }}' | adcli join --stdin-password {{ AD_Domain }} -U {{ username }} --domain-ou={{ Join_OU }}

  - name: Creating AD server admin group
    shell: echo '{{ password }}' | adcli create-group ADM_{{ adhostname }} --stdin-password --domain={{ AD_Domain }} --description="Admin group for {{ adhostname }} server" --domain-ou={{ SRV_ADM_GRP_OU }} -U {{ username }}

  - name: Configuring sssd.conf
      src: sssd.j2
      dest: /etc/sssd/sssd.conf
      owner: root
      group: root
      mode: 0600

  - name: Configuring krb5.conf
      src: krb5.j2
      dest: /etc/krb5.conf
      owner: root
      group: root
      mode: 0644

  - name: Configuring sudoers
      src: ADsudoers.j2
      dest: /etc/sudoers.d/ADsudoers
      owner: root
      group: root
      mode: 0440

  - name: Configuring PAM/SSHD to use SSSD
    shell: authselect select sssd with-mkhomedir --force

  - name: Enabling oddjobd service
      name: oddjobd.service
      enabled: yes
      state: started

  - name: Restarting SSSD
      name: sssd
      enabled: yes
      state: restarted


%ADM_{{ adhostname }}@{{ AD_Domain_alt }}      ALL=(ALL:ALL) ALL


domains = {{ AD_Domain_alt }}
config_file_version = 2
services = nss, pam

[domain/{{ AD_Domain_alt }}]
krb5_realm = {{ AD_Domain }}
realmd_tags = manages-system joined-with-adcli
cache_credentials = True
id_provider = ad
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = False
fallback_homedir = /home/%u
access_provider = ad
ad_maximum_machine_account_password_age = 30
ad_access_filter = DOM:{{ AD_Domain_alt }}:(memberOf:1.2.840.113556.1.4.1941:=cn=ADM_{{ adhostname }},{{ SRV_ADM_GRP_OU }})

dyndns_update = false
dyndns_update_ptr = false

#debug_level = 9


includedir /etc/krb5.conf.d/

    default = FILE:/var/log/krb5libs.log
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmind.log

    dns_lookup_realm = true
    dns_lookup_kdc = true
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = true
    rdns = false
    pkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crt
    spake_preauth_groups = edwards25519
    default_realm = {{ AD_Domain }}
    default_ccache_name = KEYRING:persistent:%{uid}

    {{ AD_Domain }} = {

    .{{ AD_Domain_alt }} = {{ AD_Domain }}
    {{ AD_Domain_alt }} = {{ AD_Domain }}

When copy/pasting these into actual files on your system, take care to ensure indentation isn’t lost as YAML is very particular about these things. Run the playbook with:

ansible-playbook -k -K join-ad-linux-rhel.yaml

iptables core with ssh drop logging


:LOGGING - [0:0]

-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT

# here you allow the ssh/22 connections you need
-A INPUT -p tcp -m state --state NEW -m tcp -s IP.YOU.WANT.TOALLOW --dport 22 -j ACCEPT

# remaining 22/ssh connections are sent to logging
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j LOGGING

# recieve and log
-A LOGGING -m limit --limit 6/min --limit-burst 3 -j LOG --log-prefix "[iptables dropped] "

# drop for real



# lets log the dropped connections into it's own file
:msg,contains,"[iptables dropped]" /var/log/iptables.log
# stop processing
& stop


  rotate 14
        /usr/bin/systemctl kill -s HUP rsyslog.service >/dev/null 2>&1 || true

Ansible: syncing Linux to AD time

Requirements: the PDC address is looked up from Active Directory DNS, so the machine you’re running the playbook against must be able to resolve names from the domain namespace. Assumes your timezone is set correctly.

ansible/setup-domaintime-linux/setup-domaintime-linux.yaml :

- hosts: all
  user: localuser
  become: true

      - chrony
      - bind-utils

    - name: domain_fqdn
      prompt: "Enter domain FQDN to search for PDC (example: mydomain.local)"
      private: no


  - name: Checking if running RedHat/CentOS
      msg: The system is not running RedHat/CentOS, aborting
    when: ansible_facts['os_family'] != 'RedHat'

  - name: Ensuring ntpdate is absent
      name: ntpdate
      state: absent

  - name: Ensuring chrony and bind-utils are present
    yum: name={{ pkgs }} state=present update_cache=yes

  - name: Masking ntpd service
      name: ntpd
      enabled: no
      masked: yes
      state: stopped

  - name: Looking up PDC in AD DNS
    shell: nslookup -q=SRV _ldap._tcp.pdc._msdcs.{{ domain_fqdn }} | grep _ldap | awk '{print $7}' | head --bytes -2
    register: PDClookup
  - set_fact:
      PDCfqdn : "{{ PDClookup.stdout }}"

  - name: Configuring chrony.conf
      src: chrony.conf.j2
      dest: /etc/chrony.conf
      owner: root
      group: root
      mode: 0644

  - name: Enabling chronyd service
      name: chronyd
      enabled: yes
      state: restarted

ansible/setup-domaintime-linux/templates/chrony.conf.j2 :

server {{ PDCfqdn }} iburst

driftfile /var/lib/chrony/drift
makestep 0.1 3

logdir /var/log/chrony
log measurements statistics tracking

leapsectz right/UTC

ansible/setup-domaintime-linux/ansible.cfg :

inventory           = hosts
host_key_checking   = False
become_method       = sudo

ansible/setup-domaintime-linux/hosts :

virtualhostname ansible_host= ansible_ssh_user=localuser