Pacemaker cluster running Samba as a member of a Windows Domain (Direct Integration)

General Notes

This section is dedicated to briefly describe the major topics/resources that will be used as to achieve a Pacemaker cluster, running Samba, to become a member of a Windows Domain (Direct Integration)

As seen here those are the instructions for Active Directory Integration:
- set netbios name in smb.conf to the name you want your Samba resource to have (make sure it's the same on all nodes).
- On whichever node is currently running the Samba resource, join the AD domain (net ads join, etc. refer to the Samba documentation for specifics).
- That Samba instance should now be a member of the AD domain, and also be in the DNS (assuming your AD server is your DNS server).
- Edit the DNS on the AD server and remove the node's physical IP address (only the virtual IP should be associated with the name).
- This gives your resource a "virtual name" (to use the MSCS term).

As seen with man idmap_tdb2
Samba's idmap_tdb2 Backend for Winbind
The idmap_tdb2 plugin is a substitute for the default idmap_tdb backend used by winbindd for storing SID/uid/gid mapping tables in clustered environments with Samba and CTDB.

And those are the related resources:

[root@host1 ~]# pcs resource describe ocf:heartbeat:CTDB
ctdb_manages_samba: Should CTDB manage starting/stopping the Samba service for you? This will be deprecated in future, in favor of configuring a separate Samba resource.
ctdb_manages_winbind: Should CTDB manage starting/stopping the Winbind service for you? This will be deprecated in future, in favor of configuring a separate Winbind resource.

[root@host2 ~]# pcs resource describe systemd:smb
systemd:smb - systemd unit file for smb
Cluster Controlled smb

[root@host2 ~]# pcs resource describe systemd:winbind
systemd:winbind - systemd unit file for winbind
Cluster Controlled winbind

[root@host2 ~]# pcs resource describe systemd:nmb
systemd:nmb - systemd unit file for nmb
Samba NMB Daemon

Test Environment

Below is a list with the used VMs in addition with the configured IP addresses :

Prerequisites

The following section will just mention
a. the prerequisites as someone to achieve having a high available File System, that samba will use, and
b. a brief overview of the expected results.
(Intentionally the following steps are not covering the HA cluster installation and the storage configuration as this is out of the scope of this article).
As to be more specific,
- iSCSI and multipath technologies have been used
- clustered LVM and
- on top of it, a GFS2 clustered partition has been created, mounted under /mnt/gfs2share

[ALL ~]# iscsiadm -m session -P 3 | grep -A 4 'Attached SCSI'
Attached SCSI devices:
************************
Host Number: 2    State: running
scsi2 Channel 00 Id 0 Lun: 0
Attached scsi disk sda        State: running
--
Attached SCSI devices:
************************
Host Number: 3    State: running
scsi3 Channel 00 Id 0 Lun: 0
Attached scsi disk sdb        State: running

[ALL ~]# multipath -ll
mpatha (360014057e06a97069f24891b988fbd20) dm-2 LIO-ORG ,block1          
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 2:0:0:0 sda 8:0  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 3:0:0:0 sdb 8:16 active ready running

[ALL ~]# lsblk
NAME                           MAJ:MIN RM  SIZE  RO   TYPE  MOUNTPOINT
sda                              8:0        0  20G    0    disk  
└─mpatha                       253:2        0  20G    0    mpath
  └─cluster_vg-cluster_lv          253:3        0    5G    0    lvm   /mnt/gfs2share
sdb                              8:16          0  20G    0    disk  
└─mpatha                       253:2        0  20G    0    mpath
    └─cluster_vg-cluster_lv      253:3        0    5G    0    lvm   /mnt/gfs2share

[ALL ~]# mount | grep gfs2share
/dev/mapper/cluster_vg-cluster_lv on /mnt/gfs2share type gfs2 (rw,noatime)

Basic configuration

RPMs & Units

[ALL ~]# yum install samba ctdb cifs-utils samba-client samba-winbind samba-winbind-clients krb5-workstation
[ALL ~]# systemctl stop ctdb; systemctl stop smb; systemctl stop nmb; systemctl stop winbind
[ALL ~]# systemctl disable ctdb; systemctl disable smb; systemctl disable nmb; systemctl disable winbind

config files

[ALL ~]# cat << END > /etc/samba/smb.conf
[global]
   netbios name = idmcluster
   workgroup = hacluster
   ; server string = idmcluster - %$HOSTNAME
   realm = HACLUSTER.MYLAB.LOCAL
   security = ADS

   ## winbind options ##
   idmap config * : backend    = tdb2
   idmap config * : range = 1000000-19999999
   idmap config * : rangesize = 1000000
   winbind enum users = Yes
   winbind enum groups = Yes
   winbind use default domain = No
   ; winbind use default domain = Yes
   winbind separator = +

   ##  kerberos ##
   dedicated keytab file = FILE:/mnt/gfs2share/krb5.keytab
   ;kerberos method = dedicated keytab
   kerberos method = secrets and keytab

   ## user related options ##
   template homedir = /home/%D/%U
   template shell = /bin/bash

   ## cluster-related ##
   clustering = yes
   ctdbd socket = /var/run/ctdb/ctdbd.socket

   ## logs-related ##
   log level = 1
   debug pid = true
   max log size = 0
END

[ALL ~]# cat << END > /etc/krb5.conf
includedir /etc/krb5.conf.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_keytab_name = /mnt/gfs2share/krb5.keytab
 # dns_lookup_realm = true
 # dns_lookup_kdc = true
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 default_ccache_name = KEYRING:persistent:%{uid}
 default_realm = HACLUSTER.MYLAB.LOCAL

 [realms]
 # Define only if DNS lookups are not working
 HACLUSTER.MYLAB.LOCAL = {
   kdc = win2k12r2.hacluster.mylab.local
   admin_server = win2k12r2.hacluster.mylab.local
   default_domain = hacluster.mylab.local
 }

[domain_realm]
 # Define only if DNS lookups are not working
  .hacluster.mylab.local = HACLUSTER.MYLAB.LOCAL
  hacluster.mylab.local = HACLUSTER.MYLAB.LOCAL
END

[ALL ~]# cat << END > /etc/ctdb/nodes
192.168.100.101
192.168.100.102
END

[ALL ~]# cat << END > /etc/ctdb/public_addresses
192.168.100.251/24 ens9
192.168.100.252/24 ens9
END

Defining the resources

[root@host1 ~]# mkdir -p /mnt/gfs2share/ctdb

[root@host1 ~]# pcs resource create ctdb ocf:heartbeat:CTDB \
ctdb_socket="/var/run/ctdb/ctdbd.socket" \
ctdb_recovery_lock="/mnt/gfs2share/ctdb/ctdb.lock" \
ctdb_dbdir="/var/lib/ctdb" \
ctdb_logfile="/var/log/log.ctdb" \
ctdb_debuglevel=1 \
ctdb_manages_samba="no" \
ctdb_manages_winbind="no" \
op monitor interval=10 timeout=30 \
op start timeout=90 op stop timeout=100 \
clone

[root@host1 ~]# pcs resource create samba systemd:smb --group hasmbshare
[root@host1 ~]# pcs resource create winbind systemd:winbind --group hasmbshare
[root@host1 ~]# pcs resource create nmb systemd:nmb --group hasmbshare
[root@host1 ~]# pcs resource clone hasmbshare

[root@host1 ~]# pcs constraint order clusterfs-clone then ctdb-clone
[root@host1 ~]# pcs constraint colocation add ctdb-clone with clusterfs-clone
[root@host1 ~]# pcs constraint order ctdb-clone then hasmbshare-clone
[root@host1 ~]# pcs constraint colocation add hasmbshare-clone with ctdb-clone

Direct Integration

Joining to the Domain

[root@host1 ~]# net ads join -w HACLUSTER -S win2k12r2.hacluster.mylab.local -U administrator
_or_
[root@host1 ~]# kinit administrator
Password for administrator@HACLUSTER.MYLAB.LOCAL: 
[root@host1 ~]# 
[root@host1 ~]# net ads join -w HACLUSTER -S win2k12r2.hacluster.mylab.local -k
Using short domain name -- HACLUSTER
Joined 'IDMCLUSTER' to dns domain 'hacluster.mylab.local'
Not doing automatic DNS update in a clustered setup.

[root@host1 ~]# net ads keytab add cifs -U administrator
_or_
[root@host1 ~]# kinit administrator
Password for administrator@HACLUSTER.MYLAB.LOCAL: 
[root@host1 ~]# 
[root@host1 ~]# net ads keytab add cifs -k
Processing principals to add...
[root@host1 ~]#

Defining the users & the collab folder

[ALL ~]# authconfig --enablewinbind --enablewinbindauth --enablemkhomedir --disablesssd --disablesssdauth --update

[root@host1 ~]# mkdir -p /mnt/gfs2share/domain_share
[root@host1 ~]# chown root:"HACLUSTER+domain users" /mnt/gfs2share/domain_share/
[root@host1 ~]# chmod 770 /mnt/gfs2share/domain_share/
[root@host1 ~]# ls -ld /mnt/gfs2share/domain_share/
drwxrwx--- 2 root HACLUSTER+domain users 3864 Dec 20 15:10 /mnt/gfs2share/domain_share
[root@host1 ~]#

[ALL ~]# cat << END >> /etc/samba/smb.conf
[domain_share]
   path = /mnt/gfs2share/domain_share
;   guest ok = yes
   valid users = "@HACLUSTER+domain users"
   read only = no
END

[root@host1 ~]# pcs resource restart hasmbshare-clone

Expected results

Cluster status

[root@host2 ~]# pcs status
Cluster name: idmcluster
Stack: corosync
Current DC: host2.hacluster.mylab.local (version 1.1.16-12.el7_4.7-94ff4df) - partition with quorum
Last updated: Sat Mar  3 11:58:27 2018
Last change: Sat Mar  3 11:30:13 2018 by hacluster via crmd on host2.hacluster.mylab.local

2 nodes configured
15 resources configured

Online: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]

Full list of resources:

 fence_host1    (stonith:fence_xvm):    Started host2.hacluster.mylab.local
 fence_host2    (stonith:fence_xvm):    Started host1.hacluster.mylab.local
 Clone Set: dlm-clone [dlm]
     Started: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]
 fence_all_scsi (stonith:fence_scsi):   Started host2.hacluster.mylab.local
 Clone Set: clusterfs-clone [clusterfs]
     Started: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]
 Clone Set: ctdb-clone [ctdb]
     Started: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]
 Clone Set: hasmbshare-clone [hasmbshare]
     Started: [ host1.hacluster.mylab.local host2.hacluster.mylab.local ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@host2 ~]# 

Samba status

[root@host1 ~]# ctdb nodestatus
Number of nodes:2
pnn:0 192.168.100.101  OK (THIS NODE)
pnn:1 192.168.100.102  OK

[root@host1 ~]# ctdb ip
Public IPs on node 0
192.168.100.251 1
192.168.100.252 0

[root@host1 ~]# ip add | grep 25[1-2]
    inet 192.168.100.252/24 brd 192.168.100.255 scope global secondary ens9

[root@host2 ~]# ip add | grep 25[1-2]
    inet 192.168.100.251/24 brd 192.168.100.255 scope global secondary ens9

[root@host1 ~]# nslookup idmcluster.hacluster.mylab.local
Server:        192.168.100.120
Address:    192.168.100.120#53

Name:    idmcluster.hacluster.mylab.local
Address: 192.168.100.251
Name:    idmcluster.hacluster.mylab.local
Address: 192.168.100.252
[root@host1 ~]#

[root@host1 ~]#  klist -k
Keytab name: FILE:/mnt/gfs2share/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 host/idmcluster.hacluster.mylab.local@HACLUSTER.MYLAB.LOCAL (des-cbc-crc)
   ...
   3 cifs/idmcluster.hacluster.mylab.local@HACLUSTER.MYLAB.LOCAL (des-cbc-crc)
   ...
[root@host1 ~]# 

[root@host1 ~]# net ads info
LDAP server: 192.168.100.120
LDAP server name: win2k12r2.hacluster.mylab.local
Realm: HACLUSTER.MYLAB.LOCAL
Bind Path: dc=HACLUSTER,dc=MYLAB,dc=LOCAL
LDAP port: 389
Server time: Wed, 28 Feb 2018 20:06:34 CET
KDC server: 192.168.100.120
Server time offset: -1
Last machine account password change: Sat, 23 Dec 2017 11:55:15 CET
[root@host1 ~]# 

[root@host1 ~]# wbinfo -u; echo "---"; wbinfo -g
HACLUSTER+administrator
HACLUSTER+guest
HACLUSTER+krbtgt
HACLUSTER+aduser
---
HACLUSTER+winrmremotewmiusers__
HACLUSTER+domain computers
HACLUSTER+domain controllers
HACLUSTER+schema admins
HACLUSTER+enterprise admins
HACLUSTER+cert publishers
HACLUSTER+domain admins
HACLUSTER+domain users
...
[root@host1 ~]# 

Use case

[root@host1 ~]# date; ssh -l HACLUSTER+aduser localhost
Thu  1 Mar 12:02:03 CET 2018
HACLUSTER+aduser@localhost's password: 
Creating directory '/home/HACLUSTER/aduser'.

[HACLUSTER+aduser@host1 ~]$ smbclient -m SMB3 //idmcluster.hacluster.mylab.local/domain_share -k
smb: \> exit

[HACLUSTER+aduser@host1 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1000003
Default principal: aduser@HACLUSTER.MYLAB.LOCAL

Valid starting     Expires            Service principal
01/03/18 12:02:06  01/03/18 22:02:06  krbtgt/HACLUSTER.MYLAB.LOCAL@HACLUSTER.MYLAB.LOCAL
    renew until 08/03/18 12:02:06
01/03/18 12:02:06  01/03/18 22:02:06  IDMCLUSTER$@HACLUSTER.MYLAB.LOCAL
    renew until 08/03/18 12:02:06
01/03/18 12:02:31  01/03/18 22:02:06  cifs/idmcluster.hacluster.mylab.local@HACLUSTER.MYLAB.LOCAL
    renew until 08/03/18 12:02:06
[HACLUSTER+aduser@host1 ~]$