[fedfs-utils] [PATCH] FedFS - simple setup howto
Chuck Lever
chuck.lever at oracle.com
Mon Aug 27 08:24:59 PDT 2012
Hi Ian-
FYI, we've got a patchwork repository, your patch is now here:
http://patchwork.ozlabs.org/patch/180159/
And I marked it "Under Review." Thanks for tackling this. Overall, the organization is right, but the details need a little work, and hopefully I've given you some pointers in the right direction.
Also there is an INSTALL file in the tar ball. Should we augment that instead?
On Aug 27, 2012, at 2:28 AM, Ian Kent wrote:
> Add a howto to setup a DNS domain fedfs.org, an (OpenLDAP) LDAP
> instance with context dc=fedfs,dc=org and work through the steps
> to setup a read-only FedFS root domain with a single referral.
>
> Signed-off-by: Ian Kent <raven at themaw.net>
> ---
>
> doc/howto/fedfs-simple-setup-howto.txt | 465 ++++++++++++++++++++++++++++++++
> 1 files changed, 465 insertions(+), 0 deletions(-)
> create mode 100644 doc/howto/fedfs-simple-setup-howto.txt
>
> diff --git a/doc/howto/fedfs-simple-setup-howto.txt b/doc/howto/fedfs-simple-setup-howto.txt
> new file mode 100644
> index 0000000..6454f64
> --- /dev/null
> +++ b/doc/howto/fedfs-simple-setup-howto.txt
> @@ -0,0 +1,465 @@
> +
> +Simple example of basic FedFS setup and operation
> +=================================================
> +
> +The examples in this test procedure assume a Fedora 18 install.
> +
> +Assume that the package fedfs-utils-server and fedfs-utils-client
> +are installed on perseus.fedfs.org. And that fedfs-utils-server and
> +a nfs-utils built with fedfs-utils-devel installed on zeus.fedfs.org
> +and that fedfs-utils-lib is also installed on zeus.fedfs.org.
Since fedfs-utils is nominally the "upstream" version, we should discuss the tar ball installation procedure here, and not a particular distributor's package installation.
> +
> +We want to setup perseus.fedfs.org to provide the domain root,
> +be able to use FedFS to mount the domain root, and to setup and
> +use a FedFS juntion to access an export on zeus.fedfs.org upon
> +access to a directory within the domain root. And lastly, setup
> +autofs to use the FedFS program map to mount the root of the
> +domain.
There are five roles in a FedFS domain: file server, filesystem client, NSDB, NSDB admin client, and DNS server. I find that helps organize the set-up steps.
It's not always clear below where you are switching between perseus and zeus, nor which host is running the forwarding DNS or the LDAP service. It would be good to clarify that. But also I think at least the filesystem client (which mounts the domain root or which runs autofs) ought to be a separate host.
> +
> +#Questions still to be answered:
> +#1. When and where (ie. on which server) is rpc.fedfsd used in the
> +# procedure below.
rpc.fedfsd is set up on the NFS file servers in the domain, including the file server that is sharing the domain root. Note, though, that it's optional: you can do the same operations by running "nfsref" or "nsdbparams" directly on file servers.
> +#2. What is the procedure for setting up a junction on a third
> +# unrelated machine.
Works pretty much like setting up the domain root. Pick an export on some file server, and use "nfsref" to set up a junction.
> +
> +Setup a forwarding DNS server for fedfs.org
> +===========================================
> +
> +Some of this setup may not be necessary but it is what I use to setup
> +DNS instances for testsing. Also many of the files created in step 1
> +should be included in a caching nameserver package. Installing the
> +cacheing nameserver package and making appropriate modifications to
> +turn the cacheing nameserver into a forwarding nameserver iis really
> +all that needs to be done. Of course the fedfs zone files will always
> +be needed.
> +
> +The example assumes the private IP address range 192.168.1 is being
> +used and a nameserver that is able to satisfy name resolution for
> +external names is at 192.168.1.1 and the hosts perseus.fedfs.org and
> +zeus.fedfs.org are assigned addresses 192.168.1.32 and 192.168.1.31
> +respectively.
> +
> +1. Setup zeus.fedfs.org as a forwarding DNS server:
Using a forwarding server is an interesting choice… we often do it this way for testing. But the point of the SRV record is to advertise the domain root globally. Should we describe here how to add an SRV to a non-forwarding DNS server instead, or in addition to, the below recipe?
Ah, much farther along, you set up the SRV record.
> +
> + mkdir -p /var/named/master/fedfs.org
> + if [ ! -d /var/named/data ]; then
> + mkdir /var/named/data
> + fi
> +
> +Create /var/named/named.root using:
> + dig +norec NS > /var/named/named.root
> +
> +Create /etc/named.rfc1912.zones as:
> + // named.rfc1912.zones:
> + //
> + // ISC BIND named zone configuration for zones recommended by
> + // RFC 1912 section 4.1 : localhost TLDs and address zones
> + //
> + zone "localdomain" IN {
> + type master;
> + file "localdomain.zone";
> + allow-update { none; };
> + };
> +
> + zone "localhost" IN {
> + type master;
> + file "localhost.zone";
> + allow-update { none; };
> + };
> +
> + zone "0.0.127.in-addr.arpa" IN {
> + type master;
> + file "named.local";
> + allow-update { none; };
> + };
> +
> + zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa" IN {
> + type master;
> + file "named.ip6.local";
> + allow-update { none; };
> + };
> +
> + zone "255.in-addr.arpa" IN {
> + type master;
> + file "named.broadcast";
> + allow-update { none; };
> + };
> +
> + zone "0.in-addr.arpa" IN {
> + type master;
> + file "named.zero";
> + allow-update { none; };
> + };
> +
> +Create /var/named/localdomain.zone as:
> + $TTL 86400
> + @ IN SOA localhost root (
> + 42 ; serial (d. adams)
> + 3H ; refresh
> + 15M ; retry
> + 1W ; expiry
> + 1D ) ; minimum
> + IN NS localhost
> + localhost IN A 127.0.0.1
> +
> +Create /var/named/localhost.zone
> + $TTL 86400
> + @ IN SOA @ root (
> + 42 ; serial (d. adams)
> + 3H ; refresh
> + 15M ; retry
> + 1W ; expiry
> + 1D ) ; minimum
> +
> + IN NS @
> + IN A 127.0.0.1
> + IN AAAA ::1
> +
> +Create /var/named/named.local as:
> + $TTL 86400
> + @ IN SOA localhost. root.localhost. (
> + 1997022700 ; Serial
> + 28800 ; Refresh
> + 14400 ; Retry
> + 3600000 ; Expire
> + 86400 ) ; Minimum
> + IN NS localhost.
> + 1 IN PTR localhost.
> +
> +Create /var/named/named.ip6.local as:
> + $TTL 86400
> + @ IN SOA localhost. root.localhost. (
> + 1997022700 ; Serial
> + 28800 ; Refresh
> + 14400 ; Retry
> + 3600000 ; Expire
> + 86400 ) ; Minimum
> + IN NS localhost.
> + 1 IN PTR localhost.
> +
> +Create /var/named/named.broadcast as:
> + $TTL 86400
> + @ IN SOA localhost. root.localhost. (
> + 42 ; serial (d. adams)
> + 3H ; refresh
> + 15M ; retry
> + 1W ; expiry
> + 1D ) ; minimum
> + IN NS localhost.
> +
> +Create /var/named/named.zero as:
> + $TTL 86400
> + @ IN SOA localhost. root.localhost. (
> + 42 ; serial (d. adams)
> + 3H ; refresh
> + 15M ; retry
> + 1W ; expiry
> + 1D ) ; minimum
> + IN NS localhost.
> +
> +Create /etc/named.conf as:
> + options {
> + listen-on port 53 { any; };
> + listen-on-v6 port 53 { ::1; };
> + directory "/var/named";
> + allow-query { any; };
> + forwarders { 192.168.1.1; };
> + recursion yes;
> + };
> +
> + logging {
> + channel default_debug {
> + file "data/named.run";
> + severity dynamic;
> + };
> + };
> +
> + zone "." IN {
> + type hint;
> + file "named.root";
> + };
> +
> + include "/etc/named.rfc1912.zones";
> +
> + include "/var/named/master/fedfs.org/zone.fedfs.org";
> +
> +Create /var/named/master/fedfs.org/zone.fedfs.org as:
> + zone "fedfs.org" {
> + type master;
> + notify no;
> + file "master/fedfs.org/db.fedfs.org";
> + };
> +
> + zone "1.168.192.in-addr.arpa" {
> + type master;
> + notify no;
> + file "master/fedfs.org/db.192.168.1";
> + };
> +
> +Create /var/named/master/fedfs.org/db.fedfs.org as:
> + $TTL 3D
> + @ IN SOA ns.fedfs.org. hostmaster.fedfs.org. (
> + 199802152 ; serial, todays date + todays serial #
> + 8H ; refresh, seconds
> + 2H ; retry, seconds
> + 4W ; expire, seconds
> + 1D ) ; minimum, seconds
> + ;
> + NS ns ; Inet Address of name server
> + ;
> + localhost A 127.0.0.1
> + ns A 192.168.1.32
> + zeus A 192.168.1.32
> + perseus A 192.168.1.31
> +
> + ; Assume these aren't present, they've been left to show an example
> + ; of the SRV record that would be used for a read-write FedFS domain
> + ; rather than the read-only setup we use here.
> + ;_nfs4._domainroot._tcp SRV 0 0 2049 perseus.fedfs.org.
> + ;_nfs4._write._domainroot._tcp SRV 0 0 2049 perseus.fedfs.org.
> +
> +Create /var/named/master/fedfs.org/db.192.168.1 as:
> + $TTL 3D
> + @ IN SOA ns.fedfs.org. hostmaster.fedfs.org. (
> + 199802151 ; Serial, todays date + todays serial
> + 8H ; Refresh
> + 2H ; Retry
> + 4W ; Expire
> + 1D) ; Minimum TTL
> + NS ns.fedfs.org.
> +
> + 32 PTR ns.fedfs.org.
> + 32 PTR zeus.fedfs.org.
> + 31 PTR perseus.fedfs.org.
> +
> + service named start
> +or
> + systemctl start named.service
> +
> +2. On perseus.fedfs.org and zeus.fedfs.org configure /etc/resolve.conf:
> + cd /etc
> + cp resolve.conf resolv.conf.orig
> + echo "domain fedfs.org" > resolve.conf
> + echo "search fedfs.org" >> resolve.conf
> + echo "nameserver 192.168.1.32" >> resolve.conf
> +
> +
> +Setup an NSDB (NameSpace DataBase)
> +==================================
> +
> +1. Set parameters for NSDB connections:
> + # nsdbparams(8) is used to set NSDB connection parameters
> + nsdbparams update -e "dc=fedfs,dc=org" \
> + -D "cn=Manager,dc=fedfs,dc=org" \
> + zeus.fedfs.org
Note: This command is run on file servers and NSDB administrative clients, not on the LDAP server, so those servers can contact the LDAP server correctly. Here's where I'm a little confused about whether this is perseus or zeus.
> +
> +2. Setup an OpenLDAP instance for fedfs.org:
> + service slapd stop
> +or
> + systemctl stop slapd.service
> +
> +Create /etc/openldap/slapd.conf as:
> + include /etc/openldap/schema/core.schema
> + include /etc/openldap/schema/cosine.schema
> + include /etc/openldap/schema/inetorgperson.schema
> + include /etc/openldap/schema/nis.schema
> + # Get this from the FedFS distribution
> + include /etc/openldap/schema/fedfs.schema
> +
> + pidfile /var/run/openldap/slapd.pid
> + argsfile /var/run/openldap/slapd.args
> +
> + database bdb
> + suffix "dc=fedfs,dc=org"
> + rootdn "cn=Manager,dc=fedfs,dc=org"
> + rootpw secret
> + directory /var/lib/ldap
> +
> + index objectClass eq,pres
> + index ou,cn,mail,surname,givenname eq,pres,sub
> + index uidNumber,gidNumber,loginShell eq,pres
> + index uid,memberUid eq,pres,sub
> + index nisMapName,nisMapEntry eq,pres,sub
> +
> +Create fedfs.org-naming-context.ldif as:
> + dn: dc=fedfs,dc=org
> + objectClass: domain
> + dc: fedfs
> + description: naming context
> +
> +Create an OpenLDAP instance for fedfs.org:
> + # Ensure that /etc/openldap/ldap.conf is setup to use fedfs.org
> + # by commenting out any other URI and BASE directives and then
> + # add definitions to the end of the configuration.
> + echo "URI ldap://zeus.fedfs.org/" >> /etc/openldap/ldap.conf
> + echo "BASE dc=fedfs,dc=org" >> /etc/openldap/ldap.conf
> +
> + # Setup OpenLDAP for fedfs.org
> + cd /etc/openldap
> + restorecon -v slapd.conf
> +
> + rm -f /var/lib/ldap/*
> + # supress warnings that this file does not exist
> + touch /var/lib/ldap/DB_CONFIG
> +
> + rm -rf slapd.d
> + slaptest -F slapd.d -f slapd.conf
> + # I think this will restore the correct ownership
> + # of the configuration directory tree but it may
> + # be necessary to chmod -R ldap.ldap slapd.d also.
> + restorecon -R -v slapd.d
> +
> + slapadd -l fedfs.org-naming-context.ldif
> + chown ldap.ldap /var/lib/ldap/*
> + restorecon -v /var/lib/ldap/*
> +
> + service slapd start
> +or
> + systemctl start slapd.service
Is all of this necessary to include here? I'm a detail-oriented person, but I'm little concerned about how long these instructions are (for DNS and LDAP set-up), since it gives the impression of a lot of complexity (yes, I know, there are already a lot of moving parts here). But one would hope that installing and configuring an LDAP server would be simpler…
> +
> +4. Add NCI (NSDB Container information) attributes to the
> + naming context LDAP entry:
> + nsdb-update-nci -l zeus.fedfs.org \
> + -D "cn=Manager,dc=fedfs,dc=org" \
> + -e "dc=fedfs,dc=org"
> +
> +
> +Add a FedFS junction within a domain root directory
> +===================================================
> +
> +Assuming there is a file system mounted on /vm (or just a directory
> +we can export) on server zues.fedfs.org which we want to access
> +under the domain root as <domain root mount point>/vm, we will be
> +exporting /.domainroot-fedfs.org as the domain root.
> +
> +1. Add an entry to /etc/exports on zeus.fedfs.org:
> +
> + # Add to /etc/exports
> + /vm *(ro)
> +
> + # Restart the nfs service or just re-export the table
> + exportfs -r
Why not just create and export a directory called "/.domainroot-fedfs.org" here? Alternatively you could describe how to use a bind mount to export some other FS as "/.domainroot-fedfs.org."
> +
> +2. Add a junction to the domain root on persues.fedfs.org:
Actually, the domain root is the top of the namespace tree, so you should create junctions in the domain root, not point junctions to it.
> +
> + #
> + # Tell nfsref the LDAP server (the NSDB) we are using to
> + # record file system name (FSN) and file system location
> + # (FSL) uuids. This assumes the LDAP connection parameters
> + # have been setup as in step 1 of "Setup an NSDB".
> + #
> + export FEDFS_NSDB_HOST=zeus.fedfs.org
> +
> + #
> + # Add the junction metadata to the directory and update
> + # the NSDB with uuid info of the junction.
> + #
> + mkdir -p /.domainroot-fedfs.org/vm
> + nfsref --type=nfs-fedfs \
> + add /.domainroot-fedfs.org/vm \
> + zeus.fedfs.org /vm
> +
> +
> +Setup fedfs domain root export (read-only case)
> +===============================================
> +
> +For this we are seeking to mount the domain root exported from host
> +perseus.fedfs.org.
> +
> +1. Add an SRV record for the FedFS file server to DNS:
> +
> + _nfs4._domainroot._tcp SRV 0 0 2049 perseus.fedfs.org.
> +
> +2. Restart named to make in available.
> +
> + service named restart
> +or
> + systemctl restart named.service
Or "rndc reload"
> +
> +3. Add an entry to /etc/exports on perseus.autofs.test:
> +
> + #
> + # Created when we added the junction above.
> + # mkdir /.domainroot-fedfs.org
> + #
> + /.domainroot-fedfs.org *(ro)
> +
> +4. Restart NFS:
> +
> + service nfs restart
> +or
> + systemctl restart nfs.service
Or "exportfs -rv"
> +
> +5. Mount using the FedFS mount utility on a local directory:
> +
> + mount -v -t fedfs /nfs4/fedfs.org /mnt
> + mount | grep domainroot
> + perseus.fedfs.org:/.domainroot-fedfs.org/ on /mnt type nfs4 ...
> +
> + cd /mnt/vm
> +
> + #
> + # This check assumes /etc/mtab is symlinked to /proc/mounts
> + # as it is in Fedora. Kernel automounted file systems will
> + # not be present in the text based /etc/mtab and so will not
> + # be seen in it. Look to /proc/mounts instead in this case.
> + #
> + mount | grep ^zeus.fedfs.org.*vm
> + zeus.fedfs.org:/vm/ on /mnt/vm type nfs4 ...
> +
> + # Ha, move out of the directory so it can be umounted
> + cd
> +
> +5. Lastly cleanup:
6. :-)
> +
> + #
> + # This example includes a specific umount of the junction
> + # (/mnt/vm) but such kernel automounted file systems are
> + # umounted automatically (when they are not in use) so it
> + # may not be present when this step is done.
> + #
> + umount /mnt/vm
> + umount /mnt
> +
> +Setup autofs to automount the domain root
> +=========================================
> +
> +1. Add a line to /etc/auto.master to automount FedFS root domains:
> +
> + #
> + # Note that the autofs pseudo option "nobind" probably
> + # should be used. In the case here it is required because
> + # the FedFS client also hosts the root of the domain and
> + # autofs will see the mount is local and perform a bind
> + # mount instead of an NFS mount. That, of course, means
> + # file system lookups won't be with an NFS file system
> + # so NFS referals can't be followed.
> + #
> + # Also note that the autofs mount point name must be
> + # /nfs4 to be able to mount nfs4 root domains.
> + #
> + echo "/nfs4 /usr/sbin/fedfs-map-nfs4 nobind" >> /etc/auto.master
> +
> +2. Restart or reload the autofs service:
> +
> + service autofs restart
> +or
> + systemctl restart autofs.service
> +
> +3. Check that we can mount the domain root and the referal:
> +
> + # automount the root domain.
> + [raven at perseus ~]$ ls /nfs4/fedfs.org
> + top.txt vm
> +
> + # automount the referal (from a different machine).
> + [raven at perseus ~]$ ls /nfs4/fedfs.org/vm
> + lost+found test.txt
> +
> + # Check they were mounted.
> + [raven at perseus ~]$ mount |grep perseus|grep nfs4
> + perseus.fedfs.org:/.domainroot-fedfs.org/ on /nfs4/fedfs.org type nfs4 ...
> + [raven at perseus ~]$ mount |grep zeus|grep nfs4
> + zeus.fedfs.org:/vm/ on /nfs4/fedfs.org/vm type nfs4 …
Or "findmnt -t nfs4"
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
More information about the fedfs-utils-devel
mailing list