Infrastructure

     2x10GbE Web frontend   1GbE VM/CI frontend 
       (10.202.96.12)         (10.202.96.13)
         ____|____              ____|____ 
        | R6515-1 |            | R6515-2 | 
        | Callisto|            | Ganymede|
        |_________|            |_________| 
             |                       | 
          NFS|25GbE mlxcx0        NFS|25GbE mlxcx1
             |       _________       |
             |______| R740XD2 |______| 
                    | Jupiter |
                    |_________|------- admin 1GbE bge0 (10.202.96.14)

Jupiter

Storage server configured to serve:

List available disks:

$ iostat -En
c5t22d1          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MZ7KH480HAHQ0D3  Revision: HF57 Serial No: S5CNNA0N507702 
Size: 480,10GB <480103981056 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t23d1          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: MZ7KH480HAHQ0D3  Revision: HF57 Serial No: S5CNNA0N507570 
Size: 480,10GB <480103981056 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t0d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG169GJ 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t1d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG13MXC 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t2d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG16BWT 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t3d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG10BH0 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t4d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG13KVY 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t5d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG169NB 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t6d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG0TPJK 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t7d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG169QY 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t8d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG169RK 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t9d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG13RH5 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t10d1          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG1637E 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 
c5t11d1          Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: SEAGATE  Product: ST4000NM017A     Revision: CSJ7 Serial No: WJG16ABJ 
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 0 Predictive Failure Analysis: 0 

The system is installed on mirrored 480GB SSDs:

$ zpool status rpool
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        rpool        ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            c5t22d1  ONLINE       0     0     0
            c5t23d1  ONLINE       0     0     0

errors: No known data errors

Create first RAIDZ2 volume with first six disks:

$ sudo zpool create tank raidz2 c5t0d1 c5t1d1 c5t2d1 c5t3d1 c5t4d1 c5t5d1

Create second RAIDZ2 volume with remaining six disks:

$ sudo zpool add tank raidz2 c5t6d1 c5t7d1 c5t8d1 c5t9d1 c5t10d1 c5t11d1

Check ZFS pool status:

$ zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        tank         ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c5t0d1   ONLINE       0     0     0
            c5t1d1   ONLINE       0     0     0
            c5t2d1   ONLINE       0     0     0
            c5t3d1   ONLINE       0     0     0
            c5t4d1   ONLINE       0     0     0
            c5t5d1   ONLINE       0     0     0
          raidz2-1   ONLINE       0     0     0
            c5t6d1   ONLINE       0     0     0
            c5t7d1   ONLINE       0     0     0
            c5t8d1   ONLINE       0     0     0
            c5t9d1   ONLINE       0     0     0
            c5t10d1  ONLINE       0     0     0
            c5t11d1  ONLINE       0     0     0

errors: No known data errors

Show short summary of ZFS pools:

$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool   444G  24,3G   420G        -         -     0%     5%  1.00x  ONLINE  -
tank   43,7T   428K  43,7T        -         -     0%     0%  1.00x  ONLINE  -

Callisto

Ganymede

Configuration snippets (Ubuntu)

Networking: netplan/50-cloud-init.yaml

network:
        ethernets:
                enp5s0f0:
                        addresses:
                                - 10.202.96.2/20
                        gateway4: 10.202.111.254
                        nameservers:
                                search: [cemef, cemef.mines-paristech.fr, sophia.mines-paristech.fr, corp.sophia.mines-paristech.fr]
                                addresses: [10.202.203.53, 10.202.203.54]
                enp5s0f1:
                        addresses:
                                - 172.20.128.200/24
                ib0:
                        addresses:
                                - 172.20.144.200/24
        version: 2

NFS:

# /etc/exports: the access control list for filesystems which may be exported
#       to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
/scratch 172.20.144.0/255.255.255.0(rw,async,no_root_squash)