Infrastructure
2x10GbE Web frontend 1GbE VM/CI frontend
(10.202.96.12) (10.202.96.13)
____|____ ____|____
| R6515-1 | | R6515-2 |
| Callisto| | Ganymede|
|_________| |_________|
| |
NFS|25GbE mlxcx0 NFS|25GbE mlxcx1
| _________ |
|______| R740XD2 |______|
| Jupiter |
|_________|------- admin 1GbE bge0 (10.202.96.14)
Jupiter
Storage server configured to serve:
- GIT repositories to Callisto,
- VM images to Ganymede,
- User directories exported through NFS.
- Storage: 12 x 4TB set up as stripped 2 x RAIDZ2 (6 disks).
List available disks:
$ iostat -En
c5t22d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: MZ7KH480HAHQ0D3 Revision: HF57 Serial No: S5CNNA0N507702
Size: 480,10GB <480103981056 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t23d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: MZ7KH480HAHQ0D3 Revision: HF57 Serial No: S5CNNA0N507570
Size: 480,10GB <480103981056 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t0d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG169GJ
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t1d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG13MXC
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t2d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG16BWT
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t3d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG10BH0
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t4d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG13KVY
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t5d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG169NB
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t6d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG0TPJK
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t7d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG169QY
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t8d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG169RK
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t9d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG13RH5
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t10d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG1637E
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c5t11d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST4000NM017A Revision: CSJ7 Serial No: WJG16ABJ
Size: 4000,79GB <4000787030016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
The system is installed on mirrored 480GB SSDs:
$ zpool status rpool
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c5t22d1 ONLINE 0 0 0
c5t23d1 ONLINE 0 0 0
errors: No known data errors
Create first RAIDZ2 volume with first six disks:
$ sudo zpool create tank raidz2 c5t0d1 c5t1d1 c5t2d1 c5t3d1 c5t4d1 c5t5d1
Create second RAIDZ2 volume with remaining six disks:
$ sudo zpool add tank raidz2 c5t6d1 c5t7d1 c5t8d1 c5t9d1 c5t10d1 c5t11d1
Check ZFS pool status:
$ zpool status tank
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c5t0d1 ONLINE 0 0 0
c5t1d1 ONLINE 0 0 0
c5t2d1 ONLINE 0 0 0
c5t3d1 ONLINE 0 0 0
c5t4d1 ONLINE 0 0 0
c5t5d1 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
c5t6d1 ONLINE 0 0 0
c5t7d1 ONLINE 0 0 0
c5t8d1 ONLINE 0 0 0
c5t9d1 ONLINE 0 0 0
c5t10d1 ONLINE 0 0 0
c5t11d1 ONLINE 0 0 0
errors: No known data errors
Show short summary of ZFS pools:
$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 444G 24,3G 420G - - 0% 5% 1.00x ONLINE -
tank 43,7T 428K 43,7T - - 0% 0% 1.00x ONLINE -
- Authentication: LDAP CEMEF
Callisto
- GIT/etc...
Ganymede
- VM server
Configuration snippets (Ubuntu)
Networking: netplan/50-cloud-init.yaml
network:
ethernets:
enp5s0f0:
addresses:
- 10.202.96.2/20
gateway4: 10.202.111.254
nameservers:
search: [cemef, cemef.mines-paristech.fr, sophia.mines-paristech.fr, corp.sophia.mines-paristech.fr]
addresses: [10.202.203.53, 10.202.203.54]
enp5s0f1:
addresses:
- 172.20.128.200/24
ib0:
addresses:
- 172.20.144.200/24
version: 2
NFS:
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)
/scratch 172.20.144.0/255.255.255.0(rw,async,no_root_squash)