# Main # Tech #

ZFS over VFS-FUSE (Embedded Cluster)

Testing ZFS operation using as vdevs, OBJECTBLOKCS of remote CSRB Nodes running on the CSRB Testbed's BBB Cluster.

Demonstration Videos

    Blue LED configuration (TOP-BOTTOM)
  • SD Card Activity
  • eMMC Activity
  • (unused)
  • CPU Activity

ZFS RAID-Z2 (Long Demo)

  • Z2 RAID (8 data, 2 parity)
  • ashift=15 recordsize=256k sync=standard primarycache=metadata
  • Dual screen video capture.
  • The Top screen shows the mpstat outputs of all 10 Nodes.
  • Test steps:
    1. dd a 1GB file with urandomdata
    2. zpool scrub
    3. dd the 1GB file, to /dev/null
    4. iozone auto run with DIRECT IO

ZFS RAID-Z2

  • Z2 RAID (8 data, 2 parity)
  • ashift=15 recordsize=256k sync=standard primarycache=all
  • Test steps:
    1. dd a 512MB file with urandomdata
    2. zpool scrub
    3. dd the 512MB file, to /dev/null

ZFS dRAID-1

  • d1 RAID (10 childs, 3 data, 1 parity, 0 spare)
  • ashift=15 recordsize=128k sync=standard primarycache=all
  • Test steps:
    1. dd a 512MB file with urandomdata
    2. zpool scrub
    3. dd the 512MB file, to /dev/null

ZFS RAID-0

  • Stripe RAID (10 devices)
  • ashift=15 recordsize=256k sync=standard primarycache=all
  • Test steps:
    1. dd a 512MB file with urandomdata
    2. zpool scrub
    3. dd the 512MB file, to /dev/null

System Tweaks

ZPOOL common options

	-o ashift=15
	-o feature@large_dnode=enabled

	-O dnodesize=auto
	-O xattr=sa
	-O atime=off
	-O compression=off

Network Stack tuning

# Default Socket Receive Buffer
sysctl net.core.rmem_default=262144

# Maximum Socket Receive Buffer
sysctl net.core.rmem_max=33554432

# Default Socket Send Buffer
sysctl net.core.wmem_default=262144

# Maximum Socket Send Buffer
sysctl net.core.wmem_max=33554432

# Increase the maximum amount of option memory buffers
sysctl net.core.optmem_max=268435456

# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
sysctl net.ipv4.udp_mem="65536 131072 262144"

# Increase the read-buffer space allocatable
sysctl net.ipv4.udp_rmem_min=33554432

# Increase the write-buffer-space allocatable
sysctl net.ipv4.udp_wmem_min=33554432

# enable SO_MAX_PACING_RATE
tc qdisc add dev eth0 root fq limit 100000 flow_limit 10000

ZFS tuning

echo 65536 > /sys/module/zfs/parameters/zfs_vdev_max_active

echo 1024 > /sys/module/zfs/parameters/zfs_vdev_sync_read_min_active
echo 1024 > /sys/module/zfs/parameters/zfs_vdev_sync_write_min_active
echo 1024 > /sys/module/zfs/parameters/zfs_vdev_async_read_min_active
echo 1024 > /sys/module/zfs/parameters/zfs_vdev_async_write_min_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_sync_read_max_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_sync_write_max_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_async_read_max_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_async_write_max_active

echo 4096 > /sys/module/zfs/parameters/zfs_vdev_initializing_min_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_rebuild_min_active
echo 4096 > /sys/module/zfs/parameters/zfs_vdev_scrub_min_active
echo 32768 > /sys/module/zfs/parameters/zfs_vdev_initializing_max_active
echo 32768 > /sys/module/zfs/parameters/zfs_vdev_rebuild_max_active
echo 32768 > /sys/module/zfs/parameters/zfs_vdev_scrub_max_active

echo 1024 > /sys/module/zfs/parameters/zfs_vdev_min_ms_count

echo 29000 > /sys/module/zfs/parameters/zfs_scrub_min_time_ms
echo 30 > /sys/module/zfs/parameters/zfs_txg_timeout

echo 33554432 > /sys/module/zfs/parameters/zfs_max_recordsize
# Email # Twitter # YouTube # IRC # © 2017-2042 CSRB #