Cluster Testing (Windows & Linux Clients)

This guide outlines how to validate the resilience, performance, and quota enforcement of your Ceph cluster using Windows 10 clients, as well as how to perform administrative mounts on Linux.

1. Health & Benchmarks

Before proceeding, ensure the cluster is healthy and establish a performance baseline.

1.1 Status Check

# Run on Admin Node (Linux)
sudo ceph -s

Expected: HEALTH_OK, 3 monitors, 4 OSDs (up/in).

1.2 RADOS Benchmark

Test the raw speed of the storage backend.

# Run on Admin Node (Linux)
sudo rados bench -p cephfs_data 10 write --no-cleanup
sudo rados bench -p cephfs_data 10 seq

2. Quota Enforcement

We will simulate a user trying to exceed their 5GB limit from Windows.

Setup

1. Complete the setup in windows-client.md.
2. Mount student NIM 22106050001's subvolume to drive Z:.

The Stress Test

Try to write a 6GB file into the 5GB quota space using PowerShell.

# On Windows Client: Attempt to create a 6GB dummy file
fsutil file createnew Z:\quota_test.img 6442450944

Expected Result: Windows should reject the command and throw a specific quota error:

Error: There is not enough space on the disk.

3. User Isolation (Security)

Verify that 22106050001 cannot access 23106050001's volume.

Access Control Test

Try to mount 23106050001's path using 22106050001's secret key file on Windows.

# Attempt mount with wrong credentials
ceph-dokan.exe -c C:\ProgramData\ceph\ceph.conf -n client.22106050001 --secret C:\ProgramData\ceph\student.secret -l Y -x /volumes/students/23106050001/UUID-HERE

Expected Result: The dokan utility should crash or exit immediately with an authorization error, and the Y: drive will not mount.

4. Concurrent Access (Multiple Clients)

Test if multiple Windows computers can mount and interact with the same student subvolume simultaneously without locking or corruption.

The Setup

1. Mount 22106050001's volume on Windows PC 1 (as Z:).
2. Mount the exact same volume on Windows PC 2 (as Z:).

The Test

# On Windows PC 1: Create a file
Set-Content -Path Z:\test_sync.txt -Value "Hello from PC 1"

# On Windows PC 2: Immediately read the file
Get-Content Z:\test_sync.txt

# On Windows PC 2: Append to the file
Add-Content -Path Z:\test_sync.txt -Value "Hello from PC 2"

# On Windows PC 1: Verify the changes appeared
Get-Content Z:\test_sync.txt

Expected Result: Both computers can read and write to the same subvolume simultaneously. The file changes made on PC 2 should instantly appear on PC 1, demonstrating CephFS's strong cache coherency.

5. Fault Tolerance (OSD Failure)

Verify that if an active storage drive (OSD) dies, students can still access their data seamlessly thanks to the 3x Replication strategy.

The Test

1. On Windows PC 1, start a continuous script to slowly write data:

# Write a sequence of numbers into a file endlessly
$i=0; while($true) { $i | Out-File Z:\survive.txt -Append; Start-Sleep -Milliseconds 100; $i++ }

2. While the script is writing, open an SSH terminal on your Admin Node (Linux) and forcefully kill an OSD daemon:

# Find a running OSD
sudo ceph osd tree

# Violently stop the container daemon for OSD 0
sudo ceph orch daemon stop osd.0

3. Return to Windows PC 1.

Expected Result: The PowerShell loop on the Windows client should continue writing numbers to Z:\survive.txt without hanging or throwing an I/O exception.

Remember to start the OSD back up when finished:

# On Admin Node
sudo ceph orch daemon start osd.0

6. Overprovisioning (Thin Provisioning)

Verify that an administrator can assign quotas that total more than the physical raw capacity of the cluster.

The Test

Attempt to create a student subvolume with a 50 Terabyte quota on an 8 Terabyte cluster.

# Run on the Admin Node (Linux)
ceph fs subvolume create cephfs overprovision_test --group_name students --size 50T

Expected Result: The command succeeds silently. You can verify it by mounting the test volume on Windows and checking the drive properties; Windows will report it as a 50 TB network drive.

7. Linux Client Mount (Admin Exploration)

While students use Windows to access their isolated subvolumes, administrators can mount the entire CephFS root natively on Linux to explore all data without needing containerized shells.

7.1 The Easy Way (Running on the Admin Node)

If you are already logged into your primary admin node (e.g., ceph-mgr-1), the easiest way to explore files is using our provided helper script. It handles secret extraction automatically and drops you into a temporary shell.

# Run the interactive explorer script from the project root
sudo ./scripts/explore-students.sh

(When you type exit, it safely unmounts automatically).

7.2 Manual Mount (From a Remote Linux PC)

If you want to mount the cluster from a completely different Linux computer (e.g., your personal laptop), you cannot immediately run Ceph commands. You must first copy the authentication keys.

Step 1: Install prerequisites and copy credentials On your remote Linux PC, install the Ceph tools and securely copy the keys from your admin node (ceph-mgr-1):

# On your remote PC: Install client tools
sudo apt update && sudo apt install ceph-common -y

# On your remote PC: Create ceph config directory
sudo mkdir -p /etc/ceph

# Securely copy the config and admin keyring from ceph-mgr-1
# (Replace 'deffa' and '192.168.202.221' with your admin node's details)
scp deffa@192.168.202.221:/etc/ceph/ceph.conf /tmp/ceph.conf
scp deffa@192.168.202.221:/etc/ceph/ceph.client.admin.keyring /tmp/ceph.client.admin.keyring

sudo mv /tmp/ceph.conf /etc/ceph/
sudo mv /tmp/ceph.client.admin.keyring /etc/ceph/

Step 2: Choose a Mount Method Now that your remote PC is authenticated, you can mount CephFS using either the Kernel module or FUSE.

Option A: Native Kernel Mount (Faster)

The Linux kernel module requires a raw password string, so we must extract it from the keyring we just copied:

# 1. Extract raw secret string from the keyring
sudo ceph auth get-key client.admin > /tmp/admin.secret
sudo chmod 600 /tmp/admin.secret

# 2. Create a mount point
sudo mkdir -p /mnt/ceph-root

# 3. Mount the CephFS root (Replace IP with your MON IP)
sudo mount -t ceph 192.168.202.221:/ /mnt/ceph-root -o name=admin,secretfile=/tmp/admin.secret

# 4. Unmount when done
sudo umount /mnt/ceph-root
sudo rm -f /tmp/admin.secret

Option B: FUSE Mount (User Space)

Alternatively, ceph-fuse natively understands the .keyring file we copied in Step 1, so no secret extraction is needed.

# 1. Install the Ceph FUSE client
sudo apt install ceph-fuse -y

# 2. Create a mount point
sudo mkdir -p /mnt/ceph-fuse

# 3. Mount the filesystem (it automatically reads /etc/ceph/)
sudo ceph-fuse /mnt/ceph-fuse

# 4. Unmount when done
sudo umount /mnt/ceph-fuse