Multi-Tenant Subvolumes

This guide details how to provision isolated storage for 400 users, each with a strict 10GB quota, using CephFS Subvolumes.

1. Strategy: Subvolumes

Instead of manual directory management, we use the Native Subvolume abstraction.

• Isolation: Each subvolume is managed like a separate volume.
• Quotas: Enforced at the subvolume level.
• Security: Unique cryptographic keys for each user.

2. Create Subvolume Group

Create a logical group to organize student volumes.

# Run on the Admin Node
ceph fs subvolumegroup create cephfs students

# Verify with
ceph fs subvolumegroup ls cephfs

3. Automated Provisioning

3.1 Simple Sequential Batch

Basic Batch

Use this script to create 400 subvolumes (student_001 to student_400) with a fixed quota.

View create_students.sh

#!/bin/bash
# create_students.sh
FS_NAME="cephfs"
GROUP_NAME="students"
USER_COUNT=400
QUOTA_SIZE=$((10 * 1024 * 1024 * 1024))
OUTPUT_FILE="student_credentials.csv"

echo "Username,Subvolume,MountPath,SecretKey" > $OUTPUT_FILE

for i in $(seq -f "%03g" 1 $USER_COUNT); do
    USER_ID="student_$i"
    # 1. Create subvolume
    ceph fs subvolume create $FS_NAME $USER_ID \
        --group_name $GROUP_NAME \
        --size $QUOTA_SIZE > /dev/null 2>&1

    # 2. Authorize client
    ceph fs subvolume authorize $FS_NAME $USER_ID $USER_ID --group_name $GROUP_NAME > /dev/null 2>&1

    # 3. Fetch key and path
    KEY=$(ceph auth get-key client.$USER_ID)
    PATH_ADDR=$(ceph fs subvolume getpath $FS_NAME $USER_ID --group_name $GROUP_NAME)

    echo "$USER_ID,$USER_ID,$PATH_ADDR,$KEY" >> $OUTPUT_FILE
done

3.2 NIM-Based Provisioning

Advanced Batching

This variant allows batching students by Angkatan (Year) and NIM (Student ID) with customizable counts for each batch.

View create_students_nim.sh

#!/bin/bash
# create_students_nim.sh
# Provision students based on Angkatan (Year) and NIM (Student ID)

FS_NAME="cephfs"
GROUP_NAME="students"
QUOTA_SIZE=$((5 * 1024 * 1024 * 1024)) # 5GB

# Ensure output directories exist relative to where the script is executed
mkdir -p credentials keyrings scripts/windows scripts/linux

# Configuration: "Year:Count" pairs
# Example: 22:150 means Year 2022 with 150 students
BATCH_CONFIG=(
    "22:5"
    "23:5"
)

# Handle flags
DELETE_MODE=false
if [[ "$1" == "--delete" ]]; then
    DELETE_MODE=true
    echo "Running in DELETE MODE: removing specified batches..."
else
    echo "Running in CREATE MODE: provisioning specified batches..."
fi

for config in "${BATCH_CONFIG[@]}"; do
    YEAR="${config%%:*}"
    COUNT="${config##*:}"

    if [ "$DELETE_MODE" = true ]; then
        echo "Deleting $COUNT students for Angkatan 20$YEAR..."
    else
        echo "Creating $COUNT students for Angkatan 20$YEAR..."
    fi

    for i in $(seq -f "%03g" 1 $COUNT); do
        # NIM format: YY106050XXX
        NIM="${YEAR}106050${i}"

        if [ "$DELETE_MODE" = true ]; then
            # 1. Remove subvolume
            ceph fs subvolume rm $FS_NAME $NIM --group_name $GROUP_NAME > /dev/null 2>&1
            # 2. Delete auth key
            ceph auth rm client.$NIM > /dev/null 2>&1
            # 3. Clean up files
            rm -f "credentials/${NIM}_credential.csv" "keyrings/ceph.client.${NIM}.keyring" "scripts/windows/mount.${NIM}.bat" "scripts/linux/mount.${NIM}.sh"
        else
            # 1. Create subvolume
            ceph fs subvolume create $FS_NAME $NIM \
                --group_name $GROUP_NAME \
                --size $QUOTA_SIZE > /dev/null 2>&1

            # 2. Authorize client
            ceph fs subvolume authorize $FS_NAME $NIM $NIM \
                --group_name $GROUP_NAME > /dev/null 2>&1

            # 3. Fetch key and path
            KEY=$(ceph auth get-key client.$NIM)
            PATH_ADDR=$(ceph fs subvolume getpath $FS_NAME $NIM --group_name $GROUP_NAME)

            # 4. Generate Output Files
            echo "NIM,Subvolume,Path,Key" > "credentials/${NIM}_credential.csv"
            echo "$NIM,$NIM,$PATH_ADDR,$KEY" >> "credentials/${NIM}_credential.csv"

            printf "[client.%s]\n\tkey = %s\n" "$NIM" "$KEY" > "keyrings/ceph.client.${NIM}.keyring"

            # Windows Mount Script
            cat <<EOF > "scripts/windows/mount.${NIM}.bat"
ceph-dokan.exe -l z -n client.${NIM} -c C:\\ProgramData\\ceph\\ceph.conf -k C:\\ProgramData\\ceph\\ceph.client.${NIM}.keyring --root-path ${PATH_ADDR} --win-vol-name ${NIM}
EOF

            # Linux Mount Script
            cat <<EOF > "scripts/linux/mount.${NIM}.sh"
#!/bin/bash
# Linux ceph-fuse mount script for student ${NIM}

MOUNT_POINT="/mnt/${NIM}"
CONF="/etc/ceph/ceph.conf"
KEYRING="/etc/ceph/ceph.client.${NIM}.keyring"
CLIENT_ID="client.${NIM}"
ROOT_PATH="${PATH_ADDR}"

if [ "\$EUID" -ne 0 ]; then echo "[ERROR] Please run with sudo: sudo ./mount.${NIM}.sh"; exit 1; fi
if ! command -v ceph-fuse &> /dev/null; then echo "[*] Installing ceph-fuse..."; apt update && apt install ceph-fuse -y; fi

mkdir -p "\$MOUNT_POINT"
echo "[*] Mounting CephFS subvolume to \$MOUNT_POINT..."
ceph-fuse -n "\$CLIENT_ID" -c "\$CONF" -k "\$KEYRING" -r "\$ROOT_PATH" "\$MOUNT_POINT"

if [ \$? -eq 0 ]; then
    echo "[OK] Successfully mounted."
else
    echo "[ERROR] Mount failed."
fi
EOF
            chmod +x "scripts/linux/mount.${NIM}.sh"
        fi
    done
done

if [ "$DELETE_MODE" = true ]; then
    echo "Cleanup complete. All subvolumes, credentials, and scripts removed."
else
    echo "Done! Credentials, keyrings, and mount scripts saved in their respective directories."
fi

4. Cleanup

Destructive Script

This script permanently deletes all 400 subvolumes and their associated authentication keys.

View cleanup_students.sh

#!/bin/bash
# cleanup_students.sh
for i in $(seq -f "%03g" 1 400); do
    USER_ID="student_$i"
    # Remove volume and key
    ceph fs subvolume rm cephfs $USER_ID --group_name students > /dev/null 2>&1
    ceph auth rm client.$USER_ID > /dev/null 2>&1
done

5. Mounting (Client Side)

Preparation: Client Keyring

Before mounting, ensure the client has the appropriate keyring file. The provisioning scripts automatically generate these in the keyrings/ directory.

# Example for student NIM: 22106050001
# Copy the generated keyring to the standard location
sudo cp keyrings/ceph.client.22106050001.keyring /etc/ceph/

# Or create a secret file for kernel mount (more secure than passing 'secret' directly)
echo "<SECRET_KEY>" > student.secret
chmod 600 student.secret

Kernel Mount

Use the secretfile option for better security.

sudo mount -t ceph <MON_IP>:<PATH_FROM_CSV> /mnt/storage \
  -o name=<USER_ID>,secretfile=student.secret

FUSE Mount (Non-Root)

User Space

Students can use ceph-fuse to mount their data without requiring root privileges.

ceph-fuse -n client.student_001 --client_mountpoint=<PATH> /home/student/data

6. Management

Check Quota Usage

ceph fs subvolume info cephfs student_001 --group_name students

Resize Quota

ceph fs subvolume resize cephfs student_001 --size 20G --group_name students