Ceph Cluster Guide

Appearance

Migrating Subvolumes (EC to Replicated) ​

Ceph does not allow you to change the underlying CRUSH rule (e.g., from Erasure Coded to 3x Replication) of a populated storage pool on the fly. Doing so would instantly corrupt the mapping of where data chunks are physically stored.

However, you do not need to start over or wipe the cluster. CephFS supports File Layouts, which allows a single filesystem to span multiple pools. You can create a new Replicated pool, attach it to your existing CephFS, and explicitly tell Ceph to store a specific subvolume's data in the new Replicated pool instead of the old EC pool.

This guide details the non-disruptive migration process.

1. Create the New Replicated Pool ​

First, create the new data pool with the desired 3x Replication strategy.

bash
# 1. Create the new pool (replace 32 with your target PG count)
sudo ceph osd pool create cephfs_data_rep 32 32 replicated

# 2. Enforce 3x Replication on the new pool
sudo ceph osd pool set cephfs_data_rep size 3
sudo ceph osd pool set cephfs_data_rep min_size 2

# 3. Prevent this pool from being used for generic RADOS blocks
sudo ceph osd pool application enable cephfs_data_rep cephfs

2. Attach the New Pool to CephFS ​

Tell your existing CephFS that it is allowed to use this new pool for data storage.

bash
sudo ceph fs add_data_pool cephfs cephfs_data_rep

# Verify the pool is attached (you should see multiple data pools listed)
sudo ceph fs ls

3. Create the Target Subvolume ​

We need to create a new subvolume for the user where the migrated data will go.

bash
# Create a new staging subvolume for the student
sudo ceph fs subvolume create cephfs 22106050001_rep --group_name students

# Get the UUID paths for BOTH the old and new subvolumes
sudo ceph fs subvolume getpath cephfs 22106050001 --group_name students
sudo ceph fs subvolume getpath cephfs 22106050001_rep --group_name students

(Note down the UUID paths returned by the commands above).

4. Mount the Filesystem & Set the Pool Layout ​

To route data to the new pool, we must modify the extended attributes (xattr) of the new subvolume's directory. This requires mounting the CephFS root.

bash
# 1. Install the 'attr' package required for modifying file layouts
sudo apt update && sudo apt install attr -y

# 2. Use our helper script to mount the root and drop into a shell
sudo ./scripts/explore-students.sh

Now, inside the [CephFS Explorer] shell, change the layout of the new directory:

bash
# 3. Set the directory layout to the new replicated pool
# (Replace UUID-NEW with the path of the newly created _rep subvolume)
setfattr -n ceph.dir.layout.pool -v cephfs_data_rep 22106050001_rep/<UUID-NEW>

# 4. Verify the layout was successfully applied
getfattr -n ceph.dir.layout.pool 22106050001_rep/<UUID-NEW>
# Expected output: ceph.dir.layout.pool="cephfs_data_rep"

WARNING

You must set the layout on an empty directory. If any files exist in the directory before you run setfattr, the command will fail.

5. Migrate the Data ​

While still inside the CephFS Explorer shell, securely copy the data from the old subvolume to the new one.

bash
# Use rsync to copy all data and preserve file permissions/timestamps
# (Replace UUID-OLD and UUID-NEW accordingly)
rsync -avH 22106050001/<UUID-OLD>/ 22106050001_rep/<UUID-NEW>/

Once the transfer is 100% complete, you can exit the shell:

bash
exit

6. Update Quotas and Clean Up ​

Now that the data is safely resting in the new 3x Replicated pool, apply the student's quota to the new subvolume and delete the old one.

bash
# 1. Apply the 5GB quota to the new Replicated subvolume (size must be in bytes: 5368709120)
sudo ceph fs subvolume resize cephfs 22106050001_rep 5368709120 --group_name students

# 2. Delete the old Erasure Coded subvolume
sudo ceph fs subvolume rm cephfs 22106050001 --group_name students

# 3. (Optional) Rename the new subvolume to drop the "_rep" suffix
# Note: Subvolume renaming is supported in Ceph Quincy (v17) and later.
sudo ceph fs subvolume rename cephfs 22106050001_rep 22106050001 --group_name students

7. Update Windows Client Mount ​

Because the UUID path of the subvolume has changed, you must update the Windows client's mount command to point to the new UUID.

powershell
# On Windows, remount using the new UUID path
ceph-dokan.exe -c C:\ProgramData\ceph\ceph.conf -n client.22106050001 --secret C:\ProgramData\ceph\student.secret -l Z -x /volumes/students/22106050001/<UUID-NEW>

The user's data is now fully migrated from Erasure Coding to 3x Replication with zero cluster downtime!