Ceph is a distributed Object Storage and file system, it’s a distinctive piece of software in its own category for two reasons:
- It is developed in C++ (thus, offering speed and device-level integration)
- It provides a shared network block device driver offering POSIX access to Linux (the Kernel module is the subject of this post).
The latter is included in Kernel versions > 2.6.37, for those, like me, who are a bit “left behind” on Ubuntu Stable releases, the module can be installed thanks to the backport support on Ceph’s Kernel Module distribution (Git based); exhaustive documentation can be found on Ceph’s Wiki.
Recently, Ceph Messenger workload management has changed as it now adopts the cleaner Concurrency-managed workqueue subsystem; the reason for this is to avoid deadlocks on multi-processors systems. A vanilla Lucid Lynx’s kernel does not implements Concurrency-managed workqueues, compiling the client Kernel module returns this error:
#make -C libceph
make: Entering directory `/home/gabriele/ceph-client-standalone/libceph'
make -C /lib/modules/2.6.32-24-server/build M=/home/gabriele/ceph-client-standalone/libceph CONFIG_CEPH_LIB=m modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.32-24-server'
CC [M] /home/gabriele/ceph-client-standalone/libceph/ceph_common.o
CC [M] /home/gabriele/ceph-client-standalone/libceph/messenger.o
/home/gabriele/ceph-client-standalone/libceph/messenger.c: In function ‘ceph_msgr_init’:
/home/gabriele/ceph-client-standalone/libceph/messenger.c:99: error: implicit declaration of function ‘alloc_workqueue’
/home/gabriele/ceph-client-standalone/libceph/messenger.c:99: error: ‘WQ_NON_REENTRANT’ undeclared (first use in this function)
/home/gabriele/ceph-client-standalone/libceph/messenger.c:99: error: (Each undeclared identifier is reported only once
/home/gabriele/ceph-client-standalone/libceph/messenger.c:99: error: for each function it appears in.)
make[2]: *** [/home/gabriele/ceph-client-standalone/libceph/messenger.o] Error 1
make[1]: *** [_module_/home/gabriele/ceph-client-standalone/libceph] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-2.6.32-24-server'
make: *** [all] Error 2
make: Leaving directory `/home/gabriele/ceph-client-standalone/libceph'
The fastest workaround is to, actually, revert the patch that introduced non-reentrancy. A quick check on the mailing list confirmed this, thanks a lot to Henry Chang.
A quick check on branch’s commits points to the following two entries (extract):
#git show-branch --more=20
…..
+* [master~10] net/ceph: make ceph_msgr_wq non-reentrant
+* [master~11] ceph: fsc->*_wq's aren't used in memory reclaim path
….
Now, it’s just a matter to revert the commits (your commit references might vary):
#git revert -n master~10 #git revert -n master~11
The rest of the installation is straightforward:
#make -C libceph #cp libceph/Module.symvers ceph/ #make -C ceph #make -C libceph modules_install (depmod is included) #make -C ceph modules_install (depmod is included) #modprobe libceph #modprobe ceph
Ceph’s client is ready.
Ceph is still on development stage, the authors are very vocal about it’s non-readiness for production, however the current release goes in the good direction and, judging from Ceph’s roadmap, it is evident that the team is busy implementing stability and continuity tool (e.g. fsck and disaster recovery tools). This seems to me a good sign that Ceph might reach 1.0 before Q3 2011.
