How to share the NDAS disk with OCFS2 file system

OCFS2 is a shared symmetric file system by Oracle. Unlikely OCFS, OCFS2 is for general purpose and included in 2.6.x offcial kernel tree.

We did test with Ubuntu system to use OCFS2.

Prerequisite

• One NDAS disk.
• Two systems with i386 Ubuntu 5.10 installed.

Any system that has OCFS2 kernel module/feature installed.

Step 1

Install NDAS driver for each system.
Make sure the NDAS disk is accessible.

Step 2

Install ocfs tools on both system

apt-get install ocfs2-tools ocfs2console

Step 3

Configure cluster /etc/ocfs2/cluster.conf on both system

node:
        ip_port = 7777
        ip_address = 192.168.0.100
        number = 1
        name = host01
        cluster = ocfs2

node:
        ip_port = 7777
        ip_address = 192.168.0.101
        number = 0
        name = host02
        cluster = ocfs2

cluster:
        node_count = 2
        name = ocfs2

* The file format is very strict, indent with TABs. Even then I couldn't get it to work and had to resort to ocfs2console don't forget to add the following to /etc/hosts on both system

host01 192.168.0.100
host02 192.168.0.101

Make sure port 7777 is reachable from each other (check iptables)

Step 4

Start OCFS2 service on both system

/etc/init.d/o2fs load    # depending on your system, this can also be o2cb instead of o2fs
/etc/init.d/o2fs online  # can also be o2cb

Step 5

Partition the NDAS disk and format as ocfs2 on one system.

fdisk /dev/ndas-00xxxxxx:0   # path will be given by ndasadmin enable
mkfs -t ocfs2 /dev/ndas-00xxxxxx:0p1 # p1 means the first partition

Step 6

Turn on the heat beat for the file system on both system

ocfs2_hb_ctl -S -d /dev/ndas-00xxxxxx:0p1

Step 7

Mount the file system from both systems

mkdir -p /mnt/ocfs2
mount -t ocfs2 /dev/ndas-00xxxxxx:0p1 /mnt/ocfs2

and enjoy!

Tips

The ocfs2 nodes exchanges the heartbeats each other. During the file transfer the heartbeats could be congested by the NDAS i/o transfer. This congenstion results the 40% degration of writing performance.

Therefore, we recommend the hearbeat network should be dedicated to avoid NDAS i/o tranfer congestion. or use gigibit switch/nic if your NDAS disk is 100Mbps version. (With Gigabit NDAS disk, only solution is dedicating the heartbeat network.)

Performance

512MiB DRAM 1GiB transfer, HDD cache 2MB(affecting write performance)

NDAS/ocfs2/100Mbps/heartbeat on: read 9.7 MB/s , write 6.2 MB/s
NDAS/ocfs2/100Mbps/heartbeat off: read 9.9 MB/s , write 10.4 MB/s
NDAS/ext2/100Mbps: read 10.1 MB/s , write 10.8 MB/s

Local HDD/ocfs2: read 39.8 MB/s, write 42.9 MB/s
Local HDD/ext2Ï: read 44.7 MB/s , write 42.9 MB/s

* Please note that the hearbeat network is not dedicated. that's why the write of NDAS/ocfs2 is much slower (6.2 MB/s), I believe.
** The network maxium speed of 100Mbps network is about 12.5 MB/s in theory if we ignore the overhead of packet headers metadata. so bottleneck for network test is the network speed, for local hdd is the hdd speed.