Wiki Navigation

GFS on NDAS

The most inexpensive GFS solution in the market!!

Goals

Multiple systems can write the data simultaneously on one file system on NDAS device
It is seamless to join or leave the clustering to access the GFS file system

How to use GFS on NDAS

1. System Requirements

Two Linux systems Fedore Core 4 or Fedora Core 5 installed
NDAS device version 1.1 or above

2. Set up the system

Install Fedore Core 4/5 on the each system (one, two)

Install GFS module

   yum install -y GFS GFS-kernel magma-plugins fence dlm gulm cman

For smp kernel, execute the following. (same way for xen0, xenU, kdump kernels)

   yum install -y GFS GFS-kernel-smp magma-plugins fence dlm dlm-kernel-smp gulm cman cman-kernel-smp

Note: ccs and perl will be installed to satisfy the dependency.

Disable the firewall or disable the firewall to/from the following ports
```
UDP 6809
TCP 50008
```
the above can be done by adding the following to the "Other ports:" field of the "system-config-securitylevel" tool
```
6809:udp, 50008:tcp
```

3. Set up the cluster

Edit /etc/cluster/cluster.conf on both system Two node only configuration

<cluster name="example" config_version="1">
 <cman two_node="1" expected_votes="1">
 </cman>
 <clusternodes>
  <clusternode name="one" votes="1">
   <fence>
    <method name="single">
     <device name="human" ipaddr="192.168.2.1"/>
    </method>
   </fence>
  </clusternode>
  <clusternode name="two" votes="1">
   <fence>
    <method name="single">
     <device name="human" ipaddr="192.168.2.2"/>
    </method>
   </fence>
  </clusternode>
 </clusternodes>
 <fencedevices>
  <fencedevice name="human" agent="fence_manual"/>
 </fencedevices> 
</cluster>

Three or more nodes configuration

<cluster name="example" config_version="1">
         <clusternodes>
           <clusternode name="one" votes="1">
             <fence>
               <method name="single">
                 <device name="human" ipaddr="192.168.2.1"/>
               </method>
             </fence>
           </clusternode>
           <clusternode name="two" votes="1">
             <fence>
               <method name="single">
                 <device name="human" ipaddr="192.168.2.2"/>
               </method>
             </fence>
           </clusternode>
           <clusternode name="three" votes="1">
             <fence>
               <method name="single">
                 <device name="human" ipaddr="192.168.2.3"/>
               </method>
             </fence>
           </clusternode>         
         </clusternodes>

         <fencedevices>
           <fencedevice name="human" agent="fence_manual"/>
         </fencedevices> 
</cluster>

Punch the udp port 6809 and tcp port 50008 for cman to communicate each others.
Add the following the /etc/sysconfig/iptables (see above for how to do this with the "system-config-securitylevel" tool):
```
-A RH-Firewall-1-INPUT -p udp -m udp --dport 6809 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 50008 -j ACCEPT
```
and restart the iptables
```
# /sbin/service iptables restart
```

Start ccd on both

 one# /sbin/service ccsd start
 two# /sbin/service ccsd start

Insert dlm kernel modules

 one# modprobe lock_dlm
 two# modprobe lock_dlm

Join cluster on both

 one# cman_tool join -w
 two# cman_tool join -w

Verify the cluster working
```
 one# cat /proc/cluster/nodes
 two# cat /proc/cluster/nodes
```
you should see two nodes on both systems.

Join fence domain on both

 one# fence_tool join -w
 two# fence_tool join -w

4. Install NDAS driver

5. Using GFS

Note NDAS driver is available at http://code.ximeta.com

Enable NDAS disk as a 'share' mode

 one# ndasadmin register xxxx-xxxx-xxxx-xxxx-oooo --name your_netdisk
 two# ndasadmin register xxxx-xxxx-xxxx-xxxx-oooo --name your_netdisk
 one# ndasadmin enable -s 1 -o s
 two# ndasadmin enable -s 1 -o s

Format GFS

 one# gfs_mkfs -j 2 -t example:my_lock -p lock_dlm /dev/nda1

Mount GFS

 one# modprobe gfs
 one# mount -t gfs /dev/nda1 /mnt
 two# modprobe gfs
 two# mount -t gfs /dev/nda1 /mnt

Bugs

On simuletanous write, a host died with holding the lock. and the other can't access the NDAS device after that. Will be fixed milestone:0.9.7 Fence_manual does not guarantee data integrity, you might find the above bug goes away when the "dead" host gets fenced correctly. For a real cluster setup you will want some kind of real fencing.