Distributed File Sharing

Author: Cathal O'Keeffe

Some of our applications allow users to upload files to the file system, typically photographs.

These files need to be accessible on every instance of the application, not just the handler they were uploaded on.

In the SNMP Manager trap rules and MIBs are also kept on the file system and need to kept synchronised between handlers whenever they are updated.

We use a daemon called lsyncd (“el sync”) to handle this. Our lsync configurations leverage the unison file syncing tool.

Servers that are running a master or slave sync configuration need both lsyncd and unison installed. Servers that are only receiving pushes and not generating sync updates themselves only need unison installed.

We use a pair of files to configure lsyncd on our servers.

For master servers, the first file is /usr/local/conf/lsyncd/master.conf and it specifies the log file for the process, what directories to sync, and what sync script to run whenever file modifications are performed.

The sync script, located at ~/script/master_unison_sync.sh is the second file that needs to be configured. This specifies what directories need to be synced with which servers. To start the sync process just run the command lsyncd /usr/local/conf/lsyncd/master.conf. That's it!

Now we have a master server that copies out all file changes to the slave server when it sees that a file has changed. But what if a file changes on the slave server? We still want to keep files synced across all servers so something needs to happen here two. The slave servers also have two files that make up the configuration.

One is /usr/local/conf/lsyncd/slave.conf and the other is ~/script/slave_unison_sync.sh. These perform similar jobs to their equivalents on the master server but instead of pushing changes itself when the slave server sees a change has occurred in one of the sync folders, it sends a message to the master server to start a sync event. The master server then pulls all changes from the slave server and distributes them to any other slaves. So whether the files were changed on the master or slave server it's still the master server that's running the heavy lifting.

While not really a server that's been configured. It's worth emphasising that servers do not need to have the above slave configuration running in order to receive updates from master. In this case they will receive updates from the master when a sync event is happening, but updating the files on the slave itself will not cause the master to pull the changes.

Note: This master/ slave setup is subject to change when master swaps are performed. To check if you are on the master type the commands: show master status; or show slave status; in the database.

Master - extapps1

• Syncs ~/appfiles to extslb, extapps2 and extems
• Syncs ~/appfiles/SnmpManagerFiles to extdb2

Slave - extapps2

• Syncs ~/appfiles with extapps1

Slave - extlb

• Syncs ~/appfiles with extapps1

Slave - extems

• Syncs ~/appfiles with extapps1

1. SSH to one of our QA app handler servers.
2. Locate the appfiles directory and cd into it.
3. Create a small text file in this directory named after yourself - e.g. “john.txt” or something.
4. SSH to the other app handler and verify that a copy of the file now exists on it (if it's not there straight away wait a minute).
5. Modify this copy of the file.
6. Go back to the first app handler and verify that the change is pushed back to the original file after a minute.
7. To verify that you've completed this just do an 'ls -l' on the directory with the file in it and take a screenshot of your terminal window.
8. Delete the file when you're done.