Mirroring Files in Different Places - Links, Bind Mounts, and BindFS

Introduction to the Problem

Spoiler alert: this blog post is uber-geeky. For those of you not wanting to geek out today, I'm sorry :)

Here at RedBottle, we recently had to come up with a new solution to a problem we've had to solve before: how do you make the same files appear in two different places on a Linux server?

Potential Solutions

Traditionally, there have been four ways to solve this problem:

1. Symbolic Links

This is usually every sysadmin's best friend – you use the ln -s command to put a placeholder at one place in the file system that points somewhere else, and the rest of the system happily pretends that the same files and folders appear at two different places.

In most cases, this works well, and this was the solution we went with in the past. Unfortunately, in our new environment, the files we want to mirror are in two different namespaces.

The short version is that symlinks only work for mirroring something from point A to point B when a user has access to both points A and B. In our case, when users log-in, they should only see their own files and specific, system-wide files (programs, config files, etc), but not other users' files. Meanwhile, system services like Aegir, a Drupal hosting tool, need to work with files from all users. In order to accomplish this, we use what is often referred to as a "chroot jail" for each user, and then mirror the files we want them to see inside the jail. Therefore, symbolic links simply won't work for this purpose because they require each user to have access to the locations the symbolic links would point to, which are outside the jail.

2. Hard Links

Although the same ln program that is used to create symbolic links is also used to create hard links, hard links tend to be much more rarely used in Linux. This is mainly because hard links can only be used for files, are not supported by all filesystems, and both the link and the file it references must be on the same disk. Hard links can also make some processes like replacing files that are the target of a hard link awkward. For example, if you create file A, create hard link B that points to file A, then copy file C on top of file A, you'll end up with files A and B not containing the same data.

These drawback mean that hard links won't work for our purposes because we need to mirror folders, not just files.

3. Bind Mounts

A bind mount is an incredibly powerful, flexible tool that the Linux kernel (the core of the operating system) provides for taking one part of a mounted file system and remounting it elsewhere, the same way that you would mount a disk. To do it, you run the mount command, which typically requires superuser privileges, and you pass it the --bind option, along with the source and target parameters. Then, poof!, magic happens and the source is mirrored at the target.

Unfortunately, a bind mount is a little too magical – the mirror is so convincing that common tools like mv and rm don't realize that it is anything special. This is especially dangerous if you recursively delete a folder that contains the mount point for a bind mount, because the deletion will recurse into (i.e. carry into) the source. Basically, if you have folder A that you mount inside folder B, and then you delete folder B without un-mounting folder A first, you will wind up deleting everything inside both folders A and B (except for the mount point for A inside B, since you will get an error that the resource is in use.). If you're the root user and accidentally delete a folder that contains a mount without thinking, this could be very bad.

What's worse is that you can't mount something that you have read/write access to as read-only with a bind mount, to safeguard against things like accidental deletion.

There are a few other exotic features of bind mounts that are particularly useful for servers with multiple users, but they are outside the scope of this blog article. For more information about them, see the article entitled "Shared Subtrees" from LWN.net.

Unfortunately, we're not able to use bind mounts exclusively as our solution because we use automated tools that could very easily delete folders containing mounts. If only there was something more flexible...

4. BindFS Mounts

Enter BindFS, a File System in User Space (FUSE) version of bind mounts with a lot more flexibility. Some of its key features are the ability to mount things read-only, adjust the user and group ownership information of the mount, adjust permissions, and more. Basically, it's the swiss army knife for bind mounting.

Of course, as with all things, there are tradeoffs, and BindFS does have its own – resources and performance. Since bind mounts are implemented in the kernel, they are very fast and their data structures are compact, so they use only a small amount of internal resources like kernel memory. BindFS, on the other hand, is implemented in user-space, which means that it runs like any other program that users can run normally. To accomplish this, each time you mount a resource, a new program is spawned that stays in memory until the resource is unmounted. That also means that the kernel has to use the same resources to support each mount that it would to support an entire program that's running, rather than using the miniscule amount of internal resources it would use for a bind mount. In addition, because FUSE is in user space, rather than inside the kernel, access to files on a BindFS mount go through more interfaces and layers of the kernel before being handled. Both of these factors result in BindFS not being nearly as efficient as bind mounts, but also allow it to be more flexible.

Evaluating How the Solutions Perform

Since we're discussing the performance of BindFS, you might be wondering just exactly how much slower it is than bind mounts, or any of the other options I described. Luckily, I've done the homework there for you using a nifty tool called Iozone.

Test Results

What follows are the results of running tests with Iozone.

For the test set-up, see the next few sections. A full copy of the result data from the test is available as a workbook in Excel 2010 format.

Overall Results

Overall, most of the results were expected. BindFS comes out as the slowest of the methods, with nearly an 76% drop in file access performance compared to directly accessing files. As can also be expected, mounting a BindFS mount through another BindFS mount further decreases performance, but only by about an additional 10%, rather than having twice the impact. Using a bind mount on top of a BindFS mount had almost no impact on performance, beyond the performance loss of the BindFS mount itself, but this was expected given the lightweight nature of bind mounts.

Curiously, using bind mounts, symlinks, and symlinks through symlinks had a positive impact on performance, which makes no intuitive sense, other than to be explained as test error.

Read Tests

After reviewing the overall results, there were no big surprises in the read test results. BindFS was still the slowest option, especially on the "Random Read" test, where its performance was 82.2% slower than direct access. On the "Stride Read" test, however, BindFS performed only 70.8% slower than direct access.

The results of using bind mounts or symlinks were not statistically significant enough to make a comparison among the different tests.

Write Tests

The write tests were also not terribly surprising. Here, BindFS came in at 86.9% slower on the "Record Rewrite" test than direct access, but scored only 64.9% slower than direct access for the "Writer" test.

The results for the other access methods showed an even greater positive effect on performance than in the overall results or read test results.

Conclusions

Accessing mirrored files through a BindFS mount is, on average, about 76% slower than directly accessing the files. Writing is faster than reading, but only slightly so (about 1% difference, on average). Using bind mounts or symlinks, on the other hand, has only a marginal, if any, effect on performance. I would hesitate to say that bind mounts or symlinks would improve performance, despite the fact that the tests run here demonstrated that result. I would say that it might be worth it to investigate why this happened for these particular tests.

Putting things into perspective, 76% might sound like a lot, but depending on your server environment and use case, it might not have as much of an effect on server performance as you'd expect. With a dedicated server rather than a VPS; a different file system, such as xfs or ext4; or a higher speed NAS, you might get better performance than what we got here.

For now, on our servers, we're going with BindFS because the benefits of protecting data outweigh the performance losses, and our servers are still performing within a range we find acceptable.

Test Hardware

The following is a description of the server hardware used for the test:

• Host: MediaTemple
• Service: Dedicated Virtual (dv) 4.0 512 MB
• Processor: Intel(R) Xeon(R) CPU L5630 @ 2.13GHz
• Memory: 512 MB

Test Software

The following is a description of the server hardware used for the test:

• Linux Distro: CentOS release 5.6 (Final)
• Linux kernel version: 2.6.18-028stab092.1 #1 SMP Wed Jul 20 19:47:12 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux
• Iozone version: 3.397 (32-bit)
• BindFS version: 1.8.3-3.el5.x86_64
• FUSE version: 2.7.4-8.el5.x86_64
• File system: vzfs (Virtuozzo virtual file system; most likely ext3 on the physical hardware node)

Test Process

1. A folder called "results" was created.
2. A folder called "origin" was created.
3. A folder called "inner_origin" was created inside "origin".
4. A file called "file.txt" was created inside "origin/inner_origin" using touch. This file was not part of the test; it was merely there to make it easier to determine whether mounting was performed and had worked correctly.
5. Iozone was run with the options -a -e -f ./origin/test.file -Rab "01 - Direct Access.xls" .
6. For testing mounts, a folder called "mount" was created as the mount point for "origin".
7. The access option under test (symlink, mounts, etc) was used to mount or link "origin" as "mount". For BindFS, the options --owner= and --group= were used to alter the owner and group of the mounts.
8. Iozone was run with the options -a -e -f ./mount/test.file -Rab "RESULT_FILE.xls" .
9. For testing mounts, a folder called "inner_mount" was created as the mount point for "origin/inner_origin".
10. The access option under test was used to mount or link "mount/inner_origin" as "inner_mount". For BindFS, the options --owner= and --group= were used to alter the owner and group of the mounts.
11. Iozone was run with the options -a -e -f ./inner_mount/test.file -Rab "RESULT_FILE.xls" .