Encrypted File System Performance in OpenBSD 3.2
by francisco roque

Table of Contents
Introduction
Being curious as to the performance costs of using an encrypted fs on my old 486 laptop, i decided to run some benchmarks comparing direct disk filesystems to encrypted file systems.
Steps
The following steps were performed to setup and run the tests:
- OpenBSD 3.2-release was installed on an old pentium. All the default install sets were installed. Nothing was changed from a default install with the exception of modifying various partitions for /home, which is the partition that the tests were run on. Yes this means that softupdates were not enabled.
  For the curious, here is a dmesg of the test machine.
- The Iozone package was the only package added. At the time, the package was for version 3.83.
- An additional kernel was compiled using -release source with the addition of Ted Unangst's cgd port (revision 1). This kernel is GENERIC with the addition of cgd, and was only used during the cgd tests (an oversight; probably should have used the same kernel throughout, just in case).
- Iozone filesystems tests were run on 4 filesystems implementations on the same hardware: one run to fs right on disk, one run to unencrypted vnode, one run to encrypted vnode, one run to cgd using blowfish. During each test, the same physical partition was used, but formatted in different ways. These ways were:
  1. fs directly on disk. The following commands were used to create the fs:
```
newfs /dev/rwd0g
mount /dev/wd0g /home
# edit /etc/fstab to include /home
```
  2. unencrypted vnode. The following commands were used to create the fs:
```
newfs /dev/rwd0g
mount /dev/wd0g /mnt
dd of=/mnt/vnd if=/dev/zero count=900 bs=1048576
vnconfig -c svnd0 /mnt/vnd
newfs /dev/svnd0c
mount /dev/svnd0c /home
# edit /etc/fstab to include /mnt and /home
```
  3. encrypted vnode. The same commands as for unencrypted vnode were used, except that vnconfig -ck was run instead and a password was supplied.
  4. cgd. Ted Unangst's steps were followed. /dev/wd0g was the device used (same dev as the other steps used).
  The same password was used for all the encrypted file systems.
- After the partition was created and before each test, the machine was rebooted.
- Each run was performed in single user mode with no networking running.
  Iozone was run as the following:
```
/usr/local/bin/iozone -g256m -Ra -U/home -f/home/file > run.$$.out 2> run.$$.err
```
  This runs all the read and write tests for filesizes from 64KB to 256MB, unmounting /home between each test.
  A tail -f run.$$.out was run at the same time.
- After the first 4 tests, the processor was changed to a P-100 and the same steps were used to repeat the 4 tests.
Results
Some of the results were surprising, some weren't. The graph below shows the average scores for each run for each device used.
As expected, direct disk device performed very well. However, I did not expect the unencrypted vnd device to outperform the direct disk device so well for read tests. Closer inspection of the graphs for read tests shows that unencrypted vnd's really outperform direct disk only with lower filesizes (under 4MB).
Encrypted devices perform well below the unencrypted devices, usually 2-3 times slower. The encrypted vnd device performed better than the cgd device for reads, but the cgd device performed better in writes. Closer inspection of the runs shows that for very small file sizes, encrypted vnd works much better than cgd in read tests, while cgd remains more constant throughout filesizes for write tests.

P-200 summary: large P-100 summary: small, large
From these results, we can determine what device type will best suit what application. If your application requires more read operations than write, then encrypted vnd is the way to go. If your application requires more write operations than read, then cgd should deliver better performance.
Also, it is worth noting that if you happen to have small files and happen to be performing many read operations, you may want to consider unencrypted vnd devices over direct disk devices.
Here are large graphs of all the tests run on all the filesystems.
P-100:
read, reread, random read, backward read, stride read, fread, refread, write, rewrite, random write, record rewrite, fwrite, refwrite
P-200:
read, reread, random read, backward read, stride read, fread, refread, write, rewrite, random write, record rewrite, fwrite, refwrite
cgd/vnd comparison on P-100/200:
read, reread, random read, backward read, stride read, fread, refread, write, rewrite, random write, record rewrite, fwrite, refwrite
Problems & Inaccuracies
There are various problems and inaccuracies with the process and these results that may lead to incorrect decisions for other applications.
- Softupdates were not enabled.
  The default mount options were used. I did one run of a straight disk device on P-100 with softupdates enabled, but the results were so negligible that i decided not to go any further.
- The hardware used may not map properly to hardware currently purchased for production machines.
  As stated in the introduction, my main goal was to estimate what kind of performance hit my laptop, a 486/33, would take. Current hardware may make the performance impact of encryption more negligible. Then again, these runs may display how a top-of-the-line server would react under load. If the increase of performance from faster processors continues to follow the same pattern as it did from P-100 to P-200 (almost a doubling in performance), then a P-1100 should perform fast enough to make disk I/O the bottleneck.
- Encrypted vnd devices have a longer history in OpenBSD than cgd devices.
  cgd devices are not in the default kernel, or (as of this writing) even in the -current tree. Its current status should be considered when decided what implementation to use. That said, i did not experience any difficulties with the cgd device during my tests.
- Not all implementations of encrypted file systems were tested.
  There are two other well-known encrypted filesystems available for OpenBSD: tcfs and cfs. I tried to implement tcfs but encountered difficulties in the form of a kernel panic upon umount'ing the filesystem. I did not try cfs as this would require more processes to run, and on my target platform this was quite undesirable. Additionally, cfs is marked as unsupported by its author.
- Not all forms of encryption with cgd were tested.
  cgd has the possibility of accepting both AES and blowfish as encryption algorithms. I did not try compiling with AES. I tried using blowfish with 448 bit keys enabled, but this yielded results very similar to the default settings.
Conclusion
In terms of performance, both cgd and vnd have their place. Since vnd performs so much better in reads for very small files, i feel that overall cgd devices are better. However, i am biased as i feel that, once compiled in, the setup and configuration of a cgd device is easier than a vnd device. It should also be noted that were cgd devices ported to use OpenBSD's crypt subsystem instead of a port of the NetBSD system, then cgd might also take advantage of hardware based crypto accelerators. Though this may not help my laptop, it might certainly help a server with encrypted filesystem needs.
The decision as to whether a cgd or vnd device is better for your needs will really depend on those specific needs; hopefully this document will help you decide.
References
Manpages:
vnd(4)

Other howto's and docs:
the cgd on OpenBSD page by Ted Unangst, the original porter of cgd.
TCFS the Transparent Cryptographic File System. The source shipped with OpenBSD 3.2 panic'd my kernel. Here are a couple pages about using TCFS on OpenBSD: 1, 2.
CFS an encrypted file system implemented via an NFS interface.
IOzone Filesystem Benchmark - very useful tool for measuring disk I/O.
Author, Copyright, License
The author of this document is francisco roque, who can be found at
http://www.blackant.net/
This document is copyright francisco roque 2002.
This document is released under BSD style license and may be used in whole or part for anything so long as this section remains intact.
The author takes no responsibility for damages incurred by following the steps in this document, including but not limited to the loss of data or damage to hardware. Good Luck.