Propolice Performance in OpenBSD
by francisco roque

1. Table of Contents
   1. Table of Contents
   2. Introduction
   3. Steps
   4. Results
   5. Problems & Inaccuracies
   6. Conclusion
   7. References
   8. Author, Copyright, License

    

2. Introduction

   Stack protection, via Hiroaki Etoh's Propolice, was added to OpenBSD 3.3 userland amidst much rejoicing, despite the performance loss. Soon afterwards stack protection was also enabled in OpenBSD kernel and people questioned how much more performance loss would occur. This document attempts to answer how much of a performance difference people will see with a propolice-enabled kernel.

    

3. Steps

   The following steps were performed to setup and run the tests:
   

   • OpenBSD 3.1-release, 3.2-release, 3.3-current (20030508 snapshot), and 3.3-current (same snapshot) with a propolice enabled kernel (from 20030513 cvs) were installed on two machines: an Athlon XP and a Pentium 166. The same filesystem layout was used for each test (fs's were newfs'd for each version, but the disklabel remained the same). dmesg: athlon, pentium.

   • The Bytebench port was the only port added. At the time, the port was for version 3.11 on all systems.

   • In all cases, softdep was added to each partition.

   • Each run was performed in single user mode with no networking running.
     bytebench was run with no options.

    

4. Results

   OpenBSD 3.1 and 3.2 do not use propolice. The OpenBSD 3.3 snapshot did not have a propolice kernel, but all other binaries were propolice enabled. The final run includes a propolice-enabled kernel.

   The difference between 3.1 and 3.2 is included in order to show any general trend between releases. Though more than two datapoints are necessary to really determine any general trend, these show that performance changes between 3.2 and 3.3 cannot be solely attributed to propolice.

   
   larger version

   
   larger version

   These graphs show the difference on bytebench tests across OS versions. 3.1 results are given a value of 1 and other OS versions are graphed in relation to that.

   Most test results show under 10% overall performance degradation. Two tests stand out from these results: filesystem and execl. The improvement in filesystem performance for the Athlon is most likely due to code changes between 3.2 and 3.3, apparently something affecting systems with more memory (did someone enable UBC when i blinked?). I don't know about the execl change, but it is important to note that this performance has been steadily decreasing and the decrease between 3.3-current with and without propolice enabled kernel is only 0.7% on the athlon and 1.2% on the pentium.

   More noteworthy than the overall changes are the changes between 3.2-release and 3.3-current and between current with and without propolice.
   The overall difference between 3.2 and 3.3 (sans propolice kernel) is 4.7% on the athlon and 9.3% on the pentium.
   The overall difference between current with and without propolice kernels is 2% on the athlon and 0.7% on the pentium.

   This table has all the results from the bytebench runs.

    

5. Problems & Inaccuracies

   There are various problems and inaccuracies with the process and these results that may lead to incorrect decisions for other applications.

   • Network/firewall was not tested.
     It would be nice to see the performance hit on networking of a propolice-enabled firewall, but to do this testing right would take a lot of time and thought.

   • Lots of code changes between releases.
     The benchmarks were run between 3.1 and 3.3 in order to determine what sort of performance change is to be expected between releases in order to not blame everything on the propolice-enabled system. However, since so much changes between releases, it is difficult to determine what propolice's impact is. It would have been better to compile 3.3-current with propolice disabled, but this would have taken more time. Additionally, i feel that since most people will install default releases, these tests better show real-world expectations and changes.

   • I'm trying to remember why i didn't do 3.3-release and i can't figure it out.

   • I'll think of more (and be told of more) problems as time passes, and those will appear here.
     

    

6. Conclusion

    

7. References

   Other howto's and docs:
   Propolice webpage - technical info and more about propolice.
   

    

8. Author, Copyright, License

   The author of this document is francisco roque, who can be found at
   http://www.blackant.net/

   This document is copyright francisco roque 2002.

   This document is released under BSD style license and may be used in whole or part for anything so long as this section remains intact.

   The author takes no responsibility for damages incurred by following the steps in this document, including but not limited to the loss of data or damage to hardware. Good Luck.