Last updated on August 7rd 2004
DRAFT VERSION
Contents
Introduction
Tuning the TCP/IP Stack
Enabling the NCA
Caching Compiled PHP Code
Tuning Apache
Resource Dependency
Benchmarking Improvements
Conclusion
Additional Reading
Introduction
Out-of-the-box, Solaris is not tuned for being a Web server. While you could certainly set up Apache (or any number of other HTTP software) and start sending out pages to the Web, performance will not be anywhere near what your machine is capable of. What's more is that Solaris allows by default certain TCP/IP operations that should be adjusted to provide both greater security and protection from DDoS attacks. This article shows you how to achieve all this with a system boot script and how to enable the Solaris NCA (Network Cache and Accelerator) that was introduced in Solaris 8. The latter has been steadily improved upon in subsequent Solaris releases and greatly enhances Web serving performance. Finally, if you serve PHP pages, a free accelerator to provide both RAM and file caching of compiled code.
Most of these changes entail modifying key system files including the modification of tunable kernel parameters and installing kernel modules, installing an init script and tweaking other files which require the cycling of system service daemons. It goes without saying that you should have root access, be careful of what you are doing and not to try this on a production box first time around. Keep a backup copy of configuration files that you modify so you can revert in an emergency. Lastly, after each section you might want to reboot the machine to verify that things are working as they should; otherwise you'll have too many changes at once to pinpoint issues later.
Tuning the TCP/IP Stack
This part consists of two distinct modifications to be made to the system in the way of tunable kernel parameters; one set is accomplished by modifying the /etc/system file and the other with a custom "tune_tcp" shell script, executed during system initialization (boot). It is important that you save a backup copy of your /etc/system file (e.g. "cp /etc/system /etc/system.good") in the event something is incorrect, preventing the system from booting.
The first step entails adding a few lines to your /etc/system file thusly:
# Increasing file descriptors for faster TCP/IP apps (e.g. Apache) # set rlim_fd_max=16384 set rlim_fd_cur=8192 # Determines the size of the table for TCP information # # Default: 512 # set tcp:tcp_conn_hash_size = 32768Now onto the next step by installing a copy of the tune_tcp script that I've written for this purpose. Simply copy the file into /etc/init.d (making sure it is executable with "chmod 755 tune_tcp") and link to it by changing into the /etc/rc2.d directory and running the "ln -s ../init.d/tune_tcp S99tune_tcp" command to make a symlink. During system initialization and in passing through run level 2, the script will get executed - thereby tuning some key values in the Solaris TCP/IP stack. As noted in the shell script, these values are tuned towards high performance in the running of a Web server. If your system is primarily running local apps, is a workstation or serves as a development box - the values might be a bit harsh. They deal mostly with lowering the instances of lingering "TIME_WAIT" states on connections, opening up the bottlenecks on the send/receive buffers, allowing many more connections than by default and so on. There are also entries in this script to turn off forwarding, routing redirects, broadcast/ping echoes, protect against SYN flood attacks and so on. See the fully commented shell script for more details.
You'll need to reboot to have the /etc/system file modifications take effect, but you can run the init script at any time to change the other values. You'll notice that the Web server can handle a LOT more traffic now over a stock Solaris box and is a little safer from vandals as well.
Enabling the NCA
Enabling the NCA, or Network Cache and Accelerator is a simple process that increases Web serving performance a great deal. Similar in principle to the Linux TUX implementation in that pages are served from RAM and through the kernel, avoiding user space. Enabling the NCA is accomplished by editing several configuration files in /etc/nca and modifying your HTTP start up script - most all are transparently supported. Recompiling or modifying your HTTP software is not required as the hooks are in the system-wide socket library.
To enable NCA, make sure you have the required packages installed. With the command, "pkginfo | grep nca" you should see output similar to the following:
SUNWncar Solaris Network Cache and Accelerator (Root) SUNWncarx Solaris Network Cache and Accelerator (Root) (64-bit) SUNWncau Solaris Network Cache and Accelerator (Usr) SUNWncaux Solaris Network Cache and Accelerator (Usr) (64-bit)Go into the /etc/nca directory and note the three configuration files there. The first file, nca.if contains a list of interfaces that the NCA should listen to. Depending on your system and network interface, this can be "hme0" for later systems, "le0" for earlier ones or "qfe0" for quad-port interfaces. If you use Sun's Alternate Pathing, you might even choose a "mether" virtual interface here. If you want, you can simply use the wildcard character "*" which specifies all interfaces. Ideally, you want to limit this to as few interfaces as possible and especially just the interface serving the outside world with your Web content.
The next step is to look at the ncakmod.conf file. All that needs to be done in this file is to make sure all three options are set to "enabled."
Finally, look at the ncalogd.conf file and ensure that the "status" line reads "enabled" as well. Make sure that the /var/nca directory exists, which by default does not. Simply issue a "mkdir /var/nca" command to do so.
The final modification must be done to your HTTP server start up script. We'll go with Apache since it's installed along with Solaris and is the most common. If you're using the stock Solaris Apache installation, you have nothing further to do actually, since the start up script is already modified to take advantage of NCA automatically. If you chose to install a more up-to-date version of Apache, or Apache 2.x - you'll need to copy this Sun-modified script (apachectl) over, paying attention to the paths at the top of the script, making sure they reflect the layout if it has changed. You'll see the relevent sections in the script where the Apache server is started with multiple references to NCA.
If you're using Apache, you'll want to addtionally set the "KeepAlive" parameter to "Off" and comment out the following lines, since the NCA now handles these aspects:
MaxKeepAliveRequests 100 KeepAliveTimeout 15Now we're ready to try it out. Unfortunately, you'll have to reboot the system for the changes to take effect as we're dealing with kernel modules that must be loaded at boot. To verify that the NCA has initialized, make sure that the /var/run/nca_httpd_1.door file exists and that there are fresh files in /var/nca as well. If you're curious about other settings, run the "ndd -get /dev/nca ?" command for all kernel tunable parameters affecting NCA. In Solaris 8 and even in Solaris 9 there is very little documentation about the the NCA, especially debugging and making sure you're actually using it. Hopefully Solaris 10 will have improved upon this aspect of NCA as well.
That's it! You're now serving pages right from the kernel that have been cached. The performance increase from this tweak alone should be quite noticeable, more so in later versions of Solaris such as Solaris 10 or Solaris 9 even as each version improves upon the previous one.
Caching Compiled PHP Code
If you use PHP in your Website, you've surely heard of the Zend Optimizer, which is part of the base PHP installation. The principle is to cache your PHP code that's already been compiled, saving future requests the extra burden of compiling your code all over again. While the included Zend extension works, it is a very light implementation and you'll need to purchase the Zend Accelerator product to really achieve better performance. There is a free alternative called PHP Accelerator. It provides compiled code caching from RAM as well as files. While free in the sense it is not commercial like the Zend offering, it is somewhat limited in that it is released in binary form only, applies to certain versions as mentioned below and is not GPL/Open Source. Another alternative is the Turck MMCache accelerator.
First, head over to the PHP Accelerator home page and grab the version that mirrors your installation best. You'll need to pay attention to both the version of Solaris as well as PHP that you are running. While it's best to use an exact match, one that's close will often work as well.
Put the software wherever you'd like - it's really just two files. You can stash them in your Apache directory, or make a directory just for the accelerator somewhere. In any event, you'll only have to edit your php.ini file to point to where you ended up putting the "php_accelerator_X.Y.Z.so" file. You can run with the default values, or you can add in more extensive options along with comments by cutting-and-pasting the relevant sections from the "CONFIGURATION" text file supplied with the accelerator. I chose to do the latter, so it's easy to see what the options are and have them at the ready, should they be needed later. At the very minimum, you'll need to add this line to your php.ini file, most often located in either /usr/local/lib or /usr/local/Zend/etc:
zend_extension=/path/to/php_accelerator/php_accelerator_X.Y.Z.soThe next step is to create a directory for the file cache and point that out in the php.ini file as well with a line similar to:
phpa.cache_dir = /path/to/cache/dirBy default, the cache files are stored in /tmp which I would highly discourage. You do not want to find yourself with a full /tmp directory! You can make a subdirectory under /var, put the directory on a RAID or on its own filesystem if you wish. If you do choose to use the /var directory and it is on its own filesystem make sure you don't accidentally fill up all the space available or you will lose logging information at best and potentially make the system unstable or crash as with /tmp up above.
Restart your HTTP software and let it run for a little while. Run the command "phpa_cache_admin -mv" from where you installed the accelerator and you should see output similar to the following:
shm size 16.0MB bytes mempool size 16.0MB mempool bytes allocated 512.0B mempool max bytes allocated 512.0B mempool bytes free 16.0MB mempool overhead 124.0B cache enabledYou'll also notice any number of files in your newly created cache directory. Congratulations, the accelerator is installed! Verify it further by running a phpinfo() script on your Web server and look for the text "...with the ionCube PHP Accelerator vX.Y.Z, Copyright (c) 2001-2002, by Nick Lindridge" in the Zend info box, the second one down from the top or look in your HTTP software logs for a line similar to "
Tuning Apache
Since the Apache Web server (HTTP software) is installed in most Solaris installations and/or is the most commonly used, we'll examine some tweaks that can be made to the httpd.conf file to maximize use of system resources. Again, this is something you should only do on a system that will primarily function as a Web server, or you will consume a large number of system resources for nothing.
Perform the usual Apache tuning as outlined by the Apache documentation and tweak a few parameters to take advantage of what your server hardware provides in terms of RAM so that you can effectively handle the expected connections that the above changes will now allow. Basically, you will want to up the "MinSpareServers" and "MaxSpareServers" as well as the "MaxClients" values. Depending on how much traffic you expect and how much RAM is available in the system, you might want to use values of "10" and "20" for the min/max values respectively, and "255" for the "MaxClients" value. Ideally, you'll want to add a line in your php.ini file if you're using PHP Accelerator. That line would read "phpa.shm_max_processes = 256" or similar - use a value that is equal to Apache's "MaxClients" plus one.
Increasing these values in Apache on a Web server box ensures that there are plenty of Apache processes on hand to answer connections rapidly. These values could be increased further still which depends solely on your RAM configuration. Do NOT increase these values so much that you spawn a great deal of Apache processes and begin to swap. This would knock your performance way, way down - making all these modifications moot. Use a program such as "top" to keep an eye on processes, RAM, swap and also CPU utilization. Tune Apache appropriately as you observe trends over the course of a few days or weeks.
Resource Dependency
Just a brief mention about other things that might affect Web server performance and that is the dependency on other resources. Such resources run the gamut from CPU speed, amount of RAM, disk drive specifications and so on. But don't forget that there are also "soft" dependencies that can greatly affect the speed of your Web server. If you use PHP code, you're more than likely accessing a database. Whether you use MySQL or Oracle - make sure that a perceived lack of performance in your Web server isn't due to a slow database. If you're running your database on another server, also make sure that the network connection between the two machines isn't affecting you negatively. A congested connection will hold up your Web server, as it would if you were using a SAN with the same problem. Such tuning is beyond the scope of this article, but you should carefully monitor your other resources and ensure that they are not holding up processing of Web requests.
Benchmarking Improvements
At each step of this process, you might want to run some benchmarks against your Web server to see just how much has changed in terms of performance. This will tell you exactly how you're doing, if any problems arise or if further tweaking is needed.
Again using Apache as an example, you can use the "ab" command to run a few profiles of connections and concurrency levels to gauge many performance metrics. Some things you want to keep an eye on are requests per second, transfer rate and RAM/CPU utilization of the server. To obtain the purest results, you should begin each test with a fresh restart of the Apache server and apply a few consistent browsing sessions to load up the caches. It might be interesting to see the performance differences between un-cached and cached pages/code and so on as it will take a few accesses to fully cache a particular page or script. Once in the cache, access will be at its maximum efficiency.
As with any performance tuning and tweaking, you need to establish not only a baseline to which all future benchmarks will be compared against, but also at each step of the process so that you can tell not only if you're improving things - but also if you're making them worse! Take the extra time and make sure you're moving forward.
Conclusion
The steps in this document are meant mostly to bring to your attention some minimal steps required to get greatly enhanced performance from your server. There are many variables from one site to another including programming practices, amount of code, server configuration, software versions and more - making a cut-and-dry formula somewhat difficult. However, armed with the information above it is suggested you do some research on the Internet for related articles and take what you know about your own Website(s) to come up with the right balance. Monitoring the server for a length of time, making a few adjustments and repeating this process is the best way to obtain Web server Nirvana.
Additional Reading
To help you further understand some of the tuning going on here, check out some of the references listed below for various FAQs, HOW-TOs and other information.
