Unfortunately, systematic performance tuning is time-consuming. Buying new hardware to handle increased load is often preferred over benchmark testing (and is obviously your sales representative’s ideal solution). Unix’s legendary flexibility compounds the problem, especially on Linux, where "performance tuning" can mean anything from adding memory to rewriting kernel source code. Too many choices can be paralyzing.

A lack of time for systematic testing doesn’t preclude a systematic approach to tuning, though, if the underlying issues are understood.

Practical Tuning
Tuning a system for performance is an engineering problem: given limited resources, how do you reach a goal? Performance must be balanced against other goals, of course. High availability systems may require constant uptime, but merely acceptable performance. Scalable clusters may sacrifice reliability for the ability to grow quickly. Public kiosks may prefer ease-of-use. Ignore all that for the time being.

The temptation to begin tweaking software immediately when a performance problem arises is strong. This is often a mistake. Hardware places firm limits on any system, and no amount of software tuning after-the-fact can compensate for poor design in this respect. Additionally, making multiple simultaneous software changes can backfire without the sort of systematic approach that time so rarely permits. The "quick-tune" approach may rein in the rogue process that’s spiraled out of control, but is unlikely to alleviate deeper problems, and may even interfere with your ability to identify long-term trends.

Hardware Design
Performance tuning through hardware design involves organizing your processors, network interfaces, storage, and the interconnections (the "buses") between the three. Input devices have a negligible effect on system performance, except for specialized applications, so I’m going to ignore them as well, for the moment.

Practically speaking, processor and bus speed are difficult to improve once a system is built unless you’re building the system yourself. In the age of pre-built systems, few servers are built to customer specifications at this level of detail, and techniques such as overclocking processors by or playing with BIOS bus rates tend to invalidate warranties and support contracts. Similarly, bandwidth is no longer a limiting factor for most applications. Optimizing performance via hardware design, then, is roughly equivalent to optimizing storage. Speed up access to your data, and performance will improve.

Storage
Storage decreases in price and speed roughly in proportion to its distance from the CPU (ignoring network-based storage). Electronic storage is above the line, physical storage below:

On-CPU storage
Level 2 Cache (frequently, also on the CPU)
Main Memory, aka "RAM"
-----
Disk
Tape

To simplify grossly, assume each storage medium in the above list is about half the speed and price of its parent and you’ll get the idea (as well as a renewed respect for the brave pioneers who dealt with nothing more than tape and punch cards, when things were really expensive).

The division between electronic and physical storage is the most important division in determining the performance of any computer. Once moving parts are involved, access times for data increase precipitously. The most common performance problem, then, is the input-output, or I/O problem: storage devices that are incapable of keeping up with the processor.

Divide and Conquer
If you’re fortunate enough to be designing a system from scratch, you can follow some general rules of thumb. Since tape is used primarily for backups and only rarely for actual I/O nowadays (ignoring hierarchical storage systems), and assuming that we maximized our options regarding memory and cache when we bought the system, it follows that our disk layout may be the single most important thing we can adjust to improve future performance.

Unix reads data heavily from wherever user files and frequently used applications are stored. Other directories are written to heavily—space used for temporary or log files, for instance. These limitations are common to any system, Unix or otherwise, since they’re usage-related, not operating system-related. Placing heavily read and heavily written file systems on the same disk causes drive contention -- i.e., competition between processes for a single drive.

Separating read-from and written-to file systems onto different disks (and disk controllers, if possible) is preferable. Multiple-disk (RAID -- Redundant Array of Independent Disks) file systems can improve performance as well, although it’s best to know something about the various RAID varieties before implementing one. Contrary to what the three-disk configurations most vendors sell would lead you to expect, RAID 5 file systems, which use three or more disks to provide data redundancy, actually reduce write performance in comparison to other RAID levels.

Separating the system’s swap space is also important, as it’s involved in the system’s virtual memory management. As the system shuffles processes around, anything in memory that isn’t immediately useful is written to the lower storage tiers -- i.e., "swapped" out. On systems with limited chip-based memory (RAM), disks make up the shortfall. Under heavy load, the system may be so occupied with writing memory in and out to disk that no serious work gets done -- a condition known as thrashing. On systems that provide flashing LEDs as indicators of drive activity, it’s usually readily, if unscientifically, identifiable -- the LEDs flash really fast. Swap files or partitions should be spread out across multiple disks for maximum performance, and placed on faster disks when possible. Also, swap space does not necessarily coincide with the /tmp directory used for temporary file space, although this is common on modern systems -- make sure that you know your system’s particulars, and optimize both if necessary.

Unix does have some peculiarities of its own. Unlike Windows, which writes to its equivalent of the "root" file system all the time, the Unix root file system ( the "/" directory), which contains the files the system needs to boot, is rarely read once the system is up and running. It can be safely placed on slower disks, unless quick reboots are required to minimize downtime.

The Elements of Performance
With apologies to Strunk and White, here are a few tips on the "style" of performance tuning:

Omit the Needless
A Web server’s front page may receive millions of hits a day. Don’t do something silly like make this file a symbolic link to something else, just because it’s easier than editing your configuration file properly. Every hit on the front page would then generate a double hit—one to find the link, and one to find the file the link refers to. Needlessly doubling disk activity is never a good idea. A "hard" link, of course, is fine — in Unix, two hard links actually refer to the same data, instead of pointing to it indirectly.

Know Your Subject
It helps to know something about how your particular brand of Unix works. For instance, some systems provide an option to turn off the tracking of last access time information -- not last modification time, which is considerably more useful. This is called the "noatime" option in Solaris, and is placed in the system’s table of mounted file systems (the /etc/vfstab file). For something like a mail spool or a temporary file system, where data is so transient that knowledge of the last time a particular file was accessed is next to useless, turning off this capability may increase performance significantly -- or imperceptibly. Small performance gains are still gains.

Use the Right Tool, Not Its Second Cousin
For oft-repeated tasks, use the most appropriate tool for the job. System monitoring is a common source of abuse in this respect. Graphical monitoring tools, and even some command-line monitoring tools like "top," which produces a nicely formatted display of processor and memory usage, make relatively heavy demands on the system to obtain this information, especially if they refresh the data every second. Several lower-level monitoring tools, available on most versions of Unix, produce equivalent data with fewer demands on the system, even if they’re tougher on the eyes:

• iostat -- monitors disk activity
• netstat -- monitors network activity
• vmstat -- monitors virtual memory activity on some systems
• sysstat -- monitors almost all system activity (originally known as sar on Solaris, later ported to Linux)

To obtain long-term performance statistics, the monitoring tool must have minimal impact. Although freely available or "standard" tools in the base Unix distribution are useful, be aware that the most effective tools may be proprietary.

And Knowing Is Half the Battle!
"Knowledge is power" is trite because it’s true: Know where the real work of your application is being done. Ask yourself this question: Where will the bottlenecks be? If you know this, you’re ahead of the game, or at least the documentation.

If this brief discussion of performance tuning seems overly simplistic, it is. Shelves of books are devoted to this topic. I’ve also stressed the importance of hardware design, a ground-up approach, at the expense of software tuning, which is obviously of considerable if not equal importance. For instance, Denial of Service attacks can be mitigated by alterations to a system’s network stack that can’t easily be accomplished in the network card itself—software tuning can obviously aid systems under stress. To paraphrase my erstwhile mechanic: although it’s not true that any system can offer limitless performance if properly tuned, limitless tuning can offer proper performance. Best of luck!