linux performance troubleshooting notes

vmstat <# in seconds>
  procs
    r = # of processes waiting for cpu
    b = # of processes waiting for i/o
  swap
    si and so = swap in and swap out; 
                close to 0; 
                no more than 10 blocks/sec
  io
    bi and bo = disk i/o
  system
    in = # of interrupts / sec
    cs = # of context switches / sec
  cpu
    us = % user
    sy = % system
    id = % idle
    wa = % i/o wait



iostat -dx <# in seconds>
   various rw/sec options
   await = # of milliseconds required to 
           respond to requests
   %util = device utilization


top
  load = 1m 5m 15m 
         (average # processes waiting on cpu time) 
         (quad core load of 4 is really same 
          amount as 1 core load of 1)
  cpu  = us/user sy/system id/idle wa/io_wait
  avail ram = free mem + cached

  'C' = sort processes by %CPU
  'M' = sort processes by %MEM


* high vmstat us cpu shows cpu-bound combined with 
  procs piled up in procs r column, combined with 
  low disk util in iostat
* high vmstat procs b column combined with high disk 
  util in iostat show i/o bound
* high values in swap si & so in vmstat show swapping
* idle machines show low r/b in vmstat procs, 
  high % in cpu id
* iostat to track down which device is getting 
  heavy reads/writes