Collective Intelligence Framework

Looks interesting....

collective-intelligence-framework

brain dump #1

for i in $(cat /usr/local/hadoop/conf/slaves); do scp mapred-site.xml $i:/usr/local/hadoop/conf/mapred-site.xml; done

if (grep -q "^/usr/local/lib$" /etc/ld.so.conf); then echo ""; else echo "/usr/local/lib" >> /etc/ld.so.conf; fi

lftp :~> set ftp:ssl-force true
lftp :~> set ftp:ssl-protect-data true
lftp :~> set ssl:verify-certificate true
lftp :~> connect ftp.domain.tld
lftp ftp.domain.tld:~> login my_username

test ssl anonymous auth:
openssl s_client -starttls smtp -crlf -connect some_mail_server:25 -cipher aNULL

test ssl on smtp:
openssl s_client -starttls smtp -crlf -connect some_mail_server:25

low cipher checks:
openssl s_client -connect some_web_server:443 -cipher LOW:EXP

shadow hash:
openssl passwd -1 -salt XXYYZZ11

screen rsync -avz /source/dir/ /target/dir/

rsync -avz -e ssh /source/dir/ user@target_system:/target/dir/

rsync -avz --delete /source/dir/ /target/dir/

rsync -avz --exclude=.snapshot /source/dir/ /target/dir/

linux performance troubleshooting notes

vmstat <# in seconds>
  procs
    r = # of processes waiting for cpu
    b = # of processes waiting for i/o
  swap
    si and so = swap in and swap out; 
                close to 0; 
                no more than 10 blocks/sec
  io
    bi and bo = disk i/o
  system
    in = # of interrupts / sec
    cs = # of context switches / sec
  cpu
    us = % user
    sy = % system
    id = % idle
    wa = % i/o wait



iostat -dx <# in seconds>
   various rw/sec options
   await = # of milliseconds required to 
           respond to requests
   %util = device utilization


top
  load = 1m 5m 15m 
         (average # processes waiting on cpu time) 
         (quad core load of 4 is really same 
          amount as 1 core load of 1)
  cpu  = us/user sy/system id/idle wa/io_wait
  avail ram = free mem + cached

  'C' = sort processes by %CPU
  'M' = sort processes by %MEM


* high vmstat us cpu shows cpu-bound combined with 
  procs piled up in procs r column, combined with 
  low disk util in iostat
* high vmstat procs b column combined with high disk 
  util in iostat show i/o bound
* high values in swap si & so in vmstat show swapping
* idle machines show low r/b in vmstat procs, 
  high % in cpu id
* iostat to track down which device is getting 
  heavy reads/writes