1
2
3
4
5
6
7
8
9

wget http://github.com/downloads/tiwilliam/istatd/istatd-0.5.7.tar.gz
tar xpf istatd-0.5.7.tar.gz
cd istatd-0.5.7
./configure
make
make install 
useradd istat
mkdir -p /var/{run,cache}/istat /home/istat
chown istat.istat /var/{run,cache}/istat /home/istat

Create and edit /etc/istat.conf to match your server specs.
This config is for my OpenVZ vps.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#
# /etc/istat.conf: Configuration for iStat server
#
 
# IP Address to listen
network_addr           77.88.xx.xx
 
# choose any port, Default is 5109
network_port           55666 
 
# this is password, 5 digits
server_code            12345
 
server_user            istat
server_socket          /var/run/istat/istat.sock
server_pid             /var/run/istat/istat.pid
cache_dir              /var/cache/istat
monitor_net              venet0
monitor_disk             ( /dev/simfs )

Start istatd server:

1

sudo -u istat istatd -d -c /etc/istat.conf

you might add it to /etc/conf.d/local.start to auload on startup

Then download iStat for iphone http://bjango.com/apps/istat/ it’s only $0.99
and add your server to the list.

Here are some screenshots, first is server and second image is my macbook discovered with bonjour.

Methods I have seen so far:

1. Checking content-type from file name: this inefficient, a user can just rename a file and you are fooled, or the file can have a different file format and you will not get the expected result.

2. Using Rmagick to check if the file is an image. This is so slow and uses so much Ram. You can try to initialize an rmagick object from an image file, then rescue when the file is not an image.

3. Using mini_magick to check if a file. This method is faster than rmagick. Implemen ted same way as rmagick.

A Better method for OSX and Linux,  is to use the command line tool “file” included in most UNIX operating systems.

It is very fast and very accurate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

file = "/path/to/file.ext"
if RUBY_PLATFORM.match(/darwin|linux|unix|solaris|bsd/)
 content_type = `file --raw --brief "#{file}"`.chomp
 case
  when content_type.match(/image|png|jpg|jpeg|gif/)
   real_type = "image"
  when content_type.match(/pdf/)
   real_type = "pdf"
  when content_type.match("Microsoft Word|Microsoft Office Document")
   real_type = "doc"
  else # This can go on and on
   real_type = "Unknown"
  end
end

Some examples of content types:

.doc = Microsoft Word document data

.doc = Microsoft Office Document

.pdf = PDF document, version 1.4

.pdf = PDF document, version 1.3

.psd = Adobe Photoshop Image

.png = PNG image data, 3508 x 4961, 8-bit/color RGBA, non-interlaced

.gif = GIF image data, version 89a, 195 x 109

.jpg = JPEG image data, EXIF standard

etc…

I hope this can be useful to someone.

You can do it by typing about:config in your firefox url tab.

Most blogs and tutorial will tell you to set high values to improve the speed, such as this one: www.mydigitallife.info/2007/10/16/speed-up-your-firefox-by-adjusting-your-http-pipelining/

Many people will go crazy and make values even higher such as:

network.http.pipelining.maxrequests 32
network.http.max-persistent-connections-per-proxy 128
network.http.max-persistent-connections-per-server 128
network.http.max-connections-per-server 256

These settings are very high and will create at least 32 connections to your server.

if you have many images and SSI includes, it could overload your apache webserver.
If you use apache2.2 with worker_mpm it will create 1 thread for each connection, thus you will have 32 new threads forked within just a few seconds.

Of course if you have a quad-core server with lots of ram you should not bother to read this.
But for most cheap vps and single core servers, it can really help.

so how you do it? simple, just use iptables conn_limit module

iptables -I INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above  8 -j REJECT

you may need to adjust the order or to insert or append like

# to apped to the end of the INPUT chain:
iptables -A input …
or
# to insert at position 10 of the input chain:
iptables -I input 10 …

I have tested this schema and work very well with firefox pipeline freaks.

the server will only then take up to 8 simultaneously connections per IP

to test the established connections you can try with netstat from the server:

netstat -na | grep -i esta | grep -v 127.0.0.1 | sort -n -t. -k2

Any comments, suggestions are welcome …

Update:

This can be not so good to legit users behind a proxy or firewall, because the the IP will be unique to all users behind the proxy/firewall.

in this case, you would increase the limit value.

Since you are Gentoo user, i don’t need to go to details. You know what you doing.

Update: Mod-rails now works with apache mpm-worker

1. Recompile Apache non-threaded

add this to /etc/portage/package.use

www-servers/apache -threads

and this to /etc/make.conf

APACHE2_MPMS="prefork"

2. Re emerge apache

# emerge -va apache

3. Passenger is in gentoo portage, but its in testing. Currently Version 2.0.1

# echo "www-apache/passenger" >> /etc/portage/package.keywords

4. Install Passenger

# emerge -va passenger

If it tries to install rails 2.2.2, rake, and lots of other gems that you already have installed trough rubygems, then run emerge with –nodeps option

# emerge -va --nodeps passenger

5. Edit /etc/conf.d/apache and add “-D PASSENGER” to apache options

for example mine looks like this:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D LANGUAGE -D PROXY -D PASSENGER"

That’s it.

Now just drop a similar vhost config file inside /etc/apache/vhosts.d/

This is a sample vhost file for a rails app.

<VirtualHost *:80>
  ServerName mydomain.com
  DocumentRoot /myapp/public
  Include /etc/apache2/vhosts.d/deflate.conf
  RailsBaseURI /
  # The maximum number of Ruby on Rails application instances that may be simultaneously active.
  # A larger number results in higher memory usage, but improved ability to handle concurrent HTTP clients.
  # normally 1 to 10. (1 for each 50mb ram)
  RailsMaxPoolSize 1
  # The maximum number of seconds that a Ruby on Rails application instance may be idle.
  # That is, if an application instance hasn't done anything after the given number of seconds,
  # then it will be shutdown in order to conserve memory. ( 1 hour)
  RailsPoolIdleTime 3600
  RailsEnv 'production'
  <Directory /myapp/public>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
    </Directory>
</VirtualHost>

My sample deflate.conf,
used to gzip the content

<Location />
	SetOutputFilter DEFLATE
	#
	# Netscape 4.x has some problems...
	BrowserMatch ^Mozilla/4 gzip-only-text/html
	#
	# Netscape 4.06-4.08 have some more problems
	BrowserMatch ^Mozilla/4\.0[678] no-gzip
	#
	# MSIE masquerades as Netscape, but it is fine
	BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
	# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
	# the above regex won't work. You can use the following
	# workaround to get the desired effect:
	BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
	# Don't compress images
	SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
	SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
	SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary
	# Make sure proxies don't deliver the wrong content
	Header append Vary User-Agent env=!dont-vary
</Location>

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/apache2/deflate_log deflate

* Update on July 10, 2008.

- Now using gentoo portage to install it. it’s more smooth.

Note:

Personally I found that Thin + nginx uses less memory(Nginx 4MB + each thin server) than
apache + passenger, which uses quite more. (Apache: 50MB + each rails spawner)

But 256MB of ram? what can you do with just 256?

Normally a default 256mb linux machine would not handle very well a set of Apache + Mysql + 1 mongrel/thin/ebb instance. due to the high memory usage of a default configuration, it will swap very often.

After much research and instinct i made it run one thin servers with mysql and nginx, without any swapping, and really fast as it can be.

If your linux start swapping often your performance will go down to the floor… Swapping is bad, specially on a XEN VPS.

The trick is to setup Mysql to use MYISAM and use Nginx instead of apache.

Here is the process list with the Resident Memory usage, after 30 days uptime and about 1,800 page views on the website.

and others, such as sshd, cron, iptables, bash, together about 5mb.

As you can see, total of memory usage of the applications on the server is about 83 MB, thus leaving the server with 170MB of ram for the linux itself and file cache.

this is what #free command tells:

             total       used       free     shared    buffers     cached
Mem:           256        235         20          0         54         62
-/+ buffers/cache:        128        128
Swap:          511          0        511

Nice uh?
you can also make use of the nice tool called “vmstat”
it’s very import that ’si’ (swap in) and ’so’ (swap out) stays zero all the time.

i.e. running vmstat 10 times with a 4 seconds interval. (ignore the 1st line)

# vmstat 4 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0    116  14900  56668  62692    1    1     3     3    9    5  0  0 100  0
 0  0    116  14908  56668  62692    0    0     0     0   34   53  0  0 100  0
 0  0    116  14968  56668  62692    0    0     0     0   37   54  0  0 100  0
 0  0    116  14968  56668  62692    0    0     0     0   31   52  0  0 100  0
....

Here is the recipe:

1. Use MyIsam instead of InnoDB

You can read about it more in here:

I forgot to add that you need to dump your database first:

mysqldump -u root –all-databases > dump.sql

then change my.cnf accordingly,

restart mysql and reload the database

mysql -u root < dump.sql

Change only the values for my.cnf as shown below, and delete all innodb related stuff

# can be safely set to 1M if you are really tight on Ram
key_buffer 			       = 4M

max_allowed_packet 			= 1M
table_cache 				= 32
sort_buffer_size 			= 512k
net_buffer_length 			= 8K
read_buffer_size 			= 512k
read_rnd_buffer_size 		= 512K

# can be safely set to 1M
myisam_sort_buffer_size 	= 2M  

language 					= /usr/share/mysql/english

# security:
# using "localhost" in connects uses sockets by default
# skip-networking
bind-address				= 127.0.0.1

# No logging,
# make sure you backup your database more often.
#log-bin

server-id 					= 1

# point the following paths to different dedicated disks
tmpdir 						= /tmp/

# Very important to have this here.
# otherwise it will still load InnoDB.
skip-innodb

[mysqldump]
quick
max_allowed_packet 			= 16M

[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer 					= 8M
sort_buffer_size 			= 8M
read_buffer 				= 2M
write_buffer 				= 2M

[myisamchk]
key_buffer 					= 8M
sort_buffer_size 			= 8M
read_buffer 				= 2M
write_buffer 				= 2M

[mysqlhotcopy]
interactive-timeout

If you get problems reloading the database, stop mysql delete the contents in /var/lib/mysql/* , then run mysql-installdb and start it and reload again the sql dump file.

Actually that’s the way i most prefer..

2. Use nginx

this is an example nginx config file, located at /etc/nginx/nginx.conf

user nginx nginx;
# set to 2 or 3 if you have more processors or cores.
# it will use about 3MB per worker
worker_processes 1;

error_log /var/log/nginx/error_log info;

events {
	# default is 8192
	worker_connections  1024;
	use epoll;
}

http {
	include		/etc/nginx/mime.types;
	default_type	application/octet-stream;

	log_format main
		'$remote_addr - $remote_user [$time_local] '
       	'"$request" $status $bytes_sent '
		'"$http_referer" "$http_user_agent" '
		'"$gzip_ratio"';

	client_header_timeout	10m;
	client_body_timeout	10m;
	send_timeout		10m;
        client_body_buffer_size    128k;
        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;

	connection_pool_size		256;
	client_header_buffer_size	1k;
	large_client_header_buffers	4 2k;
	request_pool_size		4k;

	gzip on;
	gzip_http_version 1.1;
        gzip_comp_level 6;
        gzip_proxied any;
        gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
        gzip_buffers 16 8k;
        # Disable gzip for certain browsers.
        gzip_disable "MSIE [1-6]\.(?!.*SV1)";
	gzip_min_length	1100;

	output_buffers	1 32k;
	postpone_output	1460;

	sendfile	on;
	tcp_nopush	on;
	tcp_nodelay	on;

	keepalive_timeout	75 20;

	ignore_invalid_headers	on;

	index index.html;

    # The following includes are specified for virtual hosts
    include /var/www/apps/bla.com/shared/config/nginx.conf;

}

this is an example vhost file

upstream mongrel_bla_com {
  server 127.0.0.1:8001;
}

server {
	listen 80;
	client_max_body_size 40M;
	server_name bla.com www.bla.com;
	root /var/www/apps/bla.com/current/public;
	access_log  /var/www/apps/bla.com/shared/log/nginx.access.log;

	location / {
                proxy_set_header  X-Real-IP  $remote_addr;
                proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect false;
                if (-f $request_filename/index.html) {
                        rewrite (.*) $1/index.html break;
                }
                if (-f $request_filename.html) {
                        rewrite (.*) $1.html break;
                }
                if (!-f $request_filename) {
                        proxy_pass http://mongrel_bla_com;
                        break;
                }
	}

	location ~* ^.+\.(jpg|js|jpeg|png|ico|gif|txt|js|css|swf|zip|rar|avi|exe|mpg|mp3|wav|mpeg|asf|wmv)$ {
		root /var/www/apps/bla.com/current/public;
	}

}

3. Remove unnecessary Services you dont need.

Some linux distros have enabled by default services we dont need.

such as cupsd, apmd, acpid, mdns, samba, nfs, ftpd… etc…

This is my make.conf in case it helps

Note that I set MAKEOPTS=”-J1″ , it will only use 1 gcc process at the time, and not disturb the system, (machine has 4 cores)

Also portage_niceness to 18, to make sure it will run smooth and not disturb thin and mysql.

from nice man page: “Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable).”

CFLAGS="-march=athlon64 -O2"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j1"

USE="3dnow 3dnowext apache2 \
     bash-completion bzip2 \
     -cups \
     gzip httpd hpn \
     innodb imagemagick \
     javascript jpeg \
     mmx mmxext mysql \
     nptl nptlonly \
     perl phyton png \
     ruby \
     screen sse sse2 sqlite sqlite3 ssl \
     threads udev unicode utf8 \
     vim-syntax \
     -X -kde -gnome -gtk -bindist \
     xml xml2 zlib "

GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.cites.uiuc.edu/pub/gentoo/ ftp://gentoo.mirrors.tds.net/gentoo http://ge
APACHE2_MPMS='worker'
PORTAGE_NICENESS=18

if you want to use mod_rails Passenger, set APACHE2_MPMS=’prefork’

note: I am positive you can throw in another thin server instance, and it will still not swap, or swap very little at all.

have fun

**************************

Updates :

Wanna know what Slicehost Manager Diagnostics says about my VPS ?

Diagnostics

    * Your slice is currently running.
    * The host server is up.
    * Your swap IO usage over the last 4 hours is low: 0.0016 reads/s, 0.0 writes/s. (Read more about swap here)
    * Your root IO usage over the last 4 hours is low: 0.038 reads/s, 0.1643 writes/s.
    * The host server's load is nominal: 0.00, 0.03, 0.00.