Car insurance

admin

Encrypted image that grows as needed, for linux backups

 osx  No Responses »
Jul 242010
 

This command will create an encrypted image on mac OS X that will grow as you need it.
Since I could not find anywhere on the net, I dig on the hdiutil man page and wrapped the command.

The image starts with about 600MB in size, which is not wasted actually.

I use this image to backup my sensitive data such as servers config files, mysql databases and repositories.
I set it to use AES 128bit encryption.
Case-sensitive, because Linux fs are Case-sensitive by default. If you don’t set it Case-sensitive you cannot use for linux backups.

hdiutil create -encryption AES-128 -stdinpass -fs “Case-sensitive HFS+” -type SPARSE -nospotlight -volname Servers Servers

it will create a file named Servers.sparseimage

VN:F [1.9.15_1155]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.15_1155]
Rating: 0 (from 0 votes)

 Posted by admin at 3:47 pm

Check guaranteed Ram, burstable Ram and Error Counts on your OpenVZ VPS

 linux  No Responses »
Jul 092010
 

Now a days there are so many good and cheap OpenVZ VPS providers out there.
Just have a look at this blog and you will see what I mean. [http://www.lowendbox.com/tag/openvz/]

The way OpenVZ virtual machines offers RAM resource is quite different from XEN.
You get Dedicated Ram (guaranteed ram) and Burstable Ram.

Burstable Ram is good when your server has sporadic load spikes, such as blogs and news sites.
Ideally you should not rely on burstable ram, it depends on the Host System. if the host system starts get low on free ram, you will lose that memory and the kernel will kill your processes to recover the ram.

Best is to keep below 90% of your Dedicated ram.

So, here I have an script that will monitor your memory limits. Also check for errors on your openVZ.
it’s perfect for an hourly cron job. It will print output on warnings and errors only.
Will suppress output if your openVZ container is behaving well. :)

for example, put on /etc/cron.hourly/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

#!/usr/bin/env ruby
#
# Program to check for failed count on OpenVZ systems
#   Failed counts indicate over commit on memory. 
#   it should always be zero.
 
 
@file = `cat /proc/user_beancounters`.split("\n")
 
# Delete the first 3 lines, which are irrelevant to us.
@file.delete_at(0)
@file.delete_at(0)
@file.delete_at(0)
 
@failcnt = []
 
@file.each do |t|
  tmp = t.split
  unless tmp.last == "0"
    @failcnt << tmp
  end
end
 
# Check that 'held' is less than 'barrier' 
#   held = current count
#   barrier = soft limit
# only checking for:
#   privvmpages @file[1] 
#   oomguarpages @file[7] 
# 
# oomguarpages : 
#   The out-of-memory guarantee, in pages. 
#   Any VE process will not be killed even in case of heavy 
#   memory shortage if the current memory consumption 
#   (including both physical memory and swap) 
#   does not reach the oomguarpages barrier.
#
# privvmpages: 
#  The size of private (or potentially private) memory allocated by an application. 
#  The memory that is always shared among different applications 
#  is not included in this resource parameter.
# 
# Sources: 
#  - http://wiki.openvz.org/UBC_secondary_parameters
#  - http://wiki.vpslink.com/Meaning_of_the_/proc/user_beancounters_Values
 
privvmpages = @file[1].split
privvmpages_current = privvmpages[1].to_i
privvmpages_limit = privvmpages[3].to_i
if privvmpages_current > privvmpages_limit
  puts "WARNING: #{privvmpages[0]} has passed the limit: #{privvmpages_current}/#{privvmpages_limit}"
  puts "WARNING: Server may crash or reboot"
end
# Also check it we are getting close to 70% of usage
privvmpages_pc = ((privvmpages_current.to_f/privvmpages_limit)*100).ceil
if privvmpages_pc > 70
  puts "WARNING: #{privvmpages[0]} is high: #{privvmpages_pc}% of limit"
end
 
oomguarpages = @file[7].split
oomguarpages_current = oomguarpages[1].to_i
oomguarpages_limit = oomguarpages[3].to_i
if oomguarpages_current > oomguarpages_limit
  puts "WARNING: #{oomguarpages[0]} has passed the limit: #{oomguarpages_current}/#{oomguarpages_limit}"
  puts "WARNING: Some Processes may be killed"
end
# Also check it we are getting close to 85% of usage
oomguarpages_pc = ((oomguarpages_current.to_f/oomguarpages_limit)*100).ceil
if oomguarpages_pc > 85
  puts "WARNING: #{oomguarpages[0]} is high: #{oomguarpages_pc}% of limit"
end
 
unless @failcnt.empty?
  puts "=================================="
  puts "Found Memory problems with OpenVZ:"
  puts "=================================="
  @failcnt.each do |t|
    puts "#{t.join(' - ')}"
    puts "=================================="
  end
end
VN:F [1.9.15_1155]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.15_1155]
Rating: 0 (from 0 votes)

 Posted by admin at 9:37 pm

Server monitoring with gentoo and iStat on the iPhone

 linux  6 Responses »
Jun 252010
 

You can monitor your servers nicely graphs using iStat for iphone.
these are instructions for Gentoo:

1
2
3
4
5
6
7
8
9

wget http://github.com/downloads/tiwilliam/istatd/istatd-0.5.7.tar.gz
tar xpf istatd-0.5.7.tar.gz
cd istatd-0.5.7
./configure
make
make install 
useradd istat
mkdir -p /var/{run,cache}/istat /home/istat
chown istat.istat /var/{run,cache}/istat /home/istat

Create and edit /etc/istat.conf to match your server specs.
This config is for my OpenVZ vps.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#
# /etc/istat.conf: Configuration for iStat server
#
 
# IP Address to listen
network_addr           77.88.xx.xx
 
# choose any port, Default is 5109
network_port           55666 
 
# this is password, 5 digits
server_code            12345
 
server_user            istat
server_socket          /var/run/istat/istat.sock
server_pid             /var/run/istat/istat.pid
cache_dir              /var/cache/istat
monitor_net              venet0
monitor_disk             ( /dev/simfs )

Start istatd server:

1

sudo -u istat istatd -d -c /etc/istat.conf

you might add it to /etc/conf.d/local.start to auload on startup

Then download iStat for iphone http://bjango.com/apps/istat/ it’s only $0.99
and add your server to the list.

Here are some screenshots, first is server and second image is my macbook discovered with bonjour.

VN:F [1.9.15_1155]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.15_1155]
Rating: 0 (from 0 votes)

 Posted by admin at 2:00 pm  Tagged with: , ,

Really checking the content-type/mime_type of a file in OSX and Linux

 osx, ruby  4 Responses »
Jun 142009
 

I have came across many projects where checking file uploads and content-type (mime-type) is poorly implemented or heavy in resource.

Methods I have seen so far:

1. Checking content-type from file name: this inefficient, a user can just rename a file and you are fooled, or the file can have a different file format and you will not get the expected result.

2. Using Rmagick to check if the file is an image. This is so slow and uses so much Ram. You can try to initialize an rmagick object from an image file, then rescue when the file is not an image.

3. Using mini_magick to check if a file. This method is faster than rmagick. Implemen ted same way as rmagick.

A Better method for OSX and Linux,  is to use the command line tool “file” included in most UNIX operating systems.

It is very fast and very accurate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

file = "/path/to/file.ext"
if RUBY_PLATFORM.match(/darwin|linux|unix|solaris|bsd/)
 content_type = `file --raw --brief "#{file}"`.chomp
 case
  when content_type.match(/image|png|jpg|jpeg|gif/)
   real_type = "image"
  when content_type.match(/pdf/)
   real_type = "pdf"
  when content_type.match("Microsoft Word|Microsoft Office Document")
   real_type = "doc"
  else # This can go on and on
   real_type = "Unknown"
  end
end

Some examples of content types:

.doc = Microsoft Word document data

.doc = Microsoft Office Document

.pdf = PDF document, version 1.4

.pdf = PDF document, version 1.3

.psd = Adobe Photoshop Image

.png = PNG image data, 3508 x 4961, 8-bit/color RGBA, non-interlaced

.gif = GIF image data, version 89a, 195 x 109

.jpg = JPEG image data, EXIF standard

etc…

I hope this can be useful to someone.

VN:F [1.9.15_1155]
Rating: 9.0/10 (1 vote cast)
VN:F [1.9.15_1155]
Rating: +1 (from 1 vote)

 Posted by admin at 6:55 pm  Tagged with: , , , , ,

Easy installing Nginx + mod_rails passenger OSX

 nginx, osx  No Responses »
Jun 142009
 

Requirements:

1. XCODE you can download xcode from http://developer.apple.com/tools/xcode/index.html

2. OSX 10.4, 10.5 or 10.6

Procedures:

1. Install Passenger

$ sudo gem install passenger

now check where is passenger installed:

$ passenger-config --root

in my case is: /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.5

2. Install nginx

If you have nginx from macports, deactivate it in case of conflicts.

You can activate anytime later

$ sudo port deactivate nginx

$ wget http://sysoev.ru/nginx/nginx-0.7.59.tar.gz

$ wget http://sysoev.ru/nginx/nginx-0.7.62.tar.gz

$ tar xpf nginx-0.7.62.tar.gz

$ cd nginx-0.7.62

I Recommend using nginx 0.6 series because I had a lot of “502 Bad Gateway” with 0.7 series.

$ wget http://sysoev.ru/nginx/nginx-0.6.37.tar.gz

$ tar xpf nginx-0.6.37.tar.gz

$ cd nginx-0.6.37

$ sudo ./configure --add-module=/opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.5/ext/nginx/ \
  --with-http_ssl_module --user=nobody --group=nobody --with-http_gzip_static_module \
  --with-poll_module --prefix=/opt/local --with-pcre

Configuration summary
+ using system PCRE library
+ using system OpenSSL library
+ md5: using OpenSSL library
+ sha1 library is not used
+ using system zlib library

nginx path prefix: "/opt/local"
nginx binary file: "/opt/local/sbin/nginx"
nginx configuration prefix: "/opt/local/conf"
nginx configuration file: "/opt/local/conf/nginx.conf"
nginx pid file: "/opt/local/logs/nginx.pid"
nginx error log file: "/opt/local/logs/error.log"
nginx http access log file: "/opt/local/logs/access.log"
nginx http client request body temporary files: "client_body_temp"
nginx http proxy temporary files: "proxy_temp"
nginx http fastcgi temporary files: "fastcgi_temp"

$ sudo make

$ sudo make install

$ cd /opt/local/conf

$ sudo cp mime.types.default mime.types

$ sudo cp nginx.conf.default nginx.conf

Edit nginx.conf

$ mate nginx.conf

or

$ sudo vi nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

 
user nobody;
worker_processes  2;
 
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  logs/error.log  info;
 
# Pid
pid  logs/nginx.pid;
 
events {
worker_connections 1024;
}
 
http {
include       mime.types;
default_type  application/octet-stream;
 
sendfile        on;
#tcp_nopush     on;
 
#keepalive_timeout  0;
keepalive_timeout  65;
 
gzip  on;
# if a precompiled gzip of the file exists, use it and force http proxies
# to use separate cache's based on User-Agent
gzip_static on;
gzip_min_length 2000;
gzip_buffers    16 8k;
gzip_types      text/plain text/html text/css image/x-icon application/xml application/xml+rss text/javascript;
gzip_disable    "MSIE [1-6] \.";
gzip_vary         on;
gzip_comp_level   2;
 
gzip_proxied any;
 
server {
listen       80;
server_name  localhost;
location / {
root   /Users/fred/Sites ;
autoindex on;
index  index.html index.htm;
}
}
 
passenger_root /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.2;
passenger_max_pool_size 8;
passenger_max_instances_per_app 1;
# The maximum number of seconds that an application instance may be idle.
# That is, if an application instance hasn’t received any traffic after the given number of seconds,
# then it will be shutdown in order to conserve memory.
passenger_pool_idle_time 3600;
 
# Project 1
server {
listen 80;
client_max_body_size 250M;
server_name project1.local;
root /Users/fred/rails/project1/public;
passenger_enabled on;
rails_env development;
access_log  /Users/fred/rails/project1/log/nginx.access.log;
error_log  /Users/fred/rails/project1/log/nginx.error.log info;
}
 
# Project 2
server {
listen 80;
client_max_body_size 250M;
server_name project2.local;
root /Users/fred/rails/project2/public;
passenger_enabled on;
rails_env development;
access_log  /Users/fred/rails/project2/log/nginx.access.log;
error_log  /Users/fred/rails/project2/log/nginx.error.log info;
}
 
# Project 3
server {
listen 80;
client_max_body_size 250M;
server_name project3.local;
root /Users/fred/rails/project3/public;
passenger_enabled on;
rails_env development;
access_log  /Users/fred/rails/project3/log/nginx.access.log;
error_log  /Users/fred/rails/project3/log/nginx.error.log info;
}
 
# And so on... as many projects as you want
 
}

Now edit your /etc/hosts and add the hosts for your local project

$ mate /etc/hosts


127.0.0.1   project1.local

127.0.0.1   project2.local

127.0.0.1   project3.local

3. Start nginx

sudo nginx

4. go to your browser and open project1.local

:)

5. Easy start/restart/stop

add this to your ~/.bash_profile file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

 
function nginx_reload() {
FILE="/opt/local/logs/nginx.pid"
if [ -e $FILE ]; then
echo "Reloading NGINX..."
PID=`cat /opt/local/logs/nginx.pid`
sudo kill -HUP $PID
else
echo "Nginx pid file not found"
return 0
fi
}
 
function nginx_stop() {
FILE="/opt/local/logs/nginx.pid"
if [ -e $FILE ]; then
echo "Stopping NGINX..."
PID=`cat /opt/local/logs/nginx.pid`
sudo kill -INT $PID
else
echo "Nginx pid file not found"
return 0
fi
}
 
function nginx_restart() {
FILE="/opt/local/logs/nginx.pid"
if [ -e $FILE ]; then
echo "Stopping NGINX..."
PID=`cat /opt/local/logs/nginx.pid`
sudo kill -INT $PID
sleep 1
echo "Starting NGINX..."
sudo nginx
else
echo "Nginx pid file not found"
return 0
fi
}

Troubleshooting

1. Nginx is not running

- check the logs

- check if it is really not running:

$ ps aux | grep nginx

2. you see the nginx error “502 Bad Gateway”

- may there is a problem with the /var/folders/ permissions on OSX:


2009/06/13 16:14:33 [crit] 1106#0: *1 connect() to unix:/var/folders/xl/xlSRYvzFHH8Fcehc51ciyE+++TI/-Tmp-//passenger.1091/master/helper_server.sock failed (13: Permission denied) while connecting to upstream, client: 127.0.0.1, server: hassan.local, request: "GET / HTTP/1.1", upstream: "unix:/var/folders/xl/xlSRYvzFHH8Fcehc51ciyE+++TI/-Tmp-//passenger.1091/master/helper_server.sock:", host: "hassan.local"

to fix it I did this:

$ sudo find /var/folders/xl/ -name “master” -exec chmod 755 {} \;

$ sudo find /var/folders/xl/ -name “-Tmp-” -exec chmod 755 {} \;

everytime I reboot my mac I had to do that… I still dont know how to fix it…

anybody knows?

That is it for now.

VN:F [1.9.15_1155]
Rating: 10.0/10 (1 vote cast)
VN:F [1.9.15_1155]
Rating: +1 (from 1 vote)

 Posted by admin at 6:39 pm  Tagged with: , , ,

Stop http Pipeline from overloading your server, using connlimit iptables

 Uncategorized  1 Response »
Jan 252009
 

Have you edited or tweaked your pipeline settings in Firefox?

You can do it by typing about:config in your firefox url tab.

Most blogs and tutorial will tell you to set high values to improve the speed, such as this one: www.mydigitallife.info/2007/10/16/speed-up-your-firefox-by-adjusting-your-http-pipelining/

Many people will go crazy and make values even higher such as:

network.http.pipelining.maxrequests 32
network.http.max-persistent-connections-per-proxy 128
network.http.max-persistent-connections-per-server 128
network.http.max-connections-per-server 256

These settings are very high and will create at least 32 connections to your server.

if you have many images and SSI includes, it could overload your apache webserver.
If you use apache2.2 with worker_mpm it will create 1 thread for each connection, thus you will have 32 new threads forked within just a few seconds.

Of course if you have a quad-core server with lots of ram you should not bother to read this.
But for most cheap vps and single core servers, it can really help.

so how you do it? simple, just use iptables conn_limit module

iptables -I INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above  8 -j REJECT

you may need to adjust the order or to insert or append like

# to apped to the end of the INPUT chain:
iptables -A input …
or
# to insert at position 10 of the input chain:
iptables -I input 10 …

I have tested this schema and work very well with firefox pipeline freaks.

the server will only then take up to 8 simultaneously connections per IP

to test the established connections you can try with netstat from the server:

netstat -na | grep -i esta | grep -v 127.0.0.1 | sort -n -t. -k2

Any comments, suggestions are welcome …

Update:

This can be not so good to legit users behind a proxy or firewall, because the the IP will be unique to all users behind the proxy/firewall.

in this case, you would increase the limit value.

VN:F [1.9.15_1155]
Rating: 5.5/10 (2 votes cast)
VN:F [1.9.15_1155]
Rating: 0 (from 0 votes)

 Posted by admin at 8:31 am  Tagged with: , ,
 Older Entries  Newer Entries
© 2012 Ruby, Rails, OSX and Linux fun Suffusion theme by Sayontan Sinha

Switch to our mobile site