Fred's Blog - Home

Very low memory VPS Linux for Rails

2008-07-03T16:21:00Z

The other day I had to set up a VPS machine at Slicehost for a client on a tight budget. I paid for 256mb VPS based on Gentoo, my distro of choice.
But 256MB of ram? what can you do with just 256?
Normally a default 256mb linux machine would not handle very well a set of Apache + Mysql + 1 mongrel/thin/ebb instance. due to the high memory usage of a default configuration, it will swap very often.
After much research and instinct i made it run one thin servers with mysql and nginx, without any swapping, and really fast as it can be.
If your linux start swapping often your performance will go down to the floor... Swapping is bad, specially on a XEN VPS.
The trick is to setup Mysql to use MYISAM and use Nginx instead of apache.

Here is the process list with the Resident Memory usage, after 30 days uptime and about 1,800 page views on the website.

Mysqld 5.0.54 :	7.5 MB
Thin server each :	66.5 MB
Nginx 0.6.29 :	3.5 MB (1 worker)
Postfix	4.2 MB

and others, such as sshd, cron, iptables, bash, together about 5mb.

As you can see, total of memory usage of the applications on the server is about 83 MB, thus leaving the server with 170MB of ram for the linux itself and file cache.
this is what #free command tells:

             total       used       free     shared    buffers     cached
Mem:           256        235         20          0         54         62
-/+ buffers/cache:        128        128
Swap:          511          0        511

Nice uh?
you can also make use of the nice tool called "vmstat"
it's very import that 'si' (swap in) and 'so' (swap out) stays zero all the time.
i.e. running vmstat 10 times with a 4 seconds interval. (ignore the 1st line)

# vmstat 4 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0    116  14900  56668  62692    1    1     3     3    9    5  0  0 100  0
 0  0    116  14908  56668  62692    0    0     0     0   34   53  0  0 100  0
 0  0    116  14968  56668  62692    0    0     0     0   37   54  0  0 100  0
 0  0    116  14968  56668  62692    0    0     0     0   31   52  0  0 100  0
....

Here is the recipe:

1. Use MyIsam instead of InnoDB

You can read about it more in here: http://blog.evanweaver.com/articles/2007/04/30/top-secret-tuned-mysql-configurations-for-rails/
I forgot to add that you need to dump your database first:
mysqldump -u root --all-databases > dump.sql
then change my.cnf accordingly,
restart mysql and reload the database
mysql -u root < dump.sql
Change only the values for my.cnf as shown below, and delete all innodb related stuff


# can be safely set to 1M if you are really tight on Ram
key_buffer 			       = 4M

max_allowed_packet 			= 1M
table_cache 				= 32
sort_buffer_size 			= 512k
net_buffer_length 			= 8K
read_buffer_size 			= 512k
read_rnd_buffer_size 		= 512K

# can be safely set to 1M
myisam_sort_buffer_size 	= 2M  

language 					= /usr/share/mysql/english

# security:
# using "localhost" in connects uses sockets by default
# skip-networking
bind-address				= 127.0.0.1

# No logging, 
# make sure you backup your database more often.
#log-bin

server-id 					= 1

# point the following paths to different dedicated disks
tmpdir 						= /tmp/


# Very important to have this here.
# otherwise it will still load InnoDB.
skip-innodb

[mysqldump]
quick
max_allowed_packet 			= 16M

[mysql]
# uncomment the next directive if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer 					= 8M
sort_buffer_size 			= 8M
read_buffer 				= 2M
write_buffer 				= 2M

[myisamchk]
key_buffer 					= 8M
sort_buffer_size 			= 8M
read_buffer 				= 2M
write_buffer 				= 2M

[mysqlhotcopy]
interactive-timeout

If you get problems reloading the database, stop mysql delete the contents in /var/lib/mysql/* , then run mysql-installdb and start it and reload again the sql dump file.
Actually that's the way i most prefer..

2. Use nginx

this is an example nginx config file, located at /etc/nginx/nginx.conf

user nginx nginx;
# set to 2 or 3 if you have more processors or cores. 
# it will use about 3MB per worker
worker_processes 1;

error_log /var/log/nginx/error_log info;

events {
	# default is 8192
	worker_connections  1024;
	use epoll;
}

http {
	include		/etc/nginx/mime.types;
	default_type	application/octet-stream;

	log_format main
		'$remote_addr - $remote_user [$time_local] '
       	'"$request" $status $bytes_sent '
		'"$http_referer" "$http_user_agent" '
		'"$gzip_ratio"';
									       
	client_header_timeout	10m;
	client_body_timeout	10m;
	send_timeout		10m;
        client_body_buffer_size    128k;
        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;

	connection_pool_size		256;
	client_header_buffer_size	1k;
	large_client_header_buffers	4 2k;
	request_pool_size		4k;

	gzip on;
	gzip_http_version 1.1;
        gzip_comp_level 6;
        gzip_proxied any;
        gzip_types text/plain text/html text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
        gzip_buffers 16 8k;
        # Disable gzip for certain browsers.
        gzip_disable "MSIE [1-6]\.(?!.*SV1)";
	gzip_min_length	1100;

	output_buffers	1 32k;
	postpone_output	1460;

	sendfile	on;
	tcp_nopush	on;
	tcp_nodelay	on;

	keepalive_timeout	75 20;

	ignore_invalid_headers	on;

	index index.html;
    
    # The following includes are specified for virtual hosts
    include /var/www/apps/bla.com/shared/config/nginx.conf;

}

this is an example vhost file

upstream mongrel_bla_com {
  server 127.0.0.1:8001;
}

server {
	listen 80;
	client_max_body_size 40M;
	server_name bla.com www.bla.com;
	root /var/www/apps/bla.com/current/public;
	access_log  /var/www/apps/bla.com/shared/log/nginx.access.log;

	location / {
                proxy_set_header  X-Real-IP  $remote_addr;
                proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_redirect false;
                if (-f $request_filename/index.html) {
                        rewrite (.*) $1/index.html break;
                }
                if (-f $request_filename.html) {
                        rewrite (.*) $1.html break;
                }
                if (!-f $request_filename) {
                        proxy_pass http://mongrel_bla_com;
                        break;
                }
	}

	location ~* ^.+\.(jpg|js|jpeg|png|ico|gif|txt|js|css|swf|zip|rar|avi|exe|mpg|mp3|wav|mpeg|asf|wmv)$ {
		root /var/www/apps/bla.com/current/public;
	}

}

3. Remove unnecessary Services you dont need.

Some linux distros have enabled by default services we dont need.
such as cupsd, apmd, acpid, mdns, samba, nfs, ftpd... etc...

This is my make.conf in case it helps

Note that I set MAKEOPTS="-J1" , it will only use 1 gcc process at the time, and not disturb the system, (machine has 4 cores)
Also portage_niceness to 18, to make sure it will run smooth and not disturb thin and mysql.
from nice man page: "Nicenesses range from -20 (most favorable scheduling) to 19 (least favorable)."

CFLAGS="-march=athlon64 -O2"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j1"

USE="3dnow 3dnowext apache2 \
     bash-completion bzip2 \
     -cups \
     gzip httpd hpn \
     innodb imagemagick \
     javascript jpeg \
     mmx mmxext mysql \
     nptl nptlonly \
     perl phyton png \
     ruby \
     screen sse sse2 sqlite sqlite3 ssl \
     threads udev unicode utf8 \
     vim-syntax \
     -X -kde -gnome -gtk -bindist \
     xml xml2 zlib "

GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://gentoo.cites.uiuc.edu/pub/gentoo/ ftp://gentoo.mirrors.tds.net/gentoo http://ge
APACHE2_MPMS='worker'
PORTAGE_NICENESS=18

if you want to use mod_rails Passenger, set APACHE2_MPMS='prefork'

note: I am positive you can throw in another thin server instance, and it will still not swap, or swap very little at all.
have fun

**************************

Updates :

Wanna know what Slicehost Manager Diagnostics says about my VPS ?

Diagnostics

    * Your slice is currently running.
    * The host server is up.
    * Your swap IO usage over the last 4 hours is low: 0.0016 reads/s, 0.0 writes/s. (Read more about swap here)
    * Your root IO usage over the last 4 hours is low: 0.038 reads/s, 0.1643 writes/s.
    * The host server's load is nominal: 0.00, 0.03, 0.00.
;)

An Mplayer Config that plays 1080p

2008-06-24T07:36:00Z

This mplayer configuration file can play 1080p perfectly and smooth

it was tested on Athlon X2 4000, and Sempron CPU, and only 1GB Ram (Gentoo Linux)

make sure you have your Video drivers configured and set
Such as nvidia, ati(fglrx) and intel.
I think it should also work on any P4 or Athlon XP

put this on ~/.mplayer/config
(create it if it's not there)

~/.mplayer/config


# Write your default config options here!

#======
# Video
#======

vo=xv
# Use double-buffering. (Recommended for xv with SUB/OSD usage)
double=yes

# Use 16MB of ram to cache the file
# Increase to 64mb if you have spare Ram.
cache=16380

fs=yes
ni=yes

# Video Output postprocessing quality 
# This sets the postprocessing into overdrive using all possible spare cpu cycles to make the movie look better
# if  you have older CPU such as Athlon XP or Celeron, comment this.
# Fast Machine: 90
# Slow Machine: 10
autoq=50

# use this for widescreen monitor! non-square pixels
# My monitor = 1440x900 == 16/10
# Change this to your monitor.
monitoraspect=16:10



#==========
# Subtitles
#==========


# Align subs. (-1: as they want to align themselves)
spualign=-1

# Anti-alias subs. (4: best and slowest)
spuaa=4
# try 3 if your cpu is not so fast.
# From mplayer man page:
#                 0    none (fastest, very ugly)
#                 1    approximate (broken?)
#                 2    full (slow)
#                 3    bilinear (default, fast and not too bad)
#                 4    uses swscaler gaussian blur (looks very good)


# Set language.
slang=en,eng

#===================
# Text-based subtitles
#===================

# Find subtitle files. (1: load all subs containing movie name)
sub-fuzziness=1

# Set font encoding.
subfont-encoding=unicode

# Set subtitle file encoding.
unicode=yes
utf8=yes

# Resample the font alphamap. (1: narrow black outline)
ffactor=1

# Set subtitle position. (100: as low as possible)
subpos=100

# Set subtitle alignment at its position. (2: bottom)
subalign=2

# Set font size. (2: proportional to movie width)
subfont-autoscale=2

# Set font blur radius. (default: 2)
subfont-blur=2.0

# Set font outline thickness. (default: 2)
subfont-outline=2.0

# Set autoscale coefficient. (default: 5)
subfont-text-scale=4.4

# OSD
#====

# Set autoscale coefficient. (default: 6)
subfont-osd-scale=4.4

Without this config the 1080p video plays terrible

NOTE:
1080p = 1920x1080 pixels

Easy installing Passenger mod_rails on gentoo Linux

2008-05-12T18:42:00Z

To install the great Mod_Rails on Gentoo linux it's as easy as 5 steps.

Since you are Gentoo user, i don't need to go to details. You know what you doing. ;)

1. Recompile Apache non-threaded

add this to /etc/portage/package.use

www-servers/apache -threads

and this to /etc/make.conf

APACHE2_MPMS="prefork"

2. Re emerge apache

# emerge -va apache

3. Passenger is in gentoo portage, but its in testing. Currently Version 2.0.1

# echo "www-apache/passenger" >> /etc/portage/package.keywords

4. Install Passenger

 
# emerge -va passenger

If it tries to install rails 2.0.2, rake, and lots of other gems that you already have installed trough rubygems, then run emerge with --nodeps option

# emerge -va --nodeps passenger

5. Edit /etc/conf.d/apache and add "-D PASSENGER" to apache options

for example mine looks like this:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D LANGUAGE -D PROXY -D PASSENGER"

That's it.
Now just drop a similar vhost config file inside /etc/apache/vhosts.d/

This is a sample vhost file for a rails app.

<VirtualHost *:80>
  ServerName mydomain.com
  DocumentRoot /myapp/public
  Include /etc/apache2/vhosts.d/deflate.conf
  RailsBaseURI /
  # The maximum number of Ruby on Rails application instances that may be simultaneously active. 
  # A larger number results in higher memory usage, but improved ability to handle concurrent HTTP clients. 
  # normally 1 to 10. (1 for each 50mb ram)
  RailsMaxPoolSize 1
  # The maximum number of seconds that a Ruby on Rails application instance may be idle.
  # That is, if an application instance hasn't done anything after the given number of seconds,
  # then it will be shutdown in order to conserve memory. ( 1 hour)
  RailsPoolIdleTime 3600
  RailsEnv 'production'
  <Directory /myapp/public>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
    </Directory>
</VirtualHost>

My sample deflate.conf, used to gzip the content

<Location />
	SetOutputFilter DEFLATE
	#
	# Netscape 4.x has some problems...
	BrowserMatch ^Mozilla/4 gzip-only-text/html
	#
	# Netscape 4.06-4.08 have some more problems
	BrowserMatch ^Mozilla/4\.0[678] no-gzip
	#
	# MSIE masquerades as Netscape, but it is fine
	BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
	# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
	# the above regex won't work. You can use the following
	# workaround to get the desired effect:
	BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
	# Don't compress images
	SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
	SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
	SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary
	# Make sure proxies don't deliver the wrong content
	Header append Vary User-Agent env=!dont-vary
</Location>

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/apache2/deflate_log deflate

* Update on July 10, 2008.
- Now using gentoo portage to install it. it's more smooth.
Note:
Personally I found that Thin + nginx uses less memory (Nginx 3MB, thin ~56MB) apache + passenger will use quite more. (Apache 40MB, mod_rails ~50MB)

Rails Session, be the fastest you can be. Libmemcached

2008-03-09T19:15:00Z

To use The new Libmemcached and memcached by Evan Weaver , drop these lines on your config/environments/production.rb


# See more on http://blog.evanweaver.com/files/doc/fauna/memcached
require 'memcached'
config.action_controller.session_store = :mem_cache_store
memcached_server = '127.0.0.1:11211'  
memcached_options = {
  :namespace => "my_sweet_app_#{RAILS_ENV}",
  :hash => :default
}
SESSION_CACHE = Memcached::Rails.new(memcached_server, memcached_options)
ActionController::Base.session_options[:cache] = SESSION_CACHE

memcached is up to 150x faster than memcache-client, and up to 15x faster than caffeine. See BENCHMARKS for details.

READ the docs at : http://blog.evanweaver.com/files/doc/fauna/memcached

ruby-mysql now Ruby 1.9 compatible

2008-03-07T23:26:00Z

Tommy has just released an new mysql-ruby package.

Actually 2 of them:

mysql-ruby-2.7.5 and mysql-ruby-2.8pre2

They are Ruby 1.9 compatible

Requirements

MySQL 5.0.51a

Ruby 1.8.6, 1.9.0

here is the link http://tmtm.org/en/mysql/ruby/

Greate Job Tommy!

Cashboard is Just Awesome!

2008-02-25T21:07:00Z

Cashboard App is just most awesome app I have been using lately…

I can’t say anything else… it’s freaking cool!!!

I got the free plan, which gives 2 projects and 2 logins. but I can’t hesitate to go for the Leather Plan, which is $12 / month

Features I love the most:

Can print PDFs of time sheets and invoices
Send those PDFs to your Employee or company.
Automatic Time tracking.
OSX Widget for time tracking.
Paypment Integration. ( you can get paid right there)
And finally, “Basecamp Integration”

Isn’t it just sweet???

Rails Playground Rocks

2007-12-31T10:36:00Z

I have been hosting many Rails applications in many different "SHARED" Hostings. and all I can say is that the best of them is Rails Playground...

Here are my reasons:

1. Uptime is very good.

05:38:15 up 108 days, 9:56, 7 users, load average: 0.46, 0.46, 0.45

2. Load is very Low

Load average: 0.46, 0.46, 0.45

for 2 x dual core cpus, this load is Awesome. The maximum load I have seen is like 1.9

3. Service is excellent

Every email I send I get the answer right back, with a good solution and help.

4. I asked for Git support

and I got it, they installed git right away.

5. FAST Hardware

in my rails application I get between 100 to 200 Req/sec using a single mongrel. I even get near results using FCGI !!! of course my app uses a LOT of caching, as well, using rails 2.0 edge from trunk. you might not get same results...

6. Fast connection

They have resonably fast connections... From the the other side of the world (Japan, Thailand, Malaysia) I get real good speed.

7. Free 2 mongrel Instances

I get 2 mongrel for the developer plan.

8. Free 1GB subversion repository

if i had to pay for http://svnrepository.com/ it would cost another 5 USD month.

9. All newer gems

On my machine, 76 gems installed. well updated.

SVNrepository.com is a real good choice to pay if you need more space to host many applications and more support.

So, for for the newbie and junior Programmer, RailsPlayground is definitely worth it for the "price".

Disclaimer :

IT IS JUST MY PERSONAL OPINION, for a shared Hosting I don't guarantee you will get the same service as I get. you might not get same results as I get... basically, you might not be lucky as i am. :)

NOTE:

If you have a critical application I do recommend to host on AMAZON EC2, or get your own Server Stack or Clusters.

Multi Search with Ferret, Pagination and Sorting

2007-12-28T23:56:00Z

I had to do a search with ferret for a single model and multiples queries, plus sorting in pagination.
for example, I wanted to search for a User with name="John Smith" AND city="atlanta" AND state="ga" AND zip=100*

here is my implementation

The model code, I got it from Igvita Ferret Pagination in Rails and slightly modified.



  def self.paginating_ferret_search(options)
    offset = (options[:current].to_i - 1) * options[:page_size]
    limit = options[:page_size]
    order = options[:order]
    count = total_hits(options[:q])
    PagingEnumerator.new(options[:page_size], count, false, options[:current], 1) do |page|
      res = find_with_ferret(options[:q], 
           {:offset => offset, :limit => limit, :sort => order})
    end
  end

This is the method in the controller



  def search
    #params[:d] is the direction of sorting
    if params[:d]
      sort_order = params[:c] + (params[:d] == 'down' ? ' DESC' : ' ASC')
    else
      sort_order = "name ASC"
    end
    current_page = (params[:page] ||= 1).to_i
    
    # check for each search field on the form 
    if params[:city] && !params[:city].blank?
      @city_query = "city:\"#{params[:city]}\""
      @city_label = "City: <blue>#{params[:city]}</blue>"
    end
    
    if params[:state] && !params[:state].blank?
      @state_query = "state:#{params[:state]}"
      @state_label = "State: <blue>#{params[:state]}</blue>"
      if @state_query.size != 2
        flash[:notice] = "State should be 2 Digits"
      end
    end
    
    if params[:zip] && !params[:zip].blank?
      @zip_code_query = "zip:#{params[:zip]}*"
      @zip_code_label = "Zip: <blue>#{params[:zip]}*</blue>"
    end
    
    if params[:name] && !params[:name].blank?
      @name_query = "name:#{params[:name]}"
      @name_label = "Name: <blue>#{params[:name]}</blue>"
    end
    
    # for ferret we cannot User "name ASC", have to be "name"
    if @city_query or @state_query or @zip_code_query or @keyword_query
    # params[:c] is the column name to sort
      if params[:c]
        sort_order = case params[:c]
          when "name"    then "name"
          when "city"       then "city"
          when "state"     then "state"
          when "zip"       then "zip"
          else "name"
        end
      else
        sort_order = "name"
      end
      
      # Conditions: can be either of both AND/OR
      @condition = "AND"
      # Queries for Ferret
      # My helper to build the query,
      @full_query = build_query(@full_query, @keyword_query,  @condition)
      @full_query = build_query(@full_query, @city_query,     @condition)
      @full_query = build_query(@full_query, @state_query,    @condition)
      @full_query = build_query(@full_query, @zip_query, @condition)
      # Labels  for flash[:notice]
      # I use comma to delimiter the labels in flash[:notice]
      @condition = ","
      @query_label = build_query(@query_label, @keyword_label,  @condition)
      @query_label = build_query(@query_label, @city_label,     @condition)
      @query_label = build_query(@query_label, @state_label,    @condition)
      @query_label = build_query(@query_label, @zip_label, @condition)
      
      # get the query
      @users = User.paginating_ferret_search({:q => @full_query, 
            :page_size => 100,
            :current => current_page, 
            :order => sort_order}
      )
      flash[:notice] = "<h3> Found #{@prospects.size} records for #{@query_label} </h3>"
    else
      # if no query, just return empty
      @User = []
   end
end

My application method, in application.rb add this method:



  def build_query(full_query, query, condition)
    if query
      if full_query && !full_query.empty?
        full_query = "#{full_query} #{condition} #{query}"
      else
        full_query = query
      end
    end 
    return full_query
  end

in application_helper.rb I have this little helper I found in this site
it helps to make the links in the sorting columns in the views.. <macro:code> def sort_link(title, column, options = {}) condition = options[:unless] if options.has_key?(:unless) sort_dir = params[:d] == 'up' ? 'down' : 'up' link_to_unless condition, title, request.parameters.merge( {:c => column, :d =>

OSX Leopard Sux big time

2007-12-20T13:14:00Z

This brand new OS, “The most advanced OS” sucks big time!!!

This OS is so damn unresponsive. it hangs all the time… i hate it.

(Update 10.5.2 doesnt hang anymore)

- Mail hangs, when I send email, search or read

- iTunes hangs, (becoming unresponsive)

- Safari hangs, and takes all my ram… up to 800 MB!!!

- The terminal console is lagging… and crashing. iTerm also becomes unresponsive.

- even textmate is crashing, not because of the textmate, but because of the OS.

It all happened after I installed Leopard..

TIGER WAS FINE….

I bought this DVD, 130 USD

and what I get??? this shit.

now I have to go back to tiger.

Use TIGER until apple fixes Leopard.

Use TIGER and wait for Snow Leopard (10.6)

Some Solutions:

Use OSX Tiger.
Use Firefox or Camino,
Use Thunderbird as mail client.

Update:

10.5.4 came out and still sux :-P

Faster Rails auto_complete with memcached

2007-12-17T10:43:00Z

Auto_complete by default uses full text search to the database.

That can be very slow if you your database is big...

so, let use memcached instead.

The original "roll your own" autocomplete action:



  def auto_complete_for_email_destination
    value = params[:email][:destination]
    @contacts = @user.contacts.find(:all, 
      :conditions => [ 'LOWER(email) LIKE ? OR LOWER(name) LIKE ?', '%' + value.downcase + '%', '%' + value.downcase + '%'], 
      :order => 'contacts.email ASC',
      :limit => 5)
    render :partial => 'contacts/destination_autocomplete'
  end

now, we change it to:



  def auto_complete_for_email_destination
    value = params[:email][:to_all]
    @contacts = @user.find_user_contacts(value,{:limit => 5})
    render :partial => 'contacts/destination_autocomplete'
  end

# User model

require 'digest/sha1'

class User < ActiveRecord::Base
  
  acts_as_cached
  after_save :reset_cache
  
  # find contacts with matching pattern 
  # the key to the cache is the user.id 
  def find_user_contacts(query,options = {})
    user_id = self.id
    limit = (options[:limit] ||= 5).to_i
    contacts = User.get_cache(self.id).cached(:contacts)
    # iterate through the contacts and use regexp to match it
    reg = Regexp.new(query)
    # find_all works with Enumerator Class.
    results = contacts.to_enum.find_all {|t| (t.email =~ reg) || (t.name =~ reg)}
    results[0..(limit-1)]
  end
end


# Contact mode, sweep the user cache if saved.
class Contact < ActiveRecord::Base  
  belongs_to :user

  acts_as_cached
  after_save :reset_cache
  after_save :sweep_cache_from_user


  def sweep_cache_from_user
     User.reset_cache(self.user_id)
  end

if you use shared host, you could as well use Ferret instead of memcached.

That's it folks...

Speed improvement:

before: ~ 20-40 req/sec

now: ~ 350-400 req/sec

on Amazon EC2 small AMI

Note: Experimental code... test it well before using in production...

Encrypt folders in Mac OSX with encfs

2007-12-17T08:08:00Z

OSX already include the File Vault functionality that allows you to encrypt your whole Home Folder. Thou the storage overhead is so small, the time to encrypt it the first time is very very long. if you have Videos, and big files, it takes even longer.

What if I don't want to encrypt my big folders like Movies, Music, Pictures, Pdfs?

I only want to encrypt my Documents folder. Be aware that VMWARE stored the virtual machine files under this folder, you should move it to outside Documents.

WARNING:

Be careful with this tutorial, Write down your password somewhere and BACKUP your data before going further these steps.

if you forget your password, say good bye to your data.

THERE IS NO WAY TO GET YOUR DATA BACK!!!

TOOLS required:


# update your ports to get the latest encfs that runs ok on OSX10.5
$ sudo port selfupdate

# install encfs
$ sudo port install encfs
<macro:code>

Lets move The Documents folder contents to another folder:

<macro:code>
$ cd
$ mkdir temp_documents
$ mv Documents/* temp_documents/

Create the directory to hold the encrypted files, it can be any name.

Run this only one time. The first time to setup the folder...



$ mkdir .documents

Setup the encryption



$ encfs ~/.documents/ ~/Documents/

you will see this:



fred@Macintosh ~ $ encfs ~/.documents/ ~/Documents/
Creating new encrypted volume.
Please choose from one of the following options:
 enter "x" for expert configuration mode,
 enter "p" for pre-configured paranoia mode,
 anything else, or an empty line will select standard mode.

now, after you pass this step, the file system will be mounted as well.

encfs uses FuseFS, so it behaves just like a mount point

to unmount it you do


$ unmount ~/Documents

to mount it again issue this command:


$ encfs ~/.documents/ ~/Documents/

# or this way, which will look with better names and a folder icon on Desktop:

$ encfs ~/.documents/ ~/Documents/ -- -o fsname=Documents -o volname=Documents -o local

to check mounted filesystems


$ mount

you should be able to see:


encfs@fuse2 on /Users/fred/Documents (fusefs, nodev, nosuid, synchronous, mounted by fred)

or this if you used the longer command.


Documents on /Users/fred/Documents (fusefs, local, nodev, nosuid, synchronous, mounted by fred)

Now, with the encrypted folder "mounted", mv the data from that temp folder to the new encrypted folder:

WARNING: be carefull here


$ cp temp_documents/* Documents/
$ rm -rf temp_documents/

that's it folks.

Final overview:

to create the encrypted folder: $ encfs ~/.documents/ ~/Documents/

to Mount it (enable) $ encfs ~/.documents/ ~/Documents/ or $ encfs ~/.documents/ ~/Documents/ -- -o fsname=Documents -o volname=Documents -o local

to Umount it (disable) $ umount ~/Documents

never remove .documents folder, or touch it. don't change anything inside .documents

remember the dot in the front means the folder is invisible you won't see it in Finder.

This also should work for Linux.

Make your Rails app faster with memcached. Part 1

2007-12-13T18:28:00Z

. Make your Rails app faster with memcached. Part 1

In this article I describe how to make your application increase considerably in performance

Required Applications:

1. cache_fu (plugin)

./script/plugin discover ./script/plugin install cache_fu

1. memcache_client

gem install memcache_client

1. memcached

(http://www.danga.com/memcached)



port install memcached
apt-get install memcached 
emerge memcached
yum install memcached

or get the source and compile it.

Setup cache_fu

Run memcached
configure config/memcached.yml

Parts to cache in rails:

Cache User Authentication restfull_authentication: User.current_user logged_in?

Those issue a DB query:

1234	SELECT * FROM users WHERE (users.'id' = 1) LIMIT 1

edit user model (app/models/user.rb)



class User < ActiveRecord::Base 
  acts_as_cached 
  after_save :expire_cache 
end

note: only if you are not using cookie session store from rails2.0

change the method loginfromsession in lib/authenticated_system.rb



# Called from #current_user. First attempt to login by the user id stored in the session. 
# Changed to read from Memcached 
def login_from_session 
  #self.current_user = User.find_by_id(session[:user]) if session[:user] 
  self.current_user = User.get_cache(session[:user]) if session[:user] 
end

Settings plugin I change the method self.object(var_name) in the file: vendor/plugins/settings/lib/settings.rb



acts_as_cached 
#retrieve the actual Setting record 
def self.object(var_name) 
  #Settings.find_by_var(var_name.to_s) 
  if Settings.cached?(var_name.to_s) 
    return Settings.get_cache(var_name.to_s) 
  else 
    setting = Settings.find_by_var(var_name.to_s)
    Settings.set_cache(var_name.to_s,setting) 
  end
end

then, I added a settings model: app/models/setting.rb



class Setting < ActiveRecord::Base 
  validates_uniqueness_of :var 
  acts_as_cached 
  after_save :expire_cache 
  after_save :expire_cached_var 
  
  # Clear Cache by var 
  def expire_cached_var 
    Settings.all.each do |t| 
      Settings.clear_cache(t[0].to_s) 
    end
  end
end

Basic model caching

An example of article.rb



class Article < ActiveRecord::Base 
  
  validates_uniqueness_of :permalink
  
  def self.find_permalink(permalink) 
    if self.cached?(permalink) 
      return self.get_cache(permalink) 
    else 
      article = self.find_by_permalink(permalink) 
      self.set_cache(permalink,article) 
    end
  end
end

THE BEST: partial caching:

here you can cache partials for example in the view code:

app/views/properties.html.erb



  <% @articles.each do |article| %>  
    <% cache article.permalink do %>  
      <%= render :partial => "each_article", :locals=> {:article => article} %> 
    <% end %>
  <% end %>

will create $RAILSROOT/tmp/caches/properties/propertypermalink.cache something like that, not sure what is the name

To sweep it after an update to the model:

property.rb



class Property < ActiveRecord::Base

  acts_as_cached
  after_save :reset_cache, :sweep_memcached
  
  def sweep_partial_cache
    cache_dir = ActionController::Base.page_cache_directory+"/.."+"/tmp/cache"
    unless cache_dir == RAILS_ROOT+"/public"
      FileUtils.rm_r(Dir.glob(cache_dir+"/"+self.permalink.to_s+".cache")) rescue Errno::ENOENT
      RAILS_DEFAULT_LOGGER.info("Cache directory '#{cache_dir}' fully sweeped.")
    end
  end

auto_complete_for ajax using memcached

it improved from 10 req/sec to 350 req/sec to one of my apps.

comming soon…

in part 2 of this article