MySQL on a dedicated server vs increasing RAM on server with MySQL + Rails app

first understand where's your bottleneck. is your app mostly reading from sql? writing? if you mostly read and your data set no longer fits in memory probably adding more ram will give you performance boost. but maybe data already fits in memory and mysql hogs the cpu [eg it does full table scans because it cannot make use of index in some circumstances like subqueries]

few advices:

• start with monitoring what's going on with your server eg using munin.
• enable slow query logging
• analyze what queries do you send to sql - eg with ¹mysql.com/mysqlsla" rel="nofollow" target="_blank">mysqsla - maybe after all you do unnecessary full-table scans?
• use explain for each of most frequent querries

only after that decide weather to:

• get more cpu power [and move mysql to another machine]
• buy more ram
• optimize your code