Php + Apache + Mysql + [anything] Server cluster - pitfalls, hints and roadmaps to efficiency

I think uesp covered the general stuff pretty well. To decide what you are going to do for your case, there are a couple of things you need to sit down and think about:

1. What is the current load on each of these components? What is the projected future load?
2. What are the failure scenarios you want to deal with? What caused your last failure?

The first questions tell you the minimum number of servers you'll need at each level in order to have a running site.

The second questions tell you how much hardware you'll actually want to have in order to ensure your site keeps running. As you lay out the failure modes, you'll find that you're going to need to consider more than just servers: firewalls, upstream internet connections, generators, physical locations and more. You'll also need to address things like having administrators on call to deal with servers crashing at 3AM and the monitoring needed to wake up the administrator and let them know something has crashed. If your failure before was due to a configuration or programming error, consider a staging environment between development and production for testing to take place after the programmers are done with their changes and before the changes go live.

A few general things that I've learned over the years:

• See this question for a list of good books on the subject of performance, scaling and high availability sites.
• "Cluster" is the correct term. You're using multiple machines to serve one site in an attempt to increase availability. You can also use cluster to refer to specific portions of your setup: for example servers 2+3+4 would be your application cluster.
• Is there any reasons why you only have redundancy on the application level? What about MySQL and static content? Especially since your static content is relatively large look at how much bandwidth you can serve to N concurrent users if needed. What happens if the MySQL server fails or if server #4 has a bad disk?
• If you're moving everything from one machine start off small unless you don't mind spending more than you need. For example, I found a larger than expected performance gain in a similar situation moving from 1 to 3 servers. After you split into multiple servers you may find the new bottleneck is in a different area.
• As you plan for scaling now don't completely forget about possible future scaling. A little forward thought and design now can save you time in the future. For example, you have one static server now but what you want multiple in a year, or several servers spread out geographically.
• Consider creating scripts to help setup specific types of servers...doing it manually each gets old and you always forget one step. I did this recently and wish I had done it from the start. Running one script that does 50 install steps automatically in a few minutes saves you much time in the long run.
• As you get more servers the likely-hood of experiencing some sort of hardware failure becomes higher. Plan for this and play the what-if game: What if the hard drive failed on server X? What would we lose? How long would the site be out? How long would it take to fix it? etc...