Load balancing, high availability for webapp setup

We have build a special interest social network and need to setup now the proper server setup for the webapp servers.

We have for the webapp 5 physical server available (2xE5645 CPU, 24 GB Ram, RAID, 4x Gbit NIC), one of the five server need to provide the API for our mobile visitors (iOS, Android). Our webapp need PHP, APC, Memcached and MySQL.

Also we have 4 more physical server inside (2xE5620, 12 GB Ram, RAID, 4x Gbit NIC):

• Comet push server (custom server for the chat)

• Queue/Mail server (beanstalkd)

• static webserver (G-WAN webserver) with 2xDAS with 12x2TB (total 40 TB in Raid 6)

• Apache webserver dedicated for HTTPS (this server can also be used for the LB cluster if the SSL secured page will be put on the LB cluster, is this better?).

For the beginning we only want to have the webapp server in LB/HA, because we don't have more servers available at the moment.

And a last is a low cost server (X3430, 4 GB Ram, RAID, 2x Gbit NIC) that is used for monitoring all Hardware.

We have 2x HP Procurve 2900 managed 48 port Gbit Switch.
All the above equipment stay in our Rack in a network center. We use RHEL 6 on all servers.

I have much ideas, but I'm not really sure which will be the best way.

My direction is go the way to install HA Proxy and Apache webserver on two boxes, have on two boxes MySQL DB Server and on one both Apache and MySQL for the API/Webservice. To use one of the switch for private network where all server will be connected and used for internal communication (MySQL, File transfer after upload).

Is it helpful to connect multiple NIC ports per server to the Switch?

Is HA Proxy the best way for our case? Or would it be better to use nginx on 2 or 3 boxes and to use it for load balancing and serving PHP and have LVS or some similar for the HA part?

I'm open for all solutions, and the 5+1 server can be used flexible.

Thanks for any help.