ElasticSearch on EC2 - runs into problem recovering when one of the nodes times out then recovers

This is on 0.15.2-1 (I can't easily move up to 0.16.* in the near
future, so just let me know if this has been fixed in a more recent
version and I'll shut up - from a quick scan I couldn't see any
candidate fixes in the change files though)

On a 2-node (test/reference) cluster running on EC2, one of the nodes
("node A") ran out of memory (due to one of the other processes) and
hung for a few minutes (eg no ssh connectivity) before the offending
process was killed and it returned to normal.

During that time "node B" detected a loss of connectivity to "node A"
and removed it. When "node A" recovered, it did not get added back to
"node B"'s list.

So at that point:
 - nodeA believed it was part of a 2-node cluster (and shared new
documents posted to it)
 - nodeB believed it was part of a 1-node cluster (and obviously did
not pass any new documents to nodeA)

The log file entries from "node B", which might make the above clearer
(just pasted below below since it's so short):



My YML configuration is very simple (just pasted below below since
it's so short):



Once I restarted node B (first manually deleting all its documents to
make life simpler), everything returned to a working state (with "node
A" as the master and "node B" as the slave).

Again - apologies if this is already fixed. If not, shall I create an
issue? Not 100% what the desired functionality should be ("node A"
regains mastership? "node A" becomes a slave?), but presumably not the
above :)

Any thoughts appreciated - and great job on ElasticSearch!

Alex
apig### @ikanow.com