Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

idea about slave snapshotting option

clean up a database
(24 lines)
Scaling out Redis
(16 lines)
Jan 19, 2012
Jak Sprats
Jak Sprats
Hi All,

I mentioned this idea for the windows port, but it may not be a bad
idea for many setups.

The idea is to have 1 master feeding 1 slave.
The master does not persist data to disk, the slave does in-process
SAVEs.
During the slave's in-process SAVE, it would still accept requests and
write CRUD commands to a Q.
After the slave's SAVE was done, it would replay the Q (in their
naturally occurring order).

The benefit this type of architecture has is it does not run the risk
of a COW memory explosion on the slave.
The downside is if the SAVE takes a long time, the Q could get very
big and also use up lots of memory.

I have the feeling (no hard data) that Q-ing commands (which are
delivered @ NIC speed) does not run the same risk of HUGE memory
explosions as COW does, or in other words, the Q-size is more
predictable.

This would be a big win for people running close to their RAM limits
on both master & slave.

Something about the idea seems a bit goofy, so I was wondering if
anyone had feedback on if this would be a good idea or if there is
something about it that makes it just a completely lame idea

- jak





Reply
Tags: windows portbad idea
Messages in this thread
idea about slave snapshotting option
reply Re: idea about slave snapshotting option
(49 lines) Jan 24, 2012 11:02
reply Re: idea about slave snapshotting option
(31 lines) Jan 24, 2012 11:59
reply Re: idea about slave snapshotting option
(27 lines) Jan 24, 2012 22:31
reply Re: idea about slave snapshotting option
(31 lines) Jan 25, 2012 12:37
reply Re: idea about slave snapshotting option
(40 lines) Jan 25, 2012 13:01
reply Re: idea about slave snapshotting option
(88 lines) Feb 3, 2012 04:59
reply Re: idea about slave snapshotting option
(106 lines) Feb 6, 2012 10:31
reply Re: idea about slave snapshotting option
(137 lines) Feb 7, 2012 06:31
Is slave-based-snapshotting necessary to maintain decent response times?
October 21, 2011 12:01:09 PM
Hi, I have a redis database which is around 1.4GB in memory / 350MB on disk. Every 5 minutes, redis would snapshot itself to disk, and at that same time, redis-stat would show latency jumping from 0.25ms to as high as 20 seconds. After a bit…
PATCH/puppet 2/3] [#4090] Git rid of the idea that run_mode is a configurable option with defaults
June 30, 2010 05:42:41 PM
Along the way this fixes an issue with 2.6 alpha that sections of the puppet config file were getting ignored. Signed-off-by: Matt Robinson <matt@puppetlabs.com> --- lib/puppet.rb | 2 +- lib/puppet/application.rb …
EBS Snapshotting
October 28, 2010 12:32:23 PM
How long do we need to lock the database when doing an initial snapshot of our EBS volume? Do we need to prevent writes to the database during this entire process? I seem to recall Eliot saying that we would only need to hold the lock while the…
Snapshotting to a different volume?
May 18, 2011 07:38:45 PM
As of 0.8.0, is it possible to take a Cassandra snapshot to a different volume (like a EBS volume dedicated for backups)? About a year ago, Jonathan Ellis said that this won't be implemented b/c snapshots are basically hard links:…
Configuring periodic snapshotting?
July 1, 2011 04:02:46 PM
Hi folks, elasticsearch/modules/elasticsearch/src/main/java/org/elasticsearch/ action/admin/indices/gateway/snapshot/GatewaySnapshotRequest.java says: * Gateway snapshot allows to explicitly perform a snapshot through the gateway of one or more…
Redis-cluster and persistence/snapshotting
October 28, 2010 05:08:54 PM
Antirez, can you talk more about cluster and persistence? Do you expect all the nodes in the cluster to have the same configuration? Could you set it up so, for example, only slaves do persistence? Would be nifty if a slave got promoted, he…
Created: (AMQ-2810) Pure Master/Slave - Expired messages not replicated to Slave
July 6, 2010 04:20:51 AM
Pure Master/Slave - Expired messages not replicated to Slave
Created] (AMQ-3376) Master/slave + cluster leaves messages on slave
June 23, 2011 06:37:16 AM
Master/slave + cluster leaves messages on slave
Pure Master/Slave Bug? Expired messages not replicated to Slave - update
July 5, 2010 08:59:17 AM
Update: Same problem with ActiveMQ 5.3.0, current 5.3.3- and current 5.4-SNAPSHOT. Any ideas? Andreas -----Ursprüngliche Nachricht----- Von: Weber, Andreas, M-ED Gesendet: Montag, 5. Juli 2010 09:47 An: 'users@activemq.apache.org' Betreff:…
Created] (AMQ-3489) Slave not replicating consumption with Pure Master Slave
September 8, 2011 10:57:32 AM
Slave not replicating consumption with Pure Master Slave
Updated: (AMQ-2485) Pure Master Slave documentation should describe how to recover from slave failur
September 18, 2010 11:47:04 AM
[ https://issues.apache.org/activemq/browse/AMQ-2485?page=com.atlassian. jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Snyder updated AMQ-2485:
Updated] (AMQ-2485) Pure Master Slave documentation should describe how to recover from slave failur
April 1, 2011 06:26:59 AM
[ https://issues.apache.org/jira/browse/AMQ-2485?page=com.atlassian.jira .plugin.system.issuetabpanels:all-tabpanel ] Gary Tully updated AMQ-2485:
Re: Tomcat does not start for slave activeMQ Broker in jdbc master slave cluster environment
January 6, 2011 09:55:45 AM
Hello Dejan, > but you cannot have a slave broker in an active Tomcat instance. thanks. That is what was troubling me. An identical config with standalone brokers worked. Is there any specific reason why embedded brokers and master/slave…
Updated: (AMQ-2485) Pure Master Slave documentation should describe how to recover from slave failur
December 2, 2010 07:11:43 AM
[ https://issues.apache.org/jira/browse/AMQ-2485?page=com.atlassian.jira .plugin.system.issuetabpanels:all-tabpanel ] Dejan Bosanac updated AMQ-2485:
Created: (AMQ-2969) Connection pool exhausted in JDBC master-slave when slave tries to acquire lock
October 8, 2010 12:19:04 PM
Connection pool exhausted in JDBC master-slave when slave tries to acquire lock
Pure Master/Slave Bug? Expired messages not replicated to Slave
July 5, 2010 02:47:11 AM
Hi, I use a Pure Master/Slave configuration with ActiveMQ 5.3.2. Master/Slave both use the same DLQ configuration with: processExpired="true" processNonPersistent="true" The normal Master/Slave processing seems to work correctly, actions on…
Querying from slave in Master-Slave Replication with PHP Client Driver
October 19, 2010 04:15:07 PM
Hi, According to this documentation: http://us3.php.net/manual/en/mongocursor.slaveOkay.php by default slave can't be queried. But I am being able to query from slave even though I haven't enabled MongoCursor::$slaveOkay flag variable. PHP Driver…
Master/slave + cluster leaves messages on slave
June 22, 2011 08:38:44 AM
I've setup 4 servers (v. 5.5.0) this way: http://activemq.2283324.n4.nabble.com/file/n3617053/amq_setup.001.jpg I set up consumers on Node 1 and Node 3. I start producing messages to Node 1. Messages are distributed (load balanced) across Node…
Slave not replicating consumption with Pure Master Slave
August 24, 2011 08:03:12 PM
I'm seeing some strange behavior with ActiveMQ 5.4.2 pure master slave setup... 5.4.2 master running on mqmaster01: <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data"…
Re: Issue 156 in redis: Slave-of-Slave replication can cause
August 24, 2010 09:06:00 AM
Updates: Status: Verified Comment #3 on issue 156 by antirez: Slave-of-Slave replication can cause http://code.google.com/p/redis/issues/detail?id=156 Just fixed in master. Thanks, Salvatore
In IntelliJ IDEA, how do I make a Tomcat Server's "Peform on 'Update' action" option sticky?
June 22, 2011
I have a Tomcat Server running as a saved configuration in IntelliJ IDEA 10.0.3. I've saved the "Peform on 'Update' action" option to "Update…
Good Idea/Bad Idea? Using MySQL RAND() outside of a small set of subquery results?
January 16, 2011
So in MySQL, I've read that for large tables with lots of rows, using ORDER BY RAND() is a bad idea (even with ~500 row tables, supposedly). Slow…
Wordpress using Easy Post Types sorting by value in option string instead of option value
March 28, 2011
I have a post type called members, which has a field called "business-type". To display the members on the page sorted by business type I'm doing…
Mk-table-sync in a master-slave scenario: Changes not replicated to the slave.
May 30, 2011
Hello to everyone. I have being employing mk-table-sync to synchronise tables from a master to slave on mysql 5.1. Unfortunately, while differences…
Create a MySQL slave from another slave, but point it at the master
April 8, 2011
Problem I have MySQL replication setup between 2 servers, master ( A ) and slave ( B ). I need to add a new slave to the mix ( C ). I want this…
Is there a way to batch create DNS slave zones on a new slave DNS server?
January 11, 2011
I currently have a DNS server which is serving as a master DNS server for a number of our domains. I want to set up a brand new secondary DNS…
Master-slave-slave replication: master will become bottleneck for writes
December 26, 2010
hi, the mysql database has arround 2TB of data. i have a master-slave-slave replication running. the application that uses the database does read…
Mysql master-slave-master-slave replication
January 20, 2011
Hi. I have two phisical servers. I copied some databases from server1 to server2 using command: server1$ mysqldump -u root -q -p…
Don't have the idea to to design my DB
May 15, 2011
don't have the idea to to design my DB how can i design a database for a currency exchange site, any help
Any idea how to implement this?
January 28, 2011
Any idea how to implement this (http://fluin.com/63) using MySQL+PHP+Javascript(mootools)? In a nutshell, it's a realtime threaded conversational…
Innodb option in my.cnf
December 29, 2010
Is innodb_file_per_table faster than innodb_data_file_path in select ?
View more.... option
March 17, 2011
I am using limit to display the latest 3 entries. If there are more than 3 entries how can i show the option "view all". HOw can i tweak the code or…
GWT App through tomcat on Intellij-Idea 10
May 13, 2011
Sounds simple, but I'm having lots of issues. I want to run a GWT project in a tomcat on Intellij-idea 10. Does anyone know how to set up the run…
Project idea for hadoop
April 4, 2011
HI Im 3rd year of college student major in software engineering and had few experiences on HADOOP.i looking for a idea of small to medium size…
Good idea to use colons in url?
February 7, 2011
Hey I'm building a small system, which has nice htaccess rewritten urls. Now I'm wondering if it's a good idea to use colons for it. Here's an…
Php-cgi causes randomly segfault, any idea?
April 24, 2011
php-cgi randomly dies and causes kernel segfault. After 1 or 2 days, system halts when I have to reboot the server then. error message are like…
Backing up a DB with Git - a good idea?
April 27, 2011
The way I see it dumping a PostgeSQL DB into one big SQL file and then committing and pushing to a remote Git repo can be a terrific backup…
Any idea how this can be read as a CSV? - Python
January 20, 2011
Hi folks, I have a CSV formatted in the following way: ID=123[DLE]KEY=test[DLE]KEY2=VAL123 where [DLE] is the "Data link escape" control character…
Need to use a Database but no idea how to do that in Android
February 20, 2011
Need to use an existing Database Sqlite (that i add in the Assets folder) but don't know how to deal with and i did not see any complete example…
Php IS NULL globalize option?
March 4, 2011
I have a query that is selecting about 20 columns, all are varchars. Some will be empty, which is just fine. But I want to put a default string of…