Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

Random slow updates causing lock% bursts

MongoDB returns ObjectId values that are different to what is stored (using GridFS)
(34 lines)
best performing driver for mongodb
(9 lines)
Oct 14, 2011
Jon Ivmark
Jon Ivmark
Hi all,

we're having some trouble with our mongodb cluster (v 1.8.2, sharded,
replica sets on EC2 instance storage), seeing random slow updates.

We are under moderate load, 200-250 updates/second, a few inserts and
maybe around 150 reads/second.  Updates are mostly either pushing a
string to an array or incrementing counters.

All indexes and data easily fit in RAM (we are using the cluster for
session tracking and we're removing the session documents after 30
minutes of inactivity). Indexes+data is below 1G on a 7G machine.

Anyway, to our problem. Lock % is mostly hovering at around 3-6 %, but
every once in a while (once a minute or so) it spikes to 30+ % causing
some socket timeouts in our app servers (we're using a 200ms socket
timeout).

I've enabled profiling for all databases using a slow limit of 10 ms,
and from what I can see the slow writes can be pretty much anything
(stuff that for the vast majority of cases is super fast).

Any help with finding what's causing the slow writes would be much
appreciated.

mongod log for 5 minutes or so: http://pastebin.com/b4jk3K9n

I've excluded lines like this (which there are a loooot of):

Fri Oct 14 19:25:41 [conn259071] info DFM::findAll(): extent 0:2f00
was empty, skipping ahead
20135658-0dde-11e0-855b-12313b032222.sessions

mongostat for the same period: http://pastebin.com/4BTZcQLs

iostat -x 2: http://pastebin.com/CfQF5R8J

Again, thankful for any help.

Jon





Reply
Tags: instance storagemoderate loadslow updates
Messages in this thread
Random slow updates causing lock% bursts
reply Re: Random slow updates causing lock% bursts
(53 lines) Oct 15, 2011 00:06
reply Re: Random slow updates causing lock% bursts
(74 lines) Oct 15, 2011 02:32
reply Re: Random slow updates causing lock% bursts
(84 lines) Oct 15, 2011 03:53
reply Re: Random slow updates causing lock% bursts
(11 lines) Oct 17, 2011 15:41
ngx_http_read_client_request_body() causing random timeouts
May 31, 2011 03:08:44 AM
Hi, I've been trying to identify what is causing timeouts towards the client when a POST request is sent to nginx. I have a custom module that reads the POST payload and runs in ACCESS phase. It's based on nginx-form-input module from agentz. …
During unittest, changes not applied causing random test failure.
January 25, 2011 03:43:00 AM
In my python unit test, I have tests that randomly failed. They look like this: 1. db = pymongo.Connection() 2. assert db.profile.find_one({'_id': 1})['c'] == 2 3. db.profile.update({'_id': 1}, {'$inc': {'c': 1}}) 4. assert…
Created] (HDFS-2152) TestWriteConfigurationToDFS causing the random failures
July 14, 2011 02:26:46 PM
TestWriteConfigurationToDFS causing the random failures
lock-free counter updates
May 24, 2011 02:32:35 PM
Hello all, In adding support for the sFlow monitoring standard to the latest memcached (see http://code.google.com/p/memcached/issues/detail?id=202) I used a scheme for lock-free counter accumulation that might make sense for the memcached stats…
concurrent updates / lock-handling
September 27, 2010 08:29:32 PM
hello, i am currently evaluating riak as data-storage for out upcoming project. the wiki provides a lot of informations and is really good - most of my questions are answered, but i did not find much information about how riak handles concurrent…
Spike of write lock percentage causing high response time
January 21, 2011 01:04:30 PM
To whom this may concern, Recently our mongodb instance (1 node) is experiencing an unusual spike of high write lock percentage (~4%). During these unusual spike periods the CPU percentage usage went up to 390%+ (it is a quad core) and the load…
When are updates slow
September 3, 2011 07:28:59 AM
Hi, Am unable to guess why the following update could be slow, there are about 53,000 such objects in the db currently. Any thoughts? Is it dependent on the size of the object saved? In this case, obj.to_json.size is 10,400. [conn863034] update…
How do index updates affect lock % (versus regular writes that do not)?
October 5, 2011 11:12:00 AM
We have a large collection of documents indexed on a single primary key. We have a process that adds and deletes several thousand of these documents a second (therefore modifies the index). We have another process that does several million updates…
How do index updates affect lock % (versus regular writes that do not)?
October 5, 2011 02:29:26 PM
We have a large collection of documents indexed on a single primary key. We have a process that adds and deletes several thousand of these documents a second (therefore modifies the index). We have another process that does several _hundred_…
Http sub protocol slow for random reads
June 21, 2011 01:07:58 PM
This is a multi-part message in MIME format. So I am doing a lot of random reads and I am getting only 1-2 lookups per second where I used to get thousands per second. I did an HPROF and found that the root cause relates to the Sun http protocol…
Created] (MATH-585) Very slow generation of gamma random variates
June 4, 2011 01:39:49 PM
Very slow generation of gamma random variates
Updates to at least synchronize with slow changing key topics
August 26, 2011 10:26:17 AM
Hi guys, I know that there's a lot of flux now in the code and that makes certain initiatives a moving target when it comes to documentation. When you actually finish writing the documentation the subsystem you are documenting might have made a…
updates getting very slow - wrong data model?
August 17, 2011 12:09:53 PM
Dear all, I am suffering from low performance over time during data import, as can be seen in the following graph: http://treebank.info/mongomonitor5.png What we're trying to do is to load Wikipedia into MongoDB and represent every sentence as…
bursts of DBException in process: socket exception
February 24, 2011 06:56:54 AM
We have four mongod shards and four mongos processes, with three configuration servers, running 1.7.6. A few times per day, our mongos logs show bursts of: .... Tue Feb 22 18:01:59 [conn2056362] DBException in process: socket exception Tue Feb 22…
Created] (HDFS-2103) Read lock must be released before acquiring a write lock
June 23, 2011 04:11:16 AM
Read lock must be released before acquiring a write lock
How can you have an Exclusive Lock on nothing? please help me understand this lock monitoring query
April 14, 2011 08:13:53 PM
Hi. I use the following query (from http://wiki.postgresql.org/wiki/Lock_Monitoring) to monitor locks; and I've got an ExlusiveLock that does not have a relation name associated with it. What is locked with the Exclusive Lock in this case,…
Is it possible to determine how many updates are in place updates?
April 5, 2011 11:59:48 AM
Are there any stats on this or logging that can be turned on ?
Long pauses during updates updates?
October 8, 2010 07:33:25 PM
I'm doing a one time batch update where I'm looping through about 250k documents and updating a field in each one with $set (using PyMongo). Every 500 documents or so, the loop freezes for 1-3 minutes. db.currentOp() shows a different item being…
Lock in_use.lock failed when using nfs in namenode
June 26, 2011 03:34:11 AM
Hi I want to use nfs to save a copy of the data in namenode, and the nfs dir /mnt/nfs/dfs/ can be read and written. Things seem to be right, but when i format the hdfs, "ERROR common.Storage: Cannot create lock on /mnt/nfs/dfs/name/in_use.lock"…
Is db.repairDatabase(); only write lock or read lock ?
August 22, 2011 08:00:34 AM
Hi guys, I am trying to find some answers whether once db.repairDatabase(); is running I can still do selects/read operations and only writes are locked or everything is locked ? many thanks, /Marcin
Can you lock updates which will change too many rows in a database?
March 22, 2011
Is there a way of implementing a rule in mysql which will block updates effecting over 10 rows? For instance, lets say i have tbl1 with 50,000 rows…
Obtaining Read/Write Lock for Sqlite Parallel Updates
April 4, 2011
Hi there! I write a lot of parallel scripts in python for research purposes and was wondering if it is possible to obtain a read/write lock manually…
CouchDB - slow updates
April 15, 2011
Hello, Im using CouchDB in my project. LVS + 2 couch node. But in DB where is 250000 documents, update of single document take up to 30s! is it…
Sqlite updates slow (15 seconds for 1720 records) on SSD disk
May 20, 2011
Dear fellow developer, for some reason updates for 1720 records takes around 15 seconds when on SSD disk (especially when having trim enabled). I…
MySQL slow query - "Waiting for query cache lock"
March 18, 2011
I am running a simple query on a simple table on the same machine as the server running 5.5. It is taking 22sec to return ~7000 rows from a 20…
MySQL table updates, but not fast enough for a select statement on the same page to access the updates - any ideas?
February 18, 2011
I am trying to write some code that updates a mysql table, and then selects out of that same table in the same page. However, I find that when I do…
MySQL InnoDB dead lock on SELECT with exclusive lock (FOR UPDATE)
March 25, 2011
I do this to ensure only once instance of this process is running (pseudo code php/mysql innodb): START TRANSACTION $rpid = SELECT `value` FROM…
Mysql sequential updates or simultaneous updates
March 5, 2011
Hello, I would like to ask, whether it is more optimum to perform a simultaneous update in 3 mysql tables using a joined query or perform each in…
Each status update in Facebook inserts a new row in table or updates a field containing a list/CSV of status updates?
February 1, 2011
analyzing Facebook. When a user posts something on his wall.. these updates are stored in newly inserted different rows or newly added different…
MySQL: How to retrieve a random row or multiple random rows?
January 3, 2011
I have a MySQL database table that stores the URLs of photos. I need to pull 5 random records from the database of a particular type. I can pull 5…
PL/SQL rownum updates
March 18, 2011
Hi, I am working on a database with a couple of tables. They are a districts table PK district_id student_data table PK study_id FK district_id…
SSL installed but no lock
February 15, 2011
I have installed ssl on a server .when i try links using https they work but i don't see a lock. Any suggestion on the issue?
SSL installed but no lock
February 15, 2011
I have open ssl installed on the server, all the key ,csr and crt on the server. Configured apache conf to the correct path for key and cert but i…
How to lock file in PHP?
February 5, 2011
Hello. I'm trying to create a PHP file, which wouldn't run if it's already running. Here's the code I'm using: <?php class Test { private…
How to LOCK a row in MySQL ?
May 2, 2011
I'm working on a CRM desktop application which is going to be used by more than one agent at a time, and all agents will go through the same list of…
If I could lock a few tables, would it be bad?
June 23, 2011
Let's create Page.php query1 query2 query3 Is there any way to lock a few tables, so if one user visits Page.php and second user opens this page a…
RSA is causing me headaches
March 1, 2011
Situation I want to encrypt/decrypt byte[] data using RSA in my C# porject. I use OpenSSL.Crypto this is my code byte[] msg =…
Row lock in mysql
February 13, 2011
I have to do some network IO based on every row in a table with more than 70 million rows. Since high TPS is needed i have created a php script that…
Is MongoDB built for a lot of updates?
January 4, 2011
I'm creating a blog. Everytime someone views the page of a blog, can I do this: post['views'] += 1 db.collection.save(post) I don't index 'views'.
Are Redis updates synchronous?
June 7, 2011
If I push something onto a list in Redis, then pop from that list, is it guaranteed that I will get the item I pushed earlier or is it possible for…