Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

Problems with slow update

backup mongodb data.
(26 lines)
Configuring virtual IPs
(25 lines)
Feb 8, 2012
Gbalthasar
Gbalthasar
Hi,

on a single MongoDB 2.0.2 instance I have collection with documents
like this:

 db.assortment.findOne()
{
	"_id" : ObjectId("4f3279b85e53cee38eefa2fa"),
	"pk" : "2j1IRLU8g0kSvJv9CBYV58-1",
	"code" : "323948-40",
	"referenceNr" : "TLP LV6",
	"status" : {
		... small subdocument
	},
	"exclusiveCompany" : false,
	"masterCode" : "324948",
	"productManagerCode" : "Y_RINGER",
	"manufacturer" : {
		... small subdocument
	},
	"modulePk" : "SPWUfGcRlHRLSp1csHvW58-1142",
	"mprgPk" : "UdWfxpZ0mHRLSp1csHvW58-1142",
	"prgPk" : "D5sezwJ3nHRLSp1csHvW58-1142",
	"visible" : [
		"iQJtnEnq0rxBxmWbsHvW58-32"
	],
	"visibleIso" : [
		"de.CH"
	],
	"searchable" : [ ],
	"searchableIso" : [ ],
	"localized" : {
		"iQJtnEnq0rxBxmWbsHvW58-32" : {
		... small subdocument
		}
	},
	"property" : {

	},
	"bios" : [ ],
	"context" : [
		... around 50 entries
	],
	"update" : null
}

and the following stats after a fresh import:

 db.assortment.stats()
{
	"ns" : "hardware.assortment",
	"count" : 263089,
	"size" : 1082690256,
	"avgObjSize" : 4115.300358433838,
	"storageSize" : 1247039488,
	"numExtents" : 22,
	"nindexes" : 9,
	"lastExtentSize" : 215420928,
	"paddingFactor" : 1,
	"flags" : 0,
	"totalIndexSize" : 1641332000,
	"indexSizes" : {
		"_id_" : 8985424,
		"pk_1" : 15755152,
		"code_1" : 9770320,
		"exclusiveCompany_1" : 8617504,
		"context_1" : 1476798176,
		"bios_1" : 16262064,
		"searchable_1" : 56921312,
		"visible_1" : 37339792,
		"referenceNr_1" : 10882256
	},
	"ok" : 1
}

When doing an update it takes 60 seconds to update 1211 document:

Wed Feb  8 14:59:22 [conn8] update hardware.assortment query: { pk:
{ $in: [ <1211 values like "DgGf03K9oUQJA5x9CBYV58-1"> ] } } update:
{ $push: { bios: "b_SY96C2vGBKRLSp1csHvW58-26" } } moved:1 69318ms

and after the update the collection stat looks like:

 db.assortment.stats()
{
	"ns" : "hardware.assortment",
	"count" : 263089,
	"size" : 1087657972,
	"avgObjSize" : 4134.1826226105995,
	"storageSize" : 1247039488,
	"numExtents" : 22,
	"nindexes" : 9,
	"lastExtentSize" : 215420928,
	"paddingFactor" : 1.409999999999991,
	"flags" : 0,
	"totalIndexSize" : 1645444528,
	"indexSizes" : {
		"_id_" : 8985424,
		"pk_1" : 15746976,
		"code_1" : 9762144,
		"exclusiveCompany_1" : 8642032,
		"context_1" : 1480133984,
		"bios_1" : 16360176,
		"searchable_1" : 56954016,
		"visible_1" : 37977520,
		"referenceNr_1" : 10882256
	},
	"ok" : 1
}

What I'm doing wrong? A find or count on the very same query object
used for update returns within 50-300 ms so why does it take that long
to update 1211 documents?

Addiotional Info:

In the time the update runs, the following lines appear in log:

Wed Feb  8 14:58:13 [conn8] old journal file will be removed: ../data/
local/journal/j._33
Wed Feb  8 14:59:19 [conn8] old journal file will be removed: ../data/
local/journal/j._34
Wed Feb  8 14:59:19 [conn8] old journal file will be removed: ../data/
local/journal/j._35

Does this mean MongoDB needs 400 Megabytes of journaling to update
1211 documents?

Greetings,
Gerhard Balthasar





Reply
Tags: assortment
Messages in this thread
Problems with slow update
reply Re: Problems with slow update
(163 lines) Feb 8, 2012 08:18
reply Re: Problems with slow update
(188 lines) Feb 9, 2012 00:52
reply Re: Problems with slow update
(857 lines) Feb 9, 2012 01:24
reply Re: Re: Problems with slow update
(873 lines) Feb 9, 2012 04:33
reply Re: Problems with slow update
(354 lines) Feb 9, 2012 09:22
Performance Problems on insert / update. growing update time
August 19, 2011 06:42:00 AM
hi, i am currently developing a big catalog applicaton. the target ammount of data is 3-4mio articles with 40mio of attributes in collections(embedded). each article has about 10 fields, 3 indexes are required for updating / finding items. I use…
Slow queries and Indexing Problems
August 2, 2010 10:18:19 AM
We are having several issues with the queries and developing an indexing strategy for a query like: query: { timestamp: { $lte: 1280752012610 }, query_tags: { $in: { 0: "group_id-10-0-1" } }, source_group_id: { $in: { 0: 551, 1: 408, 2: 285, 3:…
slow system problems after performing sharding
July 15, 2011 02:44:29 AM
We have a system running mongo as backend, this originally consisted of 1 server non-sharded. The system ran fine but it was beginning to get to the limits of the hardware. As such we decided to add some shards to spread the load. During the…
Why update with upsert is so slow?
December 2, 2010 04:05:48 AM
hi, I use $push do a update operation. Every consumer record have a message history array attribute, the cap size of the array is about 30. I need append the data. the code just like this. record = {"$push":{"hist":[ {"cid":i, "d":…
slow findAndModify() + update()
February 3, 2012 02:51:54 AM
Hi there I am loading up ~0.5M simple/small documents into mongodb and manipulating them with a simple java/groovy client. The performance of the client is adequate in the beginning, but it gets progressively slower and slower as the client…
Extremely slow bundle update under Oracle 11
December 30, 2011 03:54:11 AM
Hello, We are measuring performance of our application on the larger database under big load. Jackrabbit tables are in the same database as application tables. We use OraclePersistenceManager. Oracle: 11,2,0,1,0 Jackrabbit: 2.2.10 According to…
Closed: (FOR-1061) svn update is very slow on our zone server
January 8, 2011 04:05:31 PM
[ https://issues.apache.org/jira/browse/FOR-1061?page=com.atlassian.jira .plugin.system.issuetabpanels:all-tabpanel ] David Crossley closed FOR-1061.
Commented: (FOR-1061) svn update is very slow on our zone server
January 8, 2011 02:37:27 PM
[ https://issues.apache.org/jira/browse/FOR-1061?page=com.atlassian.jira .plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12 979196#action_12979196 ] Brian M Dube commented on FOR-1061:
update problems
February 3, 2012 04:10:51 PM
sqlite help, If someone has the time I would appreciate it if you could run this module. Everything works except the class Kupdate. For some reason I don't get any complaints from python or sqlite but when selecting from the table I cannot …
Update about subtree problems
July 19, 2010 06:34:47 PM
Hi guys, just an update about the current situation. We discussed on this ML about options regarding how to handle subtrees in the most efficient way in the server. We currently have 3 possibilities : - the first one, which is partially…
Problems with $ operator and update in PHP
December 10, 2010 03:55:10 PM
Hi, I'm trying to update the array of a document in PHP and use the following statement: $this->usuario->update(array("_id" => new MongoId("4d006dad89c950ae01000000") ), array( 'amigos.$.estatus' => '1')); But can't update, mongo…
problems running sa-update
December 29, 2010 04:46:51 AM
hi, i have a self-compiled installation of SA 3.2.5 on a rather old redhat system (RH 7.3, perl v5.6.1). SA is installed in /usr/local/SpamAssassin-3.2.5. when i run "/usr/local/SpamAssassin-3.2.5/bin/sa-update -D" i get the following error: …
Mongo one to many relation and update problems
July 28, 2010 12:28:59 AM
Hi guys, I'm implementing blog based on MongoDB. Let's look at first how Mongo guys recommend us to store blog post and its comments (http://www.mongodb.org/display/DOCS/Schema+Design): * posts should be a collection. comments should be…
Created: (MAPREDUCE-2149) Distcp : setup with update is too slow when latency is high
October 25, 2010 11:15:48 AM
Distcp : setup with update is too slow when latency is high
Problems with document update followed by attachment creation
August 17, 2010 05:53:43 PM
Hello, I am using jcouchdb in a groovy project. I stumbled into serious problems when I update a document and afterwards create some attachements for this document. I always get a "java.net.SocketException" (sometimes broken pipe, sometimes…
update, updateMulti is slow, is "find" and "save" faster?
February 24, 2011 09:13:46 AM
hello, i'm accessing mongo 1.6 via java i have to do many update,updateMulti queries, (at least 100,000) i'm using the java command "update" and "updateMulti" when possible. In java: i am using an executorService and generating a new task for…
select statment going slow and slow while using IN (xx,xx)
January 12, 2012 02:48:49 AM
I am transforming a db with millions records to anther schema. for some reason I need to select the records using IN (xx,xx). the ids in the IN was about 1000 recored every time, but I found the query was getting slow while the selection moving…
Problems renaming components inside blueprint context (Reposting due to nabble problems)
March 5, 2011 10:05:57 AM
Hi, I'm currently evaluating camel-blueprint with karaf. In my current project (to be ported to this runtime), running standalone camel-spring, I am able to give a different name to a component by manually instantiating it in the spring xml…
Slow queries(~15seconds) on cluster - but no slow queries in profiler
December 14, 2011 02:32:02 AM
I'm intermittently getting very slow response times on queries via mongos. Cluster layout: - 10 monos - one on each node - 10 shards - Each shard has 1 master, 1 slave, 1 arbiter - Currently each shard is active on 1 node - profiling is…
iPOJO and CamelContext problems (was "Problems triggering a camel route in OSGi")
May 30, 2011 07:50:18 AM
I sent the mail below to the Camel mailing list but I'm beginning to think that it is more of an iPOJO problem than a Camel problem. I would really appreciate any help since this is extremely confusing to me. /Bengt I have a very strange problem…
MySQL: UPDATE too slow
February 19, 2011
The following is a benchmark of a series of queries I have to run. As you can see, the UPDATEs are significantly slower than the other queries. What…
Android update row problems
April 17, 2011
In the app we are putting together I am trying to update rows from different Activities but it is throwing an error or not updating anything because…
MySQL update with subselect too slow
June 2, 2011
Having an issue with an update query taking more than 20 minutes (I kill it after that). Scenario: Table one has some 300K records. Table two…
Update too slow from table with one record
March 24, 2011
Please consider the table bd.nn that contains roughly 19 million records: > SELECT n, name, cdf from bd.nn limit 4;…
Problems with update statement in SQLite
May 31, 2011
I have created a database using SQLite. I want to update the value of a "features" column( type Blob)...but i do not know how to write the "update"…
Very slow update on a relatively small table in PostgreSQL.
May 26, 2011
Well i have the following table(info from pgAdmin): CREATE TABLE comments_lemms ( comment_id integer, freq integer, lemm_id integer, bm25 real )…
Complex mysql update query is slow
June 23, 2011
I have an import table with user data, and I need to mark the rows with duplicate field values because they should not be imported. CREATE TABLE…
C# + MySQL prepared statement w/ON DUPLICATE KEY UPDATE problems
March 28, 2011
I have a C# application that is having issues using a prepared statement with an ON DUPLICATE KEY UPDATE into MySQL. When I check the connection…
How can I selectively update some sqllite table to preserve user data for an iphone app update?
February 27, 2011
can you selectively update certain sqllite tables when updating an iphone app, in order to preserve user stored data? how? appreciate the help!
Mysql Workbench - Automatically update Foreign keys on Primary key update
March 11, 2011
I have just started to use mysql workbench(5.2.31) to create EER diagrams. I use exactly the same name for foreign keys and primary keys. Is it…
MySQL on "duplicate key update" skipping update on verifiably unique records
April 7, 2011
While trying to do an insert with ON DULICATE KEY UPDATE, the server will constantly skip multiple records when the query is run through PHP or run…
Cannot update multiple rows in Sqlite database update command in android.
December 30, 2010
Hi! I am facing a problem here. I am trying to bulk update my table i.e trying to update multiple rows in the database. The update is simple. I just…
Codeigniter - database : how to update multiple tables with a single update query.
January 28, 2011
I saw this on the codeigniter forum Considering the below code UPDATE a INNER JOIN b USING (id) SET a.firstname='Pekka', a.lastname='Kuronen',…
How do you update multiple field using Update.Set in MongoDB using official c# driver?
January 27, 2011
The following code will allow me to update the Email where FirstName = "john" and LastName = "Doe". How do you update both Email and Phone without…
MySql Performance: INSERT...ON DUPLICATE KEY UPDATE or UPDATE & ROW_COUNT
May 5, 2011
I query that most of the times will be updating an existing record. However, I need to check if this records exists and if not I should create it.…
Update multiple embedded documents in single update statement
June 1, 2011
Here 'Users' is the model mongo mapper document And Posts,Comments are mongo mapper embedded document. Its like Users -> Posts -> Comments for…
Performance question: ON DUPLICATE KEY UPDATE vs UPDATE (MySQL)
February 17, 2011
Is there performance difference between INSERT INTO ON DUPLICATE KEY UPDATE and UPDATE? If I know the values that can be UPDATEd - should I use…
Insert Into one duplicate update xx - Does it also update the INDEXes of the table?
February 10, 2011
I have a table with 30M+ rows, and each index update is expensive. I sometimes have to update and/or add 5000+ rows in a single insert. Sometimes…
Using ON DUPLICATE KEY UPDATE insert query for update already exist row
May 28, 2011
connection() { OleDbConnection nwindConn=new OleDbConnection(); MySql.Data.MySqlClient.MySqlConnection con= new MySqlConnection(); MySqlCommand cmd;…
SQL UPDATE : You can't specify target table 'nilai_mhs' for update in FROM clause "
February 16, 2011
I have query like: UPDATE nilai_mhs SET status_nilai = 3 where id_kode_mk = (SELECT e.id_kode_mk FROM nilai_mhs c, tbl_mhs d,master_mk…