Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

Export raw data without invalidating memory pages

Upcoming MongoDB Community Events
(53 lines)
Shard down, RHEL says: mongo dead but subsys locked
(31 lines)
Feb 7, 2012
Mihnea Giurgea
Mihnea Giurgea
From our current understanding of MongoDB, iterating through 100GB of
data will load 100GB into RAM (assuming swap is disabled) and will
invalidate 100GB worth of memory pages. Is there any way to avoid
that? Can MongoDB iterate through a collection / database without
memory-mapping objects?

Our current use-case is that we want to deplete (transfer) a part of
our MongoDB cluster to a non-MongoDB storage. This directly affects
our production working set, as the data we are moving invalidates
memory pages, resulting in a performance degradation.

Thanks,
Mihnea Giurgea @ uberVU





Reply
Tags: memory pages
Messages in this thread
Export raw data without invalidating memory pages
Pigmix: data export
August 23, 2011 08:49:54 PM
Hello, Pigmix can generate data to HDFS and I would like to find a way to convert the da to a general format by either: 1. flattening the data (would copyToLocal work here?!) 2. export the data to SQL tables (or any other non specific Hadoop…
How to make sure Redis memory pages are physics, not swap to disk by OS ?
October 4, 2010 10:51:55 AM
By looking the code, it just malloc. Will that means I need set swappiness to 0 for safe ?
Continuous export of data out of database
November 15, 2011 10:21:02 AM
Hello VoltDB has an export feature to stream the data out of the database. http://voltdb.com/company/blog/voltdb-export-connecting-voltdb-other-s ystems This is different from Cassandra's export feature…
export data from HBase to mysql
June 23, 2011 01:32:52 PM
Looking for tools to help export Data from HBase to MySQL, will need some transformations before inserting. thanks, Vishal
Export Control Data for your product
August 3, 2011 06:26:25 AM
Dear Sir or Madam, to comply with the U.S. export control laws we kindly ask you to provide us with the export relevant data (such as ECCN and CCATS) for your product: Apache ANT Thanks in advance, Werner Werner J. Ettinger Procurement…
Export data with column names
May 2, 2011 10:12:57 PM
Does anyone know how to export data out with column names? Any help here is appreciated. Thanks, Ranjith
Issue 383 in redis: A command to implement releasing of allocated memory pages
November 19, 2010 12:20:00 AM
Status: New Owner: ---- Labels: Type-Defect Priority-Medium New issue 383 by klin### @yandex.ru: [FEATURE REQUEST] A command to implement releasing of allocated memory pages http://code.google.com/p/redis/issues/detail?id=383 Some of us are…
Data export error in mongohq heroku
June 10, 2011 08:59:55 AM
Hi, When i tried to move(export) my development data to heroku mongohq, i got this weird error. Error creating index app560699.categoriesassertion: 13111 wrong type for BSONElement () 0 != 16 It seems there is a problem with creating indexes.…
Updated: (HIVE-645) A UDF that can export data to JDBC databases.
July 26, 2010 01:22:55 AM
[ https://issues.apache.org/jira/browse/HIVE-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-645:
Export Blob data from sqlite database to html
November 16, 2011 03:02:39 AM
I have a sqlite database from a iphone chat app which I would like to present in a human readable format either csv, html or other. The database has over 600 tables with multiple rows of conversation, the fields are: sender (text) - servertime…
Created: (DIRSTUDIO-703) DSML Export fails to export when the size limit is hit
October 8, 2010 08:45:58 AM
DSML Export fails to export when the size limit is hit
Updated: (DIRSTUDIO-703) DSML Export fails to export when the size limit is hit
October 8, 2010 08:45:58 AM
[ https://issues.apache.org/jira/browse/DIRSTUDIO-703?page=com.atlassian .jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre-Arnaud Marcelot updated DIRSTUDIO-703:
Updated] (DIRSTUDIO-703) DSML Export fails to export when the size limit is hit
January 9, 2012 06:35:08 AM
[ https://issues.apache.org/jira/browse/DIRSTUDIO-703?page=com.atlassian .jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre-Arnaud Marcelot updated DIRSTUDIO-703:
Commented] (DIRSTUDIO-703) DSML Export fails to export when the size limit is hit
March 26, 2011 07:43:15 AM
[ https://issues.apache.org/jira/browse/DIRSTUDIO-703?page=com.atlassian .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedComment Id=13011621#comment-13011621 ] Stefan Seelmann commented on DIRSTUDIO-703:
Upgrading or changing memcahe client without invalidating the memcache.
June 17, 2011 05:39:47 PM
Hello. We wanted to change the hashing strategy of the memcache (memcache.hash_strategy) from "standard" to "consistent". Is there a easier way to do this without invalidating all of the memcache? If we do change it on the run, All the key…
invalidating stored prepared statements like triggers at an installation.
January 17, 2012 01:42:00 PM
Is there any existing way in the software to invalidate all the existing stored prepared statements (like compiled triggers). Either a command in sane or insane or with a property on boot? The code does this now automatically on any number change…
DO NOT REPLY Memory leaks protection does not cure leaks triggered by JSP pages code
November 24, 2010 03:35:17 PM
https://issues.apache.org/bugzilla/show_bug.cgi?id=48837 Mark Thomas <mar### @apache.org> changed: What |Removed |Added
DO NOT REPLY Memory leaks protection does not cure leaks triggered by JSP pages code
November 25, 2010 03:31:43 PM
https://issues.apache.org/bugzilla/show_bug.cgi?id=48837 --- Comment #3 from Sylvain Laurent <sylvain### @gmail.com> 2010-11-25 16:31:10 EST --- Thanks Mark. Just to clarify things : the patch just improves the detection of such leaks.…
DO NOT REPLY Memory leaks protection does not detect leaks triggered by JSP pages code
December 1, 2010 08:30:53 PM
https://issues.apache.org/bugzilla/show_bug.cgi?id=48837 Konstantin Kolinko <knst.ko### @gmail.com> changed: What |Removed |Added
Issue 1244 in cherokee: It woud be nice to have a way of invalidating the IO Cache
August 23, 2011 08:18:17 AM
Status: New Owner: ---- New issue 1244 by b..### @karreman.org: It woud be nice to have a way of invalidating the IO Cache http://code.google.com/p/cherokee/issues/detail?id=1244 What steps will reproduce the problem? 1. Change a file that is in…
Memory usage export from database to csv in php
March 15, 2011
Hello. I need to export data from mysql to csv. I have to select data from several tables putting them in arrays, then process them and return them…
In C#, is there any way to have an in-memory file linked as an in-memory SQLite database with System.Data.SQLite?
May 31, 2011
What I want to do is something along the lines of the following: using System.Data.SQLite; using System.IO; //My SQLite connection SQLiteConnection…
MySQLdump no data export
April 30, 2011
I want to do a mysqldump with the table definition and table data every day, for this I config a cron job with this comand: "mysqldump -u user…
MySql export schema without data
May 30, 2011
I'm using a MySql database with a javaprogram, now I want to give the program to somebody else and I whas wondering how you export your MySql…
SQL: export table data without Primary Key
June 17, 2011
I want to export data from a sql table, but I dont want the primary key to be exported. The reason is that I have data on my localhost that needs to…
How to export table data to file
January 25, 2011
hi i just wanted to export table data a csv file can any one give me sample
Why mySQL data export hang inbetween??
May 18, 2011
I do have 1 million datas in my MySQL database and when I export whole data it is getting stuck in between and showing the download box for long…
Export MySQL data to separate txt files.
March 16, 2011
I have a MySQL database dump of about 250,000 records. I am very new to MySQL, and I need each of these records exported into their own .txt file.…
Manifold/PostGIS data manipulation and export
February 6, 2011
I'm currently working on a GIS database project using Manifold Ultimate. I am able to import data from PostGIS via the database console, and edit…
Export partial mysql meta data, possible?
March 8, 2011
I want to save some specific records from a database to a file, thus: "select, export etc" * from tbl1 where ClientName = 'DemoAccount'; should…
Extract data from a Google Chrome bookmarks export with PHP
April 11, 2011
I am wanting to get my google chrome bookmarks into a database, so my first step is taking an exported .html file from chrome with PHP and getting…
Attempting to export MySQL data to CSV & download through browser.
May 18, 2011
Hi, so, first let me state, before you start providing other suggestions on how to simply export a complete table to a CSV file, that is not what I…
Export data from database and write to HDFS(hadoop fs)
June 10, 2011
Now i am trying to export data from a db table, and write it into hdfs. And the problem is: will the name node become bottleneck? and how is the…
Importing JSONP data from HTML page then export to CSV using r
February 10, 2011
I have JSON data, of which this is a snippet: {"sweater":"15","localtime":"7:14 PM","xcoord":-61,"desc":"John Smith SHOT on Jack…
Hibernate data export / import for a single tenant
April 11, 2011
Hey Guys, I have posted quite a general question about implementing a multi-tenant solution using very specific technologies in the past. I now have…
How to write postgresql script to export data from table to csv file
January 26, 2011
hi, i Just want to write a script which finds the tables in a particular postgresql data and converts/exports whole data to individual csv files…
Pagination bug - When i use 'WHERE' in the query, pages do not remember the data
February 16, 2011
hello i'm having some difficulties in making this php pagination work. Here is the code http://www.phpeasystep.com/phptu/29.html This works fine,…
In memory database System.Data.Sqlite
May 21, 2011
I am trying to create an in memory database using vb.net and the system.data.sqlite dll. But I neeed to copy the context from my file db first I…
Why OpenSSL may keep received/written data in memory?
April 21, 2011
I'm trying to ensure that sensitive data (passwords, ...) are not kept in clear-text in process memory and I have found that all data sent to or…
Regarding stripslash in html pages how to hide... in php pages
March 21, 2011
hello, when i add the enter from the CMS admin panel then add the data before doing this php method.. $result = mysql_real_escape_string($str);…