Best unofficial Apache Server developers community
Username
Forgot password?
Sign in with Twitter account
Sign in with Facebook account
List archives

ConvertToCapped 29TB - "can't map file memory" when replication is off

Re: C# Driver: Dynamic query
(10 lines)
db.runCommand addshard is erroring out.
(32 lines)
Feb 8, 2012
Thorn Roby
Thorn Roby
I'm trying to convert an existing 4TB collection to capped 29TB. I
started on the replica set secondary, bringing it up without the
replSet argument because I'm not sure if the table lock might cause
replication problems if it's actively replicating (I did enlarge the
oplog to a size that should be able to handle the roughly 20 hours I
anticipate will be needed for the conversion). However, when I bring
up the (normally) SECONDARY without the replSet argument, and attempt
the convertToCapped, I get

db.runCommand({"convertToCapped": "log", size: 29000000000000});
{
        "assertion" : "can't map file memory",
        "assertionCode" : 10085,
        "errmsg" : "db assertion failure",
        "ok" : 0
}

 and in the logfile I see

couldn't open /data/db/lfs/lfs.1013 errno:24 Too many open files

I see the same error even if just looking at collection stats
(db.log.stats() ) . Another strange thing is that the VM size in top
is only 181M instead of the expected 8TB. This is 2.0.2 running on
Linux 64bit. I'm not sure why the absence of replication would cause
this. I verified that all datafiles and oplog files are owned by
mongod. I am starting the process with "numactl --interleave=all". The
system has 48G RAM.

I tried the conversion on the SECONDARY with replication running, but
I get "not master". I was hoping to run it on the SECONDARY so I could
get the replica set re-synced before running it on the PRIMARY.





Reply
Tags: replication problems
Master/slave replication got "caught user assertion 13440 bad offset:0 accessing file:" er
June 19, 2011 04:47:55 PM
Hi folks, We have a master-slave replication setup. Slave is started with --fastsync from physical files copied from master 10 hours ago. No unclean shutdown since then. Before we did a fastsync, the replication has been failing by crashing…
MogileFS::Client "Out of memory" when store file larger than 1G
March 8, 2011 03:45:39 PM
Hi, I guess this is a known problem. I experienced the past that mogstored does not like large file, so I replaced it with nginx, since then my storage nodes have been doing awesome. But, in 32bit machines, I ran into a problem of using…
show dbs trigger "can't map file memory - mongo requires 64 bit build for larger datasets"
April 7, 2011 07:09:05 PM
Hi I am on 32 windows and installed 32 bit mongo. Everything is good until I run "show dbs": > show dbs Fri Apr 08 10:05:08 uncaught exception: listDatabases failed:{ "errmsg" : "exception: can't map file memory - mongo requires 64 bit…
"can't map file memory" during index creation on 64bit
January 25, 2011 07:35:02 AM
Hi, I have mongodb 1.6.4 64bit running on Windows 7. Stats for collection: count: 1 400 000 size: 3.4Gb data folder: test.0 test.1 test.2 test.3 test.4 test.5 with test.0 = 64Mb and test.5 = 2Gb. Now I am trying to create an index on certain…
Closed: (DERBY-4590) You can drop a file-system database from a directory named "memory"
January 13, 2011 03:23:09 PM
[ https://issues.apache.org/jira/browse/DERBY-4590?page=com.atlassian.ji ra.plugin.system.issuetabpanels:all-tabpanel ] Kim Haase closed DERBY-4590.
64-bit system already,why "can't map file memory" again?
September 14, 2011 09:27:40 PM
I used Windows Xp 64-bit virtual machine. At first, the disk storage was not enough and mongo came up with a "can't map file memory"(If my suspection was reasonable since my partition C is only 1.7G left) However, though I make the partition C much…
"all" not inclusive of "replication" in pg_hba.conf
September 24, 2011 10:14:51 PM
Dear List , It is been found that the entry local all all trust does not renders below redundant in pg_hba.conf local replication replicator01 trust regds mallah.
Created] (HDFS-2233) WebUI doesn't handle file names with ";" and "+" characters
August 7, 2011 10:52:29 PM
WebUI doesn't handle file names with ";" and "+" characters
How to deploy-file or "upload artifact" webpage a "tests" artifacts to snapshost
November 15, 2010 03:19:39 AM
Hi, I wonder how I could upload to archiva both jars, tests-jars (and eventually sources and tests-sources, but this is optionnal) to a *snapshot* repository in archiva. I try by the webpage, or by running mvn deploy-file with tests classifier…
How to deploy-file or "upload artifact" webpage a "tests" artifacts to snapshost
November 15, 2010 03:19:37 AM
Hi, I wonder how I could upload to archiva both jars, tests-jars (and eventually sources and tests-sources, but this is optionnal) to a *snapshot* repository in archiva. I try by the webpage, or by running mvn deploy-file with tests classifier…
Customizing mod_include -- Print "virtual" or "file" attribute in handle_include
September 9, 2010 05:09:31 PM
Apache Development Gurus: I've worked with Apache for awhile now, but this is my first time tinkering with the source code for one of its modules. I'd like to alter mod_include to print out as a HTML comment all server side include references. …
about "large file" and "resume download" on reverse proxy mode
July 15, 2011 12:39:16 AM
Dear list, I am testing nginx with reverse proxy mode. I found that nginx download the file from upstream server to proxy_temp_path when I download a file from the reverse proxy server,if I stop and re-download,the nginx will download the file…
Customizing mod_include -- Print "virtual" or "file" attribute in handle_include
September 9, 2010 06:31:41 PM
Apache Development Gurus: I've worked with Apache for awhile now, but this is my first time tinkering with the source code for one of its modules. I'd like to alter mod_include to print out as a HTML comment all server side include…
??: Re: about "large file" and "resume download" on reverse proxy mode
July 15, 2011 11:19:21 AM
Thank your reply. But it is not for my question. When the item isn't cached(If it is very large over 1G,it would take a long time to be cached),many normal users download it,then the proxy server would download the item from upstream server with…
Common CGI Options" the "Check file" and "Rule" -> "Check local fil
September 3, 2011 06:51:52 AM
Hi, I'm wondering if in a high loaded server is better to have in "Handler" -> "Common CGI Options" the "Check file" enabled or not, and in "Rule" -> "Check local file" enabled or not... I would like to uncheck both for performance…
V 0.9.4: unexpected end of file, expecting "; " or "}" in /etc/mime.types:789
February 12, 2011 08:26:32 PM
Hi Folks I couldn't see a reference to this in the recent archives, and googling didn't help. I've just installed Debian 6 on a new laptop, and compiled nginx 0.9.4 with: ./configure --with-http_ssl_module --with-http_gzip_static_module \ …
Toast Blog: {"error":"not_found","reason":"missing"}
November 2, 2010 03:27:50 PM
Toast Blog: {"error":"not_found","reason":"missing"} Chris, your implementation of Toast is offline, down or broken. is that by design? as they say any smell is better than no smell! ~:" regards jonathan chetwynd www.peepo.com …
"onlyif" for resource type "file"
June 28, 2010 07:59:46 AM
Hello, How do I deploy a file, but then stop deploying it when it has been changed in a certain way? Something like: file { "/etc/ldap.conf": owner => root, group => nscd, mode => 640, content…
DO NOT REPLY excessive usage of file "/dev/random", error message "too many open fil
August 27, 2011 12:19:19 PM
https://issues.apache.org/bugzilla/show_bug.cgi?id=40787 Stefan Fritsch <sf### @sfritsch.de> changed: What |Removed |Added
Session replication fails with "Member send is failing"
December 7, 2011 10:04:36 AM
Hi all, I'm currently setting up a two-node cluster on a single machine (for testing only). I tried both on my Windows 7 box and on a Debian GNU/Linux 6.0.3 (squeeze) server. Tomcat versions are 7.0.23 and 6.0.35 Java versions are 1.6.0_18 (Linux)…
Javascript RegEx replacing "
May 10, 2011
I'm trying to patch the Drupal Panels module to fix this nasty issue , which makes the AJAX errorhandler completely unreadable. The error handler…
In C#, is there any way to have an in-memory file linked as an in-memory SQLite database with System.Data.SQLite?
May 31, 2011
What I want to do is something along the lines of the following: using System.Data.SQLite; using System.IO; //My SQLite connection SQLiteConnection…
Powershell pipe file contents into application without loading file in memory
January 24, 2011
With cmd I'd run mysql -uroot database < filename.sql to import a database dump (read from file and pass to MySQL). However, < is "reserved"…
How to regex search/replace with File::Map in largish text file with to avoid "Out of Memory"-Error?
June 10, 2011
I am in the process of converting a big txt-file from an old DOS-based library program into a more usable format. I just started out in Perl and…
PHP File upload - memory handling on CGI
February 22, 2011
I'm wondering how PHP handles a file upload. I'v read a file upload will be split in chunks and will be assembled after the upload has finished. Is…
MySql Replication Performance on Master after replication is resumed
June 4, 2011
Current configuration is basic Master->Slave replication. Each night various data import jobs run on master. During this time, slave replication…
MySQL Replication Simple Master/Slave Replication
May 19, 2011
Hi All, I have simple Master/Slave configuration. I have 8GB of RAM on both my production boxes. I was using Master for Writes only and slave for…
PHP Apache Memory Usage on file upload
May 25, 2011
When you upload a file to PHP, using a raw POST request, does the POST immediately on execution of the php thread form apart of memory usage…
Nginx uses all memory when uploading large file
June 22, 2011
I am using nginx with the upload module and when I upload a large file all of the memory is used by nginx. Anything I can do to fix that? Here is my…
File download in php, memory limit problem?
February 13, 2011
Hey. A client of my decided to move the website from a somewhat nice server to a... lets call it lesser nice server. The problem is, there's a file…
Does the PHP function filemtime use up memory lesser than it has to when it would file_get_contents() a file?
May 16, 2011
I know it's kind of obvious, but take a look at it this way: Let's say I have a text file which contains a JSON data, and this file will be opened…
SQLite database connection is making changes in memory but not saving to the file - AS3 AIR
April 5, 2011
Hello, world! I'm trying to write to a local SQLite database using the flash.data.* classes in AIR. I'm opening a synchronous connection in CREATE…
Large File Uploads Consume Disproportionate Amount of Memory
February 1, 2011
I have a VPS with 512 MB RAM, and I'm trying to upload files that are between 100-250 MB. Files up to about 100 MB work fine, but anything over and…
Problem with downloading file - web server starts consuming memory during dowload
May 25, 2011
Hi, Earlier I was able to download large files from my webserver and I never faced problem. But now, when I start a download and see the output of…
Can a C# program read a text file into memory and then pass that object to a method that requires a filename?
June 10, 2011
In a C# program I am importing a large text file (300mb) into a MySQL database via the MySqlBulkLoader function of the MySql .Net Connector. The…
Is it possible to create an in-memory SQLite database from an existing chunk of memory?
March 1, 2011
From the documentation it looks like this is not possible, but I want to be sure. Here's my use case: I want an encrypted database. It should only…
Php-cgi memory usage higher than php's memory limit
December 23, 2010
I'm running apache with a worker MPM and php with fastcgi. the following are my mpm limits: StartServers 5 MinSpareThreads 5 MaxSpareThreads 10…
Why autovacuum process takes so much memory and swap memory ?
June 15, 2011
We have a product log database which only produces insert sql? But we found that The autovacuum process took so much memory? it takes about 16 GB at…
MySQL Replication
February 17, 2011
We have an update process which currently takes over an hour and means that our DB is unusable during this period. If I setup up replication would…
Couchdb replication
January 24, 2011
From what I know, to avoid conflicts, couch will reject saving conflict document in single node, and in other hand, couch will allow conflicts in…