Questions about mongodb sharding details

1,      config server

I remembered that I have seen a statement in the document like "3
config servers, any of them are down ,meta data read only, but the
system is online as long as 1/3 is up" .
Here the "meta data" refers what kind of metadata? the metadata of
collections ? or some information about the data distribution ? What
are the differents between the metadata in the <dbname>.system.*
namespaces in MongoDB and the metadata in the config servers? I hope
there is a list showing what kind of metadata are stored in these two
places.


2,      mongos

Is the mongos just acts as a router in the system? As we know, once a
data is inserted , it will be hashed to a shard with a hash
algorithm . What I want to know is where the hash action takes? In the
mongos or in the config server ? I can't find a complete workflow
about the read action or the write action , so come here to ask for
some help .

Thanks .

hwang

  Our company is getting ready to deploy a mongo cluster using 1.6.1 
auto sharding and have a few
questions surrounding how we may be able optimize our setup.

Our application is write-heavy and will be inserting around 80 to 100 
rows per second into a single sharded
collection.  Each row represents a log entry coming from our clients and 
a typical row will have the following
info plus maybe 20 other properties that vary from row to row:
{ "e": "environmentName", "id": "clientId", "date": "timestamp in ms" }.

This basically forms a raw log that we then build metadata on by running 
a series of timed jobs.  Most of our
queries tend to involve a combination of "e", "date", and "id" 
(sometimes just e, sometimes just date, and
sometimes e + date).

Below are the questions that I have on optimizing the cluster setup, is 
there anything else we should be thinking about?

Optimizations for writes?
     -Currently running out of the box mongo config.  Are there 
parameters we should adjust to help
     with write heavy applications?

What is the recommended replica set server configuration?  Considering 
something like the following:
     -16 servers, 8 shards - each shard is a replica set with an online 
+ offline pair
     -16 servers, 16 shards - each server in the cluster is part of a 
replica set with one other server

Shard key
     -In our tests we have been sharding on { id: 1, date: 1}, should we 
add "e" as part of the shard key or create an index?
     -Is there a general rule of thumb for deciding what should be 
indexed and what should be part of the shard key config?



Regards,

--Mike

You started the mongos pointing it at a configdb using localhost - you
need to use localhost for everything or nothing at all.

On Wed, Sep 1, 2010 at 10:48 AM, Orit Alul <orit### @gmail.com>
wrote:
 Hi,

 I'd like to create 2 mongodb shards so that each one of them is
 located on different host.

 I took the following steps:
 1. Started 2 shards
    ( mongod --shardsvr --dbpath c:\data\db\a --port 10000 --rest
      and
      mongod --shardsvr --dbpath c:\data\db\b --port 10001 --rest)
    on hosts host1 and host2 correspondingly.
 2. Started the config server on host1
    (mongod --configsvr --dbpath c:\data\db\config --port 20000 --
 rest)
 3. Started mongos server on host1
    (mongos --configdb localhost:20000)

 4. Opened the mongo shell and try to perform the following commands:
 use admin;
 db.runCommand({addshard : "<host1 ip address>:10000"});
 db.runCommand({addshard : "<host2 ip address>:10001"});

 I get the following error:
 "errmsg" : "can't use localhost as a shard since all shards need to
 communicate. either use all shards and configdbs in localhost or all
 in actual IPs "

 I'd like to emphasize that I put the host ip-address!
 It seems that only when I use the addshard command with "localhost" it
 succeeds
 db.runCommand({addshard : "localhost:10000"});

 Did you ran into this issue?
 What am I doing wrong?

 Thanks,
 Orit


 --
 

There are a couple of tutorials out there on actually configuring
shards.

The best one is here:
http://blog.boxedice.com/2010/08/03/a...ver-with-mongodb/

The "localhost" problem that you've encountered is due to
documentation in several places that runs "sharding" on the same node.
http://www.mongodb.org/display/DOCS/A...iguration Session
http://www.snailinaturtleneck.com/blo...slave-is-so-2009/

For some reason both corporate docs and employee blogs seem to think
that "single-node sharding" is a good reference doc.

Obviously both Mike and Kristina knew the answer to your problem, but
it seems like that doesn't really justify fixing the docs yet. So
again, I'll point you to the Boxedice blog that has a more reasonable
demo of getting started.

On Sep 1, 10:00 am, Kristina Chodorow <kris### @10gen.com> wrote:
 > 3. Started mongos server on host1
 >    (mongos --configdb localhost:20000)

 You have to use ips everywhere, even for the config db.



 On Wed, Sep 1, 2010 at 10:48 AM, Orit Alul <orit### @gmail.com>
wrote:
 > Hi,

 > I'd like to create 2 mongodb shards so that each one of them is
 > located on different host.

 > I took the following steps:
 > 1. Started 2 shards
 >    ( mongod --shardsvr --dbpath c:\data\db\a --port 10000
--rest
 >      and
 >      mongod --shardsvr --dbpath c:\data\db\b --port 10001
--rest)
 >    on hosts host1 and host2 correspondingly.
 > 2. Started the config server on host1
 >    (mongod --configsvr --dbpath c:\data\db\config --port 20000
--
 > rest)
 > 3. Started mongos server on host1
 >    (mongos --configdb localhost:20000)

 > 4. Opened the mongo shell and try to perform the following
commands:
 > use admin;
 > db.runCommand({addshard : "<host1 ip address>:10000"});
 > db.runCommand({addshard : "<host2 ip address>:10001"});

 > I get the following error:
 > "errmsg" : "can't use localhost as a shard since all shards need
to
 > communicate. either use all shards and configdbs in localhost or
all
 > in actual IPs "

 > I'd like to emphasize that I put the host ip-address!
 > It seems that only when I use the addshard command with
"localhost" it
 > succeeds
 > db.runCommand({addshard : "localhost:10000"});

 > Did you ran into this issue?
 > What am I doing wrong?

 > Thanks,
 > Orit

 > --
 > 

 3. Started mongos server on host1
    (mongos --configdb localhost:20000)


You have to use ips everywhere, even for the config db.

On Wed, Sep 1, 2010 at 10:48 AM, Orit Alul <orita### @gmail.com>
wrote:

 Hi,

 I'd like to create 2 mongodb shards so that each one of them is
 located on different host.

 I took the following steps:
 1. Started 2 shards
    ( mongod --shardsvr --dbpath c:\data\db\a --port 10000 --rest
      and
      mongod --shardsvr --dbpath c:\data\db\b --port 10001 --rest)
    on hosts host1 and host2 correspondingly.
 2. Started the config server on host1
    (mongod --configsvr --dbpath c:\data\db\config --port 20000 --
 rest)
 3. Started mongos server on host1
    (mongos --configdb localhost:20000)

 4. Opened the mongo shell and try to perform the following commands:
 use admin;
 db.runCommand({addshard : "<host1 ip address>:10000"});
 db.runCommand({addshard : "<host2 ip address>:10001"});

 I get the following error:
 "errmsg" : "can't use localhost as a shard since all shards need to
 communicate. either use all shards and configdbs in localhost or all
 in actual IPs "

 I'd like to emphasize that I put the host ip-address!
 It seems that only when I use the addshard command with "localhost" it
 succeeds
 db.runCommand({addshard : "localhost:10000"});

 Did you ran into this issue?
 What am I doing wrong?

 Thanks,
 Orit


 --
 

Hi All,

Does anyone know how I can enable routing for mongodb databases
without sharding for collections? That is I want to query a mongos
server and have it route me to the proper mongodb shard. I am running
on a 32bit windows and I am afraid that if I enable sharding for
collections that it might push it over the 2GB limit per process
limit. Or is that not a problem in with autosharding in that it
detects if a shard will exceed its capacity?

My best guess is to

1. Create n number of  mongod --shardsvr
2. Setup a mongod --configsvr
3. Runup a mongos process and add all the shards in #1.

I only need routing so is this enough or do I also need to run the
command enablesharding for each db?

Thank you,

Phat

Hi,

how do i place config servers + can i configure the mongos servers in
each dc to use local slaveok reads for readonly connections in each
dc? Writes use a 2nd standard replica set connection but aren't of
interest in this case.

e.g. A request that is being sent to the DC_ASIA1 should use the local
replica set members in that datacenter for slaveok reads and not the
master or loadbalance it among other members.

1) As far as i understand i can place one/multiple mongos servers in
each DC and use addshard to only add local slaves to each mongos which
will only be used for slaveok reads. Does that work or will mongodb
detect a replica set member, use it as a seed -> use master/load
balanced reads?

2) How/where do i place the config servers?

Here is the setup:

- 4+ datacenters (6-9 planned in case that matters for config
servers)
  - DC_US1 -> DC_US2 ~ 40ms
  - DC_US2 -> DC_UK1 ~ 90ms
  - DC_US1 -> DC_ASIA ~ 200ms
- mongodb with 3 shards, 4 replica set members per shard, one rs
member for each shard per datacenter
- US1/2 + UK can become primary
- ASIA / others passive
- reads occur in all datecenters, must have local speed -> localhost
slaveok event. consistent reads

- DC_US1
  - shard1 - rs_member1 - prio 1
  - shard2 - rs_member1 - prio 1
  - shard3 - rs_member1 - prio 1

- DC_US2
  - shard1 - rs_member2 - prio 1
  - shard2 - rs_member2 - prio 1
  - shard3 - rs_member2 - prio 1

- DC_UK1
  - shard1 - rs_member3 - prio 1
  - shard2 - rs_member3 - prio 1
  - shard3 - rs_member3 - prio 1
  - arbiter1

- DC_ASIA1
  - shard1 - rs_member4 - passive prio 0
  - shard2 - rs_member4 - passive prio 0
  - shard3 - rs_member4 - passive prio 0


Thanks a lot
Alex

Sorry for my thoughtless , I got it now. With format "hostname:port"
as the parameter when using mongostat command, then you can look at a
certain mongod process.
Thank you for replying .

On Aug 22, 2:34 pm, ?? <joe.h### @gmail.com> wrote:
 But when I use the mongostat , the command is "mongostat -h
192.168.xxx.195"
 In the command I just specify the host , not the port number, there
are 3 mongod process running on the host on different port .

 [ad### @dw-adw1 ~]$ ps -ef | grep mongo
 admin     2459  2398  0 14:32 pts/0    00:00:00 grep mongo
 admin    31492     1 57 10:41 ?        02:11:52
/home/admin/hy/mongodb-linux-x86_64-1.6.0/bin/mongod -f
/home/admin/hy/conf/27017.cfg
 admin    31512     1  4 10:41 ?        00:10:12
/home/admin/hy/mongodb-linux-x86_64-1.6.0/bin/mongod -f
/home/admin/hy/conf/27018.cfg
 admin    31544     1  1 10:42 ?        00:03:53
/home/admin/hy/mongodb-linux-x86_64-1.6.0/bin/mongod -f
/home/admin/hy/conf/27019.cfg

 the 27017,27018,27019 are the 3 port number for the 3 mongod process.
 I'm not sure if I described it clearly , if there is any
questions,just let me know .
 Thanks .

 -----????-----
 ???: mongodb-use### @googlegroups.com
[mailto:mongodb-use### @googlegroups.com] ?? Eliot Horowitz
 ????: 2010?8?22? 11:14
 ???: mongodb-us### @googlegroups.com
 ??: Re: [mongodb-user] Questions about the results of mongostat
and Concurrency in mongoDB

 1) mongostat is just for a single process your'e talking to.

 2) what do you mean session?  if you mean connection, if A reads,
then
 B writes, then A reads, the 2nd A read will have B's update





 On Sat, Aug 21, 2010 at 10:57 PM, hwang <joe.ha### @gmail.com>
wrote:
 > Hi :

 > 1, About the result of the mongostat
 >        Supposed that I have 3 mongod process on node A :
a_1,a_2,a_3. And
 > the a_1 is the master of replica set foo1, the a_2 and a_3 are
slave
 > of replica set foo2 and foo3. Now through the mongostat I find
the
 > update / s column is high ,about 8000 . So I'm wondering the
results
 > from command "mongostat -h A" is just consist of the master
process of
 > replica set foo1 ,a_1 , or the total updates of the a_1 , a_2
,a_3  ??
 > May be a_2 and a_3 are slaves that there is not write operations
on
 > them, but there is oplog apply to them from their own master
process
 > on other nodes.

 > 2, Locking and Concurrency
 > I have not found some details materials about the Locking or
 > Concurrency in mongoDB
,exceptwww.mongodb.org/display/DOCS/How+does+concurrency+work
 > .
 > As I know ,now the write operation does not block the read ,
right ?

 > But how you deal with the Read Consistency ? if session A just
read
 > that a=2 , then session B update a=6 . Even in the same session
 > A ,query that record again will get a different result ?

 > At last, if there are 2 updates to the same record at the same
 > time(maybe this is not impossible) , how mongoDB deal with it?

 > Thanks in advance for your reply.
 > hwang

 > --
 > 

Hi :

1, About the result of the mongostat
	Supposed that I have 3 mongod process on node A : a_1,a_2,a_3. And
the a_1 is the master of replica set foo1, the a_2 and a_3 are slave
of replica set foo2 and foo3. Now through the mongostat I find the
update / s column is high ,about 8000 . So I’m wondering the results
from command “mongostat –h A” is just consist of the master process
of
replica set foo1 ,a_1 , or the total updates of the a_1 , a_2 ,a_3  ??
May be a_2 and a_3 are slaves that there is not write operations on
them, but there is oplog apply to them from their own master process
on other nodes.


2, Locking and Concurrency
I have not found some details materials about the Locking or
Concurrency in mongoDB ,except
www.mongodb.org/display/DOCS/How+does+concurrency+work
.
As I know ,now the write operation does not block the read , right ?

But how you deal with the Read Consistency ? if session A just read
that a=2 , then session B update a=6 . Even in the same session
A ,query that record again will get a different result ?

At last, if there are 2 updates to the same record at the same
time(maybe this is not impossible) , how mongoDB deal with it?


Thanks in advance for your reply.
hwang

Hi team, 
  I am new to Active MQ .I need to install and work in Active MQ for the
Chat application.Can any one say some steps or documentation of how to
work
with Active MQ to perform the chat application.

Thank you,

With Regards,
Prem

Hi Everyone,

I'm experimenting with servicemix(3.3)+camel(2.2)+cxf (rather new to them)
for implementing a WS client.
I have the following code:

        from("direct:remotesender")
            //.handleFault() 
            .onException(SoapFault.class)
//org.apache.cxf.binding.soap.SoapFault 
                .maximumRedeliveries(0)
                .process(new Processor() { 
                    public void process(Exchange exchange) throws
Exception
{
                        log1.info("INBODY
----------------"+exchange.getIn().getBody(String.class));
                        log1.info("INHEAD
----------------"+exchange.getIn().getHeaders());
                        log1.info("OUTBODY
---------------"+exchange.getOut().getBody(String.class));
                        log1.info("OUTHEAD
---------------"+exchange.getOut().getHeaders());
                        log1.info("EXC
-------------------"+exchange.getException());
                        log1.info("FAIL
------------------"+exchange.isFailed());
                        log1.info("EXCP
------------------"+exchange.getProperty(Exchange.EXCEPTION_CAUGHT,
Exception.class));
                        SoapFault faultex =
exchange.getProperty(Exchange.EXCEPTION_CAUGHT, SoapFault.class); 
                        log1.info("NODE
------------------"+faultex.getNode());
                        log1.info("REASON
------------------"+faultex.getReason());
                        log1.info("MSG   
------------------"+faultex.getMessage());
                        log1.info("DETAIL
------------------"+faultex.getDetail());
                    }
                })              
                .end()             
            .process(new Processor() { 
                public void process(Exchange exchange) throws Exception {
                    List<Element> outElements = new
ArrayList<Element>(); 
                   
outElements.add(exchange.getIn().getBody(org.w3c.dom.Document.class).getDocumentElement());
                    CxfPayload<SoapHeader> cxfPayload = new
CxfPayload<SoapHeader>(null, outElements);
                    exchange.getOut().setBody(cxfPayload); 
                    log1.info("CFXP
VVVVVVVVVVVVVVV"+cxfPayload.toString());
                }
            })
            .setHeader("SOAPAction",
constant("http://service.com/fault_ticket"))
            .to("cxf:bean:serviceEndpoint") // PAYLOAD mode is set from
spring
            .process(new Processor() { 
                public void process(Exchange exchange) throws Exception {
                    log1.info("CCFX VVVVVVVVVVVVVVV");
                    CxfPayload<SoapHeader> requestPayload =
exchange.getIn().getBody(CxfPayload.class);
                    List<Element> inElements =
requestPayload.getBody();
                    exchange.getOut().setBody(inElements.get(0)); 
                    log1.info("CFXE
VVVVVVVVVVVVVVV"+exchange.getOut().getBody(String.class));
                }
            })
            ;



1. My primary concern is, that in the case the server returns a SOAPFault,
I
can not extract the details of the fault. The SoapFault is thrown, and the
corresponding onException clause is triggered, but according to the log
below, the detail attribute is null.


    11:03:44,210 | INFO  | INBODY
----------------org.apache.camel.component.cxf.CxfPayload     headers:
nullbody: [[REQUEST: null]]
    11:03:44,210 | INFO  | INHEAD
----------------{http://service.com/fault_ticket,
CamelRedeliveryCounter=0,
CamelRedelivered=false}
    11:03:44,213 | INFO  | OUTBODY ---------------null
    11:03:44,213 | INFO  | OUTHEAD ---------------{}
    11:03:44,213 | INFO  | EXC -------------------null
    11:03:44,213 | INFO  | FAIL ------------------false
    11:03:44,214 | INFO  | EXCP
------------------org.apache.cxf.binding.soap.SoapFault: Server Error
    11:03:44,214 | INFO  | NODE ------------------null
    11:03:44,214 | INFO  | REASON ------------------Server Error
    11:03:44,214 | INFO  | MSG    ------------------Server Error
    11:03:44,214 | INFO  | DETAIL ------------------[detail: null]


The actual soap response is:

    500 Internal Server Error
    Server: MX/WEBSERVICE 3.X
    Cache-Control: private, pre-check=0, post-check=0, max-age=0
    Pragma: no-cache
    Expires: 0
    Connection: close
    Content-Length: 1551
    Content-Type: text/xml; charset=UTF-8

    <?xml version="1.0" encoding="UTF-8"?>
    <SOAP-ENV:Envelope
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
      <SOAP-ENV:Body>
        <SOAP-ENV:Fault>
          <faultcode>SOAP-ENV:Server</faultcode>
          <faultstring>Server Error</faultstring>
          <detail>
            <e:exception
xmlns:e="http://service.com/interfaces/mx_exception/">
              <code></code>
              <message>"WORK_ORDER_ALREADY_EXISTS</message>
              <callstack>"POISON_CALL_STACK" .... </callstack>
            </e:exception>
          </detail>
        </SOAP-ENV:Fault>
      </SOAP-ENV:Body>
    </SOAP-ENV:Envelope>

2. A second remark is, that the handleFault()  does not seem to have any
effect on the flow. The SoapFault is thrown regardless of it. I expected
it
to work similar to jbi faults: when it is present throw the exception,
otherwise just set the EXCEPTION_CAUGHT property on the exchange, and let
the flow continue.

3. My third question, isn't there a simple way to send a SOAP message with
CXF? I mean I have the SOAPBody prepared (as string or
org.w3c.dom.Document), and I expect cxf to wrap it in an envelope, and
also
return a SOAPBody. Do I really have to do the cumbersome payload
conversions
prior and after the request?


Thanks for your time,
attilav

Pardon I knew that, I must have been having an off day. Yes what I said is
wrong.

On Thu, Aug 19, 2010 at 12:46 PM, Bobby Jack
<bobbyk### @yahoo.co.uk> wrote:

 --- On Wed, 8/18/10, Wesley Acheson <wesley.a### @gmail.com>
wrote:

 No-one spotted the deliberate mistake? ;)

 > 4 values: are Top, bottom, left and right.

 should be

 4 values: are top, right, bottom, left


 - Bobby

Wesley Acheson wrote:
 Pardon I knew that, I must have been having an off day. Yes what I
said is
 wrong.

And you are having another, it seems.  Isn't this the wrong list for that
?
Oh well, there are just weeks like that..

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Devaraj Das reopened MAPREDUCE-1624:

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Devaraj Das resolved MAPREDUCE-1624.

Render more data for component details

Retired job info cache should contain more details than just JobStatus

     [
https://issues.apache.org/jira/browse...nels:all-tabpanel
]

Emmanuel Lecharny updated DIRSERVER-1160:

Clicking on a service link does not display the service details

Author: poirier
Date: Thu Aug 19 18:03:50 2010
New Revision: 987254

URL: http://svn.apache.org/viewvc?rev=987254&view=rev
Log:
Overhaul the detailed doc for virtual hosts:
- remove references to implementation details
- re-order information to fit the order of the page
- add more links to glossary and other pages

Modified:
    httpd/httpd/trunk/docs/manual/vhosts/details.xml

Modified: httpd/httpd/trunk/docs/manual/vhosts/details.xml
URL:
http://svn.apache.org/viewvc/httpd/ht...254&view=diff