dynamic scaling question

Hi there,

I'm new to Riak and starting to evaluate it for an application we're
building.  So far, I'm quite impressed.  But I'm having some trouble
wrapping my head around what happens exactly in the event of a node
failure.
 In this section of the wiki:

https://wiki.basho.com/display/RIAK/R...oesN=3reallymean?

it states that there's no guarantee that the replicas will reside on
different machines.  Does this mean that it's possible for all N copies of
a
given datum to end up on the same machine?  Is there any way to avoid
this,
so that a single node failure won't result in data loss?

Thanks!
Tony

When configuring network of brokers using dynamic discovery such as
multicast, does that mean the directional is 'duplex' by nature or still
one-way? The thing is that the networkConnector will only have multicast
address and not other brokers:

 <networkConnectors>
            <networkConnector name="bridge"
uri="multicast://default?group=MyGroup" />
 </networkConnectors>

Am I mssing anything here?

-J

I'm curious if there are any efforts ongoing to amortize the
background tasks in Cassandra over time?
Specifically, the cost of compaction and AE, rebalancing, etc seems to
be a problem for some users when they are expecting more steady-state
performance. While this may sometimes be the result of a cluster which
is at its marginal capacity, users are still surprised with the
performance hit or downtime required for common operations. Making the
cluster able to make finer-grained and measurable progress towards the
ideal state may help other users, too.

Is there a feasible design or enhancement which may allow these types
of background tasks to be broken apart into smaller pieces without
compromising overall consistency?
It would be excellent if the user could see the over-all state of the
storage cluster, and to choose the proportion of resources allocated
to recovering backlog vs servicing clients, etc.
Even better, if there were some basic heuristics which worked well for
the general case, and users would only have to see the scheduling plan
in special situations.

How would you go about doing that? Does the current architecture lend
itself to this type of optimization, or otherwise?

Hi All,
I've got a couple questions that have come up about how Cassandra works
and
what others are seeing in their environments.  Here goes:

1.) What have you found to be the best ratio of Cassandra row cache to
memory free on the system for filesystem cache?  Are you tuning it like an
RDBMS so Cassandra has the vast majority of the RAM in the system or are
you
letting the filesystem cache do some of the work?

2.) Is the Cassandra cache write-through (ie are new records held in the
row
cache as they're written to disk?

3.) When using the random partitioner how much difference should be
expected
(or has been observed) between nodes?  2%? 10%?

3.5) Can a load balance be expected to bring the data distribution pretty
close to even among all nodes in the ring?  Is the correct process for a
loadbalance to run the loadbalance operation on each node in the ring?


Thanks!  I'm curious to hear what other's have observed.
-Aaron

 
 1.) What have you found to be the best ratio of Cassandra row cache
to memory
free on the system for filesystem cache?  Are you tuning it like an RDBMS
so
Cassandra has the vast majority of the RAM in the system or are you
letting the
filesystem cache do some of the work?

This depends on your exact case: how much rows are in a hot set. Throwing
too
much memory to JVM cache results in slower garbage collection with no
effect on
performance. There are cases (for ex, large rows, which are read mostly
partially using get_slice), for which row cache will do things worse. I
did a
try and watch approach, changing size of row cache and watching for row
cache
hit ratio and op/s. Hit ratio of 0.9 was enough for my case.

 
 2.) Is the Cassandra cache write-through (ie are new records held in
the row
cache as they're written to disk?

Not exactly. Cassandra keeps recent writes (not rows) in memory, but after
flushing memtable, it will reread from disk (and reconstruct) whole row to
row
cache on 1st read if data. 

 
 3.) When using the random partitioner how much difference should be
expected
(or has been observed) between nodes?  2%? 10%?

This depends on data. It will distribute keys almost equal between nodes,
nut
sizes of row data could be different for different keys. In my case it was
about
0.2% 

We are working with a trunk of hive hive-0.6.0-957988.

Dynamic partitions are working for us in testing, we tested with about
100 dynamic partitions.

For our production run we have about 1000 (offer_id)'s.

HQL="set hive.exec.dynamic.partition.mode=nonstrict;
     set hive.exec.max.dynamic.partition.pernode=200000;
     set hive.exec.max.dynamic.partitions=20000;
        insert overwrite table bco PARTITION (dt=20100721,offer)
                SELECT * FROM (
                   SELECT browser_id, country, dma_code,
                          offer_id from baseline_raw where
gen_date=20100721 and blt='bco' DISTRIBUTE BY offer_id
                   ) X;
"

2010-07-22 22:36:21,499 Stage-1 map = 100%,  reduce = 50%[Fatal Error]
Operator FS_6 (id=6): Number of dynamic partitions exceeded
hive.exec.max.dynamic.partitions.pernode.. Killing the job.

As you can see we always die in the reducer with the same exception.
Is it possible that hive.exec.max.dynamic.partitions is not being used
in the reducer code?

Thanks,
Edward

In guide, there are
title like #{title}

but how to do with
<sql id="xxx">
select something from table where things like '%thing%'
</sql>

Thanks

Hi,

It's somehow possible to dynamically change the sql statements in the
connectionInitSqls?

 

Thanks,

Pedro Mendes

 

Hello,

Is there anyway to dynamically setup routes? For example, I'm trying to
setup a route using a File component. In that file there will be something
like the following:
MQ_USERS_DATA_TOPIC=/the/file/location

I would like to read this file and then setup a route from above folder to
MQ_USERS_DATA_TOPIC

The content of the above file would be dynamic and the mapping of TOPIC /
file location could grow or shrink.

Thanks

See this wiki page
http://camel.apache.org/discussion-forums.html

You need to be a subscriber on the mailinglist to have the mails
forwarded.

A colleague just sent over a link to a blog about the Hibernate Validator
4.1.0 release:

http://musingsofaprogrammingaddict.bl...validator-41.html

In particular, they now offer a dynamic API.  Does anyone have any
thoughts on the API they've created?  IMO if they have dynamic constraints,
and bval has dynamic constraints, eventually the spec will have dynamic
constraints.  I personally don't think that dynamic constraints are that
powerful unless you have ways of "branching" constraint mappings from
globally-applicable down to instance-level (and possibly steps in between).
 I'm also not thrilled with their fluent builders:  they seem nice to use
at compile-time, but the fact that they would require a different builder
per constraint type to be written or generated seems less than optimal.  I
am wondering if we could borrow some ideas from e.g. Mockito to stay
typesafe and compiler-checked yet avoid the need to write a builder per
constraint.

Thoughts?

-Matt

Hi,

How can I get a security realm name per deployed context ? We deploy 
several versions/instances of our software on the same Geronimo server, 
and want the browser to remember username/password per url.

When deploying a web app we create a security realm name (in the plan) 
containing the context to which we deploy, e.g. secure-context1, 
secure-context2. In the Geronimo console I can see all the realms 
created, so that seems to be fine.

My web.xml (and geronimo-web.xml) contain 'secure' as (generic) realm 
name, and that is what the browser reports when I connect.

Maybe a bigger question (for me) is: how does the security definition in 
the plan.xml link back to the (geronimo-)web.xml or how can I link my 
web.xml/geronimo-web.xml to the realms available ?

Thanks in advance.

Regards,
Kees

should i include .ece and .cms type of files while analysing the response
time and throughput??

Hi

You can find the example here[1]
[1] http://camel.apache.org/recipientlist-annotation.html

Willem

Then you're not doing it the way I have described.

I have been battling with this for ages. Every time I want to import
dynamically generated xml document it doesn't work. The URL gets mangled
by
the import tag.

I can use the request parameters to form a full URL but that doesn't work
for a secure app because the credentials aren't shared.

Create a Jersey web service and use the tags as I have described. Just
doesn't work at all. I have resorted to writing my own tag. 

Hey ho.



Hassan Schroeder-2 wrote:
 
 On Wed, Sep 16, 2009 at 7:05 AM, paulbrickell
 <paul.b### @evolvedintelligence.com> wrote:
 
> If I create a url with a context relative path (e.g. /x/y/z) and
the I
> use
> this in the url attribute of an import tag, the import tag seems
to
> attempt
> a lookup of a file in the web application.
>
> So given this...
>
> <c:url var="aURL" value="/x/y/z"/>
> ${aURL} Get resource at z
> <c:import url="${aURL}" var="xml" />
>
> The link in href attribute in the browser is a valid url, but the
url
> attribute in the import tag is not. I get a file not found
exception for
> /x/y/z. Import seems to want to lookup a static resource (html
page for
> example) located on the file system and not open connection to
some
> network
> resource. I am assuming from the name (import) that this is
probably what
> is
> expected.
 
 No, what you're describing works exactly correctly for me -- the
 URL is accessed, and the rendered output, not a file, is fetched.
 
 What value does ${aURL} display?
 -- 
 Hassan Schroeder ------------------------ hassan.s### @gmail.com
 twitter: @hassan
 
 

I'm trying to do a dynamic partition insert like so:

        FROM (
            FROM (
                SELECT project, file, os, country, dt
                FROM test_downloads WHERE dt='2010-06-30'
                CLUSTER BY project, file, dt
            ) a
            SELECT TRANSFORM(project, file, os, country, dt)
            USING '/var/local/sfpy/dstat/dstat/transform/reduce.py --test'
            AS (project, file, downloads, os, country)
        ) b
        INSERT OVERWRITE TABLE test_daily_file_totals PARTITION
(dt='2010-06-30', project)
        SELECT project, file, downloads, os, country

The test_daily_file_totals table is created like so:

CREATE TABLE IF NOT EXISTS {{ table_name }} (
    file STRING,
    downloads INT,
    os STRING,
    country STRING
)
PARTITIONED BY (dt STRING, project STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE

But I get an error with the dynamic partition syntax:

FAILED: Parse Error: line 12:89 mismatched input ')' expecting = in
destination specification

Can someone see something I'm doing wrong in the sql? I have set both
hive.merge.mapfiles and hive.merge.mapredfiles to false in hive
configuration.

Thanks,
-L

So ok, I traced through where isDynamic is called, and thought about it a
little, and guessed the answer.

If you do not define a set of version-matchers explicitly, then you get
the usual ones, Latest, Sub Revision, Version Range, etc.  But if you do
define a custom matcher, say:

  <version-matchers>
     <pattern-vm>
        <match revision="versionconfig"
pattern="${versionbase}\.\d+-(${config}|complete)"
args="versionbase,config" matcher="regexp"/> 
     </pattern-vm>
  </version-matchers>

it is interpreted as a complete definition of ALL version matchers -- and
you no longer have available the default version matchers.   1.0.+ is no
longer interpreted as a dynamic lookup.

So the correct definition is instead:

  <version-matchers>
     <exact-vm/>
     <latest-vm/>
     <sub-vm/>
     <range-vm/>
     <pattern-vm>
        <match revision="versionconfig"
pattern="${versionbase}\.\d+-(${config}|complete)"
args="versionbase,config" matcher="regexp"/> 
     </pattern-vm>
  </version-matchers>

A little more verbose than expected.   <chain-vm/> might be what I
really want here, but I haven't found a description of it.


Nathan

 

 

 -----Original Message-----
 From: Nathan Franzen
 Sent: Tuesday, July 06, 2010 12:31 PM
 To: 'ivy-use### @ant.apache.org'
 Subject: PatternVersionMatcher breaks dynamic revisions
 
 I've been experimenting with the version matchers, and after some
 experiments got a PatternVersionMatcher that was doing the lookups on
a
 particular dependency the way I wanted it too.   Unfortunately, I
then
 saw other dependencies -- which we working before -- now failing.
 
 Trying to break it down into the simplest test case, I have in my
 ivy.xml, either
         <dependency org="com.mmodal" name="devTools" rev="[4.0,)"
/>
 or
         <dependency org="com.mmodal" name="devTools" rev="4.0.+"
/>
 which should (and do) give me version 4.0.102
 
 
 now in ivy settings, I add or remove
      <version-matchers>
         <pattern-vm/>
      </version-matchers>
 which turns off or on the successful lookup.
 
 I glanced at the source of
 org.apache.ivy.plugins.version.PatternVersionMatcher, but I don't see
 anything obvious fishy.  I don't know what code is involved in
 initializing the set of version matchers, so it's hard for me to know
 where something's gone wrong.
 
 Of course I first used -verbose, but don't see anything helpful in
the
 output.
 
 Nathan
 
 
 
 
 

Does anyone know how to dynamically adjust a running endpoint's policy on
the fly, at runtime, without restarting anything in CXF? We have a DOM
tree
of the policy, but don't know what kind of interceptors, or actions to
write. We don't want to modify the source of CXF, but just leverage its
pluggability.

Thanks,
Ravi

Hello folks,

I'm trying to program a tool that gets an instance of a class that is a
description of a service that should be exposed. It doesn't seem that I
could use the simple frontend as the class is not the service but just a
description of the service. It includes informations about name,
inputparameters, types of the results and where to find the correct method
to call. 

Does anyone perhaps know how I could get a service out of that at runtime
using cxf?

curious regards
    Martin

Hi lovely people :)

I was wondering, within the context of the Spring DSL, is it possible
given a bean with @RecipientList to be used in
a <recipientList> element ?

TIA

Yiannis