Tuning garbage collection

Data store garbage collection: interrupt mark

CollectionUtils.removeAll(Collection collection, Collection remove) calls
ListUtils.retainAll(collection, remove)

Hi Gurus,

Any advise on initial tuning values for apache MinSpareServers,
MaxSpareServers, and StartServers, tcp tunings, ulimits, sysctl....

I'm running svn 1.6 over apache2 pre-fork. System load goes high as
much as 10 during heavy usage.

What would be your recommended MaxRequestsPerChild for heavy svn usage?

Thanks,


West

Weijunli,

I also have an environment that has similar very large datasets with
strict
latency.  Can you please elaborate on the custom changes you added to
cassandra to meet these sla, either code or configuration.  i am very
interested in learning more about the internal workings of cassandra and
performance.

Thanx,
Artie

On Thu, May 6, 2010 at 10:06 AM, Weijun Li <weiju### @gmail.com>
wrote:

 Our use case is a little different: our server is a typical high
volume
 transaction server that processes more than half billion requests per
day.
 The write/read ratio is close to 1, and the cluster needs to serve
10k
 write+read with strict latency (<20ms) otherwise the client will
treat it as
 failure. Plus we have hundreds of millions of keys so the generated
sstable
 files are much bigger that the ram size. In this case using mmap will
cause
 Cassandra to use sometimes > 100G virtual memory which is much
more than the
 physical ram, since we are using random partitioner the OS will be
busy
 doing swap.

 I have finally customized cassandra to meet the above requirement by
using
 cheap hardware (32G ram + SATA drives): one thing I learned is that
you have
 to carefully avoid swapping, especially when you need to cache most
of the
 keys in memory, swap can easily damage the performance of your in
memory
 cache. I also made some optimizations to reduce memory/disk
consumption and
 to make it easier for us to diagnose issues. In one word: cassandra
is very
 well written but there's still huge potential for you to improve it
to meet
 your special requirements.

 -Weijun


 On Wed, May 5, 2010 at 9:43 AM, Jordan Pittier
<jordan.p### @gmail.com>wrote:

> I disagree. Swapping could be avoided. I don't know Cassandra
internals
> mechanisms but what I am expecting is that whenever I want to
read rows that
> are not in RAM, Cassandra load them from hard drive to RAM if
space is
> available, and, if RAM is full to reply my query without saving
rows in RAM.
> No need for swapping.
>
> I have no try yet to change DiskAccessMode to standard, I hope it
will
> help me.
>
> Another thing : please dont post your benchmark figures without
any
> explanation on the work load generator or your cluster settings.
It really
> doesn't make any sense...
>
>
> On Wed, May 5, 2010 at 6:16 PM, Weijun Li
<weij### @gmail.com> wrote:
>
>> When you have much more data than you can hold in memory, it
will be
>> difficult for you to get around of swap which will most
likely ruin your
>> performance. Also in this case mmap doesn't seem to make much
sense if you
>> use random partitioner which will end up with crazy swap too.
However we
>> found a way to get around read/write performance issue by
integrating
>> memcached into Cassandra: in this case you need to ask
memcached to disable
>> disk swap so you can achieve move than 10k read+write with
milli-second
>> level of latency. Actually this is the only way that we
figured out that can
>> gracefully solve the performance and memory issue.
>>
>> -Weijun
>>
>>
>> On Wed, May 5, 2010 at 8:19 AM, Ran Tavory
<rant### @gmail.com> wrote:
>>
>>> I'm still trying to figure out where my slowness is
coming from...
>>> By now I'm pretty sure it's the reads are slow, but not
sure how to
>>> improve them.
>>>
>>> I'm looking at cfstats. Can you say if there are better
configuration
>>> options? So far I've used all default settings, except
for:
>>>
>>>     <Keyspace Name="outbrain_kvdb">
>>>       <ColumnFamily CompareWith="BytesType"
Name="KvImpressions"
>>> KeysCached="50%"/>
>>>
>>> 
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackAwareStrategy</ReplicaPlacementStrategy>
>>>      
<ReplicationFactor>2</ReplicationFactor>
>>>
>>> 
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>>>     </Keyspace>
>>>
>>>
>>> What does a good read latency look like? I was expecting
10ms, however
>>> so far it seems that my KvImpressions read latency is
30ms and in the system
>>> keyspace I have 800ms :(
>>> I thought adding KeysCached="50%" would improve my
situation but
>>> unfortunately looks like the hitrate is about 0. I
realize that's
>>> application specific, but maybe there are other magic
bullets...
>>>
>>> Is there something like adding cache to the system
keyspace? 800 ms is
>>> pretty bad, isn't it?
>>>
>>> See stats below and thanks.
>>>
>>>
>>> Keyspace: outbrain_kvdb
>>>         Read Count: 651668
>>>         Read Latency: 34.18622328547666 ms.
>>>         Write Count: 655542
>>>         Write Latency: 0.041145092152752985 ms.
>>>         Pending Tasks: 0
>>>                 Column Family: KvImpressions
>>>                 SSTable count: 13
>>>                 Space used (live): 23304548897
>>>                 Space used (total): 23304548897
>>>                 Memtable Columns Count: 895
>>>                 Memtable Data Size: 2108990
>>>                 Memtable Switch Count: 8
>>>                 Read Count: 468083
>>>                 Read Latency: 151.603 ms.
>>>                 Write Count: 552566
>>>                 Write Latency: 0.023 ms.
>>>                 Pending Tasks: 0
>>>                 Key cache capacity: 17398656
>>>                 Key cache size: 567967
>>>                 Key cache hit rate: 0.0
>>>                 Row cache: disabled
>>>                 Compacted row minimum size: 269
>>>                 Compacted row maximum size: 54501
>>>                 Compacted row mean size: 933
>>> ...
>>> 

Hi All,

I noticed using Firefox with a plugin yslow that it recommends using only
4 virtual hosts instead of the 16 I have for serving images. I am using a 6
core system and want to make sure I take advantage of it using Apache APR
so I setup more than 4 virtual hosts.

I am using multiple virtual hosts for serving images since I heard that
can improve user response time where they have browsers with 4 or more
concurrent connections supported. I have a screenshots page where I have 24
thumbnails being loaded into the browser and I am trying to tune APR for
the best user response time.

So any best practice for this you all would recommend?

Thanks,
-Tony


      

https://issues.apache.org/bugzilla/show_bug.cgi?id=49230

--- Comment #1 from Mark Thomas <mar### @apache.org> 2010-06-22
05:25:02 EDT ---
Thanks for the patch. It has been applied to 7.0.x and will be included in
7.0.1 onwards.

I have also proposed the patch for back-port to  to 6.0.x

https://issues.apache.org/bugzilla/show_bug.cgi?id=49230

--- Comment #2 from sylvain.### @gmail.com 2010-06-22 15:36:52 EDT ---
IMHO, logging an error if the class is not found might induce some users
of
non-Sun JVMs in error...

an INFO level is probably enough, with a message explaining that it might
be
normal with some JVMs.

https://issues.apache.org/bugzilla/show_bug.cgi?id=49230

--- Comment #3 from Mark Thomas <mar### @apache.org> 2010-06-23
03:44:14 EDT ---
I've aligned the patch with how we handle other Sun specific issues for
non-Sun
JVMs.

https://issues.apache.org/bugzilla/show_bug.cgi?id=49230

Mark Thomas <mar### @apache.org> changed:

           What    |Removed                     |Added

Every so often I see intances of ModSecurity set a session variable and
then
not be able to retrieve the variable later on. In other words, I have a
rule
that creates a SESSION.username variable, which gets set as follows:

[08/Jul/2010:09:30:24 +1000]
[xxx.xxx.xxx/sid#8fb6de7][rid#cfa8c10][/index.asp][9] Set variable
"SESSION.username" to "123456".

but sometimes (and only sometimes) when I try to use the SESSION.username
variable the variable does not resolve to a value. When this happens I
don't
see "Resolved macro %SESSION.username" in the logs, but I do see the "Set
variable "SESSION.username to ..." which tells me that the variable is
set,
but sometimes seems to disappear. Has anyone experienced this before?

Thanks,
Chris

Is there a way to set a collection variable from a Lua script? Perhaps
something like m.setvar. Alternatively, can I return the value I want to
set
in a collection value to modsec and use the rules language to set a
collection variable?

Thanks

Hi,

I have this class generated by wsdl2java (comments remove):
@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "RunExtensionArgumentsType", propOrder = {
     "extensionArgument"
})
public class RunExtensionArgumentsType
     implements Serializable
{

     @XmlElement(name = "ExtensionArgument")
     protected List<RunExtensionType> extensionArgument;

     public List<RunExtensionType> getExtensionArgument() {
         if (extensionArgument == null) {
             extensionArgument = new ArrayList<RunExtensionType>();
         }
         return this.extensionArgument;
     }

}


I have been serialising it for ages using the following code:
     static {
         aegisContext.initialize();

         TypeMapping tm = aegisContext.getTypeMapping();
         Type aegisType = tm.getType( RunExtensionArgumentsType.class );
         if( aegisType == null )
         {
             // Make a new type and register it for the future.
             log.debug( "Registering type for RunExtensionArgumentsType"
);
             TypeCreator tc = tm.getTypeCreator();
             aegisType = tc.createType( RunExtensionArgumentsType.class );

             QName qName = new QName( "http://groupgti.com/s-service", 
"RunExtensionType");
             tm.register( RunExtensionArgumentsType.class, qName, 
aegisType );
         }
     }

     public static String extensionArgsToString( 
RunExtensionArgumentsType extensionArgs )
     {
         try
         {
             StringWriter stringWriter = new StringWriter();
             XMLStreamWriter xmlWriter = 
xmlOutputFactory.createXMLStreamWriter( stringWriter );

             AegisWriter<XMLStreamWriter> writer = 
aegisContext.createXMLStreamWriter();

             TypeMapping tm = aegisContext.getTypeMapping();
             Type aegisType = tm.getType( RunExtensionArgumentsType.class
);
             writer.write( extensionArgs, aegisType.getSchemaType(), 
true, xmlWriter, aegisType );

             xmlWriter.close();
             return stringWriter.toString();
         }
         catch( Exception ex )
         {
             log.error( ExceptionStringConverter.ExceptionWithCauses( ex 
) );
             return null;
         }
     }

Now I come to deserialize it using this:
     public static RunExtensionArgumentsType extensionArgsFromString( 
String extensionArgsString )
     {
         try
         {
             StringReader stringReader = new StringReader( 
extensionArgsString );
             XMLStreamReader xmlReader = 
xmlInputFactory.createXMLStreamReader( stringReader );

             AegisReader<XMLStreamReader> reader = 
aegisContext.createXMLStreamReader();

             TypeMapping tm = aegisContext.getTypeMapping();
             Type aegisType = tm.getType( RunExtensionArgumentsType.class
);
             Object result = reader.read( xmlReader, aegisType );

             xmlReader.close();
             return (RunExtensionArgumentsType)result;
         }
         catch( Exception ex )
         {
             StringBuilder msg = new StringBuilder();
             msg.append( "Exception whilst trying to convert \" 
").append( extensionArgsString ).append( "\" to 
RunExtensionArgumentsType:\n\t" );
             log.error( ExceptionStringConverter.ExceptionWithCauses( 
msg, ex ) );
             return null;
         }
     }

but it hits this exception:
         org.apache.cxf.aegis.DatabindingException:
  No write method for property 
{http://groupgti.com/s-service}extensionArgument in class 
com.groupgti.s_service.RunExtensionArgumentsType:
  No write method for property 
{http://groupgti.com/s-service}extensionArgument in class 
com.groupgti.s_service.RunExtensionArgumentsType:
  org.apache.cxf.aegis.type.basic.BeanType:writeProperty @
BeanType.java:272
  org.apache.cxf.aegis.type.basic.BeanType:readObject @ BeanType.java:163
  org.apache.cxf.aegis.AegisXMLStreamDataReader:read @ 
AegisXMLStreamDataReader.java:82
  org.apache.cxf.aegis.AegisXMLStreamDataReader:read @ 
AegisXMLStreamDataReader.java:31
 
com.groupgti.services.service.XmlConversionSupport:extensionArgsFromString
@ XmlConversionSupport.java:162
  com.groupgti.services.XmlConversionsTest:testStringToExtensionArgs @ 
XmlConversionsTest.java:65


Now clearly this class can be deserialized 'cos it's used in by the web 
service, so what do I have to do to make it use a get().add() instead of 
a set() for collections?

Thanks

Jim

On 24/06/2010 17:41, Finn Bock wrote:
 2010/6/24 Adrian A:
> What if we make OptionGroup extend Option?
> Wouldn't that simplify the problem?
 
 Indeed it would, but with a tiny risk of breaking existing user code
like:

I don't think we need to enforce inheritance onto Option and OptionGroup.
The issue at hand is minor
at best, no need to complicate the framework.

Bob

Hi All,

I been trying mapReduce and I'm currently stuck because the resulting
collection looks like:

{ _id: { MY_KEY_STUFF }, value: { MY_VALUE_STUFF } }

But it happens that I would like to have only the contents of
MY_VALUE_STUFF
in my permanent collection, because I want it easy to index by a couple of
fields in there and it makes it diffcult by having them embedded.

I tried the finalize function but it doesn't seem to have tha purpose as
my
resulting collection looks the same.

Thanks in advance
Kevin

Use typesafe collection

Good Morning,

I am seriously stumped on this issue.  I am collecting a single object
from
MyBatis, which contains, among other things, an ArrayList of another
object.  I have tried a single SQL query with a collection type that maps
to
a ResultSet, and one that maps to a select (in essence running a second
query).  In both cases, if a single result is returned, I am given either
a
null or an empty ArrayList... never an ArrayList containing a single item.
If I add a second item in to the table, I get the expected results.  Of
course, when there are no results, I also get an empty ArrayList (what I
want).

I don't get it.

Here's snippets of what my XML mapper looks like:

    <resultMap id="aarResultMap" type="AARBean">
        <id property="aarID" column="aar_id"/>
        <result property="atOffice" column="at_office"/>
        <result property="processingComplete"
column="processing_complete"/>
        <result property="comments" column="comments"/>
        <association property="updateDate" javaType="TimeBean">
            <constructor>
                <arg column="update_date"
javaType="java.util.Date"/>
            </constructor>
        </association>
        <collection property="players" javaType="ArrayList"
column="aar_id"
ofType="AARPlayerReviewBean" select="selectReviews"/>
    </resultMap>

    <resultMap id="reviewResultMap" type="AARPlayerReviewBean">
        <id property="player.playerID" column="player_id"/>
        <result property="playerComments" column="player_comments"/>
        <association property="player" resultMap="playerResultMap"/>
        <association property="playerRole"
resultMap="playerRoleResultMap"/>
    </resultMap>

    <select id="selectReviews" parameterType="int"
resultMap="reviewResultMap">
        SELECT
          p2.first_name || ' ' || p2.last_name player_name,
          p2.player_id,
          p2.email_address,
          p2.phone_number,
          p2.aor_lead_order,
          aors2.aor_id,
          aors2.aor_name,
          aors2.color aor_color,
          aors2.is_current,
          pr.role_id,
          pr.role_desc,
          pal.comments player_comments
        FROM
          player_aar_lines pal,
          players p2,
          player_roles pr,
          aors aors2
        WHERE  pal.player_id      = p2.player_id(+)
        AND pal.role_id           = pr.role_id(+)
        and p2.aor_id             = aors2.aor_id(+)
        AND pal.aar_id            = #{aarId}
    </select>

Am I crazy or should this be working?

Rob

Hi,

Is there a way to integrate the JAXB 2.1 Collection setter injection
plugin[1] to cxf-codegen-plugin ?

Regards.

[1] https://jaxb2-commons.dev.java.net/co...-setter-injector/

Hi all!

I'm trying to understand the correct use of collections mapping of the
jackrabbit OCM.

	@Collection(proxy=true, 
			  elementClassName=SomeClass.class,
collectionConverter=NTCollectionConverterImpl.class)

I found something like this but I really don't understand what's going
on with this.

Is there any good explanation about how to use? The main page is almost
empty...

Thank you.

You cannot use '$' or '\0', it shouldn't start with "system.", and it can't
be the empty string.


On Thu, Jul 29, 2010 at 3:23 AM, Alatriste <ess### @gmail.com>
wrote:

 Hi,

  is there any restriction on using '@', '.', '%' and so on in a
 collection name?.
 I've checked with "use### @contactLists.com" and it properly creates the
 collection.

 Thank you for this great software.

 --
 

Thanks, I created a ticket:

http://jira.mongodb.org/browse/SERVER-1502

On Jul 27, 6:35 pm, Eliot Horowitz <eliotho### @gmail.com> wrote:
 Interesting - but no :(
 If you add a jira would be pretty easy to add though.



 On Tue, Jul 27, 2010 at 12:31 PM, Thijs Cadier
<th.### @80beans.com> wrote:
 > Hi,

 > We've been looking at using Map/Reduce to generate a collection
of pre-
 > formatted data to a permanent collection so we can do some
complicated
 > stuff on every page view that we wouldn't be able to do
otherwise.
 > We're planning to re-generate the collection every day. This is
 > working out great so far.

 > We'll be doing a lot of queries on an Array field in this
generated
 > collection and we would need an index to get the performance we
want.
 > My question is: Is it possible to add indexes to this collection
 > before the permanent collections are swapped?

 > Thanks

 > Thijs

 > --
 >