Best unofficial Apache Server developers community |
arbitrary structure for joins redux
| |||||
| Jun 20, 2010 | |||||
 Borja Martín |
|
||||
| Tags: | |||||
Similar Threads
redux FOR-1200 (Was: Re: svn commit: r979542)
Gav... wrote: Obviously something is not right with my setup. This was an otherwise clean and up to date checkout, only changing the live-sites file. This is on my linux Ubuntu machine. I'll look into it. When the production of PDF docs was disabled (see dev list). Each of us who has/is committing the site changes needs to clean out our local site "deploy" directory. I mean remove the PDFs that are lingering from an old run, so as not to re-deploy them. IIRC then this is the same problem with any "deleted" documents. IIRC, i used "diff -rq" and some grep to get a list of differences between the "generated" set of docs and the "deploy" set of docs (sorry cannot remember the directory names under site-author), then turned that list into a shell-script of 'svn rm' commands. See FOR-1200 around 21 June 2010. -David > Author: gmcdonald > Date: Tue Jul 27 05:45:13 2010 > New Revision: 979542 > > URL: http://svn.apache.org/viewvc?rev=979542&view=rev > Log: > Publish from forrestbot > > Added: > forrest/site/asf-infrastructure.pdf (with props) > forrest/site/committed.pdf (with props) > forrest/site/compliance.pdf (with props) > forrest/site/contrib.pdf (with props) ...
Puppet Camp Redux - October 7th-8th in San Francisco
Hi all After the success of last year's Puppet Camp and Puppet Camp Europe we're going to be running Puppet Camp again in the United States. The current dates are October 7th and 8th (Thursday and Friday) and the location will be San Francisco - currently TBD but downtown San Francisco. So mark your diary! Like previous Camps there will be a small cost to cover food and venue (TDB but minimal). We're also looking for input on structure, format, speakers, and social events. We're presently looking at replicating the previous format of organized speakers in the morning and un-conference sessions in the afternoons. So if you'd like to speak, nominate someone else to speak, have an idea for a session (panels anyone?), or just some general feedback we'd love to have it. We're also looking for expressions of interest in a potential "advanced" or "developer" Puppet training earlier in that week. Thanks James Turnbull -- Puppet Labs - http://www.puppetlabs.com C: 503-734-8571
Adding arbitrary property on record field
It looks as though it's not possible to add an arbitrary property on a
record field. e.g., in the following example, although the schema
parses fine, the "alias" property gets thrown away:
{
"name": "KVPair",
"type": "record",
"fields" : [
{"name": "key", "type": "int", "alias": "EventTime"},
{"name": "values", "type": "bytes"}
]
}
I had read the Avro spec and thought this was actually allowed.
("Attributes not defined in this document are permitted as metadata, but
must not affect the format of serialized data.")
Am I wrong, or is this a bug/omission? Seems like it would be a really
useful feature to have. And near as I can tell, the only other way to
achieve the same behavior would be to do some kind of hack using the
field's "doc" attribute.
Thanks,
DR
execute arbitrary remote shell command against running instance
Largly based on mfojtik's previous patches, this simply adds the
ability to ssh into a running instance and run a command against
it, reporting the results.
Most likely will change to merge with Michal's keypair patch from
today and to include client support
---
server/deltacloud-core.gemspec | 1 +
.../drivers/mock/data/instances/inst3.yml | 9 ++++++
.../lib/deltacloud/helpers/application_helper.rb | 20 +++++++++++++
server/lib/deltacloud/models/instance.rb | 18 ++++++++++++
server/server.rb | 29
++++++++++++++++++++
server/views/instances/run.html.haml | 17 +++++++++++
server/views/instances/run.xml.haml | 14 +++++++++
7 files changed, 108 insertions(+), 0 deletions(-)
create mode 100644
server/lib/deltacloud/drivers/mock/data/instances/inst3.yml
create mode 100644 server/views/instances/run.html.haml
create mode 100644 server/views/instances/run.xml.haml
diff --git a/server/deltacloud-core.gemspec
b/server/deltacloud-core.gemspec
index 6b5a966..53571e4 100644
--- a/server/deltacloud-core.gemspec
+++ b/server/deltacloud-core.gemspec
@@ -65,6 +65,7 @@ require 'rake'
s.add_dependency('thin', '>= 1.2.5')
s.add_dependency('rerun', '>= 0.5.2')
s.add_dependency('json', '>= 1.2.3')
+ s.add_dependency('net-ssh', '>= 2.0.23')
s.add_development_dependency('compass', '>= 0.8.17')
s.add_development_dependency('nokogiri', '>= 1.4.1')
s.add_development_dependency('rack-test', '>= 0.5.3')
diff --git a/server/lib/deltacloud/drivers/mock/data/instances/inst3.yml
b/server/lib/deltacloud/drivers/mock/data/instances/inst3.yml
new file mode 100644
index 0000000..a84feca
--- /dev/null
+++ b/server/lib/deltacloud/drivers/mock/data/instances/inst3.yml
@@ -0,0 +1,9 @@
+:name: InstanceWithLocalhostAddress
+:state: RUNNING
+:image_id: img3
+:owner_id: mockuser
+:public_addresses: [ localhost ]
+:private_addresses: [ localhost ]
+:realm_id: us
+:instance_profile: !ruby/object:InstanceProfile
+ id: m1-small
diff --git a/server/lib/deltacloud/helpers/application_helper.rb
b/server/lib/deltacloud/helpers/application_helper.rb
index 94396d2..86b5c09 100644
--- a/server/lib/deltacloud/helpers/application_helper.rb
+++ b/server/lib/deltacloud/helpers/application_helper.rb
@@ -106,4 +106,24 @@ module ApplicationHelper
end
end
+ def store_private_key(id, key)
+ path = File.join('tmp', 'keys')
+ FileUtils.mkdir_p(path) unless File.directory?(path)
+ filename = File.join(path, "instance_#{id}.key")
+ File.open(filename, 'w') do |f|
+ f.puts key
+ end
+ return filename
+ end
+
+ def remove_private_key(key)
+ File.delete(key) if File.exists?(key)
+ end
+
+ def cdata(&block)
+ text = capture_haml(&block)
+ text.gsub!("\n", "\n ")
+ "<![CDATA[\n #{text}\n]]>"
+ end
+
end
diff --git a/server/lib/deltacloud/models/instance.rb
b/server/lib/deltacloud/models/instance.rb
index 9cf69b8..c2c9ff4 100644
--- a/server/lib/deltacloud/models/instance.rb
+++ b/server/lib/deltacloud/models/instance.rb
@@ -16,6 +16,7 @@
# License for the specific language governing permissions and limitations
# under the License.
+require 'net/ssh'
class Instance < BaseModel
@@ -35,4 +36,21 @@ class Instance < BaseModel
self.public_addresses = [] if self.public_addresses.nil?
self.private_addresses = [] if self.private_addresses.nil?
end
+
+ def run_command(cmd, username='', opts={})
+ hostname = self.public_addresses.first
+ return "No hostname/IP address specified" unless hostname
+ output = ""
+ Net::SSH.start(hostname, username || 'root', opts) do |session|
+ session.open_channel do |channel|
+ channel.on_data do |ch, data|
+ output += data
+ end
+ channel.exec(cmd)
+ end
+ session.loop
+ end
+ return output
+ end
+
end
diff --git a/server/server.rb b/server/server.rb
index 2516d3e..4271094 100644
--- a/server/server.rb
+++ b/server/server.rb
@@ -161,6 +161,12 @@ get "/api/instances/new" do
end
end
+get "/api/instances/:id/run" do
+ respond_to do |format|
+ format.html { haml :"instances/run"}
+ end
+end
+
collection :instances do
description <<END
An instance is a concrete machine realized from an image.
@@ -226,6 +232,29 @@ END
param :id, :string, :required
control { instance_action(:destroy) }
end
+
+ operation :run, :method => :post, :member => true do
+ description "Run command on instance"
+ param :id, :string, :required
+ param :cmd, :string, :required
+ param :private_key, :string
+ param :username, :string
+ control do
+ @instance = driver.instance(credentials, { :id => params[:id] })
+ private_key = store_private_key(params[:id], params[:private_key])
+ begin
+ @output=@instance.run_command(params[:cmd], params[:username], {
:keys => private_key })
+ rescue Exception => e
+ @failed = true
+ @output = "#{e.message}\n#{e.backtrace.join("\n")}"
+ ensure
+ remove_private_key(private_key)
+ end
+ respond_to do |format|
+ format.xml { haml :"instances/run" }
+ end
+ end
+ end
end
collection :hardware_profiles do
diff --git a/server/views/instances/run.html.haml
b/server/views/instances/run.html.haml
new file mode 100644
index 0000000..488322c
--- /dev/null
+++ b/server/views/instances/run.html.haml
@@ -0,0 +1,17 @@
+%h1 Run command
+
+%form{ :action => "/api/instances/#{params[:id]}/run", :method =>
:post}
+ %label
+ Command:
+ %input{ :name => 'cmd', :size => 30}/
+ %h3 Private key
+ %label
+ Paste private key here:
+ %p
+ %textarea{ :name => 'private_key', :cols => 30}
+ %h3 Authentication options
+ %label
+ Username:
+ %input{ :name => 'username', :size => 30}
+ %br/
+ %input{ :type => 'submit', :value => "Run"}
diff --git a/server/views/instances/run.xml.haml
b/server/views/instances/run.xml.haml
new file mode 100644
index 0000000..b91dbd7
--- /dev/null
+++ b/server/views/instances/run.xml.haml
@@ -0,0 +1,14 @@
+!!! XML
+%instance{:href => instance_url(@instance.id)}
+ %id<
+ =@instance.id
+ %name<
+ =@instance.name
+ %status
+ =@failed ? "FAILED" : "SUCCESS"
+ %command<
+ =cdata do
+ =params[:cmd]
+ %output<
+ =cdata do
+ =@output
Created: (AVRO-601) Enhance schema parser to allow arbitrary properties to be defined on a record fi
Enhance schema parser to allow arbitrary properties to be defined on a record field
Doing joins between column familes
So I am not sure if you guys are familiar with OCM . Basically it is
an ORM for Cassandra. Been testing it
So I have created model that has the following object relationship.
OCM generates the code from this that allows me to do easy
programmatic query from Java to Cassandra.
Object1-(Many2Many)->Object2-(Many2Many)->Object3-(Many2Many)->Object4-
(Many2Many)->Node
So my app gets the NODE and tries to query the dependency relationship
from Node->Object4->Object3->Object2->Object1.
I have compared the performance between Cassandra(with OCM) vs DB2.
The result is not very encouraging since the DB2 performance is
showing at least 3X faster than Cassandra. DB2 basically is just a
single call with a number of inner joins ..
Looking at the code, I think we might get a better performance if
somehow we can do the joins between objects within Cassandra server
rather than the client side. Right now , I am basically doing the
following.
Node node = new Node(connection,"nodeidentifier");
node.loadInfo(); // going to the wire ...?
node.loadObject4(); // this goes to the wire too ..
object4Keys = node.getObject4().getColumns.keys();
while(object4Keys.hasNextElement)
{
object4Key = object4Key.nextElement();
object4 = node.getObject4().get(object4Key);
object4.loadInfo(); // this goes to the wire
too ..
object4.loadObject3(); // this goes to the wire too ..
object3Keys = object4.getObject3().getColumns.keys();
while(object3Keys.hasNextElement)
{
object3Key = object3Key.nextElement();
object3 = node.getObject4().get(object4Key);
object3.loadInfo(); // this goes to the wire too ..
object3.loadObject2(); // this goes to the wire too ..
...
..
... until you get object1//
}
I think there is a lot of going back and forth between the client and
cassandra and if we can only move the relationship joins to Cassandra
server I think we can minimize the latency and improve the overall
performance on the query.
Is there a way to do a join across ColumnFamilies in Cassandra ?
Sanjay Radia joins the Hadoop PMC
The Hadoop PMC has voted to add Sanjay Radia to itself. Congratulations, Sanjay! Thanks for all of the hard work on Hadoop. -- Owen
cluster throwing errors when new or existing node joins
Hi
I have a setup of 4 nodes, whenever I am restarting any of the nodes, even
after deleting the data directories and commit log I get the following
error
ERROR 18:46:41,296 Fatal exception in thread
Thread[COMMIT-LOG-WRITER,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NullPointerException
at
org.apache.cassandra.db.Table$TableMetadata.getColumnFamilyId(Table.java:131)
at org.apache.cassandra.db.Table.getColumnFamilyId(Table.java:364)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103)
at
org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:475)
at
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 1 more
ERROR 18:46:41,297 Error in ThreadPoolExecutor
java.lang.NullPointerException
at org.apache.cassandra.db.Table.apply(Table.java:407)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
ERROR 18:46:41,299 Fatal exception in thread
Thread[ROW-MUTATION-STAGE:5,5,main]
java.lang.NullPointerException
at org.apache.cassandra.db.Table.apply(Table.java:407)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
ERROR 18:46:51,309 Error in ThreadPoolExecutor
java.lang.NullPointerException
at org.apache.cassandra.db.Table.apply(Table.java:407)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
ERROR 18:46:51,310 Fatal exception in thread
Thread[ROW-MUTATION-STAGE:6,5,main]
Kindly suggest what can be the reason for this error.
list of structure..
Hello everybody! We have puppet infrastructure integrated with ldap. I can pass different variables via puppetVars attribute in ldap. Currently, I need to write puppet class to configure mod_proxy server. mod_proxy servers has 5-10 configured sites. Every site configuration has some variables: site name, listen port, destination site name, destination port, protocol.. Does anybody has an idea, how can I place all this variables out of puppet classes? In ldap? Or maybe puppet have some internal database, like chef?
Erlang Doc structure
I've been playing with Erlang views on and off in CouchDB, and find the information on the internal Erlang structure of Docs lacking. Is there somewhere I can read up on this? I had initially assumed that a Doc was a list of tuples, but I am starting to suspect that I am wrong. Mike
Updated: (HIVE-158) table aliases dont work for sampled tables in joins
[
https://issues.apache.org/jira/browse...nels:all-tabpanel
]
Carl Steinbach updated HIVE-158:
Created: (DERBY-4679) Several left outer joins causes unstable query with incorrect results
Several left outer joins causes unstable query with incorrect results
Delete document Tree Structure
Hi,
I design document in couchdb to have relation something like
directory
|
A---------D
|
B---------C
|
|
E ------F
if i would like to http (DELETE B) will delete all down documents
what is the possible solution to use.
1. can i create view and find relation and delete all
of them ?
2. get relation by higher programming api (C#,PHP) and
then delete each one?
3. Do you have any solution to suggest on this.....?
Thanks,
A.
RE: Delete document Tree Structure
hi ,
I am very new to couchdb, in my design , i just keep parent id
of each documents.
such as if i would like to delete B, i know that to delete C(have B id)
and then I have find and
to delete E(have c id).... etc.
my question is ....
1. is this possible ????
2. how do i pass the document id (such as B) to view that i would
like to search and delete document under its?
3. dose couchdb support delete document in view or i have to query
all of them than use http api to delete ?
thanks for every ideas
A.
-------- Original Message --------
Subject:
Re: Delete document Tree Structure
Date:
Thu, 27 May 2010 06:43:58 +0200
From:
J Chris Anderson <jch### @gmail.com>
Reply-To:
use### @couchdb.apache.org <us### @couchdb.apache.org>
To:
use### @couchdb.apache.org <us### @couchdb.apache.org>
On May 26, 2010, at 8:48 PM, Aun... ??????? wrote:
Hi,
I design document in couchdb to have relation something like
directory
|
A---------D
|
B---------C
|
|
E ------F
if i would like to http (DELETE B) will delete all down
documents what is the possible solution to use.
1. can i create view and find relation and delete
all of them ?
2. get relation by higher programming api (C#,PHP)
and then delete each one?
3. Do you have any solution to suggest on
this.....?
It is common to store the full path to each item, on the item, so you'd
have
B > C > E > F stored on F
then you can view easily across the tree.
however, reparenting a node, (say, moving B to become a child of D)
require asynchronous processing or a bulk docs request and is not
transactional.
Chris
Thanks,
A.
JMS-MAP-JSON Improper Data structure
I am getting an odd error in my data from my broker. Using perl I am
changing
a map message to json using jms-map-json. This returns an improperly
structured version of my data.
'map' => [
{
'entry' => [
{
'string' => [
As you can see it is creating an array where the hash should be.
I now have to access data like @{@{$json->{map}}[0]->{entry}}; Which
is very
unpleasant. We did not have this issue prior. I used to be able to simply
do
@{$json->{map}{entry}} to access my data
This is with amq 5.3.2
Need advice about docs structure design.
Hy guy's. My model: Users has Bookmarks and Posts have idea keep Bookmarks and Posts in separate DB (for speed views). But in view i want fetch Bookmarks and Posts with user name, In what way i can do this? keep copy of users in each DB? or may be keep users in separate DB, fetch bookmarks or posts and then fetch users and then merge? Yes i know i can copy user name to doc, but have a trouble when user name is changed (it's force me use stale=ok in request view +background process to track changes and run rebuild index). I think it's will be cool crazy stuff if i can update docs(cached fields - not have effect to index) and say Couch you don't need rebuild index relax ;). Any advice?
adding request structure access to mod_header
We were dancing around trying to get RequestHeader (mod_header) to add
the authenticated user to a proxy request. The approaches we found
involved tricking other modules into calling ap_add_common_vars to
apr_table_addn(r->subprocess_env, "REMOTE_USER", r->user);
I decided to simplify by adding %{}r directives à la
RequestHeader set "x-webobjects-remote-user" "%{user}r"
which interrogate the request structure directly. I re-used
the request structure names 'cause, well, why not?
Please try to reply to <er### @w3.org>. <eri### @gmail.com> is a
temporary hack as ezmlm seems to subscribe the address in the
Return-Path: instead of the From: (grr!).
adding request structure access to mod_header
We were dancing around trying to get RequestHeader (mod_header) to add
the authenticated user to a proxy request. The approaches we found
involved tricking other modules into calling ap_add_common_vars to
apr_table_addn(r->subprocess_env, "REMOTE_USER", r->user);
I decided to simplify by adding %{}r directives à la
RequestHeader set "x-webobjects-remote-user" "%{user}r"
which interrogate the request structure directly. I re-used
the request structure names 'cause, well, why not?
Please try to Cc: <er### @w3.org>. <eric### @gmail.com> is a
temporary
hack as ezmlm seems to subscribe the address in the Return-Path:
instead of the From:
Assigned: (ARIES-353) SPI-Fly needs changes in project structure, so that itests and other subprojec
[
https://issues.apache.org/jira/browse...nels:all-tabpanel
]
David Bosschaert reassigned ARIES-353:
| |||||