pmxbot IRC Log Viewer

[00:20:30] <cheeser> joannac: i am now. back home for the evening.

[01:08:33] <decimal_> could someone do me a huge favor and explain why the following function isn't returning any documents when fired with a proper id? http://pastebin.com/QiiHr6Xx

[01:10:48] <joannac> decimal_: checked the _id exists by running in the mongo shell?

[01:11:14] <decimal_> id is not the document _id

[01:12:08] <joannac> okay, checked the eventid exists...

[01:12:58] <decimal_> it does

[01:14:33] <joannac> and the eventid is ingested correctly?

[01:15:27] <joannac> what's the dataBase variable?

[01:18:40] <decimal_> the databasevar si the database in question

[02:20:38] <lazypower> ls

[02:53:34] <decimal_> does this make sense for returning a document from a findonequery using nodejs + mongodb? callback(undefined, eventDoc);

[02:53:45] <decimal_> undefined would be err

[02:53:52] <decimal_> and eventDoc is the returned document

[04:43:05] <Bluefoxicy> Everybody is stuck on this "don't touch mongodb if you EVER want to do relational stuff" dogma

[04:43:07] <Bluefoxicy> it's bullshit.

[04:43:42] <Bluefoxicy> you can select a document out with find(), then find({'$in':{'id': $objectidlist}})

[04:44:14] <Bluefoxicy> you can write a map-reduce to do that and just return the whole document with a list of objectids expanded to documents from another collection--that's a join operation.

[04:44:20] <Bluefoxicy> relational is doable.

[04:45:46] <cheeser> or use something like morphia that will atuo-fetch references for you

[04:46:41] <retran> often you dont need a map-reduce

[04:46:59] <retran> just use a simple cursor/ while loop

[04:49:04] <retran> there's some really bad blog articles about people wanting to use Mongo as a RDBMS and being disapointed

[04:49:45] <retran> here's a famous one

[04:49:46] <retran> http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/

[04:50:04] <retran> read this and you'll want to ring this guys neck

[04:53:05] <rcombs> I was really hoping that was a truncation in the URL slug

[04:53:34] <rcombs> "Why you should never use MongoDB for <thing an relational DB would be better for>" or somesuch

[04:53:59] <retran> nope

[04:54:06] <retran> it's a really fully ignorant article

[04:54:12] <retran> pure FUD

[04:55:47] <retran> the author makes some weird declaration that "social data *is* relational" and therefore Mongo is not suitable

[04:56:20] <rcombs> wat

[04:56:34] <retran> and only evidence provided for these points are cute little graphics and photographs

[04:58:24] <retran> strangely, also seems to imply that the Mongo tutorial regarding TV shows is misleading people into thinking Mongo's is ideal when it's really not

[04:58:44] <retran> the article should really be titled

[04:58:56] <retran> "how i fucked up a Mongo DB design"

[04:59:35] <retran> or maybe "how i got frustrated using a system novel to me"

[05:11:02] <halfamind> Can anyone help me understand why this matches docs where the 2 fields are equal?

[05:11:02] <halfamind> db.coll.find({"$where": "this.fooID != this.barID"})

[05:15:55] <retran> i believe mongo more than i believe you that they're equal

[05:23:54] <joannac> halfamind: works for me. Pastebin a document that incorrectly matches?

[05:24:31] <joannac> halfamind: http://pastebin.com/ATuicjii

[05:24:42] <Bluefoxicy> also that article shows more fundamental misunderstanding.

[05:25:03] <Bluefoxicy> What you want for a social network is some sort of graph based database like node4j or something

[05:25:19] <Bluefoxicy> that shows relationships between things and other things

[05:25:49] <Bluefoxicy> The nodes themselves would carry IDs that may function as _id fields in MongoDB (even maybe being objectid()) or keys in PostgreSQL

[05:26:05] <Bluefoxicy> because yeah

[05:26:13] <Bluefoxicy> you're storing a bunch of different types of data that need different data models

[05:44:32] <halfamind> joannac: Thanks! My fooID and barID are objectiDs… maybe that's to blame?

[05:50:18] <joannac> halfamind: correct, you can't directly compare objectIDs

[05:50:43] <joannac> http://pastebin.com/jF51vYdR

[05:50:58] <joannac> that relies on them being objectIDs now, which is not idea

[05:51:01] <joannac> ideal*

[05:51:20] <joannac> You should probably come up with a different way of doing whatever you're doing

[05:53:02] <halfamind> joannac: Thanks. My thoughts as well, especially since the .equals() comparison fails when fooID is null.

[08:09:01] <bin> guys a question..

[08:09:04] <bin> if my primary is down

[08:09:10] <bin> and it reconnect as secondary

[08:09:20] <bin> how to set rs.slaveOk() to be set automatically

[08:09:21] <bin> ..

[08:43:09] <naillwind> is it somehow possible to create an array before $addToSet if it turns out that the array doesn't already exist in a document? eg. { _id: "abc", data: { items: "" } } db.colleciton.update({ _id: "abc" }, { $addToSet: { data: { friends: [{ friendId: "xyz" }] }}}); <- example doesn't work, but is there something like this that does?

[08:58:23] <bin> anyone alive ?

[09:09:53] <Nodex> naillwind : it's a watse of space to create it so I highly doubt it

[09:11:25] <tiller> naillwind: I'm not sure your addToSet is correct

[09:11:58] <tiller> $addToSet: { "data.friends": {friendId: "xyz"}}}

[09:21:49] <bin> is it okey to use db.getMongo().setSlaveOk()

[09:21:51] <bin> on master

[09:21:59] <bin> the idea is to make the process automatic ..

[09:22:12] <bin> everytime node is connected to set setSlaveOk()..

[09:48:52] <arvin_> hello guys

[09:49:05] <arvin_> i would like to seek help on my replicaset-shard setup problem

[09:49:19] <arvin_> Adding a replicaset to a shard says that servers are not part of data1 replica set

[09:49:32] <arvin_> But when you add a single server, it specifically says that it is part of "data1"

[09:49:58] <arvin_> here are the shell output, http://pastebin.com/MsJ3XS0d i hope someone can help me

[09:58:03] <[AD]Turbo> hola

[10:09:31] <arvin_> Anyone here?

[10:09:53] <Derick> sure

[10:10:04] <Derick> just ask your questions, and people that know an answer will reply

[10:10:13] <Derick> IRC is asynchronous

[10:35:02] <Nodex> lol @ Derick - top answer

[10:36:10] <Derick> ?

[10:37:25] <ron> Derick: Nodex was being asynchronous.

[10:38:33] <Nodex> your "async" answer ;)

[10:47:51] <arvin_> i got a problem with adding a replica set on a shard

[10:48:01] <arvin_> here is the link http://pastebin.com/MsJ3XS0d i hope someone can help me

[10:50:51] <joannac> arvin_: WHat happens if you just specify one member instead of all three?

[10:54:28] <arvin_> joannac: it say it is part of the data1 replicaset

[10:54:56] <joannac> sh.addShard("data1/s5.dev:28001)

[10:55:02] <joannac> oops, close the quote

[10:58:06] <arvin_> it work

[10:58:09] <arvin_> it worked

[10:58:37] <arvin_> it also added all the servers that was part of the replicaset

[10:59:39] <arvin_> so i guess the wiki is outdated, you dont need to add all servers but just one server that is part of a set to add the whole replicaSet.

[10:59:43] <arvin_> thanks joannac

[11:00:45] <joannac> what wiki?

[11:01:04] <arvin_> http://docs.mongodb.org/manual/tutorial/convert-replica-set-to-replicated-shard-cluster/

[11:01:51] <joannac> ...that's not a wiki

[11:02:11] <joannac> anyway, it's not wrong to add all 3, you just don't need to

[11:02:21] <joannac> something is definitely weird with your hosts

[11:02:31] <arvin_> ohh

[11:02:53] <joannac> when you add "s5.dev:28001" by itself it thinks it's part of the "datahub1" replica set

[11:03:08] <arvin_> yeah i forgot to change it to data1

[11:03:31] <arvin_> company policy not to use internal names outside

[11:04:05] <arvin_> Anyways i'll try to replicate it on another occasion to rule out it is not my hosts Thanks @joannac

[11:46:38] <bin> woop woop guys :)

[11:46:50] <bin> is it a problem to call slaveOk on master ?

[11:56:10] <dawik> hey. i see references to "cond" and "op" what do this mean?

[12:16:03] <Nodex> cond and op ?

[12:16:12] <Nodex> perhaps add some context to that?

[12:34:29] <runa_> heyas. I have 3.5 MM json docs (in separated files) I want to load in a mongo collection. any hints?

[12:34:58] <runa_> mongoimport --jsonArray says it will load up to 16MB

[12:37:46] <runa_> um. afaik, mongoexport --collection output is just a concatenation of json docs?

[12:40:20] <Nilshar> Hello all, there is something I don't understand in my mongo setting. I'm using php driver to connect to a replicaset, giving to php master and slave host. all works fine when all servers are up, but if I shut down the slave host (not only the mongod, but all the host), then all connection to mongodb are slow (>3s), any idea why ?

[13:09:55] <Mephastos> g'day

[13:16:43] <Nodex> d'good

[13:22:12] <F|shie> hello all im trying to install luamongo, but i get this error while running make. mongo_bsontypes.cpp: In function ‘int bson_type_ObjectID(lua_State*)’:mongo_bsontypes.cpp:76: error: ‘gen’ is not a member of ‘mongo::OID’ make: *** [mongo_bsontypes.o] Error 1. Got source from https://github.com/moai/luamongo

[13:32:45] <ninkotech> hi... how many concurrent clients can single mongodb handle well?

[13:45:03] <Nodex> ninkotech : connections are pooled by drivers

[13:45:11] <Nodex> that's hard to answer

[13:45:59] <ninkotech> is it bad idea to use mongodb server from 500 computers? (up to 5 running apps on each computer)?

[13:46:39] <Nodex> I've thrown 45,000 concurrent requests at a mongod and it stood up

[13:46:47] <Nodex> (in a benchmark)

[13:46:53] <ninkotech> sounds good :)

[13:47:07] <Derick> Nodex: how does that work with a 20k connection limit? :-)

[13:47:13] <ninkotech> but i am concerned more about number of connections..

[13:55:52] <Nodex> pass

[13:56:02] <Nodex> ab -n 45000 .....

[13:56:08] <Nodex> it didn't die ;)

[13:56:22] <Nodex> sorry -c 45000

[13:56:34] <Nilshar> ab isn't really reliable for that...

[13:57:03] <Nodex> always works fine for me

[13:57:20] <Nodex> perhaps it throttled it, who knows

[13:57:29] <Nilshar> probably depends what you call "works fine".

[14:01:07] <ninkotech> hmm i will prolly do small test, running it on 100+ nodes to see how it will work

[14:02:00] <Derick> ninkotech: mongod will handle that, but your network might not

[14:02:23] <ninkotech> those changes will be small...

[14:02:36] <ninkotech> just 2k documents updated a lot

[14:02:52] <Derick> are those in place changes?

[14:03:03] <ninkotech> 4/5 will be

[14:03:06] <Derick> or are the documents growing?

[14:03:17] <ninkotech> ha, growing prolly

[14:03:27] <Derick> you might want to look at preventing that

[14:03:32] <ninkotech> why?

[14:03:44] <Derick> first of all, there is a 16MB limit per document

[14:03:45] <ninkotech> i am still designing the architecture...

[14:03:55] <ninkotech> Derick: growing max to 4k :)

[14:04:05] <Derick> and secondly, lots of growing documents are not the fastest as things might need to be moved on disk

[14:04:17] <Derick> ninkotech: can I see an example of such a document, and which part grows?

[14:05:21] <ninkotech> Derick: i dont have that yet... tomorrow maybe, i just started

[14:05:25] <Derick> ok!

[14:05:36] <Derick> ninkotech: pass it by here/me if you want somebody to look at it

[14:05:44] <ninkotech> i just wanted to check out if 500x5 connections are doable

[14:05:51] <ninkotech> before i will shard to another server

[14:06:08] <ninkotech> as my number of connections might grow up to 30k

[14:06:43] <ninkotech> (x5)

[14:06:55] <Derick> with only 4k * 2k data it's not a lot of data of course

[14:07:32] <ninkotech> Derick: its 40GB now in postgres, with all metadata, and its slow like hell :)

[14:08:29] <Derick> how can 4000 x 2000 be 40GB? it's more like 8 MB

[14:08:32] <ninkotech> redesigning app to be able to scale, but i am not sure if hundreds (later thousands) computers connecting will work well

[14:08:57] <ninkotech> Derick: its 200.000.000 * 2-4 kb

[14:09:01] <ninkotech> + metadata

[14:09:07] <Derick> 14:00 <ninkotech> just 2k documents updated a lot

[14:09:13] <Derick> I probably misread that

[14:09:24] <ninkotech> i have 200 millions of them now

[14:09:32] <ninkotech> almost

[14:09:44] <ninkotech> and they are bigger in postgres

[14:09:52] <Derick> I would probably suggest to use mongoS and shard from the start.

[14:10:07] <ninkotech> Derick: sounds good

[14:10:08] <Derick> mongoS can be used for connection pooling *I think*.

[14:10:10] <ninkotech> thanks for tips

[14:10:26] <Derick> but starting with a sharded setup (even though it's just one shard) seems like a clever idea in your case.

[14:11:49] <ninkotech> Derick: i am still relatively new to mongo... i just played with it for few days

[14:12:32] <Derick> ok :)

[14:12:37] <Derick> feel free to ask here!

[14:12:53] <ninkotech> thanks... so i will make a shard and run some tests now :)

[14:13:04] <ninkotech> i will soon see how well it behaves :D

[14:13:06] <Derick> playing = fun™

[14:17:29] <ninkotech> :)

[14:34:48] <Nodex> Nilshar : the normal definition really you know - works?

[14:37:26] <Nilshar> Nodex: are you sure it send 45k simultaneous connections ? no you can't :)

[14:38:45] <bin> guys

[14:38:55] <bin> is ot ok to invoke slaveOk() on primary?

[14:39:13] <cheeser> that setting applies more to your connection than the host.

[14:39:18] <kali> bin: yes

[14:39:26] <kali> bin: it's a noop

[14:39:29] <Nodex> Nilshar : that's not what you asked

[14:39:49] <Derick> bin: the drivers won't send that onwards anyway

[14:40:02] <Derick> they will only send that bit to secondary nodes

[14:40:13] <Nilshar> Nodex: that was not me asking, but that was your answer :)

[14:40:25] <Derick> setting slaveOk in a driver means that they can select a secondary, however, slaveOk is depcreated in favour of read preferences.

[14:41:06] <bin> well i want to automates the process

[14:41:17] <bin> if primary got down and after that got back into the game

[14:41:31] <bin> as a secondary i want to read from it

[14:41:40] <Derick> yeah, for that slaveOk works... but your app needs to deal with replication lag then too

[14:41:57] <bin> what do you mean ?

[14:42:09] <Derick> replication is asynchronous - secondaries can be behind

[14:42:19] <Derick> with slaveOk you *can* get outdated data

[14:42:35] <kali> s/can/will/

[14:42:38] <Derick> also, slaveOk does do nothing for writing/updating data, as that *has* to go to a primary

[14:42:54] <Derick> kali: well, it depends ... I've never seen it - but you do indeed need to take care of it.

[14:43:59] <bin> Derick well but i set writng concerns ..

[14:44:21] <kali> Derick: i've seen it... as many other subtle problem, it only manifests on loaded replicated system, like a production environment. never on development :)

[14:44:35] <Derick> bin: you can set write concerns, but that has nothing to do with slaveok/read preferences

[14:44:41] <Derick> kali: of course :-)

[14:45:10] <bin> well if you set write concerns you wait till for example replicated all write operations to secondaries..

[14:45:27] <Nodex> Nilshar : you asked a question earlier.

[14:45:32] <bin> so i send write operation ... my primary goes down ..

[14:45:33] <Nodex> Guess another one for the ignore list

[14:45:36] <bin> there is a rollback

[14:45:47] <Derick> bin: yeas, but that write will still fail then

[14:46:03] <bin> well yes but the current information from read will be up to date

[14:46:10] <bin> because no other write operation has been executed..

[14:46:56] <Derick> a write concern of w=3 will *not* guarantee that *other* connections will see the new data on all nodes at the same time - it only works for the connection/client that does the w=3 write btw

[14:49:24] <bin> sorry but can you explain it a little bit with an example ..

[14:49:25] <bin> sorry mate

[14:50:12] <Derick> sure

[14:50:22] <Derick> so, say script A does a write with w=3

[14:50:32] <Derick> the call won't return until all three nodes are written to

[14:50:34] <Derick> right?

[14:51:00] <bin> yeap

[14:51:09] <Derick> If script A has written two *two* nodes, a script B can read from the third node (either through slaveOk or directly) and still see outdated data

[14:51:14] <bin> well w=3 isnt it at least 2 secondaries to have the write operation replicated ?

[14:51:19] <Derick> sure

[14:51:34] <bin> ah i see your point ..

[14:51:37] <Derick> all three = primary + two secondaries

[14:51:49] <bin> i got it i got it

[14:51:53] <Derick> ok :)

[14:51:53] <bin> but what is the best solution

[14:52:00] <bin> if you want to read from secondaries.

[14:52:04] <Derick> your app needs to be able to deal with the lag

[14:52:06] <bin> and have consistency

[14:52:14] <bin> how's that ..

[14:52:21] <Nilshar> Nodex: indeed, but I don't see the point sorry..

[14:52:26] <bin> how do you measure that lag

[14:52:31] <Derick> if you want fully consistent reads, you have to read from the primary

[14:52:34] <Derick> bin: one sec

[14:52:38] <bin> ok

[14:53:52] <Derick> if you have the server started with --rest, then you can check on http://localhost:28017/replSetGetStatus?text=1 what the lag is

[14:54:54] <bin> ahhhhhhhhhhh yeah

[14:54:56] <bin> rest services ..

[14:55:03] <bin> good point

[14:55:14] <Derick> i usually enable that only on my arbiter (I have two data nodes, and an arbiter)

[14:55:25] <bin> you mean 1 p , 1 s , 1a

[14:56:23] <bin> so if i call rest service and get the lag time .. i guarantee the consistency .. or at least 99.9

[14:56:23] <bin> :D

[15:00:23] <bin> well anyways Derick thanks a lot mate :)

[15:00:29] <Derick> np :-)

[15:00:34] <Derick> yes, 2 nodes and an arbiter

[15:00:47] <Derick> in general, there is not a dedicated primary of course, as it can change without warning really

[15:01:07] <bin> well still thanks

[15:01:08] <Derick> bin: i wouldn't suggest to check up on the lag, but make sure your application doesn't die when there is a lag

[15:01:42] <bin> well it wont die but i guess it will lag with the response to the client ..

[15:02:17] <Derick> or you might show outdated data to them

[15:06:05] <bin> one more last question

[15:06:09] <bin> this lag is based on ?

[15:06:17] <bin> time for replicating

[15:06:21] <Derick> yeah

[15:06:22] <bin> or electing the new primary

[15:06:28] <Derick> replication time

[15:06:45] <Derick> there is a number of facts that can influence it - network latency, or slower drives, etc

[15:06:46] <bin> you are the man :)

[15:06:51] <Derick> or read/write contention

[15:10:39] <tomasso> I need to do some operation every time a document is inserted into a collection. To do so, i wrote a ruby process that polls the mongo collection doing a find({}).count() , the thing is that polling mongo seems not to be good.. since mongo doesnt support triggers, what would be the best way to implement smth like that?

[15:12:28] <sflint_> tomasso : are you using a data access layer over top of mongo?

[15:12:38] <sflint_> or just doing straight writes to mongo?

[15:13:03] <tomasso> sflint_: using the ruby driver

[15:13:28] <tomasso> the writes also from another ruby process

[15:13:40] <bin> Derick: thanks once again :)

[15:14:56] <sflint_> tomasso....i mean a little higher than the driver....another abstraction layer of code between your code and the database aka DAL(Data Access Layer)?

[15:15:23] <sflint_> if you were then you could implement a method that does this operation for you in that layer....kina like a trigger

[15:15:38] <sflint_> I am not a big fan of triggers in the database anyway...any database...

[15:15:58] <banzounet> When you do db.collection.remove() does mongo return the number of rows deleted?

[15:16:07] <sflint_> no

[15:16:46] <Derick> tomasso: I would suggest doing something like is described here: http://derickrethans.nl/mongodb-and-solr.html — it's in PHP but you can do the same through ruby. You wouldn't need to actively poll then.

[15:17:41] <sflint_> Derick : do you know about "mongo-connecto" ?

[15:17:49] <sflint_> "mongo-connector"

[15:18:01] <banzounet> I'm trying to delete all my contacts before a certain date I tried this : db.contacts.remove({ createdAt: { $lt: { '$date': '2013-12-03T01:00:20.000Z' } } }) but it doesn't delete anything

[15:18:06] <banzounet> Any idea?

[15:18:14] <Derick> sflint_: I know that it exists, but have not looked at it

[15:18:37] <banzounet> "createdAt" : ISODate("2013-05-26T15:38:43Z")

[15:18:37] <Derick> banzounet: delete only deletes one item per default, you need to turn on multi too

[15:18:42] <sflint_> Derick....we actively use it for a lot projects to move data around..it has solr built in as well

[15:18:52] <Derick> sflint_: making a note to look at it

[15:19:51] <banzounet> Derick: http://docs.mongodb.org/manual/reference/method/db.collection.remove/

[15:20:07] <sflint_> banzounet...for a lot of delete operations I use a .js file and verify what I am deleting first. And instead of deleting actually just move to seperate collecton (soft delete)

[15:20:14] <tomasso> i understand.. good idea.. the thing is that they both are different processes the one that writes and the one that consumes, the collection is a queue for a messaging system, that i dont want to collapse for receiving too many requests, i want the first process to iterate on the new elements, to sent the requests, so im able to control even delays if necessary

[15:20:21] <tomasso> Derick: let me check

[15:20:59] <Derick> banzounet: justOne boolean Optional. To limit the deletion to just one document, set to true. The default value is false.

[15:21:12] <Derick> that seems wrong

[15:21:40] <banzounet> sflint_: I messed up with my data, I need to erase 58k of rows and re insert the one from a backup

[15:22:18] <Derick> banzounet: better to drop your collection then

[15:22:25] <sflint_> banzounet...you can still do it in javascript....wouldn't take that long.

[15:22:32] <Derick> oh wait, you only want to delete partial things?

[15:22:55] <banzounet> Derick: Yes I need to keep the data from today

[15:22:59] <Derick> ok :-)

[15:23:05] <banzounet> but it's just around 4k rows

[15:23:18] <Derick> '$date': '2013-12-03T01:00:20.000Z'

[15:23:20] <Derick> looks wrong

[15:23:22] <Derick> I think you need:

[15:23:28] <sflint_> banzounet: you can dump with a query

[15:23:29] <Derick> ISODate('2013-12-03T01:00:20.000Z')

[15:23:31] <sflint_> then drop the collection

[15:23:39] <sflint_> and restore the backup

[15:23:40] <banzounet> sflint_: Yeah that what I was thinking about

[15:23:43] <sflint_> then restore the dump

[15:31:56] <banzounet> I still have problemes with my query

[15:31:57] <banzounet> -q "{ 'createdAt': { '$gte': { '$date': 'ISO(2013-12-03T01:00:20.000Z)' } }}"

[15:32:08] <banzounet> Wed Dec 4 16:34:00.467 0 objects

[15:33:45] <banzounet> (ISODate not working either)

[15:34:10] <cheeser> you're passing a string there not an ISODate

[15:34:55] <Nodex> quotes need to be inside the ISODate('....') not on the outside

[15:35:09] <banzounet> when I do that I got assertion: 16619 code FailedToParse: FailedToParse: Bad characters in value: offset:26

[15:36:14] <Nodex> pastebin your whole query

[15:38:08] <banzounet> http://pastebin.com/W1qPK9QU

[15:38:09] <Nodex> you do also know that you can do $gte : ObjectId() don't you?

[15:38:39] <banzounet> I don't, I'm pretty new to no sql world

[15:39:02] <banzounet> I didn't know that objecId was incremental in no sql :(

[15:39:07] <banzounet> (If it's what you meant)

[15:39:23] <Nodex> it's not, it's timestamp based but it's as if it was incremental for the purpose of this and sorting

[15:39:37] <Nodex> plus it has an index on it already ;)

[15:40:16] <Nodex> grab your object id from your find() and use that ...{createdAt:{$gte:ObjectId("......")}}

[15:41:16] <banzounet> okay it worked

[15:44:12] <Nodex> ;)

[15:45:30] <banzounet> thanks

[15:46:01] <Nodex> no problemo

[15:49:37] <banzounet> When I tried to resotre the data from today I get Wed Dec 4 16:51:19.487 ERROR: ERROR: root directory must be a dump of a single database

[15:49:40] <banzounet> Wed Dec 4 16:51:19.487 ERROR: when specifying a db name with --db

[15:50:03] <banzounet> using mongorestore

[15:52:44] <banzounet> ( I used : mongorestore -h xx -d zzz -u uu -p pp -c contacts new.bson)

[16:08:21] <banzounet> Works now, I wasn't pointing at the right file my bad :)

[16:08:25] <banzounet> Thanks again guys

[16:13:15] <whaley> rubber ducking

[16:28:58] <patweb> i'm attempting to use a tailable cursor against a capped collection, but whenever the collection gets flooded the cursor disconnects from the collection. i've looked into doing batches, but it still ends up failing. is there something that i should do to ensure that the cursor stays alive?

[16:36:15] <morfin> hello

[16:36:52] <morfin> can you suggest Mongodb for clients database where each client can have different fields number?

[16:37:41] <morfin> as example it have fields name,last name,patronymic and number, other fields like email,skype, icq and something else are optional

[16:38:21] <morfin> there can be as much fields as i need for each client

[16:40:34] <liquid-silence> hmmm

[16:40:41] <liquid-silence> how can I pull an item out of an array

[16:40:55] <liquid-silence> db.books.update({ _id: "529d8a9e000000182b000003"}, { $pull: { "books.$.permissions.group_id": "529d8002000000d00a000013"}})

[16:41:07] <liquid-silence> apparently that is not valid

[16:41:21] <liquid-silence> I just want to remove the permission where the group_id is x for the current book

[16:42:57] <Nodex> morfin ?

[16:57:01] <morfin> i am not sure how mongodb will behave on millions "rows"(documents)

[16:59:21] <morfin> anyone did use MongoDb with huge databases?

[16:59:43] <ron> 'huge'

[17:00:21] <morfin> ?

[17:00:31] <ron> morfin: http://nosql.mypopescu.com/post/14461628088/mongodb-in-numbers-foursquare-wordnik-disney

[17:00:36] <ron> just.. to give you an idea.

[17:01:44] <morfin> i do not have much resources

[17:02:18] <morfin> like lots of servers etc but queries number is not so huge

[17:02:39] <morfin> just 200-400 queries per second(usually updates)

[17:03:10] <ron> okay, but that answers your question.

[17:03:28] <morfin> and sometimes fat queries to extract 1000-500000 rows

[17:03:39] <morfin> hmmm so it can do that

[17:03:51] <morfin> i am not sure will that work for me on my hardware

[17:04:16] <ron> I'm not sure anything would.

[17:04:37] <morfin> what do you mean

[17:05:02] <ron> you can't expect to do wonders with limited resources.

[17:05:13] <Nodex> ++

[17:05:27] <morfin> actually i am not sure abot queries per second

[17:05:33] <Nodex> morfin : you do realise that Mongodb Scales horizontally right?

[17:05:48] <morfin> yes

[17:06:03] <Nodex> so you don't need masses of hardware on any one machine

[17:06:41] <morfin> i was using PostgreSQL but it's limited: i can do anything i need but that will be slow(with tonns of shitty JOINS etc)

[17:07:41] <Nodex> if your idea needs relational data then you will hit that wall with Mongodb too

[17:08:29] <morfin> problem is usually about storing different fields number for different "rows"

[17:08:51] <Nodex> but what problem, I don't understand

[17:09:37] <morfin> i need to store lots of rows with variable number of fields

[17:09:56] <Nodex> mongodb doesn't have rows, it has documents

[17:10:00] <morfin> i know

[17:10:05] <fakewaffle> The system.indexes.json exported by Mongo don't include all the fields I have specified in Mongoose. It use to export all of them, but no longer does.

[17:10:42] <Nodex> fakewaffle : is there a question coming or just that statement?

[17:11:09] <fakewaffle> Ha. /exit

[17:11:18] <ron> Ha. /mock

[17:11:35] <Nodex> lol

[17:11:40] <fakewaffle> What might cause this?

[17:11:54] <ron> stupidity would be my guess.

[17:11:55] <fakewaffle> Invalid data? or maybe our ODM?

[17:12:01] <Nodex> Mongoose would be the guess

[17:12:03] <ron> oh, you mean your statement.

[17:12:39] <Nodex> check the indexes on each collection with db.foo.getIndexes();

[17:13:21] <fakewaffle> I've done that, and they seem incomplete. We think that Mongoose is deferring the creating of them until they are needed?

[17:13:37] <Nodex> hmm that sounds weird. Are they set as background?

[17:13:39] <fakewaffle> Or is that part of Mongo, and the background creation and maintenance of them.

[17:13:41] <fakewaffle> Yeah

[17:14:00] <Nodex> background just means it wil create them when the system is not busy

[17:15:31] <morfin> with relational databases there is a problem: use lots of joins to gain data from different tables or use MongoDB with documents

[17:24:23] <Nodex> morfin : I still don't see your problem

[17:24:34] <Nodex> you're complaining about what exactly?

[17:24:56] <morfin> no

[17:24:58] <morfin> i am not

[17:25:11] <morfin> i am not sure should i move to mopngodb with that structure

[17:25:17] <Nodex> but you've yet to say why you have a problem

[17:25:28] <Nodex> 17:07:57] <morfin> i need to store lots of rows with variable number of fields

[17:25:49] <Nodex> that doesn't make sense as a question

[17:26:47] <morfin> right now i am using postgresql but as i said there some problems(actually i think that's problems of any RDBMS) with storing different fields numbers so is that good idea to rewrite everything to MongoDB and use it as basckend for my app

[17:27:55] <Nodex> in your orginal example you wanted to store some contact details for a person e.g skype,phone,email etc etc correct?

[17:28:01] <morfin> yes

[17:28:34] <Nodex> in mongo you woulld achieve this with a sub document ... contact : {email:...,phone:....,skype:.....}

[17:28:59] <Nodex> you can do the same thing in postgres afaik using the Json type

[17:29:01] <morfin> there can be lots of fields, some of them exists, some of them does not exist for clients also i'll need to upload from xlsx

[17:29:28] <Nodex> mongodb doens't require anything ... if Bob has skype then great else it doens't matter

[17:30:06] <Nodex> plus your upload method is part of your app not the database

[17:32:02] <du2x> Hello you guys. I have a question. I have an api that access mongodb and make stuff with it's data. Should I open a new connection for each request, or keep an opened connection forever?

[17:34:01] <morfin> basically i think i'll use Symfony 2 and Doctrine 2 with MongoDB driver and probably native app(for some critical operations)

[17:53:23] <Nodex> morfin : Personaly I would advise against bloated frameworks

[17:53:35] <Nodex> you've already mentioned you don't have system resources

[17:53:42] <Nodex> + many system resources

[17:57:56] <azathoth99> ok

[17:58:08] <azathoth99> say I want to add a member to a replicaset

[17:58:24] <azathoth99> I have to sync copy the files in /data/mongo including the local db?

[17:58:32] <azathoth99> does that include the journal?

[17:58:43] <azathoth99> or just the localblah.1 ..N

[18:14:03] <azathoth99> hello

[18:14:12] <azathoth99> I am trying to rsycn the fiels from primary and setup 2 and 3

[18:14:15] <azathoth99> primary is fine

[18:14:21] <azathoth99> but other 2 nodes are pissed and crash mongo

[18:15:33] <lazypower> When defining a model map in MongoDB, are collectins of nested bson documents represted as a list? I'm not finding this in the documentation and I know its there, i just dont see it :|

[18:20:26] <azathoth99> how do I print the size of the oplog?

[18:22:02] <cheeser> use local;

[18:22:03] <cheeser> db.oplog.rs.stats()['size']

[18:23:57] <du2x> Hello you guys. I have a question. I have an api that access mongodb and make stuff with it's data. Should I open a new connection for each request, or keep an opened connection forever?

[18:26:59] <cheeser> most drivers, iirc, maintain an internal connection pool in whatever their version of MongoClient is.

[18:38:53] <azathoth99> ok

[18:39:56] <azathoth99> http://pastebin.ca/2492178

[18:40:55] <azathoth99> size 288

[18:41:01] <azathoth99> but storage size wht 5g?

[18:41:03] <azathoth99> hm

[18:41:09] <azathoth99> my storage fodler is 300G

[18:41:12] <azathoth99> hm

[18:43:27] <azathoth99> replica02.PROD:PRIMARY> db.printReplicationInfo()

[18:43:27] <azathoth99> configured oplog size: 4851.6091796875MB

[18:43:27] <azathoth99> log length start to end: 108secs (0.03hrs)

[18:43:27] <azathoth99> oplog first event time: Tue Dec 03 2013 17:12:07 GMT-0800 (PST)

[18:43:27] <azathoth99> oplog last event time: Tue Dec 03 2013 17:13:55 GMT-0800 (PST)

[18:43:28] <azathoth99> now: Wed Dec 04 2013 10:41:25 GMT-0800 (PST)

[18:43:30] <azathoth99> 5G?

[18:43:37] <azathoth99> thats less thatn 5% of 300G

[18:45:52] <azathoth99> hm

[19:05:42] <ranman> cheeser listens to taylor swift.

[19:05:53] <cheeser> i do

[19:06:09] <cheeser> i'm gonna queue some up right now and unplug the headphones.

[19:06:17] <cheeser> this is joy that everyone should share.

[19:06:30] <ranman> hahaha, please no.

[19:06:33] <ranman> I'll do anything

[19:18:43] <cheeser> ranman: YOU BELONG WITH ME!!!!

[19:29:01] <patrickod> mongo-perf doesn't take a --port option any more even though it's mentioned in the readme?

[19:29:04] <patrickod> regression ?

[19:29:48] <ranman> patrickod: which version?

[19:31:34] <patrickod> latest master

[19:31:36] <patrickod> just cloned it

[19:57:50] <clever_> is update supposed to delete all fields i didnt supply it with?, its behaving more like save?

[19:58:52] <kali> clever_: there are two ways of using update: either provide a full document replacement (as you did) or use the $ modifiers ($set $addToSet, ...) to make changes in place

[19:59:18] <clever_> ah, *reads docs again*

[19:59:41] <clever_> ah, i want $set

[20:00:19] <clever_> thanks

[20:00:30] <kali> clever_: this should be more explicit, or come early in the update page. i reported that as an issue, and it improved a bit, but not enough to my taste, so feel free to notify the author

[20:00:54] <kali> clever_: little buttons top right

[20:02:47] <clever_> i'm just used to the sql style, where update only sets the fields you give it

[20:03:25] <kali> clever: most people who read these pages are in the same position :)

[20:03:43] <clever> ive also had some minor problems with the array operators

[20:03:50] <clever> let me find an example

[20:03:53] <joannac> kali: feel free to suggest changes

[20:04:29] <clever> var query = {$or:[{owner:this.userid},{members:{$all:[this.userid]}}]};

[20:04:43] <clever> kali: it seems to only work if i pass in an array of elements, even if i'm only looking for a single element

[20:05:21] <joannac> clever: huh?

[20:05:34] <joannac> members: this.userid doesn't work?

[20:05:42] <clever> members is an array

[20:05:52] <joannac> and?

[20:05:54] <kali> joannac: i think the first paragraph should mention the two kinds of updates. probaly as the second sentence, before starting the "multi" discussion. you want me to open a new issue ?

[20:05:58] <clever> i would not expect an array to ever match a single object

[20:06:17] <clever> or is mongo doing the unexepcted?

[20:06:35] <kali> clever: try it :)

[20:06:48] <joannac> clever: or your expectations are wrong :p

[20:08:35] <ranman> clever: the best way I've found to think about arrays of embedded objects in mongodb are to imagine them being unrolled for the purposes of querying

[20:09:03] <clever> that is strange, it does work

[20:09:10] <OneAngryDBA> !temp

[20:09:22] <clever> i'll have to keep that in mind for any future array work i do

[20:09:37] <joannac> kali: "Modifies an existing document or documents in a collection, by either updating specific fields in the existing document(s), or replacing the document(s) entirely."

[20:09:46] <joannac> kali: something like that?

[20:09:50] <ranman> s/are/is

[20:10:08] <joannac> ohai there ranman

[20:10:17] <ranman> heyo :D

[20:10:18] <kali> joannac: i would add "depending on the form of the "update" parameter." as it is a bit... well, unatural

[20:10:24] <kali> un-natural

[20:10:32] <joannac> kali: but we haven't gotten to that yet

[20:10:44] <joannac> kali: 'what's the update operator?'

[20:10:57] <joannac> that's the reason it's where it is to begin with

[20:11:59] <joannac> Introduce parameters, THEN discuss them

[20:12:17] <kali> yeah, i agree on the principle

[20:13:47] <kali> but the two behaviours of update are so different it is really confusing, close to an api misdesign, imho, so i would bend the principle and put this closer to the beginning

[20:14:14] <kali> but we can try your version and iterate if/when we see another beginner confused :)

[20:16:27] <joannac> kali: Modifies an existing document or documents in a collection. Can be used to either modify only specific fields, or to replace the document or documents entirely, depending on the parameters passed.

[20:16:43] <joannac> kali: you know, we should have this discussion in a docs ticket :p

[20:17:49] <kali> joannac: yeah, i'm in lazy mode

[20:18:29] <kali> joannac: but yes, i think this is definitely an improvement

[20:20:24] <joannac> kali: Gotta be unlazy to get things improved :)

[20:31:14] <joannac> kali: can i pm you?

[20:53:23] <clever> http://pastebin.com/dsZk6vWp is it valid to both $pull and $addToSet the same array in a single update?

[20:58:18] <clever> kali: ?

[21:07:22] <retran> when i give mongo console javascript, is it executed on the server side?

[21:07:34] <azathoth99> probly

[21:07:51] <retran> what kinda answer is that X(

[21:08:05] <azathoth99> one from someone whos not paid

[21:08:11] <azathoth99> lol

[21:08:16] <retran> my question is held for ransom

[21:08:17] <azathoth99> unregulated capitalism is best

[21:08:22] <azathoth99> 0 debt

[21:08:37] <azathoth99> 0 cap gain tax 0 death tax use atomic pwoer end unoinx

[21:08:46] <azathoth99> private union free trains

[21:08:48] <azathoth99> :)

[21:09:01] <retran> what the hell are you jabbering about

[21:09:13] <azathoth99> economics

[21:09:15] <azathoth99> dont worry

[21:09:22] <retran> sounds more like politics

[21:09:23] <azathoth99> subject might be kinda hard 4u

[21:09:29] <ranman> ?

[21:09:31] <ranman> kick

[21:09:34] <retran> please

[21:09:36] <azathoth99> well economics started as polittical economy

[21:09:39] <azathoth99> with adam smiths work

[21:09:50] <cheeser> Number6: mstearn ?

[21:09:51] <azathoth99> an inquiry into the nature and cuases of the wealth of nations

[21:10:00] <cheeser> joannac?

[21:10:10] <retran> sounds like you are a psuedo intellectual

[21:10:22] <rafaelhbarros> retran: do not feed the troll

[21:10:32] <retran> fine

[21:10:35] <rafaelhbarros> retran: I believe it's on the server side, yes, let me look it up

[21:10:40] <retran> heh... thanks dude

[21:10:45] <rafaelhbarros> retran: I remember reading something like that a while back

[21:11:01] <retran> cool, just interested. i figure a map-reduce is executed server side

[21:11:09] <retran> but i wonder if same for ad-hoc javascript

[21:11:24] <ranman> retran: certain javascript is executed serverside but things you run in the shell are executed in the shell, stuff you pass in for where query or things like that will run on the server

[21:11:43] <ranman> make sense?

[21:11:47] <rafaelhbarros> exactly what ranman explained

[21:11:48] <retran> so like iteratating through a cursor

[21:11:52] <retran> is client side

[21:11:58] <retran> on the mongo console

[21:11:59] <rafaelhbarros> if you query something, it's server

[21:12:07] <ranman> if the mongo shell is your client yes

[21:12:13] <rafaelhbarros> if you get the result of that query and put into a var, and manipulate that var, client side

[21:12:22] <retran> the db.whatever command is server side

[21:12:35] <retran> but iterating through the result (cursor) is client, i gues?

[21:12:38] <azathoth99> s/psuedo/young/

[21:12:41] <rafaelhbarros> rs.whatever too

[21:12:54] <retran> i see. that means it would be best to put it all into a map/reduce

[21:13:07] <rafaelhbarros> retran: perhaps, if you fetched everything at once, then yes

[21:13:23] <retran> yah

[21:13:39] <rafaelhbarros> retran: also depends on your bottleneck =) do you want to keep your mongodb server running business logic?

[21:14:19] <retran> right, that's a consideration

[21:16:55] <clever> http://pastebin.com/dsZk6vWp is it valid to both $pull and $addToSet the same array in a single update?

[21:17:08] <cheeser> valid? or does it work?

[21:17:23] <clever> it throws back an error

[21:17:28] <clever> which is in the pastebin

[21:18:25] <retran> i should be careful about declaring what is "best" i guess

[21:23:03] <joannac> clever: can you change it to member: {$pull: ..., $addToSet: ...}

[21:23:04] <joannac> ?

[21:24:26] <clever> joannac: [MongoError: fn[0] == '$']

[21:25:19] <clever> thats even more cryptic then the last error

[21:25:33] <clever> oh wait, i'm missing an s, hmm

[21:25:48] <clever> nope, same error

[21:28:07] <clever> joannac: http://pastebin.com/PKMwhb3d

[21:29:04] <joannac> Okay,then no you can't

[21:29:24] <clever> any other atomic way to add and remove items from an array?

[21:29:30] <clever> or am i stuck doing 2 update queries?

[21:32:47] <joannac> 2updates

[21:32:56] <joannac> what you are doing is inherently 2 updates

[21:33:39] <clever> ok, i'll just have to do it that way then

[21:37:04] <clever> {members:{$addToSet:this.userid}}

[21:37:25] <clever> joannac: seems this is also invalid, it replaced the entire array with an object

[21:37:34] <clever> so your idea was probly never going to work right

[21:47:46] <azathoth99> msg retran

[23:10:21] <retran> you know what i think? i think cursors are really slow

[23:33:00] <qswz> http://stackoverflow.com/questions/5719408/mongo-find-items-that-dont-have-a-certain-field actually which option is better?, 2nd answer look better to me

[23:35:36] <qswz> because I'd use a field "share" that when not existing to null, will be returned in searches

[23:40:30] <Joeskyyy1> As long as your share field will always have that schema you should be safe with the second

[23:40:47] <Joeskyyy1> cause $exists can't use indexes if you're trying to index to help speed up the query

[23:40:53] <Joeskyyy1> ^ correct me someone if I'm wrong.

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 4th of December, 2013