PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 28th of December, 2012

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:02:15] <mchammer> hi! i am new to mongodb and have a question about storing large amounts of data via a nodejs-shell-script. on the client i have a data-scructure that looks like this: i've got a 2d-array of chunks (100x100) each chunk containing another 2d-array of 64 * 16 "tiles" and in each of tile some strings and numbers. what would be the best way to save this in mongodb?
[00:03:47] <mchammer> my (noobish) try was to use mongoose, create some schemes and save it in this exact format into mongo but it runs out of memory and / or exceeds the maximum bson-size
[03:47:17] <miskander> Anyone have any suggestions or documentation as to how to handle data migrations for an RoR app that uses MongoDB. I am dropping a collection and moving the data into another collection.
[05:30:19] <mattbillenstein> hi all
[05:31:17] <mattbillenstein> anyone know why querying by just a bare objectid string in python?
[05:47:25] <_aegis_> mattbillenstein: import bson; db.coll.find({'_id': bson.ObjectId('503efbfb123b4fb1b3')})
[05:49:25] <mattbillenstein> of course - I meant to ask why the bare string form stopped working...
[05:49:34] <mattbillenstein> this seems recent to me
[06:07:03] <_aegis_> I never got it working on 2.x
[06:07:13] <_aegis_> don't remember which versions I tried it on
[06:07:28] <_aegis_> it's required in the javascript console too
[06:08:56] <synchrone> Hi all! I was wondering, what happens if in my .NET application I try to use MongoDBFileStream after IDisposable RequestStartResult object was disposed ?
[06:18:32] <mattbillenstein> _aegis_: i see -- I used to do this all the time, so it has a subtle way of breaking things if this actually changed
[09:15:55] <yarco> always get "db object already connecting, open cannot be called multi"…when using nodejs with mongodb
[10:05:53] <salentinux> Guys, is it possible using the php driver to execute custom code that returns array of data? I'm able to get a scalar value but not an array of data. For example not able to access to result of this code execution: http://pastebin.com/r52jv7Gh
[11:52:42] <arussel> is there a way to add multiple document in a single query ?
[11:53:21] <arussel> I get as json an array of doc, I would like to insert it in a collection without having to deserialize it
[11:54:37] <kali> arussel: you'll have to deserialize it anyway, mongodb does not deal with json but bson
[11:55:14] <arussel> is it just about "a":"b" -> a:"b" ?
[11:56:26] <kali> nope, bson, is a very binary encoding
[11:58:35] <arussel> I can't really deserialize, I don't know what the doc are, but I could request that the sender send valid bson. Is Bson valid Json ?
[11:58:58] <kali> no, bson is very different
[11:59:12] <kali> it does not look like json at all at the byte level
[12:01:03] <arussel> I hope lift-mongodb has something that can translate from json to bson
[12:02:25] <arussel> kali: thanks for your help
[12:03:05] <kali> arussel: you usualy don't explicitely encode in bson, you push MongoDBObject into the driver, and the driver encodes to bson
[12:03:22] <kali> arussel: http://stackoverflow.com/questions/7708499/convert-lift-json-jvalue-into-mongodbobject-using-casbah
[12:06:13] <arussel> it is what I usually do. But I also have a use case of receiving collections as json and having to create the collection and inserting the docs.
[12:06:40] <arussel> Then I should be able to run queries on this collection, but I have no idea what the docs are.
[12:08:14] <kali> what's the problem with not knowing what the docs are ?
[12:10:07] <arussel> knowing what the doc is, I can create a MongoRecord and read json, write to bson
[12:11:54] <arussel> but if I can translate from Map[String, Any] to Bson, that should do
[12:12:03] <kali> ho sure
[12:12:17] <kali> you want a deserialization, not odm mapping
[12:38:27] <igotux> is it possible to log all the requests processed by mongo server to some file ?
[12:39:46] <kali> igotux: this is not exactly what you're asking for, but it may help http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/
[12:40:40] <kali> igotux: mongosniff may also help
[12:41:42] <igotux> chekcing...
[12:44:17] <igotux> ty kali ..
[12:53:06] <Dededede4> hi / salut
[13:06:26] <Dededede4> Il n'y a personne qui parle, du coup je ne sais pas si c'est un salon anglais ou français. :-°
[13:06:43] <kali> english
[13:08:31] <Dededede4> ok
[13:09:30] <Dededede4> Is it possible with mongodb prevent inserting duplicates?
[13:10:07] <kali> Dededede4: http://docs.mongodb.org/manual/core/indexes/#index-type-unique
[13:10:27] <FrankJLhota> Is this an appropriate platform for discussing the MongoDB C driver?
[13:10:58] <kali> FrankJLhota: you can try, but very few people actually have experience with it
[13:12:19] <FrankJLhota> I may end up going the unofficial C driver route.
[13:12:34] <algernon> what's the question? :)
[13:13:48] <FrankJLhota> I cannot use mongo C driver 0.6 because of a leak, so I tried to upgrade to 0.7, only to discover that it did not support adding custom metadata to grid files.
[13:14:47] <algernon> you can get around that by manually updating the document in the fs.files collection, I suppose. though, that's not too optimal.
[13:15:26] <FrankJLhota> We actually use that feature, so we are seriously contemplating the creation of a 0.6 / 0.7 hybrid to bring that feature back.
[13:16:15] <algernon> fwiw, my unofficial lib supports adding custom meta-data to gridfs files O:)
[13:16:48] <FrankJLhota> Was that a mistake on the part of the mongo C driver team, or did they want to get rid of custom metadata for some reason? Who should I ask?
[13:18:04] <algernon> I'd suspect it was a mistake.
[13:20:02] <FrankJLhota> I think we may go the route of re-inserting the 0.6 implementation of this feature into 0.7. I would be happy to upload the resulting code so that this feature can be restored in the next revision.
[13:22:53] <FrankJLhota> The parameter for custom metadata were removed from the functions gridfile_writer_done() and gridfs_insert_file().
[13:26:27] <algernon> are you sure that was part of any C driver?
[13:26:57] <algernon> I mean... I went through the history of src/gridfs.h, and gridfile_writer_done() only ever had one single parameter
[13:27:14] <FrankJLhota> It was part of 0.6.
[13:27:21] <Dededede4> kali : There is a need to use ensureIndex only once a collection?
[13:29:15] <algernon> FrankJLhota: the v0.6 tag has MONGO_EXPORT int gridfile_writer_done( gridfile *gfile ); in src/gridfs.h
[13:35:28] <FrankJLhota> This is odd; it was part of the 0.6 mongo C driver that we downloaded months ago, leaving me to wonder how could custom metadata ever be added via the C driver.
[13:37:59] <FrankJLhota> Is the C driver the only one where adding custom metadata is an issue?
[13:46:27] <kali> Dededede4: yes
[13:55:17] <Dededede4> cool ! thank you ! :)
[13:55:23] <benpro> I'm trying to build an index in sub-documents but it fails (http://paste.evolix.org/paste/94/), any idea ?
[13:56:28] <benpro> Assertion: 16393:reading DiskLoc for external sort failederrno:0 Success <<< what does that mean ?
[13:58:46] <benpro> memory issue ?
[13:59:07] <kali> memory, file descriptor numbers ?
[13:59:12] <kali> permissions ?
[13:59:33] <benpro> oh fd limit may be at 1024... checking, thx
[14:16:45] <benpro> fd limit set to unlimited but same
[14:18:19] <benpro> kali: which permissions should I check ?
[14:18:49] <benpro> and for memory, how to see if this is the issue ?
[14:19:45] <benpro> (no oomkill)
[14:31:35] <kali> benpro: i'm not sure... this looks like a low level issue... so i was probing for some obvious issues
[14:41:13] <arussel> is 50ddae6d9628c8bc8ece16e9stats a valid collection name ?
[14:42:01] <arussel> it autocompletes on the shell but: db.50ddae6d9628c8bc8ece16e9stats.count(); throws:
[14:42:11] <arussel> SyntaxError: missing ; before statement (shell):1
[14:44:02] <algernon> try db['50ddae6d9628c8bc8ece16e9stats'].count()
[14:44:14] <algernon> it is a valid name, but js is picky.
[14:44:28] <arussel> db['50ddae6d9628c8bc8ece16e9stats'].count() is working
[14:44:48] <arussel> is it the number at the start ?
[14:44:57] <algernon> I suppose yes.
[14:45:06] <arussel> ok, thanks
[14:45:51] <benpro> kali: gandi network fail :x
[14:46:45] <juajuaa> hi, when im reading with my php driver from a remote mongodb, does the remote mongodb on its localhost has a write lock?
[14:55:18] <kali> benpro: just hosting my screen and irssi, there :)
[14:56:42] <benpro> kali: It was funny 'cause I've lost contact with my test server... at gandi.
[14:58:03] <retro|cz> strnadj, :-*
[16:41:32] <wereHamster> arussel: you can avoid that problem if you name your collections 'stat<id>' instead of '<id>stat'
[18:34:05] <hever> Hello, how can I simply count all values for a given "key" or better get the keys for the three most occouring keys ?
[18:37:36] <wereHamster> db.foo.count(condition)
[18:41:54] <hever> wereHamster, ok but I don't really have a condition. I just want to count the data using one key or grouping by one key
[18:43:14] <hever> ah it seems to be aggregate
[18:43:19] <wereHamster> keras: what is one key?
[18:43:49] <hever> just one identifier present in all documents
[18:44:01] <wereHamster> db.foo.count({key: value})
[18:45:47] <hever> wereHamster, yes but there may be multiple identifiers and I want those, which are most occouring
[18:46:06] <hever> "AAA", "BBB", "CCC", "AAA" should give me "AAA"
[18:46:30] <hever> I mean assuming those "data"
[18:47:39] <wereHamster> so you have documents {key:"AAA"},{key:"BBB"},{key:"AAA"},.. ?
[18:48:01] <hever> yes
[18:48:25] <hever> but I'm not going to select them based on the key, I just want the three most occouring
[18:48:28] <wereHamster> so you need map/reduce or the new aggregation framework
[18:51:38] <hever> wereHamster, or can I do it in two steps and just count the occourances in the first step?
[18:52:29] <wereHamster> which first step?
[18:53:35] <hever> I mean counting it in a first step and then retrieving the data in a second one
[18:53:55] <rekibnikufesin> using the aggregation framework- you should be able to get it like this:
[18:53:56] <rekibnikufesin> db.foo.aggregate( [
[18:53:56] <rekibnikufesin> { $group: { _id: "$key",
[18:53:56] <rekibnikufesin> count: { $sum: 1 } } }
[18:53:56] <rekibnikufesin> ] )
[18:55:40] <hever> rekibnikufesin, ok thank you, I'll try that
[18:55:54] <rekibnikufesin> toss a $sort on the end:
[18:55:55] <rekibnikufesin> db.foo.aggregate( [
[18:55:55] <rekibnikufesin> { $group: { _id: "$key", count: { $sum: 1 } } },
[18:55:55] <rekibnikufesin> { $sort: { count: -1 }}
[18:55:55] <rekibnikufesin> ] )
[18:56:04] <rekibnikufesin> and that should sort it for you as well
[19:27:32] <pringlescan> Hello all, I'm trying to decide between SQL and MongoDB for a project. It's a testing platform for schools that tracks metrics for different skills (i.e. multiplication, long division, etc). There are a ton of reporting requirements SQL will be well-suited for… but the kind of rapid changes we'll want to make and the table layout (i.e. test -> questions -> answers -> grades) would quickly create tons of rows. Any input?
[19:29:05] <wereHamster> pringlescan: you'll either have tons of rows or tons of documents.
[19:29:31] <wereHamster> ask yourself how you will query the data, and then pick sql or a document oriented database based on which one is more suitable
[19:30:00] <pringlescan> I don't see scalability being an issue. I just don't want things to be too rigid. I can't anticipate the way we'll query it yet, it needs to be easy to create new reporting types.
[19:30:55] <wereHamster> does the data schema change a lot or only the reporting types?
[19:32:06] <jollyBoy> Would anyone please be able to help me?
[19:32:31] <wereHamster> have you asked your question?
[19:33:25] <jollyBoy> I'm struggling to use the mongorestore to upload my collection on the local machine to mongoLabs
[19:34:13] <jollyBoy> I tried this but no luck: mongorestore -h ds047057.mongolab.com:47057 -d my_DB -c my_Coll -u xxxx -p xxxx /dump/my_database/my_collection
[19:37:04] <pringlescan> wereHamster, initially the schema, but once that settles down, reporting types will be the more frequent change
[19:40:05] <arussel> is this: "foo": /^K/ valid json ?
[19:46:27] <zastern_> Is it possible to handle replica set elections in a way that is transparent to my application (excluding the brief downtime)? E.g. not having to have the application try to hit mongo1, then if it fails, hit mongo2, etc. Is a mongos what I need?
[19:47:33] <wereHamster> zastern_: mongos is for sharding. And the drivers should automatically reconnect when a new master is elected
[19:47:49] <wereHamster> the driver may throw an exception or return an error if an election is underway
[19:47:55] <zastern_> wereHamster: but . . . reconnect to what, thats the thing. how does the driver know about my multiple mongo servers
[19:48:04] <wereHamster> your application should handle that exception/error
[19:48:13] <wereHamster> it does.
[19:48:15] <zastern_> wereHamster: so the answer is that no, as a sysadmin, there's no way for me to handle that
[19:48:22] <zastern_> my devs need to handle it as an exception
[19:48:27] <wereHamster> yes
[19:48:35] <zastern_> ok, thanks
[19:48:48] <wereHamster> you as sysadmin only have to care about the mongod running
[19:48:59] <wereHamster> everything else you can blame on the s/w devs
[19:49:03] <zastern_> lol no
[19:49:07] <zastern_> i was hoping there was a way for me to handle it
[19:49:13] <zastern_> by putting something like a mongos in front of hte replcia set
[19:49:19] <zastern_> to route requests to servers that are up
[19:49:26] <zastern_> maybe i could use haproxy.
[19:49:35] <wereHamster> your devs have to handle errors/exceptions anyway
[19:51:00] <jollyBoy> Why does "mongorestore -h ds047057.mongolab.com:47057 -d my_DB -c my_Coll -u xxxx -p xxxx /dump/my_database/my_collection" give an error and not let me import the collection in my dump folder?
[19:51:39] <wereHamster> jollyBoy: we can only guess. And gussing without seeing the error is difficult.
[19:53:16] <jollyBoy> This is the error: boost::filesystem::file_size: No such file or directory: "/dump/my_database/my_collection.bson"
[19:53:47] <wereHamster> mongorestore presumably wants the bson file
[19:53:51] <wereHamster> you're giving it a directory
[19:55:56] <jollyBoy> The bson file is in the dump directory but it says no such file or directory
[19:56:18] <wereHamster> so why don't you give mongorestore the full path to that file?
[19:56:38] <manflo> Looking for info on mongodb + glusterfs...
[19:57:07] <manflo> does it make sense to compare glusterfs vs GridFS?
[20:01:50] <jollyBoy> Still doesn't recognise the directory....I'm using the command for import from MongoLabs to do this: "mongorestore -h ds047057.mongolab.com:47057 -d blah -c my_Coll -u <user> -p <password> my_Coll.bson"
[20:02:37] <wereHamster> jollyBoy: and the error is.. ?
[20:04:44] <zastern_> wereHamster: ok, so what if I was sharding accross two replica sets, how would you handle exceptions then? since it's behind a mongos the devs dont even really know about it right
[20:05:32] <wereHamster> it depends on the driver how it handles that situation
[20:06:14] <zastern_> wereHamster: php
[20:06:44] <wereHamster> well, go read in the driver documentation
[20:07:12] <zastern_> lol
[20:07:17] <arussel> anyone know of a java lib that can parse a mongo query string into a DBObject ?
[20:08:17] <jollyBoy> So based on the import command given by monogLab: I did "mongorestore -h ds047057.mongolab.com:47057 -d my_DB -c my_Coll -u xxxx -p xxxx /dump/my_database/my_collection". But I get the No such file or directory error
[20:08:29] <jollyBoy> Even if I include the full pathname
[20:33:13] <blaizco> Hello World :D
[20:42:34] <blaizco> In what environments you use mongodb? is suitable for a website?