Log file Viewer
#mongodb logs for Friday the 21st of September, 2012
[00:00:12] <hadees> i guess if my site gets big enough where foliowers size is a problem I should switch to something other then mongo
[00:01:54] <Oddman> I mean if you're talking hundreds of thousands of followers, and you're worrying about document size and/or limits, then just separating out to a collection would work
[00:04:06] <hadees> is there a good way to monitor that? like an alert if my document sizes get too big?
[00:51:39] <planas> I have not been able to get php to work with Ubuntu 12.04 after editing the php.ini file
[01:08:19] <hdm> somewhat random performance question; given 4 spinning disks, would it be more or less efficient to raid-0 them or run 4 shards each with one disk, if the machine has godly amounts of ram and processor cores? (256G ram, 16 cores)
[01:09:07] <hdm> guessing disk overhead is higher with 4 shards, but processing and i/o would be 4 x faster with shards
[02:05:29] <niriven> hi, if i find myself making very large documents (well, small at first in mongo then they get bigger, and bigger, etc) shoul di be breaking them apart?
[02:06:16] <niriven> eg, users -> events. events might get massive. should i make a users collection and a users collection and relate to them in code isntead of users containg events (which might get large, eg. large document)
[02:10:09] <mrpro> massive doesnt sound good cause i'd think that the document would need to grow and things would need to be rearranged
[02:15:29] <niriven> mrpro yeah i have one doc per event, and one doc per user, but if i store all the users events in the user doc, it'd probably be bigger than the recommended 16mb or whatever mongo recommends.
[02:24:18] <Oddman> if events are something you query for regularly (aka, they're a top-level data structure), have them in their own collection
[02:24:43] <Oddman> one thing to note is that mongodb has a recommended max size (it doesn't have a technical limit) of 16mb per document
[02:25:29] <niriven> Oddman: yeah the events are queried very often, but with some users as an and critera
[02:26:00] <niriven> eg. get all events that have this, this, and this, but also that are associated with this set of users
[02:26:32] <Oddman> cos it might be better to have an events collection, with a users subset of data on the events collection
[02:27:05] <Oddman> imho, and again this is pretty shallow considering I don't know the extent of your requirements - but sounds like events should be it's own collection, with a user_id
[02:27:27] <niriven> Oddman: event and user have a id that can relate (i know, bad word here!). so, my plan was to have a users, and events, find the users im interested in, then find the events im interested from there
[02:27:49] <niriven> Oddman: ok cool, yeah i though that was best too, but i figured it might be against the whole mongo thing to relate in code.
[02:34:25] <niriven> details tho are, 1000 users with some information in the user, 151 million events, and 6 million which relate to a valid user. right now i have a users, and events_assigned and events_unassigned, since all the queries are targeted to events that have valid users, and unassigned events might make it into assigned collection later if i find a user. from there ill find all users that fit some profile, then look at their events.
[02:40:50] <niriven> 151 million events captured. each event might or might not relate to one user in my db (there are 1000 of them), and there are 6 million with a user id, so thats 6 million / 1000 :)
[02:44:17] <niriven> so hmm maybe its better ot have two collections, users (each with on average 600 events), and events, one that don't tie to users
[02:46:26] <niriven> but if my document in mongo gets bigger than 16mb, thats not a prob? i know i cant send a doc with 16mb, which means i cant just post save a full user if it exceeds 16mb
[02:47:23] <Oddman> I think what you'd best do here - is get all the events into an event collection
[02:48:26] <niriven> tried that, didnt work. had to at least split out events that are assigned to a user and ones that are not, since i don't care about complex queries for the majority of the events (those that are not assigned to users). so indexing events im not interested in is a waste :)
[03:25:42] <tinhead> If am using mongoid to maintain a cache. I insert like: Cache.where(:key => key).add_to_set(:route, route)
[03:27:20] <tinhead> Honestly, I would expect Cache.where(:key => key).pull_all(:route) to work, but it doesn't.
[05:20:30] <johnny_rose> when using mongoid, is there a way to only fetch the fields defined in a subclass and not its parent?
[07:45:40] <Gargoyle> Is it possible to have an index changed without creating a new one if the ensureIndex doesn't match? by specifying an index name explicitly?
[07:47:40] <Gargoyle> Then I want ensureIndex('col':1, 'another':1), but I no longer want to keep the other index.
[07:48:41] <Gargoyle> However, If possible, I don't want to have a separate script - I just want the ensureIndex() statement as part of my db access code for that collection.
[07:54:33] <Gargoyle> And if you really want a friday morning brain teaser, any thoughts on how I can make this faster? http://pastebin.com/RZjh2mh5
[07:56:59] <Gargoyle> NodeX: Kind of. Rarther than having a separate "database maintenance script", if the overhead to ensureIndex() is minimal when the index exists, I was thinking about just having ensureIndex calls inside the class that is querying that collection.
[07:58:30] <Gargoyle> So if you have a large collection, calling ensureIndex() unessicarrily would be a bad idea?
[08:01:11] <Gargoyle> Is there a fast way to check for an index? (Trying to make code that sets up the db automatically, without requiring a "install" as such)
[08:08:11] <Gargoyle> any thoughts on that query? the collection only has 300,000 items, and 100+ms seems a long time
[08:14:13] <NodeX> the docs dont say if it does or not so I assume it does as they normaly tell you if they donr
[08:22:02] <Gargoyle> Final one on indexes for now. If the results are being sorted, does the sort column need to be in the same index as the search params?
[09:56:51] <Gargoyle> Any PHP users with a free 10 mins, I would appriciate any feedback you might have on: https://github.com/gargoyle/MongoSession/blob/master/MongoSession.php
[10:10:03] <Gargoyle> ahh. that was hanging from previous version that had a separate session_id as a unique…
[10:15:18] <NodeX> it requires the same user loads a page at exactly the same time on the same browser
[11:37:32] <noordung> I have a question on Mongoose... If a document is new, I cannot call populate, but I need the populatable-field to be here. How can I do this?
[11:38:42] <noordung> If the document is new, I'd generally have an ObjectId, instead of a full document...
[12:43:12] <Seidr> Heya, I'm doing a mapreduce on a dataset which results in an object (containing data like count, deviation values and various other bits)..is it possible to filter on the values within this object when doing a find on the resulting collection? I seem to be getting no results back - would I need to run a second mapreduce run on this data? Cheers for any advice =)
[13:00:58] <Seidr> Ahh, figured it out! :) I had to use dot-notation to reach into the inner document. Huzzah!
[13:37:20] <Gizmo_x> hi i ineed example for pagination in mongodb with php for example when i go to page which is in the middle of the pages for example if i have from 1 to 1000 pages using skip()->limit() as i hear will be slow how can i implement fast paging?
[13:52:20] <noordung> Gizmo_x, you can employ 'fake' pagination, by chunking... create a document with an array of objectids which will hold a specified amount...
[13:56:35] <noordung> Well, lets assume that a chunk is a document with an array of object ids, and holds exactly 100 objectids. You can even call this chunk a 'page'. Assign an index property to it which will identify the page number. Then findOne by this index, and load all of the documents referenced by the ObjectIds in the array...
[13:59:30] <Gizmo_x> noordung: for every pageSize i will have new document in the colection and if i have 500 pages i have to create 50 documents in the array? because the page size will be 10
[14:01:02] <Gizmo_x> also there is something like here http://stackoverflow.com/questions/9703319/mongodb-ranged-pagination
[14:01:29] <Gizmo_x> when i say fast from paging ways in mongodb which is the best to deal with high amount of documents
[14:04:03] <Gizmo_x> NodeX: i tihnk that too but lets say im idiot and i want to find if app have bugs or if i can make the app to crash or to spam
[14:04:06] <NodeX> personaly if your going past page 5 then somehting is wrong with the query because the result(s) shouldv'e been found by then
[14:04:07] <noordung> NodeX, maybe implement an index property on every document, and then find on a lowerRange < index < upperRange?
[14:04:53] <noordung> NodeX, conventional logic tells me it would be faster, especially if you employ indexing on the index property...
[14:04:59] <NodeX> Gizmo_x : an expensive part of the page is the count() so firstly you need to cache that
[14:05:46] <Gizmo_x> NodeX: hmm but if i cache it i will never have the real count in real time right?
[14:07:46] <Gizmo_x> NodeX: you suggest to use only prev and next links for pagination this way i will not use the last page as count() and i can use ranges if that way is faster then skip()?
[14:08:43] <NodeX> the problem lays with where your documents lay on disk/ram.. the first few pages will more than likely live next to each other, the rest could requie massive disk seeks
[14:09:14] <NodeX> if you need pages all the way to the end then you should really throw a cache in the middle
[14:13:13] <Gizmo_x> NodeX: this noSQL is more complicated when you get in deep. a little offtopic which db is faster in this situation using offset and limit? sql or nosql like mongodb vs mysql?
[14:15:40] <Gizmo_x> NodeX: i try to get result if in the features if i need that option to list a lot of documents that's why im calculating now to optimise my db like that
[14:18:04] <NodeX> if you need to page from 1 to 10,000 you're going to have to accept that it's an expensive operation
[14:19:13] <Alir3z4> What web-framework is out-there that will work very well and integrated with mongodb?
[14:24:39] <Alir3z4> Basically i guess every web-framework out-there were created to work with SQL databases
[14:25:34] <Alir3z4> NodeX: http://en.wikipedia.org/wiki/Web_application_framework, http://en.wikipedia.org/wiki/Comparison_of_web_application_frameworks
[14:26:15] <Quantumplation> Are there any known issues with using both sort and limit on a mongo query? On my local machine, the code behaves perfectly (selects the items in descending date, so the newest items, then limits the results to 50), however once I deploy (same code, same database), it's selecting some number of items, limiting them to 50, then sorting those items by descending date.
[14:29:41] <Alir3z4> NodeX: If you look to my first question you will see i'm looking for a web-framework that works really integrated with mongoDB, Alot of these web-frameworks around will loose couple of their main functionality when going to work with MongoDB and other NoSQL data[base|store]
[14:31:27] <NodeX> most of the discussion with any sort of framework in this chan revolves around the drivers themselves
[14:32:06] <Gizmo_x> Alir3z4: http://code.google.com/p/morphia/ http://mongolink.org/ check this web sites will help you
[14:32:10] <NodeX> I have never seen anyone specifically model their app around a framework, to do so is silly
[14:33:52] <Alir3z4> NodeX: You have to do that if you want to us that framework, But designing the database model for both nosql and sql will be damn confusing
[14:34:23] <Alir3z4> Gizmo_x: Seems it's work with Play framework also, let me check it out and try it for couple of hours
[14:36:52] <Quantumplation> Sure, getting it from the deployed server is going to take me a minute though, hold on. In the mean time, here's from my local machine: http://pastebin.com/wFWTipQp
[14:38:16] <Quantumplation> er, oh. Is there a way, with the C# drivers, to output the query from the cursor object?
[14:40:15] <NodeX> Alir3z4 : dont waste your time trying to explain it to me, it's not somehting I use or would ever need
[14:43:42] <NodeX> I go on the idea that the quickest path is that of least resistence so why would anyone ever want to put frameworks, ORM's etc in the way and cause friction
[14:44:29] <Gizmo_x> NodeX: it's more easy to manage db? because there is standards and team integration in enterprise prjects
[14:44:58] <Gizmo_x> NodeX: it's a performance issue but team work is more inportant in big projects
[14:45:07] <TTimo> hello. I am using mongoengine/pymongo .. are there tools to log/time the mongodb traffic?
[14:45:11] <Alir3z4> NodeX: The same reason why folks using GUI library to create desktop application
[14:45:33] <TTimo> e.g. similar to logging ORM queries in Django when working against a traditional RDBMS
[14:50:09] <Alir3z4> Performance shouldn't be number one priority, If that so, we should all create program in 0101010010101
[14:50:55] <NodeX> I feel sorry for the users of your apps if you dont prioritise it, or the admins of your servers
[14:51:58] <Quantumplation> NodeX: I can't seem to distill out exactly what the C# drivers are querying, but if I were to type it manually it'd be http://pastebin.com/9P1JY4Rc. Without knowing exactly what query C# is running though, that's not much help.
[14:53:22] <Quantumplation> it's using the same database. (both connecting remotely to another server)
[14:53:30] <Alir3z4> But i guess because we are not from the same industry, we can't be agree on this part
[14:55:32] <Quantumplation> NodeX: I'm not sure. All I know is that the results turned when i run it on my local machine are correct, but the results returned when it's run from my deployed environment are incorrect.
[14:56:27] <ppetermann> and when you end up serving enough concurrent requests, you will realize that performance is an integral part of whats called scalability
[14:57:35] <NodeX> you can only optimise so much before you realise that your framework is the bottleneck
[14:59:24] <ppetermann> sure i can say something about django. django is a nice framework to build average content serving websites on
[15:02:01] <Alir3z4> ppetermann: I wrote complete accounting app with it, and that was prefect with django, specially with it's ORM
[15:03:05] <Alir3z4> i start the web with php and then went to rails for a year and half and then jump into django/python
[15:03:44] <Alir3z4> ppetermann: Of course i didn't, accounting itself is full relatonal data that makes it hard to work with mongo and nosql
[15:06:50] <ppetermann> because i wouldnt want anything that needs transactional safety done in mongo :)
[15:07:01] <Alir3z4> About 1 minute here, but it's been couple of week that i trying to fit it into my stack
[15:10:08] <airportyh> example: https://gist.github.com/3762060 how would I convert that to use runCommand, so that I can get the "explain" stats?
[15:15:14] <airportyh> I saw this page titled Aggregation http://www.mongodb.org/display/DOCS/Aggregation and I assumed it was the page for aggregation framework
[15:16:50] <NodeX> db.collection.aggregate({{$group:{_id:"$element_id",total:{$sum:1}}},{$match:{browser:'Chrome',os: 'MacOS', device_type: 'Desktop'}});
[15:21:22] <ppetermann> i just saw your nick and was like "oh, right, james, you havent seen that guy in ages"
[16:26:51] <diegoviola> i'm trying to have an ajax/websocket thing that will retrieve changes in a collection, every time there's a change in a collection, any ideas how i can accomplish this from the mongodb side?
[17:22:18] <EatAtJoes> I installed this sonata admin bundle distro, and the demo page has no css applied. Is that normal?
[20:14:02] <diegoviola> i'm working with some code that saves data to a mongodb database, and i want to show notifications with websockets or ajax when data is saved in a collection... but i don't think mongodb supports triggers or anything like that, what do you guys recommend?
[20:37:44] <crudson> you can add websockets support at the tier that collects and saves data to mongodb
[20:38:28] <crudson> rather than saving, then waiting for some notification from mongodb (which doesn't exist)
[20:39:11] <crudson> I wrote a blog on this a while ago for saving and broadcasting gps data with ruby/mongo/websockets/html5
[20:55:35] <crudson> found it on a machine here that isn't connected publicly at the moment - do you use ruby? I can pm you a copy of page
[21:21:31] <crudson> diegoviola: I have to pull together a few repos and blog posts to make it nice and pretty. pm your email address and I will send you a msg in next couple days.
[22:53:03] <jiffe98> is there a procedure for replacing a replica with another machine with the same hostname and IP?