Log file Viewer
#mongodb logs for Friday the 22nd of November, 2013
[00:03:36] <pzuraq> qsd: They would see the number of unread for each issue... having a separate model may work
[00:03:39] <retran> i've removed displaying exact number of (fill in blank) in many UIs, and no complaints from users
[00:06:24] <retran> business reporting is one place that exact counts seem to be important, i guess. but that's kind of a different use case than our discussion
[00:07:29] <pzuraq> true. I was mostly modeling the app off github issues since it's similar to what I want to accomplish.
[00:13:24] <pzuraq> ok last question, would it be faster to select a collection of records by there ids or by the value of a field?
[00:15:02] <retran> but if you run ensureIndex on the criteria of the value, should be similar speed
[00:31:53] <qsd> guys, it's maybe stupid but is there a way to insert with a condition? couldn't find it in doc, or I simply do the find before
[01:19:21] <silne30> When I create a mongodb document, there is a default record that doesn't contain any of the fields I have in the collection. Can I drop it?
[01:49:16] <Fieldy> hello, i'm running 2.4.6. I changed my dbpath and now I get a permissions error. I've gone as far to chmod 777 every directory down the path including the one dbpath is, and I still get the error. the directory is owned by the user and in the group for mongo that my distro (fedora 19) set up. i'd post the log line but it's an offline system.
[01:49:37] <Fieldy> i've searched and found people with the same issue but haven't found a resolution. have you run into this?
[01:50:55] <Fieldy> harttho: yes, it's owned by the user the distro set up for mongo, and in the group as well
[01:51:44] <Fieldy> the error specifically says the exact dbpath dir i gave, it doesn't have permissions to, but it does (including the directories dbpath is in)
[01:54:06] <harttho> The dbpath must exist before you start mongod. If it does not exist, create the directory and the permissions so that mongod can read and write data to this path.
[01:59:30] <Fieldy> harttho: not a symlink. it existed before I started it, all I did was a mv from the original to the new, then updated the config dbpath variable
[02:01:46] <laphlaw> this might be more of a git question, but i'm trying to clone the project, and it seems like its 'stuck' at 'remote: Compressing objects'
[02:02:10] <retran> one use case is i used a powerful more powerful system to run a ton of indexes on a set of "fixed" data
[02:02:14] <Fieldy> harttho: okay, so move it out, don't create a (now empty) dir that it can access, and try to start it?
[02:02:37] <retran> but it's epxensive to have powerful systems attached with great internet connectiivity
[02:06:20] <harttho> Fieldy: And just double checking, user permissions are good for the dir? chown mongodb /data/db
[02:11:36] <Fieldy> harttho: correct. it's owned by mongodb user, in the mongodb group, which is what it's to be using. dir perms are 770. I moved the old one out of the way, created a new one, set up the perms the same, and I still get the error.
[02:13:04] <harttho> Fieldy: Other than su mongodb and try to make a file in the dir from home dir or something
[02:20:32] <Fieldy> harttho: yeah I don't either. i found other people having the same issue with no clear path to resolution. i've been at it hours now, i have about 15 years of admin experience and this has me stumped.... thanks for your time
[02:21:23] <Fieldy> work time too... folks getting impatient. might have to move on as annoyed as that makes me :/
[02:43:08] <Fieldy> harttho: got it figured out, though i'm not entirely sure why. i removed mongo entirely using the package manager, removed the dbpath dir, re-installed, and changed dbpath before starting it the first time...
[04:04:08] <Fieldy> i've read a lot of documentation today. it's good stuff. however I also feel a little buried. how can I begin learning about how to have mongo look at gzip compressed text log files, with a predictable field order and type, for use in queries which will then output the name of the file to hand off to another app for further in-depth querying?
[04:05:56] <retran> i'm curious, how did you come to the idea that Mongo will look at your "gzip compressed text log files"
[04:12:35] <Fieldy> retran: i didn't really, then it was poorly worded. i'm not a coder, or script writer, or anything like that, is there something canned out there i can start with and try to figure out for my own purposes?
[04:20:35] <retran> so no, nothing you can do unless you're interested in application development / programming
[07:38:01] <proximity> is there a possible way to pass a collection to find. For example i have documents for cars and i pass a collection with models of cars and i want mongo to find me all cars matching models in the collection
[10:04:48] <bui> Hi ! I'm pretty new to mongodb and thinking about it for a project. I will very often need to be able to know how much % of the total documents in collection are returned by my find(). Sometime according to criterias (ie. shares a common property / value). Is there a more elegant way than having several find() with counts() to get statistics ? thanks :)
[12:01:29] <Repox> Hi. Could anyone please share some ideas how I can solve the issue with my level of interests on this question? http://stackoverflow.com/questions/19989197/dynamic-fields-and-slow-queries - Any input would be gladly appreciated.
[12:19:23] <dongodb> Hey guys, trying to figure this one out https://gist.github.com/anonymous/a03f332bedcd22fb4078
[12:29:12] <alen> <alen> i have a 3 node mongo cluster with mongos and confserver [14:16] <alen> i want to take one node offline to do HW maintenance [14:16] <alen> what is the best way to close a node?
[13:46:50] <tl> I asked on the MongoDB google groups forum but git no response. https://groups.google.com/forum/#!topic/mongodb-user/bAg-drbEnHw I would be glad if somebody could look at my command - if I did somethong wrong
[13:49:35] <ron> yeah, but do it nicely. Something like - Sorry to bump the question, but there was no replied for a few days. Any suggestions would be appreciated.
[14:04:35] <tl> My question about mapReduce streaming through Node.js was so unspecific because I actually wonder if anybody is using it at all. There are barely no questions let alone answers about it in the forum or on stack overflow.
[14:24:04] <scrdhrt> As a newbie my self, I've found that as soon as you stray from the basic stuff, it's really hard to get an answer
[14:24:39] <scrdhrt> Dunno if it's because people dont know, or if they don't want to share the answer
[14:25:19] <Nodex> [13:53:52] <scrdhrt> How do I show the last queryies by time? <------------- Doesn't make sense
[14:27:51] <scrdhrt> Yeah sure, your right about that. But how hard is it to ask for a clarification? No diss, I'm just reflecting on the subject
[14:28:39] <scrdhrt> ofc questions should be clear, but even then I get answers like "look it up your self"
[14:28:43] <Nodex> it's pretty implied that you should explain in as much detail as possible a question to get the help you need
[14:30:08] <tl> well I didn't realize that my question is unclear. writing too much can lead to tl;dr
[14:30:53] <scrdhrt> How can I look up the lastest queries made to the db, using system.profile.find(), sorting by time showing only the latest ones, like tail /some/file
[14:32:04] <Nodex> scrdhrt : I don't think that information is available, it will certainly be in the journal so you would need to query that
[14:33:22] <tl> but I still don't see what the problem with my question is: I'm posting my command and ask if somebody can spot a problem. I post the error which basically says "permission denied" and remark that the permissions AFAICT are alright.
[14:33:49] <Nodex> for me I just append a "_lu" (last updated) field to each doc so I can see what was inserted/updated
[14:34:07] <tl> what's wrong abot that question? The project is on GitHub but I doubt that soembody would want to read through it
[14:35:14] <tl> Nodex: the question is "why do I get an error?" and what could I possibly do to find the real issue (because the error seems misleading to me)
[14:36:30] <[AD]Turbo> when setting mongod to user syslog, is it possible to specify the syslog facility (in order to redirect logging to a specific file) ?
[14:42:37] <Nodex> I don't know a lot about java but I would say tthat you should make sure the owner of the file is probably the owner of the java process
[14:54:45] <tl> Nodex: Thanks! I set the scripts to the (quite unreasonable) -rwxrwxrwx@ 1 777 (everybody can do anything with tem) and now I don't get any more permission issues. I do get more errors but I'll examine them now
[15:08:46] <singh_abhinav> how good is to store payment or revenue info in mongodb .. I have one use case .. I am storing revenue which each user should get in mongodb ... sample schema is like this http://paste.ubuntu.com/6458942/ ... There will be cron job which will be running say every day and it will transfer amount specified in "currentrevenue" field via paypal and set currentrevenue to 0 and add a record in payment history ... can there be any problem if I use mo
[15:13:39] <singh_abhinav> Nodex: yeah I understand that .. but in this use case I will not need rollback .. i will update mongodb document (to decrease currentrevenue) of user x only if paypal response is that money is transferred to user x
[16:10:47] <richthegeek> hey, does anyone have a quick idea of what happens if I do a $set and then an $inc (or any two atomic modifiers) on the same field?
[17:05:45] <revoohc> Does anyone know how count works? Does it just use the index or does it actually scan the table? (i.e. db.coll.find().count() )
[17:32:21] <jmar777> revoohc: looks like it can be optimized as well if you match against an indexed field. see https://jira.mongodb.org/browse/SERVER-1752
[17:33:23] <jmar777> papajuans: don't think so. a lot of mongodb uses in #node.js, though (and you may have some luck in #mongoosejs, which uses node-mongodb as the driver)
[17:55:17] <paulfryz_> there seems to be a something off with the latest node-mongodb-native release (V1.3.20)
[17:58:14] <paulfryzel> as supposed patch release (1.3.19 -> 1.3.20) it's breaking packages that depend on it (mongojs) and the release SHA itself seems to point to an orphaned commit (maybe the 1.4 branch was force-pushed over)
[18:46:39] <harttho> Fieldy: Good to hear it worked. I've moved the dbpath after installation so it sounds like you may have found a small bug or something
[19:03:00] <jmar777> paulfryzel: sorry, had to run off earlier. did you ever get the node-mongodb-native thing sorted?
[19:07:36] <disorder> find({$query: {'expiry_date': {$gt: new Date()}},$orderby:{'create_date':-1},$maxScan: 10})
[19:11:43] <jmar777> disorder: you're ordering by create_date, but filtering on expiry_date. Do you really want to use limit instead?
[19:12:28] <jmar777> disorder: you're telling the query engine to only look at a max of 10 documents, which in theory could result in 0-10 matches before the scan terminates
[19:12:54] <paulfryzel> jmar777: nah, nothing so far. i just put in an issue on the mongojs project though: https://github.com/mafintosh/mongojs/issues/105
[19:13:29] <paulfryzel> for now, we're just shrinkwrapping (locking to V1.3.19) any packages that use these modules
[19:14:59] <jmar777> disorder: try: `.find({ expiry_date: { $gt: new Date() } }).sort({ create_date: -1 }).limit(10)
[19:17:44] <jmar777> disorder: limit() will be applied within the query engine itself, so it won't return all matching documents first
[19:18:41] <disorder> btw, for large documents this should be quite inefficient right? the sort forces the db to compute all the documents to sort them
[19:21:07] <disorder> maybe I need to split the collection and maintain only newer documents (which satisfy the filter)
[19:44:37] <Goopyo> how are ints stored in mongodb? Would 100 occupy the same space as say a timestamp: 1356998400?
[19:47:45] <Fieldy> harttho: thanks. it could be a distribution packaging issue too, my guess is it was trying to be helpful at first start, but unknowingly caused issues. though I still don't understand why, but it works now. thanks for your time
[20:31:32] <jmar777> disorder: if you don't have any indexes, then yes - that's a pretty expensive query
[20:32:02] <jmar777> disorder: since you want to filter by expiry_date and sort by create_date, you'd want the following index to make it efficient:
[20:33:26] <jmar777> disorder: the most important part there is that in the compound index, you specify the field you want to sort by *last*
[20:33:55] <jmar777> disorder: and, of course, that you indicate the correct sort order (-1 in this case)
[20:53:56] <Goopyo> is it alot of data points? I've only dealt with that kind of data at higher frequencies
[20:55:16] <x0nic_> i started with the history embedded, but my server fell over after a few stocks where added
[20:55:22] <Goopyo> well honestly that kind of data isn't ideal for monogo anyway but for one I'd trim all the keys to o,h,l,c,d,t
[20:57:37] <Goopyo> most programming languages convert them really quick so the data save will be worth it
[20:58:37] <Goopyo> something like {'t' : appl , start_dt, end_dt, timestamps : [], o : [], h : [], l :[] … }
[20:59:04] <Goopyo> and then every write do a push query to all the keys so $push : { o : 123, h: 1262, timestamps : now … }
[21:23:15] <joeroot> hey guys, sorry to be annoying, but does anyone know how to batch inserts in casbah
[21:32:30] <drjeats> Goopyo: what do you mean by timestamps? ISODates are stored as 64bit int millisecond timestamps
[21:38:24] <x0nic_> so I narrowed down my problem to an index. If i have an index on a field the sort is broken
[21:39:19] <x0nic_> am i doing something wrong? the docs say that mongo can still sort no matter the direction of the index
[21:49:32] <joannac> x0nic_: going to need more information here. you've changed your schema since the last pastie?
[22:45:06] <drjeats> Goopyo: well I can't argue with that, now im off to go read the spec/docs/impl and understand why
[22:46:05] <Goopyo> oops pinged the wrong person. Let me know when you do, Im going to make sure these are the same length
[22:51:41] <Fieldy> can I get some pointers for apps that will use mongodb where my use case is several TB of gzip compressed proxy log files? I'd like to stick a few key fields in mobgodb, including the filename it came from, so we can search for terms in fields and then know what filename to be searched. versus searching all of them as we do now.
[22:58:39] <Goopyo> one digit larger its 17. So I guess for second granularity timestamp is a more compact store
[23:26:38] <drjeats> Goopyo: isn't __sizeof__ the size of the python object in memory, not the number of bytes that get sent over the wire?
[23:30:46] <drjeats> yeah, it's also a BSON object (which does inherit from bytes/str) so there's probably other metadata stuffed in it