Log file Viewer
#mongodb logs for Thursday the 28th of August, 2014
[00:10:13] <pm2> Hi, I'm playing around with using the mongoc library to access a database from C code. I'm having trouble storing a date object in the db. The docs say dates are stored as signed 64-bit ints as the # of milliseconds since the epoch. So in C I'm basically doing "$date": time(NULL)*1000 - but that seems to end up storing "\u001" in the database. Any idea on what I'm doing wrong here?
[00:29:31] <ramfjord> is there a way to get an aggregate call to return a collection of documents instead of one giant one?
[01:26:57] <ramfjord> In the projection arguments for find, is there any way to rename a field to something else?
[01:29:07] <Boomtime> http://docs.mongodb.org/manual/reference/operator/aggregation/project/#pipe._S_project
[01:29:52] <Boomtime> find just emits the documents it finds, you can use $project as the next stage to reform the document stream if you need to before feeding to the next stage
[01:30:39] <ramfjord> I may need to use a traditional one, at least until we upgrade to 2.6 to get a cursor back from aggregate
[01:31:18] <Boomtime> i do not believe there is a $project equivalent for find, only redaction and filter
[02:28:07] <tpayne> hello. I need to create a user on a new db called dev_myDb, but it says I need to be auth to create one. I have a admin user on the admin db which I can auth, but how do i make a user on dev_c0smo
[02:31:12] <tpayne> getting this: Error: couldn't add user: User and role management commands require auth data to have schema version 3 but found 1 at src/mongo/shell/db.js:1004
[04:28:11] <keeger> i am considering using mongo for an application, but i come from a rdbms background and am wondering about whether mongo would fit my use case well
[04:28:53] <keeger> i have a lot of users who can schedule certain types of jobs to run at a future point in time, so i need to build some sort of queue system
[04:30:07] <keeger> seems simple enough. but how would my system be able to gather jobs at the right time? Ie, i poll every 5 seconds and go "give me jobs < now()"
[04:32:30] <Boomtime> it affects your data model.. if you allow users to just keep scheduling jobs with no end in sight, then your system needs to tolerate that
[04:33:16] <Boomtime> per user however, you could put jobs for a single user in an array of a single document representing that user
[04:35:24] <Boomtime> the question is whether you have a seperate users collection and a jobs collection, or if you can store these together in a combined collection
[04:36:08] <keeger> if it's standalone job doc, it would need a pointer back to the owner of the job
[04:37:24] <Boomtime> there are no joins here, so every "reference to another collection" you make is just maintenance on you
[04:37:27] <keeger> i know. i actually think the user doc containing their jobs is a nice layout, which is far easier for mongo thatn sql
[04:40:10] <keeger> can i write a query that runs across the users->jobs collection and returns jobs based on datetime? that sounds expensive in mongodb but i could be wrong
[04:43:21] <keeger> for my app, 90% of the data is contained within the user, which is why i thought mongo fit very nicely. just a matter of figuring out the queue part :)
[05:01:22] <Boomtime> if you use a single collection, with a document structure like: { _id: <id>, username: "Some guy", jobs: [ { jid: <x>, cmd: "", at: <datetime> }, { jid: <y>, cmd: "", at: <datetime> } ] }
[05:02:02] <Boomtime> then you can index "jobs.at", and run a find on "$gt > now" every however many seconds
[05:03:14] <Boomtime> you could run an aggregation even to spin out each initial match into individual job documents for processing
[05:05:01] <Boomtime> so the trouble with a basic find() is that t finds each document only once, no matter how many jobs that document might contain which need to be started
[05:05:52] <Boomtime> you can either walk the jobs array on the client, looking for which jobs caused the match to occur, or use aggregation to find them and seperate them into their own document
[05:09:35] <Boomtime> on the other hand, having a job collection means very tiny documents and requires you to maintain a "reference" to a users collection, a very sql like design
[05:11:06] <Boomtime> for each job that you found you would run an update to remove that specific job entry, using the job id as part of the find
[05:11:58] <Boomtime> that way you can even multi-thread the retrieval of jobs and know that they can detect each other removing pending jobs, and each job would only be run once
[05:13:26] <keeger> that would be great. so i can aggregate to find just the jobs, and then mark the found jobs as removed? would the jobs have a pointer back to the user in their document?
[05:13:58] <keeger> i mean, if i do a callback, when it is done, i'd like to mark the job complete in the user's document somehow
[05:14:24] <keeger> i can either update the job (status flag) or make a completedjobs collection inside it
[05:24:31] <keeger> i think i might do the jobs collection, with some references ala sql. seems like a simple setup to me
[07:21:57] <Boomtime> so? test what you are trying to do using the mongo shell, it runs from anywhere
[07:22:55] <Boomtime> it's just a test, i'm not asking you to switch to mongo shell, i'm trying to determine if the problem is pymongo or server-side
[07:23:21] <Boomtime> if you see the same problem in mongo shell, then it is at least not pymongo, maybe it is network etc
[07:24:10] <Boomtime> if the mongo shell does not have this problem then maybe the connection-string is a bit wonky
[08:37:11] <SubCreative> I just want to come in here and give my recommendations for mubsub for anyone needing to listen for changes on your collections.
[08:58:16] <SubCreative> The only thing I can't seem to figure out is how to listen to changes to documents already inserted into the collection
[09:04:41] <jordana> You probably only need to inspect how it listens to the oplog and take it from there but I'm happy to help you out if you need it
[09:31:32] <SubCreative> looking into mongo documentation it seems replicaset is for multiple instance server
[09:32:09] <jordana> You could setup another instance that doesn't house any data and see if it does create an oplog
[09:32:52] <SubCreative> http://loosexaml.wordpress.com/2012/09/03/how-to-get-a-mongodb-oplog-without-a-full-replica-set/
[10:07:08] <SubCreative> Namespace for emitting, namespace format is database + . + collection eg.(test.posts).
[10:07:47] <jordana> Yeah I saw that but I'm not convinced that the replica set name has nothing to do with it
[10:13:24] <jordana> Does it have any debug out? Sorry I don't have time to try this on my machine right now :(
[10:18:06] <jordana> It's using the debug package, and seeing as I can't see a wrapper around it inside the source I think it should definitely be outputing debug messages.
[11:59:12] <nevan> hi anybody working with ruby driver...i'm facing issue with some mongodb command being run from ruby
[13:12:30] <nickpresta> When would I want to use an Arbiter in my even replica set? Why not just use an additional mongod that holds data? Is it strictly a "arbiters don't need dedicated hardware" thing?
[13:14:14] <nickpresta> kali: thanks. How terrible would it be to use an additional data-storing mongod (on crappier hardware) with a lower priority (so it can't elect to be primary) instead of an arbiter?
[13:15:23] <cheeser> you should read the documentation on setting those priorities. there are subtle gotchas involved.
[13:16:58] <nickpresta> kali: agreed, but due to requirements I cannot run an arbiter on another machine I have in my cluster -- it has to be "dedicated" anyways... :-(
[13:19:50] <nickpresta> kali: I was thinking I may be able to use it as a read slave for analytics or something similar
[13:33:35] <kali> Const: mmm yeah... it's not rocket science, so i assume you have something that does not work to show us :)
[13:36:21] <Const> It's link to the PHP connector, As I use $user['$inc']["history.${index}.${value}.all"] = 1 to increment a value, when the value does not exists, it create it, but not as an array: http://pastebin.com/3qvfAqji
[13:36:51] <Const> Then, I'd need to perform some aggregates on it, to return most seen author per example
[13:40:28] <Const> ah but it's not variables like, user's input. it's just because I use it into a loop
[13:41:07] <kali> Const: everything in the aggregation framework, the query language and optimiser are designed with document where the key name are keywords
[13:44:30] <kali> the write ops will be one tad more complex: you'll have to use the positional operator ($) to make the update
[13:55:04] <kali> edrocks: because the shell will get confused... you'll have to use db["team.members"].find() instead of db.team_members.find()
[13:57:45] <edrocks> db.system.indexes.find() works but it might be special case for system collections
[13:58:23] <kali> I actually can never remember which one of database or collection can have a dot in its name, so i find dot in collection names confusing
[13:59:09] <edrocks> do you make multiple dbs for one project or just different collections? I've been using multiple collections in one db
[13:59:46] <Serge> Is there any way to access original document data from a double $group aggregation?
[14:00:16] <Serge> pretty much the second $group loses all data that it does not create, and $project wont let me access it
[14:01:10] <kali> edrocks: it can make sense to group collections in more than one database when the project becomes bigger
[14:01:59] <edrocks> kali: ok i was just wondering because i couldnt think of any real reason besides the namespace limit but even that you can make bigger. It would make sense though if you do have a ton of collections
[14:03:30] <kali> edrocks: it may also be argued that it is a workaround for the lock-per-database issue
[14:07:24] <Const> Kali, I managed to get this into my document: http://pastebin.com/VWnUuBfP but now, I don't know how to target it fo the înc. You know, the path with dot, like history.authors.all but to target where "authors" : "myvalue"
[14:08:35] <ssarah> what's up with this error guys? http://pastebin.com/HWA0MfiX it's probably something obvious i'm missing
[14:09:12] <edrocks> Const: I think you would use {"$inc":{"counter.all":"1"}} (format may be wrong but your probably not using counter.all)
[14:10:04] <kali> Const: mmmm... you may need two queries actually... one to add the author if it's not there, and then one to $inc
[14:11:01] <kali> Const: the second one you can write by matching the author in the first find() argument, and using the positional operator (.$.) to refer to it in the $inc
[14:13:54] <Const> right kali, but I think I can't. Here more information: it's a script who add multiple authors to the document, but to update thousands of document from the collection. So I'm using a bulk (in php MongoUpdateBatch()). So the $inc is added to the batch, in a loop where it add other $inc. How could I add a query to the $inc, then?
[14:17:17] <Const> yes kali, I found example like db.collection.update({versions.v:'some_version'},{"$inc":{"versions.$.count":1}}); But, within a batch update, where hould I put the query to match the index?
[14:19:42] <kali> Const: well, each update in the bulk comes with a criteria and a modifier, right ?
[14:20:08] <dberry> If I migrate a replica set(move config servers, promote secondaries) will the shard info in sh.status() reflect the changes or do I need to add new shards too?
[14:22:43] <Const> well, exaclty: foe each update, there is mor than one increase of authors. For exeample, I read a list of posts favorites by users. So yes, for each user, there is more than one post, so I must add author of each post.
[14:23:32] <kali> Const: i though you were just updating the 4 ou 5 counters for one author in each of them
[15:47:41] <Const> kali, what I tried did not work... but I may find something. For now, if I add my $inc using something like history.authors.me.all: 1, history.authors.you.all: 1
[20:03:26] <ssarah> hei guys i started my replicate set and i'm getting this [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
[20:12:16] <ssarah> digicyc: no wait, i thought it was weird because i had to do"export LC_ALL=C" for it to start. but now i just ssh into another shell and mongo -> rs.initiate() and it seems happy.
[20:48:51] <Chaos_Zero> is it possible to limit a user so that they can only insert but not delete or modify documents?
[21:15:58] <Chaos_Zero> how can I make a service like auditd available to start but not run on startup?
[21:56:08] <SpNg> I’m trying to use the aggregation framework to see how many records are created on a date. Currently the “created” field is using Unix time stamps. Is there a way to convert these to mongo dates?
[21:58:51] <SpNg> cheeser: what’s the best way to do aggregation against that value? can I write a function for the grouping?
[22:01:00] <SpNg> cheeser: I’m looking to take the last week of records and calculate the number of records created on each day.
[22:01:25] <SpNg> I can do it in code, but I was hoping to use mongo’s aggregation framework. maybe it’s not the right tool for this though