Log file Viewer
#mongodb logs for Tuesday the 24th of February, 2015
[00:36:19] <joannac> dacuca: sorry, i should clarify. for the purposes of upgrading, supported. Long-term, not supported
[00:59:13] <bros> joannac, How do I define an index that should be unique only within a subdocument?
[01:09:44] <Boomtime> bros: can you elaborate on what you mean? unique indexes only care about uniqueness between documents
[01:24:33] <joannac> I also wouldn't be running 2.2.0 long term in any case, it's now 3(?) years old
[02:25:20] <kburkitt> NBC decided not to proceed with March Madness.... but wants to do the Religion targeting thing. GOD IS NOT ON OUR SIDE.
[03:55:27] <supershabam> I started an experiment to tail the oplog and write stat summaries automatically over 5m windows. Has anybody else worked with storing metric datapoints in mongodb?
[04:48:18] <quadmasta> I've got a MEAN app that runs on distributed hardware. The nodes try to connect to the master and failing that they connect to their own instance. I need to, on occasion, grab the log collection from the nodes and push them into the master database. I don't know how to describe that simply enough to become a search query
[07:02:02] <morenoh149> how can I tell from explain output whether the whole collection was scanned or not?
[07:06:24] <morenoh149> I'm just going to assume that because it says b-tree cursor it didn't do a full scan
[07:08:03] <therrell> Hi everyone, I have a mongo cluster and I write to it. I then immediately try and read that document I just wrote and mongo can’t find it. Is this a common issue with default write concern behavior on mongo clusters?
[07:08:22] <therrell> Here’s a gist of the read & write code using a java client: https://gist.github.com/anonymous/3768f4f276e0a5e916a0
[07:08:59] <therrell> Its worth noting this is an intermittent issue where the read operations return null
[07:09:32] <therrell> so I’m thinking my write concern isn’t set correctly for my use case of write than immediately read
[07:28:19] <bo_ptz> [Please help ] mongo>>> db.objects.find({siteId: "boris"}) return infinite character '\'
[07:34:52] <bo_ptz> { "_id" : ObjectId("54d8a60b5ec1936857071553"), "id" : "1c344b19f305aa61", "siteId" : "www.svyaznoy.ru", "userId" : "boris", "country" : "Корея, республика", "price" : "12990", "name" : "Видеодомофон Commax CDV-71AM (серебристый)"}
[07:36:46] <joannac> Pharmboy: ask your question instead of meta-asking? then at least you could maybe get an answer
[07:41:11] <Pharmboy> favitem", "owner": "derek" }, { "$addToSet": { "fixtureids": { "$each": ["BATH POD END CAP SHELF", "FAUCETS 2-FUNCTION SHOWERHEAD SIGN"] } } }, function ... )
[07:45:21] <Pharmboy> sorry for posting javascript question in mongo forum, you are absolutely right it is not mongo
[07:51:26] <therrell> In case anyone paid attention to my question I figured it out. I’m using 2.2 java client which as a default write concern of normal (now known as unacknowledged) that’s fire and forget and provides no guarantee that the write operation completed before mongo responds successfully.
[07:52:37] <therrell> So my issue can be fixed by upgrading to the 2.10 client which has the default write concern of acknowledged or I can configure my 2.2 client to use the non-default write concern of my choosing
[07:54:08] <Boomtime> bo_ptz: your problem is most likely that there is invalid UTF-8 in a document field and the shell is failing to render it
[07:54:55] <Boomtime> bo_ptz: please understand the server does not care, it can handle whatever crazy string you push at it even if it's invalid, the only rule it has is that the first zero-byte marks the end of the string
[07:57:45] <Boomtime> anyway, if you want to find the document that is causing this, you can grab each one of the docs in that list one at a time by _id and see if the shell can print it
[08:04:33] <flusive> Hi, I have problem with disk space on one shard (is 98% used), I have also another shards (shard1: 50% disk space free, shard2: 20% disk space free). What can I do to increase free space on shard3 or how can I force mongo to don't used more space on shard3?
[08:06:47] <Boomtime> do you understand that the problem is just displaying the result in the shell?
[08:07:39] <Boomtime> you may find one document, or you may find several are a problem, you will need to play around a bit now to find out why the problem documents can't be printed in the shell
[08:08:13] <Boomtime> it is very common, because there are just tonnes of libraries that do not handle UTF-8 correctly
[08:09:04] <bo_ptz> Boomtime, it's not only display problem when I do it in nodejs with native driver get data from mongo , it return object with \\\\\
[08:10:54] <Boomtime> that's pretty much proof that the data is not valid UTF-8, that node is trying to interpret it as such and the translation comes out as garbage
[08:11:52] <Pharmboy> are mongoose questions acceptable here if I get crickets in the mongoose channel?
[08:12:59] <teoo> hello.. could anyone please check this SO question, and maybe answer, if you have experience? http://stackoverflow.com/questions/28684462/what-is-faster-to-search-in-hashmap-object-or-array-in-mongodb
[08:18:20] <Pharmboy> I asked it awhile ago, it is in regards to $addtoset $each ... someone confirmed by syntax was correct, but when I pass it in node.mongoose, it seems there is a bracketing issue
[08:46:32] <flok420> hi i have a problem converting a mongo query into the PyMongo equivalent. on the forum nobody knows the answer but maybe someone in this channel knows it? the posting is at https://groups.google.com/forum/#!topic/mongodb-user/32m6T_et3MY
[08:58:37] <joannac> flok420: in one you have { $gt: ["$ts" , 0] }, in the other { "$gt" : [ "$ts" , max_age ] }
[08:59:12] <joannac> flok420: in any case, do the aggregation step by step. leave just the match part. do you get results?
[09:17:39] <flok420> it looks like that it works with the match and the unwind. if i add the $redact statement then nothing is returned. I verified that max_age is 0. also with an explicit 0 (instead of that variable) I get nothing. if i replace "$ts" by "ts" then all data is returned as if the whole $gt is not executed
[09:25:08] <teoo> could anyone please check this SO question, and maybe answer, if you have experience? http://stackoverflow.com/questions/28684462/what-is-faster-to-search-in-hashmap-object-or-array-in-mongodb
[09:40:05] <flok420> joannac: if I remove the redact, then I get all data in the collection. so something is wrong with that
[10:03:09] <flusive> Please help me to solved problem with free space on shard. What happen when my shard will be full? Whole sharding will crash? What can I do to increase free space on shard3 or how can I force mongo to don't used more space on shard3? I need some solution to store around 25GB/week
[10:35:03] <styles> I'm trying to organize game cards (magic etc). I want a Card object to belong to a Set. I need to search on different factors. My idea was to add the Set to each card object.
[10:35:34] <styles> Or keeping cards seperate from sets and jsut linking the ID and requried info in each card into the card object itself
[10:44:29] <joannac> if you want to do things like "find all cards in this set", just embed the setname/setID in the document
[12:10:26] <esko> i have a collection with title: and anchor: i would need to insert a incrementing id: to every (title: anchor:) in the collection, could someone help?
[12:40:37] <andrewjones> Hi can anyone help with Mongo MMS , using it on a standalone server no internet connection as the guides imply you can do this but we cant find out what the default login credentials would be on a standalone instance of mms
[12:42:46] <StephenLynx> I'm not familiar with MMS, but isn't it the same as a default install, with no auth?
[12:43:16] <andrewjones> no we have it setup but it asks for user login details and not sure how you set this up
[12:43:52] <andrewjones> Tried to contact mongo direct as purchasing support but cant get hold of them
[13:27:20] <jayjo> What's the best way to write json data to mongodb? Would this be a bulk write operation from my language of choice and associated driver, or can I do this through mongo shell?
[13:28:18] <hayer> I have a document which looks like this http://pastebin.com/hevptGEf .. how can I select all the elements in data which has a date between X and Y?
[13:35:23] <joannac> hayer: if you are ever in a situation where you think "I want a subset of this array" your array should be split into top-level documents
[13:38:56] <hayer> joannac: okey, so I should have a document called "data", where each item has a "item_id" that points to the ID of the document containing things like name and somevalue?
[13:40:08] <hayer> StephenLynx: but how will this be if the data array is massive? Like million of rows.
[13:40:33] <StephenLynx> even better to have as a separate document, because BSON has a limit on the size of documents.
[13:41:10] <hayer> a separate document for each data entry? Aka each entry in the data-array should be one document?
[13:41:13] <StephenLynx> and since you can index, performance will not be much of an issue. just make sure you don't make somethat that will need multiple queries.
[13:42:21] <hayer> Wouldn't that mean evetually I will have millions of documents? I've been told that document = table, but with no structure set in stone.
[13:43:31] <StephenLynx> the posts collection used to be a sub array in the threads collection. but because I regularly had to query for just some posts in that thread and the 16mb limit, I chose to put them on a separate collection.
[13:44:19] <hayer> okey, at I've created one collection for each vechile. Each vechile has data. Each element of data should be a separate document inside the vechiles collection..?
[13:46:52] <StephenLynx> now you either use _id or create your own unique field for each vehicle and use it to track to which vehicle each data document belongs to.
[13:47:27] <StephenLynx> you do have field references, but in the driver they just perform multiple queries behind the scenes.
[13:48:02] <hayer> So I can then ask the vehicle_data collection to return all documents where vehicle_id is X and date is between Y and Z! This makes so much more sense.
[13:48:32] <jayjo> Is there a reason when I imported .5gb file with mongoinsert my db is now 1gb? is that standard?
[13:49:48] <hayer> thanks StephenLynx - that cleared up alot. I'll probably be back later this week with more dumb questions :)
[13:49:49] <jayjo> cheeser: is 2x the normal rate though? I'm about to do the same process with 4gb file, should I expect 8gb db after?
[13:51:58] <cheeser> most databases preallocate "extents" so that information is stored close together.
[14:04:26] <jayjo> just fyi - I just wrote the next file and it is almost exactly 2x the import file size
[14:04:53] <jayjo> so now I'm at 7.95 GB, I wonder if that does have to do with space being allocated
[14:41:02] <flusive> Please help me to solved problem with free space on shard. What happen when my shard will be full? Whole sharding will crash? What can I do to increase free space on shard3 or how can I force mongo to don't used more space on shard3? I need some solution to store around 25GB/week
[14:48:48] <bros> I have a key called "store_id" in a lot of tables. I'm finding myself having to do an extra query every time I call store_id, to verify that the store_id belongs to account_id. Would it be smarter to add account_id as well to avoid the extra query?
[14:49:39] <bros> I want to make sure the user isn't trying to "spoof" a store ID and gain access to somebody else's data.
[14:50:00] <bros> Every time I do a query with a store, I precede it with a query for that users account and stores
[14:50:57] <bros> I considered doing that but couldn't come up with a reliable way for logged-in clients to get notification of the refresh while already logged in.
[14:54:38] <bros> db.accounts = collection. db.accounts.users for users, db.accounts.stores for stores
[14:58:17] <bros> An account can have 5 stores, and 10 users. The 5 stores belong to all 10 users of the account.
[15:00:13] <StephenLynx> so you can just cache the user validation data, including the account it belongs to. so you can easily check against the account the store belongs to when you retrieve it.
[15:04:00] <bros> What's the proper way to verify something you don't want duplicated exists? Set a unique index, try to insert, catch error
[15:05:08] <StephenLynx> and since you are using node/io, I assume, you should be checking for the error anyway
[15:05:54] <StephenLynx> but for that you will need a separate collection, I guess. I tried to make an index for documents in a sub array and failed. but I could be wrong.
[15:08:06] <bros> db.items.ensureIndex({ 'barcodes.barcode_id': 1 }) { safe: undefined, background: true, unique: true }
[15:08:38] <GothAlice> Standard practice: attempt the change (insertion/update) and check for failure, handle failure. Atomic, simple, fast.
[15:10:07] <GothAlice> If you're dealing with sub-arrays, you can handle that as part of the update query.
[15:10:30] <GothAlice> It simply won't "match" a document that conflicts, thus nUpdated will be zero, and you can catch that.
[15:12:23] <GothAlice> (Since "inserts" to sub-arrays are actually $push operations, they're updates. In your update, query for $ne or $not matching the potential duplicate value, and your record will either push the value (nModified=1) as there is no conflict, or conflict and not update the record. An index will speed up that check, potentially. The only way to tell if it actually helps is by telling MongoDB to explain the query with and without the index for
[15:14:35] <bros> When I insert/update then check for failure, will I be returned the ID of the culprit?
[15:15:12] <StephenLynx> if you have an unique index, then yes. but if you just write a query that will match 0 documents, then no.
[15:16:02] <StephenLynx> you can only get the updated document with an operation that I forgot it's name, but it will only return IF it actually updates the document.
[15:17:12] <GothAlice> (Returns document prior to modification, amongst other goodies. The manual page has more.)
[15:17:20] <StephenLynx> not to mention you will have to just assume the query conflicted with exiting values. it could be the query parameters were wrong in the first place.
[15:24:24] <styles> I'm trying to organize game cards (magic etc). I want a Card object to belong to a Set. I need to search on different factors. My idea was to add the Set to each card object.
[15:25:05] <styles> Or keeping cards seperate from sets and jsut linking the ID and requried info in each card into the card object itself
[15:27:19] <StephenLynx> when query for cards, will you have to query for the set of said card for additional data? if you do, will you have to perform just one additional query or multiple queries depending on the number of cards?
[15:30:52] <StephenLynx> depending on what you answer, you might be better with separate collections or subarrays.
[15:33:51] <bros> StephenLynx, how would you store an item that sometimes has a subarray variants, but someitmes doesn't
[15:37:00] <NoOutlet> The best practice is to fit your data schemas to your application requirements. There isn't an inherent best way to store data without the application context.
[15:38:02] <bros> I simply want to be able to store/retrieve barcodes for items. Sometimes, items has variants. If that's the case, the barcodes belong to the variant, not the item. I'm not sure how to best represent that.
[15:38:41] <winem_> hi. I have some trouble understanding the rollback process of mongods in a replica set. is this right: the primary node writes the stuff into it's datafile and keeps the documents in the rollback file since it got an ack from the majority of nodes, e.g.? depending on the write concern of course
[15:39:15] <winem_> would just like to understand what happens in detail if the primary can't reach any of his slaves and a file system backup from the datadirectory is running...
[15:41:41] <jayjo> but is it true that no other language is 'enabled' working directly with it in the shell? all of the examples in the docs are using js functions
[15:42:31] <cheeser> jayjo: the shell is a js shell. so, no, writing c#, say, in the shell probably wont' ever happen
[15:42:53] <StephenLynx> so yeah, any other language pretty much has to mimic it. you can use json on the shell client because it just parses text.
[15:44:31] <jayjo> I like js- i'm just getting a feel for this. If i'm writing mapreduce or other aggregation tasks, this is _always_ done through the shell, right?
[15:44:51] <NoOutlet> winem_, the primary gets some writes but is (for some reason) unable to send those writes to the secondaries. One of the secondaries becomes primary and takes some new writes. When the initial primary comes back into the replica set, it will see that there are new writes which it doesn't have and that the replica set doesn't have the writes that it got before the disconnection. That's when a rollback occurs.
[15:46:28] <winem_> ok. so the initial primary node writes it's stuff to the datafile before it gets the ack from the secondaries or does the primary just write it to the rollback file and moves the docs into the datafile when it receives the ack
[15:47:28] <NoOutlet> I don't know what you mean by the ack. The acknowledgement? or like the "ACK! I'm dying!"
[15:48:22] <NoOutlet> If the primary gets acknowledgment from the secondaries, then there won't be a rollback.
[15:50:49] <winem_> if yes, it would mean that the primary writes to the datafile -> sends the documents to the secondaries and waits for their acknowledgement -> primary rolls back if it gets no acknowledgement and touchs the datafile again
[15:51:21] <NoOutlet> They may or may not have been written to a datafile in the primary. But the writes will be rolled back (taken out of the datafile) and written to the rollback folder.
[15:55:14] <jayjo> last time I'll ask about the js - this page from the pymongo documenation has the mapreduce functions put in as js. is this unique to the python driver?
[16:07:58] <NoOutlet> Okay. Let's say you have a three node replica set. One is the primary. It's in a separate data center from the other two.
[16:09:28] <NoOutlet> A network partition causes lost connections between the primary and the two secondaries. This happens immediately after a couple writes which have not been replicated to the secondaries.
[16:10:32] <NoOutlet> And finding out that the secondaries aren't reachable causes an election on both sides of the partition.
[16:11:00] <NoOutlet> On the primary side, there is only one voter, so there is no majority and the primary steps down to a secondary, no longer able to take writes.
[16:11:16] <NoOutlet> On the secondaries side of the partition, there are two voters and one of them becomes a primary.
[16:12:14] <NoOutlet> If there are no writes to the new primary before the partition ends, then there will be no need for a rollback.
[16:14:26] <NoOutlet> When the partition ends, the datasets will be compared and if it's seen that the initial primary (the one in a data center by itself) has writes that the others don't, those writes will be replicated to the others then.
[16:15:24] <NoOutlet> However, if some writes are made _during_ the network partition, then when the partition ends the writes that the initial primary had will be rolled back.
[16:16:44] <winem_> but how does the rollback work? I saw that there is a folder called rollback in the data-directory
[16:16:47] <NoOutlet> It's irrelevant to the situation about whether it wrote to the datafile or the journal.
[16:17:24] <winem_> ok, I understand. So I just have to think a bit abstract and don't have to care if it's written to the data file
[16:18:09] <NoOutlet> Essentially, the data in the initial primary will be synced to the initial secondaries (one of which became a primary and took writes).
[16:18:59] <NoOutlet> So the initial primary will not have the writes that it took immediately before the partition, but it will have the writes that the other primary took during the partition.
[16:19:31] <NoOutlet> This is so that if an administrator wants to apply those writes, they are available.
[16:20:45] <winem_> so the rollback file is written when the initial primary went back online again (as a primary e.g.) and rolls back some operations because the new primary did not receive them
[16:20:50] <NoOutlet> If you know much about revision control like git or subversion or anything like that, it's a similar problem.
[16:21:02] <winem_> and now, it's up to the admin to decide whether he would like to process the rollback file manually?
[16:21:24] <NoOutlet> Basically, when there are writes to two systems within a replica set, mongodb doesn't know how or if the writes can be merged.
[16:22:40] <winem_> great. so it was that "easy"... I could not explain when he writes the rollback file or if the documents are kept in the journal / data file or the rollback file at the same time, etc...
[16:22:50] <NoOutlet> I think it's not likely that the initial primary would become primary again unless it had some priority.
[16:23:24] <winem_> thank you very much. I will go in the background and play around on the dev environment to see if I got everything right now. this was very helpful - thanks for your time!
[16:23:53] <NoOutlet> Because when it comes online, if there is a rollback, that means that it needed to take writes from the new primary.
[16:34:07] <freeone3000> I have three servers pat of an 8-member replicaset who seem to be stuck in recovery. Their state is listed as "RECOVERING", but their optimeDate doesn't seem to increase. What could cause this?
[16:57:29] <StephenLynx> because they don't actually provide any functionality, they just add bloat to your project.
[16:58:11] <styles> StephenLynx, It's going to be multiple things. All cards based on a set. But also (here's the tricky part) a set can be "hidden" (not public). So I need to be able to query for all cards w/ no hidden set
[16:58:12] <StephenLynx> have been using mongo in two projects with nothing but the driver and having no issues at all.
[16:59:37] <domo> eh, for the amount of functionality we get out of our bongo service, the tradeoff is negligible
[17:00:17] <ra21vi> I am trying to build a complex aggregation query, but failing. The required aggregation and JSON data structure is at - https://gist.github.com/anonymous/78028b8b42ce2044f64b
[17:01:34] <ra21vi> Right now I have to hit multiple queries and then apply logic at app side to get the result. Need help in building aggregation query for same.
[17:02:45] <ra21vi> In given gist (https://gist.github.com/anonymous/78028b8b42ce2044f64b), the record file describes how data is stored in mongodb. In query, the required aggregation query is explained
[17:40:33] <GothAlice> Indeed, a 64-bit integer would take up 4 fewer bytes. What you lose if you do that is everything, however.
[17:41:41] <GothAlice> Siamaster: MongoDB has no concept of "auto increment", which means you need to jump through hoops and you encounter many more race conditions than before: http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
[17:42:51] <GothAlice> Notably, if you have multiple servers how do you synchronize ID creation between them to prevent duplication? ObjectId doesn't suffer this problem. (In fact, each client application connection can deal with ID creation in complete ignorant bliss of any other server or application creating IDs.)
[17:44:09] <Siamaster> and as far as mongo is concered, it doesn't matter if I use ObjectID or String in my java code right?
[17:44:57] <GothAlice> If you accidentally store an _id that is the hex-encoded string representation of the ObjectId, suddenly creation-order sorting goes out the window, as does range querying them.
[17:45:51] <GothAlice> (Pro tip: ObjectId includes the creation time allowing you to $gt and $lt query them to find records in date ranges. Also, unless you need to extract date-like values from a creation time, you won't need to store an additional creation time.)
[17:46:16] <GothAlice> Rather, extract date-like values from a creation time within an aggregate query. (Things like $hour, $minute, $day, etc.)
[17:57:04] <Siamaster> what if, you already have a unique String id entity then? Would you create an ObjectId anyway?
[17:57:46] <GothAlice> If you have external ID sources, then no worries using that unique value as the _id in your collection. Making them unique is Facebook's problem. ;)
[18:02:21] <jayjo> If I have a query running a text search on an 8gb database and it's ~10 minutes in, should I be mad? ;) I didn't test it on a subset first
[18:03:55] <GothAlice> I'm not entirely sure how MongoDB's FTI search operates, however it's common to boolean search first, that way only a subset of the documents are ranked. If MongoDB does do this initial boolean search for you, then either your query is matching a whole lot more than expected or something's up. If it doesn't, then you might want to do that initial filter yourself. ;)
[18:04:30] <GothAlice> jayjo: There is the potential issue that your index doesn't fit in RAM… if this is the case then you may be there until doomsday. ;^P
[18:36:01] <eaSy60> Hello. Is there a way to create a query on the last element of an array inside a document?
[18:38:17] <GothAlice> eaSy60: There are several methods to almost do what you want. $elemMatch will let you query the contents of an array, but that only works if you store in index with the records, as you demonstrate. You could store a copy of the "last" record pushed outside the array, then it becomes trivial to get back. You could also do an aggregate query, unwind the array, and pick the $last of each of the fields when re-grouping.
[18:39:31] <GothAlice> (Your $elemMatch example also has the race condition of needing to look up the size of the array first, to determine the index of the last element.)
[18:40:09] <eaSy60> GothAlice: that was an exemple, I don't want to store an index or the last record :)
[18:41:11] <GothAlice> Storing the last record outside the array, and $set'ing it when you $push new values to the array, while a minor increase in processing (one extra operation) will allow you to easily query (and even index) the fields for that "last" record.
[18:41:40] <GothAlice> If I needed to query them, that's what I would have done for the forums I wrote. (As it stands, I just slice during projection since when I need the last reply to a thread I just need the last reply, no query.)
[18:43:11] <eaSy60> I need to get all the documents in a collection where document.anArray[last].status === 'online', I don't need to $push any thing.
[18:43:34] <GothAlice> I'm referring to how you update that data, not query it, when I refer to pushing. Are new values ever added to anArray?
[18:47:49] <GothAlice> That is the simplest method to achieve what you want, a queryable "last" value from an array.
[18:48:43] <GothAlice> Yeah, with this discrete last value, you don't need a getter/setter any more.
[18:49:32] <eaSy60> Yes but I need to update it and I don't really want that, this would reduce the flexibility of the data
[18:50:44] <GothAlice> This is an example of what I often refer to in MongoDB as non-duplication of data. It isn't technically a duplication of the data because it's pretty much required to query in the manner you wish.
[18:53:21] <GothAlice> Doing it the pure aggregate way would require a rather substantial $group projection that includes every field you care about, and the results come back in a manner different than standard queries. It would be more difficult to implement that way. Doable, but seriously, the $set when you $push makes everything extremely easy. You could limit the $set to a subset of the embedded document's fields if you really wanted, too.
[18:55:33] <GothAlice> You could $set lastStatus if that's the only value you care about on that last record. :)
[18:56:02] <GothAlice> This is also referred to as a form of pre-aggregation. (You're keeping what would be the result of an aggregate query up-to-date in the record itself during normal use.)
[18:57:20] <eaSy60> Maybe I should do one request with $elemMatch, and then postprocessing the result in order to filter the documents where the last element of the array match my condition
[18:57:41] <GothAlice> … again, that's even more work. And work that would require roundtrips from the database to your application.
[18:59:50] <GothAlice> Schemas aren't sacrosanct… especially in MongoDB which is effectively schema-less. Your model should model how you need to query your data, you shouldn't be falling back on application-side bulk processing basically ever.
[19:00:50] <GothAlice> (Bulk processing of multiple records application-side eliminates the point of even having a database. Might as well use on-disk JSON files. ;) (Ohgodspleasenobodydothis…)
[19:12:08] <hayer> I store a time field as .Net DateTime for when the data was created/inserted. I use this to select all data between date X and Y. Can I use the ObjectId.getTimestamp for the same?
[19:14:42] <GothAlice> hayer: Yes. Most ObjectId implementations have a factory method "from_datetime" or similar to allow you to construct an ObjectId for a particular moment in time, and you can then $gt/$lt range query _id.
[19:15:34] <GothAlice> It's very useful, and for most cases (where you aren't using $hour/$year/etc. projection against the creation time field in aggregate queries) eliminates the need for a discrete creation time field.
[19:17:39] <hayer> but is the time from when the item was inserted or when it was actually written to disk?
[19:18:17] <GothAlice> In most cases the client driver is what constructs the ID, so it would be the moment of the .insert() call, not the time it was committed to disk. (The ID needs to already exist at that point.)
[19:18:40] <GothAlice> As an example, using MongoEngine in Python: ohist = Invoice.objects(criteria).filter(id__gt=ObjectId.from_datetime(now-timedelta(days=30)), state__ne='voi').count()
[19:19:24] <GothAlice> (Count the Invoice documents matching mostly security-related search criteria, i.e. ownership, whose ID indicates the invoice was created within the last 30 days and state isn't void.
[19:33:40] <iksik> hmm, is it possible to execute mongo query directly from cli (with eval) without mongodb shell header/banner?
[19:35:04] <Siamaster> without the ultimate edition, you wouldn't be able to use the high availability features :S
[19:38:24] <StephenLynx> Siamaster just avoid becoming a fanboy. mongo is pretty good, but is not a panacea.
[19:39:11] <Siamaster> yeah, one need to be flexible in using stuff. nothing is always the right thing
[19:41:08] <Siamaster> not only that, it makes you too depeneded on GAE and you need to always think about the pricing when designing your db
[19:42:58] <Siamaster> so if you have a many to many relationship like , User, Store, UserVisitStore
[19:43:23] <Siamaster> and you want to be able to query stuff like which stores does a user visit or which users are visiting a store
[19:43:32] <freeone3000> I'm getting "RS101 reached beginning of local oplog" when adding a new member to a replicaset after running rs.reconfig() on primary.
[19:44:05] <freeone3000> How can I add a new member to the replicaset with no data and have it just load?
[19:44:24] <Siamaster> i'm using it now, but I'm trying to design my project so that it's easy to migrate away if/when i have many users
[19:55:12] <ra21vi> Need help with following aggregation : https://gist.github.com/anonymous/78028b8b42ce2044f64b
[19:56:37] <fewknow> ra21vi: because there are faster engines out there that are easier to use. Really depends on the aggregation, but complicated ones should be done in other engines
[19:56:51] <ra21vi> fewknow: just for single query, I won't opt to handle two DBs. I just moved my DB to mongo. and trying to map queries in mongo as much as possible
[19:56:59] <freeone3000> fewknow: https://gist.github.com/freeone3000/6c705b8979b0df843e54 is the logs. Error is "RS101 reached beginning of local oplog", which is the last message from "heartbeat" in rs.status()
[19:57:31] <fewknow> ra21vi: if you abstract away the db with a "data access layer" you don't need to know what DB the data comes from
[19:58:44] <fewknow> ra21vi: really depends on what you are doing....i just don't use mongodb for aggregation(not fast enough for me)
[19:58:52] <ra21vi> fewknow: oh I see:) by maintaining all these just for a simple chat app, I will soon be begger :) LOL
[20:01:49] <freeone3000> fewknow: Yes. It's in FATAL immediately after adding, with the complaint "RS101 reached beginning of local oplog" in rs.status().
[20:05:49] <fewknow> also when you add a node to the set you can specify where it replicates from...you should probably do that.
[20:06:54] <freeone3000> fewknow: We know. We read secondary-preferred. There's actually another four secondaries coming up.
[20:07:07] <freeone3000> fewknow: We didn't want to do a sharded cluster because that ties user data to its geographic shard.
[20:07:29] <freeone3000> fewknow: If we don't use that shard key, it's not geographically distributed.
[20:08:39] <freeone3000> fewknow: It's for a company whose customers, do, indeed, move that often. We also have some very strict time requirements.
[20:09:21] <fewknow> you can use a caching layer or something to solve that issue...if you want to distribute the data and scale it you will need to shard
[20:10:23] <fewknow> there is also a point where you will not be able to shard and you will be stuck with slow responses from secondaries
[20:11:26] <fewknow> but for the issues at hand you need to set priorities on all nodes...also make sure you always have odd number of nodes.
[20:11:56] <freeone3000> fewknow: I set them as https://gist.github.com/freeone3000/50e21076527bd1808be9 .
[20:12:43] <fewknow> each one needs a priority.....and you never duplicate it...so 100,75,50,25, etc
[20:16:29] <freeone3000> Changed it to https://gist.github.com/freeone3000/50e21076527bd1808be9 with unique priorities. Have an odd number, since id 3 is a non-voting member. Still have id 7 in FATAL.
[20:19:46] <fewknow> freeone3000: just changing the priority is not going to fix the issue. Once you have the config correct you will need to remove the node and add it back. I would tail the log file when you add it back to see what is going on....tail the primary and the node you are adding
[20:31:00] <freeone3000> Any other suggestions? "RS101 reached beginning of local oplog" seems to mean that it's ahead of primary - how can I force it to understand that it's not?
[20:33:04] <fewknow> i am not sure why it is in FATAL...seems like a config issue where it doesn't know that it is a secondary
[20:40:02] <freeone3000> fewknow: Can't recover from backup - next backup window isn't available yet, and this replset had ran out of disk at last backup.
[20:40:21] <freeone3000> (Trying to restore those backups got a secondary stuck in RECOVERY with a last optime date of last friday, with no updates.)
[22:01:19] <jordanpg> #addUser correctly adds a user, however I cannot get this to work with #command
[22:01:47] <jordanpg> db.command("db.createUser( { user: \"" + DatastoreFactory.DEFAULT_MONGO_DB_USERNAME + "\", pwd: \"" + DatastoreFactory.DEFAULT_MONGO_DB_PASSWORD + "\", roles: [] } )");
[22:02:30] <jordanpg> results in the following response: { "serverUsed" : "localhost:27017" , "ok" : 0.0 , "errmsg" : "no such cmd: db.createUser( { user: \"my_username\", pwd: \"my_password\", roles: [] } )" , "code" : 59 , "bad cmd" : { "db.createUser( { user: \"my_username\", pwd: \"my_password\", roles: [] } )" : true}}
[22:20:25] <jordanpg> added question to SO: http://stackoverflow.com/questions/28707415/what-is-the-correct-syntax-for-the-dbcommand-method-that-is-replacing-the-conve
[23:59:59] <ra21vi> is mongo suggested as spatial geo db with 10% write and 90% read load scenario (geo queries mostly in radius search)...