Log file Viewer
#mongodb logs for Monday the 13th of April, 2015
[00:36:56] <FrenchieUser12> i'm new to mongodb, how do i go about checking when was the last time that something was inserted into a collection?
[00:38:33] <FrenchieUser12> so i guess my question is whether there is a method that returns a date/time of the last insertion of a collection
[00:39:10] <FrenchieUser12> are object IDs created automatically upon the creation of a document or do i have to do that explicitly
[00:41:24] <FrenchieUser12> so i did not specify a specific id, is that what i should do in order to get a date/time of last insertion?
[01:51:11] <FrenchieUser12> and any update to the document has no effect on the result of that method right?
[01:56:54] <FrenchieUser12> if i dont specifically set _id for my document, are the ones that mongodb generate for my document in increasing order?
[02:08:12] <FrenchieUser12> so these documents are indexed by the object id for a b-tree. If it sorts by object id and created documents have increasing id, why doesn't mongodb have a method to get the last document? wouldn't it always be to the right of the tree?
[02:09:00] <FrenchieUser12> i'm just trying to get the last document created so i can get its timestamp, is there no easy way?
[02:28:38] <FrenchieUser12> what's wrong with this python code? cUsers.find().sort({_id:-1}).limit(1).getTimestamp()
[02:41:55] <FrenchieUser12> if i put quotes it says TypeError: if no direction is specified, key_or_list must be an instance of list
[02:45:14] <FrenchieUser12> still complains of TypeError: if no direction is specified, key_or_list must be an instance of list
[02:47:41] <FrenchieUser12> okay well cUsers.find().sort('_id',pymongo.ASCENDING).limit(1)) worked but if i try to get its timestamp with .getTimestamp() it complains of AttributeError: 'Cursor' object has no attribute 'getTimestamp'
[02:53:51] <joannac> FrenchieUser12: http://stackoverflow.com/questions/10404633/iterating-over-mongo-document-in-python
[02:55:50] <macwinner> i've updated to 3.0.2.. but have been holding off on wiredtiger until it's been in the wild for a bit..
[03:35:22] <FrenchieUser12> i get an error saying AttributeError: 'ObjectId' object has no attribute 'getTimestamp'
[03:39:55] <FrenchieUser12> print(ObjectId((docobject[0])['_id']).getTimestamp()) AttributeError: 'ObjectId' object has no attribute 'getTimestamp'
[06:07:00] <wday> Are there any rules of thumb for how long the initial provisioning of an mms backup can take?
[06:43:02] <fontanon_> Hi everybody, I have a clustering question: 2 servers, one with limited ssd disk and another with tons of non-ssd space. I would like having the most recent records in my mongodb collections in the ssd one and the older in the non-ssd one. Does it makes sense? Would an specific sharding config help?
[08:08:14] <Nomikos> morning all. I'm trying to sort a contact list by company, from a-z, but it starts with all documents where company has no value or is unset
[09:53:34] <vagelis> I want to check a field called URL if it starts with a url and if yes change the url to something else but keep the path i want to change only the host address
[09:55:03] <vagelis> Btw I am not sure if you can see my messages or I should first identify or something I am not familiar with irc etc :S
[13:37:45] <jr3> is there any way to return a cursor or something that I can process 10 results from mongo, t hen get the next 10
[13:40:58] <StephenLynx> it just won't hit the db for the next set of results until it finished the batch, I think.
[14:02:02] <pagios> hi guys, so i want to insert a new row if finding that row returns null, is this possible in 1 command?
[14:35:16] <fontanon> Hi everybody, is there any sharding strategy to keep recent records in an SSD disk and historical records in HDD ??
[15:13:07] <itisit> i use command rs.add("mongo2") to add secondary instance for replica set, but then rs.status() shows both instances are now as "PRIMARY", what could be the reason?
[15:14:37] <StephenLynx> cheeser "these people who I never gave anything in any shape or form to them are obliged to spoonfeed me because I can't RTFM" That is what it means.
[15:28:21] <fontanon> cheeser, btw I found the key blog post for tag aware sharding based to keep recent records in one machine and the older in another: http://blog.mongodb.org/post/85721044164/tiered-storage-models-in-mongodb-optimizing
[15:54:27] <pagios> hi i am using ruby binding and doing: result = client[:'userTable'].find(:subscriberToken => token).update_one( "$pull" => { :subscribedTo => topic } )
[16:07:34] <greyTEO> whats the proper way to setup auth for mongo? Should the users be added to the corresponding db or all inside admin?
[16:14:38] <deathanchor> greyTEO: need to setup admin user in admin db first, then you can setup users per db if you want.
[16:14:57] <itisit> deathanchor: i think the problem is all three members runs rs.initiate() before join
[16:15:27] <deathanchor> you only need to run rs.initiate on one member, which becomes primary, then add the other members.
[16:16:41] <greyTEO> deathanchor, I setup the admin and a user, but I cannot connect from anywhere outside the shell
[16:30:51] <DragonPunch> like in the subdocs i have a Date and i want to get the latest item in that subdoc by date
[16:40:56] <greyTEO> deathanchor, I made a quick gist showing the error I keep getting: https://gist.github.com/dmregister/346aa4402a6fae4f2d54
[16:47:54] <GothAlice> greyTEO: Please read: http://docs.mongodb.org/manual/release-notes/3.0-compatibility/#security-changes
[16:48:06] <GothAlice> See also: http://docs.mongodb.org/manual/release-notes/3.0-compatibility/#driver-compatibility-changes
[17:22:13] <GothAlice> greyTEO: Then the thing to check is that the users are actually contained in the database you expect them to be. MongoDB users are database-specific, and if the user has permissions granted on a different database, you need to explicitly tell it which database to authenticate against, separate from which database to use for data access.
[17:51:12] <greyTEO> I guess this is the localhost exception i am facing..which should be mean that I do not have a valid admin. but in fact I do
[17:51:46] <GothAlice> Well, it could be that the Java driver is attempting TCP communication, and the command-line version uses UNIX domain sockets. What's your listen configuration directive set to?
[17:53:09] <GothAlice> http://docs.mongodb.org/manual/reference/configuration-options/#net.bindIp — this thing. Ugh. Too many meetings on Monday = brain screwed up all day.
[18:20:46] <StevePerkins> Anyone using Spring Data MongoDB? When a document contains one or more nested documents, all of the code examples I’ve seen show that nested collection mapped to a Java “List”. I’m wondering if you can use Java’s “Set” instead (which doesn’t allow for duplicate elements)… to ensure that you never write a non-unique collection to Mongo, without having to make that field indexed?
[18:22:07] <StephenLynx> no idea. most people here frown upon frameworks. You could have a better chance on a place dedicated for the framework itself.
[18:25:33] <GothAlice> I often preach of the benefits of a good abstraction. MongoEngine being my preferred one for Python due to the seamless nature of its additions like triggers, referential cascading, etc., etc.
[18:26:25] <GothAlice> StevePerkins: In terms of treating lists like sets, MongoDB provides $addToSet to do the "add only if missing" part. The rest is no different than handling any other list.
[18:29:23] <StevePerkins> Yup, I see that the “addToSet” option is supported for the Spring API’s update operation. But it only seems to support auto-mapping of a nested collection to “List” containers. Presumably because if the field isn’t indexed, then there are no guarantees that it won’t be populated with duplicate values by something outside of your application code.
[18:30:45] <StevePerkins> I can live this as-is, or just scrap the Spring wrapper and roll my own solution with the raw Java driver. But since my application code will be the only thing ever writing to this database, it would be nice to “cheat” and effectively get uniqueness without incurring the penalty of an index.
[19:02:55] <Biganon> Hi everyone. I've never used MongoDB before, but I have a question : would it be possible to have something like a 1440 × 200 × 6 array (tridimensional as you can see) in mongodb, and quickly (and I mean really quickly) access its elements ?
[19:10:23] <Biganon> you answered "no" to the question "would it be possible to do it quickly", so I asked why it wouldn't be possible to do it quickly
[19:11:00] <Biganon> well I guess it depends what we mean by "quickly", guess I'll just have to try on my server to see
[19:12:13] <GothAlice> You can only $elemMatch one array in a query, thus having more than two levels deep = making your data effectively un-queryable. Effectively. There are ways around anything, but unrolling the unrolled unroll of an unroll in an aggregate query = asking to be committed for insanity. ;)
[19:13:17] <GothAlice> Well, no, it would require too much time and effort to simply wrap your head around the query at all, let alone it being fast, which it won't be.
[19:13:58] <GothAlice> Instead, a reasonable and "naive" approach would be to have a document with the vectors separated, e.g. {foo: 1, bar: 2, baz: 3} where foo/bar/baz are the different dimensions you are modelling.
[19:14:29] <GothAlice> (And the rest of the document would represent whatever value you were intending to store at that combination of vectors.)
[19:15:26] <GothAlice> http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework measures the difference between several different nested storage structures, for comparison of speed and data storage requirements.
[19:16:36] <GothAlice> Separating the values out like that would also let you benefit from indexes, and make any queries you need to fetch or manipulate that data quite trivial.
[19:17:08] <Biganon> I can't say I understand everything, but thank you for the link, and for your advice !
[19:17:44] <GothAlice> someArray[x][y][z] = {name: "Bob"} — how is this different than {name: "Bob", x: x, y: y, z: z} ?
[19:18:24] <cheeser> in order traversal gets trickier but then what "in order" means could be anything...
[19:21:28] <Biganon> (this is the perfect channel, I've been there for 10 minutes and 2 sysops answered with good ideas)
[19:46:37] <ep1032> Stupid question of the day: my password starts with a - character. Whenever I try to log into mongo (command shell), it thinks my password is an argument, and fails on "ambivilous argument"
[19:48:38] <GothAlice> ep1032: On UNIX machines (or in any POSIX-compatible shell) you can use "--" (double hyphen) to terminate the argument list. This lets you use arguments beginning with a '-', for example, if you accidentally name a file like that: rm -- -foo
[19:49:07] <GothAlice> Or, finally, you can have the password prompted interactively (without echo) by supplying it as the last argument.
[19:50:46] <NoOutlet> On Biganon's 3d matrix question, if an order could be determined before indexes were created, you could make in order returned results very quickly as well.
[19:52:02] <GothAlice> ep1032: Please read the manual: http://docs.mongodb.org/manual/reference/program/mongo/#bin.mongo
[19:53:02] <GothAlice> As a wholly unrelated issue to your command line problem, any particular reason you're using a version with known defects instead of the most recent for that release, or, preferably, most recent?
[19:54:05] <GothAlice> ep1032: From when you first joined and were impatient within the same minute, I think you just need to read the manual more slowly. ;)
[19:57:35] <ep1032> still failing, but now at least for authentication reasons, not cmd line parsing reasons
[20:02:29] <tehgeekmeister> how important a limitation is it that mongodump doesn't back up the local database?
[20:02:49] <GothAlice> tehgeekmeister: The local database contains mongod-specific configuration details; it's not even replicated.
[20:02:52] <tehgeekmeister> I'm doing an emergency rollover of a mongo, and hadn't had the time to read the docs previous to this in much depth, so I'll need to rely on IRC to some extent.
[20:03:14] <itisit> how to revert a member of replica set back to standalone mode? or how to clean all config data so mongo can be restarted clean?
[20:03:37] <tehgeekmeister> GothAlice: but if I just want to restore the content, and our config wasn't special/etc., it seems that may not be important?
[20:03:43] <cheeser> seems like you'd be better off reading the docs *now* since you know you'll get *something* done.
[20:05:05] <tehgeekmeister> as in, the person who built this system is not around, and i have to redeploy it. part of the redeploy process as it exists is migrating the database from one instance to another.
[20:06:03] <GothAlice> In general my own "migrating" processes are as simple as adding a new replica, letting it settle, then turning off the old node. Is your process more involved, tehgeekmeister?
[20:07:26] <GothAlice> itisit: Simply rs.remove() the node, then restart it without the "--replSet" option or configuration directive.
[20:07:53] <tehgeekmeister> fewknow: i didn't design the previous deploy process. I wouldn't have done it that way, but what's done is done, so. until I have a good chance to come up to speed and make more informed decisions, it's not the time to change that drastically, I don't think.
[20:08:28] <tehgeekmeister> the server also hosts an application. that application needs to be upgraded. we do upgrades by deploying entirely new images.
[20:10:01] <fewknow> tehgeekmeister: there is no reason to dump and restore. Unless you need recover data or migrating 1 DB to separate it from others on the replica set
[20:10:58] <fewknow> tehgeekmeister: there is also a tool for doing this called mongo-connector. Using this can migrate data without downtime.
[20:12:44] <tehgeekmeister> fewknow, i understand what you're saying, but i don't want to change a deploy process i don't understand, with a database I do not really understand, on the day when the system needs to be used for a pretty critical function
[20:14:28] <fewknow> tehgeekmeister: I am confused on "a deploy process" deploying is different that migrating data. If you need to deploy a new replica set that is one issue. Having to migrate data is separate. I don't know anything about your data or your process, but I know how to move data with MongoDB
[20:15:04] <tehgeekmeister> but i'm trying to minimize risk by minimizing the number of changes i make, while still allowing myself some disaster recovery options. mongodump seems to fit those criteria, so long as I can be sure it will preserve all of what I care about.
[20:15:24] <tehgeekmeister> fewknow: it's a deploy process for an application+db pair. unfortunately, the deploy process manages both as one thing.
[20:15:58] <tehgeekmeister> since we deploy by creating an entirely fresh box, databases are tricky. the data migration is already handled as part of the deploy process.
[20:16:01] <fewknow> tehgeekmeister: you are increasing the risk by messing with the EBS mount and backing up restoring data. If you add in a node to the replica set, mongoDB will do everthing for you
[20:16:56] <tehgeekmeister> the point is that I have a process, I don't understand it or the systems around it well, and I don't want to change the whole thing until I understand. I just want to provide myself a safety in case things die terribly.
[20:17:32] <tehgeekmeister> yeah, i think that's fine. i can confirm that we don't have any interesting config.
[20:17:40] <fewknow> but unless you lose the entire replica set then you will be fine iwth just the dump
[20:18:27] <GothAlice> tehgeekmeister: As a deployment comparison, many of my DB hosts don't have permanent storage; the entire /var/lib/mongodb is RAMdisk. High availability is handled through replication failover. (Including two offsite replicas, one delayed by 24h.)
[20:18:53] <GothAlice> Worst-case all nodes on Rackspace die, I start a new one up and it replicates from the in-office copy, starts up, and the app is back online.
[20:19:08] <tehgeekmeister> as we switch to a more sophisticated setup, i'll probably consider stuff like that
[20:19:29] <GothAlice> tehgeekmeister: https://mms.mongodb.com/ < free for < 8 hosts. Handles rolling upgrades and everything for you.
[20:32:18] <GothAlice> If only I had a time machine and a cattle prod to use on certain developers… ;)
[20:36:47] <GothAlice> I like the solution that prevents the initial confusion, even if it defies physics. ;)
[20:39:15] <webjay> it doesn't seem to be Mongoose as the .collection gives me the Mongo driver, but it seems the my prod env connects slower than my dev, and that might be the problem, so I'm trying the wait for conn open
[20:47:24] <MacWinner> GothAlice, upgrade to mongo 3.0.2 was pretty much flawless... still holding off on wiredtiger though.. hope you don't mind if i periodically checkin to see if it gets your seal of approval
[20:47:52] <GothAlice> MacWinner: It likely will once it is scheduled to become the default: 3.2 or so.
[21:29:02] <unholycrab> the docs say to pull each secondary out of the replica set to build indexes on them
[21:29:19] <unholycrab> but why can't i just build in the background on each secondary, then build in the background on the primary?
[22:09:42] <joannac> unholycrab: if you're taking the secondaries out, why bother building in the background?
[22:15:32] <pagios> i want to provide authentication for users to query my mongodb, should i store the user/pass in a document and check for authentication on that or is there any better way
[22:17:52] <GothAlice> pagios: Could you clarify what you mean? a) You're exposing MongoDB to your users to let them use arbitrary tools against your data, or b) You're exposing MongoDB via your web application and need user authentication / authorization at that layer.
[22:20:05] <GothAlice> pagios: First, grab a copy of an "scrypt" library for your language of choice. (Best not to try to roll your own password security.) Then you can have "user" documents which store the scrypt hash of their password. To check if a user login is valid, compare the hashes. (The library will have a "validate" or "verify" method for this.) You're going to also want to make sure you app is served over HTTPS only.
[22:20:38] <joannac> unholycrab: well your choices are take the secondaries out, or have very high load and potential downtime
[22:21:06] <pagios> GothAlice: correct but i mean i store the user and encrypted pass in a document and it is fine
[22:24:15] <GothAlice> pagios: scrypt is such that if someone steals your entire database contents, it'll still take them _hundreds_ of milliseconds _per attempt_ when brute-forcing. I.e. it'll take longer than the remaining lifespan of the universe to get a 50% chance of breaking one password… as long as they aren't dictionary words or otherwise weak.)
[23:22:06] <joannac> maybe http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.__getitem__
[23:24:26] <joannac> "Raises IndexError when the slice has a step, a negative start value, or a stop value less than or equal to the start value."
[23:24:31] <GothAlice> FrenchieUser12: next(cursor) will iterate the cursor one step and return the next record. This is effectively what a for loop does: it just calls next(cursor) until there's nothing left. Better, if you only want one record, is to use collection.find_one() instead of collection.find() to issue the query. See: http://api.mongodb.org/python/current/tutorial.html#getting-a-single-document-with-find-one
[23:25:11] <FrenchieUser12> i don't think i can use .find_one() because i want the first doc in the collection
[23:26:25] <GothAlice> http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find_one < takes the same arguments as find, which means you can sort your results.
[23:26:59] <GothAlice> In fact, there's an open ticket to request "random" as a feature. Such a feature currently does not exist.
[23:27:09] <GothAlice> See http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find for the additional arguments find_one() supports, including sort=.
[23:28:14] <GothAlice> Then you'll have no control over which document you get. It might be the first doc entered, it might not be. If you truly want the first document created that matches your query, sort by _id ascending.
[23:28:23] <FrenchieUser12> or would sorting sort by _id and since they are in increasing order it would give me the first one
[23:29:09] <GothAlice> (As long as you're using true ObjectId values as your _id. If you're not, you'll need to find something else to sort on, i.e. an explicit creation time field.)
[23:48:42] <DragonPunch> okay so when i query, my subdocs are returning as [Object] when I console.log them
[23:48:54] <GothAlice> FrenchieUser12: To write that, I read the documentation and followed what it said exactly. http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find_one I read "All arguments to find() are also valid arguments to find_one()" clicked "find()" to jump there, then read the description of the "sort" argument/parameter which says "a list of (key, direction) pairs…" with a link to sort() for details.
[23:50:03] <DragonPunch> GothAlice: I tried JSON.stringify and toJSON however, they didn't fix my problem
[23:50:35] <GothAlice> ("Normally" for JavaScript, that is.) If you want it in another format, you'll have to encode it. Alas, I don't JS, so this is the extent I'll be able to assist with that. :(
[23:51:06] <FrenchieUser12> ugh... when i print (collection.find_one(sort=[('_id', pymongo.ASCENDING)]) it just says None.
[23:57:46] <GothAlice> FrenchieUser12: I often find it useful to test things out using an interactive shell. For Python I use "ipython" as it gives nice tab completion and syntax highlighting. After some playing around, this is what I get: http://showterm.io/08680692e051220002d26 — it works as intended. Thus, if you are getting no (or the wrong) data, it's your data.