Log file Viewer
#mongodb logs for Tuesday the 6th of January, 2015
[00:39:31] <vagueBrother> hey yall, still trying to get an answer to my question about storing sub documents inside documents in a collection. are there any best practices regarding how to do this? it seems simple enough but i’m having trouble figuring out how to query an episode.id in this example http://plnkr.co/edit/6E7vOFTfcvPYWC73h77g
[00:46:43] <vagueBrother> can i query just on the id of the episode object without knowing its parent documents?
[00:51:39] <mordof> vagueBrother: wouldn't you have to do a query for the show, then query the seasons, then episodes? i'm fairly new to mongo
[00:52:20] <vagueBrother> that sort of structure makes more sense to me conceptually, rather than having an episodes collection and a shows collection
[00:53:11] <mordof> i know mongoid allows the use of full querying on subdocuments, so it's gotta be possible somehow
[00:53:49] <mordof> vagueBrother: http://stackoverflow.com/questions/20519073/mongodb-embedded-document-querying
[00:55:24] <mordof> vagueBrother: they come out as arrays, but if they are actually embedded documents it works
[00:56:01] <Boomtime> the rest i pasted directly, minus the ... bits (also invalid), then ran the query i pasted here
[00:56:43] <mordof> Boomtime: is there antything special one has to do for creating embedded documents?
[00:56:47] <Boomtime> { "_id" : ObjectId("acfb087a8efcb83086bfec03"), "title" : "Star Trek", "id" : 1234, "seasons" : [ { "id" : 2345, "seq" : 1, "episodes" : [ { "id" : 34
[00:59:55] <vagueBrother> i sort of assumed getting stuff out of this nested object would be similar to how you do it in normal js
[01:00:17] <vagueBrother> i already have an episodes collection that works fine when querying by episode id
[01:01:30] <vagueBrother> i’m dumb, and in my real data the episode id is called “episodeId” AND it’s a string, not a number, b'oh
[01:02:51] <vagueBrother> Boomtime: is there a best practice for this sort of thing? would it be better or faster somehow if i had a discrete episodes collection and then a separate shows collection?
[01:03:28] <vagueBrother> but i’m trying to get a feel for how best to deal with this sort of data from people who are more experienced with mongo
[01:05:03] <Boomtime> vagueBrother: firstly, it is not generally recommended against using multiply nested arrays where you want to query on the items inside the deepest level
[01:05:35] <Boomtime> * we recommend against multiply nested arrays where you want to query on the items inside the deepest level
[01:06:23] <Boomtime> the source of this recommendation is a limitation in the array positional operator ($) which can only reference the first level array position in a succesful match
[01:07:19] <vagueBrother> so you might recommend to have an episode object that has an associatedShow id or something?
[01:08:25] <Boomtime> you can contruct a season still if you need to, but you're more likely interested in the individual episode or the show generally anyway
[01:09:48] <vagueBrother> one final question and i’ll leave you alone, and if you want to point me to docs, i’m happy to read them
[01:13:13] <Boomtime> just provide all the matches that you need and they will all be required (by default) for a document to match
[01:14:19] <vagueBrother> so i can just put in a comma separated list of things i want to match and it’ll AND them all together?
[01:14:45] <Boomtime> mongodb requires a _id be present, you should provide a value for it and make use of it if you have something applicable
[01:15:09] <vagueBrother> that was a side question, actually. say i don’lt want to use _id and want to, instead index on showId?
[01:15:49] <Boomtime> mongodb requires a _id be present, you should provide a value for it and make use of it if you have something applicable
[01:19:32] <mordof> I've got a 'legacy' 2d index on location: [x, y] fields in my collection. i tried doing a location greater than [3, 3] but it only matches the x portion for the greater than. any way i can get both to match?
[01:20:09] <mordof> or would i need to separate the location into two individual keys and index each individually?
[01:21:56] <mordof> mhmm - i'm learning that gradually. i started out using mongoid so that knowledge is a bit further ahead
[01:23:43] <Boomtime> documents have a "location" field specified as [longitude,latitude] ? (or [x,y] if you just want dimensionless euclidean distance)
[01:27:10] <Boomtime> if you want to cheat to get the query that mongoid is running you can increase logging on the server to print it out to the logs, which will emit in shell format
[01:27:34] <Boomtime> but that might be more trouble than just learning to convert whatever mongoid does
[01:29:22] <charform> is there anyway to apply an index to a subset of documents based on the value of a field rather than the existence of a field? sparse index on documents where field == x?
[01:31:10] <charform> i have a users collection that contains users that signed up with their email and users that login via social media... I want a unique index on type: 'localUser', email: 1 but for that to not apply to social users..
[01:31:51] <charform> social users may have an email field that is the same as other social users ie if they login with google then facebook where the accounts are botha attached to the same email
[01:32:44] <charform> I guess I could create a separate collection for localUsers and social login users but other than this difference they have mostly the same fields and it would mean checking for user types before quering for users client side etc
[01:33:37] <Boomtime> charform: it sounds like you have an underlying unique condition that you have not fully captured
[01:34:43] <charform> is a sparse 3 feild index a decent solution? userType: 1, email: 1, socialId: 1
[01:36:12] <Boomtime> mordof: you are querying the field as though it were a regular array, you should look up the spatial query operators
[01:37:17] <Boomtime> charform: if you create a unique sprase index where one of the fields is only present when the other two fields are combinatorially unique, that should work for your case
[01:39:01] <charform> does mongo short circuit on such indexes? ie if after checking userType and email, if they are unique will it immidiately go ahead with the write operation?
[01:39:07] <mordof> Boomtime: so this: db.stars.find({ location: { $geoWithin: { $box: [ [ 3, 3 ], [ 7, 8 ] ] } } }) is for suer using the index?
[01:39:43] <Boomtime> mordof: to check if a query is using a particular index, add .explain() to the end
[01:41:01] <Boomtime> charform: the index will short-circuit as fast as it can, it doesn't need to check any specifics - the index is effectively a binary search path
[01:41:20] <charform> or what about concatenating email + socialId and using that as _id? for local users it would jsut be their email social users would break ties with their socialId and index on a single field
[01:42:29] <charform> although emails may be quite long I suppose and I would have to store the email again in a separate field
[01:43:13] <Boomtime> charform: that's an interesting idea, it certainly sounds plausible, you have options to test by the sounds of it
[01:43:42] <Boomtime> considering the data replication, consider that an index effectively replicates the data from the document
[01:44:59] <newmongouser> Hi, does anyone know where ListField is defined. I'm getting error : "NameError: name 'ListField' is not defined
[01:46:47] <Boomtime> you will need to provide more information, where are you seeing this error? what are you trying to do when it happens? what libraries/environment/versions etc
[01:49:21] <newmongouser> It's python 2.7.6, I'm actually trying to work with Flask, and I'm defining my database schema. I have class Org(db.EmbeddedDocument) and class Category(db.Document), in class Category() I've defined orgs=ListField(db.EmbeddedDocumentField('Charity')) and this is where I get the error
[01:53:12] <charform> Boomtime: would the standard ObjectId be more efficient for subsequent lookups though (as opposed to the concatenated email + socialId string)? I also have many documents in different collections that reference these user _id fields
[01:54:27] <Boomtime> objectid is assured unique inside of 12 bytes - that's pretty fast binary divergence compared to virtually any data you're likely to have on hand
[01:54:52] <Boomtime> but if that field is not being used consistently by you then why have it at all?
[01:55:18] <charform> they are appearing to be mostly a wash when running them now, mongoId slight faster but I have to create the object each time client side from the string my mongoclient returns to the app
[01:57:32] <Boomtime> i still doubt the small difference in speed will make any tangible difference in reality
[01:58:28] <charform> yah I suspect I will just go wit the sparse 3 field index unless there is a solution that is an order of mag better the concat seems mostly identical if not slightly worse
[02:00:07] <charform> a separate collection would mean two smaller indexes I guess, smaller b trees to traverse but I doubt that would be of any consequence unless the worlds population was using this service
[02:00:34] <Boomtime> sparese index is small, it contains only those documents which have the fields
[02:01:54] <Boomtime> you may consider that your sparse index only protects against the localuser duplicates, it does not prevent duplicates of combinations that should logically be unique (email:facebook for example)
[02:06:46] <charform> true, I was just thinking of the entire index, ie, every user is guaranteed to have at least one of the three fields
[02:07:12] <charform> therefore the unique check runs on the entire collection rather than the one it needs to only
[02:07:34] <charform> although if I put the socialId field first in the index it would short circuit liek you say
[02:14:24] <newmongouser> I'm using the Tumblelog tutorial on Mongo's website, and I've verified everything up to the point asking for a ListField, but I'm still getting a name error. Is there a known reason for this or should I just reinstall all my extensions?
[03:28:17] <chetandhembre> Hi I am storing array of object in mongodb but when i want to fetch it does not return result in proper formate,
[03:29:52] <newmongouser> on the mongo site comparing sql to mongo, it says these statements are equivalent: "SELECT user_id, status FROM users" is equal to "db.users.find( { }, { user_id: 1, status: 1, _id: 0 } )", what's the purpose of the empty document in the db.users.find(), it's in some of their queries and not in others
[03:32:00] <newmongouser> Oh OK, is it necessary? If I if I ran the query without it would it result the same?
[03:35:38] <newmongouser> Is the empty where clause necessary when i only want to return objects with those fields present?
[03:56:56] <newmongouser> On mongos docs it read: "Not specifying a query document to the find() is equivalent to specifying an empty query document." Do you think including it is just being explicit. Even from the page (http://docs.mongodb.org/manual/tutorial/query-documents/) it's not entirely clear
[03:57:35] <newmongouser> But in the snippet above - "db.users.find( { }, { user_id: 1, status: 1, _id: 0 } )", isn't the second document the query document?
[04:19:17] <rh1n0> Having trouble. I have a user that can connect to the mongodb but i keep getting an authorization error on a specific query - can you grant privileges to queries?!?
[04:34:23] <newmongouser> Do you need an empty where clause on a query or does mongo supply it itself? in the docs it says find() will supply it if not given, but in all of their examples it is present
[04:37:11] <Boomtime> newmongouser: the docs use the shell find() method as the example, which can be called with no parameters and it will assume an empty document
[04:38:00] <Boomtime> not that if you need to pass a value for one of the other parameters to that method (filter or options) then you will have to supply an empty document in the first parameter because, well, javascript
[04:38:46] <Boomtime> in short, it depends on the language and driver you are using - what language and driver are you using?
[04:42:39] <newmongouser> Boomtime: is it a good habit to use the first blank document in case you invoke other methods later down the road (other than the find() method?
[04:44:43] <Boomtime> newmongouser: your question also comes with the assumption that you'll be doing a lot of "find all" queries, which is kind of defeating half the reason of having a database - why would you always do this?
[04:46:56] <newmongouser> I won't. I was just going through Mongo's site in order to get more comfortable with best practices.
[04:52:19] <newmongouser> For my particular use I have roughly 40 'categories' with organizations listed in their respective category, so I'm running something similar to find( {category: 1}) or find({category: 3}) (category indexed)... There won't ever be more than 1500 organizations, so is the find syntax something I shouldn't bother getting more familiar with?
[04:53:03] <cheeser> you're going to need to know the "native" query language regardless of what driver your app uses.
[04:55:01] <newmongouser> When does the blank query document become something I need have concern for?
[04:59:19] <newmongouser> Ok, thank you cheeser. I'll read more on the read operations and projections docs. I appreciate your help.
[07:06:06] <jimbeam> Is it ok to declare two different schema in two files for the same object? I only want to access a subset of the properties in each file
[07:08:30] <Boomtime> jimbeam: it sounds like your question is regarding a library you are using, mongodb is schemaless - what library are you using?
[08:00:46] <charform> if a field is part of a composite index does that it is also indexed individually or should i create a separate index on it if I plan to query based on this field regularly?
[10:33:57] <Abluvo> hi, I am trying to install mongodb with a YAML config file which includes the storage.nsSize property but I keep getting parsing errors. Is there anything wrong with this YAML file http://pastebin.com/LAa5uapg?
[10:34:26] <Abluvo> The error I keep getting is: Error parsing INI config file: unrecognized line in 'storage:'
[12:44:20] <theRoUS> Derick: i'm mining a ticketing system, which means daily fetches of about 100K documents of at least 2K each from the ticketing system, and putting them into a mongodb.
[12:45:11] <theRoUS> i only want to do insert-new and update-if; is there a recommended best practice for checking to see if a document in mongo is different from one in memory?
[12:45:58] <Derick> I don't know... I guess you could calculate a clever hash - that you also store in the documents in the collection? I think that would be fastest
[12:46:08] <theRoUS> atm, i'm working along the lines of including a shasum field in the db documents, and just doing a find() of the key field and the shasum
[12:46:36] <theRoUS> and comparing the shasum from the find() with the one calculated for the in-mmemory data fetched from the ticketing system
[12:47:49] <theRoUS> update() has :upsert; i wish insert() had an :update_if (if the insert document already has an _id field)
[13:35:40] <Naeblis> Hi. How can I count the *total* of all nested arrays for my collection items which match a certain query? Eg: { $match: { foo: bar }} and get a total count of all the result.myArray items
[13:43:30] <leotr> i tried to create nfs share and move some of files to that share and then created symlinks
[13:54:58] <leotr> cheeser: so it's better to use iSCSI or if possible to connect some external drive
[14:26:00] <aliasc> just asking though. a guy in this channel dropped me a link on a project using mongodb
[14:49:03] <SpeakerToMeat> Before I google (this might be stupid), I wanna ask here... is there a "futton" for mongo?
[15:21:39] <xissburg> Is it possible to constrain the value of a property such as 'gender' to make sure only 'm' or 'f' are inserted?
[18:59:55] <Snoopy> Hey guys, I had a quick question about the php driver for mongo if that if this it the right place to ask it..
[19:26:08] <proteneer> in GridFS - if I’m updating an existing file, and a read comes in between the update, does it load the old file?
[20:06:01] <bhahn> does anyone know whether attempting a failover during a long foreground index creation will cause any problems (mongo 1.8)?
[20:29:46] <mng7> why doesn't that work to set processed to false on a single document where processed doesn't exist?
[20:30:00] <mng7> (i know i should specify multi to make it work for all of them, but let's stick to this for now..)
[20:44:02] <hrrld> Hello, I'm using the node.js driver and I'm calling .each() on a cursor for the first time. How do I signal that I've accumulated as many documents as I want, and that I don't want any more calls to the callback I passed to .each() ?
[20:53:11] <hrrld> Hm. There may not be a way: https://github.com/mongodb/node-mongodb-native/blob/2.0/lib/cursor.js#L420
[22:27:39] <harttho> sh.status() shows the primary for the DB is on shard 2, where the data is actually on shard 1
[22:37:18] <Boomtime> harttho: did you issue a movePrimary or have you dropped the database and re-created it any point in the past?
[22:47:52] <harttho> running mongostat on the other mongos (with missing collections) shows no reads/writes/etc
[23:17:02] <harttho> Boomtime: any thoughts? It appears mongos2 is routing to the incorrect shard, and mongos1 is reading from the correct, but empty, shard
[23:19:17] <Boomtime> may be easier to read this one first: https://jira.mongodb.org/browse/SERVER-15213
[23:20:04] <Boomtime> the good news is that you have detected it and can fix your situation very easily now; restart the mongos
[23:21:05] <Boomtime> you can avoid a restart of the mongos by issuing a flushRouterConfig if you prefer
[23:21:08] <harttho> Question is, when restarting the mongos, will the data from shard 1 be moved? or will it just start to write on the correct shard
[23:21:44] <harttho> We were going to move data from shard 1 to 2 (new primary), flushRouterConfig, enjoy writing data to existing collections on correct shard
[23:21:59] <Boomtime> you will need to manually access that data, connect to the shard directly and reconcile it manually