Log file Viewer
#mongodb logs for Tuesday the 16th of June, 2015
[01:11:42] <MrAmmon> is there a simple way to create a collection in mongo from a subset of documents in another collection? For example, if I just want to create a table of 1000 documents from my collection of millions so I can test an index?
[01:12:32] <GothAlice> Aggregation lets you transform the documents however you wish during the migration.
[01:14:23] <MrAmmon> it's ok, I promise not to use my new found skills to assist my marketing department
[01:25:33] <GothAlice> Mine calls a 1-column CSV a database. Considering SQL, she's not wholly incorrect. (Zzing!)
[01:31:13] <GothAlice> Irony: my highlight color is green, and your name color is yellow, on my client. Excellent combination.
[02:23:09] <MrAmmon> does mongo have to actually find something on a query in order to return an accurate explain?
[03:40:06] <loth> Hey all, getting a "exception: need most members up to reconfigure, not ok" error even though I can reach the node im adding just fine https://pastebin.mozilla.org/8836929
[03:59:16] <madeti> why is this not working: find({'someArrayField': {$elemMatch: {$in: [{'k1': 'v1', 'k2', 'v2'}, {'k1: 'b1', 'k2': 'b2'}] } } })
[04:03:46] <Boomtime> @madeti: there are a few curious things going on in your query there.. first, you only have 1 predicate to the $elemMatch, so $elemMatch is superfluous
[04:05:21] <madeti> Boomtime: 'someArrayField' is a field in my document which has an array of objects/dicts,
[04:06:35] <madeti> either one of the objects from the array [{'k1': 'v1', 'k2', 'v2'}, {'k1: 'b1', 'k2': 'b2'}] in the 'someFieldArray'
[04:16:41] <Boomtime> @madeti: $elemMatch applies to it's own parameters, not to the parameters of operators that are nested deeper in - so you actually need each 'or' clause to be: { $elemMatch: {'k1': 'v1', 'k2', 'v2'} }
[04:17:06] <Boomtime> this also implies that $in is not the right operator, since it expects to apply direct matching
[04:17:40] <Boomtime> i.e you want an arrangement more like this: $or: [ $elemMatch: {}, $elemMatch: {} ]
[06:52:36] <Jonno_FTW> hi, I'm having trouble with pymongo, I can't get an index from a cursor, eg: readings[1234]
[09:28:20] <circ-user-22hxs> I have a collection with 2 millions documents, I'd like to move documents older than 6 months to a backup collection, how can I do this efficiently?
[09:29:51] <Lujeni> circ-user-22hxs, you can dump the document which need a backup, then remove it with a basic query ?
[09:31:23] <circ-user-22hxs> @Lujeni: it's multiple documents, around 1 million, so I'm wondering what's the best way to do this.
[09:35:15] <Lujeni> circ-user-22hxs, 1million is not a big deal. You can simply remove it. Your database lock will increase during this operation
[09:38:15] <circ-user-22hxs> @Lujeni : I'd like to move them to a backup collection, not remove them.
[09:40:01] <mtree> "errmsg" : "exception: the operator must be the only field in a pipeline object (at '$scrapes.ota'",
[10:23:01] <moqstheone> Is it possible in mongodb with mongoose to create a schema that contains an array with different schemas?
[11:01:29] <GoCrazy> I want to ask about a better way to implement autocomplete using mongodb , currently i am using simple RegEx , i want to make it more efficient .
[12:39:36] <GoCrazy> i want to know is there any way to use array variable with $each in aggreagtion in mongodb
[15:38:21] <GothAlice> ehershey: Anything worth doing twice is worth writing a tool to do it for you.
[15:38:52] <deathanchor> GothAlice: if you do it once, you'll probably have to do it again, so write the tool the first time.
[15:39:24] <GothAlice> deathanchor: Aye, my corollary is usually: Anything worth doing right is worth doing twice.
[15:41:39] <deathanchor> When you do things right, people won't be sure you've done anything at all. -Futurama Binary Galaxy
[15:45:17] <StephenLynx> I remember a full update would clock around 80mb, but its 50mb this time around
[17:04:07] <BadHorsie> I have a collections ipNets with entries like: {"cidrs":{"10_0_0_1/24":{"ip":{"10_0_0_2":{name:"mygw"}}}}}, I'm curious about how to search for {"name":"mygw"}, I usually do like "cidrs.<range>.ip.<ip>.name", but in this case I don't know the <range> or <ip> from the query...
[17:06:31] <StephenLynx> instead of making cidrs an array and the ip a field of the objects inside the array?
[17:08:22] <BadHorsie> We thought of that over "cidrs":[{"range":"10.0.0.0/24","ips":[{"ip":10.0.0.1,"name":"mygw"}]}] because it was kinda hard for updates, like the find would return one positional operator and you would need that trick of foreknowing the itemNumber...
[17:08:57] <BadHorsie> TBH we are just learning and trying to find a reasonable model and failing at doing so, part of learning I guess
[17:12:58] <BadHorsie> Interesting, so have an array of IPs per key-ed range... cidrs in your example is "10_0_0_0/24" and there are multiple cidrs expected, right ?
[17:14:25] <StephenLynx> there are? I don't know much about networking to figure your requirements just by what I understood so far.
[17:15:40] <StephenLynx> "cidrs":[{"range":"10.0.0.0/24","ips":[{"ip":10.0.0.1,"name":"mygw"}]}] is much better
[17:18:15] <StephenLynx> it depends more on how you will query it rather than how you will update it.
[17:19:30] <StephenLynx> if you want to query the ips from multiple ranges along with data that is pertinent to the range, you will have to perform an additional query to get the range documents after obtaining the ip documents and then manually append the data on the ips to fake a join.
[17:19:47] <BadHorsie> How would you build the update instruction? I have a db.ipNets.update({"cidrs.ips.ip":"10.0.0.1"},{$set:cidrs.$.ips.0.name:"mygw"}); which works in my case because I know beforehand that it's item subzero...
[17:21:19] <BadHorsie> So I use twice $ and it knows which array itemNumber I mean ? AFAIK there is a JIRA for it to work with like $1, $2 and so on... But so far is not possible...
[17:23:33] <StephenLynx> I know its not me because youtube loads fine, I am checking on the network.
[17:23:47] <asturel> hi, how can i .find( "target" : "xx" ) case-insensitive? regexp doesnt always works because there's sometimes special characters
[17:25:02] <GothAlice> asturel: Due to use of extended characters, you may be forced to save a second copy of the field, lower-case'd, and query on that.
[17:25:46] <GothAlice> But if you're allowing anything outside [a-zA-Z_][a-zA-Z0-9_-]+ as a username… you're gonna have issues.
[17:26:07] <GothAlice> asturel: Correct. {username: actual_username, username_lower: to_lower(actual_username)}
[17:26:44] <asturel> http://docs.mongodb.org/manual/reference/operator/aggregation/strcasecmp/ something like this doesnt work?
[17:27:00] <StephenLynx> Can't find it. But GothAlice should be able to confirm, you are able to use multiple $ to indicate indexes in an update, right?
[17:27:43] <GothAlice> asturel: Would your system let me use "🚸" as a username? (Yes, there's a unicode symbol between those quotes.)
[17:29:01] <GothAlice> https://en.wikipedia.org/wiki/Unicode_equivalence and http://www.unicode.org/faq/normalization.html should be interesting to you.
[17:29:19] <GothAlice> No, I mean, that type of query can't use an index. It has to load and iterate all possible matching records.
[18:46:25] <fllr> Hey guys! I'm running into a weird issue with mongo. As soon as I set up a user on mongo using db.userCreate(), it doesn't authenticate me anymore
[18:47:37] <StephenLynx> if you have authentication turned on but no users, you can log in from localhost without authentication.
[18:47:48] <StephenLynx> as soon as you create an user, you have to authenticate even from localhost.
[19:33:42] <tehgeekmeister> is there a way to compare two collections/mongo dumps for document level equality efficiently?
[19:51:14] <fllr> Hey guys! I'm running into a weird issue with mongo. As soon as I set up a user on mongo using db.userCreate() with the following roles: ['clusterAdmin', 'readWriteAnyDatabase', 'userAdminAnyDatabase', 'dbAdminAnyDatabase'] (in the admin db), it doesn't authenticate me even if I run db.auth() (in the database I want to use)
[19:52:53] <GothAlice> fllr: Were you describing that the user was created in a DB other than the one the user needs access to?
[19:53:06] <GothAlice> I.e. the user was created in the admin database, but needs to use database "foo"?
[19:53:39] <GothAlice> --http://docs.mongodb.org/manual/reference/program/mongo/#mongo-shell-authentication-options
[19:54:14] <GothAlice> fllr: Yeah, then you need to tell MongoDB's tools which database to connect to first and authenticate with, via the --authenticationDatabase option.
[20:34:03] <kopasetik> QUESTION: for some reason i cannot save to my users collection with this code: https://gist.github.com/kopasetik/5cc8ea545b4179277886
[20:34:54] <kopasetik> https://gist.github.com/kopasetik/5cc8ea545b4179277886#file-route-js Lines 31-38
[20:36:57] <deathanchor> is there docs about performance issue with queries that use $all and $nin on an array field which is indexed?
[20:40:58] <GothAlice> deathanchor: In general, issuing multi-value array queries will multiply the "difficulty" of a given plan by the number of values, i.e. worst-case it can't use an index and needs to manually compare each value passed in against the actual records. With an index, it still compares each, but these are much faster btree lookups. (Not hashtable lookups, but one step back from that in terms of performance.) The more values, the slower.
[20:45:15] <GothAlice> deathanchor: http://codyaray.com/2014/11/mongo-multi-key-index-performance goes into some detail, though the unicode errors make it hard to read on my display. XP
[20:48:37] <StephenLynx> GothAlice, how do you sort pinned threads in your forum? If I use two sorts in the aggregation, it will mess up sorting all unpinned threads. if I use one sort with the two fields, it will not put the pinned on top.
[20:49:03] <GothAlice> Pinned threads by definition only matter on the first "page" of thread results, to my forums. Thus: separate query.
[20:51:16] <GothAlice> Having more than a "page" of pinned threads, though, utterly defeats the purpose of thread pinning and obstructs normal usage by hiding ordinary threads.
[20:52:10] <GothAlice> (But we also do extensive "new since the last time you visited" checking and highlighting on the interface, so even if you refresh the "first page" of nothing but pinned threads, if an ordinary thread was updated, you'll still be able to see that fact.
[20:54:15] <GothAlice> … they'd all have the same value and it'd rely entirely on the modified order, wouldn't it?
[20:57:11] <GothAlice> In my forums, pinned are rendered separately from ordinary, too, so double reason for two queries, but in the general case, yeah, sort is good.
[21:19:19] <GothAlice> Hmm, okay, maybe it sorta does. I've never seen a support example that made overt use of it, I guess.
[21:50:49] <ruffyen> anyone know if there are any intentions to support raspberrypi with official MongoDB builds?
[21:55:38] <GothAlice> ruffyen: MongoDB uses memory-mapped files extensively, thus no, very difficult to abstract away while keeping performance.
[21:56:38] <rendar> GothAlice: you mean that MongoDB uses mmap to memory-map files and write binary data (e.g. integers) into files, which are for example in little endian? what you mean exactly?
[21:57:14] <GothAlice> rendar: Correct. Even things you might not expect to be integers are integers, such as ObjectIds, which contains four, I think?
[21:57:55] <rendar> GothAlice: the strange thing i can't get is: even if rasp is big endian, the I/O of those data will be big endian into the rasp file system, so what? :)
[21:57:57] <GothAlice> rendar: Basically, having the wrong endianness won't just garble your data files to be non-portable off that architecture. It'll also garble the BSON wire protocol.
[21:58:44] <GothAlice> It's engineered for one, and in a single codebase supporting both is, as mentioned, not simple.
[21:59:55] <GothAlice> I'd prefer to not have the added complexity, and thus bugs, that broad support for edge case platforms would bring, too. :P
[22:03:23] <Progz> Hello, I've git a trouble with a simple aggregate query. I Match on a domain name and sum the number of items. I had an index but nothing to take. It's pretty slow. 3 to 5 seconds.
[22:04:32] <GothAlice> db.collection.find({domain: "toto.com"}).count() < should give you the answer near-instantly.
[22:05:22] <Progz> db.collection.find({domain: "toto.com"}).count() => this would return the number of page in my website
[22:06:48] <GothAlice> http://docs.mongodb.org/manual/reference/method/db.collection.aggregate/#db.collection.aggregate < first documented option.
[22:07:37] <bmillham> How many times have you had coffee coming out of your nose after reading someone else's code GothAlice ;-) (And Hi, long time no see :-) )
[22:08:02] <GothAlice> bmillham: Howdy! Indeed, long time. Rarely. It takes a truly special gem of code to phase me, these days.
[22:10:57] <GothAlice> (When you have a linked list that's also a packed array that's also a ring buffer… you're in for fun times.)
[22:12:02] <rendar> union { void * ptr; uintptr_t integer; } u; u.ptr = some_pointer(); u.integer &= ~mask; user_the_pointer(u.ptr);
[22:13:33] <GothAlice> In fact, it chose an index that doesn't "cover" the query, even though one is available. (Likely because it's an aggregate and can't actually benefit from index query covering.)
[22:14:01] <GothAlice> $match, $order, and $limit, if not preceded by anything other than each-other, can use an index.
[22:15:13] <Progz> The best way to do it, it's to calculte them asynchronously and update the value in my document
[22:16:07] <GothAlice> var count = 0; db.foo.find({}, {nb_items: 1}).forEach(function(doc)}{ count += doc.nb_items; }) — I'd see how quick that is compared to the aggregate, for you. (Running an explain, or hinting, to use the index that includes nb_items.)
[22:16:44] <GothAlice> Then consider: either you do the loop, or MongoDB is doing the loop, but someone, somewhere is spending the time to loop over all that.
[22:17:02] <GothAlice> So pre-aggregation of the counts becomes a much more viable alternative to figuring out the total later.
[22:19:17] <GothAlice> Progz: The trick is that whenever you update nb_count on a document, also update the nb_total or whatever on some document somewhere that represents the "site" or other grouping you want to get stats on.
[22:20:50] <Progz> GothAlice: indeed. With the Page collection, we have the Domain collection with information about this domain. Il will add in the domain document nb_total