Log file Viewer
#mongodb logs for Friday the 10th of April, 2015
[00:19:20] <GothAlice> luffy_: http://docs.mongodb.org/manual/reference/command/mapReduce/#output-inline < this page has many mentions of the "BSON document size limit", which you are encountering.
[00:19:48] <GothAlice> luffy_: You can explicitly not perform the map/reduce inline, i.e. the equivalent of the aggregate pipeline $out operator.
[06:04:36] <joannac> compeman: do you have an example on what kinf of daily tip you would find useful?
[06:22:53] <compeman> joannac: or some help scripts would be written in this channel. (getting from mongo docs maybe).
[06:25:39] <Boomtime> where would this daily tip appear? this sounds like a good suggestion for the mongodb blog
[06:26:54] <compeman> joannac: ok, the cost of 'searching in mongo docs' for a IRC bot is useless. ok. but maybe channel bots can daily share some tips or links from mongo blog.
[06:27:26] <compeman> it is just a suggestion, of course there are many people who are willing to help eachother here
[06:28:02] <Boomtime> this is a good forum for interactive help, i think the blog is more suited to the broadcast-style of daily tips
[08:41:53] <jokke> i'm having some trouble.. what does this error message tell me? failed with error 16837: "cannot use the part (3 of sections.4.page_modules.3._id) to traverse the element ({3: null})"
[10:24:39] <wawrek> how to query and retrieve embedded documents in mongodb? I am using the aggregation framework in order to query values of embedded documents and I would like to return these grouped by their parent's ids. is this possible ?
[12:46:59] <wawrek> I am looking for a way of querying embedded documents within an array. I want to return these (sub documents) grouped by their parent id. are there any examples showing how to aggregate and retrieve embedded docs within a list?
[13:14:26] <GothAlice> wawrek: http://docs.mongodb.org/manual/reference/operator/query/elemMatch/ http://docs.mongodb.org/manual/reference/operator/aggregation/unwind/ http://docs.mongodb.org/manual/reference/operator/aggregation/group/
[13:16:28] <GothAlice> It basically duplicates the main record once for each list item; then the field that was a list becomes a single element from that list for each resulting document.
[13:18:09] <deathanchor> let me rephrase, for data I have, I haven't had a use case for unwind yet. had no real world exp with it yet
[13:19:15] <deathanchor> now I got this exception: exception: assertion src/mongo/db/repl/rs_initiate.cpp:119
[13:20:16] <deathanchor> should upgrade eventually, but it's a sharded replset... (so many machines)
[13:20:20] <GothAlice> So, uhm, any reason you are't updating that to, I don't know, a supported version? ;)
[13:20:55] <GothAlice> (It can handle zero-downtime rolling upgrades for you. But, uh, I don't think it supports 2.2 any more.)
[13:22:08] <GothAlice> pamp: Depends on the dataset. mongodump/mongorestore are 12% faster on mine if I use WT. (OTOH, my dataset can kill it… so, I still use mmapv1.)
[13:26:08] <wawrek> I would also use $redact to get rid of embedded documents that do not match the query
[13:31:19] <pamp> dump: mongodump --db newModel --collection propertiesBased --out "C:\Program Files\MongoDB\Server\3.0\bin\dumps\planned_network_test"
[13:32:54] <GothAlice> pamp: http://docs.mongodb.org/manual/administration/production-notes/#mongodb-on-windows — Please install the hotfix.
[13:38:05] <GothAlice> pamp: Are you able to get MMS in there? It would be able to provide a lot of detail as to the operation of the mongod service during the dump and restore.
[13:43:16] <GothAlice> Also, as a note, I can't recommend WiredTiger for production use at the moment.
[13:44:17] <GothAlice> (Though not because of any performance issue; on non-Windows WT kinda rocks socks in the performance department.) Because of reliability and memory management. (I.e. it doesn't conform to its own documented limits and configurable options, and tends to get OOM-killed in my test cases.)
[13:45:03] <deathanchor> hmmm.. 2.2.6 doesn't like adding and removing a member in a single rs.reconfig()
[13:49:53] <GothAlice> … two years of at least one 64GB Azure node? (Which I can't seem to find, heh.) *insert low whistle instead of surprised expletive here* $36K (at a minimum) to test something?
[13:50:11] <GothAlice> I sometimes spin up 1000 VMs for an hour to test things, but that only costs $7.
[13:52:47] <GothAlice> pamp: Don't think I'm only down on MS, here. I'm also unimpressed by things like Compose.io's pricing. ;)
[13:53:49] <StephenLynx> if I had people that oblivious at management on my company I would be looking for a new job ASAP.
[13:54:01] <GothAlice> StephenLynx: Due to longer lasting hardware on average, the increase cost is amortized over a longer time and thus the value is actually greater than Windows machines. (I.e. we still have Macs that are running a-ok after 8 years.)
[13:55:10] <GothAlice> Indeed; generic "pc" commodity hardware (towers) and laptops a la Thinkpads, etc.
[13:56:08] <StephenLynx> even if it is a health 8 year obsolete hardware, you are still locked up like hell.
[13:57:06] <GothAlice> (Though mostly that install runs as a VM to let me orchestrate server updates and things.)
[13:57:20] <StephenLynx> last information I got from a mac developer is that installing any OS that is not OSX on a mac is not trivial.
[13:57:42] <GothAlice> It's EFI. Or you can ignore pretty much the whole thing by opening up Bootcamp once to enable BIOS emulation.
[13:58:09] <GothAlice> Sure the built-in keyboard is USB, and that may take a bit of planning in order to have a working keyboard on startup. ^_^
[13:58:10] <StephenLynx> then you got incompatibility issues, I also heard that you can't just use any hardware and install on a mac.
[13:58:39] <GothAlice> Could you clarify that last bit? "use any hardware" "install on a mac" — do you mean "install Mac OS on any hardware"?
[13:59:41] <GothAlice> Also, the three USB network interfaces I have both work A-OK. (One of them is a rather open WiFi chip to enable proper promiscuous mode.)
[14:00:32] <StephenLynx> and if something breaks you have to replace the whole thing because its all bundled like a laptop.
[14:01:27] <StephenLynx> also, it doesnt matter if it works after 8 years, it is obsolete by then and you could easily buy new hardware with the amount of money they overcharge for it.
[14:01:36] <GothAlice> I walk into an apple store, hand them my laptop, they take 5 minutes to confirm it's broke, they hand me a new one, and at home or the office I log in and *bam* everything is pretty much ready to go again.
[14:02:33] <StephenLynx> no, I am sounding like someone who doesn't like to be treated like a dog by its products.
[14:11:25] <deathanchor> My workstation only needs be powerful enough to run a terminal and browser. My real work is done on servers.
[14:15:40] <GothAlice> sadmac: I've used pretty much every OS known to humankind since 1989. ;) I switched from Windows to Linux, then Linux to Mac because I realized I wasn't paying myself to be my own syop, and the frequent need to "fix" things (video, audio, etc.) was hurting productivity.
[14:17:13] <GothAlice> sadmac: A few months ago I had a user chat me up who updated their desktop Ubuntu install. And broke the video driver. And re-UUID'd the partitions. The box got utterly hosed during a standard update. :thumbs up:
[14:18:38] <GothAlice> (And the number of people I chat with who are *constantly* wrangling with multiple audio routing daemons just to get everything working is very non-zero.) I'm a musician, photographer, sysadmin, and programmer (amongst other things) and need a computer that works for all of those tasks without needing to be "fixed". ;)
[14:20:31] <sadmac> Also you didn't say videographer, which I would have had to give you because linux NLE software is all crap right now
[14:21:49] <sadmac> Yeah, trying to use PiTiVi was the last time I felt like I was in the bad old days of linux
[14:22:07] <GothAlice> When I'm in the zone it takes me about two or three hours per minute of produced video… and I used to be able to go for days straight just editing. It's like a form of meditation. ^_^
[14:23:22] <sadmac> GothAlice: so what do you do for mongodb? Seems like every time I'm in here you're half the chat.
[14:25:39] <Derick> GothAlice: and that pointed me at https://www.youtube.com/watch?v=7JmveaEopfI :-)
[14:26:10] <GothAlice> Heh, another of mine. I flaied/failed a bit there after running into a parallelization deadlock. ¬_¬ Could have done it in half the time if that didn't happen.
[14:26:33] <GothAlice> I got a recommendation for "MTV - The Head" and had a sudden wave of nostalgia. XP
[14:27:00] <D3add3d> hello I would like to ask how to fix this error: http://pastebin.com/VZEPSej5 (I'm trying to compile the mongo-nonx86)
[14:28:01] <GothAlice> D3add3d: MongoDB only supports little-endian platforms for the server. The tools *may* (possibly should) work, though.
[14:29:12] <D3add3d> On https://github.com/skrabban/mongo-nonx86/tree/master it states that it is "version of mongodb for big endian processors (sparc, powerpc) and processors that care about alignment (sparc, arm)."
[14:29:51] <GothAlice> Unfortunately, that's a fork. I don't know if the developer of that fork hangs out here, or is awake.
[14:33:42] <D3add3d> Too bad MySQL eats up all the resources so I wanted something little less RAM and CPU intensive
[14:35:49] <D3add3d> ok... but I need only really small db for what I'm working on currently so that would not be a big issue
[14:37:23] <GothAlice> D3add3d: All database engines will attempt to consume all available RAM for use as caches. Simple fact; and a good one. Do not mix database services onto hosts doing other things.
[14:38:47] <D3add3d> GothAlice: I have only one Raspberry Pi and I don't have the money to pay for a hosting
[14:39:16] <GothAlice> D3add3d: On a Pi, use SQLite. It'll work way better in that constrained environment than just about anything else.
[14:41:33] <GothAlice> Similar to attempting to use a monolithic XML file on-disk. Neither of those are useful "at rest" storage systems as to do anything with it requires effectively full deserialization.
[14:42:07] <sadmac> the old tokyotyrant website just says "We are rearing children" and has a picture of a couple of kids on it
[14:42:14] <GothAlice> YAML, which supports streamed record separators, is only _slightly_ better. (You can seek around in the file and extract just the "record" you want.)
[14:47:24] <GothAlice> pagios: Tags are typically done as an array within your document: {_id: ObjectId(…), …, tags: ['foo', 'bar', 'baz']}
[14:47:51] <GothAlice> You can also index this type of field and it'll do "the right thing" (i.e. index the nested values).
[14:48:19] <pagios> GothAlice: are they dynamic? can i add them anytime so for example start with 2 tafs, and then if user adds a new tag that array dynamically stores it? no need to predefine the number of tags?
[14:49:02] <GothAlice> (The first {} should be your query to update the correct record, i.e. {_id: ObjectId(…)})
[14:50:16] <pagios> to select all rows that have baz and bar i do db.collection.find({tag: 'baz' || 'bar' }) ?
[14:54:22] <pagios> GothAlice: i have users subscribing to topics, say topic1 topic2 topic1 and something called alltopics
[14:56:04] <pagios> like everytime the users sub/unsubscribe from a topic, remove/add the corresponding tag on his row GothAlice
[14:56:17] <GothAlice> Yes, but I must first highly recommend running through a few tutorials, first. Get the basics down ('baz' || 'bar' — not a good sign ;) then definitely search around for specific tutorials on managing tags.
[14:58:38] <GothAlice> pagios: For example, http://docs.mongodb.org/manual/reference/operator/update/#array is the documentation for all of the array update operations. ($addToSet and $pull to add/remove tags.)
[14:59:13] <GothAlice> pagios: Because MongoDB is so different compared to people's typical relational background, it's important to get comfortable thinking the way MongoDB does about your data before you try to really use it.
[14:59:20] <GothAlice> I.e.: http://www.javaworld.com/article/2088406/enterprise-java/how-to-screw-up-your-mongodb-schema-design.html
[15:02:31] <GothAlice> pagios: For an example that isn't really directly applicable to your particular use case, http://www.devsmash.com/blog/mongodb-ad-hoc-analytics-aggregation-framework demonstrates how certain changes in how you store your data can affect query performance and storage efficiency. (Naive approach = 133MB data size, 94MB index size. Hybrid approach = 39MB data size, 23MB index size.)
[15:03:40] <pagios> i understand that mongodb has no schema means i can start with no array , add some arrays to some rows only and then tags on some othese rows etc
[15:04:47] <GothAlice> pagios: While MongoDB "basically doesn't care", developers should. For example, consider a simple boolean flag in your document. {} (flag missing), {flag: false}, {flag: true} — this greatly complicates your queries!
[15:06:00] <GothAlice> Missing and null can be interpreted as "use the default", true/false are explicit.
[15:07:34] <GothAlice> For sanity sake, it's also important to make sure documents within a collection conform to the same (pseudo-) schema. Storing completely disparate structures in the same collection screws up some indexing assumptions, even if MongoDB lets you do this.
[15:08:01] <GothAlice> pagios: If you have no tags in a document, {tags: []} — an empty list — will help avoid some query pitfalls.
[15:10:36] <pagios> btw is there anything an oracle/mysql.relationanl db that can do that mongo cant do?
[15:11:31] <GothAlice> Though MySQL is barely worthy of inclusion in any list that also includes Oracle. ;)
[15:13:57] <pagios> what is faster to have a join using oracle say or 2 queries into 2 tables in mongo and do the logic on application level?
[15:14:27] <GothAlice> That's the naive approach, and Oracle will be faster due to the extra round-trips. (Maybe; as with anything, this would need to be measured to confirm.)
[15:15:16] <GothAlice> However, faking "joins" in MongoDB is generally frowned upon. There are better approaches. In the example of forums, I store replies to a thread within the thread itself, and store a reference in the thread to the forum it belongs to.
[15:15:48] <StephenLynx> on the other hands, if the limitations of a sub-array are getting you, fake joins are better.
[15:16:01] <GothAlice> Because when you're looking at an individual thread you clearly want the replies, and when you delete a thread you want to delete the replies, this structure makes sense. I can then query for "replies 20-39" of the thread, and get the thread details, in one query, for presentation to the web.
[15:17:12] <pagios> heard of REddit that does an awesome job since it stored the db in RAM vs mongo on disk
[15:18:31] <GothAlice> pagios: http://www.mongodb.com/blog/post/high-performance-benchmarking-mongodb-and-nosql-systems
[15:19:19] <GothAlice> Somewhat relevant, but doesn't include redis. Redis, BTW, contributed to the failure of a five year project I was a member on, a while back. (Django+Redis+Celery = death when you get slashdotted/hackernewsd/lifehackered the same week.)
[15:22:15] <GothAlice> pagios: As a weird note, in several deployments I run MongoDB off of ramdisk. (Memory mapped ramdisk files = direct RAM access = blindingly fast.)
[15:22:27] <GothAlice> Most of my database servers don't have permanent storage. (I do this with Postgres, too.)
[15:27:06] <pagios> i think you need a loadbalancer on frontend to send the database updates to the different nodes of the cluster
[15:27:19] <pagios> coz if the system fails, no replication is possible at that specfic point in time
[15:27:52] <GothAlice> In both cases, I stream the per-minute oplogs to off-site deep storage with weekly snapshots. Worst-case (and I mean very wort-case) if there are zero nodes online (including off-site replica) when a new DB host is spun up it'll see the cluster as down, pull the latest snapshot, and replay all of the logs posted after the snapshot. If there is at least one other node (typically our office off-site replica) it'll stream the latest data off
[15:28:53] <GothAlice> You may have noticed a trend of placing most of the "complex code" in the client drivers. :)
[15:29:31] <pagios> yea but i mean at the minute the node servigin the client fails, the transaction goes with it
[15:30:44] <GothAlice> If there's an election called (to determine a new primary in the event of a failure of the current primary) all clients are disconnected and instructed to re-connect post election. It's up to your own code on what to do when this happens: retry, etc. Some queries you might not want to retry.
[15:32:03] <pagios> query server, when receive reply, commit on client side that data has been written for example otherwise retry..
[15:32:41] <GothAlice> As there are no "transactions" the concept of a "commit" doesn't apply. Update operations are individually atomic.
[15:33:06] <GothAlice> (Excluding multi-document updates, which if they fail mid-way can cause problems. Thus you can use two phase commits in that situation.)
[15:33:57] <GothAlice> In terms of how much checking the client driver does when executing a query or update, there's something called the "write concern". This determines the depth of checking from "ignore every problem, even socket problems" all the way to "make sure this data is propagated to a majority of the cluster, and is committed to the journal" (slowest).
[15:34:31] <GothAlice> pagios: I'm averaging 0.1s replication lag, with occasional 15s spikes I have yet to run down and fix. (I think I still have some unfortunate memory defragmentation options turned on.)
[16:01:32] <pagios> GothAlice: can you add remoe tags from a row without reinjecting the unmodified ones every time?
[16:02:11] <GothAlice> pagios: My previous examples, including links to $addToSet and $pull, all demonstrate how to do that.
[16:02:11] <pagios> for example in mysql if i want to update a json element i need to update["unmodifiedelemtn","modifiedelement2"]
[17:33:54] <Nomikos> is there a way to sort documents on some key, ascending, while keeping the documents with empty values for that key at the back?
[17:40:28] <razieliyo> so I guess I have to insert the exact element, not just the fields you want updated
[17:44:20] <razieliyo> Nomikos, I guess there's no way to do what I wanted, or maybe I could handle a requested object, modify it, then replace existent by that one
[17:44:26] <Nomikos> there is a modifier to make it behave, I think, it's been a while, just diving back into this
[17:46:32] <Nomikos> razieliyo: after 'query' and 'update', there is a third parameter options, an object where one of the keys is 'upsert', set it to true to tell update() to "not remove" existing fields
[17:46:45] <Nomikos> razieliyo: see http://docs.mongodb.org/manual/reference/method/db.collection.update/
[17:49:03] <Nomikos> I've had fun debugging php issues where, if some other variable referenced the inserted $data variable, $data would /not/ get the _id key after insert
[17:49:27] <Nomikos> this after finding $data /would/ get the _id key whether the insert was successful or not
[18:08:04] <Nomikos> I wrestled mongodb for 1.5 years, then did nothing for a while, then had to learn&use laravel/angular for 3 months, as of last week it's back to the old project
[18:39:09] <MacWinner> GothAlice, do you recommend goign to mongo 3.0.2 but not wired tiger at the moment? or just staying at 2.6.9.. i do want to eventually make the switch the wired tiger for teh storage compression
[18:39:44] <GothAlice> MacWinner: There are many, many improvements in general making 3 a worthwhile upgrade. I've upgrade all of my clusters but one, and I'm doing that one this week-end.
[18:40:27] <GothAlice> I typically stay with defaults; when WiredTiger becomes the default, I'll consider it for production use. :)
[18:46:38] <MacWinner> GothAlice, cool. any particularly salient improvement? reliability is the key for me
[18:48:02] <GothAlice> MacWinner: https://jira.mongodb.org/browse/SERVER-17444?filter=17503 < 1754 resolved tickets in the 2.7.0->3.0.2 so far.
[21:15:50] <chairmanmow> I'm trying to update a document in a collection from an object that has a value that will be stored in an array. For values that aren't in the array I want to overwrite them, but for the array values, I want to push to the array that's in the document. I've seen the $push and $addToSet operators, but I'm not exactly sure how to structure my query so the other non-array fields will also update. I'm trying to insert an object that lo
[21:24:34] <GothAlice> chairmanmow: db.collection.update({…query…}, {$push: {somearray: somevalue}, $set: {otherfield: othervalue}})
[23:06:20] <SoNewbie> Creating an application with erlang, as my web app and creating andriod and ios apps which all are data driven based on mongodb, I would like to know what is the best way to request/minipulate info from these three apps, create an api from them to call? or a web service? is there a common solution to this?
[23:07:30] <SoNewbie> my question do I create a layer on the top of mongodb to make these apps call it to get info and set info to mongodb? or any other database for that matter
[23:11:56] <markin> any ideas on how to make a query like: Model.find(where A=B and if(C=D) { then X = true})
[23:15:03] <markin> SoNewbie: normally if you have a bunch of client side apps, you want an api so that people don't communicate directly to your db
[23:16:27] <markin> SoNewbie: you'd normally expose like a rest or soap api and communicate just over http