Log file Viewer
#mongodb logs for Monday the 12th of January, 2015
[01:42:25] <anybroad> Well, now there are some items in a collection which should be sorted next to each other.
[01:42:45] <anybroad> I use the reference-number of item 1 to indicate it should be sorted right next to item 1.
[01:51:43] <drags> is there anyway to run or simulate 'db.<collection>.getIndexKeys()' in mongo javascript?
[01:52:47] <drags> I'm setting up sharding and I need to enumerate all of my collection index keys so I can plan my shard keys
[01:53:31] <drags> cheeser: how can I concat my collection names (read into an array using db.getCollectionNames()) into the db.<collection>.getIndexKeys() call?
[05:55:50] <anybroad> Do I still need this?: http://stackoverflow.com/questions/11303294/querying-after-populate-in-mongoose
[08:32:24] <Folkol> Hello, mongodb is logging (what seems to be?) all queries - filling up my disk. The db.getProfilingLevel() is set to "0". How can I disable this logging?
[08:35:40] <Folkol> I have read that it always logs slow queries, but I do not know how to distinguish "normal" profiling logs from "logs due to slow queries".
[09:30:20] <queretaro> Hi - In http://docs.mongodb.org/manual/core/backups/ it says that mongodump is not ideal for backing up large systems, might anyone provide a little bit more of an explanation regarding this?
[09:31:16] <kali> well, mongodump means that all the data has to flow through one mongos -> mongobackup connection, so it has a finite bandwidth
[09:31:34] <kali> you can in a situation where you need more than say, 24h to get all the data through
[09:49:28] <Spea> Hey there. I want to create a new index in the background on one of my collections. The documentation states that with 2.4 indexes on secondaries will always be built in the foreground. Does that mean, that when the index creation on the primary finishes, all secondaries will block the collection?
[09:51:57] <kali> 2.6 solves it, AFAIK. and there is a "procedure" to avoid that (basically, you build the index on offline secondaries)
[09:52:51] <Spea> i wonder what is less pain now, upgrading to 2.6 or the procedure you have mentioned :)
[09:53:27] <lqez> kali: you're right. http://docs.mongodb.org/manual/core/index-creation/#building-indexes-on-secondaries
[09:53:35] <lqez> It says “Changed in version 2.6: Secondary members can now build indexes in the background. Previously all index builds on secondaries were in the foreground.”
[09:54:37] <lqez> (Actually, I was stuck on simliar problem at 1.6 lol http://stackoverflow.com/questions/11649767/mongodb-copydatabase-runs-index-creation-on-foreground-not-background)
[09:57:34] <Spea> does anybody have a clue how long it would take to create an index on a collection with 17M rows?
[10:05:29] <Folkol> kali (and others who might be interested): The log-spam was due to slow queries, it seems like mongod is logging slow queries despite profilingLevel being set to 0. I added an index and the log went silent.
[10:26:14] <Folkol> http://docs.mongodb.org/manual/reference/configuration-options/#operationProfiling.slowOpThresholdMs
[10:43:31] <lqez> if you just use (not hack) mongodb itself, visit http://docs.mongodb.org/manual/installation/
[11:03:28] <giuseppesolinas> I'm getting Could not stat /dev/xdf --- No such file or directory when trying to install on EC2, what are the elegible partitions that I can use for mongo as in http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/#deploy-mongodb-ec2 ?
[11:12:51] <lqez> giuseppesolinas: ‘Individual PIOPS EBS volumes for data (1000 IOPS), journal (250 IOPS), and log (100 IOPS)’
[11:13:46] <lqez> https://aws.amazon.com/marketplace/search/results/ref=gtw_navgno_search_box?searchTerms=mongodb
[11:15:00] <lqez> did you create an instance via '$ ec2-run-instances ami-05355a6c -t m1.large -g [SECURITY-GROUP] -k [KEY-PAIR] -b "/dev/xvdf=:200:false:io1:1000" -b "/dev/xvdg=:25:false:io1:250" -b "/dev/xvdh=:10:false:io1:100" --ebs-optimized true'
[11:15:11] <giuseppesolinas> I've read that it could have different names but I don't want to overwrite useful stuff
[11:17:24] <lqez> and the document is just a case for using 3-different EBSs to get better performance than using only 1 EBS.
[11:19:23] <lqez> giuseppesolinas: and do not change /etc/mongod.conf as like as example on that page.
[11:25:24] <giuseppesolinas> lqez, btw yes, I should change mongod.conf if I've set up those folders
[11:38:18] <giuseppesolinas> is it normal for mongod to take a while to boot up? A couple of minutes and it hasn't booted up yet
[11:43:01] <kali> giuseppesolinas: you may want to check that you've put the data on a ext4 partition and not ext3
[11:44:03] <giuseppesolinas> otherwise I won't be able to change it because it's the main partition and I have this ami
[11:49:59] <giuseppesolinas> of course, if I am running my app and my database on the same machine I should refer to the mongo address as "localhost", right?
[12:31:55] <adrian_lc> I'm getting this error no matter what I do Error: couldn't add user: User and role management commands require auth data to have schema version 3 but found 1 at src/mongo/shell/db.js:1004
[12:32:40] <adrian_lc> I don't wanna update the auth schema cause ansible doesn't seem to support the new version yet
[12:51:11] <tommy_the_dragon> I want to do an update using the java driver. I have the JSON for the update I wish to make in a string. Is there an easy way to turn that into a DBObject?
[12:54:04] <tommy_the_dragon> I'm probably struggling with the basics on how to use the driver. Can someone link me a decent tutorial?
[12:56:22] <tommy_the_dragon> Of course I can work with what I have, but I'd rather not re-invent the wheel
[13:05:54] <tommy_the_dragon> kali: well, how am I supposed to deal with the result of a query? Maybe that's a better question.
[13:11:17] <kali> either pull one, modify it and save it, or make a new DbObject with your modifiers
[13:12:52] <tommy_the_dragon> That's what I mean, what's the best way to modify/pull values from them, when subdocuments are involved too
[13:15:14] <kali> also, note that if you build modifiers objects, you can make the as BasicDBObject, and they implement Map<> already
[15:40:00] <jiffe> Mon Jan 12 08:25:24.159 Invalid access at address: 0x706eeb27dcc8 from thread: repl writer worker 1
[15:41:49] <StephenLynx> I just saw some benchmarks where postgresql nosql tools outperformed mongodb by over 2 twice the speed. anyone here has something so say about that or knows if anything changed in the last year? or does really postgre outperforms mongo?
[16:10:58] <theRoUS> i'm having trouble formulating a query using regexes. i want to find all records where {'short_description':/CRITICAL/} AND {'$not':{'short_description':/ACK/}}
[16:12:23] <theRoUS> kexmex: isn't $and implicit? i've tried "{'short_description':/CRITICAL/,'$not':{'short_description':/ACK/}}" but get no records
[16:15:38] <theRoUS> yah, the '$not':{'short_description':/ACK/} returns null, so that must not be the right syntax
[16:17:19] <Derick> you probably would like to stay away from regular expressions like that though, as they can't use an indexed lookup
[16:25:07] <theRoUS> comprends.. but how? 'short_description' can't be the selector key, since regexes are dyadic. can't say 'short_description':{'$and':[{'$not':/ACK/},/CRITICAL/]} can i?
[16:27:00] <StephenLynx> https://gitlab.com/mrseth/bck_leda/blob/master/pages/search_user.js line 32
[16:29:12] <Derick> db.col.find( { $and : [ { 'short_description': /CRITICAL/ }, { 'short_description': { $not: /ACK/ } } ] } );
[16:32:10] <theRoUS> Derick, StephenLynx: got it. that works. probably totally ignores the index on that field, but once cached it's pretty fast. about a second to pick out 26_000 records from 100_000
[16:45:07] <Derick> StephenLynx: regex anchored to the start of the strin uses index, anything with $not does not
[16:49:35] <StephenLynx> if I use $regex : {field: '^foo'} will it understand it is anchored to the start of the string?
[17:47:26] <StephenLynx> because stuff like mongoose change way too much how you deal with the database itself.
[17:49:41] <anybroad_> For example, I want to assign categories from categories selection to books in books collection.
[17:52:39] <anybroad_> with other words, when I change something in category, it should be changed for the category the books got, too.
[17:53:19] <StephenLynx> or use a relational database since the problem requires a relational database
[17:54:10] <anybroad_> Pardon my ignorance, what would be the use cases for mongodb? I heard about it and like its flexibility - how can I make it useful for me?
[17:56:24] <StephenLynx> from what I heard it is easy to just add more servers to a cluster with mongo and you don't even have to stop the servers
[17:56:43] <anybroad_> but I will need this when I got many, many visitors and so a highload on the database
[17:58:53] <anybroad_> so when I change something in the category I would have to update it for each book, right?
[17:59:05] <anybroad_> So I need some hook-update system which does this updating each time the category is modified
[18:04:38] <anybroad_> so you are using the login and when you encounter a field with login you do a 2nd lookup to find the associated login of it?
[18:06:37] <StephenLynx> you either shape your user interface because of it or you make tons of queries
[18:08:30] <StephenLynx> non-relational db would be better? probably. but it is way longer and I'm lazy
[18:10:53] <StephenLynx> http://stackoverflow.com/questions/9412341/mongodb-is-dbref-necessary "Dbref in my opinion should be avoided when work with mongodb, at least if you work with big systems that require scalability.
[18:10:53] <StephenLynx> As i know all drivers make additional request to load DBRef, so it's not 'join' within database, it is very expensive. "
[18:11:46] <anybroad_> so one normalizes the data in nosql in a way that multiple queries aren't needed?
[18:13:16] <anybroad_> so updating or inserting in nosql database requires traversing and updating the related records, right?
[18:13:53] <StephenLynx> so you deal with the lack of relations by keeping your relations to a minimum
[18:14:23] <StephenLynx> and will just be using a crippled version of what relational databases already do.
[18:15:09] <StephenLynx> it doesn't surprises me. mongodb is not designed for relations, period. even if it implements the semantics, it does not work around that.
[18:16:24] <StephenLynx> because both you and his posts don't give any data, but his makes more sense because of how mongo is designed.
[18:16:36] <cheeser> there's nothing inherently wrong with DBRefs. the choice to use embedded documents versus references is a bit more nuanced than "never do this" or "always do that"
[18:18:12] <StephenLynx> "To resolve DBRefs, your application must perform additional queries to return the referenced documents. Many drivers have helper methods that form the query for the DBRef automatically. The drivers [1] do not automatically resolve DBRefs into documents."
[18:19:14] <cheeser> it can be perfectly valid to model your data like that in mongo provided you don't need those referenced docs all the time.
[18:19:40] <StephenLynx> the point is not that. is the lost of performance because you are not using a tool designed for that job.
[18:20:24] <StephenLynx> if you are not referencing the docs all the time, you might as well just use as little of the relations as I suggested.
[18:20:41] <cheeser> the point is he's making generalized, sweeping pronouncements absent ant context in which to make an informed decision.
[18:21:47] <cheeser> is mongo a bad fit? maybe. but maybe not. and no single sound bite from an internet forum is going to adequately answer that question.
[18:23:41] <anybroad_> alright, so when I add a new book, some code would update the category property in book j/bson, too.
[18:23:53] <anybroad_> and when I update a category, the code must go through each book and update the fields I changed in category
[18:24:07] <anybroad_> but I spare a lookup when fetching books (no pun intended), as the category is already there
[18:25:31] <StephenLynx> or you just have one field in the book for the category and you don't update the book.
[18:26:00] <drager> How do I make mongodb accept external ips? Because I have one website running that db and a site and I have one android application which needs that data from that server. But if I accept external ips will that result in a security flaw?
[18:26:24] <StephenLynx> there is a command, but if you look for mongo ip binding it should appear easily
[18:27:17] <anybroad_> StephenLynx: so a second lookup would be necessary but wouldn't be very expensive as mongodb queries are very fast?
[18:27:21] <drager> cheeser: I'm reading here; http://docs.mongodb.org/manual/administration/security/#SecurityandAuthentication-Ports
[18:27:44] <StephenLynx> a good architecture is to have an application running on the server that will accept http requests, query the database and output a response.
[18:28:09] <StephenLynx> anybroad_ a second lookup? if you wish to see the details of the category, yes.
[18:28:28] <anybroad_> StephenLynx: ah, I use the category name in category field, so I know it already
[18:29:22] <anybroad_> StephenLynx: as I do the same (the problem there is that your fron-end is connecting to your database), how should this function then?
[18:29:28] <StephenLynx> so I don't have to have two fields and update the books if I change the display name
[18:29:48] <StephenLynx> you are not connecting the FE to the database, you are just using an unique field as any other
[18:29:59] <anybroad_> StephenLynx: ok, so only the category name, and 2nd query is performed to look up the slug of category.
[18:30:28] <anybroad_> slug means in this case a short name of category to be used in urls for example
[18:31:29] <StephenLynx> I would either use the long name or use the body of the request for the parameters instead of the url
[18:32:42] <StephenLynx> it doesn't use the url for the request, it sends a string in the body of it
[18:34:05] <StephenLynx> https://gitlab.com/mrseth/lynxchan/blob/master/doc/Back-end%20interface.txt
[18:35:06] <anybroad_> StephenLynx: in this case it is not an app but a website, so I have to use slugs or normal urls for the user/seo
[18:36:33] <StephenLynx> https://gitlab.com/mrseth/lynxchan/blob/master/src/fe/js/common.js line 704
[18:37:21] <drager> cheeser: Shouldnt it work just binding mongo to 0.0.0.0 for it to accept external ips?
[18:37:54] <StephenLynx> drager if you accept external ips make sure you enable authentication. but you really should just put a webserver running to perform the queries
[18:41:24] <StephenLynx> I know because my company has some legacy apps that use that and they do http requests
[18:42:09] <anybroad_> StephenLynx: right, web apps are based on javascript, they wouldn't work without it, (plain html), right?
[18:46:11] <anybroad_> I use javascript for normal websites for polyfilling missing browser feature (e.g. css, etc)
[18:47:55] <StephenLynx> you may have reasons to do it, like wanting every user to be able to use the page even without js.
[18:49:40] <anybroad_> I use javascript for interactivity and filling missing browser functionality (polyfill) and it is very handy for this.
[18:49:53] <anybroad_> But it can be disabled and the html/css and link stuff would still be functional.
[18:51:14] <StephenLynx> and node can still do whatever you do with express. don't think express is a necessity
[18:51:56] <anybroad_> but when the future is javascript thick clients for the browser, using the dom (html) only for some rendering stuff, ok
[18:55:12] <anybroad_> StephenLynx: ok, that's interesting. So app.use(function(req,res,next){ can be replaced by some other middle-ware module?
[18:56:40] <StephenLynx> node just executes the function you gives it when you start listening to a connection
[18:58:08] <StephenLynx> express just hides it under a bunch of crap that nodes have to dig through to reach your code
[19:01:03] <anybroad_> StephenLynx: So I use express-validator, express-session, express-error-handler and express-enrouten
[19:01:14] <anybroad_> But those are standalone modules, they had been taken out from express to do one thing.
[19:05:27] <drager> StephenLynx: I have a webserver running with websockets pushing the data but Im currently getting; (android:http://meteor.local/:0) XMLHttpRequest cannot load http://domain.se/sockjs/info?cb=firnycyccl. Origin http://meteor.local is not allowed by Access-Control-Allow-Origin.
[19:15:24] <StephenLynx> that is a fucking pain in the ass indeed when you are dealing with browsers
[19:16:12] <StephenLynx> https://gitlab.com/mrseth/lynxchan/blob/master/src/be/operations.js line 194
[19:16:59] <StephenLynx> anybroad_ I don't know, I don't care, I wouldn't touch it with a mile long pole.
[19:25:19] <anybroad_> StephenLynx: So the html in https://gitlab.com/mrseth/lynxchan/tree/master/src/fe are served statically as normal html to browser?
[19:25:41] <anybroad_> StephenLynx: and could you give me a line in your example where references are de-normalized? Because I want to learn more about this.
[19:26:54] <anybroad_> StephenLynx: this?: https://gitlab.com/mrseth/lynxchan/blob/master/src/boot.js#L181
[19:29:15] <anybroad_> StephenLynx: e.g. when a login is removed and then this login also removed from boards staff field
[19:30:38] <anybroad_> StephenLynx: because I would have to do the same when a category is removed, the products must forget that category then or even removed altogether.
[19:31:30] <StephenLynx> https://gitlab.com/mrseth/bck_leda/blob/master/pages/delete_forum.js line 18
[19:31:51] <StephenLynx> in your case you would have to remove the field, there is an operator for that
[19:33:23] <StephenLynx> https://gitlab.com/mrseth/lynxchan/blob/master/src/be/public/setGlobalRole.js line 24
[19:37:38] <LuckyBurger> hey quick Q: if i have a node.js object with functions and i save it to a mongodb collection, when I pull it back out will it still have the prototype functions still on the object ?
[20:18:38] <winem_> hey guys, did anyone ever participate in the courses from mongo university? unfortunately, I was not able to watch all the videos of the first chapter, due to some critical issues at work.. chapter 2 starts tomorrow evening. would just like to know whether the videos and quizzes from chapter one will still be available after chapter 2 started
[20:19:38] <cek> what's the sense of having $elemMatch operator if you can just use {$gte: xx, $lt: yy} compound statement,for ex.,?
[20:20:50] <octoquad> hi winem_ I did, just completed homework for this week. Not sure if it will be, but maybe you can ask the question on the discussion tab.
[20:55:38] <centran> would anyone be able to help me with coping indexes from one server to another? (just the indexes)
[21:11:19] <jiffe> one of my mongod's is crashing right after I start it and it looks like it happens right after it starts background syncing
[21:18:47] <cek> for THIS document: for (var key in this.products) { emit(key, null); } . What's the stanza for THIS.subdoc?
[21:22:18] <cek> found it! ----for (var key in this.products) { for (var kk in this.products[key]){ emit(kk, null); }}
[22:00:46] <sellout> Is { "$elemMatch": { "$eq": 4 } } the right thing to do? "$eq" doesn’t seem to be documented, but { "$elemMatch": 4 } errors with something like “argument to $elemMatch must be an object”.
[22:04:38] <suupreme> Hi, I am using mongodb together with Go. I am writing some test now where I create a struct of a user. In the test I set the ObjectId. When I insert it in the test-db and try to query it back I can't by id but I can see the exact same user in there with exact same Hex of the id but in the test-struct it was a "1" at the end and in the db it became an "a" any ideas?
[22:05:42] <StephenLynx> you will have to deal with mongo doing stuff with the _id field. for one I don't touch it
[22:06:39] <StephenLynx> because find returns an array of pointers, aggregate returns an array and both are methods of the collection poiter
[22:08:48] <StephenLynx> I set whatever field I want as unique and pre-create, based on an incremented value somewhere
[22:09:09] <StephenLynx> and if you want to have any relation, you will have just an unreadable id instead of something like a name
[22:09:48] <StephenLynx> then you either make two readings for the readable name of the related object or you show the _id or show nothing at all
[22:10:22] <StephenLynx> so lets say I have a list of forums the user joined. instead of making an array with _ids, I set the forum name as unique and make a list of forum names in the user.
[22:13:27] <sellout> StephenLynx: But I have a field with an array of numbers … there are no subfields.
[22:19:43] <sellout> If I have a doc that looks like { …, nums: [4, 2, 1], …}, I want to aggregate([{ $match: { nums: { $elemMatch: { $eq: 4 } } } }, …]).
[22:21:30] <sellout> $eq seems to have appeared in 2.6, but I could find no mention of it, which makes me worry that it might break.
[22:26:49] <winem_> could need some help once more. wondering if it's just too late and I'm too tired.
[22:27:02] <winem_> mongoimport --db mc_mw --collection products --type json -p -u c_mw --port 37017 --file /tmp/products.json ... this return ins error 18 auth failed
[22:30:14] <sellout> StephenLynx: I don’t have a model. I’m working on a SQL->MongoDB compiler (http://slamdata.com) and am basically implementing `where 4 in someField`. Zips will work, though: { $match: { loc: { $elemMatch: { $eq: 43.058514 } } } } is what I currently generate from `select * from zips where 43.058514 in loc`.
[22:30:50] <sellout> Previously, we just used a $where in this case, but I saw $eq used somewhere, and it worked.
[22:42:30] <winem_> got it. helped to read the documentation about the authorization process itself once again
[22:43:32] <sellout> StephenLynx: Can you write out the full $match, because I think I’m missing something.
[22:44:54] <sellout> StephenLynx: someField doesn’t contain a number, it contains an array of numbers.
[22:47:39] <StephenLynx> the aggregate method will receive a callback, in this callback it will have 2 arguments: error and results.