Log file Viewer
#mongodb logs for Thursday the 5th of June, 2014
[00:59:30] <codenado> hi all. I have a question about many-to-many relationships. I have users and groups, users can be in many groups, groups contain many users. What’s the preferred way to store that? One option is a person list within the group document, and a group list on the user document, but that means data is duplicated
[01:04:25] <joannac> d you need to know who is in a group, as well as what groups someone is in, on a regular basis?
[01:05:50] <joannac> seems like both of those can be satisfied by having a list of groups in a use document
[01:07:08] <codenado> if i went that route, would the query to get all users in a group be efficient?
[01:08:00] <codenado> it seems like you’d be looping through every single user for that data, or is mongo efficient at that type of query?
[01:12:22] <joannac> you could index on the "groups" field, and then search db.users.find({groups: foobar})
[02:31:14] <nezt> i'm a graduate student, and i have to build a web application that basically serves as a formatted viewport for a mongoDB database. i was wondering what some of you recommend i use as the backend language/framework
[06:57:03] <voidhouse> I wan't to sync a mongo document with the client (browser, at least for now) in real time? Something like firebase.
[07:11:16] <kali> voidhouse: mongodb is not designed to be accessed directly from a browser, apart except for monitoring and audit
[07:11:56] <voidhouse> kali: Yeah, that is what I am looking for, something with Node.js, GoLang, or even Python. Or whatever really.
[07:30:16] <mipo> I checked mongodb official website , but i didn't find where is code description part?
[07:40:17] <noqqe> kali: do you have expierence using dns-roundrobin for (lets say 3) mongos instances ?
[07:40:52] <rspijker> you can just have a connect string with multiple mongos instances in there, can;t you?
[07:41:25] <rspijker> kali: not 100% sure… I would have to ask our dev guys how we actually do it :P
[07:42:00] <kali> noqqe: i would not dare messsing with hostnames and dns and stuff around the mongodb discovery protocols
[07:42:41] <kali> noqqe: i know for sure it leads to all kind of weird (hear "bad") things in replica sets. maybe for mongos it's less awful, but i would not do it
[07:44:47] <rspijker> or just run them on the DB servers… it’s slightly less optimal, but shouldn’t matter much in terms of reliability
[07:55:48] <rspijker> db.games.aggregate([{$project:{“f”:{$add:[“$Attrint1”, “$Attrint2”]}}}, {$match: {“$f” : {$gt: 1, $lt: 2000}}])
[09:06:22] <voidhouse> rspijker: No, something like firebase, to keep objects at client side (browser) and the database in sync.
[09:07:33] <voidhouse> rspijker: Yeah, meteor is nice but it's a fullstack, I just want a simple API.
[09:08:30] <voidhouse> Perhaps I can dig through their code and see how it works, problem is though, it is using sock.js/socket.io
[09:10:47] <voidhouse> I have lots of processes that push their output to mongodb and I want a way to have the most recent version in the database.
[09:12:35] <voidhouse> Nodex: Nope. Say I have `log { line_a: { text: "Download Progress" , value: 0.9 }; line_b: {....} }`
[09:13:47] <voidhouse> Nodex: Perhaps that is a way, but that means I will have to keep ask the server for possible changes.
[11:10:12] <jekle> I´m considering to replace mysql in our new webshop/cms because we need to define attributes dynamically for several entities within the system.
[11:10:58] <jekle> is this a common use case for mongodb? I have been reading articles and watching yt videos since days.and I am so unsure
[11:11:22] <asido> any ideas why I have such behavior when specifying variable in $projection: http://paste.kde.org/pxhf0fgje ?
[11:19:14] <Nodex> jekle : if your shop needs transactions then you shouldn't move the whole app to Mongodb
[11:25:30] <jekle> the apps audiences are quite small and we aren´t the greatest devs, in short: we haven´t used transactions yet :) but no problems so far. thus I think its maybe not so important for us. Thinking about using both I am concerned about too much complexity. managing two permanent related datastores sounds somehow like a challange
[11:25:34] <rainerfrey> I just setup a replica set for testing, with all nodes on the same hostt, and the logs of the mongods are full of entries like connection accepted from xx.xx.xx.xx, and end connection xx.xx.xx.xx
[11:26:33] <rainerfrey> each connection seems to live for 20-30 seconds. Is this normal that a replica set uses so short-lived connections among each other?
[11:27:14] <rainerfrey> and: is there a way to reduce that log output, as this seems to be quite unnecessary information
[11:28:03] <rainerfrey> (I read about configuration option "quiet", but that seems to suppress a lot more information and is knot recommended in the docs)
[11:30:13] <rainerfrey> I use mongdb 2.6.1 installed from debian package, and did not change any configuration options that are related to log level.
[11:39:23] <kali> rainerfrey: logging connection management is important, as connection can quickly become a scarse resource, so i would not recommend silencing this
[11:40:09] <kali> rainerfrey: as for the replica chatting, i think it is a relatively recent behaviour of 2.6, but i have seen it in working clusters, so i would not worry too much about it
[11:47:52] <rainerfrey> wow ... this is like 900-1000 log entries per hour, on an unused replica set. That is a multitude of the log volume of 6 application servers and micro service web applications together
[11:52:44] <rainerfrey> yes I do worry about 100 log lines per hour of just connection opened and connection closed events in an unused system. And I do worry on how to notice important information among this, especially as mongodb does not tag log entries with any kind of level or priority
[11:54:53] <rainerfrey> BTW can anyone explain the connection (or difference) between the logLevel parameter and the verbosity configuration file option?
[13:35:48] <AlexejK> if my server is under heavy load (and for instance has huge IOWait %), when i have my write concern set to 1 (acknowledged) will that mean that operation will return as soon as a read for that object will not yield in an empty value?. Basically I'm having the problem that even though i have ACKNOWLEDGED set, and read pref to primary on this entity, when my server is overloaded my operation returns but next read for the same object fails (gets no
[13:40:18] <AlexejK> rspijker: I know.. that's why I'm wondering :-/ For me Acknowledged means that this server is ready to respond with it BUT may have not flushed down to disk yet
[13:44:12] <AlexejK> that's whats confusing me and right now keeps me banging my head on the keyboard
[13:44:55] <rspijker> AlexejK: and you are sure the java driver is executing things in the order you expect?
[13:47:27] <AlexejK> it's not async processing in that case.. its rest call that generates the object, saves it, when save operation (non-async) returns, it returns the key (kind of like _id) back to the client. Client in the next call uses the key to make another call, and when we process that call we check for existance of the object with that key in the DB.. that's were only under heavy load of mongo I'm getting no results, a but a sec or 2 later I can get a hit
[13:48:09] <AlexejK> "it returns the key" = "we return the key to the client, key is generated before save operation and is part of the object saved
[13:50:14] <rspijker> AlexejK: ok… not 100% sure about the java driver. But I assume that it blocks until the write is acknowledged if you tell it to
[13:52:17] <AlexejK> rspijker: yes, that's what it says. It basically uses the {w:1, wtimeout:0, fsync: false, j:false } But w:1 should mean that server should return a hit on that one from that moment onward, right?
[13:58:26] <AlexejK> that would mean longer response time on initial call but will offload primary.. but thats another topic :-)
[14:03:22] <Fishy> Is this the correct place to ask mongoose questions? Getting my pastebin ready...
[14:04:43] <rspijker> Fishy: there are usually some people around that use mongoose. So while it’s not the official place (not sure there is one), it’s often a good enough place to ask
[14:07:04] <AlexejK> rspijker: This *may* be Spring-Data issue that we use in some places.. I think it does some mumbo-jumbo with preparting it on some operations... will dig in more
[14:07:38] <rspijker> AlexejK: ok, all I can say is that afaict you are doing everything correctly from a mongo POV
[14:30:08] <AlexejK> rspijker: thanks, that at least eliminates that problem.. So i'll try to figure out if there is something else going on
[14:47:50] <michaelchum> I'm trying to insert over 40M documents with a script in PyMongo but the process slows down and stops at 3M documents
[14:48:27] <michaelchum> I found something weird in my mongod.log: [clientcursormon] mem (MB) res:1426 virt:13045 mapped:6094
[14:49:10] <michaelchum> The virtual memory (virt:) keeps increasing but stops at 13045, mapped stops at 6094. Everytime I try running the script
[14:52:15] <michaelchum> And I tried to erase my db and run the script multiple again, I found the correlation of the virt: mem when the insertion stops
[14:52:52] <michaelchum> htop, doesn't use a lot of RAM only 1gb/4gb, CPU drops when insertion stops
[15:14:42] <modcure> "Tracklist" : [ { "Track" : "1", "Title" : "Smells Like Teen Spirit", "Length" : "5:02" }, { "Track" : "2", "Title" : "In Bloom", "Length" : "4:15" } ] <-- i only want to get back the embeded document that contains "In Bloom" .. db.spreadsheets.find( { "Tracklist.Title" : "In Bloom" }, {Tracklist: 1} ) but its returning all the embedded documents within the array
[15:25:14] <sqlnoob> I'm trying to get number of unique users per month with logged in activity between two dates. more specifically month-wise. I tried db.runCommand() with distinct on ApplicationLog and userId as key with query for activityType and date field but I get 0 as the result. Can anyone think of or suggest me better approach for this? thanks
[15:27:44] <sqlnoob> I feel like aggregation is a good way to go for it. trying to figure out using documentation but no success so far
[15:31:05] <rspijker> modcure: you can do “Traacklist.$”:1, but it will still give you a wrapping array, just with only 1 element
[15:34:33] <rspijker> sqlnoob: not sure what it is you are after exactly… Would the month be an input and the number of unique users during that month the output?
[15:37:35] <sqlnoob> I'm trying aggregate with $match: {date: {$lte: lowerDateValHere, $gte: higherDateValHeree}}
[15:37:52] <mistawright> hi guys i need some help. i need to clear my mogodb as I currently have filled this server up and did not realize it. I am however unable to run the mongod daemon as i am out of space. how can clear these db's so i can resume operation?
[15:38:35] <rspijker> sqlnoob: maybe do a group on the month, then addToSet for userIds and then project a count of that
[15:40:08] <sqlnoob> thanks rspijker. not sure what that exactly means but I'll work on it. need to do quite a reading. thanks
[15:40:12] <rspijker> sqlnoob: {$group: {“_id”:{$month:”$date”}, “uids”:{$addToSet: “$userId”}},{$project:{“logins”:{$size:”$uids”}}}
[15:41:32] <rspijker> in your dbpath there will be a bunch of files or directories (depending on your directoryPerDB setting) which you can simply remove
[15:42:42] <mistawright> rspijker, i have two files there that relate to graylog. do i just delete them and if that is the case does that remove the db as well? I need to continue to aggregate messages/logs
[15:47:13] <rspijker> pd: that’s really strange, since you have very little processes runnable and uninterruptable sleep...
[18:18:52] <BadHorsie> In an aggregate, how can I do something like {$group:{_id:"$field1"+"$field2","maxBlah"... to groug by two fields?
[18:45:09] <michaelchum> Hi, I'm trying to insert 40M documents but at some point, the insert pauses every 5 minute and then stops eventually stops at 3M documents. I've also noticed that mongotop always says 0 in everything during insert, there's 0ms in write and read
[18:59:54] <michaelchum> I don't think so well, it's just a big loop which iterates and every iteration computes a document and inserts it
[19:00:29] <michaelchum> But do you think it's normal that mongotop always says 0 while the db.collection.count() increases?
[19:02:23] <amcrn> general question: is there a downside to defaulting to a sharded deployment with a single shard (a replica-set of three members) vs. using a vanilla replica-set with three members? if you know you're going to have some deployments that absolutely need shards, why not just default to a sharded strategy for all to simplify operational complexity (assuming you're willing to eat the cost of config-servers and query routers)
[19:04:56] <michaelchum> oh ok, I do have 500 inserts per second, going to wait until 3M and see what happens, thanks jzuijlek!
[19:12:53] <amcrn> (i found a mailing list entry @ https://groups.google.com/forum/#!msg/mongodb-user/FyYw2jczyHA/PIkvU8fpwFAJ on 11/20/13 seeming to indicate it's a reasonable idea, but was curious if anything has changed)
[19:26:17] <biggbear> hello folks. iterating over a collection is sending a lot of the same object. I have a lot of results identicaly like this { "_id" : { "$oid" : "4ffb8b1724acb9a87fbcd4b5"}...
[19:26:57] <biggbear> and this log: info DFM::findAll(): extent 2:3d48000 was empty, skipping ahead.
[19:34:53] <insanidade> hi all. what access permission/role should I fix in order to solve the following error when trying to connect to mongodb: Error while trying to show server startup warnings: not authorized on admin to execute command { getLog: "startupWarnings" }
[20:04:21] <BlakeRG> can anyone tell me why i get this error? ReferenceError: birthday is not defined
[20:31:19] <biggbear> what could be the reason for getting lots of documents whit the same "_id"? (info DFM::findAll(): extent 2:3d48000 was empty, skipping ahead.)
[22:21:39] <biggbear> List<DBObject> list=collection.findAll()..toArray(); iterating over this list i'm getting a lot of exactly the same documents (same "_id"). Driven me crazy