Log file Viewer
#mongodb logs for Wednesday the 14th of September, 2016
[07:12:11] <himanshu> Hi, I am new to mongodb. I have saved Object IDs as string without reference, at places where I want to tag. I did it in wrong way. Now I want to convert those string to object ID and reference them to respective Collections. Please help me out.
[07:15:42] <joannac> get the document, modify the document to make the string into a ObjectID, save the new document
[07:17:14] <himanshu> joannac: I see, I can convert these strings to object ids. But then how to add reference back to them ?
[07:24:32] <himanshu> joannac: For example, a list of user id : [ user1_objectID, user2_objectID ]. So now I want to reference these user IDs to User Collection. Isn't this thing called referencing ?
[09:52:30] <hdkiller_> hello! i have a collection with 8000 small documents and i am trying to do around 400-500 qps with queries like this: db.getCollection('pool_ip').find({ $and:[{ long_start:{$lte:3111560450}, long_end:{$gt:3111560450} }]}) and when i start mongodb host start loading. I have an index long_start_1_long_end_1 { "long_start" : 1.0, "long_end" : 1.0 }
[09:55:23] <hdkiller_> the load on the host running mongodb jumps to load average: 190.85, 67.92, 72.44
[09:55:59] <Derick> what does iostat tell you? what's your memory usage vs swap? How much data do you have? There can be so many reasons... it's not a simple answer.
[09:57:40] <hdkiller_> this collection is just 8000 small documents, i have no idea how large the whole collection, but i don't think it's significant to cause any problems. ram 64G/59G used no swapping at all
[10:59:38] <hdkiller_> Derick: 2016-09-14T12:47:10.942+0200 I COMMAND [conn45887] query pool.pool_ip query: { $and: [ { long_start: { $lte: 3211228110 } }, { long_end: { $gte: 3211228110 } } ] } planSummary: IXSCAN { long_start: 1.0, long_end: 1.0 } ntoreturn:0 ntoskip:0 nscanned:6494 nscannedObjects:0 keyUpdates:0 writeConflicts:0 numYields:51 nreturned:0 reslen:20
[10:59:40] <hdkiller_> Derick: locks:{ Global: { acquireCount: { r: 104 } }, MMAPV1Journal: { acquireCount: { r: 52 } }, Database: { acquireCount: { r: 52 } }, Collection: { acquireCount: { R: 52 } } } user_key_comparison_count:0 block_cache_hit_count:0 block_read_count:0 block_read_byte:0 internal_key_skipped_count:0 internal_delete_skipped_count:0 get_from_memtable_count:0 seek_on_memtable_count:0 seek_child_seek_count:0 585ms
[11:50:59] <crazyphil> if I have a sharded 2.6 setup, and I need to turn auth on (mongodb-cr specifically), do I need to set every mongod instance for --auth, or only my config servers, or everything?
[12:10:17] <joannac> https://docs.mongodb.com/v3.2/tutorial/enforce-keyfile-access-control-in-existing-sharded-cluster/
[12:23:13] <admin0> hi all..if i have 3 datacenters for 3 mongodb servers, and need data in all of them, where do I put my arbriter ?
[12:25:36] <admin0> and does all 3 need to be in a layer2 domain or is layer3 (unicast) also works fine ?
[13:17:33] <StephenLynx> what are watches? what kind of field are they? how did you queried for them?
[13:18:20] <greatgg> db.watches.find({type: 'episode', user_id: user._id}, {episode_id: true, last_watch: true}).limit(30).sort({last_watch: -1}, callback
[13:21:53] <StephenLynx> keep in mind that indexes on a collection that large will use up quite some resources.
[13:27:10] <ashp> 2016-09-13T18:17:26.600-0400 [conn2481792] query local.oplog.rs query: { ts: { $gt: Timestamp 1473804205000|273 } } planSummary: COLLSCAN cursorid:1028688818334 ntoreturn:0 ntoskip:0 exhaust:1 nscanned:24368965 nscannedObjects:24368965 keyUpdates:0 numYields:48119 locks(micros) r:96563812 nreturned:101 reslen:411143 439730ms
[13:28:29] <StephenLynx> when a document changes size too often, indexes must be rebuilt, if I am not mistaken.
[13:29:56] <StephenLynx> the logic is very simple: will this index impact performance in a positive manner and I got resources to keep it? then you use.
[13:39:50] <StephenLynx> on compound indexes I THINK it might matter if they are both the same or not.
[13:52:08] <Derick> but it's not for storing "BSON documents" that you would store in a collection, for that to work, you need to serialize to a string
[13:53:09] <Derick> Pinkamena_D: in any case, if your normal documents are that big, you probably should rethink your schema
[13:54:04] <Pinkamena_D> I dont need to handle it like a document (while it is stored), I just thought the serialization would reduce preformance slightly
[13:54:42] <StephenLynx> if you don't have to handle it unlike any other binary blob, then yes, you can use it.
[13:54:52] <Pinkamena_D> most documents are not that large, this piece I am working on would act as a cache in case we were not able to connect to the actual in-memory cache
[16:08:32] <Naeblis> Is there any way to query for a specific value that occurs n times in an array?
[16:09:30] <Naeblis> or well, if there are duplicates, only take the most frequent or first value etc. But need to find them first
[17:35:07] <deathanchor> mongo3.2: what's the shell command for allowing secondary? so I can run show dbs.
[17:49:59] <bros> Did not find local voted for document at startup; NoMatchingDocument: Did not find replica set lastVote document in local.replset.election
[17:50:02] <bros> Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset
[17:51:52] <bros> 2016-09-14T13:41:30.791-0400 I REPL [ReplicationExecutor] Error in heartbeat request to 10.8.0.10:27017; ExceededTimeLimit: Operation timed out
[17:56:28] <bros> Derick: because I am trying to avoid having my production database publicly accessible
[18:05:04] <StephenLynx> bros configure the firewall so only your application servers will be allowed by the OS to connect to the db.
[18:17:56] <Derick> can you make it so that the node you're connected to can talk to *one* of the 10.8.x nodes?
[18:18:38] <Derick> because with 4 nodes, and two nodes not working, the 2 left over nodes can not form a majority, and hence not have a primary
[18:19:44] <Derick> bros: your fuckups are not my responsibility, so play nice when people try to help
[18:20:04] <Derick> I think your best bet is: change the connection string to only have the 10.128.15.141 node in it
[18:21:33] <Derick> if you have done that, you can empty the local.system.replset collection - and then start over again.
[18:27:26] <Derick> you would have found about the problem, and because 2 is more than the majority of 3, it would have worked
[18:33:04] <pihpah> I have to insert about of 700 nested documents and it takes about 2 minutes. With every insert the insert time is getting worse, and that's really sad.
[18:53:25] <AlmightyOatmeal> I'm running a seemingly expensive query, is there any way to determine exactly what it's doing? Also, is there a way to make mongod utilize multiple processors? I'm currently running a query against a collection within a DB but it seems to be sticking to only one core :(
[18:55:08] <synthmeat> hey, is there some kind of mini mongo i could run? i like mongo, and i'd like to have it on really small instance (512MB) with possibly other stuff alongside it
[18:56:37] <synthmeat> hah. there actually is mini mongo, but that's something from meteor for clients. prolly single rw
[18:58:23] <synthmeat> totally another db with exact same surface (query language, i guess) would do too
[19:04:26] <AndrewYoung> AlmightyOatmeal: You can use the query analyzer. From the mongo shell you can run "var explain = db.foo.explain()" and then do things like "explain.find({ /* query */})"
[19:05:28] <AndrewYoung> The best way to make mongo use more than one processor for a single query is to implement micro-sharding, where you split your data across multiple mongod instances.
[19:05:51] <AndrewYoung> However, it's complicated and there can be serious problems with it if you machine isn't well provisioned.
[19:06:12] <AndrewYoung> It has some very specific use cases where it works well, but it's not generally used.
[19:06:40] <AndrewYoung> You would want to run the shards in some sort of container so that they are memory limited, etc.
[19:07:36] <AlmightyOatmeal> AndrewYoung: that's one complaint that i have is that i can't seem to limit resource utilization of mongod which has caused OOM issues
[19:07:48] <AndrewYoung> Then you make the query to a mongos and it does a scatter/gather operation on the shards, splitting the work up amongst the shards.
[19:09:10] <AndrewYoung> MongoDB will give either 60% of available RAM on the box - 1GB (it looks in /proc to find that info, I believe) or 1GB, whichever is larger.
[19:09:48] <AndrewYoung> Also, swap is important for WiredTiger, whereas it was not as important for MMAPv1.
[19:12:47] <AndrewYoung> synthmeat: You might try running it with the MMAPv1 storage engine instead of WiredTiger. You'll lose the benefits of WiredTiger though.
[19:13:20] <AndrewYoung> Under provisioning can be a problem though. If your "working set" (the combination of indexes and commonly used data) doesn't fit in memory it can cause performance issues.
[19:14:20] <AlmightyOatmeal> FWIW, i've played with a number of WiredTiger cache sizes and have tried to slowly decrease trying to prevent OOM. i tried limiting the sources granted to the process but that really upset mongod
[19:19:04] <AndrewYoung> OOM killer is worse under WiredTiger than MMAPv1 because when the kernel is running low on memory under MMAPv1 it just flushes stuff back to disk because it knows it can do that, but with WiredTiger it has to move stuff to swap. If it can't do that then OOM killer will kick in.
[19:22:18] <zylo4747> if i'm going to create a hidden replica set member with priority 0, can i put those options in the config file or do i only define them when adding the member
[19:23:04] <AlmightyOatmeal> AndrewYoung: i'm not running Linux so OOM Killer isn't a problem, however limiting available resources outside of mongod gave me http://pastebin.com/QVnbQbKc and to answer the swap question: http://pastebin.com/xDE8HFb8 but i try to avoid the extra disk I/O for swapping if at all possible as i would rather save that for actual data I/O
[19:24:32] <AlmightyOatmeal> i really hate to say that CouchBase was pretty sweet in terms of clustering and resource limiting/kindness but MongoDB is easier, faster, and the filesystem footprint is the smallest of any document stores i've tested
[19:25:02] <AlmightyOatmeal> but it seems resource control and ease of clustering has a long way to go :(
[19:26:25] <AlmightyOatmeal> even if it has the "whole box", that means it can choke off other essential system services as well which is not a good thing if you want to guarantee Y percent of Z resources for system management :(
[19:27:47] <AndrewYoung> Well, I meant no other major services running on the box. That's why it assigns 60% to the cache.
[19:29:56] <AlmightyOatmeal> AndrewYoung: in a jail, no, but i created a custom login class for the mongodb user running mongod (which is where i received the OOM)
[19:30:14] <AndrewYoung> synthmeat: Yes, it still requires a good bit of memory, but because of how it works (using the system mmap calls) it plays nicer in small memory environments I think.
[19:44:44] <synthmeat> AndrewYoung: i think better solution would be to set up one large instance that serves multiple projects, actually
[19:45:32] <AndrewYoung> Yes, that is better, but I thought you were asking specifically about using a small instance. :)
[20:38:36] <combat> Guuyyyys what's that mean? -> MongoError: Syntax error, incorrect syntax near '48' ? any tips?
[20:58:32] <combat> idk what's '48' I'm running tests in nodebb connected to mongodb on Azure servers (DocumentDB service)