Log file Viewer
#mongodb logs for Friday the 24th of January, 2014
[01:18:00] <bmw0679> I've run into a write lock problem because of lack of disk space. This prevents me from running db.dropDatabase() which would solve my disk space problem. If I remove the database files manually from the file system would this cause a problem?
[02:29:22] <darius93> if you want to store the db in a ramdisk you can easily but you must make backups to prevent data lost
[02:29:45] <Semor> In my application ,I save my data by executing SQL sentences , is mongodb advanced than this ?
[02:31:18] <Semor> I save my data in ram by c++ struct ,then in another db thread execute sql transections to do persistence
[02:34:12] <darius93> possibly dump it to disk and clean it out to of memory and call on it when needed
[02:35:20] <darius93> well you can go on and try mongo out if you want. check out that link to see the comparison between the two
[02:35:49] <darius93> but i wouldnt recommend saving it every second unless your server is that fast
[02:41:53] <darius93> well i havent used mongo in C/C++ yet but overall e.g INSERT INTO EXAMPLE ('section') values ('data'); would be like doing db.example.insert({section:'data'});
[03:03:36] <Semor> darius93:I am building a network server ,receiving client's operations and calculating results , then saving to db
[03:14:58] <darius93> Any database can be fast, its just how you use it to make it perform better. I think MongoDB may be worth a try then, or MariaDB
[04:50:11] <SlexAxton> hi, if I have a sparse index on two properties, and one is nil, and the other exists, does it still get indexed?
[06:42:53] <b0ss_> so mongodb does not implement a many-to-many relationship in something similar to the JOIN TABLES, right? It just place the other object's ID onto the required object ?
[06:46:41] <b0ss_> However, Managing these relationships is the difficult part. Say, if you delete one side you have to manually update the other side. Right ?
[07:39:52] <omid8bimo> i need some guidance on something. i have 3 mongodb servers in my replica set, 1 master mongodb, 1 slave mongodb and one as arbiter; my slave server crashed yerterday and after lots of file system repair, server become ok but mongodb couldnt start, so i removed the /var/lib/mongodb, created an empty directory for it and started the service to get the data back from master, but only half of the data got
[08:48:40] <omid8bimo> first of all, how can i know how much data has been saved in oplog? so that i can understand when i will start overwriting
[08:49:16] <omid8bimo> second of all, why mongodb would not replicate the whole dataset when a fresh mongo instance joined the replicaset?
[08:55:49] <Gargoyle> omid8bimo: I can't think of any reason why under a normal configuration the whole data would not be sync'd. Are there any errors in the logs?
[08:56:10] <Gargoyle> Dunno about checking the oplog size, but don't think that would make any difference ether.
[08:57:20] <omid8bimo> Gargoyle: no errors. db.printSlaveReplicationInfo() on master said secondary is 20 hours behind
[08:59:34] <omid8bimo> Gargoyle: master db db.printSlaveReplicationInfo() says syncedTo: 1 seconds ago, but data on secondary that came up like 1 hour ago is still 30 GB behind master!
[09:10:58] <kali> omid8bimo: have you checked the database from inside, or just look at the file size ?
[09:12:26] <omid8bimo> becase of two factors, one is that total size of /var/lib/mongodb is different on both servers. (secondary is aroudn 20GB less) and also, based on my collections, i have kcollection.48 on master but only kcollection.43 on secondary
[09:13:27] <Gargoyle> omid8bimo: Don't think you can count disk space size. Your master will probably have "holes" in it's data files, your secondary will be nice and neat to start with.
[09:14:16] <omid8bimo> Gargoyle: so to be sure that my data on secondary is up to date, can i do this?
[09:15:16] <omid8bimo> start mongod on secondary with replicaSet config and wait until it sync the entire /var/lib/mongodb on master which is 180 GB
[09:15:44] <kali> omid8bimo: yes. just don't expect the size on disk to be similar, it has no reason to be
[09:17:04] <omid8bimo> like if i have up to kcollection.48 on master, i must have kcollection.48 on secondary as well?
[09:18:37] <Gargoyle> omid8bimo: Perhaps you should stop looking at the files and use the shell to check things and leave mongo to "do its thing" with your files!
[09:23:17] <kali> but bottom line is, the procedure you followed is very really robust (as long as the oplog is deep enough, and it sounds like it is)
[09:23:55] <omid8bimo> kali: ok, here is my rs.status() output on secondary http://paste.debian.net/78117/
[09:25:41] <omid8bimo> one more thing, how much approximately does it take to do a initial sync on a LAN connection for 180 GB data?
[09:29:28] <omid8bimo> so im guessing several days to do a initial sync of 180GB data from master to a new secondary
[09:31:29] <Gargoyle> So. do 180GB * 8 = 1440Gbits/sec * 1024 = 1,474,560Mbits / 750mbps = 1,966 seconds = about 30 mins.
[09:35:22] <Gargoyle> Other factors will make a difference, disk speed, other network traffic, etc. But theiretically, your two machines should be about to shove 180GB down the wire in about 30 mins (theoretical max).
[09:36:41] <omid8bimo> ok i wanna do something for test. i just stopped secondary, renamed /var/lib/mongodb, created an empty folder with permissions set, started and now it start syncing data from master
[09:38:39] <omid8bimo> even i tried grep on mongodb.log for ip address of master, and im getting liek 10 connections per 5 mins
[09:39:54] <Nodex> if the replication has high traffic there will be a lock while indexes are built stopping new data arriving which will resync once the index has built
[09:41:40] <omid8bimo> ok fair enough. i have MMS service configured on my servers. which graph on secondary host i should pey attention for the progress of replication/sync?
[10:29:59] <tiller> Is there a way to check it -a field- has a specific value? A field being: "It doesn't matter which one".
[10:31:05] <tiller> For example, if I have: {_id: 1, subDocumentToTest: [ {a: "test"}, {b: "thing"}, {a: "hello", c: "test"} ] }
[12:23:43] <byalaga> I mean, we need to take backup from a secondary node and we want to ensure the oplog on the primary node is not rotated/slided?
[12:23:44] <byalaga> so that whenever we bring this secondary node.. it should catch up with the primary
[12:25:02] <kali> byalaga: db.printReplicationInfo() on the primary will tell you what are the first and last event of your oplog
[12:26:51] <gnagno> is it possible to create a replicaset using diffent mongodb versions? say I have a 1.8.1 and 2.4.x ?
[12:28:24] <kali> gnagno: heteogeneous replica sets are meant for version migration only. they will work for some combination but it is not recommended, and not supported as far as i know
[12:30:29] <byalaga> how can we say that?? the host may have 5operation in 1 sec sometimes.. and 50operations sometimes..
[12:30:53] <gnagno> kali, thank you, actually version migration is what I need, so you suggest me to migrate first from 1.8 to 2.0 then from 2.0 to 2.4 ?
[12:31:55] <kali> gnagno: look at the release notes for the release branch (2.0, 2.2, and 2.4). they will tell you for sure which one can be skipped. Don't forget 2.2
[12:32:26] <byalaga> kali: how can we say that exactly?? the host may have 5operation in 1 sec sometimes.. and 50operations sometimes..
[12:34:47] <kali> byalaga: it's just based on what happened in the past. it holds 62 hours of backlog, but if your load change, it will obviously cover less time
[12:36:43] <byalaga> kali: my oplog size on all nodes is 2048MB. for how log can i keep my secondary node down for backups? so that.. if i bring it back it should be able to catch up with primary?
[12:39:57] <byalaga> "oplog first event time" on primary is not changing.. does that mean.. oplog is not sliding now?
[12:41:45] <kali> that's a bit weird... unless it is actually the beginning of the existence of the primary and every write since is fitting in the oplog
[12:44:48] <_pash> hello, how long would it take to find a match of 12000 entries in a db and whats the best way to do that?
[12:48:08] <kali> count is the count of documents, size and storageSize are quite close... so i'm not sure
[12:48:29] <gnagno> quick question: assume that I have a mongo server with a lot of data in it already running, if I create a replica set will it automatically migrate all the data also?
[12:49:58] <kali> gnagno: it's a relatively painless and well documented procedure: http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/
[12:53:33] <_pash> how long would it take to run a search through 12000 entries and mongo and whats the best way of finding a match?
[13:01:57] <_pash> Nodex: it would literally be like 12000 entries of usernames that still have not registered, and as soon as the user types that username, i need to check whether it is or not registered
[13:03:25] <kali> less than a tenth of milliseconds if you have an index on the username and it fits in RAM
[15:42:20] <trueneu> Hi. Is it possible that PRIMARY and SECONDARY servers in the same replica set have different collections in the same db?
[15:42:52] <trueneu> No replication lag whatsoever. I did an initial sync for the SECONDARY server and it doesn't seem to be synced.
[15:49:54] <joannac> I would say more likely you hit some problem, possibly corruption, or maybe you're not really up to date
[15:51:39] <trueneu> I actually have 3 servers in this repl set, PRIMARY and one SECONDARY are 2.0.smth, and the second SECONDARY is 2.4.smth. Both secondaries have the same data sets
[16:17:31] <joannac> trueneu: erm, both secondaries have the same data and both were synced from scratch from the primary?
[16:18:25] <trueneu> I've checked the other secondary, the one I did initial sync for -- it synced from the second secondary :)
[16:19:12] <tiller> joannac> I think it won't work. Because if a subscription already exists for the (user, entity) but without the new tag I want to insert, the search query will fail, and I'll upsert a whole new thing. Don't I?
[16:19:23] <trueneu> I still do not understand why the replication works, but I guess I will just bring down both secondaries and do 2 initial syncs.
[17:12:39] <harrymoore> hi, all. can someone tell me if there is way to use $pull to remove a single matching array element? it looks like it pulls all matching elements
[17:22:42] <Joeskyyy> harrymoore: Could probably use the $ position operator, assuming you're wanting to remove the first occurence
[17:29:18] <Joeskyyy> even with that though, it'd be setting it to a new value :\ there's no way to pop it out from the looks of it
[17:30:05] <Joeskyyy> Otherwise you may want to have an array that each element has something like an id of sorts, and the content.
[17:30:48] <Joeskyyy> i.e. if it were an array of comments, each new comment has the comment field, and the id field (incremented by one)
[17:35:06] <harrymoore> thx @joeskyyy! i will just have to handle it in app code (java) instead: read the doc, remove the element in question and $set the entire array
[17:44:44] <NyB> Hi, does anyone have any tips for speeding up the Java driver? It seems that it has become the bottleneck in my application...
[17:49:16] <Joeskyyy> maybe harrymoore has something on that :D he was just talking about the java driver
[17:52:32] <NyB> I am thinking about using two or more driver threads in parallel, but I was hoping to avoid the complexity involved if there is a more standard way.
[17:53:23] <harrymoore> @Ny8: you can distribute your read requests if you have a replica set. the java driver is smart enough to read from the fastest responding node
[17:53:40] <NyB> as it is, my DB input thread seems to be spending 70% of its time in BasicBSONDecode.decode() :-/
[17:54:37] <NyB> harrymoore: the server is not the problem - the limiting factor seems to be the fact that I am reading a collection from a single thread
[17:56:33] <NyB> I believe I cannot use the result of find() safely from parallel threads, right? I could not find any relevant synchronization calls in the source coe...
[17:56:34] <harrymoore> i haven't done any multi-threaded work. assume the drive is ok with it though
[18:01:09] <kali> there is one thing to be aware of, also. if your database is accessed by real time stuff (like a web app) at some point you'll manage to move the bottleneck to mongodb, and you web app will die
[18:02:55] <kali> then you'll need to reorder them once decoded before pushing them through the main thing
[18:04:00] <NyB> kali: hmm... are you suggesting fetching them out-of order from several threads and then sorting them?
[18:06:05] <kali> well whatever you do, if you decode them in parallel, you'll get them in disorder, at least locally
[18:08:43] <NyB> kali: I guess part of my problem is that the bottleneck does not lie on the MongoDB server - if it did I would have a couple of ways to handle it... as it is mongod consumes about 30% of a single CPU core and the I/O is nowhere near saturation yet...
[18:17:47] <NyB> kali: is there a way to have the server add an "order" field automatically after sorting? something that I could use with $mod to split the output?
[19:28:04] <ekristen> looking for some advice for hosting mongodb in aws, I’ve got in development 400-500k documents right now, looking to be in the 10s of millions in a few months
[19:31:52] <harrymoore> @NyB if you're still there here is a gist showing threaded access to the driver. I don't write threaded code so take it for what it's worth: https://gist.github.com/harrymoore/8604212#file-app-java you could add a synchronized data structure for defining which thread will process which records
[20:00:39] <whaley> I missed something... what's wrong with just setting connectionsPerHost on MongoClient?
[20:46:04] <module000> ekristen: late reply...but i host in AWS with mongo. you need to look at the size of your data sets, and pick accordingly. for example, i have a 60GB data set, so i host it on a m2.4xlarge, which has 68GB of memory. if performance is less important, scale down your instance to include less RAM
[20:46:59] <ekristen> module000: my data set is currenly only like 6gb on disk, thats roughly 1GB per 100k documents we are storing at the moment
[20:47:45] <module000> ekristen: it's sharded on a m1.large, but that shard only exists to mitigate an outage if the primary shard fails
[20:48:21] <module000> ekristen: if you only have 6gb,, you could use a m3.large (7.5GB memory), and make a much smaller instance to shard it with, so if you have an outage your tiny node keeps you from having an outage (albeit you move slower)
[20:50:32] <ekristen> maybe I misunderstand the term shard, as what you are saying doesn’t line up with my understanding of replicasets and sharding
[20:51:28] <module000> ekristen: you could do the same with replica sets though, just elect a smaller instance to serve as your secondary
[21:06:55] <luimemee> if have a collections and each elements have a tags list. how can i have a list of all the tags on all the element please ?
[21:31:58] <beardage> hi, how can I tell mongodb in python not to index my data im importing. Thing is that it's OOM while importing millions of records.
[21:46:17] <luimemee> this should work ? db.runCommand ( { distinct: "mycollection", key: "tags", query: {'tags':{$regex:'*toto*'}}} )
[22:27:22] <the-erm> I need some advice, is there a better way to do this? http://pastie.org/8664985 I'm trying to store the hours of business so it can be looked up, but the challenge is hours of operation are from 12pm-2am the next day.
[22:31:16] <the-erm> I was thinking I could possibly store the values 'Monday': [ [0,1], [1200,2359] ] but I'm not sure how to write a query for that.
[23:41:40] <JeffC_NN> unholycrab: If you're comfortable editing the PHP, edit app/controllers/db.php and remove the links/capabilities
[23:43:57] <JeffC_NN> (for very easy edits, just search the file for "drop" and add a simple "return;" at the beginning of functions named things like doDropDatabase() (line 488) and doDropDbCollections() (line 451) or whatever you want to disable. Pretty easy.