PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 4th of November, 2013

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[00:02:25] <Djim> jblack i did one simple thing, i compiled my single.cpp file with -std=c++11 flag and without it. Without that flag it compiles ok
[00:03:22] <Djim> i think that my workaround will be compiling that single.cpp without c++ 11 support while whole the rest of code will be C++ 11 enabled
[00:46:39] <jblack> cool!
[00:46:57] <jblack> I was actually playing with the idea of suggesting you just use the c library, but that seemed... evil.
[01:04:39] <_Heisenberg_> Hello guys. I'm not familar to JIRA so can someone acknowledge that there is no release of mongodb now, which fixes this bug: https://jira.mongodb.org/browse/SERVER-7246
[02:15:37] <george2> I'm using a mongo database to store documents containing a userID, along with some other information, and I need to find the top N userIDs. Is there an easy way to do this?
[02:17:15] <george2> Or even better, the top N% of userIDs by occurance.
[02:24:42] <joannac> aggregation framework for the first.
[02:25:09] <joannac> For the second, you would probably need to post-process
[02:25:25] <joannac> Aggregation framework to count, then peel off the top N%
[02:28:37] <george2> ok, I'm reading the aggregation framework docs now
[02:49:00] <Goopyo> joannac: who maintains the pymongo client? want their thoughts on this: https://github.com/pnegahdar/MongoRedis
[03:21:44] <Steve009> do all dates in Mongo get automatically get converted to uTC?
[03:24:45] <cheeser> dates are stored internally in UTC, yes.
[03:25:01] <cheeser> i don't know that there's a way to coerce it.
[03:26:21] <Steve009> so when you bring back date times is there timezone support?
[03:26:26] <Steve009> been looking but docs are unclear
[03:30:38] <Steve009> I am using the Agg framework
[03:30:57] <Steve009> and when running a query i would want dates to come back in the correct timezone
[03:39:38] <Steve009> Timezone support seems to be a ongoing problem.....https://jira.mongodb.org/browse/SERVER-6310
[04:02:30] <joannac> Steve009: That ticket goes through some possible workarounds -- are any of them suitable for you?
[04:03:01] <Steve009> They are okay. But most are a big annoyance
[04:03:21] <Steve009> I am looking at this at the moment: http://stackoverflow.com/a/18854229/2801327
[04:03:27] <Steve009> but i have concerns about performance
[04:05:10] <Steve009> It means that the pipeline would be modifying a lot of dates
[04:12:40] <Steve009> Anyone know if 2.6 is on schedule for end of year release?
[04:19:20] <fg3> is there a way to display docs with the properties sorted?
[04:19:41] <joannac> fg3: you mean the fields?
[04:19:52] <fg3> ok fields
[04:20:53] <joannac> No. JSON is a set of field/value pairs. No order is specified.
[04:21:14] <fg3> I know what json is but mongo is not json
[04:21:31] <fg3> mongo is an app that uses json
[04:22:02] <fg3> so, isn't it possible to write a simple function to do this?
[04:22:20] <jblack> I suppose you could convert a response to a hash, and sort the hash in the language that you're using.
[04:22:24] <Steve009> Whats your use case?
[04:22:52] <joannac> Okay, let me step back. If you have a document like {"a": 1, "c":3, "b":5} -- you want output like {"a":1, "b":5, "c":3} -- is that correct?
[04:23:08] <fg3> Steve009: my use case is that I'm in a terminal comparing records and it's difficult if they are in random order
[04:23:27] <fg3> joannac: yes
[04:24:01] <Steve009> fg3: can you only work with the mongo shell?
[04:24:28] <fg3> Steve009: Is it better to work with something else?
[04:24:51] <Steve009> you could write a simple app to do this with something like Ruby
[04:24:52] <jblack> if you do it in ruby, you can easily convert the result into a sorted hash.
[04:25:05] <Steve009> ya was just about to say the same thing
[04:25:32] <fg3> mongo is js based isn't it
[04:25:42] <fg3> and js is capable also correct?
[04:25:57] <jblack> it has javascript embedded, yeah, and you can write and store functions that would probably do what you want.
[04:26:16] <fg3> I was hoping they already had these functions made
[04:26:32] <jblack> Storing functions within the db is generally avoided though.
[04:26:37] <Steve009> you could make this in ruby with about 6 lines of code
[04:27:08] <fg3> you can do this on the commad line with one line of code
[04:27:14] <fg3> err command
[04:27:36] <jblack> What's your goal in comparing? Equivilance? Perhaps you could write a query that does what you want instead?
[04:27:56] <jblack> it almost sounds as if you're attepting to comparing A to B in bash.
[04:28:02] <fg3> the goal is simply to look at records
[04:28:08] <fg3> something that is done often
[04:28:14] <jblack> so presentation?
[04:28:35] <fg3> easily look at docs
[04:29:12] <fg3> if you have a doc with 100 fields, it could get crazy
[04:29:33] <fg3> I know you can filter the fields shown - but if you want to look at the whole thing
[04:29:34] <jblack> I would probably just select the ones I wanted.
[04:30:05] <jblack> yeah.
[04:30:35] <jblack> How about using irb?
[04:30:52] <fg3> what's irb?
[04:31:40] <fg3> I guess I'll have to hunt around for the right tool
[04:31:43] <jblack> irb is the interactive ruby debugger. It allows you to run ruby commands by hand. In the same way that everything was a basic command.
[04:31:45] <Steve009> here is some code you could use
[04:31:51] <Steve009> def simpleHashSort (hash)
[04:31:52] <Steve009> sortedDates = hash.keys.sort
[04:31:54] <Steve009> sortedHash = {}
[04:31:55] <Steve009> sortedDates.each do |y|
[04:31:57] <Steve009> sortedHash[y] = hash[y]
[04:31:58] <Steve009> end
[04:31:59] <Steve009> #puts sortedHash
[04:32:00] <Steve009> return sortedHash
[04:32:01] <Steve009> end
[04:32:12] <joannac> Steve009: next time please pastebin?
[04:32:28] <fg3> thanks everybody for the feedback
[04:32:29] <Steve009> was blocked atm. give me a mine to activate it
[04:32:56] <Steve009> http://pastebin.com/ZPs3ra4X
[04:33:21] <Steve009> In this example it uses a hash that would have Dates as Keys and Numbers as values
[04:33:28] <Steve009> but the dates are out of order
[04:33:34] <Steve009> and you need them in order
[04:34:28] <Steve009> So all you need to do is convert your json to a hash and run the method
[12:27:11] <st0ne2thedge> Say guys, I noticed how our mongodb is located on a fairly small partition, it still has about 40% space left though, but is there any way to estimate when it will allocated the next datafile (2G)?
[12:48:03] <st0ne2thedge> really new to mongodb btw
[12:49:19] <Goopyo> st0ne2thedge: I'm not sure you can do that, but if its a concern you can run it in --compact mode so the allocated space isnt that big
[12:51:39] <st0ne2thedge> Goopyo: thank you, Is there a way to see how full the allocated datafile is then? that would be awesome
[12:53:13] <Goopyo> st0ne2thedge: its part of collection stats
[12:53:28] <Goopyo> st0ne2thedge: http://docs.mongodb.org/manual/reference/command/collStats/
[12:53:48] <Goopyo> size = colleciton size, storageSize = allocated size
[12:54:51] <st0ne2thedge> Goopyo: Thx! I'll look into those
[13:11:39] <st0ne2thedge> ns
[13:19:17] <st0ne2thedge> Goopyo: So if I understand correctly, every collection gets a set of dedicated allocated files?
[13:19:46] <Goopyo> correct they in your db folder labled as <col_name>.N
[13:24:55] <Derick> no, every database has <dbname>.N filenames
[13:25:10] <Derick> not per collection.
[14:06:25] <st0ne2thedge> Derick: but it seems I can only get the 'size' and 'storageSize' variables on collections though?
[14:30:41] <franck34> hi all, can i use a field named "type" in a collections ?
[14:56:43] <IrishGringo> can someone look at this... give me a hint please... http://stackoverflow.com/questions/19770389/how-to-set-a-mongodb-cursor-in-node-js
[14:57:08] <IrishGringo> can someone look at this... give me a hint please... http://stackoverflow.com/questions/19770389/how-to-set-a-mongodb-cursor-in-node-js
[15:03:59] <IrishGringo> no one is home today?
[15:43:03] <NodeX> IrishGringo : you can't do a set operation on a cursor
[15:43:20] <NodeX> a cusor just itterates, you need to perform an update
[15:43:42] <IrishGringo> NodeX: yea... I noticed that... I was just trying a findAndModify...
[15:43:58] <IrishGringo> a couple seconds ago... no errors.
[15:44:24] <NodeX> yu can do that but you cn also just issue an update - no differnce in your use case except one is atomic
[15:45:09] <IrishGringo> it would be intresting to see how the update would be done...
[15:45:23] <IrishGringo> but It looks like I got the findandupdate working....
[15:45:32] <NodeX> findandmodify
[15:45:52] <IrishGringo> yea.... findandmodify....
[15:46:05] <IrishGringo> I am looking at data... looks like it did work
[15:46:12] <IrishGringo> it
[15:51:28] <dillon_> hi all, I'm new to mongo and have a quick question. I have some document objects with attributes like { startX: 5, width: 10 }, but I'd like to change that to { x1: 5, x2: 15 }, where x2 is startX + width. I feel like I'm missing something very obvious here. -_- How would I do this? o__O
[15:53:28] <dillon_> I know I'll have to iterate over the cursor containing all the documents requiring updating, but I'm not sure if there's a best-practice to use
[15:53:44] <dillon_> ^ "iterate" might be the wrong word :)
[16:23:39] <Mastine> Hello!
[16:52:33] <locojay> hi anyone using mongodb driver for erlang?
[17:34:26] <IrishGringo> is anyone doing M101JS class... working on HW 2.3
[17:34:28] <IrishGringo> ?
[18:44:23] <jkitchen> I have a question about mongodb's delayed replica set member functionality.
[18:45:17] <jkitchen> I am using a nagios check to check the replication delays, and this check is smart about intentionally delayed replication, yet it's still complaining a lot, it seems that the delayed replication fluctuates a bit in how far behind it is sometimes?
[18:46:11] <jkitchen> is it just doing lazy updating sort of thing or is a lack of write activity on the cluster potentially causing the delay to seem to fluctuate a bit?
[18:49:41] <jkitchen> aha!
[18:49:45] <jkitchen> I see what the problem is :P
[18:49:56] <jkitchen> the warning/critical interval is in *seconds*
[18:50:06] <jkitchen> and I'm specifying 15/30
[18:50:16] <jkitchen> thank you rubber ducks :)
[18:50:37] <cheeser> nicely done, everyone!
[19:24:26] <cirwin> Hey guys, I'm trying to work out the datasize of my chunks. I've found the datasize command, but I don't know how to use it with a hashed shard key
[19:24:32] <cirwin> any pointers or ideas?
[19:29:13] <andrew91_> i'm trying to use mongodb in an ecosystem of .NET developers and they all hate it. mainly because of this reddit post: http://www.reddit.com/r/programming/comments/1ouiml/the_genius_and_folly_of_mongodb/
[19:29:18] <andrew91_> how can i convince them otherwise? :D
[19:31:52] <kali> andrew91_: http://www.indeed.com/jobtrends/Mongodb.html maybe ?
[19:32:14] <andrew91_> hehe
[19:32:39] <kali> what do you expect from people sold to M$, anyway ? :)
[19:33:55] <jblack> That particular issue seems to come up a lot.
[19:36:30] <kali> to be fair, it's much more easier to start and like mongodb coming from a dynamic language
[19:36:34] <jblack> I love mongodb and use it for a variety of projects. The unsafe operations is a valid point though.
[19:37:33] <kali> which was adressed...
[19:41:15] <jblack> I think the underlying concern for newer people is that it's awfully high risk for a default setting for a data storage tool. sure, it's easily reconfigured. It's definitely out there as a gotcha, and I think it leaves people with a taste of "what other dangerous corners did they cut off?"
[19:42:18] <andrew91_> one of the major concerns is data loss, but i think with the new release of mongo, it should be fixed...
[19:42:25] <jblack> Sure, should people rely on tools they don't understand very well in production? Probably not. Many do, though, and when they get bit due to their inexperience, they can get vocal about it.
[19:45:24] <jblack> Oh wait, safe writes is the default these days?
[19:48:02] <kali> jblack: it's been for one year
[19:49:34] <kali> nearly. http://blog.mongodb.org/post/36666163412/introducing-mongoclient
[19:51:43] <jblack> I completely missed that. Oddly, I was doing most of my mongo work about 6 months ago, and read up on it quite a bit.
[19:51:55] <jblack> Down with google populism!
[21:42:29] <cirwin> is there any way to get the hash of an object id that will be used in a sharded index?
[21:42:51] <cirwin> I'm trying to map between a list of jumbo chunks and the data they contain
[21:43:05] <saml> i don't get it
[21:43:26] <saml> jumbo chunks is mongodb tech term?
[21:43:33] <cirwin> yup
[21:43:47] <cirwin> chunks that cannot be split
[21:43:49] <saml> out of my league :P
[21:43:54] <cirwin> and that are too large to rebalance
[21:43:55] <cirwin> :)
[22:23:28] <angasulino_> say I have a date stored as a string in format yyyy-mm-dd and I want to set yyyy to be current_year + 1 on every document for which yyyy-mm-dd is less than the current date, can I do this in the server? maybe with a stored js function?
[22:23:49] <Derick> you need to iterate over all the documents in your client
[22:28:11] <angasulino_> hmm, at least one SO post says it's a really bad idea to use stored js anyway
[22:32:48] <tripflex> so I have a group of objects in collection for promo codes, what would be the best way to go through all the objects and set the oldest promo code to expired? Each object for promo code has a "DateCreated: EPOCHTIME" field, i can get the oldest one through find no problem, but want to set the oldest one "Expired: true"
[22:33:42] <tripflex> so as example, { "Package": "Package1", "DateCreated": "1383604334", "Expired": false }
[22:33:58] <Derick> "DateCreated": "1383604334"
[22:34:00] <tripflex> but there could be like 5 entries with Package1
[22:34:03] <Derick> make sure you store it as a number
[22:34:07] <tripflex> yeah sorry it is
[22:34:08] <Derick> and not as a string
[22:34:14] <tripflex> just did that as example
[22:34:14] <cheeser> findAndModify() ?
[22:37:35] <tripflex> ah okay
[22:37:43] <tripflex> and i can just sort by the package
[22:37:58] <tripflex> so sort by package and datecreated
[22:38:08] <tripflex> to get package1 and oldest date
[22:38:15] <tripflex> then package2 and oldest date
[22:38:16] <tripflex> etc
[23:05:33] <pasky> Hi! Is there a way to match documents that contain a null item in a certain array attribute? (Actually, all items in the array are null and the other documents contain no null elements in their array.)
[23:13:52] <eyda|mon> What's the standard way of doing backups for mongodb data? Is it done at all? IOW do extra shards make it redundant?
[23:15:02] <tripflex> mongoexport?
[23:15:38] <bjori> do not use mongoexport for backup
[23:15:48] <bjori> and please be careful with mongodump with shards
[23:15:50] <eyda|mon> nevermind, I have the mongodb in action book by kyle
[23:15:54] <eyda|mon> I forgot I had it. just reading that now
[23:17:18] <eyda|mon> Is there a standard for the backup frequency? the book doesn't mention
[23:19:22] <eyda|mon> once a day?
[23:23:02] <azathoth99> are basic mongo files binary ?
[23:23:37] <bjori> azathoth99: yes
[23:23:47] <bjori> well.. depending what you mean with "basic mongo files"...
[23:23:54] <bjori> but the most likely answer will be yes :)
[23:24:37] <eyda|mon> bjori: what do you mean to be careful with mongodump with shards? The book says to choose a secondary one, to lock it, then do the dump. Do I need to do anything else?
[23:25:14] <bjori> eyda|mon: your application may be insertin lots of data, which hits different shards
[23:25:32] <bjori> edouardb: so when you take backup of shard#1, and then shard#2 they could have "half the writes"
[23:25:57] <eyda|mon> bjori: how can I mitigate that issue then?
[23:26:01] <bjori> and if the balancer is running you could get into loads of problems
[23:26:16] <eucalyptus> stop the balancer
[23:26:16] <bjori> you need to ensure the backup you are taking is from the same timestamp
[23:26:22] <eucalyptus> and disable one of the config servers
[23:26:34] <bjori> right, that solves the balancer problem
[23:26:44] <bjori> but not the problem of the application being in the middle of updating multiple records
[23:26:56] <bjori> (in difference collections)
[23:27:07] <eucalyptus> well, you can't have 100% data loss backup
[23:27:15] <eucalyptus> but you can get awfully close
[23:27:21] <azathoth99> rsync SEEMS to run faster with -z moving some mongo files, I wonder if its an illusion
[23:27:33] <bjori> it isn't
[23:27:35] <azathoth99> well the numbers it reports for transfer rate are bigger
[23:27:38] <bjori> the datafiles aren't compressed
[23:27:43] <azathoth99> oh ok
[23:27:58] <bjori> if you are rsyncing then you need to copy the journal too
[23:28:11] <azathoth99> what do you guys think about cfq vs noop vs deadline linux io scheduler?
[23:28:19] <azathoth99> oh where is the journal?
[23:28:20] <azathoth99> heh
[23:28:33] <azathoth99> I have all data seemingly in /data/mongo.....
[23:28:54] <azathoth99> (if you think im bad bear in mind the devs come to me)
[23:28:57] <bjori> the jounral is there too
[23:32:47] <azathoth99> ok
[23:33:07] <bjori> (assuming you are running with journalling enabled!)
[23:33:23] <azathoth99> www.prevayler.org what do you think of this?