pmxbot IRC Log Viewer

[00:42:47] <Guest16196> quick noob question “mongo mongo1.prod.cryptkeypr.com:27017,mongo2.prod.cryptkeypr.com:27017,mongo3.prod.cryptkeypr.com:27017/shared_prod?replicaSet=keypr_rs0”

[00:42:49] <Guest16196> keypr_rs0:PRIMARY> db.getName()

[00:42:55] <Guest16196> shared_prod?replicaSet=keypr_rs0

[00:43:12] <Guest16196> why do the query options show up in the db name?

[00:43:27] <Guest16196> is the formatting incorrect?

[00:43:29] <Guest16196> thanks

[02:04:24] <rburton-> I have an embedded array for comments and I want to ordering to be sorted be reverse (new-to-old) is that possible or must I just do $slice when pulling the comments

[02:06:28] <joannac> rburton-: http://stackoverflow.com/questions/10131957/can-you-have-mongo-push-prepend-instead-of-append

[02:06:47] <cheeser> you can use an aggregation to reorder that, i think.

[02:07:00] <cheeser> g'day joannac!

[02:07:06] <joannac> You can do it directly in 2.6

[02:07:08] <joannac> hey cheeser

[02:07:14] <cheeser> joannac: i had kangaroo the other day. :D

[02:07:20] <joannac> yum!

[02:07:26] <cheeser> indeed.

[02:07:59] <cheeser> 'sup, chubbs?

[02:08:00] <cheeser> :D

[02:08:14] <rburton-> Okay so there's really no effective way other than $slice and sort the set return in memory and then render

[02:08:24] <rburton-> That's muscle

[02:08:45] <joannac> rburton-: um, 2.6?

[02:08:54] <rburton-> joannac: ?

[02:09:27] <rburton-> I'm returning the parent + child and the child I need sorted.

[02:09:35] <rburton-> basically findOne(_id: xyz)

[02:09:42] <joannac> rburton-: are you running 2.6? you can fix it upon insert in 2.6

[02:09:51] <joannac> although I guess that doesn't help with your retrieval

[02:09:54] <rburton-> I'm using 2.6

[02:10:28] <rburton-> It seems like I need to say $slice and get the last x number of elements, in memory reverse those elements and then render.

[02:10:52] <joannac> yeah, you can't reverse slice :(

[02:11:20] <rburton-> yea, but $slice + reverse in memory isn't bad.

[02:11:32] <rburton-> Not perfect, but not overly complex either

[02:11:57] <rburton-> cheeser have you heard from sarah in while. Since she went down to tx it's been a while since I talked to her.

[02:12:09] <cheeser> not since MongoWorld

[02:12:25] <rburton-> how have you been?

[02:12:52] <rburton-> I'm just working on release 1.0 for this website and I have a few companies that want to pay to use it.

[02:14:12] <pycasso> Hello, Im Looking for helpful tutorial on running a monogdb service for a web application just for learning purposes..any suggestions?

[02:14:45] <rburton-> mongo's website is good

[02:17:42] <cheeser> rburton-: not bad. working on updating ophelia so i can use it work (more)

[02:18:06] <rburton-> what project is that?

[02:18:40] <cheeser> and old version: http://www.antwerkz.com/ophelia

[02:19:24] <rburton-> using angular.js?

[02:19:31] <cheeser> and dropwizard

[02:19:58] <rburton-> DropWizard, Spring Boot, etc. there's a lot of similar frameworks

[02:29:31] <lqez> hmmm

[02:29:32] <lqez> http://docs.mongodb.org/manual/tutorial/expire-data/

[02:29:57] <lqez> I added TTL index for expiration but no documents were removed after expireAfterSeconds

[02:30:20] <cheeser> did you define that index against a date property?

[02:32:17] <lqez> cheeser: http://pastebin.com/NHq2HknT

[02:32:55] <joannac> lqez: check again

[02:32:58] <lqez> And it still exists lol

[02:32:59] <cheeser> the cleaner thread runs once a minute

[02:33:03] <joannac> the cleanup thread runs once a minute iirc

[02:33:07] <joannac> dammit cheeser

[02:33:24] <lqez> ok got it

[02:33:52] <lqez> and do I have to define TTL before inserting docs?

[02:34:02] <cheeser> you can define it at any time.

[02:34:29] <lqez> thanks-

[05:01:56] <leo_> have a performance related question. would appreciate if someone can give me a pointer on whats going wrong

[05:02:07] <leo_> I am relatively new to mongo so forgive my ignorance. I have a setup with one client (approximately 300 connections) to a sharded mongo cluster in ec2. All the 300 connections are from single client machine. This gives me approximately 300 inserts/sec. If I add multiple clients running on separate machines the inserts fall to around 70-90. I do see the queue size for readers seems to have gone up. The lock %age seems to be avg aroun

[05:03:14] <leo_> explain the drop?

[05:08:54] <Boomtime> "The lock %age seems to be avg aroun..."

[05:10:53] <Boomtime> also, what are the instances?

[05:40:12] <leo_> @boomtime

[05:40:24] <leo_> around 40%

[05:41:11] <leo_> the shards are running on pretty high end machines hi1

[05:43:48] <leo_> and the router is running on m2*2x

[05:47:22] <Boomtime> 300 inserts/sec is atrocious, are the machines doing anything else? do they have provisioned IOPS?

[05:48:41] <Boomtime> i don't see the Hi1 listed for automatic PIOPS allocation, despite being backed by SSDs, so that might be part of it

[05:51:17] <leo_> no

[05:51:24] <leo_> they aren't doing anything else

[05:51:42] <leo_> the size of the inserts is close to 4k with around 20 indexes in it

[05:51:53] <leo_> *4kb

[05:51:59] <Boomtime> 20 indexes?

[05:52:09] <Boomtime> ok, and what is the shard key?

[05:52:25] <leo_> i am just sharding on userId for now

[05:52:44] <leo_> hash based

[05:52:53] <Boomtime> userid: hashed?

[05:52:57] <leo_> yes

[05:53:11] <Boomtime> and what is the source cardinality? how may userids are there?

[05:53:19] <Boomtime> *many

[05:53:22] <leo_> close to 50k

[05:53:39] <Boomtime> ok, so distribution should be pretty good, how many shards?

[05:54:25] <leo_> 2

[05:54:37] <leo_> this is still a test cluster

[05:54:49] <leo_> and the only aim is to see how mongo scales

[05:54:54] <leo_> for my data

[05:55:43] <Boomtime> what about provisioned iops?

[05:56:10] <leo_> i haven't done anything specific for that

[05:56:27] <leo_> so it might be the default

[05:57:05] <Boomtime> which is rubbish

[05:58:10] <leo_> may be thats what is limiting

[05:58:24] <Boomtime> you can check, run iotop for a bit

[05:58:40] <leo_> on the shard

[05:58:49] <Boomtime> yep

[05:59:01] <Boomtime> on the primary of one or both shards

[05:59:12] <Boomtime> you may want t compare, to observe if there is any imbalance

[06:01:11] <leo_> nothing bad I see

[06:02:38] <Boomtime> post a line or two

[06:03:12] <Boomtime> also, are the shards replica-sets?

[06:03:42] <leo_> no

[06:03:48] <leo_> its not replica-sets

[06:04:05] <leo_> DISK READ: 0.00 B/s | Total DISK WRITE: 1849.46 K/s

[06:04:18] <leo_> since I am doing only inserts

[06:05:23] <leo_> i see 20 or 40 m/s in total disk write

[06:05:40] <leo_> may be background activity

[06:06:31] <Boomtime> what is the io % ?

[06:06:40] <Boomtime> (according to iotop)

[06:07:37] <leo_> its in 0.6 - 0.8%

[06:08:07] <leo_> some times uses 6%

[06:09:14] <leo_> write transactions on 2004.00 kB_wrtn/s

[06:09:47] <leo_> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn xvda 0.00 0.00 0.00 0 0 xvdb 17309.00 0.00 173848.00 0 173848 xvdc 0.00 0.00 0.00 0 0

[06:10:06] <leo_> iostat shows this sudden burst

[07:19:30] <coffe> Hi. FYI. the init script for 2.6.4-1 is not working on RHEL systems.. start works but stop dont as its adding a " " to it and then it cant find the pid file

[07:20:43] <joannac> umm, adding a space to what?

[07:22:13] <coffe> #PIDFILE=`awk -F= '/^pidfilepath[[:blank:]]*=[[:blank:]]*/{print $2}' "$CONFIGFILE"` <-- this one .

[07:27:59] <joannac> coffe: SERVER-13595 ?

[07:28:07] <joannac> jira.mongodb.org/browse/SERVER-13595

[07:29:49] <coffe> joannac, will check

[07:31:00] <coffe> joannac, yes .. thats it.. then its reported .. good.

[07:31:41] <joannac> thanks for letting us know anyway, just in case

[07:41:43] <coffe> joannac, NP . just spent 3h trying to understand why ..

[08:05:24] <arussel> I'm running db.currentOp(true) on a db that should have no activities and was surprised to see some results. Is there something like a 'stale' query or something like it ?

[08:08:13] <arussel> they're all active: false, waitingForLock: false and some have stuff in lockStats

[08:09:21] <arussel> some are mongodb query, but some are mine

[08:52:45] <eliDM> I everyone

[08:53:03] <eliDM> quick ques. regarding structures in mongo

[08:53:36] <eliDM> http://stackoverflow.com/questions/25522254/mongodb-best-practice-structure

[08:54:01] <eliDM> not sure what is the recommended method when dealing with 3 levels of data

[08:54:17] <eliDM> is it better to keep 1 collection or seveal?

[09:02:47] <stefandxm> guys, am i completely stupid or should i use ScopedDbConnection in a multithreaded application? and if so; why is it factory style and returns a pointer to a smart pointer?

[09:15:40] <stefandxm> http://api.mongodb.org/cxx/current/classmongo_1_1_d_b_client_interface.html#a5dfea5d2d6019ee1a67c3413a64ecf42

[09:15:45] <stefandxm> is this really the proper documentation?

[09:16:57] <stefandxm> since code header says it thros AsssertionException

[09:17:08] <stefandxm> and wtf is the ns parameter? :o

[09:18:00] <stefandxm> ah namesapce

[09:18:03] <stefandxm> documented in save

[09:18:07] <stefandxm> lets hope it means the same :D

[09:23:21] <_valle_> I having problem running mongodb. I have tried both version 2.6.0 and 2.6.3 with the following result: http://pastebin.com/ggTBVXcT

[09:25:21] <jordana> _valle_: what are you running it on?

[09:27:26] <_valle_> beaglebone black

[09:28:57] <jordana> Is it an ARM chip?

[09:29:00] <jordana> on it

[09:29:18] <jordana> Yeah - cortex 8

[09:29:28] <jordana> Does mongodb even support ARM?

[09:30:29] <jordana> There's an experimental build up I think of 2.4

[09:30:34] <jordana> http://stackoverflow.com/questions/22706738/is-there-a-mongodb-2-4-installation-available-for-arm-chips and this might help:

[09:30:42] <jordana> https://jira.mongodb.org/browse/SERVER-1811

[09:30:53] <jordana> In terms of progress

[09:31:16] <jordana> so basically _valle_, ARM isn't supported yet

[09:32:07] <Derick> the drivers support it though, not the server

[09:33:42] <jordana> Cool

[09:40:21] <_valle_> jordana: Okay I thought it was supported since it was in the meta-oe layer for yocto

[09:40:31] <_valle_> jordana: Thank you

[10:28:52] <jordana> _valle_: no worries, I think there are some builds out that are experimental so you might be able to get something up!

[10:29:02] <jordana> _valle_: or host it on a different system

[10:43:22] <rdsoze> I have a large update to run: around 8 million in a collection containing 10 million records. Should I run a single update or break it up into smaller updates using $in ?

[10:43:30] <rdsoze> Which would run faster ?

[11:17:34] <kali> rdsoze: both will be awfully slow :)

[11:18:02] <kali> rdsoze: use the single update, if it is practical

[11:38:51] <kali> :q

[11:43:25] <kali> grrrrr

[11:43:34] <kali> something wrong with my terminal

[12:14:21] <rdsoze> kali: Thanks. Any article or some sort of time comparison for the two ways ?

[12:17:58] <kali> rdsoze: nope, just experience, or good sense, talking here: most of the expensive work will fetching, updating and saving the documents. you'll update most of the documents in the collections so there is not much to be gain in optimizing documents selection. so i can't see any point in flooding your client and server memory with buffer full of ids

[12:26:46] <rdsoze> kali: thanks! That makes a lot of sense.

[14:29:41] <bendersalt> I'm migrating some data from a postgres database to a mongo database, I kind of naively just outputted to csv and loaded it, but now my time stamps are text and I can't really index them properly, what would be a good way to modify all those rows to be Date types instead? Does this question even make sense.

[14:32:50] <kali> bendersalt: it kinda make sense: http://stackoverflow.com/questions/6475987/importing-date-datatype-using-mongoimport

[14:33:59] <kali> bendersalt: that said, mongodb will probably only be really helpful once you'll have redesign your data model according to mongodb strong points

[14:34:45] <bendersalt> That link was a god send my google fu was deficient

[14:35:21] <kali> bendersalt: na, I did send it

[14:35:51] <bendersalt> kali: Yeah at the moment I'm kind of just doing some weak prototyping, I'll be doing a hard look at data structure and denormalization before I move forward with making it production quality

[14:37:01] <bendersalt> kale: Thank you!

[14:37:13] <bendersalt> Can't even get names right.

[14:46:56] <wiitard> I need help installing php mongodb drivers on mamp.. pain in the ass!

[14:47:05] <wiitard> has anyone done this?

[14:47:27] <jordana> You're better off using vagrant :p

[14:48:57] <jordana> wiitard: have you got homebrew installed?

[14:49:12] <wiitard> yes

[14:49:48] <jordana> and pecl?

[14:49:57] <jordana> pear*

[14:50:22] <wiitard> yes, im running via mamp pro

[14:51:13] <jordana> If you're path's setup correctly

[14:51:22] <jordana> run sudo pecl install mongo

[14:52:03] <wiitard> ok..one sec

[14:53:09] <jordana> your*

[14:53:43] <wiitard> here is what I did: 1. downloaded this package https://github.com/mongodb/mongo-php-driver

[14:54:44] <wiitard> 2. created a folder named "include" and copied the package files to /Applications/MAMP/bin/php/php5.5.3/include/

[14:55:15] <wiitard> 3. followed the install instructions here https://github.com/mongodb/mongo-php-driver#installation

[14:56:10] <Derick> wiitard: why did you do step 2?

[14:56:17] <Derick> wiitard: please use "pecl install mongo"

[14:56:25] <Derick> you can't just put it in some include/ path

[14:57:51] <jordana> +1 for pecl

[14:59:47] <wiitard> without step 2 I get an error message http://bpaste.net/show/f207f8276380 and with step 2 i get an error message http://bpaste.net/show/a51cb1096563

[15:00:08] <wiitard> @Derick: ^

[15:00:46] <Derick> wiitard: you need to have the PHP headers installed too

[15:01:08] <Derick> do you have /Applications/MAMP/bin/php/php5.5.3/include/php/main/php.h ?

[15:02:00] <wiitard> @Derick: no Mamp doesn't ship with an include folder which is why i created one

[15:02:32] <Derick> you need the include folder that the version of PHP that MAMP uses

[15:02:40] <Derick> otherwise you can not compile extensions

[15:08:13] <wiitard> @Derick: do i simply download the latest php version and paste into the php folder? confused to how to make that happen

[15:08:18] <Derick> no

[15:08:28] <Derick> you need to match the one that MAMP was built with

[15:08:47] <Derick> *or* do everything yourself (Starting with removing MAMP completely)

[15:10:36] <wiitard> here is the link to the mamp php version 5.5.3 http://www.mamp.info/en/downloads/older-versions/ I see this package doesn't have the include php headers

[15:10:54] <Derick> https://thomashunter.name/blog/getting-the-php-mongodb-driver-installed-with-mamp-on-os-x/

[15:10:57] <Derick> seems to explain it

[15:11:39] <Derick> actually, not really either.

[15:11:51] <Derick> My recommendation would be to learn how to setup PHP and Apache yourself.

[15:20:16] <jordana> You're best doing that through vagrant

[15:20:48] <jordana> install it, boot up a centos/ubuntu machine and install PHP/PEAR/Mongo

[16:02:56] <wiitard> @Derick:following this http://verysimple.com/2013/11/05/compile-php-extensions-for-mamp/

[16:12:44] <joer> I want to do a selective field selection when I use mongo's find. Particular situation includes a large associative object within my document for each user so I only want to select the object key relevant to the user. However I also want to select all other fields in the document (which are not linked to the particular user and are not well defined enough to allow me to simply list them and mark them off in my field selection list). I

[16:19:00] <joer> Answered my own question - $elemMatch will do. Incase anyone comes across and is curious!

[16:46:24] <jblancett> where do the init scripts for the mongodb .deb come from? I deleted them on my system and now when I reinstall the .deb it doesn't put them back

[16:46:38] <jblancett> I extracted the .deb and don't see them in there either

[16:48:34] <joer> I am looking at http://docs.mongodb.org/manual/reference/operator/projection/elemMatch/#proj._S_elemMatch and see how to use elemMatch to filter the elements of an array selected. However, is there a way to then select all other fields in my document too without necessarily individually flagging them to be returned?

[17:38:46] <edrocks> does $add

[17:39:06] <edrocks> does $addToSet return an error if the element is already in the array?

[17:41:54] <cheeser> "The $addToSet operator adds a value to an array only if the value is not already in the array. If the value is in the array, $addToSet does not modify the array."

[17:41:56] <cheeser> http://docs.mongodb.org/manual/reference/operator/update/addToSet/

[18:09:12] <feathersanddown> Hi!, how to add an element in an array that is an atribute from a document in java?? http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-java-driver/#getting-started-with-java-driver <--- there is no example for that

[18:09:36] <feathersanddown> what I'm askink is an update operation?

[18:11:20] <kali> feathersanddown: update with $set

[18:11:36] <kali> feathersanddown: no. not $set, but $push

[18:11:40] <feathersanddown> uhm...

[18:47:43] <wiitard> alas! monog php driver are installed properly with MAMP!

[19:46:32] <JesseObrien> hi mongods

[19:47:10] <JesseObrien> anyone here have any experience with mongo on the scale of 100,000+ writes in short order?

[19:47:43] <JesseObrien> i have some surveys going out that could get 5-6,000 hits in a few hours time, each survey has between 20-30 questions

[19:48:34] <cheeser> doesn't sound problematic

[19:49:15] <JesseObrien> cheeser: would a box with ~8G of ram and a quad core be alright under that?

[19:49:57] <JesseObrien> has an SSD as well

[19:50:17] <JesseObrien> the reason i'm wondering is i have it set up to write into redis, i know it'll handle it

[19:50:18] <cheeser> my dev machine is beefier

[19:50:39] <JesseObrien> but i would rather put the data into mongo if i can, to use the aggregation framework

[19:53:06] <davejfranco> #JesseObrien scale is not only related to the box capacity you have

[19:53:40] <JesseObrien> davejfranco: righto, what am i limited by?

[19:53:54] <JesseObrien> disk space, obviously, but yeah i'm interested in the real bottlenecks

[19:54:11] <davejfranco> network, availability

[19:54:16] <JesseObrien> network throughput is another thing, obviously with a cluster it would be better

[19:54:33] <JesseObrien> i don't have the budget for another box or three though

[19:55:00] <JesseObrien> davejfranco: when you say availablity do you mean like write-locking?

[19:55:29] <davejfranco> try to maintain a really good backup and disaster recovery strategy

[19:55:34] <JesseObrien> mongo has global write-lock doesn't it?

[19:55:45] <JesseObrien> alright

[19:55:51] <JesseObrien> those are basics of database interaction though lol

[19:56:15] <davejfranco> but are really important when you're trying to scale

[19:56:24] <JesseObrien> fo sure

[19:56:30] <davejfranco> is on a cloud environment?

[19:56:43] <JesseObrien> the web server and db server have private networking enabled, so theoretically it should be high bandwidth between them

[19:56:46] <JesseObrien> davejfranco: yeah digital ocean

[19:57:13] <davejfranco> do you use mms?

[19:57:25] <JesseObrien> mms?

[19:57:41] <davejfranco> is a monitoring system for mongodb

[19:57:46] <JesseObrien> ah, haven't seen it

[19:57:48] <JesseObrien> i'll check it out

[19:57:49] <davejfranco> created by mongo

[19:57:56] <davejfranco> I'm using it

[19:58:07] <davejfranco> is really good and free

[19:58:21] <JesseObrien> free for a year?

[19:58:43] <JesseObrien> oh i see just the backup is $$ after a year

[19:58:44] <JesseObrien> cool

[19:58:45] <JesseObrien> will check this out

[19:58:54] <davejfranco> I think for ever, they have this option for paying if you want to make back up

[19:59:12] <davejfranco> and is not that expensive

[20:00:07] <cheeser> less expensive than data loss for sure :D

[20:00:13] <JesseObrien> heh yeah

[20:00:23] <davejfranco> that's right

[20:07:01] <JesseObrien> davejfranco: cool

[20:07:02] <JesseObrien> hooked that up

[20:07:03] <JesseObrien> thanks man

[21:07:36] <SkramX> i could use some help with an aggregation query. I have a DB of tweets. I am counting the them grouped by user with $sum right now. Now I want to count not only the total but the number by each user that has hashtags which is based on the length of an array

[21:07:46] <SkramX> how do I sum based on if an array is empty or not?

[21:13:27] <sssilver> Hey guys... after getting the results using find(), I need to iterate over the results and add some attributes for each result. How does this work with mongo?

[21:13:36] <sssilver> if the results is basically a cursor object

[21:17:45] <joannac> .forEach

[21:18:24] <sssilver> joannac: but then I'm gonna be forced to use Javascript right.... whereas now I'm using pymongo

[21:19:36] <cheeser> you could iterate over it in python and send it back. it'd just mean a round trip.

[21:20:04] <sssilver> cheeser: I don't mind iterating over it, but the problem is that I don't seem to be able to modify it in place

[21:20:29] <sssilver> so I seem to be forced to copy each element into another set after modifying it

[21:20:54] <sssilver> is that the accepted pattern for doing something like this?

[21:21:05] <cheeser> you can modify in place...

[21:22:02] <sssilver> basically my problem is that I have a collection of teachers and a collection of students and every time someone requests a list of teachers, each teacher has to have a field like num_students, with the number of related students

[21:24:03] <sssilver> cheeser: this is what I do: http://bpaste.net/show/ab64cfb1b455

[21:24:25] <sssilver> but I don't see how that modifies data

[21:27:08] <cheeser> do you ever save it back?

[21:27:13] <cheeser> doesn't look that way

[21:28:10] <sssilver> I don't need to save it back

[21:28:16] <sssilver> I just need to return it via HTTP

[21:28:25] <cheeser> what?

[21:28:27] <sssilver> I don't want to denormalize the data

[21:28:38] <sssilver> what do you mean by saying "save it back"?

[21:28:44] <sssilver> I may be getting you the wrong way

[21:28:58] <cheeser> what makes you think you can't edit in place?

[21:29:40] <sssilver> line 4 in my paste, that doesn't actually modify data does it

[21:30:11] <cheeser> looks like it does to me.

[21:30:25] <cheeser> print out your list when you're done with that loop and see

[21:30:43] <cheeser> anyhoo, i gotta go catch a train.

[21:30:47] <sssilver> cheeser: it becomes an empty list

[21:41:10] <leo_> I see frequent spikes in disk io http://bpaste.net/show/0ca94f40ee9a while performing some load testing in mongo

[21:42:36] <leo_> after every 4 ticks I see some this spike in disk io

[21:42:53] <leo_> what could mongo be doing

[21:43:15] <leo_> does it construct index in background and this be a disk flush?

[22:04:40] <trco> Hello all. I am receiving a duplicate key error on insert on my _id. It seems after I added some new test data to my test DB this started to happen. Any idea how to reset the _id to the appropriate position so inserts start working?

[22:06:03] <trco> I imagine it's some sort of counter, but I can't seem to get it to reset

[22:08:54] <joannac> the default is a counter based on timestamp

[22:09:16] <joannac> so unless you set your system clock back...

[22:14:07] <Naum> Important

[22:14:07] <Naum> If you are running any edition of Windows Server 2008 R2 or Windows 7, please install a hotfix to resolve an issue with memory mapped files on Windows.

[22:15:35] <Naum> alguem poderia me ajudar?

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 27th of August, 2014