pmxbot IRC Log Viewer

[00:09:43] <aminethedream> hello

[01:13:20] <mikael> quick question... how do i set the timezone for localTime corect? i didnt find any about it on the docs for mongod.conf

[10:47:22] <felixjet> what is a sub-document, [] or {} ?

[10:49:27] <Nodex> [] is an array {} is an object

[10:49:42] <felixjet> yes, but in mongo ?

[10:49:49] <Nodex> yes in mongo

[10:49:59] <felixjet> how i do create a sub-document then?

[10:50:32] <Nodex> db.foo.insert({name:"Bob",profile:{phone:1234,email:"123@456.com"}});

[10:50:58] <felixjet> so.. with {}

[10:51:08] <Nodex> :)

[10:51:31] <felixjet> just to be clear, because my english sucks as much as my ability to understand mongo (it seems so xD)

[10:51:50] <felixjet> in, for example a blog, every post have tags. i should create them with [], right?

[10:51:57] <felixjet> to avoid creating a sub-document

[10:52:05] <felixjet> since its only to store "a list of things"

[10:52:21] <Nodex> it really depends on how you want to create things, but yes, in general tags would go in an arra

[10:52:24] <Nodex> array*

[10:53:43] <felixjet> thank you

[11:23:11] <Mickelodeonion> How, how can I calculate the future size of my db ?

[11:23:46] <Mickelodeonion> I need to store around 150.000.000 documents

[11:24:16] <Mickelodeonion> which each document has around 500 pairs

[11:26:21] <Mickelodeonion> ;/

[11:29:42] <kees_> just create a db with 150 documents, see how big that is, take that number, do it times 1000, look how big that number is, do it times 1000 again and there is your answer ;)

[11:32:15] <Mickelodeonion> so, there is no way to pre-calculate

[11:35:35] <Nodex> no

[13:00:57] <hallucynogenyc> Hey guys, trying to get started with mongo

[13:01:49] <hallucynogenyc> installed via homebrew in mac, however the mongod process exits with status 2 (bad options according to the docs) but without writting anything to the logs

[13:02:01] <hallucynogenyc> have noidea how to debug this any further

[13:03:04] <joannac> hallucynogenyc: what ptions are you starting it with

[13:03:33] <hallucynogenyc> deafult brew mongod.conf

[13:04:09] <hallucynogenyc> here it is

[13:04:10] <hallucynogenyc> https://gist.github.com/hallucynogenyc/9324455

[13:05:19] <hallucynogenyc> funnily enough

[13:05:26] <hallucynogenyc> if I run it like mongod --config /usr/local/etc/mongod.conf

[13:05:31] <hallucynogenyc> then it runs

[13:05:42] <joannac> erm, yes?

[13:06:01] <joannac> what were you trying before?

[13:06:04] <hallucynogenyc> however if I do a launchctl list | grep mongo

[13:06:12] <hallucynogenyc> - 2 homebrew.mxcl.mongodb

[13:06:17] <hallucynogenyc> so no pid, error code 2

[13:08:11] <joannac> and there's nothing in the log?

[13:08:31] <hallucynogenyc> nope

[13:08:35] <joannac> `ps -Alf | grep mongod` for me and pastebin

[13:08:47] <hallucynogenyc> this is my plist file

[13:08:48] <hallucynogenyc> https://gist.github.com/hallucynogenyc/9324515

[13:08:49] <hallucynogenyc> ok sec

[13:09:47] <hallucynogenyc> here you go http://pastebin.com/Aj2SghsX

[13:10:29] <joannac> you're redirected stdout and stderr?

[13:11:23] <hallucynogenyc> yeah the plist file declares stadardErrorPath

[13:11:28] <hallucynogenyc> but nothing is in there either

[13:14:36] <joannac> okay

[13:14:42] <joannac> status 2 is SIGINT

[13:15:03] <remonvv> Hi guys, we have a weird issue; we have a database that takes 6.4Gb filesize for 58Mb storage size (according to db.stats()). This margin is inexplicably huge. The data are simply 2 field tuples, no growing documents.

[13:15:43] <joannac> remonvv: deleting documents?

[13:15:45] <hallucynogenyc> not according to the mongo docs?

[13:15:48] <hallucynogenyc> it said bad options there

[13:15:53] <joannac> hallucynogenyc: link?

[13:16:27] <hallucynogenyc> https://jira.mongodb.org/browse/DOCS-19

[13:16:49] <remonvv> @joannac: Certainly not to that extent I think. Shouldn't it start reusing at some point? I know usePowerOf2Sizes helps but I was under the impression it reuses file extents at some point.

[13:17:00] <hallucynogenyc> and also

[13:17:02] <hallucynogenyc> http://ftp.isr.ist.utl.pt/pub/roswiki/doc/unstable/api/mongodb/html/namespacemongo.html

[13:17:55] <joannac> hallucynogenyc: list returns the signal, not the mongod error code

[13:18:02] <joannac> https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/launchctl.1.html

[13:18:06] <hallucynogenyc> oh

[13:18:18] <hallucynogenyc> so what to do?

[13:19:22] <joannac> remonvv: yes it will, but free lists are populated by buckets.

[13:19:59] <joannac> remonvv: and we don't exhaustively search because that leads to performance problems. So we give up if we haven't found an appropriately sized extent and create a new one.

[13:20:37] <joannac> we give after after looking at X extents in the free list, I mean

[13:20:43] <remonvv> @joannac: Okay, but that seems to happen pretty much always.

[13:21:12] <remonvv> @joannac: So what would you suggest we do? Short term to correct this state and long term to avoid it happening again.

[13:21:50] <hallucynogenyc> SIGINT 2 Term Interrupt from keyboard

[13:22:00] <hallucynogenyc> What? I mean look at this

[13:22:50] <joannac> remonvv: powerof2sizes

[13:23:08] <remonvv> @joannac: Can that be enabled on an existing collection?

[13:23:22] <joannac> remonvv: sure can, but it'll only aopply to new extents

[13:23:29] <hallucynogenyc> joannac http://pastebin.com/7wPGhGQF

[13:23:40] <hallucynogenyc> how can it clain I stopped it?

[13:23:43] <joannac> you'll need to repair it... or maybe compact... i can't remember off the top of my head

[13:24:20] <remonvv> @joannac: I'll look it up. Thanks. I'll assume there are various JIRA tickets open for this being improved already?

[13:26:13] <remonvv> Hm, can't run compact on mongos apparently and both compact and repair lock the database.

[14:31:56] <hallucynogenyc> ok joannac I figured it out, the brew installation was using a wrong path for the mongod executable, it was missing that mongod is inside /bin

[14:32:11] <hallucynogenyc> why was this showing up as a keyboard interrupt I have no idea

[14:44:52] <hallucynogenyc> this is the pull request to fix it

[14:44:53] <hallucynogenyc> https://github.com/Homebrew/homebrew/pull/27155

[15:10:31] <isart> hi!

[15:11:39] <isart> does anyone know if dropping a collection will acuire a write lock on the whole db?

[15:12:55] <cheeser> i would expect so

[15:14:47] <isart> thanks cheeser

[15:55:37] <MrSavage> Hi, how do you edit mongo's session's data?

[15:55:57] <MrSavage> can you just do something like req.session.WHATEVER = WHATEVER; ???

[15:56:14] <MrSavage> sometimes it works for me, sometimes it does not. I am not sure why

[16:03:45] <Nodex> eh?

[16:10:07] <Jonas_> Hello

[16:11:31] <MrSavage> hi

[16:12:15] <Guest27520> I would like to be able to say in C#: GetDatabase("test").nameOfCollection.Find()

[16:12:23] <Guest27520> Anyone that can help me?

[16:13:03] <Guest27520> Instead of GetDatabase("test").GetCollection("nameOfCollection").Find()

[16:13:38] <MrSavage> isn't that impossible?

[16:14:01] <kali> it is

[16:14:27] <Guest27520> Why is it that?

[16:14:46] <kali> because . is a method or property call in c#

[16:15:01] <kali> and they have to be statically defined

[16:17:29] <Guest27520> What if a make a List of the collection, can i somehow on that list say: List.nameOfCollection ?

[16:17:39] <Guest27520> of the collections*

[16:39:06] <kali> you can't have .anything_variable, this is not javascript

[16:43:47] <bryanlarsen> I'm trying to add a member to a replica set. I followed http://docs.mongodb.org/manual/tutorial/expand-replica-set/. Now rs.conf() on primary shows a stateStr of "UNKNOWN" for the new member and rs.conf() on the new member shows a startupStatus of 4, and an errmsg of ""can't currently get local.system.replset config from self or any seed (EMPTYUNREACHABLE)""

[16:44:12] <bryanlarsen> The database is large and recovery will take a while, but I don't think that it's recovering. Any pointers?

[16:49:32] <kali> can the two hosts speak to each other ?

[16:50:15] <bryanlarsen> yes, I used telnet to make sure that the ports are visible from each machine

[16:51:26] <bryanlarsen> I also was able to successfully move a configsvr onto the new machine

[16:51:46] <kali> no trivial mistake on hostname, port number ? because it really sound like something like that

[16:56:08] <bryanlarsen> in my mongodb.log on the new server, I get a bunch of "connection accepted" messages from all my mongos servers

[16:57:39] <bryanlarsen> the log says "trying to contact". I double checked that I was able to telnet to the host/port listed for all of those, and that works

[16:57:54] <morsik> hello there

[16:58:14] <morsik> i'm thinking about using mongo with hadoop backend. is this really that good as i read?

[16:58:27] <kali> morsik: wat ?

[16:58:35] <bryanlarsen> it then says "replSet info Couldn't load config yet. Sleeping 20sec and will try again."

[16:58:45] <morsik> i just calculated, and i think it could be about 5T of (not compressed) json per month

[16:58:56] <morsik> kali: what what? :D

[16:59:26] <morsik> kali: 1: http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/ 2: http://docs.mongodb.org/ecosystem/tools/hadoop/

[17:00:10] <kali> morsik: well, it's rather the other way around. you store data in mongodb, and hadoop will use it to do something

[17:01:12] <edsiper> what do you usually put as a HTTP interface on front of MongoDB ?

[17:01:15] <bryanlarsen> ah hah, found it: http://osdir.com/ml/mongodb-user/2011-12/msg01280.html

[17:01:31] <morsik> kali: ouch… my mistake.

[17:01:43] <kali> bryanlarsen: i wasn't that far :)

[17:01:44] <bryanlarsen> the /etc/hosts file got updated by all my servers except for the dead server.

[17:02:00] <Nodex> edsiper : for what purpose?

[17:02:32] <bryanlarsen> now it's showing "initial sync cloning db: studio_production". It'll probably be like that for a day or so. Thanks for your help kali, much appreciated

[17:02:33] <edsiper> Nodex, mobile apps

[17:02:50] <Nodex> I don't understand what a http interface has to do with a mobile app

[17:03:00] <ron> O_O

[17:03:13] <ron> You want to expose mongodb directly to mobile apps?

[17:03:16] <edsiper> Nodex, your mobile app requires to consume MongoDB data, so you may use HTTP as an interface to query the database

[17:03:25] <edsiper> ron, yes

[17:03:26] <ron> wtf

[17:03:40] <Nodex> edsiper : remind me to not use your mobile app, that is seriously unsecure

[17:03:44] <ron> haha

[17:04:03] <edsiper> unsecure would be to talk to mongodb directly

[17:04:18] <Nodex> most normal people write an API that spits out json or whatever

[17:04:27] <ron> even exposing it plainly using an 'http server' would be insecure.

[17:04:29] <kali> bryanlarsen: good :)

[17:04:31] <Nodex> so yes in terms of http that would be what most people d

[17:04:32] <Nodex> do

[17:04:56] <edsiper> https/spdy ,,

[17:05:05] <ron> no..

[17:05:12] <Nodex> the transport is not relevant. the exposure is

[17:06:07] <edsiper> and that API to send out the json to the database, what service type is being used ? nodejs ?

[17:06:37] <Nodex> it's platform agnostic, build it in whatever you like

[17:09:52] <edsiper> i am gonna re-state the question, is there anybody around using some HTTP/HTTPS service to query their MongoDB database ?

[17:10:17] <ron> I wonder if RockMongo counts.

[17:10:18] <ron> :p

[17:10:29] <Nodex> lol

[17:11:57] <Nodex> in all seriousness, just build one yourself, take a day or so

[17:12:14] <edsiper> thats not my point, i need to find people using HTTP interfaces to query mongo

[17:12:30] <edsiper> someone using something like this: http://docs.mongodb.org/ecosystem/tools/http-interfaces/

[17:13:06] <Nodex> I doubt you will find people because it's trivial to build one yourself. Why rely on someone else's blated code when it can be streamlined to your app

[17:13:50] <edsiper> why do yo assume that i need someone code ?.

[17:14:04] <Nodex> are you not listening to me?

[17:14:08] <edsiper> pointless discussion, well, if anyone is using some rest interface please ping me

[17:14:26] <Nodex> again, you will find it HARD to find someone becuase PEOPLE BUILD THIER OWN

[17:15:21] <edsiper> well, "build their own" and "use some existent HTTP interface as the listed above" applies...just need to contact people using/creating HTTP interfaces to query mongo

[17:16:14] <Nodex> ok good luck with that :)

[17:26:36] <morsik> kali: ok, anyway… continuing, does mongodb could be good for big data things? as i read on mongodb page, it's already used for things like that, also i found that some company is using mongo for monitoring data

[17:26:43] <morsik> …the same thing i needs it.

[17:26:47] <morsik> i need it*

[17:28:56] <morsik> this looks promising. http://www.mongodb.com/presentations/humongous-data-server-density-approaching-1-billion-documents-mongodb + https://blog.serverdensity.com/server-density-stats-in-april-2011/

[17:47:10] <kali> morsik: mongodb is for big data, no doubt about that. but not at all for batch processing (where hadoop can help)

[17:48:15] <kali> morsik: for monitoring, as you're likely to manipulate time series all over the place, I personally think the document model is not a good fit. That said, there are blog entries by a mongodb staffer stating otherwise.

[17:52:12] <morsik> kali: well, we monitor our web pages. download times, http return codes, all html (when error occured). also mail checks (so different structure)

[17:52:24] <morsik> images times/codes, css etc.

[17:52:58] <morsik> i already have mysql for this, and it is pain in the ass ;x

[17:53:37] <morsik> (and only 1.6mln records, because we don't store correct results, only failed)

[17:53:58] <morsik> kali: for calculating SLA or things like what i could use hadoop

[17:54:11] <morsik> i'm thinking also about HBase directly here.

[17:56:43] <morsik> about 1k input data per minute

[18:04:40] <Aartsie> Hi all

[18:05:52] <Aartsie> a hosting company asks me to work for them and what my price is by hour but i have no idea what the markt is at the moment, anyone a suggestion ?

[18:06:31] <morsik> depends on country, region in that country, what you should to there, etc. etc.

[18:06:47] <morsik> should do*

[18:07:41] <Aartsie> In the Netherlands, and i have to build some MongoDB clusters and sometimes i have to program some applications for them

[18:12:14] <Aartsie> morsik: you have any idea ?

[18:13:55] <morsik> nope. i'm from poland.

[18:17:16] <Aartsie> whats the price in poland ?

[18:17:43] <morsik> depends on everything, but about from 4k to 10k PLN.

[18:18:49] <Aartsie> 4k for 1 hour ?

[18:18:53] <morsik> or from 3k for begginers

[18:18:59] <morsik> ouch, no. per month.

[18:19:15] <Aartsie> hahaha thats more realistic :P

[18:19:37] <morsik> :D

[18:19:54] <morsik> sorry, different countries counts salary differently

[18:20:25] <morsik> in poland we count it per month, afaik in usa per year, and i heard that somewhere (england?) they count per week

[18:21:14] <morsik> and i'm at full-time work now, so we have always 40h per week to do, so almost nobody counts here salary per hour

[18:22:11] <rafaelhbarros> where do I get the job that pays 4k/hour

[18:22:28] <morsik> nowhere.

[18:22:32] <rafaelhbarros> or like jon stewart would say, cocaine and unicorns

[18:22:37] <morsik> haha :D

[18:22:39] <rafaelhbarros> and how do I get there?

[18:22:52] <rafaelhbarros> is it a road or a midnight train?

[18:27:20] <Aartsie> anyone else ?

[18:28:02] <rkgarcia> hi Aartsie

[18:29:11] <Aartsie> hi rkgarcia

[18:30:01] <rkgarcia> what's happening Aartsie ?

[18:31:18] <Aartsie> rkgarcia: a company asks me to work for them by hour but i don't know what i have to ask for price by hour

[18:31:54] <sflint> my rate is 150/hr

[18:33:20] <rkgarcia> sflint: skills?

[18:34:15] <morsik> sflint: In what currency?

[18:34:24] <sflint> I specialize in scaling. Whole stack...I am a mongodb master.

[18:34:30] <sflint> US currency

[18:35:03] <rkgarcia> Are you certified?

[18:35:06] <morsik> fuucckk… about 20k per month?

[18:35:06] <rkgarcia> D:

[18:35:17] <morsik> or you're freelancer admin?

[18:35:39] <sflint> I have a current job, but I so contract work for a lot of other companies in NYC

[18:36:10] <sflint> not certified in anything. I don't think

[18:36:21] <sflint> I have my six sigma (lol) from GE

[18:53:19] <kali> morsik: honestly, i think hbase would be a better fit

[18:53:32] <morsik> mhm.

[18:57:36] <rkgarcia> kali: what are you doing here? >_>

[18:58:21] <def-lkb> Hi, I am a complete beginner with mongo. I think my setup is suboptimal… So far I have used only one instance, but I think I am reaching performance limit.

[18:59:08] <def-lkb> I bought a dedicated box for mongo (8 cores), only one is used. Queries often timeout if there are concurrent writes.

[18:59:38] <kali> rkgarcia: trying to avoid one more "mongodb sucks" blog post six months from today :)

[19:00:34] <def-lkb> (only one core). What is the common setup if I want to use multiple cores?

[19:02:07] <kali> def-lkb: you may have rush into buying this box. writes have a per-db lock. you can improves thing by splitting your data across more than one database, or sharding.

[19:02:19] <kali> def-lkb: how many writes/s are we talking about ?

[19:11:17] <def-lkb> kali: the common workload is no write at all for around ~10 minutes, then around 60000 documents are written in a burst, nothing for 10 minutes, and so on

[19:12:38] <kali> mmmm... what kind of write ? insert ?

[19:13:01] <def-lkb> kali: maybe explaining the use can helps: there is small core dataset (around ~1GB), which are processed by external tools, generating a few gigabytes at a time after a few minutes.

[19:14:23] <def-lkb> only on first iteration. then, its mostly updates

[19:14:49] <def-lkb> so, I use upserts

[19:14:56] <kali> all right. by id ?

[19:15:40] <def-lkb> no

[19:17:02] <def-lkb> there is an index on a specific row, but I am not using mongo's _id during upserts.

[19:17:19] <kali> okie. are you insert growing documents ?

[19:17:55] <def-lkb> nop, it's almost always the same size

[19:18:15] <kali> ok. no low hanginf fruit :/

[19:18:50] <kali> is your memory bigger than the dataset, index included ?

[19:19:03] <def-lkb> Do you think upserting by _id would be better? If needed, doing a first pass to map data specific index to mongo _id is possible.

[19:19:10] <def-lkb> Yes, and yes.

[19:19:21] <kali> nope, i was hunting for a missing index

[19:19:49] <kali> inserting by _id may marginally help as there will be only one index to maintain

[19:19:54] <def-lkb> Ok, I'll do some stats to make sure all indexes are relevant and correctly used.

[19:20:44] <def-lkb> And you don't think that sharding on the same box will help?

[19:21:12] <kali> it may help

[19:21:37] <def-lkb> Ok, I'll try. Anyway, many thanks for your help :)

[19:21:47] <kali> you're welcome

[20:23:40] <tpanum> Does anybody have experience with the erlang driver?

[20:30:28] <rkgarcia> tpanum: no =(

[21:05:51] <McSorley> Hey #mongodb, looking for some advice on importing some large datasets into Mongo. I have a number of large CSV files ranging in size from 15-20GB that i would like to load in, just curious to see if anyone can offer any advice/strategies for dealing with some large data files..?

[21:07:16] <ranman> McSorley: mongoimport can handle csv files pretty well: http://docs.mongodb.org/manual/reference/program/mongoimport/

[21:07:29] <ranman> http://docs.mongodb.org/manual/core/import-export/

Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 3rd of March, 2014