pmxbot IRC Log Viewer

[02:42:24] <vsmatck> Is there a way to have a selector get the size of a field (a array field) instead of returning the field? Do I need to have a counter in my doc to do this?

[02:46:23] <TkTech> You do.

[02:46:32] <TkTech> $inc as needed when you modify the array.

[02:47:09] <vsmatck> TkTech: Ah thank you sir. I searched the docs a lot but wanted to be sure. I'll do a counter.

[04:21:38] <circlicious> hello, i am facing some problems.

[04:22:10] <circlicious> i am making an insert, and its not happening. var_dump() on the insert query gives me this http://pastie.org/4346663 - i guess there's nothing wrong, yet no inserts?

[04:22:39] <circlicious> also sometimes it works, sometimes it doesnt on other pages. is mongo reliable :S

[04:27:04] <circlicious> $db->lastError() dumps the same thing

[04:27:21] <circlicious> :(

[04:28:13] <circlicious> someones plzz....

[05:49:19] <taipres> where does mongodb shine?

[05:55:38] <circlicious> no idea, right now, its not shining for me

[05:57:06] <taipres> :\

[06:00:04] <taipres> well what problem does mongod solve that other relational dbs haven't?

[06:00:22] <taipres> trying to find its focus, postgresql for instance is data integrity, mysql is a mix but mostly speed

[06:04:07] <Lawnchair_> mysql is mostly for rage

[06:08:22] <taipres> lol rage?

[06:09:04] <circlicious> lol

[06:09:10] <circlicious> can you help me Lawnchair_

[06:09:14] <circlicious> i need help badly tbh

[06:09:32] <Lawnchair_> with being mad at mysql?

[06:11:38] <circlicious> oh you just joined it

[06:11:46] <circlicious> my mongodb is not accepting writes for some reason

[06:11:53] <circlicious> no error thrown, i am using php

[06:11:56] <circlicious> can you help me ?

[06:12:47] <Lawnchair_> have you checked the logs to see what it says the error is?

[06:13:10] <circlicious> logs says nothing

[06:13:42] <circlicious> Lawnchair_: just these - http://pastie.org/private/fvuqwzekyabqgjgf9m8za

[06:17:10] <circlicious> basically all it says is [initandlisten] connection accepted from 127.0.0.1:52853 #12

[06:17:11] <Lawnchair_> I'd try a safe insert, see if it throws an exception, make sure it's actually not inserting

[06:17:20] <circlicious> i am always using safe insert

[06:17:37] <circlicious> the insert or db->;lastError returns this, http://pastie.org/4346663

[06:17:49] <circlicious> $coll->insert($data, ['safe' => true]);

[06:17:54] <circlicious> safe

[06:22:47] <Lawnchair_> you're doing a batch insert ya? don't know why off hand but one of the records isn't valid, so the entire batch fails to insert

[06:23:29] <taipres> is mongodb based off anything?

[06:27:34] <circlicious> i am not doing bantch inserts, dont leave

[06:28:32] <circlicious> god, someone please help, NodeX u there?

[06:43:35] <deoxxa> taipres: happiness and rainbows

[06:46:35] <circlicious> i was talking about same prboelm in ##php, there's a guy, he really hates mongo :P

[06:46:56] <deoxxa> ##php hates making sense

[06:48:36] <circlicious> maybe he has pointsa, and maybe i couldnt prove why i chose mongo over mysql for a part of my app

[06:48:46] <circlicious> but i really think it led me t develop my app faster is all i know

[06:49:04] <circlicious> and schema changes often, i needed little faster writes, mongo proved it. but now this bug :/

[06:49:49] <deoxxa> circlicious: run mongodb in verbose mode, check what it says

[06:50:30] <circlicious> deoxxa: ok, thanks, one sec

[06:50:53] <circlicious> deoxxa: should i stoip the service, then execute monogd myself?

[06:51:00] <deoxxa> that's what i'd do

[06:51:35] <circlicious> ok, done, let me test the app now

[06:53:12] <circlicious> [conn2] command cssdeck.$cmd command: { getlasterror: 1 } ntoreturn:1 reslen:67 0ms

[10:59:33] <jrm2k6> Hi guys, stuck on this TypeError: Cannot use 'in' operator to search for 'title' in sjfhewjf

[10:59:34] <jrm2k6> at String.IN (native)

[10:59:41] <jrm2k6> any idea, i googled it without any success

[11:03:53] <deoxxa> paste your query

[11:04:12] <deoxxa> (in a pastebin)

[11:05:04] <jrm2k6> ok one sec

[11:05:54] <jrm2k6> http://pastebin.com/P27i5cK1

[11:06:04] <jrm2k6> i put the schema and the request (using coffeescript)

[11:07:03] <wereHamster> jrm2k6: new Post({ title: 'arst', body: 'qwpf' }).save -> ...

[11:07:16] <jrm2k6> ok lt me check

[11:07:31] <wereHamster> also, what does Post have to do with authentication?

[11:07:37] <jrm2k6> even using coffeescript

[11:07:42] <jrm2k6> wereHamster: nothing just a test

[11:09:43] <jrm2k6> hm ok sounds there is some progress, but this is infinitely loading

[11:20:31] <jrm2k6> any idea why it stays stuck on save?

[11:20:44] <jrm2k6> loading indefinitely

[11:24:16] <jrm2k6> it is like the save doesnt find where to write the data or something

[11:28:11] <circlicious> if i store text in a document, how long of a text can it store?

[11:31:25] <jrm2k6> ok maybe a stupid question.

[11:31:43] <jrm2k6> when you have created one schema, you use it, but what happens when you call save?

[11:31:56] <jrm2k6> does it create automatically the collection files and stuff?

[11:32:06] <jrm2k6> i m still thinking relational database so

[11:36:29] <jrm2k6> nobody?

[13:04:02] <augustl> circlicious: document size is capped to 16mb iirc

[13:04:41] <augustl> jrm2k6: there's no such thing as creating a collection, it will be created for you

[13:05:07] <augustl> actually, for all I know there are APIs to explicitly create collections ;)

[13:05:25] <jrm2k6> ok

[13:05:51] <jrm2k6> augustl: can you check that really quickly? http://stackoverflow.com/questions/11701193/save-on-mongoose-schema-stuck-in-coffeescript

[13:05:53] <augustl> but it will be created for you

[13:06:01] <jrm2k6> i explained what is the problem exactly

[13:06:14] <jrm2k6> dunno if it is a config problem/ callback problem or something... really, totally stuck

[13:06:33] <augustl> jrm2k6: so the callback to Post.prototype.save is never calld?

[13:07:04] <augustl> also, are you listening to callbacks for errors on "connect" etc?

[13:07:17] <jrm2k6> nope

[13:07:23] <jrm2k6> really beginnoing using mangoose

[13:07:23] <augustl> try that then

[13:07:27] <jrm2k6> and coffeescript

[13:07:28] <jrm2k6> so..

[13:07:40] <jrm2k6> ok let me check how to do that

[13:11:59] <jrm2k6> ok

[13:12:08] <jrm2k6> augustl: i got Error connection closed

[13:13:20] <jrm2k6> hm

[13:13:56] <jrm2k6> how should i fix that

[13:14:08] <jrm2k6> does it mean that the url i gave is not the good one?

[13:28:22] <augustl> jrm2k6: probably yeah

[13:28:28] <augustl> jrm2k6: the default port isn't 3000 for example :)

[13:28:51] <augustl> what we're doing now is the biggest problem with stack overflow btw

[13:29:00] <augustl> quite often dialogue is required to answer a question

[13:54:56] <jrm2k6> yes

[13:55:03] <jrm2k6> i agrree wit you

[13:55:10] <jrm2k6> sorry for the delay

[13:55:41] <jrm2k6> so let me try with not 300

[13:55:45] <jrm2k6> what is the default port?

[13:57:51] <jrm2k6> still this error Error: failed to connect to [localhost:27017

[13:57:57] <jrm2k6> the default one is 27017 i guess

[13:58:02] <jrm2k6> thats what i found everywhere

[14:02:23] <augustl> jrm2k6: so is the server running at localhost?

[14:02:26] <augustl> or running at all?

[14:02:28] <jrm2k6> yes

[14:02:33] <jrm2k6> no

[14:02:33] <jrm2k6> not runnign at all

[14:02:40] <jrm2k6> i want to test it locally before deploying on heroku

[14:02:53] <jrm2k6> mongoose.connect 'mongodb://localhost/mydb', (err) ->

[14:02:53] <jrm2k6> if err

[14:02:53] <jrm2k6> console.log 'wtf '+ err

[14:02:57] <jrm2k6> sorry to paste it directly

[14:02:59] <jrm2k6> :(

[14:03:28] <augustl> wait, you don't have a mongodb server running locally?

[14:04:32] <jrm2k6> i guess, yes let me check

[14:08:19] <jrm2k6> arf brew install slow ahahah free wifi

[14:08:20] <jrm2k6> anyway

[14:17:47] <jrm2k6> augustl: sounds like it was the problem... i cleaned it up and install a new version

[14:22:34] <jrm2k6> augustl: hmm show collections doesnt return anything

[14:22:45] <jrm2k6> i guess that means there is a problem

[14:22:50] <jrm2k6> the save doesnt stay stuck though

[14:26:59] <jrm2k6> nvermind

[14:27:00] <jrm2k6> work

[14:27:16] <jrm2k6> thanks

[14:27:16] <jrm2k6> i will come back with new problems :)

[14:47:46] <sunshadow> Need help in choosing shard key

[14:48:12] <sunshadow> Could anyone guide me through this pls?

[14:50:24] <mids> tell us more

[14:51:26] <sunshadow> I have a URI shortening service

[14:52:01] <sunshadow> Now we have around a million entries of long url to short url maps

[14:52:42] <sunshadow> shortURL is in the format baseURL/<handle>

[14:53:03] <sunshadow> For each entry in the DB the <baseURL,handle> should be unique

[14:53:45] <sunshadow> The most common use case is lookup where the user enters a shortener URL and wants the corresponding long URL

[14:53:58] <sunshadow> Thus

[14:54:23] <sunshadow> We are currently using a single table to maintain this short url to long url

[14:54:34] <sunshadow> We are planning to make it to 10 tables

[14:55:03] <sunshadow> The shard key could be a hash/module of baseURL and the handle

[14:55:31] <sunshadow> Since baseURL and handle are unique for each entry inthe db

[14:55:58] <sunshadow> mids: Could you get the scenario?

[14:58:12] <mids> yeah

[14:58:47] <mids> hash sounds good, should prevent hotspots

[14:59:10] <mids> not sure why you'd need 10 tables though

[14:59:31] <mids> or do you mean have it sharded in 10 pieces

[14:59:32] <sunshadow> So could you suggest a good hash function for a compoud key <baseURL,handle> to shard across 10 tables?

[14:59:43] <sunshadow> Ya 10 pieces..

[15:03:03] <mids> I'd just pick md5 or sha1

[15:03:36] <mids> some non-cryptographic hash functions might be a lot faster if that is a concern

[15:05:31] <sunshadow> mids: Thanks.. I am pretty new to handling hash functions. Could you points a few references?

[15:06:15] <sunshadow> There is one more challenge, while creating a new <longURL,shortURL> entry, first the system has to check whether an entry already exists

[15:06:21] <mids> https://en.wikipedia.org/wiki/Md5

[15:07:25] <sunshadow> Thus when creating a new entry, control has to traverse through all the shards before confirming there are no duplicates

[15:07:59] <sunshadow> Is there any solution for that. I thought of spawning 10 threads one for each table. I know there must be better ways to handle this

[15:08:03] <mids> can ppl pick their own short url?

[15:08:14] <sunshadow> Yes

[15:08:18] <mids> ah

[15:08:28] <sunshadow> Thats the challenge

[15:08:52] <sunshadow> If ppl had no control, we could derive a logic between the longURL and the generated shortURL

[15:09:23] <mids> yeah or you could precompute a list of available ones and pick one from the list with findAndModify

[15:09:46] <sunshadow> Yeah.. exactly

[15:09:58] <mids> but for you; just let mongo deal with it

[15:10:03] <mids> do a findOne on the entry

[15:10:41] <sunshadow> Ok.. Let mongo take care of that part then..

[15:10:52] <mids> or rather; check with the md5, based on the shard key it will know which shard to query

[15:11:35] <sunshadow> md5 on the longURL?

[15:11:57] <sunshadow> The shard key is based on the md5 on <baseURL,handle> right?

[15:12:17] <mids> ah I see

[15:12:25] <mids> you dont want two short urls pointing to the same long one

[15:13:33] <mids> guess you could have another collection with the mapping the other way around

[15:13:35] <sunshadow> No.

[15:13:58] <sunshadow> Oh ya..

[15:14:05] <mids> and that you can shard as well

[15:14:43] <sunshadow> And what should me the fields in The other collection?

[15:14:52] <sunshadow> LongURL and ?

[15:15:24] <mids> <baseURL,handle> I guess

[15:16:14] <sunshadow> Ahh.. Ya. That would work.

[15:16:41] <sunshadow> So, I should make two db entries for each short URL creation.

[15:17:54] <mids> think so

[15:18:06] <mids> might also depend on your usage

[15:18:14] <mids> how many reads vs writes

[15:18:27] <sunshadow> Ya I get it. Thanks a lot mids

[15:18:39] <mids> maybe the scatter / gather query on the longurl wouldnt be too bad if ppl only create urls rarely and use them a lot

[15:19:28] <sunshadow> Currently its like 90,000 new shortURLs and 1,50,000 lookups every day

[15:20:10] <mids> 1.5M?

[15:20:28] <sunshadow> Nope.. its 150,000

[15:20:31] <mids> ah

[15:20:54] <mids> what happens if the longurl is already used elsewhere?

[15:21:06] <mids> do you just give them an error message; or do you tell them the matching shorturl

[15:21:18] <sunshadow> It returns the shortURL already present in db

[15:21:27] <mids> ic

[15:21:28] <sunshadow> the maching shortURL

[15:21:52] <sunshadow> And could you suggest a few non cryptographic hashes that are fast?

[15:22:52] <mids> CityHash comes to mind

[15:23:02] <mids> but with your numbers I am sure md5 is plenty fast

[15:23:27] <sunshadow> Thanks mids , I will look into them.

[15:23:45] <mids> good luck!

[15:23:59] <sunshadow> Sure. Thanks for the help.

[18:40:14] <sspy> totalSize() shows ~ 54 MB but files are occupying 512 MB. I know about preallocating, but it's unclear why all these files are full of data. What it can be ?

[18:43:00] <sspy> any thoughts ?

[19:45:05] <vsmatck> sspy: This may be useful. http://www.mongodb.org/display/DOCS/Excessive+Disk+Space#ExcessiveDiskSpace-DatafilePreallocation

[19:46:01] <vsmatck> If you wanted you could write a program to scan the file and tell you what % of bytes are equal to zero.

[19:46:50] <vsmatck> That may not be totally accurate.

[19:48:34] <sspy> it's a good idea

[19:49:31] <sspy> also will try to disable prealloc

Log file Viewer

Help | Karma | Search:

#mongodb logs for Saturday the 28th of July, 2012