pmxbot IRC Log Viewer

[01:55:08] <nicoenarg> hey folks, i'm pretty new to mongodb and trying to figure out this problem that seems pretty easy on RDBMS for me but not sure how to go about it with mongo. I have a project management system where each project can have multiple users and each user can have multiple projects, each user for each project is assigned a projectrole. its this project role relationship that i am having a hard time figuring out because users and projects can

[01:55:08] <nicoenarg> exist without each other as well, i'm a little lost as to how to implement this and also as to how to then read it back efficiently

[01:56:03] <nicoenarg> example read would be: get projects with users and their project roles

[02:47:49] <bros> I have 8m docs in a collection already.

[02:47:52] <bros> What can I do to make this not suck?

[02:48:32] <Boomtime> nice problem description - what sucks exactly? 8 million is nothing

[03:23:49] <bros> Boomtime: http://i.imgur.com/Lc3rIqx.png

[03:23:56] <bros> That's Mongo profiling (different issue)

[03:23:59] <bros> Anything I can do here?

[03:29:05] <Boomtime> what am i looking at?

[03:29:24] <Boomtime> are you seriously using a debugger to try to optimize how you use mongodb?

[03:31:55] <Boomtime> .. ok, whatever the heck i'm looking at - what exactly are you observing that sucks? - ignore the profiler/profiling/logs/etc, unless there is a real world effect on something there is no problem

[03:32:17] <Boomtime> if you have a slow query, or you think response times should be faster, etc, let's start there

[03:32:18] <bros> Ignore the 8m doc comment

[03:32:26] <bros> The response times are fine out of the database

[03:32:30] <bros> getting it over the wire

[03:32:31] <bros> and into memory

[03:32:35] <bros> in javascript

[03:32:39] <bros> is apparently the issue

[03:32:45] <bros> "issue", there might not be much i can do about it

[03:32:51] <bros> I'm parsing a lot of data

[03:32:52] <Boomtime> yes, javascript sucks, use C if you performance at the lcoal side

[03:33:29] <Boomtime> that's the thing about scripted languages, if you actually start manipulating large volumes of data they break down really fast

[03:33:50] <bros> What's the second most performant language? I do a shit load of object related logic with the docs.

[03:33:56] <Boomtime> if you use them to just push instructions around, direct traffic with other optimized tasks, then they are fine

[03:34:29] <bros> C, Java, what's third I guess?

[03:34:47] <Boomtime> java and C# will probably be next, and likely equal once the JVM has warmed up

[03:35:11] <bros> Fourth is node, then Python?

[03:35:17] <bros> Fifth, sorry

[03:35:33] <Boomtime> node is just javascript

[03:35:44] <bros> I'm aware.

[03:36:14] <Boomtime> python is in the same bucket, it's data large manipulation is terrible - but that is expected, the language is intended to be easy to write and manipulate module libraries that do the heavy lifting

[03:36:28] <bros> How far off is Java from C?

[03:37:10] <Boomtime> well, it runs in a VM effectively, so it's technically interpretted, but it can be very good

[03:37:47] <bros> Boomtime: https://gist.github.com/brandonros/dcc253b13082e18825fabfcd50f3d784

[03:37:50] <bros> it would probably be no joke to write that in C

[03:40:09] <Boomtime> https://gist.github.com/brandonros/dcc253b13082e18825fabfcd50f3d784#file-1-js-L1-L104

[03:40:16] <Boomtime> why?

[03:40:28] <bros> why require mongdb?

[03:41:15] <Boomtime> no, why that line? that bizarre json diff'ing thing

[03:41:20] <bros> 104?

[03:41:36] <bros> oh, objectIds

[03:41:41] <bros> they suck

[03:44:29] <Boomtime> that tells me nothing

[03:44:38] <Boomtime> what are you trying to do?

[03:45:00] <bros> diff two collections

[03:45:03] <bros> one live, one old cache

[03:45:07] <bros> find out which documents are invalidated

[03:45:11] <bros> issue is, it's mainly 10k+ docs

[03:45:13] <bros> per collection

[03:48:25] <Boomtime> if only you had a version number or a last_update field or something useful like that in your schema - or even just a record of which documents have changed

[03:48:38] <Boomtime> how long is the oplog in production?

[03:48:38] <bros> I was thinking of doing that

[03:48:45] <bros> the oplog seem delayed, which sucks

[03:48:47] <bros> What woudl be really cool

[03:48:50] <bros> is if the mongo writeresult

[03:48:54] <bros> returned what _ids it updated

[03:49:03] <bros> I wrote hooks for my functions, but I think that a lot of my operations

[03:49:08] <bros> will update the last timestamp

[03:49:12] <bros> when the doc doesn't really get updated

[03:50:08] <Boomtime> you specified the query for which documents get updated - you know which documents got updated

[03:57:04] <bros> Boomtime: not entirely

[03:57:07] <bros> The update might be useless

[03:59:19] <Boomtime> the writeresult tells you the count of affected documents, and if you don't know ahead of time what documents either will be affected, or have no way of determining the effect after the fact, then either you don't care what the effect is or the effect is meaningless by definition

[03:59:46] <bros> I think a document hash would be extremely cool

[03:59:57] <Boomtime> yes, that would actually be useful

[04:00:01] <bros> How would I do that?

[04:00:07] <Boomtime> there is a collection hash command oddly

[04:00:10] <bros> I can involve redis if needed

[04:00:17] <bros> what about a query hash?

[04:00:41] <Boomtime> i think i've wanted a document hash before.. imma check if there is a feature request for it

[04:03:18] <bros> Boomtime: could I do it client side?

[04:03:25] <bros> It just gets awkward with updates/upserts, etc.

[04:06:46] <bros> might this be better served for a server side javascript function? hmm

[04:10:28] <sector_0> I'm defining a user model using the mongoose client for node, and I'm defining a userID field, should I simply merge the _id field with the userID field and make them one and the same thing?

[04:10:36] <sector_0> or would this pose a security risk?

[04:13:24] <bros> Boomtime: I have an idea

[04:13:30] <bros> I could have a collection called "hashes"

[04:13:49] <bros> With the following fields for each document: document_id, new_version_hash, old_version_hash

[04:13:58] <bros> On every write, make sure new_version_hash stays up to date

[04:14:16] <bros> Then, would I do an aggregation or a map/reduce to find the differing docs?

[04:15:24] <sector_0> p.s. the user would be able to see his/her own id, as well as other users

[04:15:48] <sector_0> ...not publicly, but it would get passed around to clients

[04:37:51] <Boomtime> @bros: if you're going to change your schema, then add an updated field - if you only want the field changed if certain fields are changed, then add those fields to the predicate for update

[04:38:15] <bros> updated isn't invulnerable to the "the timestamp changed, but the hash most likely didn't" which probably plagues my app

[04:39:56] <Boomtime> example -> update( { likes: "teddybears" }, { $set: { status: "cool", last_updated: $currentDate } } ) (or whatever the syntax is)

[04:40:12] <Boomtime> your problem is that you don't want last_updated to change unless there was a change to "status" right?

[04:40:42] <Boomtime> so -> update( { likes: "teddybears", status: { $ne: "cool" } }, { $set: { status: "cool", last_updated: $currentDate } } )

[04:41:02] <Boomtime> now it only changes documents whose status wasn't already 'cool'

[04:42:04] <Boomtime> you are in control of the predicates - by definition you know ahead of time what is about to change, so make your predicates depend on what you're about to change to know if a change is about to happen

[05:04:37] <bros> Boomtime: https://gist.github.com/brandonros/0840d245575b2a54dc8e48114f124267

[07:19:30] <vharsh> I can't start mongod process on fedora23

[07:19:32] <vharsh> https://paste.fedoraproject.org/377498/62859714/

[07:21:49] <php> vharsh, something is using port 27017

[07:22:02] <vharsh> How can I fix that?

[07:22:15] <vharsh> Can I ask mongodb to use another port?

[07:22:18] <php> Change the port, or use nmap -sT -O localhost | grep 27017

[07:22:24] <php> yeah

[07:22:39] <php> I think it's in the main configuration for mongo

[07:24:58] <vharsh> How can I stop 27017 from being used by another app?

[07:25:26] <php> Change the port for that app in its configuration

[07:25:41] <php> Run this command: nmap -sT -O localhost | grep 27017

[07:25:51] <php> It will tell you what is using port 27017 hopefully

[07:25:56] <vharsh> Didn't

[07:26:04] <php> What does it say?

[07:26:26] <vharsh> blank line

[07:26:36] <vharsh> I mean not even a newline

[07:27:28] <vharsh> I think it's got to do with SELINUX, I think I have allowed 27017 port

[07:29:48] <php> wouldnt be that

[07:29:53] <php> hmm

[07:30:15] <php> netstat -tulpn | grep 27017

[07:30:17] <php> Try that

[07:30:34] <php> wait no

[07:30:41] <php> dunno if that helps

[07:30:47] <php> yes, it does

[07:30:48] <php> go for it

[07:32:27] <vharsh> tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1027/mongod

[07:33:24] <vharsh> I think only mongod is using that port( but it is stoppped)

[07:34:37] <php> nope

[07:34:39] <php> LISTEN

[07:34:42] <php> it's listening

[07:34:43] <php> :D

[07:34:49] <php> Mongo is already started

[07:35:16] <php> To connect to the Mongo console, use: mongo

[07:35:30] <vharsh> Failed

[07:35:39] <vharsh> leemme send the output of mongo

[07:36:11] <vharsh> http://paste.fedoraproject.org/377506/30054146/

[07:36:17] <php> if you don't have important data on mongo, try this: kill 1027

[07:37:05] <vharsh> Killed ot already, I am a newbie. I am trying to learn a NOSQL db after having tried Oracle in college.

[07:38:03] <vharsh> And I am trying to become more proficient with linux. So I will try my best to install it in linux only.

[07:38:42] <php> Use: kill 1027... then try starting Mongo again

[07:38:45] <vharsh> Also I have downloaded Redhat7's RPM files, The 3.2 mongodb repo for fedora doesn't work

[07:39:23] <vharsh> http://paste.fedoraproject.org/377508/65630247/

[07:39:51] <php> ok

[07:39:56] <php> now you have a new thing to fix

[07:40:08] <php> You need to make your /data/db directory

[07:40:18] <php> Mongo is looking for it but cannot find it

[07:40:23] <php> 2016-06-11T13:00:46.421+0530 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating

[07:40:28] <vharsh> hmm.

[07:40:38] <vharsh> Can I change it to something in /var ?

[07:40:46] <php> Yep

[07:40:58] <vharsh> I mean creating a dir in '/' is something I'd like to avoid..

[07:40:59] <php> Just make sure its in its own directory

[07:41:07] <php> yeah, that makes sense

[07:41:22] <php> mkdir /var/mongodata/db/

[07:41:25] <php> try that out

[07:41:48] <vharsh> done

[07:42:58] <vharsh> It's still looking out for it in /

[07:43:00] <vharsh> http://paste.fedoraproject.org/377509/30424146/

[07:43:14] <php> You must configure Mongo to use the new folder

[07:44:50] <vharsh> I used the --dbpath flag, saw it in stackoverflow :) It worked :D

[07:45:31] <vharsh> I'll make an alias for it in .bashrc file, can you suggest a better/permanent way?

[07:55:25] <php> mongodb configs

[07:55:29] <php> probably in /etc/

[10:41:36] <php> cheeser, hey ther

[10:41:40] <php> += e

[10:41:41] <php> xD

[10:42:19] <php> Getting this error: https://sysvoid.co/u/SysVoid_2016-06-11_11-33-38.png

[10:43:04] <php> Code: https://gist.github.com/anonymous/fb582c15d9e8b2c3703acfd693633daa

[10:43:15] <php> Equal line lengths.

[10:43:18] <php> I am a happy man.

[11:25:10] <sinkmanu> hello, it is possible in javascript save the output of printjson to a variable?

[11:30:28] <sinkmanu> okay, i found tojson hehe

[12:13:21] <cippaciong> Hello, do you know if a well formed bson document can have any embedded document or array with size 0?

[13:03:35] <php> Getting this error: https://sysvoid.co/u/SysVoid_2016-06-11_11-33-38.png

[13:03:37] <php> Code: https://gist.github.com/anonymous/fb582c15d9e8b2c3703acfd693633daa

[15:09:26] <php> Getting this error: https://sysvoid.co/u/SysVoid_2016-06-11_11-33-38.png

[15:09:28] <php> Code: https://gist.github.com/anonymous/fb582c15d9e8b2c3703acfd693633daa

[15:09:33] <php> cheeser, able to help with this? :P

[18:50:05] <php> Getting this error: https://sysvoid.co/u/SysVoid_2016-06-11_11-33-38.png

[18:50:06] <php> Code: https://gist.github.com/anonymous/fb582c15d9e8b2c3703acfd693633daa

[19:44:29] <php> cheeser, sorry for all these pings, but you seem to be the only person that talks here

[19:44:49] <php> If you're there, I'd be thankful if you could help (assuming you're not busy)

[19:46:06] <Zelest> php, I don't know java, but I know php and take a wild guess, it's not about "" vs '' ?

[19:46:12] <Zelest> '$eq' vs "$eq" ?

[19:47:10] <php> nope

[19:47:16] <Zelest> Ah

[19:47:16] <php> In Java, '' would be char

[19:47:19] <php> 'c'

[19:47:19] <php> etc

[19:47:21] <Zelest> aah

[19:47:30] <Zelest> Well than, worth a try at least :)

[19:47:44] <php> Thanks anyway :)

[19:47:54] <Zelest> :)

[19:47:59] <php> Shocking how inactive it is here

[19:48:00] <kali> Zelest: java is like C: "" for string, '' for single characters

[19:48:09] <php> been here 12 hours, seen about 13 messages

[19:48:24] <Zelest> ahh

[19:48:49] <Zelest> Next question, what is pxlpvp? ;)

[19:49:00] <php> it is

[19:49:02] <Zelest> YAY!

[19:49:11] <php> In-progress network thingy

[19:53:29] <Zelest> :)

[19:53:42] <Zelest> kiddo is addicted to minecraft :)

[20:15:44] <php> Zelest, you are, or a child of yours?

[20:18:27] <Zelest> php, my son

[20:21:09] <php> Zelest, neat

[20:22:02] <Zelest> i'm just as addicted myself so... :P

[20:22:17] <Zelest> but yeah, seeing i'm raised by lego it makes me glad to see his creativity :)

[20:22:18] <php> ah

[20:23:08] <Zelest> usually not much of a pvp fan though

[20:23:25] <Zelest> (mostly because i'm the worst loser ever :P)

[20:23:57] <php> I'm not much of a PvP fan either

[20:24:04] <php> I just enjoy making things people will enjoy

[20:25:58] <Zelest> yeah :)

[22:23:13] <php> Getting this error: https://sysvoid.co/u/SysVoid_2016-06-11_11-33-38.png

[22:23:17] <php> Code: https://gist.github.com/anonymous/fb582c15d9e8b2c3703acfd693633daa

Log file Viewer

Help | Karma | Search:

#mongodb logs for Saturday the 11th of June, 2016