pmxbot IRC Log Viewer

[00:05:19] <wfaulk> thanks; I'll assume fs corruption for now. would still like to have seen those errors passed back to clients in some way

[02:01:06] <thefirstdude> hi, what is a good web framework biult around php and mongodb

[02:02:23] <culthero> Anyone have any qualms with TokuMX?

[02:28:07] <talus46> greetings

[03:08:30] <calvinx> I am trying to return only specified fields but I don’t seem to be getting the expected results. (reference http://docs.mongodb.org/manual/tutorial/project-fields-from-query-results/#return-the-specified-fields-and-the-id-field-only )

[03:08:44] <calvinx> This is what I have done in my shell http://dpaste.com/1TEXXND

[03:09:22] <calvinx> I am expecting `db.roles.find({_id:1})` to rertun me the same results as above minus the “name”: “developer” key-value pair.

[03:09:41] <calvinx> why does it not do that?

[03:17:20] <rkgarcia> calvinx: with find you need use filter, and project

[03:18:05] <rkgarcia> calvinx: you need to use the db.roles.find({name:"developer"},{_id:1})

[03:18:19] <rkgarcia> .find( FILTER , PROJECT, OPTIONS )

[03:58:49] <calvinx> oic. got it.

[03:58:55] <calvinx> thanks rkgarcia

[03:59:13] <calvinx> is there a wildcard that I can use though?

[03:59:28] <calvinx> something like db.roles.find({name:”*”},{_id:1})

[03:59:37] <calvinx> Because I don’t really care about the value of name.

[03:59:53] <joannac> then leave it out?

[04:00:29] <calvinx> if I leave it out, no results come back

[04:00:36] <calvinx> I do want ALL the results

[04:00:52] <joannac> okay, i think i missed the first part of this question

[04:01:09] <calvinx> : I am trying to return only specified fields but I don’t seem to be getting the expected results. (reference http://docs.mongodb.org/manual/tutorial/project-fields-from-query-results/#return-the-specified-fields-and-the-id-field-only )

[04:01:14] <sachinaddy> Hi.. all .. i m need some small help.

[04:01:16] <joannac> but db.roles.find({}, {_id:1})

[04:01:22] <joannac> will definitely return all documents

[04:01:27] <sachinaddy> i m unable to connect to localhost of mongodb

[04:01:34] <calvinx> AH. that’s what I want to achieve. cool.

[04:02:31] <joannac> sachinaddy: um, okay. but the mongod is there?

[04:03:35] <sachinaddy> warning: Failed to connect to 127.0.0.1:27017, reason: errno:10061 No connection could be made because the target machine actively refused it.

[04:03:54] <joannac> firewall?

[04:04:02] <sachinaddy> yes joan. MongoD is there.

[04:04:17] <sachinaddy> I m on Windows. I disabled all firewall and antivirus.

[04:05:00] <joannac> my name is joannac

[04:05:12] <sachinaddy> sorry.

[04:05:17] <joannac> most clients let you tab-complete if you don't want to type the whole thing

[04:05:22] <sachinaddy> what can I do Joannac

[04:05:59] <sachinaddy> wow, I never new tab-complete works on IRC also. Thanks.

[04:06:05] <sachinaddy> *knew

[04:06:23] <joannac> verify it's actually listening on 27017?

[04:07:25] <sachinaddy> How do I verify that in Windows?

[04:07:55] <sachinaddy> There is no config file either.

[04:08:39] <joannac> how did you start the mongod process?

[04:09:54] <sachinaddy> I just installed full msi. Then from CMD (run as admin) locate to mongodb/bin folder and typed mongo.exe

[04:10:24] <sachinaddy> I can't find config file... It suppose to bind ip and port in the conf file.

[04:11:43] <joannac> http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/#start-mongodb

[04:12:10] <joannac> you've installed mongodb but haven't started the mongod.exe process

[04:15:35] <sachinaddy> Thanks joannac.

[04:28:05] <LouisT> so, what do you guys think the best web panel for mongodb management is?

[04:37:58] <cortexman> would it be smart to create huge numbers of collections to avoid having to have collections that point to collections that point to collections etc.

[04:43:29] <joannac> why would having large numbers of collections help?

[04:44:48] <sachinaddy> I m successfully able to import a csv into a collection. Is there a way to autoload the csv ? I mean, if I put a file in some location, mongodb acutomatically import it?

[04:45:37] <cortexman> joannac, suppose you have As, which contain links to Bs, which contain links to Cs, which contain links to Ds. suppose there are many of A, B, C and D. if there are only four collections, then it seems like you're just doing random access on disk.

[04:45:59] <joannac> sachinaddy: write a script

[04:46:18] <cortexman> suppose instead that for each A you multiplex the number of collections so that you have A_id, B_id, C_id, D_id. the latter scheme could be faster because mongo knows where all those collections live on disk.

[04:46:19] <joannac> cortexman: um, okay. so how would lots of collections fix that?

[04:46:43] <joannac> so you'd store one document per collection?

[04:46:50] <cortexman> yeah

[04:47:41] <cortexman> as far as i can tell it's not a problem of limits: http://docs.mongodb.org/manual/reference/limits/

[04:47:50] <joannac> sure it is

[04:48:00] <joannac> you'd be limited to about 12000 collections

[04:48:15] <joannac> also, you'd have to generate collection names on the fly

[04:49:04] <cortexman> how did you compute 12000

[04:49:31] <stefandxm> ive been about doing research of # collections that is not a performance loss compared to # documents in a collecion with semi decent indexes

[04:49:55] <cortexman> i get 2047 megabytes (max namespace size) / 123 bytes (size per namespace) = 17,450,691

[04:50:15] <joannac> The limitation on the number of namespaces is the size of the namespace file divided by 628.

[04:50:23] <joannac> A 16 megabyte namespace file can support approximately 24,000 namespaces. Each collection and index is a namespace.

[04:50:42] <cortexman> ok that's about 3.5 million namespaces

[04:50:53] <sachinaddy> joannac, would it be good if I use mongodb for dashboarding along with node.js? The data in mongodb would be huge.

[04:50:55] <cortexman> not infinite, but large

[04:51:31] <cortexman> stefandxm do you get good lookup performance with many collections? i.e., is it fast to find a collection by name?

[04:51:59] <stefandxm> so far its decent, but ivent put it under stress yet so cant tell

[04:52:13] <cortexman> my real question is, what's the normal way to handle the situation i'm in

[04:52:26] <stefandxm> my main concern is probably lockings of the namespace (collection?)

[04:52:42] <cortexman> collection A contains a bunch of links to random documents in collection B, B contains links to random Cs, Cs contain random links to random Ds, etc.

[04:52:48] <cortexman> that sounds like lots of thrashing on disk.

[04:53:15] <joannac> Not unless it won't fit in RAM

[04:53:46] <stefandxm> cortexman, i have approx the same issue

[04:54:26] <stefandxm> our data layer is kindof like a file system and the "files" often have random links to other "files"

[04:54:54] <stefandxm> currently each folder has a collection

[04:54:58] <cortexman> is that why you are experimenting with lots of collections?

[04:55:05] <stefandxm> yeah

[04:55:18] <stefandxm> since, the data layer is transparent to other databses aswell

[04:55:22] <stefandxm> including ldap

[04:55:34] <stefandxm> and in ldap you really create "folders" fast

[04:56:23] <stefandxm> so the alternative is to either use lookup documents for path->collectionname or always ensure a collection per folder

[04:57:02] <stefandxm> currently we do the latter

[04:57:16] <stefandxm> but it might be a bad move as the indexes will often be the same

[04:57:24] <stefandxm> so you end up with collection+index

[04:58:10] <cortexman> ok, thanks for the input. i'm not sure what i'm going to do.

[04:58:26] <stefandxm> me neither :)

[04:58:40] <stefandxm> ill implement both i guess and try to figure out what works best in practice

[05:00:22] <stefandxm> in our situation we also have a few other hints that gives us more options (ie even more test cases)

[05:00:36] <stefandxm> since the documents represents classes (with inheritance)

[05:00:57] <stefandxm> so most likely we will want approx the same indexes for each class

[05:01:06] <stefandxm> and we might want to inheritate indexes somehow

[05:01:36] <stefandxm> which would imply (imo) that we would want to have collections based on types rather than path

[05:10:10] <noose> using mongo-connector to sync into Elasticsearch. i can see a lag. how can I set sync frequency

[05:10:25] <noose> in mongo-connector

[05:12:06] <LouisT> with MMS, is there a way to change the group name?

[05:16:20] <garmet> Hello! I trying out tagRanges. But all data ends up at Shard2. Anyone know what I am doing wrong? http://pastebin.com/vch742rG

[05:19:06] <joannac> split your chunk at 2000

[05:20:06] <joannac> also, I hope this is a test instance

[05:24:28] <garmet> joannac, so you always have to split? It is for testing, is it that bad?

[05:26:00] <joannac> yes, you do need to split it. those tag ranges both cover the same chunk right now

[05:26:20] <LouisT> what.. can i not remove a group on MMS? it says i have to remove all users and hosts, to do so but i can't remove myself from the group?

[05:27:08] <joannac> and yes, unless you're generating random _ids, your shard key is monotonically increasing

[05:27:15] <joannac> which means no write distribution

[05:27:31] <joannac> LouisT: open a MMSSUPPORT ticket and someone will help you remove it

[05:28:38] <garmet> ok, I'll look that up. Although I have managed to put documents on the correct shard, it's just that the ranges didnt exactly match my interpretation of min-max range.

[05:28:43] <joannac> LouisT: offhand, where does it say that? I thought we fixed all our docs, let me know if we missed one and I'll fix it up

[05:29:25] <garmet> joannac, I dont per say want an even write distribution. Just a segmentation of data between physical machines.

[05:29:29] <LouisT> https://mms.mongodb.com/help-hosted/current/core/user-management/#remove-group

[05:30:10] <joannac> LouisT: that's the onprem MMS docs. You're running onprem?

[05:30:30] <LouisT> oh, probably not, woops

[05:31:07] <garmet> joannac, would it be better that I create multiple collections instead? one for each machine and apply the full _id range as tagRanges?

[05:31:35] <joannac> garmet: I'm not sure I understand what you mean

[05:33:19] <garmet> Well what I want to do is to partition data to a specific machine depending on a certain property. Not read/write balancing.

[05:33:50] <noose> great if i can get help on mongo-connector

[05:33:52] <noose> anyone?

[05:38:28] <LouisT> hm, i assume i can't use my mms account with jira..?

[05:39:01] <garmet> joannac, but I guess split/pre-splitting is the answer? Could one option be to reduce chunk sizes? If so, how can I know what is appropriate?

[05:40:48] <ani_stark> hii

[05:41:48] <ani_stark> Anyone can guide me to get my mongodb to run my system ?

[05:50:53] <joannac> LouisT: you sure can! same credentials as MMS

[05:51:17] <joannac> garmet: you could... not sure how reducing chunk size is relevant though

[05:51:20] <LouisT> yea, i realized that now, i was apparently forgetting the last char in my password like a noob

[05:51:27] <joannac> ani_stark: sure, but more details needed

[05:52:52] <ani_stark> @joannac: My mongodb not setting up after installation. First I'm getting this warning

[05:52:54] <ani_stark> ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000

[05:53:41] <ani_stark> on trying to set-up the dbpath

[05:54:12] <joannac> that's a ulimit problem http://docs.mongodb.org/manual/reference/ulimit/

[05:54:18] <joannac> and that's a warning, not an error

[05:55:32] <ani_stark> yes That's a warning.. so I ignored it. But then it shows connection:0

[05:55:49] <joannac> what shows connection:0?

[05:56:05] <ani_stark> [clientcursormon] connections:0

[05:56:17] <joannac> in the logs?

[05:56:40] <ani_stark> in the terminal during executing same cmd.. yes

[05:57:34] <joannac> that probably means you're not redirecting to a log file

[05:57:49] <joannac> that's also not necessarily a problem

[05:58:31] <ani_stark> well I installed using brew.. N gave 777 permission to /data/db as mentioned in doc

[05:58:45] <ani_stark> then while running mongod it gives this err too

[05:59:04] <joannac> what error?

[05:59:26] <ani_stark> exception in initAndListen: 10309 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied

[05:59:49] <joannac> ls -la /data/db

[05:59:53] <joannac> check the permissions

[06:00:41] <ani_stark> total 163840

[06:00:41] <ani_stark> drwxrwxrwx 7 root wheel 238 Jun 26 20:10 .

[06:00:41] <ani_stark> drwxr-xr-x 3 root wheel 102 Jun 26 20:07 ..

[06:00:41] <ani_stark> drwxr-xr-x 2 root wheel 68 Jun 26 20:10 _tmp

[06:00:41] <ani_stark> drwxr-xr-x 2 root wheel 68 Jun 26 20:56 journal

[06:00:41] <ani_stark> -rw------- 1 root wheel 67108864 Jun 26 20:11 local.0

[06:00:42] <ani_stark> -rw------- 1 root wheel 16777216 Jun 26 20:11 local.ns

[06:00:42] <ani_stark> -rwxr-xr-x 1 root wheel 0 Jun 26 20:56 mongod.lock

[06:00:45] <ani_stark> is this ok ??

[06:01:06] <joannac> next time pastebin

[06:01:09] <LouisT> is there an MMS monitoring agent for freebsd?

[06:01:15] <joannac> also, are you running mongodb as root?

[06:01:18] <joannac> if not, no

[06:02:06] <ani_stark> I gave permissions so that i dont ve to run it in root :/

[06:02:21] <joannac> well, your mongod.lock is 755

[06:02:31] <joannac> not writable by anyone other than root

[06:02:56] <joannac> and your local.* files are 600

[06:03:17] <ani_stark> Alryt let me try with sudo

[06:04:33] <ani_stark> does it normally take so much time starting ?

[06:05:53] <joannac> depends

[06:05:58] <joannac> what do the logs say it's doing?

[06:06:28] <ani_stark> last line says connection:0

[06:06:42] <joannac> that means it's started

[06:11:22] <garmet> joannac, splitting certainly did the trick! thank you. Now I just need to understand pre-splitting ;)

[06:19:39] <ani_stark> Thanx @joannac :D Great help from ya :D

[09:49:47] <pgustafs> ping mongodb, i want to find all documents in a collection where to fields is identical

[09:52:47] <pgustafs> is it possible to group in two fields ??

[10:18:40] <Nomikos> pgustafs: http://stackoverflow.com/questions/14770170/how-to-find-mongo-documents-with-a-same-field

[10:34:38] <rasputnik> just ran movePrimary to shift a fairly small (20Gb odd) db from one shard to another (it's unsharded data)

[10:34:52] <rasputnik> now both secondaries in the replica set are RECOVERING

[10:35:21] <rasputnik> and getting lots of 'replset not trying to sync .... it is vetoed' type errors

[10:36:54] <rasputnik> also says it's too far behind to catch up, which is a bit odd because the oplog goes back 50-odd hours and it only happened about 45 minutes ago

[11:04:43] <pgustafs> Nomikos, thanks, but i only wants the docs where both fields are identical, that query finds the document where one or two fields is identical

[11:20:08] <Industrial> Hi.

[11:21:33] <nalum> Hello all.

[11:25:36] <nalum> I'm trying to set up mongodb ssl and am having trouble. When I start it with /opt/mongodb/bin/mongod --config /etc/mongod.conf I get a warning about missing the sslCAFile parameter and then it exits. Is this required? I didn't think it was when using a self signed cert.

[11:25:43] <nalum> I'm running 2.6.3

[11:40:40] <nalum> Figured it out it was permissions on the pem file, though now when I run it as service mongodb start it's not starting (wasn't before either) and it's not throwing any errors that I can see. I'm tailing the log file but that's not showing any activity. Would it log anywhere else?

[11:53:04] <nalum> Okay, figured that out, it didn't have access to the mongod command. Now I'm getting an error about permissions on the pem file again, I think they are correct the pem file is owned by the mongodb user and group and the user has read write access to it, group and other have no access to it. I'm getting Permission Denied, which is leading me to believe that the user is not correct.

[12:03:23] <nalum> Got it sorted the folder /etc/ssl/private only allows root access once the pem and key files were moved out of there it all worked.

[12:46:05] <ArSn> Is there a way to use index hinting for multikey indexes?

[13:35:53] <cheeser> new mongo-hadoop hot out of the oven: https://github.com/mongodb/mongo-hadoop/releases/tag/r1.3.0

[13:49:02] <ernetas> Hey guys.

[13:49:42] <ernetas> We recently reseted one of our MongoDB servers. It now does not get to PRIMARY status, although rs.status() on other nodes show it as primary.

[13:50:05] <ernetas> http://pastebin.com/raw.php?i=EYBqaHZL - this is what we get. What could be the issue?

[14:00:27] <q85> ernetas: you restarted the server?

[14:14:27] <ernetas> q85: yes.

[14:15:01] <q85> what is the replication status of the secondaries?

[14:15:15] <ernetas> They were down for way too long for me to failover to them.

[14:15:53] <ernetas> But rs.status() shows "still initializing" next to my main server. What's struggling is that nothing is happening.

[14:16:03] <q85> But restarting the primary would've initiated a failover situation.

[14:16:21] <ernetas> No, my failover is only for backups.

[14:16:31] <q85> So they are hidden?

[14:16:40] <q85> or priority 0

[14:16:43] <ernetas> Yes, and have 0 voting rights and priority 0.

[14:16:54] <kali> ernetas: what do you mean "reseted" ?

[14:17:02] <q85> you want them to have voting rights

[14:17:07] <ernetas> kali: as in pulling the power cord and powering back on.

[14:17:20] <ernetas> q85: that's not the point now here, is it... :)

[14:17:30] <kali> ernetas: have you changed IP, hostname in the process ?

[14:17:38] <ernetas> kali: no.

[14:18:58] <q85> what is in the logs after it booted back up?

[14:19:08] <kali> ernetas: yeah. i suspect it is re-syncing

[14:20:05] <q85> kali I think it was primary prior to shutting down and it is the most current member.

[14:20:29] <q85> Also, another member hasn't taken primary, so there isn't anything to resync to.

[14:21:12] <kali> ha. hidden. sorry for not having read everything

[14:21:22] <ernetas> kali: the failover nodes are delayed and actually were down when this whole mess happened.

[14:21:25] <ernetas> q85: http://pastebin.com/raw.php?i=r9ymZHbF

[14:23:04] <kali> ernetas: mmm interesting setup

[14:24:58] <q85> total of 2 members in the set?

[14:25:04] <ernetas> q85: yes.

[14:25:07] <q85> k

[14:25:17] <ernetas> I wouldn't mind recovering it as a standalone either...

[14:26:36] <q85> have you tried reconfiguring it with rs.reconfig()?

[14:27:06] <kali> ernetas: standalone should be easy: start it without the rs option. I advise you kill the other server or set a different port to avoid creating more chaos

[14:28:59] <ernetas> kali: I then get "WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset"

[14:29:45] <kali> ernetas: yeah, that warning makes sense :)

[14:29:57] <kali> except for the plural form, maybe :)

[14:30:15] <kreantos> hi, is this the right channel also for morphia related questions?

[14:30:31] <cheeser> sure

[14:32:02] <kreantos> cool, so i have a java-object, but I'm not mapping all the elements of a document to it. do i have to map all the elements in order to use the datastore.save method?

[14:33:09] <cheeser> no, by default morphia will use the java field name for the document property name.

[14:34:07] <kreantos> no, i mean i have not all properties of a documents as java fields

[14:34:47] <cheeser> oh, there's more in the db than in your java class.

[14:34:50] <kreantos> as i am using an already existing collection which has a lot of overhead i don't need right now

[14:34:54] <kreantos> exactly

[14:36:00] <cheeser> currently, i think you'll need to map them all. morphia will overwrite the document on save(), iirc

[14:36:45] <kreantos> oh :-(, this is what i feared

[14:37:05] <KushS> I am using mongoose for my node app. Earlier I used it to connect to a mongolab db but now I am using the same code to store data on my local db but the it does'nt seem to register the schema I created. Should I change something else apart from the db url ?

[14:39:54] <kreantos> cheeser: thanks for the info

[14:40:30] <cheeser> np

[14:40:34] <cheeser> sorry the news isn't better.

[14:40:47] <cheeser> this issue came up the other day, though, so you're not the only one needing this.

[14:42:33] <kreantos> yeap, would be awesome

[14:43:08] <kreantos> this means no a lot of work -.-, but better this way than woring with dbObjects

[14:43:22] <kreantos> -woring+working

[14:43:43] <cheeser> you might write a simple test to confirm, though.

[14:43:48] <ernetas> kali: ghm, seems like I'm back... The first time I tried to launch it as a single node, I couldn't even "show collections" in local database, was stalled completely. Thanks (for motivation to try it again :D )!

[14:44:13] <cheeser> i'll take a look. i know we do something different if the id field is already populated. maybe we already issue an update with $sets in that case.

[14:44:19] <cheeser> if not, that's what we *should* be doing. ;)

[14:44:38] <cheeser> if(id exists) upsert();

[14:46:07] <cheeser> we're halfway there.

[14:46:22] <cheeser> that logic mostly exists but only for versioned entities.

[14:47:22] <kali> ernetas: that's not good. it might be the right time to check that your disk and file system are not misbehaving

[15:08:41] <Chaos_Zero> I have a single mongo instance running on a web server. I would not open the port at all execpt the manager would like to admin outside of ssh with a gui tool. I just need one admin user to connect to this database. How do I create this user?

[15:17:55] <cozby> in a replica set, if the primary gets overloaded with traffic/requests and becomes unavailable an election will occur and a new master will be elected, but what if that traffic blast doesn't stop

[15:18:16] <cozby> wouldn't that cause a vicious cycle of elections

[15:18:24] <cozby> until the request die down?

[15:19:37] <cheeser> potentially, i'm sure.

[15:19:40] <cozby> my main concern is having single point of failure - the master. Yes replica sets fail over but then that single point gets shuffled and the same thing can occur to the next instance

[15:19:57] <cozby> cheeser: then how do people scale mongo?

[15:21:08] <cozby> does sharding alleviate the example I provided? Isn't sharding more of a solution to scaling data not really requests?

[15:22:49] <cheeser> sharding is part of it, yes

[15:30:10] <nalum> Hey, I've setup mongodb with ssl and it seems to be working fine except that it doesn't start with system boot. Any one got pointers on what the issue could be?

[15:37:42] <cozby> nalum what flavour of linux are you running?

[15:39:04] <nalum> ubuntu 14.04, I have copied https://github.com/mongodb/mongo/blob/master/debian/mongod.upstart to /etc/init/mongodb.conf

[15:56:23] <grosbob> hello

[15:56:43] <grosbob> what can i do for the dot in key please ? i need to store for example hostname/fqdn :(

[15:57:04] <cheeser> wrap it in quotes

[15:57:55] <grosbob> that's it ? oh okay thx

[16:01:18] <sellout> If I have two map/reduce jobs that both {out: {reduce: x}} to the same “x”, can they be run in parallel?

[16:03:13] <cheeser> i would expect not

[16:03:43] <sellout> Yeah, figured I was expecting a bit much, there ;)

[17:33:15] <juergen_thomann> Hi, Does anyone of you have some links for where I can look how mongodb does shuffle and sort as it is called at hadoop? I haven't found anything in that direction so far.

[17:42:32] <tscanausa> juergen_thomann: are in map reduce?

[17:42:38] <tscanausa> as in map reduce*

[17:44:20] <juergen_thomann> tscanausa: yes. I try to understand how mongodb handles for example to big results which do not find memory during map

[17:45:13] <tscanausa> it uses temporary collections

[17:48:10] <tscanausa> I lied it uses temporary files if you allow it to

[17:50:49] <juergen_thomann> tscanausa: If I now have a sharded mongodb database, does this still work? at the end the same keys have to be reduced somewhere. With collection I could imagine it with a sharded collection but on disk temporary files have to be transmitted in another way.

[17:51:28] <tscanausa> so each shard will do its own work then the work will be merged together at the end

[17:53:50] <sigurding> guys, is there any option to have db.collection.insert({ with “$binary”…}); like for example with ObjectId?

[17:57:07] <juergen_thomann> tscanausa: one last question: Does this last merging happen at a mongos or on one shard when the results are written to a new collection

[17:57:32] <tscanausa> mongos

[17:57:56] <juergen_thomann> tscanausa: OK, thanks!

[17:58:02] <tscanausa> sigurding: I do not understand your question

[17:58:28] <sigurding> tscanausa: will give you an example just a moment

[18:00:29] <sigurding> tscanausa: solved it….I had db.collection.insert({ … data" : { "$binary" : ….}});, which threw an exception. Had to replace it through: BinData()

[18:05:02] <tscanausa> awesome

[18:11:23] <Tug> Do you guys know a simple way to sync my production database locally? I was using db.cloneDatabase('REMOTE_SERVER'); but it usually fails now that my db is a bit large

[18:11:59] <cheeser> i use mongodump and rsync

[18:12:23] <cheeser> but my data is non-critical and static that time of night.

[18:13:25] <Tug> I should try that again but it's quite a pain

[18:13:46] <cheeser> cron++

[18:14:35] <cheeser> you could try a hidden replica set that you could lock, copy data files, unlock

[18:15:00] <Tug> yeah that's what I was looking for

[18:15:11] <Tug> but last time I tried I think I messed up the prod db

[18:15:31] <Tug> if someone wrote a script for that

[18:16:49] <tscanausa> Tug

[18:17:55] <tscanausa> Tug: you could read the oplog and send it your local instance. it how i keep my staging environment up to date with 10% of the production data.

[18:19:22] <Tug> tscanausa, so you're suggesting having a script on the prod server which connects to your local computer and watches for changes

[18:20:02] <tscanausa> just a suggestion. but I generally do non standard things

[18:22:38] <Tug> tscanausa, yeah it's not bad. Problem is sync is a punctual task for me. I usually copy it for debug only and my computer is not always online

[18:23:31] <Tug> problem with mongodump is the amount of data it duplicates + the extra time induced

[18:23:54] <Tug> in fact I don't have the space for twice my prod db on my local HDD

[18:24:04] <Tug> I should buy a new hard drive though

[18:24:19] <tscanausa> Tug: you could keep a short connect time and if it fails to write it fails to write

[18:26:09] <Tug> tscanausa, ok but then when my computer goes back online it only sync new data right ?

[18:26:38] <tscanausa> depends on how you write your script but in simpliest terms yes

[18:27:53] <Tug> It would be great if I could use mms' backup technology locally

[18:30:27] <Tug> Or have cloneDatabase doing incremental update :)

[18:30:35] <Tug> s/doing/do/

[18:45:50] <cheeser> onprem++

[18:48:51] <saml> http://www.onprem.com/team ?

[18:52:02] <cheeser> mms on premisis

[19:03:42] <Tug> cheeser, I see. So I have to grow my business first :)

[19:04:04] <cheeser> venture capital to the rescue!

[19:04:39] <Tug> sure ^^

[19:05:03] <ekristen> if I have a collection.find and I am using a cursor to loop over all the entries, and it was looking for state: 1, if an entry that hasn’t been looped over yet gets updated by another process to state: 2, will the first find still iterate over it?

[19:10:39] <q85> ekristen: yes

[19:11:09] <ekristen> q85, awesome, that is what I was hoping :)

[19:11:12] <q85> ekristen: I missread that. no

[19:11:15] <ekristen> damn

[19:11:26] <ekristen> so the cursor is real-time essentially

[19:11:42] <ekristen> if it hasn’t reached a document yet and that document gets updated then the next find won’t find it

[19:11:45] <q85> better stated, there are no transactions

[19:12:30] <ekristen> q85: so let me put it to you another way

[19:13:16] <ekristen> if I have 1000 documents with state: 1 and I start a find and start iterating over the documents using a cursor and another script adds another 1000 documents with state: 1 will the first find find the new ones, or will I have to start another find

[19:14:41] <q85> Hmm, I think in that case it depends on where the documents are added and the query pattern. In short, you could find the new documents.

[19:15:08] <stefandxm> you could also get the same document twice or more

[19:15:16] <q85> ^^

[19:15:16] <Tug> ekristen, there is an option to keep the cursor open I think

[19:15:22] <Tug> tailable cursor

[19:15:50] <ekristen> ah tailable cursor, let me see if I’m using that

[19:15:53] <ekristen> I’m using the Node.JS client

[19:16:59] <q85> Tug a tailable cursor would heed the same results.

[19:17:52] <Tug> q85, I believe you, never tried it

[19:18:57] <q85> I supposed it depends on if ekristen wants to see the new inserts.

[19:19:05] <ekristen> I really don’t want to

[19:19:15] <stefandxm> then you need to use snapshot

[19:19:23] <stefandxm> but it will fail on sharded environments

[19:19:44] <stefandxm> so then i believe you are back to making a manual back-end lock (in your application)

[19:20:25] <q85> the $snapshot wouldn't prevent the cursor from iterating over the new insert though.

[19:20:55] <stefandxm> really?

[19:21:36] <q85> The would be new documents, not documents that have been reread

[19:21:42] <q85> They*

[19:21:57] <q85> So the cursor would still iterate over them.

[19:22:02] <stefandxm> ah

[19:22:06] <stefandxm> its not really a snapshot

[19:22:10] <sigurding> guys what does the error message: db side execution is disabled mean?

[19:22:25] <stefandxm> its just an iterator hack

[19:22:26] <stefandxm> i see

[19:22:39] <q85> nm, you are correct

[19:23:20] <q85> wait

[19:23:22] <q85> grr

[19:23:30] <q85> here is the docs, "Even in snapshot mode, objects inserted or deleted during the lifetime of the cursor may or may not be returned."

[19:23:54] <q85> not very descriptive

[19:23:58] <stefandxm> yeah

[19:24:14] <stefandxm> but as its just an iterator

[19:24:34] <stefandxm> if the new document is inserted with an id sorted > current iterator position it would be returned

[19:24:36] <q85> Basically, mongo doesn't have transactions. So you can't guarantee these things.

[19:24:38] <stefandxm> if its sorted < it wont be

[19:24:58] <stefandxm> q85, yeah. i thought the snapshot actually did a snapshot

[19:25:07] <stefandxm> wich would be easy to make even in a non-transactional database

[19:25:12] <q85> ah, so you could use the mongo _id!

[19:25:23] <stefandxm> thats what snapshot does

[19:25:42] <stefandxm> and you could use any other index aswell to make it iterate in an unique order

[19:26:55] <stefandxm> but its not really useful for this kind of problems, imo it sounds like an optimization trick

[19:27:37] <q85> yeah

[19:28:13] <stefandxm> iam not sure if theres a find into in mongodb but thats whats needed

[19:28:24] <stefandxm> i think theres something like that in the aggregation framework

[19:28:39] <stefandxm> then you could select whatever you need and put into a temporary collection

[19:28:47] <stefandxm> all in mongodb and then set a cursor onto that one

[19:29:20] <stefandxm> of course it wont be cheap but.. if you have to do it will probably be better than trying to solve it in js ;p

[19:29:28] <q85> If you used $snapshot with $lte : {_id : <current_time as object id>} you could ensure that you don't read any new writes.

[19:30:26] <stefandxm> as long as they are sorted as that yeah. dunno how the mongo trees works for date times

[19:30:32] <stefandxm> i guess an int always would be tho

[20:24:13] <Tug> <cheeser> you could try a hidden replica set that you could lock, copy data files, unlock

[20:24:13] <Tug> ^^^ Does it matter if my replicaset has an even number of members while I synchronize the hidden member ?

[20:56:37] <tinix> is it normal for mongodump/restore to not work and cause all sorts of query errors?

[20:56:57] <tinix> for example, dumping from server and importing on local for development, just plain doesn't work.

[20:57:08] <tinix> tried not importing indexes and re-indexing, no luck

[20:58:27] <duckchat> mongoimport csv files. 4 millions rows. need to manipulate data after import. can it be done with javascript? is it fast?

[20:59:11] <stefandxm> 4 million rows.. how many columns?

[20:59:20] <duckchat> about 20

[20:59:26] <stefandxm> whats your max time?

[20:59:31] <stefandxm> (how long can you wait?)

[20:59:41] <duckchat> as long as needed.

[20:59:45] <stefandxm> then no problem

[20:59:52] <stefandxm> its not a large data set

[21:00:35] <duckchat> using ruby to update the data now. takes about 3 hours. would like something faster.

[21:00:49] <stefandxm> then "as long as needed" is not a very accurate number ;)

[21:00:58] <stefandxm> personally i would just parse the csv files myself

[21:01:09] <stefandxm> and create the mongo objects in whatever language + do the fixing

[21:01:20] <stefandxm> (or do you need the entire data set to do the fixing?)

[21:01:53] <duckchat> kinda. build relationships, etc. calculations and add new fields.

[21:02:19] <stefandxm> what is taking time? ruby pulling data or writing the data?

[21:02:28] <stefandxm> if its the write then changing language wont matter much

[21:02:29] <duckchat> writing it

[21:02:45] <stefandxm> you probably need to chunk it up and make larger writes somehow. with some kind of bulk operation

[21:02:56] <duckchat> that is an idea...

[21:03:36] <stefandxm> http://docs.mongodb.org/manual/reference/method/Bulk/

[21:03:52] <duckchat> is there some way to tell mongodb a field type during mongoimport csv?? importing as numeric and would like string

[21:04:12] <mchapman> Can anyone help with what seems to be a bug illustrated by this gist: https://gist.github.com/mchapman/078818d5229129c8cce5 ?

[21:05:09] <stefandxm> dukchat, do you really need the mongo import for csv files? csv parsing is like 5 lines of code in any language

[21:05:29] <mchapman> At least enough to say whether they think it is a mongodb, node driver or mongoose bug

[21:05:46] <stefandxm> dukchat, with only 4 Mrows (if you can keep it like that) i would have it loaded into memory and then written with bulk operations after having done the fixing

[21:06:14] <duckchat> stefandxm: well, the mongoimport is blazing fast

[21:07:03] <stefandxm> with the bulk operations in the driver i doubt it would make a difference

[21:08:31] <duckchat> stefandxm: thx. going to look hard at bulk update.

[21:09:00] <stefandxm> just make sure your driver has support for it :)

[21:09:13] <stefandxm> (that might force you to use another language)

[21:17:58] <mchapman> Running the query in mongo shell gives same response in 2.4 and 2.6 (in case anyone is interested)

[21:19:19] <chanced> does anyone know if I can rely on the integrity of order when it comes to embedded docs (received by node)

[21:20:24] <chanced> i need to allow for re-ordering and kind of hesitant to create a separate field on the container

[21:54:52] <richthegeek> so I keep hearing about TokuMX but it's always given me a sort of intangible "Don't touch this dodgy software" smell.... does anyone have opinions on it? Probably a biased place to ask this!

[21:58:51] <q85> richthegeek: I haven't used it myself, but I've also gotten the same vibe.

[21:59:25] <richthegeek> i've seen plenty of write ups that seem to be positive (http://blog.parse.com/2014/06/20/fun-with-tokumx/) but my gut keeps saying "no"

[22:00:06] <dArF1k__> hi

[22:00:30] <dArF1k__> i have question about native mongodb driver for nodejs

[22:00:40] <richthegeek> fire away dArF1k__

[22:00:51] <dArF1k__> for some reason i'm not able to do find by _id

[22:00:52] <q85> I remember reading something in some of Mongo's jira tickets comments section about people having issues with it. I don't recall any details though.

[22:01:13] <richthegeek> dArF1k__: put your code into pastie.org?

[22:01:30] <dArF1k__> sure, hold on

[22:02:04] <dArF1k__> http://pastie.org/9332408

[22:02:33] <dArF1k__> it works if I use any other field in query but not _id

[22:03:07] <richthegeek> verify what the query is: console.log(_id, {_id: new mongodb.ObjectID(_id)})

[22:03:56] <richthegeek> it's generally rare to have to case an ID as an actual ObjectID - either you're not using an auto-generated ID (ie your id is "bob") or the ID is already an object of that type

[22:05:07] <dArF1k__> 53add923b1ed176a363425be { _id: 53add923b1ed176a363425be }

[22:05:29] <dArF1k__> I'm passing _id here from api call

[22:05:41] <dArF1k__> so it should be a string

[22:06:08] <richthegeek> sure... nothing about it looks wrong! try running it from the CLI?

[22:06:23] <dArF1k__> i did.. and it worked

[22:07:08] <richthegeek> curious, very curious... i'm stumped

[22:07:10] <dArF1k__> i tried with new BSON.ObjectID(id) too and nothing

[22:08:04] <dArF1k__> anything i may be missing in connection options?

[22:09:31] <richthegeek> nothing i can think of! assuming the document exists and the query is correct then there's no reason it shouldn't return something.

[22:09:52] <richthegeek> as you've verified both of them, it's a thorough mystery

[22:10:29] <richthegeek> correct collection? does it work if you do findOne (which you should be doing with an ID query anyway)? try logging the output directly in the return (console.log(arguments))?

[22:11:10] <dArF1k__> I'm using same function to find all documents with account: XXX or _id: id... account: XXX works OK

[22:11:16] <dArF1k__> so collection is ok

[22:13:07] <aaronds> (Java API) Hi I'm having some problems trying to use a WriteResult - I'm trying to get the id of the document I just inserted. Sample: http://pastebin.com/4GnPHWqE could anyone point out what I'm doing wrong?

[22:13:36] <dArF1k__> hmm... findOne returned document

[22:13:56] <dArF1k__> does that mean find(_id: ...) does not return cursor but document?

[22:14:05] <richthegeek> find returns a cursor

[22:14:12] <richthegeek> find().toArray(callback) is the standard form

[22:15:02] <dArF1k__> find({_id:id})toArray(callback) - does not return document or array

[22:15:16] <richthegeek> https://github.com/mongodb/node-mongodb-native/blob/master/lib/mongodb/collection/query.js#L150

[22:15:22] <dArF1k__> findOne({_id: id}, callback) - did return it

[22:15:28] <richthegeek> that's the breakdown of findOne, it's just syntactic sugar around find()

[22:19:06] <richthegeek> completely OT: watching the Great Gatsby... does it stop being rubbish at any point?

Log file Viewer

Help | Karma | Search:

#mongodb logs for Friday the 27th of June, 2014