PMXBOT Log file Viewer

Help | Karma | Search:

#mongodb logs for Thursday the 4th of September, 2014

(Back to #mongodb overview) (Back to channel listing) (Animate logs)
[02:06:05] <ohshitdacops> So I have a strange problem. I would provide logs, but there are none to provide
[02:06:53] <ohshitdacops> So I have MongoDB installed on one of my EL7 development environments and have noticed that whenever I restart the box, mongodb refuses to start and requires a full reinstall
[02:08:10] <ohshitdacops> The little info I can provide: [BOX: "CentOS 7", PACKAGE: "yum, via mongodb repo", ERROR: "failed state, no logs"]
[02:08:21] <ohshitdacops> any additional info can be provided per request
[05:26:53] <royaa1> Which style of mongoose query should I be using? The first or the second? https://gist.github.com/anonymous/922459bdcac5482d3132
[05:31:53] <Boomtime> they are the same
[05:32:00] <Boomtime> which do you prefer?
[05:32:33] <catNode> Boomtime: I prefer the second one, but for some reason I have in my head that's an old way of doing it? I must've read that in documentation or on SO somewhere?
[05:32:57] <Boomtime> nope, the second one is merely making use of local helpers
[05:34:07] <Boomtime> the second tends to be eaier to read, construct, and maintain - the first one is more like what the server actually gets sent
[05:34:28] <catNode> Boomtime: that makes sense. thanks for the clarification
[05:34:54] <catNode> Boomtime: the first isn't quicker or more flexible than the other?
[05:35:29] <Boomtime> not really, the work to convert the second one is done on the client - but if that tiny amount of work affects you then you shouldn't be running a javascript interp in the first place
[05:38:28] <catNode> good point. thanks for the clarification
[09:12:31] <krion> damn !
[09:13:15] <krion> for those who follow my trouble, even with cleaning all /var/lib/mongo dbs, i still have trouble
[09:13:41] <krion> when i try to shard a database, it say "{ "ok" : 0, "errmsg" : "already enabled" }"
[09:14:26] <rspijker> config server?
[09:14:33] <rspijker> (I have not been following anything, btw :))
[09:14:59] <krion> i got two mongos , three config server and two vm shard
[09:15:35] <krion> i rm -rf var lib mongo on three config server and two shard
[09:15:49] <krion> not on the two first mongos
[09:15:59] <krion> that act as a "gateway"
[09:16:31] <rspijker> Well, sharding being enabled is stored in the cfg servers iirc. The mongos processes do cache the config, but you can invalidate that
[09:16:45] <krion> hum, interesting
[09:19:30] <rspijker> you can have a look at the config db
[09:19:41] <rspijker> when you connect to the mongos that is
[09:20:06] <rspijker> there is a “databases” colleciton in there
[09:20:29] <krion> hum...
[09:20:32] <rspijker> each document represents a db and will say whether or not it’s “partitioned”
[09:21:35] <krion> i only have admin config data if i do a show dbs;
[09:22:52] <rspijker> admin, config, and data?
[09:24:05] <krion> yes, on the config server
[09:24:16] <rspijker> use config
[09:24:27] <rspijker> db.databases.find({partitioned:true})
[09:24:31] <krion> alright, got it
[09:24:36] <rspijker> that will show you a list of DBs with sharding enabled
[09:25:06] <krion> yes, but it doesn't show the db that tell me "already enabled"
[09:25:38] <rspijker> hmmm, that’s strange
[09:26:02] <rspijker> can you pastebin the output of that command and the command you are using to enable sharding?
[09:29:39] <krion> http://pastebin.com/UGJ06Rdz
[09:30:29] <rspijker> well.. it says { "_id" : "mydatabase", "partitioned" : true, "primary" : "rs1" } in sh.status()….
[09:30:39] <krion> sorry..
[09:30:43] <krion> i obfuscated it
[09:30:46] <krion> wrongly
[09:31:15] <krion> the mydatabase partitioned true is in facte mydatabaseeeeee
[09:31:37] <rspijker> hmm
[09:32:46] <rspijker> and what does:
[09:32:50] <rspijker> use config; db.databases.find({"partitioned":true})
[09:32:52] <rspijker> give you?
[09:34:12] <krion> mydatabaseeeeeeee only, not mydatabase
[09:36:16] <krion> hum
[09:36:21] <krion> as you said, it was in cache
[09:37:47] <krion> i restart the mongos process and it works now
[09:38:39] <krion> thanks rspijker
[09:39:03] <rspijker> cool :)
[10:39:42] <A13> hi there, I got a simmple question: can I copy the data files (from /data/db) to another server and start mongod there? will it work? I cannot start a damaged installation of mongodb to do a proper export..
[10:57:23] <remonvv> \o
[10:58:54] <tehmoon> Hi
[11:00:00] <tehmoon> does anybody know why the {files_id: "hashed"} index for gridfs does not work on a sharded cluster ? v2.6.2
[12:01:43] <A13> exit
[12:27:20] <shambat> I have data stored with entries like this: http://bpaste.net/show/2f6fd25f8306 -> How can I make a query to find entries where "sources" array is larger than 1. I have tried this, but it returns nothing: db.sigs.find( {sources : {$exists:true}, $where:'this.sources.length>1'} )
[13:00:42] <tehmoon> shambat: maybe you'll find the solution with aggregate()
[13:01:34] <tehmoon> $unwind the sources a group by _id and count
[13:01:45] <tehmoon> and group by _id and count
[13:01:53] <rspijker> the where should work fine as well
[13:02:33] <tehmoon> i never used the where so i cant tell :(
[13:02:50] <rspijker> it’s not very efficient, since it uses js
[13:03:28] <rspijker> if you only want larger than 1, you can also use {“sources.1” : {$exists:true}}
[13:03:45] <rspijker> assuming you;re not stuck on a very old version
[13:04:02] <tehmoon> true :)
[13:04:28] <rspijker> caveat is when you could have sources that aren’t arrays but documents and could contain a field named 1…
[13:04:41] <rspijker> but bar that, it;s the most efficient way
[13:26:56] <remonvv> Hm, detail perhaps, but does anyone know why the shell prints NumberLong(1084752864) for numbers < 2^32 and NumberLong("10665270000") so with quotes for numbers > 2^32
[13:28:59] <jordana> remonvv: It's the do with the way JS handles ints
[13:30:19] <jordana> remonvv: I think JS only has a type that supports 8byte ints so it's a workaround
[13:31:08] <remonvv> Figured it was something like that, thanks
[13:31:19] <jordana> 8bit*
[13:31:24] <jordana> byute
[13:31:27] <remonvv> And you mean 4byte ints I assume
[13:31:56] <jordana> Yeah I think its 8byte floating point numbers
[13:32:02] <remonvv> Jep
[13:32:38] <remonvv> The single reason all languages with a best guess type should have been retired decades ago ;)
[13:32:49] <remonvv> best guess type system*
[13:33:07] <jordana> ha ;p
[13:33:36] <topwobble> running db.repairDatabase() for the Nth time this week :(
[13:35:45] <remonvv> jordana : print(200000000000000000000+1) ;)
[13:53:20] <shingara> Hi
[13:53:30] <shingara> I run a js script on mongoDB server
[13:53:42] <shingara> if the script failed the output is return in STDOUT
[13:54:09] <shingara> it can't be possible to out in STDERR and STDOUT when I add a log.info ?
[15:16:11] <tscanausa> what is the largest dataset anyone has heard of being used in mongodb?
[15:18:57] <topwobble> tscanausa: Parse has some pretty huge ones
[15:21:16] <tscanausa> I have looked but never found an actually data size.
[15:25:15] <cheeser> tscanausa: http://www.mongodb.com/press/mongodb-delivers-multi-petabyte-data-store-option-adobe-experience-manager
[15:25:57] <lha13> Hello, I've got the following stackoverflow question (http://stackoverflow.com/questions/25666187/mongodb-nested-group/25667256) and I 'm looking at
[15:26:28] <lha13> I'm looking at limiting the amount of results to 5 per array. How would I go about doing this? Any advice would be great thans!
[15:26:35] <lha13> *thanks in advance!
[15:59:54] <tscanausa> cheeser: Thanks. but not exactly specific
[16:03:22] <cheeser> when you're talking about data in the petabyte range, how specific do you *really* need to be?
[16:03:40] <cheeser> unless you think you're gonna have more data than that...
[16:14:26] <oli47> Hi guys
[16:15:07] <oli47> Can someone help me with a collection update issue ?
[16:17:20] <oli47> anyone here ?
[16:17:42] <kali> oli47: yes. but we're not good with meta-questions
[16:17:52] <kali> oli47: ask the real question you want an answer to
[16:18:06] <oli47> aw thanks
[16:18:53] <oli47> well basically I am trying to do an update of some documents, and I need to match only the ones that do NOT contain a specific value in an array value
[16:19:18] <oli47> I tried by combining $elemMatch and $not but that didn't work
[16:19:20] <oli47> any idea ?
[16:19:46] <kali> oli47: $ne should do it
[16:23:32] <oli47> worked, thanks!
[17:21:53] <doug1> Why would I get, on the router, "can't authenticate to <some-cfg-server> as internal user, error: { ok: 0.0, errmsg: "auth failed", code: 18 }" ?
[18:08:37] <doug1> sigh
[18:43:44] <doug1> Why would I get, on the router, "can't authenticate to <some-cfg-server> as internal user, error: { ok: 0.0, errmsg: "auth failed", code: 18 }" ?
[18:49:50] <harttho> I'm seeing mongos proc's taking up different ports than 27017
[18:49:52] <harttho> for instance, 9099
[18:50:15] <harttho> Where is the configuration for the mongos connection ports that would result in taking up 9099?
[18:59:16] <harttho> Note: this is on a server with mongos running connecting to an external mongo server
[19:11:21] <doug1> Why would I get, on the router, "can't authenticate to <some-cfg-server> as internal user, error: { ok: 0.0, errmsg: "auth failed", code: 18 }" ?
[20:12:42] <skot> doug1, check to make sure you are using the same keyfile or x509 cert for both the mongos and config server.
[20:14:01] <skot> harttho: that is how ephemeral ports work when a process has to make outgoing connections; it is expected.
[20:18:26] <mattkremer> Alright, I'm in the middle of a meeting with my team and there are some crazy ideas floating around regarding speed of lookup with indexes. Would hashing the values of data in MongoDB actually increase the speed of lookups?
[20:19:32] <cheeser> are those values already indexed?
[20:20:10] <mattkremer> Yes, let's say that an index already exists. Are queries against that index going to be different if the data was stored as full string or as hash when it was initially inserted and indexed.
[20:20:23] <mattkremer> *are queries going to be different speeds
[20:20:29] <cheeser> i doubt it
[20:20:51] <cheeser> you'd have to query using that hashed value and at that point, you're just doing an index lookup either way
[20:20:57] <mattkremer> That's what I thought...since the data is already indexed in an actual lookup, that the query actually won't differ in speed.
[20:21:29] <cheeser> now, if you're talking strings, larger strings are only indexed up to a certain length, iirc.
[20:21:52] <mattkremer> How "large" is larger?
[20:22:07] <cheeser> https://groups.google.com/forum/#!topic/mongodb-user/1jiql-9N1w4
[20:23:15] <shadfc> I'm getting an OperationFailure "can't get runner for query". Trying to upload a file to gridfs using pymong. Mongdb 2.4 currently. Any ideas?
[20:23:24] <mattkremer> Alright, so if I know our data isn't going to be over 1KB, then it's not really going to matter at all?
[20:48:14] <akp> is there a findAll for mongo in php
[20:48:34] <skot> mattkremer, there is going to some measurable improvement with smaller strings since they will result in a smaller indexes and faster lookups.
[20:48:44] <skot> The faster lookup part is probably the less important part.
[20:49:20] <skot> akp: find by default will find all, unless you specify a limit.
[20:58:42] <akp> hmm
[20:58:54] <akp> i tried that and i seemed to return the empty set
[21:01:20] <akp> skot: http://pastie.org/9527788
[21:01:24] <shadfc> anyone have an idea on what might cause a "can't get runner for query" error?
[21:02:13] <skot> shadfc: do you have noCollectionScans enabled?
[21:02:39] <skot> It is a mongod option. It would prevent you from having queries unless they use an index, and that error message.
[21:03:25] <skot> akp: read the docs: http://php.net/manual/en/class.mongocursor.php
[21:03:54] <doug1> is the KeyFile option only used for SSL?
[21:04:04] <skot> doug1, no
[21:04:11] <skot> It is used for all auth
[21:04:22] <doug1> ok
[21:04:23] <shadfc> I'm seeing it in two mongo installations. the common options between them are --noprealloc --smallfiles
[21:06:15] <akp> skot: thanks for the link.
[21:06:56] <skot> shadfc: Can you run this from the shell/app getting the error? db.adminCommand("getCmdLineOpts")
[21:07:11] <skot> It will return the active command line options from that instance.
[21:07:29] <skot> The option is actually called noTableScan
[21:07:57] <shadfc> skot http://dpaste.com/00SY786
[21:08:22] <shadfc> i don't see it though
[21:08:51] <skot> nope me either, must be another cause.
[21:09:21] <skot> Can you post a paste with the query, output, list of indexes and stats for that collection?
[21:09:44] <skot> Are you using 2.6.4?
[21:10:39] <shadfc> 2.6.3 looks like
[21:11:24] <shadfc> upgrading to 2.6.4 now.
[21:13:50] <shadfc> same errors on 2.6.4
[21:14:43] <shadfc> OperationFailure: command SON([('filemd5', ObjectId('5408d55cbaba325b82964932')), ('root', u'fs')]) failed: exception: Can't get runner for query { files_id: ObjectId('5408d55cbaba325b82964932'), n: { $gte: 0 } }
[21:20:19] <doug1> which boxes would get the Keyfile? All of them? cfg servers? data nodes? router?
[21:21:21] <skot> If you can post that info I can take a look.
[21:21:38] <skot> doug1: all
[21:21:48] <doug1> skot: tanx! Ok
[21:21:59] <skot> It is the shared credentials which secure intra-node communications
[21:27:12] <akp> is there anyway to just return it as a list or something so i can get a count from it and other things?
[21:28:16] <akp> or is that what the vardump thing will do for me
[21:30:50] <skot> I'm no php programmer, so I can't help there but would assume this would be a simple thing. In python it is just list(cursor), but php, don't know.
[21:35:24] <tscanausa> akp: you have to iterate the entire cursor. if you want a count call the count function with the query
[21:37:21] <tab1293> is there a faster way to convert a field type other than looping through every document with a foreach?
[21:37:47] <skot> Are you converting to the same value, or different values?
[21:38:07] <skot> If the latter, no, if the former, with an update.
[21:38:23] <tab1293> same value, they are numbers stored as strings and I want them to be integers
[21:38:39] <tab1293> I have about 350,000 docs in the collection. the foreach has been running for like 10 minutes
[21:38:58] <akp> tscanausa: thank you. i can call the count function with each find query?
[21:39:42] <skot> tab1293: Do you mean the same type, not same value. String is the type, "13", "15" or different values of strings.
[21:39:58] <skot> s/or/are
[21:40:03] <tab1293> I mean I want “15” to go to 15
[21:40:15] <skot> Does every document have "15"?
[21:40:15] <tab1293> skot ^
[21:40:21] <tab1293> no they are all unique
[21:40:30] <skot> lol, so they all have different values... :)
[21:41:05] <tab1293> I thought you meant was I changing the actualy value like “15” to 13
[21:41:19] <tab1293> so a foreach is the only way?
[21:41:20] <akp> or would that be something that i run on the object i get back from my query?
[21:42:17] <skot> tab1293: yes, but you can update just that field, and run multiple threads to do the work since your client is probably the slow part.
[21:42:34] <skot> (also, just retrieving the field you want to change will reduce the network traffic)
[21:43:23] <skot> iterator_to_array(query) will return an array in php, so the internet says.
[21:43:38] <skot> akp --- ^
[21:44:50] <tab1293> skot: this is the command I ran in the shell http://pastebin.com/GgqmCg9D
[21:45:12] <tab1293> so are you saying project only the tpb_id field to reduce the network traffic?
[21:46:34] <skot> yes, and don't use save, use db.update({_id:x._id}, {$set:{tpb_id:new NumberInt(x.tpb_id)}})
[21:47:04] <skot> Then run a few shells doing the same thing.
[21:48:04] <tab1293> am I safe to open a new shell and run the modified command while this once is running?
[21:48:28] <skot> yes, why wouldn't it?
[21:49:03] <tab1293> dunno just asking
[21:50:00] <skot> You can also use the bulk api to get a bunch, group them, and send them back as a list of individual updates.
[21:50:12] <skot> but running multiple clients should work just as well.
[21:50:54] <skot> you can start with just one and see how much faster it is.
[21:53:07] <tab1293> okay
[21:53:11] <tab1293> this look good http://pastebin.com/Wx0e7QGV
[21:53:12] <tab1293> ?
[21:54:23] <tab1293> woops minus that first line in the foreach
[22:13:40] <harttho> skot: is there a way to restrict which ports are being used for the outgoing connections?
[22:14:01] <harttho> or is there a set range of ports ephemeral ports can be on?
[22:14:01] <skot> harttho: consult your operating system docs.
[22:14:31] <skot> also, not sure why you would need to.
[22:15:02] <skot> they do not listen and are not incoming ports
[22:15:20] <harttho> Using Ubuntu 12.04 if you happen to know off the top of your head. We've run into some port collision of processes we have on the same ports and get eaddrinuse
[22:18:22] <skot> I searched the internet to return you these results: cat /proc/sys/net/ipv4/ip_local_port_range , echo "X Y" > /proc/sys/net/ipv4/ip_local_port_range
[22:19:20] <skot> I'd suggest you do that same or consult your sysadmin who manages the server.
[22:33:13] <freeone3000> I'm trying to shard *some* databases on the server, but not others. I've successfully (probably) sharded a database called "company-prod", but I'm attempting to *not* shard "company-config". However, mongo gives me https://gist.github.com/freeone3000/9cd6b4089246707dd91e . What am I doing wrong?
[22:34:22] <harttho> skot: thanks
[22:58:39] <tab1293> can you run a mongo operation in the background of a shell?
[22:59:21] <joannac> tab1293: why not just open a different shell?
[22:59:56] <tab1293> joannac: i am connecting to the server over ssh and want the operation to run when I logout
[23:00:10] <joannac> run it in screen?
[23:36:16] <unholycrab> i want to add a new empty SECONDARY to a replica set, let it sync up all the data and join the replica set.... but i want to control when the election happens
[23:36:33] <unholycrab> ie: if i just let it go, it will force an election as soon as its done copying, whenever that happens to be
[23:42:53] <joannac> unholycrab: no, it won't force an election at all, assuming it doesn't ahvea higher priority than the current primary
[23:44:15] <unholycrab> joannac: mongodb 2.4 ?
[23:44:43] <joannac> any version
[23:45:01] <unholycrab> ok
[23:45:19] <joannac> well, any version 2.4+. I cdon't have enough experience with 2.2 and below to say for sure
[23:45:42] <unholycrab> thanks joannac
[23:50:32] <freeone3000> I'm trying to shard *some* databases on the server, but not others. I've successfully (probably) sharded a database called "company-prod", but I'm attempting to *not* shard "company-config". However, mongo gives me https://gist.github.com/freeone3000/9cd6b4089246707dd91e . What am I doing wrong?
[23:51:50] <joannac> sh.status()
[23:59:25] <freeone3000> joannac: https://gist.github.com/freeone3000/1a34915d4799d1654817