pmxbot IRC Log Viewer

[06:01:46] <safani> Hello

[06:02:22] <safani> I have a very weird problem going on with aggregation. On my local pc I have mongo3.0.3 and on my dev server i have the same version.

[06:03:02] <safani> When I try an aggregation on a map reduced table on my local server all is well. I get the expected results, but on the dev server I only get one result!

[06:03:18] <safani> They are the same version and I don't understand why or how this could happen?

[06:03:39] <safani> Any clues or insight would be greatly appreciated.

[06:27:02] <joannac> safani: pastebin the aggregtion and the output on dev and local?

[06:28:03] <safani> nevermind i just fixed it...two things, one was pure stupidity. I didn't update my db dump so dev and local weren't the same. All is well, I do have another weird issue though...

[06:28:25] <safani> When attempting to page the results I just get an empty array even though i know there are more results

[06:28:43] <safani> I send skip 25 and then there are no results all of a sudden

[12:43:07] <StephenLynx> aww, find and X ops are limited to a single document :|

[12:43:08] <StephenLynx> lame

[12:57:05] <koren> hello, I have a massive collection (oauth token) that takes a large amount of my index. I decided to clean it up a bit to free some space and now my queries takes longer

[12:57:17] <koren> using wiredtiger, does my DB need some time to reorganize the index?

[12:57:26] <koren> I dropped 25% of documents, and my index is now larger

[13:03:31] <koren> was 5GB is now 8GB

[13:10:50] <koren> so I think latency is due to disk access to fetch some of the documents

[13:16:41] <deathanchor> koren: use mms? it can tell you about historic page faults and other things

[13:19:31] <koren> deathanchor: I use it yes, that's where I seen for index size bump

[13:20:02] <koren> I have 0 page fault

[13:20:32] <koren> i had one pike à 0.2 for 1 minut but latency is constant

[13:22:45] <koren> previously my requests took ~2ms max, after delete had a pike à 30-50ms and now it stabilized at 8ms

[13:23:43] <koren> I just made a delete on a non-indexed field, to delete expired tokens. this may be the cause?

[13:29:52] <deathanchor> try a .explain() on the query?

[13:31:50] <koren> it goes well using, IDHACK

[13:32:00] <koren> but the queries on other collections are affected too

[13:32:04] <koren> whole db gone a bit slower

[13:33:19] <koren> I have 10 times more read on disk than before the delete

[13:33:52] <koren> now my queries are 4ms average

[13:33:56] <koren> it stabilizes but takes time

[13:34:47] <koren> I will do this at night next time but I'm surprised this happened and maybe it is related to a bad index since index is on a string field and I removed documents based on a date

[13:35:06] <koren> so maybe index data is spaced a lot in memory now but I don't know mongodb enough

[13:36:17] <koren> index size is reducing very slowly so I think i'll just wait

[13:49:59] <StephenLynx> afaik

[13:50:02] <StephenLynx> koren

[13:50:10] <StephenLynx> wt is not 100% yet

[13:50:30] <StephenLynx> is not the default engine, but 10gen uses it on their service though.

[13:57:30] <luaboy> how to count the number of doc records？

[13:59:58] <koren> StephenLynx: yes I know that but maybe I did something wrong so I'd rather ask here so I can fix it, we are already rolling back to defaut engine on some of our db for issues related to disk IO and poor performances of IO on cloud hosting

[14:00:20] <koren> luaboy: db.col.count()

[14:00:31] <koren> luaboy: db.col.find({}).count()

[14:00:37] <StephenLynx> you don't need the find

[14:00:46] <koren> find if he want to count with filter

[14:00:50] <StephenLynx> you can just use db.col.count({query})

[14:00:54] <koren> ho ok

[14:04:50] <dede> hi, anybody knows why there is not enough documentation for gridfs?

[14:05:15] <dede> there is a high level documentation but all the advanced use cases spread over some unoffical tutorials

[14:36:09] <StephenLynx> dede what kind of advanced use cases?

[14:37:54] <dede> trying to implement file server over gridfs and need some guide on how to setup bucket/s and what are the restrictions and corner cases

[14:38:03] <dede> just like amazon s3

[14:39:12] <dede> but documentation is really basic and all I could find is native-javascript driver documentation https://mongodb.github.io/node-mongodb-native/api-articles/nodekoarticle2.html

[14:41:20] <StephenLynx> first

[14:41:32] <StephenLynx> this is the link to the latest driver http://mongodb.github.io/node-mongodb-native/2.0/api/GridStore.html

[14:42:06] <StephenLynx> second, it is not very intuitive, but its simple, you just have to use the 'root' option and set it there.

[14:42:23] <StephenLynx> third, amazon is amazon problems.

[14:46:20] <dede> lets say I am creating bucket with heirarchy /bucket1/bucket2/object_key/file_name

[14:47:05] <dede> object_key is a random number and I will map buckets to bucket1.bucket2.object_key

[14:48:00] <dede> seems like there is a restriction on bucket named being numbers

[14:48:10] <dede> db.bucket1.101010

[14:48:17] <dede> 015-06-01T17:46:18.317+0300 E QUERY SyntaxError: Unexpected number

[14:49:03] <StephenLynx> IMO that is because of most languages not allowing variables starting with digits.

[14:49:56] <StephenLynx> and from what I noticed

[14:50:11] <StephenLynx> it seems that you can just use a bucket that is a collection.

[14:50:20] <StephenLynx> I have never tried to change bucket name though

[14:50:23] <StephenLynx> so I am not sure.

[14:50:55] <dede> yes thats true and most languages does not allow that but at the end its a valid use case and I think should be in the documentation

[14:50:58] <StephenLynx> and have you considered using also the filename for path?

[14:51:21] <StephenLynx> probably mongo separates the bucket into pieces.

[14:51:37] <StephenLynx> so if you use db.10 both will have to be valid on their own.

[14:52:00] <dede> yes if you put "." it considers them distinct collections and have own validity check

[14:52:07] <StephenLynx> so there you go.

[14:52:13] <StephenLynx> working as intended.

[15:15:48] <adsf> if i had a set of strings i wanted to search for in mongo, where i needed to know which existed/which didnt, is there a bulk operation i could use?

[15:15:52] <adsf> or should i just do one at a time?

[15:16:19] <StephenLynx> I think you could use the $or operator.

[15:16:35] <adsf> then just parse the response?

[15:16:45] <StephenLynx> hm, let me read with more care

[15:16:49] <adsf> hehe

[15:16:56] <StephenLynx> " which existed/which didnt"

[15:17:01] <StephenLynx> what do you mean by that?

[15:17:16] <adsf> so 10 strings in a set, how many exist and tell me which ones

[15:17:25] <StephenLynx> what do you mean by exist?

[15:17:34] <adsf> have a matching document

[15:17:39] <StephenLynx> in what sense?

[15:17:53] <adsf> say "someidentifier" : "string"

[15:17:58] <StephenLynx> ah

[15:18:12] <StephenLynx> $in

[15:18:23] <StephenLynx> someField:{$in:[array of strings]}

[15:18:28] <adsf> okie dokie, time to do some research :)

[15:18:29] <StephenLynx> then

[15:18:33] <StephenLynx> you can use $group

[15:18:52] <StephenLynx> to know which ones you got matching documents.

[15:19:00] <adsf> perfect, then i can just loop through the response

[15:19:01] <adsf> party on

[15:19:32] <adsf> thank you for your help mate :)

[15:19:36] <StephenLynx> np

[15:22:18] <ndb> hi, I'm trying to add a new Date() field and get UTC, there is some way to force mongo to use my local?

[15:22:54] <StephenLynx> where are you instantiating the date?

[15:23:19] <ndb> on mongo shell

[15:23:32] <ndb> db.col.insert({'time': new Date()})

[15:24:13] <StephenLynx> hm

[15:24:23] <StephenLynx> indeed, it seems mongo will get the date and store as utc.

[15:24:33] <StephenLynx> I just checked in my application

[15:24:48] <ndb> using the python driver it stores ok

[15:25:29] <StephenLynx> so probably is a driver setting.

[15:25:36] <ndb> yea

[15:28:42] <deathanchor> http://docs.mongodb.org/manual/core/shell-types/

[15:28:54] <deathanchor> new Date() constructor which returns a Date object using the ISODate() wrapper

[16:46:18] <_ari> who created mongoosejs

[16:57:14] <StephenLynx> hitler, probably ¯\_(ツ)_/¯

[17:46:13] <goldstar> how do I remove everything in a db that is older than one month ?

[17:46:46] <StephenLynx> you will have to perform this query on each one of its collections, if I am not mistaken

[17:48:05] <goldstar> StephenLynx: ah, there are 20+ collections :( - I need to free up some space; how would I find out will collection is consuming the most space ?

[17:48:28] <StephenLynx> don't know.

[18:12:06] <jr3> is there a way to tell mongoose, to query on some constraint say that returns 500 documents, but I only want to process 100 at a time,

[18:12:20] <jr3> then go into the next batch size

[18:43:03] <GothAlice> goldstar: db.collection.stats() on each collection will give you various statistics, including dataSize.

[18:43:49] <goldstar> GothAlice: thanks; is it possible to get the stats in MB or GB ?

[18:44:14] <GothAlice> goldstar: db.collection.stats(1024) < supply a factor and it'll do the divisions :)

[18:44:25] <goldstar> GothAlice: ah, thanks

[18:44:59] <goldstar> GothAlice: one other question; can I set expiries for db ? i.e. only keep data that is 1 month other

[18:45:02] <GothAlice> Note that to actually free up disk space, you need to run a repair operation. http://docs.mongodb.org/manual/reference/command/repairDatabase/

[18:45:13] <StephenLynx> you can set it for individual documents.

[18:45:27] <GothAlice> goldstar: Yes, you can create a TTL (time-to-live) index against a date field in your documents.

[18:45:39] <GothAlice> goldstar: http://docs.mongodb.org/manual/core/index-ttl/

[18:45:49] <goldstar> GothAlice: thanks once again :)

[18:46:27] <GothAlice> Note again that to free up currently allocated disk space, you'll need to repair the database. After that, once you get a TTL index in, hopefully the collection won't grow too quickly.

[19:13:29] <GothAlice> But I get the general idea that iterating QuerySets is bad for some reason.

[19:27:54] <grazfather> hey guys, can a aggregate be used to set arbitrary keys in the group? e.g. http://pastebin.com/xYqbMCZ7 is the sample in/out, but i get "A pipeline stage specification object must contain exactly one field"

[19:47:14] <seion> exit

[19:47:51] <StephenLynx> bye

[19:47:54] <StephenLynx> ah

[20:25:37] <carver404> hi.. i'm trying to install mongodb on my Fedora.. though why this baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.0/x86_64/ is down?

[20:26:00] <carver404> how do i proceed?

[20:48:44] <learns> is mongodb faster than rethinkdb?

[20:51:01] <deathanchor> describe "faster"

[20:52:01] <willbutton> trying to export a collection from my dev server: `mongoexport --db test --collection bulkOperation --out bulkOperation.json --authenticationDatabase admin -u myUserName -p myPassword`

[20:52:12] <StephenLynx> Not many of us know every single db performance in existance.

[20:52:14] <willbutton> get the error: Failed: error connecting to db server: server returned error on SASL authentication step: Authentication

[20:52:19] <StephenLynx> I have never heard about rethink

[21:05:21] <deathanchor> mongodb is faster in some things than tokumx, but tokumx is faster in other things

[21:09:40] <GothAlice> willbutton: Double check that the client tools you are using match the version of the server.

[21:12:18] <willbutton> GothAlice: they do, both v. 3.0.2, attempting the export from the mongodb server

[22:08:39] <willbutton> I think it is permission related- I can't drop the database either. The user account currently has the following permissions: clusterAdmin, dbAdmin, readWriteAnyDatabase, userAdminAnyDatabase

[22:08:47] <willbutton> What permissions are required to export?

[22:11:20] <willbutton> fyi: I added dbAdmin after discovering I can't drop the database.

[22:33:22] <joannac> willbutton: if you open a shell and authenticate, does it work?

[22:33:31] <joannac> (the authentication part, I mean)

[22:34:48] <willbutton> yup- it does, username and password confirmed

[22:36:14] <joannac> confirm the roles array for me?

[22:38:39] <willbutton> sure: https://gist.github.com/rekibnikufesin/09d748617c136c53866f

[23:26:07] <kexmex> does an index help with $exists:true?

[23:28:53] <StephenLynx> not sure, but I guess so.

[23:28:59] <kexmex> i am getting

[23:29:00] <kexmex> Btree::insert: key too large to index, failing

[23:29:01] <kexmex> me

[23:29:02] <kexmex> meh

[23:29:13] <joannac> willbutton: use bulkOperation; db.bulkOperation.findOne()

[23:29:20] <kexmex> joannac

[23:29:25] <kexmex> i am getting "key too large"

[23:29:31] <joannac> make your key smaller

[23:29:36] <kexmex> what if i create a more specific index

[23:29:39] <kexmex> to make it smaller say

[23:29:40] <joannac> key size for indexing is max 1024

[23:29:44] <joannac> nope

[23:29:45] <kexmex> will $exists still be able to use it?

[23:30:00] <kexmex> like

[23:30:07] <joannac> kexmex: say you're indexing {a:1}

[23:30:15] <kexmex> createIndex({ myhugekey.somesubfield : 1 })

[23:30:17] <joannac> in one of your documents, the value for "a" is > 1024 bytes

[23:30:24] <kexmex> yea but

[23:30:27] <kexmex> what about a.b

[23:30:29] <kexmex> where b is small

[23:30:32] <joannac> sure

[23:30:39] <joannac> if that satisfies your query pattern

[23:30:48] <kexmex> well i am using $exists

[23:30:49] <kexmex> on a

[23:30:54] <kexmex> but can't it use the index tho?

[23:31:07] <kexmex> ohhh... that's still one field, i am thinking like index(filed1, field2)

[23:31:12] <kexmex> where it can use field1 part of it

[23:32:19] <joannac> huh?

[23:32:57] <kexmex> i was thinking since index is { a.b : 1 }

[23:33:14] <kexmex> it can use it for { a : { $exists: true }}

[23:33:24] <joannac> no

[23:33:28] <joannac> why would it?

[23:33:36] <kexmex> cause if a doesn't exist

[23:33:41] <kexmex> then b wont exist and thus no index entry

[23:33:47] <willbutton> joannac: gist updated with findOne() results: https://gist.github.com/rekibnikufesin/09d748617c136c53866f

[23:33:53] <joannac> so if the document is a: {c:1}

[23:33:56] <joannac> a.b doesn't exist

[23:33:59] <joannac> but a exists

[23:34:09] <kexmex> shit :)

[23:34:30] <joannac> willbutton: so the find works... but mongoexport doesn't work? weeeeeird

[23:34:59] <joannac> willbutton: wait. that's in test. try the actual database

[23:35:10] <willbutton> test is the actual database

[23:35:17] <willbutton> the db name is test

[23:35:24] <joannac> then why was your mongoexport using bulkOperation?

[23:35:26] <willbutton> the collection is bulkOperation

[23:35:36] <joannac> wait.

[23:35:41] <joannac> never mind, i can't read

[23:35:45] <joannac> >.>

[23:35:50] <willbutton> all good :-)

[23:35:55] <joannac> okay, that's still weird

[23:36:13] <willbutton> yeah- that's the conclusion I came to.

[23:36:34] <willbutton> I have another mongodb server, but can't try to replicate the issue until later tonight

[23:37:14] <willbutton> as a work-around, I moved the collections to a new db, dropped the test db, then moved them back

[23:38:21] <joannac> willbutton: does mongo -u user -p password --authenticationDatabase admin work?

[23:41:34] <willbutton> oh wow- so this is interesting. If my password is in the command, it fails but if I just use -p and enter my password when prompted, it works

[23:42:05] <joannac> do you have a weird character in your password?

[23:42:19] <willbutton> Yup. A $

[23:42:20] <kexmex> mongo works pretty slow with Blob Storage

[23:42:26] <joannac> if you quote your password, does it work?

[23:42:29] <kexmex> are there any optimization tricks?

[23:42:55] <willbutton> it does with single quotes, but not double quotes

[23:43:06] <willbutton> trying the export again with a single-quoted password

[23:43:59] <willbutton> Argh. That was it- the $ was causing it to fail, works when single-quoted

Log file Viewer

Help | Karma | Search:

#mongodb logs for Monday the 1st of June, 2015