Log file Viewer
#mongodb logs for Tuesday the 7th of June, 2016
[02:06:29] <wwwd> I'm trying to use mongoexport to generate a csv file. I can get a csv file with all of my records but want to use a --query parm to limit the records returned. I am using http://pastebin.com/knCY4gKq as my query. It is giving this error " error validating settings: query ... is not valid JSON: invalid character 'd' looking for beginning of value" Where ... is the original query. Can anyone tell me why this is not valid?
[02:15:39] <joannac> https://docs.mongodb.com/master/reference/program/mongoexport/#cmdoption--query
[02:29:24] <wwwd> joannac: Can't even tell you how many times I read them...just kept doing what I thought instead of what I was told! Story of my life!
[03:11:23] <Jonno_FTW> hi, what's the best way to put an image (~100kb) into a document using pymongo?
[03:18:33] <Jonno_FTW> cheeser: "Furthermore, if your files are all smaller the 16 MB BSON Document Size limit, consider storing the file manually within a single document instead of using GridFS. You may use the BinData data type to store the binary data."
[03:48:57] <macwinner> what are some common MongoDB misconceptions or FUD that is spread.. I'm doing a small presentation and I wanted to talk about them and why they are invalid points.
[03:53:30] <Boomtime> if you find some FUD on mongodb this channel might be an appropriate place to dispel it, but it's not really appropriate to relay it
[03:55:25] <wwwd> Though I would think discussion of pros & cons would be in order. It is impresive but not omnipotent!
[03:57:25] <wwwd> mcwinner: I would say it takes a bit more effort to keep your data clean. But in exchange you get flexability and maybe speed, depending on what your doing.
[04:40:06] <syrius> hey guys. i'm trying a fairly automated approach to searching a mongo collection. i'm using mongoengine (python) to define classes and using reflection to generate possible filters to populate jQuery QueryBuilder.. this just generates the appropriate mongo query for me (json) which I can pass as a raw query and it *does* work right now.. I'm curious if there's a way to only return items for an embedded
[04:42:06] <syrius> http://pastebin.com/LFhgSb1j here's a quick example of my schema (as python objects) but conceptually it shouldn't matter that it's python
[04:49:37] <Boomtime> @syrius: https://docs.mongodb.com/manual/tutorial/project-fields-from-query-results/#project-specific-array-elements-in-the-returned-array
[04:50:49] <syrius> thanks Boomtime - I might have to check my design. this is still poc so i haven't uploaded all the data yet (and the schema is fake.. just trying to evaluate how i want to do it)
[04:51:22] <syrius> i just need to be able to search and get back only what matched from those nested lists
[05:16:23] <Lope> mongoDB acting funny. I'm running it manually. I've set the shell of mongodb to /bin/bash. And with su I can browse to the dbpath, make files etc (have permissions). But when I enter the console "show dbs;" shows everything as 0.000GB. If I try insert new test data use foo; db.bar.insert({a:1}); it can't recall it when I try db.bar.find() afterwards.
[05:17:08] <Lope> I'm running it with this command. su -c '/usr/bin/mongod --replSet rs0 --oplogSize 50 --wiredTigerCacheSizeGB 1 --smallfiles --port 27017 --dbpath /mnt/mongodb --logpath /var/log/mongodb/test.log --logRotate rename --fork' mongodb
[05:45:44] <Jonno_FTW> macwinner: you could talk about how people use it for the wrong reasons (they need a relational db or ACID), then complain it doesn't meet their needs. People just don't know when mongo is the right tool for the job
[05:51:23] <Lope> for example if I run a CT from an image named imgfoo. let's say it's UID 33 is 10033 on the host. Now if I destroy that CT, and create another one, will the UID mapping be the same, or different?
[07:47:35] <Boomtime> @chris|: by sh.* i think you mean the shell helpers - these are javascript functions defined in the javascript of the shell
[07:47:59] <Boomtime> if you want to emulate these in any driver, take a look at the implementation in the shell
[07:50:19] <chris|> Boomtime: yes, I was already looking at those and was hoping to save me some time
[08:00:18] <Boomtime> many of the helpers don't have a generic means of implementation in a driver; consider sh.status() which ouputs a formatted table-ish sort of arrangement to the console - what would you expect the java driver to do?
[08:01:24] <Boomtime> even if there were an analogue to this that took say a textstream or some such, would it really be so useful that any single driver would want to maintain it?
[08:02:13] <Boomtime> the range of possibilities is wide, but the shell has a narrow scope so it can implement there extra little helpers
[09:11:47] <chris|> what is the default mode of the balancer? is it enabled by default or do I have to enable it explicitly?
[09:25:44] <chris|> so I assume isBalancerRunning is only true if the balancer is currently doing work?
[09:41:02] <kurushiyama> chris|: Was this a general question or do you experience problems of some sort?
[09:41:52] <chris|> general question, I wanted understand under which conditions running != state applies
[09:45:15] <kurushiyama> Well, the balancer only kicks in when balancing threshold are met and balancing is enabled.
[12:55:52] <kees_> is it possible for the mongodb driver in debug mode to print mili/nanoseconds instead of just full seconds?
[12:58:41] <kees_> and getting TRACE output, it is timestamped, but only with like: 2016-06-07T11:45:23+00:00
[13:03:30] <rpap> hello , I am looking for recommendations. I am planning to store user web activity in mongodb. There can be hundreds of requests per second and data payload will vary from page to page. Is mongodb the best choice
[13:04:13] <Derick> kees_: it uses gettimeofday already, but uses the second resolution localtime/strftime for printing
[13:05:30] <Derick> kees_: if you make it work, I'm sure they'd love a PR and Jira ticket: https://jira.mongodb.org/browse/CDRIVER
[13:21:12] <darkfrog> I am trying to diagnose a bug in my project using ReactiveMongo, and when I dump out the query it references "$oid" but when I attempt to use the query directly it says, "unknown operator: $oid". I presumed ReactiveMongo was converting "$oid" to "_id" and tried that and the query then worked, but it then doesn't return any results.
[13:38:55] <jordonbiondo> I'm looking for something like $ but updates all matching array elements, does that exist? {foo: [{a: 3}, {a: 1}, {a: 3}]} => {foo: [{a: 9}, {a: 1}, {a: 9}]}
[13:40:05] <jordonbiondo> I can use $ to update the first foo.a to 9 where foo.a == 3, but can't find a way to update all foo.a to 9 where foo.a == 3
[13:59:29] <jordonbiondo> kurushiyama: normally yes, but this was for a migration, I guess I'll just script it with mongoose, thanks.
[14:06:57] <saml> i need to implement GET /toppages?user={userid}&duration={lastday|lastweek|lastmonth}&limit=5
[14:08:15] <saml> so, in mongo, I was thinking about creating three collections: db.countsLastDay countsLastWeek countsLastMonth
[14:12:52] <saml> db.countsLastDay.createIndex({ts:-1}, {expireAfterSeconds:24*60*60}); db.countsLastDay.createIndex({url:1, user:1}, {unique:true}); db.countsLastDay.update({url:url, user:user}, {$set:{ts:ts}, $inc:{count:1}});
[15:29:10] <kenalex> never realized the power of mongodb until I used it in my first app and realized how straight forward development is with it
[15:36:47] <Frenchiie> i have mongod running on the default port and then i try to start a new mongod on port 27018 but i get an error
[15:36:57] <Frenchiie> Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2016-06-07T11:25:33.690-0400 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty. 2016-06-07T11:25:33.690-0400 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to lock file: /data/db/mongod.lock errno:35 Resource temporarily unavailable. Is a mongod
[15:37:12] <Frenchiie> i tried mongod --repair but it tries to repair on the default port so that doesn't help
[15:42:31] <Derick> or rather, you can't start two mongod's on the same machine with the same database path
[15:45:47] <kenalex> whats a good windows gui tool for managing collections and other aspects of mongodb ?
[15:49:20] <ily> hello there. i have a question. for the people that use the Mongoose API with nodejs. if i have a index on say first name like FirstName: 1
[15:49:46] <ily> and i use mongoose to do a search by first name. do i have to do a normal kind of search? or do i have to pass arguments to make it aware to use the index?
[15:50:20] <StephenLynx> first, its performance is horrible. about 6x slower than the native driver.
[15:52:09] <ily> StephenLynx, ok alright. it will be a bit of work but i think im tired of mongoose enough to switch over. so for mongodb how does that one handle indexes?
[15:52:45] <ily> StephenLynx, as in, is the use of an index transparent? as in if i do a search without index and one with, it will be the same command?
[15:59:12] <Derick> the page does say "https://docs.mongodb.com/ecosystem/drivers/node-js/#object-mappers"
[16:01:26] <StephenLynx> it could be crippled monkeys carving runes on rocks as far as the FE is concerned.
[16:02:38] <StephenLynx> my main project currently is a node back-end that also uses a web front-end
[16:03:09] <StephenLynx> it currently has about 35k loc on the back-end alone and I can tell you, that is not a realistic scenario
[16:04:09] <StephenLynx> I am just telling you that it doesn't share anything more than the language being used.
[16:05:15] <Frenchiie> Derick: so are you saying to do something like this? mongod --port 27018 --dbpath /data/db2 ?
[16:06:06] <StephenLynx> the only instance where the front-end matters to a back-end is when you have dynamic html
[16:06:22] <kurushiyama> ily So then why call it mean? Should I call it ReMGo just because I use React, mgo , Mongo and Go?
[16:06:58] <StephenLynx> from people who shouldn't be allowed near a computer, let alone to develop software.
[16:08:28] <ily> yeah theres no special create command. and if there is one existing. mongod will just read those. it wont delete anything
[16:39:45] <edrocks> kurushiyama: I'm working on my reporting stuff a little bit this week so we'll chat
[16:43:13] <darkfrog> I have a query I'm trying to optimize running against MongoDB 2.6.9 and I'm just trying to count results, but it has an $in array of like 500 ObjectIds. It takes roughly three minutes to count them all. Obviously this should be completely re-written, but I'm hoping for some low-hanging optimizations in the short-run if anyone has any suggestions?
[16:44:15] <kurushiyama> darkfrog Most do hope for that ;) can you pastebin the original query (preferably in shell syntax) and the indices of the collection in question?
[16:45:41] <darkfrog> kurushiyama: https://gist.github.com/darkfrog26/44f7e0d99d728a046f1110093ae49f3e
[16:49:33] <kurushiyama> darkfrog From what I can see, your problem most likely stems from a data model not fitting your use cases.
[16:50:24] <darkfrog> kurushiyama: agreed...I just inherited this project and am about to start a complete re-write, but I have to get the performance to "limping" status before I can do so.
[16:51:01] <darkfrog> kurushiyama: I get: "TypeError: db.analysisresult.count(...).explain is not a function"
[16:53:48] <darkfrog> kurushiyama: no, it's not Mongoose driven, but I think it might be nearly as bad. It currently uses Casbah (Scala toolkit for MongoDB)
[16:56:24] <darkfrog> kurushiyama: https://gist.github.com/darkfrog26/e491601332a414202e02df297786b423
[17:00:29] <darkfrog> kurushiyama: yeah, they cloned the database on a cloud hosted MongoDB server so I could test.
[17:01:54] <darkfrog> I wasn't sure if it would be possible to do a String contains type of call instead of $in with oids?
[17:01:56] <kurushiyama> darkfrog Ah, ok. Well, you can remove the $and clause (not that this would help much, but it is simply unnnecessary, since clauses are always inclusive). Aside from that... Remodel?
[17:02:07] <darkfrog> ...wasn't sure if that would be faster either, but I thought it might be worth a try.
[17:07:09] <darkfrog> kurushiyama: okay, I have another code-related work-around, but was hoping someone here might have some suggestions before I make my code uglier. :-p
[17:09:26] <darkfrog> I thought about trying a mapReduce scenario, but wasn't sure it would make any difference
[17:10:31] <kurushiyama> darkfrog In general, the aggregation pipeline outperforms m/r. In this case maybe not, but I would try.
[17:10:34] <darkfrog> kurushiyama: how might I re-write that query to use aggregation where it wouldn't still be a massive $in?
[17:11:39] <kurushiyama> darkfrog well, yeah, but still might be faster. You have to try. My suggestion: Try aggregations first, and if that fails do an m/r (which tends not to be incredibly fast)
[17:14:02] <kurushiyama> darkfrog But best ofc would be to start the rewrite instead of putting a lot of effort into something which might be in vain.
[17:20:09] <wwwd> I am trying to figure out how to return specific fields from an array in a sub-document based on the date. That is I want to return fields from a document inside the array of registrations which is nested in pets. E.g. https://gist.github.com/johnhitz/cfec74eb7d33a20bce4715a045269bc1. Can anyone tell me how to do this. I have tried using $slice and including specific fiels with projections. The closest I can get is to return the entire
[17:20:10] <wwwd> array with only the fields I want. As I said, ideally I would like only the fields for the specific registration.
[17:22:51] <darkfrog> kurushiyama: something like this? https://gist.github.com/darkfrog26/44f7e0d99d728a046f1110093ae49f3e
[17:29:31] <darkfrog> kurushiyama: well, aggregate slightly improved the performance, but not enough. Thanks anyway.
[17:45:47] <wwwd> poz2k4444: I followed https://github.com/mongodb-labs/mongo-connector and found it to be very easy to get going with.
[17:46:21] <poz2k4444> wwwd: when you modify a document on a mongo collection, does it sync automatically with elasticsearch?
[17:49:38] <poz2k4444> wwwd: that is the problem I'm facing, I can't get it to sync the updates, I've read and re-read the docs and everything seems fine, but the logs stays trying to sync (apparently)
[17:53:49] <poz2k4444> wwwd: it actualy syncs the data, creations and deletions, only the updates are missing
[17:55:26] <poz2k4444> wwwd: do you have any special configuration on elasticsearch or mongo? that I might missed
[18:00:01] <wwwd> In fact...anyone care to take a look at my previous question and https://gist.github.com/johnhitz/cfec74eb7d33a20bce4715a045269bc1 ;)
[18:03:35] <kurushiyama> wwwd Erm. $elemMatch might help. Or proper modelling. But I feel that is my personal mantra.
[18:04:43] <kurushiyama> I have no clue whatsoever what the parent document is supposed to represent.
[18:05:06] <wwwd> Derick: I'm kinda stuck with the model for now. And I tried $elemMatch in various ways!
[18:05:37] <kurushiyama> wwwd You are aware of the fact that you can do an $elemMatch in projections?
[18:07:28] <wwwd> Kurushiyama: It is a pet with a list of registrations nested in it. The registrations reflect all the registration events in the pets history. And isn't line 77 - 83 the projection?
[18:09:30] <wwwd> Kurushiyama: And if so I did an $elemMatch on the date and it ruturned a list of _id's. And, yes...I did not set it up I just have to use it for the time being. Eventually I would like to flatten it so that registrations and vaccinations are collections with references by _id...but not today!
[18:11:39] <wwwd> I can get back the correct pets...i.e just the pets that have registrations done in June. So maybe I just need to do post processing to pull out what I need for now. I was just hoping to find a way to do it easilly with mongo.
[18:14:04] <wwwd> Kurushiyama: Ok pretty new to mongo and programming in general, but does'nt it return the pet based on 'registrations' that match the date >= june 1 and < june 30?
[18:21:12] <kurushiyama> wwwd https://gist.github.com/mwmahlberg/3c8b73f500b83aca491652b625711e53#file-mongoexport-L81
[18:29:55] <kurushiyama> wwwd Extra points for style: Sample doc, code in shell syntax, result, expected result. If everybody followed that pattern, it would be much easier to help people.
[18:31:53] <kurushiyama> wwwd May I give you a well meant advice? Not meant as patronizing, but something I noticed.
[18:33:38] <kurushiyama> wwwd Your data model is horrible, to say the least. What language is your main one?
[18:34:48] <wwwd> Python. Problem is I have an ember front end that is set up to use it and can't change it now.
[18:36:38] <wwwd> I am hoping that very soon I will get it fixed. I inherited the data and can only do so much at one time!
[18:36:46] <kurushiyama> wwwd Please do not take this wrong, but I guess https://university.mongodb.com/courses/M101P/about might be a good idea.
[18:38:00] <wwwd> Was I correct when I said I want to flatten this out so that the registrations and vaccination are in their own collections and I use references by id?
[18:38:43] <wwwd> I will deffinitly look at it! I am really interested in learning. I do have the problem that my front end is using this model and I cant realy change it now!
[18:39:38] <kurushiyama> wwwd Maybe. Depends on your use cases. First thing to learn. You do not model by ERM, you model to have the questions on your data for your most common use cases answered as efficiently as possible.
[18:42:34] <kurushiyama> wwwd Most people coming from RDBMS conciously r unconciously follow that or at least similar patterns.
[18:46:32] <jayjo_> I have a s3 bucket with json files... I'm trying to import into my local mongodb. I'm using s3cmd and piping it to mongoimport, but I get this error: "Failed: error processing document #1: invalid character 'd' looking for beginning of value"
[18:55:13] <jayjo_> so my entire command currently is: s3cmd get --recursive s3://<bucket-name>/path/- | mongoimport -d <db> -c <collection> --type json
[19:07:20] <wwwd> kurushiyama: Can I use this in a mongoexport command? Something like https://gist.github.com/johnhitz/c9128e52871a482d13d9f2c7cf30548d
[19:08:27] <kurushiyama> wwwd I stay away from mongodump and mongoexport as far as I can. I do not say they are bad tools, but personally, I dislike them.
[20:03:06] <kurushiyama> StephenLynx Actually not so much. I tend to mount the snapshots, pump them into a tar file which (if no compression was used) is pumped through snappy and then either via nc or ssh to a backup server.
[20:03:41] <kurushiyama> StephenLynx But yes, in general you have a point in time copy of the complete data. Which is usually what I want.
[20:04:57] <kurushiyama> StephenLynx Well, I do. But in a standard setup, you would loose a replication factor.
[20:05:18] <dino82> What would cause a node to stay in STARTUP2 and never become a secondary? optimeDate stays at 1970
[20:05:28] <jayjo_> saml: by itself it works and prints the output of the data, but not on mongoimport. Could it be that the files are json files that are json separated by newlines. can i handle that in the code?
[20:06:34] <kurushiyama> StephenLynx And think of a sharded cluster. Doing a dump comes with its own intricacies, there.
[20:08:29] <saml> jayjo_, you mean you can do it in two steps? s3cmd > wow.jsons; mongoimport wow.jsons ?
[20:09:06] <jayjo_> I'm thinking it may be becauuse the file 1.json has many json objects separated by newlines
[20:12:01] <kurushiyama> jayjo_ You could simply pipe it through sed to remove all newlines between a closing and an opening bracket. Or all newlines, thinking of it, in case those are the problems.
[20:29:32] <dino82> I'm getting this warning: WARNING: the collection 'testdb.Db' lacks a unique index on _id. This index is needed for replication to function properly
[20:32:26] <kurushiyama> That would have to be really funny – actually, you can not even delete the _id index. And if you import documents, those without an _id are simply assigned one.
[20:33:16] <kurushiyama> jayjo_ Well, if it does so, then it is fine. "_ think_ thats what the trailing - does" Assumptions have bitten my lower back more than once.
[20:43:48] <kurushiyama> dino82 I can only imagine that data files were copied, and some where missing. An _id index can not be deleted, and every collection gets one when it is created. o.O
[21:02:13] <dino82> I have no idea how this got built as it is, I just know it's giving me headaches now
[21:04:46] <saml> i'm working with a system that writes bunch of documents for "write". and reads bunch of documents to combines documents for "read"
[21:09:16] <kurushiyama> saml Hm, can I see a sample doc? and could you describe your use case a bit?
[21:10:12] <kurushiyama> dino82 I know it could be done programatically, but I am no JS wiz and it is late.
[21:11:25] <kurushiyama> dino82 But a missing _id index is something I would take !*_VERY_*! seriously.
[21:11:47] <kurushiyama> dino82 I would probably stop my instance first and create a backup of the data files.
[21:21:48] <poz2k4444> hey guys, how can I replicate the oplog of one of my collections? apparently mongo-connector can update data because of this, I've tried with a new collection and everything works fine and with a dumped collection and the updates doesn't work, so after a while I realized that this has something to do with the oplog
[23:38:06] <jayjo_> I'm still trying to use s3cmd and mongoimport to transfer data from s3 to mongodb. When I use s3cmd if I put it to stdout then there are other bits of the s3cmd output that is present and throws errors. If I use --quiet it supresses output entirely. any ideas?