Log file Viewer
#mongodb logs for Wednesday the 20th of November, 2013
[00:08:44] <lyxsus> Hi. Why my mongodb collection is limited by 41322 documents? I need to import 161604. What do I miss?
[00:19:38] <harttho> lyxsus: There shouldn't be a limit that low then. What made you aware of a perceived limit?
[00:23:24] <regreddit> is there a way to determine what is causing this: MongoError: Field name duplication not allowed with modifiers
[00:24:58] <regreddit> its a fairly large document, and convert it to an Object, merge it with an updated object I get from a client, then run doc.set(object)
[00:28:22] <lyxsus> harttho: Seems like it was somewhere inside nodejs. Splitted by 4, imported well.
[00:30:51] <harttho> lyxsus: 16MB document limit size is a hard limit, for moving in large documents look for gridFS.
[00:34:03] <regreddit> lyxsus, if you are journaling mongo, the sync to disk interval could possibly be extended a bit at the cost of a bit of possible data loss
[00:52:32] <regreddit> nm, i see what was wrong, sicne I was passing in a complete document, it had bot __v and _id set, so i delete them from obj before doing doc.set(obj) and the error goes away
[02:21:44] <DoubleAW> so is there ever a time when it's better to replace _id with something else? or is it generally better to leave it as an ObjectId?
[02:22:49] <joannac> Sure, if your documents already have a unique identifier, you may as well use that
[02:24:36] <DoubleAW> I was at the conference today and a lot of the examples used some other form of id so I was wondering if there were general use cases when it's advantageous
[03:57:35] <dougb> is it possible to tail /var/log/mongodb/mongodb.log or to turn on profile and filter only by queries that aren't using an index in the system?
[04:19:30] <regreddit> DoubleAW, mainly if you already have a unique ID then using that will certainly save space and be more contextual in your app/db
[04:20:21] <DoubleAW> so, say, if I have a users collection where I require the username to be unique
[05:06:59] <saganbyte> Hello, i m kinda stuck at probably a very silly thing... i ve installed mongod db on a new aws ec2 instance and have opened ports 22, 27107 and 28107. I m able to connect to the db from my local app using mongoose.
[05:07:11] <saganbyte> I m stuck where I d like to create a username and password to connect to the db
[05:07:59] <saganbyte> When I log in to ec2 server and do 'sudo mongod --dbpath /data/db --smallfiles I think the daemon startsup
[05:31:54] <Drewch> For example, using mongodb for hot storage, and then as data gets colder, I want to move it to colder storage
[06:24:56] <cheeser> Drewch: not at the moment. you'll have to write your own cron job to archive that data.
[06:27:03] <cheeser> there's a ticket open somewhere that's along those lines but I couldn't find it just now
[07:11:57] <martin_kou_> I'm getting very slow query performance on a large collection (30GB+, 10M+ rows)
[07:12:26] <martin_kou_> the query is properly indexed, and there's only one btree index there (timestamp)
[07:13:04] <martin_kou_> but mongostat shows a query rate of only ~40/s. the hardware isn't that bad.
[07:14:09] <martin_kou_> actually, I find that as I go through the collection from the db cursor over time, the query rate drops gradually
[07:44:30] <moian> I'm having an error "QueryFailure flag set on getmore command" when retrieving some data
[08:05:19] <JuPe> Hi all! I've a question related to mongo clusters. What are the options to create a database system on several continents, so that could be written on every continent ?
[08:06:39] <JuPe> If I've understand right, it is not directly possible with mongodb, is it true ? So how I do this another way ?
[08:07:38] <balboah> you can utilize shardig to have it split up to closest db depending on where you are as a client. If that's the kind of database system you're referring to
[08:10:38] <JuPe> from reader point of view, data should be stored every continents (because between continents, there is very slow connection)
[08:21:14] <JuPe> I think that sharding is not the best option, because data is splitted, and reading from other continent is then quite slow
[08:58:33] <moian> kali: Yes this is a long running query as I'm asking lots of data point (between 100k and tens of millions), ordered by a date (I have relevant indexes)
[09:01:28] <kali> moian: you need to disable the timeout on your cursor. the way to do it depends on your client library
[09:31:56] <JuPe> kali: by the way, do you know is there any visual mondo configuration generators available ?
[09:54:24] <JuPe> np, it just would be nice to generate mongod/mongos configuration files with visual tool.. :)
[10:39:20] <NaN__> Hi, I'm on windows, I have this install structure: users\me\mongodb\bin users\me\mongodb\data\db\, I've added bin to the PATH, when I run mongod, it still says "ERROR: dbpath (\data\db\) does not exist."
[11:32:55] <LoonaTick> Hi. I have a collection which has references to another collection (user: { $ref: "user", $id: ObjectId(...) }) - I setup the user as shard key, but if I try to insert a document into the collection, a document which _does_ have the shard key in the query, it is unable to insert it because it says it doesn't have the shard key in the query. Is this expected behavior on MongoDB 2.2.0?
[11:51:42] <LoonaTick> NaN__: Not according to the documentation, the the user thin is actually an object Id
[11:54:54] <Nodex> Yesterday someone asked the exact same question, perhaps some new tutorial has appeared on the interwebs
[12:24:50] <tomasso> why could that be that text index does not return results even when i pass a search string that is available at desired fields
[13:32:22] <xxtjaxx> If you have a collection with documents that are about ~40 keys mostly strings not longer than 100-150 characters how terribly would it be ensureIndex() half of them? I'd have a troubling case here where based on the search I need to do several keys would need indexing
[13:57:26] <NaN__> what's the difference between db.users.find({name: "admin"}, {"status": 1}) and db.users.find({name: "admin", "status": 1}) ?
[15:14:37] <ypetit> Hi everybody, does any one knows a way to perform an aggregate on an associative array. I'm able to perform one on an array using unwind but not on an associative array. Any ideas ?
[15:22:05] <ypetit> but i'd like to group through any of the key whatever they are and sum/avg attributes from the values grouped
[15:23:43] <ypetit> Here's an exemple : {"slices": {"first": {"count": 4,"amount": 21,"mark": 3},second": {"count": 4,"amount": 21,"mark": 3},"anythingelse": {"count": 4,"amount": 21,"mark": 3}}}
[15:24:58] <ypetit> I'd like to do something like db.myCollection.aggregate({ "$unwind" : "$slices" },{ "$group" : {"_id" : "$slices.amount","numberOf" : {"$sum" : "$slices.count"}}} );
[15:26:13] <ypetit> it works on : {"slices": {{"count": 4,"amount": 21,"mark": 3},{"count": 4,"amount": 21,"mark": 3},{"count": 4,"amount": 21,"mark": 3}}} because it's a simple array not an associative one (i mean a Map)
[15:27:27] <ypetit> Well if anybody knows if it's possible to achieve that aggregation over subdocuments in an associative array I'd pleased to hear how ...
[15:44:37] <ypetit> kali: basically map/reduce takes 6s to execute, same written as a basic recursive JS over array and subdocuments takes 3s, another trick was to launch as many aggregates as differents keys, it then takes 1.5s ...
[15:45:13] <ypetit> kali: I was hoping to find a way to do it in one aggregate ... which if I change my structure takes 150ms !!!
[15:45:58] <ypetit> I'm trying to fool the parser using this technique : http://stackoverflow.com/questions/11189243/unwind-an-object-in-aggregation-framework las answer
[15:59:37] <epi1> hello, does anyone know in python, using mongoengine; if this is possible to perform a "GROUP BY" equivalent using the aggregation
[17:05:57] <deepender> I have a mongodb document, document contain a time field . now i want to delete the documetn after the particualr time which in field how i can do that
[17:12:20] <jawsoflife> Hey guys, I am converting a single replica set to a sharded cluster and am having some issues. My replica set is on 2.4.6. I fired up some mongo config servers and have a copy of the replica set running for testing. When I start mongos to start configuring things, however, I get an error. (Following guide here: http://docs.mongodb.org/manual/tutorial/convert-replica-set-to-replicated-shard-cluster/ )
[17:12:42] <jawsoflife> [mongosMain] ERROR: error upgrading config database to v4 :: caused by :: could not load config version for upgrade :: caused by :: 11010 count fails:{ note: "from execCommand", ok: 0.0, errmsg: "not master" }
[17:22:59] <n3b> i'm having exactly the same issue as this guy http://stackoverflow.com/questions/17527606/mongodb-failing-on-debian
[17:43:27] <quickdry21> is it typical for a balance round to take upwards of 30 minutes? I'm trying to stop the balancer - I do a sh.setBalancerState(false) - sh.getBalancerState() returns false, but for ages no sh.isBalancerRunning() still returns true.
[17:45:57] <kali> quickdry21: the changelog collection of the config database (iirc) could hint you about what's happening
[18:07:22] <Tobi1> after a while my mongodb and other programs are running well. I also understood the c++ example how I can write data into my database.
[18:07:22] <Tobi1> now I want to write some code in QTcreator but I have always problems with the include list etc.
[18:15:05] <scrdhrt> Can I have mongo log to a logfile instead of system.profile? I want to see what a client is sending to the server, like what collection and what query
[19:15:24] <TheComrade> I feel like I'm missing something basic. I want to authenticate, and run a variety of .js queries. I'd think you might do this by 'mongo query.js' but this requires re-authorizing on each query, which is obviously a PITA if you are doing this repeatedly. I could wrap it in another script but that feels inelegant. What am I missing?
[19:39:13] <harttho> Question regarding the Aggregation Framework: If I have a document with an array of numbers, how can I sum all of the values within the array?
[19:50:15] <harttho> kali: Currently am unwinding then grouping, but don't know which property to sum on
[19:57:28] <harttho> cheeser: Thanks, I figured out my issue is trying to $sum on an array of strings
[20:02:21] <tiamot> I currently am trying to load data into mongo, first I create 30 million records each with a unique _id that I create, then I attempt to update each of those records several times each time adding a new key value pair. Anyone done something similar?
[20:03:08] <tiamot> i'm trying to research if there is some way to pre-allocate document size, since I know the document is going to grow significantly after the initial insert.
[20:05:33] <harttho> tiamot: You can change the padding scalar if you know by how much it will increase in size
[20:07:22] <tiamot> yes we know the values of the id's and how many there are, so we are pre-splitting the chunks 180 chunks
[20:07:32] <brockhaywood> I'm trying to figure out the best approach for something like this. I have a collection of "messages" which all have a "thread" and I would like to generate an inbox of this which would just contain the most recent message for each thread. I've investigated using a group on a collection which would, of course, group then correctly, but this SO http://stackoverflow.com/questions/3851046/limit-results-of-group-in-mongodb suggests that group does not
[20:07:32] <brockhaywood> offer a way to limit the results. As a result, it seems like running a MapReduce would be the way to go but it seems that you don't want to run MapReduce on the fly and given that this is real-time data, running the MapReduce as a cron is not a great option. Am I missing something? is there a better approach?
[20:08:38] <harttho> brandonblack: group is an older call in Mongo, look up Aggregation Framework and it's pipeline
[20:13:42] <brockhaywood> cheesed, if I new threads ahead of time, that would work but I just want to return the most recent item in each thread.. hence the group
[21:06:35] <padan> is there a setting somewhere that i can set how a date is returned? i'd really like it returned in local time instead of utc. i dont care how its stored.
[21:59:25] <rafaelhbarros> idnc_sk: and how is the architecture, only replicas, shards with replicas, single mongod, whats going on there?
[21:59:56] <rafaelhbarros> idnc_sk: oh, another thing, are you trying to do 10k connections to the server and trying to read a bunch of stuff at the same time?
[22:00:39] <rafaelhbarros> idnc_sk: I got a server with about 2M [and growing] recs/collection, 6 collections, ~3gb memory usage on a 3 node replica, no shards
[22:01:52] <idnc_sk> rafaelhbarros: curr @112719763 records @79G on disk Virt/Res/Shared 152G/11.3G/11.2G
[22:04:50] <idnc_sk> rafaelhbarros: I'm just setting up a mr playground, still waiting - will get here if I find something fishy..
[22:06:52] <rafaelhbarros> idnc_sk: dude, I seriously love this db, I'm newbie at some stuff but I drool over any mongodb stuff, that's why I asked, immediately, a million things. =)
[22:08:11] <idnc_sk> rafaelhbarros: newbe myself - thats why I'm testing this, also - neo4j/arrangodb(although I don't realy like the concepts/standards behind graphdb's that much
[22:18:17] <idnc_sk> ..ok too soon wor that kind of talk as well - don't understand it enough to write any opinions..
[22:42:53] <ruffyen> alright i dont really know where to start with this question but it deals with the mongo shell
[22:43:06] <ruffyen> a while back on one of my servers i modified the shell to output some really useful data