Log file Viewer
#mongodb logs for Wednesday the 20th of January, 2016
[00:01:46] <Boomtime> @regreddit: yes, describe here and if you can provide a repro that would be even better
[00:05:24] <regreddit> Here is the complete process to replicate: https://gist.github.com/nikmartin/fc329a15d0173df47306
[00:06:00] <regreddit> the issue is in that collection, the property FEATURE_ID is not usable in the shell, nor in any driver
[00:06:26] <regreddit> and it's not the column name thats the issue, or data type, it's the fact that its the first property imported
[00:08:38] <regreddit> so no query by FEATURE_ID works either: db.getCollection('PLACES').find({FEATURE_ID:1397658})
[00:10:21] <regreddit> when I inspect the object using something liek node inspector, the property FEATURE_ID is surrounded by double quotes in the object explorer, but no other properties are
[00:13:00] <regreddit> that last comment probably made no sense, but here's a grab: http://i.imgur.com/AOUN448.png
[00:15:02] <regreddit> this has been working for me, but didn't when I started working on code that was querying by FEATURE_ID and it wouln't work
[00:18:06] <regreddit> so if I query the collection using the node driver, and inspect one of the objects in the returned dataset, FEATURE_ID is wrapped in ""
[00:23:16] <regreddit> on my platform, it always says 0 documents, but it creates the colelction, and the docs are always there
[00:24:18] <Boomtime> "docs are always there" <- none of your queries show anything returned, so what do you mean by this?
[00:24:19] <regreddit> the full version of that import file has 200k rows, and if i import that, i get: http://pastebin.com/CY41B5yA
[00:26:51] <regreddit> so, if i import 200k records into a new db and new collection, I see this: http://pastebin.com/CY41B5yA
[00:26:53] <Boomtime> can you give an example of it returning records? i haven't seen one yet - did i miss something?
[00:29:34] <Boomtime> can you try importing your small file (6 records i think?) to a new collection, but run mongoimport with --verbose and capture the output
[00:30:03] <Boomtime> all the outputs you've provided thus far claim that no actual insertions occurred
[00:31:02] <jiffe> [repl writer worker 15] Invariant failure n < 10000 src/mongo/db/storage/mmap_v1/btree/btree_logic.cpp 2075 <== that ring a bell with anyone? two of the three members in a replica set are dying with this and won't start up again
[00:34:42] <regreddit> if you have a public key, i'll be glad to give you ssh access to my desktop PC where I'm doing this
[00:35:48] <jiffe> I'm guessing switching from mmapv1 to wiredtiger will require 50% free space to rewrite the db?
[00:40:40] <jiffe> looks like one of the replicas died on sunday and the second died yesterday about 19 hours after the first one
[00:42:26] <Boomtime> the error is specifically occurring because some invariants in an index were violated - the index is invalid, the storage is probably corrupt
[00:43:43] <Boomtime> are all three members hosted seperately? do they share a disk/storage-layer at any point?
[00:45:45] <Boomtime> then that is surprising - i'd suggest starting a re-sync on one of them immediately, it will vote while sync'ing and induce the other to become primary again
[00:46:11] <jiffe> I've never been able to get the internal syncing to work I've always had to rsync
[00:46:31] <jiffe> I should say I've never been able to get it to work with this database, its rather large
[00:52:07] <cornfeedhobo> Boomtime: you have been helpful. before you are done with the day, would you ping me. this problem has me about ready to ditch mongo
[01:07:56] <Sht0> Hi! I'm having a collection of around 200GB in size and mongo queries seem extremely slow; anyone can please tell me if this is good method to check queries and how can I further troubleshoot?
[01:07:56] <Sht0> time mongo scrappstore --eval 'db.collection.find({"language":"en", "app":"com.my.appp"}).forEach(printjson)'
[01:12:18] <Boomtime> @Sht0: do you have an index on 'language' and 'app'? an indexd like: {language:1,app:1} for example
[01:15:51] <Sht0> @Boomtime I see this set https://bpaste.net/show/791aad00e5e1, so I think it's set
[01:17:48] <Boomtime> be aware the shell is intended as an admin tool - it won't be the best at running scripts
[01:18:40] <Boomtime> from the look of the outputs you got it's likely you waited mainly for the data to come back
[01:19:34] <Boomtime> in a collection of 200GB, unless you have 200GB of ram it probably doesn't fit anything like a decent portion in ram, so if you ask for a lot of documents, then the server has to read from disk a lot
[01:27:56] <regreddit> back from reboot and starting dinner, and still having issues with the first column of imported DB
[01:28:18] <regreddit> im going to create some collections manually and see if that makes a difference
[01:28:56] <tsturzl> So I configured SSL, and all is fine and well. So I go in and try to `rs.initiate()` and it fails, says I need to be admin. Try to create an admin(on admin DB), and it fails cause its not master.
[01:29:26] <tsturzl> So I'm scratching my head. I cannot create an admin because I have no master because I have no replset, yet I cannot create one without an admin
[01:31:39] <Sht0> @cheeser I guessed the same about printing; is there any way to output to file or something similar instead? I tried `time mongo scrappstore --eval 'db.collection.find({"language":"en", "app":"com.my.appp"}).pretty()` and I don't know if it really does the query or not as total time is around 0.1s
[01:32:20] <Sht0> @Boomtime I tried running it several times, doesn't seem to improve much, getting almost same values
[01:32:32] <tsturzl> Why is it so difficult to create a replset with SSL, seems like things change. Like I have to have an admin user on each machine all the sudden.
[01:35:20] <regreddit> so, db.createCollection('foop'); db.foop.insertOne({"FEATURE_ID" : 1397658, "FEATURE_NAME" : "Ester"}); db.foop.find({FEATURE_ID:1397658}); work fine
[01:35:42] <Sht0> @cheeser I'm getting `DBQuery: db.collection -> {"language":"en", "app":"com.my.appp"` as output when running it with pretty() so I think it does the query, but would like to see the actual data if possible by any easy means
[01:37:16] <Boomtime> tsturzl: are you connecting remotely or running mongo on the localhost where the server is?
[01:44:00] <tsturzl> Boomtime: I did this before and got it working, but I had to change the config to not have a replset, create and admin, then change it back
[01:44:57] <Boomtime> certainly sounds it - when you run mongo shell, you specify no hostname or 'localhost' right?
[01:45:22] <regreddit> so I guess my issue is possbly related to importing a CSV, since manually creating the collection an a record works fine
[01:46:19] <regreddit> i'll be glad to create an account forsomeone to ssh in to by pc to prove I'm an idiot
[01:52:44] <gardllok> Hey guys, sorry for the probable noob question but...I am working from two connections to the same ( or what I thought the same ) database on localhost default port. One being a direct connection via cli mongo client ( debian system, just typing "mongo" in command prompt bringing up a mongo cli ) the other is from a python script. However with the python scrupt adding documents to a collection I cannot see
[01:52:46] <gardllok> said added documents from the cli connection. I assume I am not "commiting" as SQL programming would be, but I cannot find how to do such googling. So I think I am asking the wrong question lol. Any points in right direction?
[01:53:44] <regreddit> there's no commit, only a write concern, but case matters, and the db name case matters
[01:55:21] <gardllok> I believe I have identical name, it works the other way around. Where I could add to the database and see it on the python side.
[02:00:26] <gardllok> con2db.predicate_collection.insert_many([{"key":key, "value":value, "initial": True} for key, value in initial.items()])
[02:01:55] <gardllok> I have also tried doing a single insert, which I am able to count for on the python end, by not the mongo cli
[02:03:04] <coconut> i ran the ensureIndex on a field, any future insert will automatically be indexed ?
[02:10:09] <regreddit> gardllok, what if you roll back to something much simpler first, like an insertmany right out of the pymongo docs?
[02:12:19] <coconut> i have an algorithm written in haskell that compares the similarities of two text. i use it to find the most similar text among a pool of artciles. now i put everything inside mongodb, so i need to reimplement everything with mongodb query language?
[02:13:20] <gardllok> Ack, I figured it out, you called it...didn't call the right database from python. Not sure why I couldn't see the alteration from mongo cli's show collections. but whatever, it works now....sorry to waste your time :/
[02:21:41] <kl0neD> Hi, looking for some help. I'm trying to export some data from a table that's populated by some web forms. JSON data looks like this: "Fields" : [ { "_id" : ObjectId("000000000000000000000000"), "FieldFriendlyName" : null, "FieldName" : "email", "FieldValue" : "email@yahoo.com" }
[02:22:19] <kl0neD> Is there anyway to get the FieldFriendlyName as a column header in a CSV export?
[02:52:49] <regreddit> cheeser, I guess i'll just whip up a script to do the import for me, I'm not going to waste any more time on the issue, it's probably local to my dev box, so no reason for me to lose any brain cells over it
[05:25:03] <Waheedi> so i can do --prefix for mongodb with scons? and then scons install will get them in the right place?
[06:34:46] <Waheedi> error, using scons with --prefix https://gist.github.com/dakwak/926aca15d40dd1ead2c4
[09:44:14] <Rumbles> Hi, I set up an arbiter last week, and configured it with log rotation to test I had all the config right. The logs there are rotating as I would expect, so I then took that config and applied it to another db-node at the end of the week, I've checked this morning and although the config is the same, I am getting 2 files created each day, 1 mongodb.log-20160116.gz and mongodb.log.2016-01-16T06-43-30 also created as an em
[09:44:14] <Rumbles> pty file, can anyone help me understand why? Further info: http://fpaste.org/311910/
[09:45:24] <Rumbles> I can't see anything in the logs about this happening, but I'm guessing mongo is creating the extra log file as part of the -SIGUSR1 call...
[10:13:49] <Rumbles> anyone got any idea why I'm getting extra log files on log rotation? http://serverfault.com/questions/750434/mongodb-log-rotation-creates-an-extra-empty-log-file-each-time-it-runs
[11:20:24] <Tzahush3d> I was wondering how would i design the database of a forum with catagories, in it forums, and in it posts.
[12:05:51] <Stevko> Hello. I have a question. I want to use mongoexport to export results of the following into csv: “find({"lastModified" : {$gt : new Date((today = new Date()).getFullYear(), today.getMonth()-1 , 1)}})” (everything that has lastModified newer than some calculated date – in this case first day of last month). How to do it? mongoexport does not like my query.
[15:08:01] <admiun> hi, does anyone here know if it’s possible to configure wiredTiger cache size in mb instead of gb in the config yaml file?
[15:14:07] <admiun> i did find a workaround on the commandline that seems to work for others -> https://groups.google.com/forum/#!topic/mongodb-user/GLrp-H31YWg
[15:15:18] <m3t4lukas> admiun: remember that you probably don't want to use it that way in production
[15:15:19] <admiun> yup, got this one: ‘Error parsing option "storage.wiredTiger.engineConfig.cacheSizeGB" as int: Bad digit "." while parsing 0.24’
[15:15:45] <admiun> well, don’t really have a choice, it uses too much ram now so it’s either this hack or the OOM killer ;)
[15:16:35] <m3t4lukas> admiun: if 1GB of RAM is too much usage for you in production you should consider a stronger machine
[15:17:21] <admiun> it’s actually not production, dev and staging machines have both mongo as well as our app server on them, hence the cache size tweaking.
[15:18:05] <admiun> you got to admit it’s a bit weird to be able to only specify the cache size in whole gigabytes ;)
[15:18:29] <tantamount> When doing a projection in aggregation, is it possible to add a field only to the first element in a collection field?
[15:19:10] <m3t4lukas> admiun: if the chache size is waay too small it could be that it gets way too slow. What also could problems is querying a collection larger than the cache size (or better: pulling data from a collection where the amount of data exceeds the cache size)
[15:19:54] <m3t4lukas> admiun: if that's the case something went wrong with your querying or indexing anyway
[15:20:25] <m3t4lukas> but the more cache you have the better. Especially with a raising number of requests
[15:20:35] <admiun> yeah that will probably not be an issue as these are development servers. and if it is i’ll just use a bigger instance type. but i’d like to first tweak the cache size and then take it from there
[15:20:59] <admiun> it’s not production loads so yeah, there are like 2 clients connected to it at all times ;)
[15:23:33] <admiun> anyway, it’s a bit weird that they chose whole gb as an integer for this field. you can’t raise it to 1.5 either, for example. but i’ll try the commandline workaround then and perhaps file a ticket
[15:37:53] <cruisibesares> hey mongoers. I have a really old client driver that doesn't support replsets. I need to make our mongo setup able to fail over automatically while the devs work on updating the driver. Im planning on using haproxy with health checks against the rest endpoint to determine the primary
[15:45:48] <Ben_> I don't know whether that async driver provides the whole featureset of the sync driver
[15:47:25] <NotBobDole> Hi all. Mongorestore has been taking 3 days to restore a bcakup of 273GB that was accross 9 shards to one shard.Anything I can do to make this run faster? Recommended specs, parameters, etc? We're doing the restoration on 1TB of SSD storage in AWS
[15:48:27] <NotBobDole> Doing the restoration of one shard took 30 minutes, but my scripted loop has been going for over 4 days. Shouldn't have taken but 5 hours unless I missed something
[15:49:32] <someone235> Hi, when I'm using db inside $where I get this error: "mongodb ReferenceError: db is not defined near"
[16:16:02] <tantamount> Is it possible to run an update query that replaces the value of a field using a value found elsewhere in the document?
[16:16:09] <Ange7> i use mongoexport → 1100 rows exported. and i use mongoimport → 600 rows imported. I don't understand why ? and i don't know how to debug this ?
[16:22:16] <cheeser> you can do a find(), iterate over the results, update that value, then save it back.
[16:31:16] <tzahush3d> Hello, so i am writing a forum application which has the following db realted: Catagory in it a forum, in it threads, and in it posts, what would be the best way of making that kind of system?
[16:40:36] <tantamount> Also how do I disable echo in Mongo scripts? Every time I do an assignment like `i=0` it prints the result to the console
[17:42:37] <nawadanp> Hi guys. one of our instance is starting since 1 hours. It's storage engine is WT, with snappy compression. Logs are stuck on wiredtiger_open config: create,cache_size=80G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0)
[17:42:56] <NotBobDole> Hi all. Mongorestore has been taking 3 days to restore a bcakup of 273GB that was accross 9 shards to one shard.Anything I can do to make this run faster? Recommended specs, parameters, etc? We're doing the restoration on 1TB of SSD storage in AWS
[18:07:31] <regreddit> if i create a compound index of say three properties, and only query on one of those properties, is that index any less effective than an index on atht property only? will it be used at all?
[18:07:59] <regreddit> should i create overlapping compound AND single property indexes to cover both cases?
[18:10:47] <regreddit> im tryin to figure if im just wasting space creating indexes on compound properties AND individual properties
[18:11:07] <regreddit> since i have a very large collection that gets queried a LOT by several parameters
[18:13:08] <regreddit> so, conversely, is a compund index unnecessary if i also index each property individually, and do multi-property AND single property queries?
[18:14:25] <cheeser> but some indexes might apply to multiple queries depending on how they're defined.
[18:16:04] <regreddit> i see - the set of queries against the collection are fixed and known, so that should be easy enough
[18:34:32] <bros_> I am using the Javascript bindings. Converting an ObjectId to a string is *killing* my loops performance wise. Anything I can do to just get the string?
[18:36:10] <cheeser> you're converting the ObjectId to a string ... why? (not a js user so i'm probably missing something)
[18:48:24] <cheeser> ObjectId("507c7f79bcf86cd7994f6c0e").equals(ObjectId("507c7f79bcf86cd7994f6c0e")) ?
[18:49:58] <bros_> I'm profiling a really tight loop that is running 20k times and taking 800ms, so I wonder if this will be any better
[19:15:23] <bros_> That's what I thought. I'd also be worried that writing 1mb or so in/out to cache might not be the fastest
[19:44:03] <cycliam> cheeser: I set it to enabled just now in the config file. I'm asking if there's a way to tell the server to request auth every time without having to restart.
[20:30:22] <mvno_subscriber> hi, i'm pretty new to mongodb. i'm having an issue where the json returned by mongodb has values wrapped in NumberInt(). for example {"stuff": NumberInt(2)}. this is really annoying since it's not valid json. how can i just show the number without the wrapper?
[20:34:04] <mvno_subscriber> can i convert it to float somehow then? i don't mind if it ends with .0
[20:36:12] <GothAlice> I.e. what gets printed out in the REPL represents JavaScript objects, not JSON. You'd need to encode it as JSON to get JSON.
[20:38:38] <GothAlice> NumberInt is a special case because JavaScript doesn't do integers. (All numbers are floats, giving 54 bits of integer accuracy.)
[20:38:53] <mvno_subscriber> i tried that, but the returned value still contains the NumberInt()s. I'm using MongoChef and intelliShell if that is any help
[20:39:39] <GothAlice> The ODM/DAO I use offers .to_json() on all documents, which does The Right Thing™.
[20:40:30] <mvno_subscriber> i switched to proper shell output and it worked, no need for json.stringify even. awesome!
[20:45:55] <mvno_subscriber> i have another question if you don't mind.. i've spent so many hours on this. i have a date that is a NumberLong (epoch time). i want to convert this to a date, i.e. year, month and date (time must be truncated or 00:00 since i use it for grouping). I can't for the life of me figure out how i do this, although i assume it should be pretty basic. http://pastebin.com/MJTsYv5B.
[21:00:17] <GothAlice> mvno_subscriber: If your "date" is actually a number, then it's not a date, it's a naive UNIX timestamp. This makes it extremely difficult to work with.
[21:01:03] <GothAlice> If it were a date, https://docs.mongodb.org/manual/reference/operator/aggregation-date/ would apply.
[21:03:52] <GothAlice> Luckily, for your use (grouping by dropping hours) you _might_ be able to get away with a modulus subtraction; i.e. remove the "remainder" (in seconds). (value - (value % 86400) would strip the time from the timestamps.)
[21:07:17] <mvno_subscriber> seems like new Date("$field") and getting $year, $month and $day kind of works.. think i have to shave off milliseconds though
[21:07:55] <mvno_subscriber> i tried to $concat them, but since they're ints $concat threw a fit, and according to stackoverflow i have to use $substr to convert them to a string
[21:11:04] <GothAlice> Or, since you're just dropping the entire time aspect of the date+time represented by the timestamp integer, the modulo approach will give you an integer "snapped" to midnight the morning of whatever date/time it represents.
[21:11:18] <GothAlice> No string manipulation. No object construction, and can be entirely done within a projection mongodb-side.
[21:14:54] <GothAlice> {snappedDate: {$subtract: ["$originalDate", {$mod: ["$originalDate", 86400]}]}}
[21:18:09] <GothAlice> UNIX timestamps are just the number of seconds (potentially floating point, thus microseconds) since the UNIX epoch. I _believe_ that's Thursday, January 1st 1970.
[21:18:39] <GothAlice> UNIX timestamps also explicitly state that each day is a regular number of seconds; leap seconds are added by having the same second happen twice. This makes it very math-safe.
[21:19:06] <GothAlice> Though, without a timezone, your timestamps _better_ be in UTC because otherwise leap days can screw absolutely everything up.
[21:19:42] <GothAlice> UTC has no daylight savings, unlike GMT, so UTC is always the safest way to store dates.
[21:21:15] <mvno_subscriber> yes, we use UTC all over the place in backend. but mongodb is totally new to me and really different from sql. the syntax is also quite different from what i'm used to
[21:21:35] <mvno_subscriber> for some reason, sitting here in the night it starts to sink in (i've been fighting this all day) :)
[21:21:49] <GothAlice> https://docs.mongodb.org/manual/reference/operator/aggregation/mod/#exp._S_mod < the documentation tends to be extremely good, for MongoDB.
[21:22:17] <GothAlice> Sometimes it just takes another set of eyeballs on a problem. Especially if there's sleep depravation in play. ;)
[21:23:26] <mvno_subscriber> hmm, i tried using new Date({$subtract…}), but it got printed as ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ")
[21:24:21] <GothAlice> Uh, yeah, that's a MongoDB projection, not something you can apply to JavaScript objects.
[21:28:41] <GothAlice> Once you get the results back, you're in your application which I'm assuming is JavaScript, because of the "new Date" references you've made.
[21:29:37] <GothAlice> https://docs.mongodb.org/manual/core/read-operations-introduction/#projections is the documentation for this.
[21:30:11] <GothAlice> So you can choose to do the date rounding/snapping within MongoDB itself at the time you make your initial query, or you can do it after in your JavaScript code.
[21:31:50] <GothAlice> Personally, I always try to make MongoDB do as much of the "heavy lifting" as possible. ;)
[21:32:01] <mvno_subscriber> ok, so i can't use javascript in the query or in the projection. i got a bit lost in the JS talk, then
[21:32:32] <mvno_subscriber> i've used the mongodb documentation extensively, but i often find that there are some basics that have eluded me. takes time to get it..
[21:33:21] <GothAlice> mvno_subscriber: Correct. MongoDB expects, effectively, certain JSON structures for queries and projections. https://docs.mongodb.org/manual/core/crud-introduction/
[21:37:12] <mvno_subscriber> i often seem to find methods that aren't documented in docs.mongodb (at least when i search). i stumbled upon $dateToString, but this is not found when i search the doc site. i have to use google for that
[21:39:19] <GothAlice> https://docs.mongodb.org/manual/reference/operator/aggregation/project/#pipe._S_project < many of the fancy / most useful ones are "aggregation pipeline" operators.
[21:39:31] <GothAlice> I.e. $dateToString is found under https://docs.mongodb.org/manual/reference/operator/aggregation-date/
[21:43:12] <mvno_subscriber> all that is left is having the output be comma separated json objects in an array - but i guess i can do that with a regexp
[22:21:13] <GothAlice> I'm sorta building my own method to serialize complex objects to MongoDB: https://gist.github.com/amcgregor/c6ae92325a941ba70568?ts=4
[22:24:11] <regreddit> i did have to write an intermediate script to parse the JSON and convert it into the doc I needed to insert into DB