Log file Viewer
#mongodb logs for Monday the 4th of April, 2016
[02:41:08] <kurushiyama> dddh_ https://www.mongodb.com/blog/post/introducing-mongorover-a-new-experimental-mongodb-driver-for-lua
[03:38:33] <Frenchiie> does doing an update on a document mean it has to find the match every time? is there a way to keep a reference to a document once you find it?
[03:38:46] <Frenchiie> so that an update to it doesn't mean having to go through the collection to find it
[05:55:54] <dddh> do not like that there are no prebuilt packages for lua mongo clients in current distros ;(
[05:56:04] <dddh> playing with mongorover: "...ua/luarocks/share/lua/5.2/mongorover/MongoCollection.lua:100: No suitable servers found (`serverselectiontryonce` set): [connection error calling ismaster on '10.14.243.131:27017']"
[10:56:12] <Derick> pamp: not from the database, try removing system log files or something like that first
[12:12:28] <Ange7> kurushiyama: i want insert doc but i don't want wait that insert is finished, i want my code continue,
[12:13:44] <kurushiyama> Ange7 Option 1: reduce the write concern (maybe not ideal, depending on your use case).
[12:14:34] <kurushiyama> Ange7 Option 2: Use an async driver, which depends on your language if such is available
[12:16:13] <kurushiyama> Ange7 Option 3 Spawn a thread (I have no clue how to do that in PHP nor wether this is even possible) and have the thread to the insert.
[12:17:28] <kurushiyama> Ange7 So how does your backend look like? How many frontends do you have? Can / Do you use bulk inserts?
[12:18:18] <Ange7> kurushiyama: found on stackoverflow http://stackoverflow.com/questions/19813713/are-mongo-php-write-operations-sync-or-async PHP Driver can write in async
[12:20:36] <kurushiyama> Ange7 The reasoning there is that the writes will be async only with a writeConcern = 0
[12:23:49] <kurushiyama> Ange7 A write concern of w=0 actually means that your writes will go unacknowledged.
[12:25:20] <kurushiyama> Ange7 Only network and socket exceptions will be returned. To make a long story short, you can not be sure that the write will be accepted. For example, duplicate key errors and alike are _not_ returned.
[12:27:28] <kurushiyama> Ange7 By all means, unless your data is extremely cheap (easy to reobtain), I would _not_ suggest lowering the write concern below 1. More often than not, I even suggest a higher write concern.
[12:43:24] <kurushiyama> dddh__ He pmed me. Will do that later. I just wonder how ops work, nowadays. Not using LVM for a dedicated partition of highly volatile and potentially growing data is... ...dangerous.
[15:23:31] <jetole> Hey guys. Is there a way for me to set multiple users in mongorc and have it auth with the credentials for the specific server depending on what server I am connecting to?
[15:43:09] <krion> i'm running mongo 2.6 and after a mass poweroff one of my configsrv doesn't start (ask for repair)
[16:20:24] <macwinne_> saving 80% on storage. and more importantly went down from 6GB of ram for indexes to 1.2GB.. that's quite a unexpected, yet pleasant, benefit.. not sure why it's not talked about more
[16:37:47] <kurushiyama> macwinne_ There is a subtle thing to keep in mind, though. All those compressions make your instance rely on CPU more than you are used to with MMAPv1
[16:39:10] <macwinne_> cheeser: I was asking this on the weekend.. not sure if you know. Is there an easy way to tell if an index is actually being used without having access to the application code?
[16:39:57] <macwinne_> if I'm just mongo dba, and i don't have time to deal with devs who may be changing queries.. i want to keep an eye out on unused indexes that are unnecessarily eating memory
[16:43:06] <kurushiyama> There isnt. I answered him originally, and I took quite some time to research.
[16:43:17] <StephenLynx> you have 2 options: inspect the log and look for a pattern on queries and compare to the existing indexes
[16:43:58] <StephenLynx> if I were you, I would just take a look on the existing indexes, see the most resource intensive ones and see if they are actually needed.
[16:44:24] <StephenLynx> if its an index that is unnecessary but isn't a big deal, it might not be worthwhile to try and eliminate it.
[16:50:26] <kurushiyama> In general, an since we are repeating this discussion I have the urge to restate it for completeness: I think this is looking for a technical solution for a non-technical problem, namely lack of communication and proper database change management.
[16:51:13] <cheeser> still, technical solutions can't be lazy and not bother to double check themselves or forget to send emails.
[16:55:25] <kurushiyama> We once hit the RAM barrier with a cluster and asked dev to recheck their indices, since some of them looked very odd (indices over non-existing fields and such). They admitted they lost overview. We dropped all indices.
[17:06:15] <saml> Field names cannot contain dots (i.e. .) or null characters, and they must not start with a dollar sign (i.e. $). See How does MongoDB address SQL or Query injection? for an alternate approach.
[17:06:30] <saml> https://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names hehehehhe
[17:06:56] <saml> so i'm writing newsletter signups. we have multiple newsletter accounts (different vendors)
[17:07:05] <cheeser> https://docs.mongodb.org/manual/faq/fundamentals/#how-does-mongodb-address-sql-or-query-injection
[17:08:18] <saml> so i can do db.subscriptions.update({email:email}, {$set: {'lists.vendorkey23.some list name':true}}})
[17:09:05] <StephenLynx> why not just have an array of _ids on each subscriber with the newsletters he is subscribed to?
[17:09:11] <saml> otherwise, i need to fetch a document by email. reconstruct it. and update the whole doc
[17:10:24] <saml> {email:'foo@bar.com', lists: [{vendor:'sailthru', name: 'wow list 1'}, {vendor: 'cheetahmail', name: 'yolo list 2'}]}
[17:11:10] <saml> let's say foo@bar.com unsubscribed from {vendor:'sailthru', name: 'wow list 1'} but subscribed to a new list
[17:12:08] <kurushiyama> Was just about to say that. And you usually need the emails for a given mailing list, like db.subscriptions.find({list:"bar@baz.com",unsub:false},{_id:0,email:1})
[17:17:09] <saml> db.docs.update({email:'a'}, {$addToSet: {lists: {vendor:'s',name:'foo bar'}}, $pull: {lists: {vendor:'a', name:'foo bar'}}})
[17:18:14] <saml> kurushiyama, in your flat model, what's _id? just mongo generated? just set index on email field?
[17:19:27] <kurushiyama> saml No. Store the subscriptions flat. db.subscriptions.insert({email:"subscriber@foo.com", list:"sender@bar.com"}) for example.
[17:20:58] <saml> this is for a web page where a user with an email can subscribe/unsubscribe to a bunch of mailinglists by clicking checkboxes and submit
[17:21:21] <kurushiyama> saml Been there, done that. For KISS (Keep It Simple and Scalable), a flat model is good enough.
[17:22:00] <saml> kurushiyama, in your flat model, you'd just delete the doc for each new unsubscription
[17:36:47] <kurushiyama> The thing is that the most common use case is "Get all non-suspended subscribers to a _given_ newsletter_"
[17:39:04] <kurushiyama> Instead of putting the list name into "subscriptions", it may be viable to even put the email in there. In that case, one query is saved, for very simple mailinglists. Personally, I needed some list metadata to process, hence I had one query for the list and one for the subscribers per incoming mail.
[17:48:59] <kurushiyama> saml Had some smoked turkey sandwiches with home made mayonaise, salad and fried onions.
[19:30:47] <ing8> I need to copy like 7kk rows from one mongo collection in one server to another mongo collection on another server. I cannot place result in the variable because I dont have enough mem and slices and limit/skip kinda slow at 10000 already. What is the best way to do this?
[21:34:00] <uuanton> if I have just primary without secondaries and in connection string secondaryPreferred would it still be able to deliver if I have no secondaries ?
[21:41:17] <uuanton> WARNING: Readahead for /data is set to 4096KB We suggest setting it to 256KB (512 sectors) or less
[21:42:20] <uuanton> i was able to change it to smaller value using sudo blockdev --setra but after instance restart it goes back to 4096
[21:53:43] <uuanton> i've read somewhere that rc.local backward compatible and its better to avoid it on centos 7.
[21:56:29] <kurushiyama> uuanton Whut? I'd like to see that source. rc.local is a mere shell script executed with root permissions.
[21:57:47] <kurushiyama> uuanton You might want to have a look at https://docs.puppetlabs.com/puppet/latest/reference/man/apply.html
[21:58:26] <kurushiyama> uuanton As for your script: you _have_ to check the return values to see if something went wrong.
[23:15:10] <Echo6> Hey, can you guys help me get a count of the number of records created from a start date to an end date?
[23:48:05] <MacWinner> i want to eventually move to a mongo sharded cluster.. i currently have a 3-node replicaset with about 4 application servers talking to it. Should I spin up mongos instances and have my apps talk to mongos as the intermediary before moving to the sharded cluster?
[23:48:39] <MacWinner> i want to minimize the number of changes I do at the time of actually sharding.. so i thought getting the application architecture to talk to mongos completed first may be prudent
[23:49:34] <MacWinner> basically questin is does mongos require an actual sharded cluster, or can it talk to a simple replicaset and then automatically talk to a sharded cluster once it's configured