pmxbot IRC Log Viewer

[00:38:34] <tejasmanohar> hey! is it possible to make objectid's (or a similar id?) have a field that starts at say 0 or 1 and goes 2 3 4 5 etc

[00:38:41] <tejasmanohar> rather than the default long strings

[00:38:50] <tejasmanohar> basically, i want to track a "position"

[00:39:56] <Boomtime> https://docs.mongodb.org/v3.0/reference/object-id/

[00:40:23] <Boomtime> _id is just a field like any other, you can put whatever you want there - including a simple number

[00:40:45] <tejasmanohar> oh i see, just override it rather than have mongo generate one for you

[00:40:56] <Boomtime> alternatively, if you really really want ObjectID as the type, the you can generate your own - i recommend following the format as specified in the above link though

[00:41:18] <tejasmanohar> right so not a 0 -> 1 -> 2 -> 3 etc model like say postgresql does it

[00:41:31] <tejasmanohar> so there's no such thing as an auto-incrementing value in mongo? no way to do that easily?

[00:42:13] <Boomtime> no, the only value the server will generate for you is the _id as an ObjectID, though most drivers will detect that missing and insert it themselves before it even gets to the server

[00:42:34] <tejasmanohar> ah

[00:42:47] <Boomtime> for the most part, the server attempts to be a "what you store is what you get back" type of engine

[00:42:52] <tejasmanohar> ah this may do https://github.com/codetunnel/mongoose-auto-increment

[00:42:53] <tejasmanohar> yea

[00:43:09] <Boomtime> so, any framework that does it is simulating and may have conditions on how that works

[00:43:27] <tejasmanohar> yea, looking @ that

[00:43:37] <Boomtime> you should make sure you understand the caveats - such as what happens if multiple clients attempt to insert new documents

[00:43:39] <tejasmanohar> race conditions is all im afraid of

[00:43:42] <Boomtime> right

[00:43:44] <tejasmanohar> right

[00:43:56] <tejasmanohar> that said i'm generating a finite number of things _once_ and storing it in mongo

[00:44:02] <tejasmanohar> so that's not as much of a concern i suppose

[00:44:17] <Boomtime> indeed, you are in a strong starting position

[00:44:24] <tejasmanohar> yep

[01:09:52] <tejasmanohar> trying to gauge whether i should store the data in mongodb or static json file that's read into memory by my app at start

[01:09:55] <tejasmanohar> it's all static Boomtime

[01:10:26] <tejasmanohar> structure is roughly like this https://gist.github.com/tejasmanohar/aed1e9cc340cc0544cdb but maybe ~500k objects in the array

[01:10:35] <Boomtime> how big is it? it might just come down to convenience

[01:10:53] <Boomtime> 500,000 objects X avg.size?

[01:12:05] <tejasmanohar> 6 1-2 digit numbers in the array, position is 1-500k, batch is 1-10, stack is A-J String (more like character), line is A-e string (character)

[01:12:14] <tejasmanohar> but yeah nvm position is number from 1-1k

[01:12:25] <tejasmanohar> but yeah not _much_ data i suppose, not too good at size estimation

[01:12:33] <tejasmanohar> https://gist.github.com/tejasmanohar/aed1e9cc340cc0544cdb

[01:12:46] <tejasmanohar> that's the structure that i described, array of 500k length let's say

[01:12:52] <tejasmanohar> would you recommend in-memory for that?

[01:14:09] <tejasmanohar> and why ? :) (so i can learn :D)

[01:15:21] <Boomtime> so, those look tiny

[01:16:03] <tejasmanohar> yea

[01:16:08] <Boomtime> but if you have a mongo shell nearby - you can make up one 'document' (one entry from the data array) and ask mongo shell what it's size would be in bson

[01:16:10] <tejasmanohar> i wonder how much memory that'd take up hmm

[01:16:15] <tejasmanohar> oooh nicee

[01:16:17] <tejasmanohar> lemme do that

[01:17:43] <Boomtime> in the shell -> Object.bsonsize( {your_json: "full object defintion here" } )

[01:18:04] <Boomtime> i assume you are semi-decent with json

[01:18:08] <Boomtime> json/javscript

[01:27:00] <tejasmanohar> yeah haha

[01:29:53] <tejasmanohar> 162 was returned. that's bytes i guess? Boomtime

[01:30:08] <Boomtime> yes

[01:30:12] <Boomtime> tony

[01:30:14] <Boomtime> tiny*

[01:30:24] <tejasmanohar> 80 megabytes yea

[01:30:55] <tejasmanohar> performance wise, is in memory better than database for this?

[01:31:03] <tejasmanohar> assuming it's gonna be accessed frequently Boomtime

[01:31:05] <Boomtime> right, so mongodb will mean you don't need to load it really - it'll jjust be present and searching could be faster etc

[01:31:11] <tejasmanohar> yea

[01:31:13] <tejasmanohar> ah i see

[01:31:18] <tejasmanohar> that's what i was getting at yea

[01:31:57] <Boomtime> you won't need to re-implement searching - although with a good library brute-force might be ok across that little data anyway

[01:35:06] <tejasmanohar> ah

[04:38:35] <Freman> someone so kindly told me how to get the size of a document... but I forgot and I'm only just getting around to 'fixing' the 'issue' that required the knowledge... Can anyone please jog my memory?

[04:40:13] <cheeser> Object.bsonsize(db.test.find({test:"auto"}))

[04:40:37] <Freman> thank you

[04:41:04] <cheeser> np

[04:58:45] <Boomtime> findOne

[04:59:05] <Boomtime> if you use find you'll just be measuring the size of a cursor when encoded as BSON

[05:04:33] <cheeser> true

[09:45:27] <pehlert> Hey folks. Is there anything like a $setOnUpdate operator, similar to $setOnInsert to set a property if the document has changed during an upsert operation?

[09:46:54] <Zelest> Derick, oh, what's the difference? :o

[09:54:18] <Derick> Zelest: new APIs, leaner driver, to be used with a PHP library on top

[09:55:37] <Zelest> Derick, Ah, how much effort does it require from the developers to modify the old code?

[09:55:45] <Zelest> As in, will it be compatible with MongoDriver ?

[09:57:44] <Derick> nope, it's not compatible (on purpose)

[09:59:16] <Zelest> Ah

[09:59:50] <Zelest> Any ETA when it's being released?

[10:04:59] <Derick> Zelest: the php 7 one? *hopefull* in the next few days, but likely not :)

[10:08:00] <Zelest> Derick, Nice :)

[11:15:29] <livcd> i set environment vars for pw,username and dbname and started mongodb container but can't login using this credentials

[11:26:58] <livcd> doh i found out that i passed test, as dbname...damn colon

[13:12:31] <yopp> what does "SHARDING [LockPinger] handled late remove of old distributed lock with ts" mean?

[13:12:45] <yopp> This started after 3.0 -> 3.2 upgrade

[13:13:04] <yopp> And it's like tens of log entries per second on each mongos :|

[15:07:39] <evgeniuz> Hi. Can someone help me, please: what are the downsides of running sharding config servers on the same hosts as shard servers?

[15:09:30] <deathanchor> evgeniuz: imagine one of those machines spontaneously combusted.

[15:10:11] <deathanchor> you would have to recover a shard member repl and a config svr.

[15:10:26] <evgeniuz> ok, then secondaries become new primaries, and they too run config servers

[15:10:40] <evgeniuz> seems ok to me

[15:10:48] <deathanchor> evgeniuz: you can only have 1 or 3 configsvr, not more or less.

[15:11:03] <evgeniuz> deathanchor: that's not true as of 3.2

[15:11:08] <deathanchor> ah

[15:11:13] <evgeniuz> config server is just replica set

[15:12:45] <evgeniuz> I mean imagine this config (shard 1: [pri + conf] [sec + conf] [sec + conf]) (shard 2: [pri + conf] [sec + conf] [sec + conf]) etc.

[15:13:03] <evgeniuz> where [ * ] is host

[15:14:31] <evgeniuz> I understand that if any two servers in one shard is down, then some of them becomes primary, at least one config is intact, so it becomes primary config as well and everything is ok

[15:15:21] <evgeniuz> and even if one shard is totally down: config is present in another shard too, so that becomes primary and single shard still functional

[15:26:15] <deathanchor> evgeniuz: there is nothing wrong with it. just a slightly more complex setup. We used to have our configs on the same host as some replmembers, we eventually split it out just for ease of setup

[15:26:29] <deathanchor> we use m1.small aws instances.

[16:09:08] <ogoffart> Hi, i would like to create a compilation_commands.json for mongodb to be used with the clang tools.

[16:09:14] <ogoffart> by googling i saw there is a file site_scons/site_tools/compilation_db.py

[16:09:17] <ogoffart> but i don't understand how it is supposed to be used. Anyone can help me?

[16:26:56] <ogoffart> (and btw, i tried do build mongo db with a git clone with --depth 1, but this failed because of some version check)

[17:58:55] <Rashad> Hello.

[17:59:01] <Rashad> What's a way to store sets in mongodb?

[17:59:09] <Rashad> Sets = arrays without order.

[17:59:10] <Zelest> sets?

[17:59:28] <Rashad> The reason for this is that I need to store conversations in a chat app.

[17:59:49] <Rashad> The conversation will be identified by the user ids of the two people chatting.

[18:00:05] <Rashad> This is ideally a set.

[18:00:06] <Zelest> In what way is an array bad?

[18:00:17] <Zelest> In MongoDB you store a JSON object..

[18:00:27] <Zelest> Either store a json object per message

[18:00:27] <Rashad> It's bad because when you try to query the db later, you will need to remember the order of the ids in the array.

[18:00:51] <Rashad> Or check the db two times for this array: [1, 2] or this array [2, 1], where 1 and 2 are user ids.

[18:01:38] <Rashad> JSON object will require naming the ids.

[18:02:15] <Rashad> I can try something like this: {"_1": "_1", "_2": "_2"} and it'll solve the problem.

[18:02:26] <Rashad> But you understand how sets would be more ideal.

[18:03:27] <Rashad> So is this my only option?

[18:03:32] <Rashad> Or should I even reconsider the schema?

[18:03:53] <Zelest> wut?

[18:04:08] <Zelest> in what way is {"_1": "_1", "_2": "_2"} not naming the objects?

[18:04:10] <Rashad> What is it that is confusing to you?

[18:04:24] <Zelest> that *is* a JSON object

[18:04:45] <Rashad> It is naming.

[18:04:51] <Rashad> I have adobted your solution in that example.

[18:04:58] <Zelest> Ah

[18:06:05] <Zelest> Personally I would probably store each "line" in the chat as an object..

[18:06:22] <Zelest> with the user posting it along with whatever other metadata it require

[18:06:34] <Zelest> Not sure if that's the best solution though

[18:06:52] <Rashad_> Sorry I lost connection.

[18:07:00] <Zelest> [19:00:52] < Zelest> Personally I would probably store each "line" in the chat as an object..

[18:07:04] <Zelest> [19:01:09] < Zelest> with the user posting it along with whatever other metadata it require

[18:07:21] <Rashad_> OK I will do that.

[18:08:33] <Rashad_> Thanks.

[18:35:27] <Rashad_> I have the following structure: http://hastebin.com/vipaxoyowi.json

[18:35:33] <Rashad_> How do I append to the messages array?

[18:36:26] <Rashad_> I am using Node.js

[18:37:10] <Rashad_> I basically want to find the embedded array.

[18:39:19] <StephenLynx> updateOne({your query},{$push:{messages:{new object}})

[18:41:03] <Rashad_> StephenLynx: Thanks.

[19:04:25] <dman777_alter> does mongo automatically index timestamp field?

[19:17:29] <cheeser> the only field automatically indexed is _id

[19:18:12] <dman777_alter> cheeser: thanks

[19:40:31] <jjulian> hey does anyone know how / why even with --disable-warnings-as-errors the build is failing for a warning? im trying to build mongo 3.2 with scons on a freebsd 9.2

[19:42:08] <jjulian> 9.3 sorry

[20:43:59] <jjulian> fun fact, i played around with the SConstructs file and now it seems to compile BUT now -Werror is added and it does not fail anymore. thats weird

[20:58:25] <jjulian> in fact on freebsd adding -Werror stops cancelling on warnings

[20:58:32] <jjulian> so it seems the other way around

[21:07:52] <jjulian> im sorry that was wrong. it was just no cache, i thought it came further in compiling but did not. -Werror also just changes warnings into errors

[21:26:59] <codecab> are there any limitations in terms of file size of the dumps when using mongorestore?

[21:27:35] <codecab> to be specific, the dump file in question is about 2 mb

[21:39:20] <StephenLynx> not that I know, why?

[21:39:30] <StephenLynx> can't restore?

[22:46:21] <codecab> yes i'm having trouble restoring my database

[22:47:39] <codecab> i'm trying to migrate from meteor.com to modulus.io and can't figure out what i'm doing wrong

[22:48:51] <codecab> i used mongodump to create a dump from the database provided by meteor.com

[22:49:27] <codecab> but when i try to upload it to modulus.io, parts of my data get lost

[22:51:03] <codecab> at first i thought its just one collection that is affected, but i just found another error in my data so my guess is i'm doing something fundamentally wrong

[22:52:34] <StephenLynx> oh

[22:52:56] <StephenLynx> I thought you were dealing with an actual server.

[22:53:00] <StephenLynx> is something with modulus

[22:53:03] <StephenLynx> not mongo

[22:54:05] <codecab> i am using mongorestore to upload my dump to a fresh database

[22:54:14] <StephenLynx> ah

[22:54:33] <StephenLynx> what kind of data is missing?

[22:54:51] <StephenLynx> does the operation finishes or you get an error?

[22:54:58] <codecab> a whole collection

[22:55:12] <StephenLynx> are you sure all the collections are on the same database?

[22:55:21] <codecab> yes

[22:57:01] <codecab> the collections i'm using were all created in the same way, all of them sow up, except one

[22:57:15] <codecab> (that happens to be the most important one, of course)

[22:57:16] <StephenLynx> on the source database

[22:57:19] <StephenLynx> when you use show collections

[22:57:22] <StephenLynx> they all show up?

[22:57:45] <codecab> is that a mongo shell cmd?

[22:57:58] <StephenLynx> yes

[22:58:10] <codecab> yes they do

[22:58:12] <StephenLynx> hm

[22:58:30] <StephenLynx> what arguments are you using on mongodump?

[22:59:00] <codecab> mongodump -u -d -h -d

[22:59:04] <codecab> eh

[22:59:07] <codecab> mongodump -u -p -h -d

[22:59:22] <StephenLynx> user, port, host and -d stands for destination directory?

[23:01:29] <codecab> well, the db url is something like mongodb://xxx:123@foo.meteor.io:27017/bar_baz

[23:01:38] <codecab> and i used > mongodump -u xxx -p 123 -h foo.meteor.io:27017 -d bar_baz

[23:02:13] <codecab> so i ended up having a directory dump/bar_baz/

[23:02:17] <StephenLynx> yes

[23:02:24] <StephenLynx> ah, yes, -d is the database

[23:02:36] <StephenLynx> ok, go in bar_baz and see which files you got there.

[23:03:07] <codecab> a .bson and .metadata.json per collection

[23:03:15] <codecab> with reasonable file sizes

[23:03:21] <StephenLynx> including the one missing on the restore

[23:03:21] <StephenLynx> ?

[23:03:35] <codecab> yes

[23:03:42] <StephenLynx> hm

[23:03:48] <StephenLynx> what command are you using to import?

[23:03:52] <StephenLynx> parameters*

[23:05:25] <codecab> mongorestore -u USER -p PW -h HOST --authenticationDatabase DBNAME --db DBNAME bar_baz

[23:05:42] <StephenLynx> weird.

[23:05:45] <StephenLynx> no idea, then.

[23:05:59] <StephenLynx> what is the name of the missing collection?

[23:06:14] <codecab> orders

[23:07:36] <StephenLynx> is the version of the source db the same as the one on the destination db?

[23:07:53] <codecab> how do i check that?

[23:12:41] <codecab> nvm, worked now

[23:12:57] <codecab> thank you anyways, for your time and all

[23:13:46] <codecab> i never really expect anything when asking on irc, so

[23:13:46] <codecab> thank u sir

[23:14:37] <StephenLynx> :v

[23:14:43] <StephenLynx> im on vacations

[23:14:56] <StephenLynx> yesterday I even logged on a VPS of a dude trying to install my software

[23:15:11] <StephenLynx> found out he completely broke the software's directory permissions

[23:16:32] <codecab> but why?

[23:16:59] <StephenLynx> because the software is a web back-end

[23:17:03] <StephenLynx> so you dont run to run as root

[23:17:27] <StephenLynx> and personally I dont like to run it as an admin user either

[23:17:54] <StephenLynx> so what I told him I do is to have a directory which both the user that runs the software and the admin can interact with by using group permissions.

[23:18:32] <StephenLynx> so he screwed with permissions and broke in a way you could TELL the system to start the software but it wouldn't actually execute the executable.

[23:19:21] <StephenLynx> oh, and he didnt realized immediately the server wasnt running.

[23:19:46] <StephenLynx> I think I spent at least two hours trying to figure stuff out

[23:20:02] <StephenLynx> because he told me the server was running, I thought he couldnt connect on localhost

[23:20:19] <StephenLynx> then an hour later I as for a ss and the server is not running

[23:21:42] <codecab> that was the moment u wanted to scrag him gently?

[23:21:53] <StephenLynx> no, I wanted to kick his nuts

[23:22:04] <cheeser> /1/1

[23:22:12] <cheeser> bah :)

[23:38:41] <shaileshg> Hi, what does client: secondary means in mongoid?

[23:38:49] <shaileshg> for eg. its written like this in documentation

[23:38:59] <shaileshg> https://www.irccloud.com/pastebin/Mw7VxZC9/

Log file Viewer

Help | Karma | Search:

#mongodb logs for Wednesday the 16th of December, 2015