Log file Viewer

[02:57:06] <RyansWorld> hello everyone I have a question about mongodb with mongoose. Is there a way to prevent duplicate records? Here is what im faced with. I have a CSV sheet that I have to grab from a website and update to my database. There is one field in the schema that is unique and should only ever have one. Is there a way to prevent an insert if the field is already in the DB? or is there a way to scrub the DB after insert and

[02:57:06] <RyansWorld> delete any duplicates?

[04:41:23] <kambei> Morning

[08:18:31] <Derick> moin

[09:32:49] <Booli> I have a bit of a aggragation question

[09:33:44] <Booli> I have documents with an created _id on {id: "someidentificationnumber:YYYYMMDDHH" } so ie. a020a61acec0:2018051809

[09:34:08] <Booli> Its great, because I can match days or months with an easy regex

[09:34:35] <Booli> { $match: { _id: /^ecfabc8593ce:20180513/ } } matches a day specifically, and { $match: { _id: /^ecfabc8593ce:201805/ } } matches the month.

[09:34:55] <Booli> But now I want to group all the days together when I query a whole month

[09:34:58] <Booli> and thats where I am stuck

[09:35:30] <Booli> (I mean I could just query all the days seperatly etc, but I first want to know if I can make the grouping work)

[09:36:02] <Booli> So when I query on /^ecfabc8593ce:201805/ I want to group all /^ecfabc8593ce:20180501XX/ where XX is the hours

[09:36:41] <Booli> so this should be a group /^ecfabc8593ce:20180501/ all the way up to /^ecfabc8593ce:20180531/

[10:57:32] <Booli> maybe mapReduce would be better for it, I don't know

[11:18:45] <Booli> maybe an easier question, how can I create a $date object from an _id like ecfabc8593ce:20180513

[11:18:54] <Booli> I guess it's done with string expression into a date expression

[11:54:39] <Derick> mapReduce is never the answer.

[11:54:54] <Derick> your data model can likely not do this, as you're using values as keys.

[11:55:32] <Derick> you can turn strings into dates by using the aggregation frameworks $dateFromString

[11:57:12] <Booli> sadly I need 3.6 for that

[11:57:24] <Booli> and I only have 3.4.1 available on my IBM cloud

[11:57:29] <Booli> as max version

[11:57:41] <Derick> that I can't help you with :-/

[11:58:12] <Booli> is there a way I can polyfill the dateFromString somehow in a function and load it

[11:58:15] <Booli> probably

[12:00:41] <Derick> no

[12:00:55] <Derick> not in the database itself, that is

[12:04:24] <Booli> not with db.system.js.save and then use it in a $where ?

[12:05:38] <Booli> nope nvm

[12:05:44] <Booli> cant be used in aggregatio

[12:05:45] <Booli> n

[12:08:56] <Derick> also, you probably should avoid any sort of javascript running on the server side

[12:09:03] <Booli> yeah I read that too

[12:09:12] <Booli> just wanted to monkeypatch until Compose can update to 3.6

[12:09:16] <Booli> but hey

[12:09:23] <Booli> I might jsut change my schema

[12:09:31] <Booli> As I am still in the position to do so

[12:09:41] <Derick> I think you should do that in any case, as you're using a value (the date string) as a key

[12:09:46] <Derick> that's often a bad design

[12:10:00] <Booli> well its proposed by mongodb

[12:10:05] <Booli> so you should then remove that from the website

[12:10:09] <Derick> where?

[12:11:29] <Booli> https://www.mongodb.com/presentations/mongodb-time-series-data

[12:11:34] <Booli> slide 43/66

[12:11:57] <Booli> It's one of the first couple of hits when you google mongodb and timeseries data

[12:11:58] <Derick> yeah, we need to do something about these old presentations on time series data

[12:12:11] <Booli> those presentations are the only thing popping up

[12:12:20] <Derick> they're no longer very relevant, especially with the wiredtiger storage engine

[12:12:28] <Booli> exactly

[12:12:34] <Booli> Ok! Off to the drawing board

[12:12:43] <Derick> thanks for that, I'll follow up on that

[12:13:14] <Booli> would you recommend just having a default _id, and adding a "sensor_id" and a time: ISODate()"

[12:13:30] <Derick> sorry, got to run now - prsentation to give. Back in ~1¾ hour

[12:13:35] <Booli> Good luck

[17:02:33] <dougs> hi there, got a strange problem with indexes here. suddenly our mongodb queries with backward sorting started all returning no results. i could fix it only by creating a new index with descending order on the same field of the previous index. tried to search but could not find anything. what could that be?

[17:03:04] <dougs> "suddenly" may be some change i'm not aware, but anyway i don't know where to look...

[18:29:28] <hotpot33> Well, is sorting on a embedded object field as easy as {$sort: { 'x.y': 1 }} ?

[18:29:46] <hotpot33> I guess I'll just try lol

[19:01:25] <alfredocambera> Hello fellas, I'm trying to filter documents (example data https://pastebin.com/gNjmGsHj) from a collection where 'gd' should contain an array, and that array may be named: "SD" or "SOMETHING". I've tried using: '{"gd": {"$in": ["SA", "SOMETHING"]} }' to filter but it seems like I'm doing something wrong

[19:09:28] <alfredocambera> I'm very new to mongo. Could anybody take a look at the example data? Any hint on how to extract the information would be highly appreciated

[20:44:54] <synaptech> alfredocambera: it looks like you're trying to get documents that contain an element 'gd' that contains either an element named 'SD' or 'SOMETHING', but in your query you're looking for docs that have the value SA or SOMETHING in the gd element

[20:46:34] <synaptech> I think you'd want something like {"gd.SA": { $exists: true }}

[20:47:45] <synaptech> note that that doesn't check if the element is an array, just that the element exists

[20:49:03] <synaptech> do the same for gd.SOMETHING and wrap both expressions in an $or: [{<exp1>}, {<exp2>}]

[21:45:41] <alfredocambera> synaptech: thanks a lot!

[21:49:50] <synaptech> np :)