[06:30:43] <jbu> hi all, I have an object which must keep track of a potentially long (worst case) list of IDs (ints) and possibly check for existence in this list as well as remove elements from the list. I'm new to document DBs, and realize this might be better suited for RDBMS, but is there a good way about doing this for mongodb?
[06:33:03] <jbu> nevermind, i was able to find a solution to this common issue
[10:12:39] <mick27> Hey folks, I am trying out cloud manager, my replica set has issues, for instance the primary has high cpu all the time north of 90%, where should I start to understand what's up
[10:28:43] <Zelest> Who may I contact regarding a typo/bug in the documentation at http://mongodb.github.io/mongo-php-library/tutorial/crud/ ?
[10:30:19] <Derick> jira.mongodb.org/browse/PHPLIB - or make a PR against https://github.com/mongodb/mongo-php-library/blob/master/docs/tutorial/crud.md :-)
[16:43:57] <serff> i'm trying to insert a json document in a file into mongo that has a "$date" / "$currentDate" and it's telling me fields can't start with "$", yet every example I see does it this way. I need the document to be a valid json document, so I can't use ISODate()/new Date(). can anyone help?
[16:44:31] <StephenLynx> in some places you can refer to a field using $
[16:44:40] <StephenLynx> that doesn't mean the name of the field contains it.
[16:44:45] <cheeser> field names in documents that are stored can't start with "$"
[16:45:12] <serff> so how would you insert a document with a date?
[16:45:18] <cheeser> in an aggregation pipeline "$foo" means "use the value of the field named 'foo' rather than the literal value of 'foo'"
[16:57:28] <serff> cheeser, i can’t put new Date() in the json document. it’s not valid json.
[16:57:49] <cheeser> the java driver doesn't work on json documents
[17:01:11] <serff> like i said, my real problem is im using a JOLT transform first. so I go json -> JOLT Transform -> Java convert to Document -> Mongo
[17:01:18] <serff> the json has to be valid json to transform it
[17:01:46] <cheeser> how are you parsing the json to get it in to the driver?
[17:02:30] <serff> final Document doc = Document.parse(new String(content, charset));
[17:11:58] <cheeser> if it was going to fail, it'd fail on the parse. after that, there's a java.util.Date in the Document not a subdocument with "$date" in a field name.
[17:14:00] <serff> ya, that’s what i would think…weird…looking at what’s different..
[17:14:34] <cheeser> can you post a recreation of this somewhere?
[17:15:07] <serff> does $currentDate work in that same test?
[17:27:38] <saml> actual task is find all <iframe> of certain video hosts. and find canonical id of the video using each video provider's api
[17:27:52] <hardwire> you could use pymongo in a snap
[17:27:56] <saml> so such html is in various different fields
[17:28:24] <cheeser> you'd be better off pulling the html through parser and pattern matching the DOMs
[17:28:50] <saml> field names are generated dynamically. so it's like components.fieldtype[i].fielddata
[17:29:30] <saml> components is an array of objects where a key could be video[0] or video or videoembed or video[2] or movieembed or html[24] ...
[17:30:25] <saml> i could walk components tree of each document. find html-like fields. make an html element out of them. run html parser. find <iframe> . filter by hosts
[17:31:37] <hardwire> or you could use mongodump or mongoexport
[19:27:59] <kurushiyama> "Safe" in the meaning "You are guaranteed to get you data back"? No, not at all. The database is guaranteed to be in a usable state after you ran --repair.
[19:28:04] <cheeser> probably your only option short of a restore from backup. nothing in the logs?
[22:03:04] <alexi5> what type of applications is mongodb normally used in ?
[22:05:44] <kurushiyama> alexi5: That is quite broad qustion. Personally, I have sse
[22:06:21] <kurushiyama> seen it from booking data over social media analysis to time series data
[22:06:37] <alexi5> i think it is better for me to tell what I am thinking of using mongodb for
[22:06:55] <kurushiyama> Aye, that would be easier..
[22:07:17] <alexi5> Intrested in developing a catalog application for our Engineering department that can store information about various pieces of equipment at various mobile sites
[22:07:50] <alexi5> as well as their settings , and images of the layout of the sites
[22:08:24] <alexi5> i did a Er model on paper and came up up with about 25 tables
[22:09:10] <alexi5> i came up with 3 collections for 3 different models using json
[22:10:41] <kurushiyama> alexi5: Well, from the start: Get your user stories done first, derive the questions you need the data to answer and then do optimized data modelling, focussing on squeezing out the best performance for the the most common use cases
[22:10:42] <alexi5> i like the document model better as it can be slow extended as new equipment are added with different attributes
[22:11:25] <kurushiyama> Upfront data modelling more often than not leads to tears and pitas.
[23:17:52] <kurushiyama> tombin: Yes, and the docs, as far as I can see, require quite a bit more: https://docs.mongodb.com/manual/reference/configuration-options/#net-ssl-options
[23:27:20] <kurushiyama> mick27: Is there something specific you want to know?
[23:29:52] <mick27> kurushiyama: Yeah I checked that. well I am still having issues when switching to a new primary, the CPU goes at 100% forever, and I can't seem to find out what's wrong, I can pretty surely rule out i/o issue for the disks. so I am looking at what gets activated on the new primary that could cause this
[23:30:21] <mick27> I googled all king of variations of my problem and haven't found something solid
[23:31:08] <kurushiyama> define "switching" and "new", please.
[23:39:07] <kurushiyama> mick27: The thing is that fiddling with priorities can have side effects. May that cause your problems? Maybe. May your VMM's real time clock be off, making your newly elected primaries virtual head spin? Very possible ;)
[23:39:58] <cheeser> add new rs members on the new hardware. once the replication is caught up (you can copy files to speed the sync) you can do an rs.stepDown() and remove the old members.
[23:40:12] <mick27> but somehow the new ones are geting crazy with the cpu to the point it causes a disruption
[23:40:19] <kurushiyama> So why do you need to have a selected member to be primary?
[23:40:33] <cheeser> reset the priorities to the defaults (1, iirc) and try again
[23:40:38] <kurushiyama> mick27: Well, as I said: check VMMs real time clock, for starters
[23:41:28] <kurushiyama> mick27: if members are more than a second apart, things can become pretty nasty.
[23:54:35] <kurushiyama> mick27: What I'd do is remove the new members, remove their dbpath's contents, reset the priorities (if necessary), then add a member on the new machines and see what happens