Log file Viewer
#mongodb logs for Sunday the 9th of November, 2014
[00:02:45] <wsmoak> so then… giuseppes where did you telnet from? your desktop, or inside your ssh session?
[00:03:53] <joannac> wsmoak: if you have auth on but no users, anyone can connect on localhost and have full access. Anyone connecting from outside will need to auth except they can't because no users
[00:04:03] <GothAlice> wsmoak: It means that after enabling authentication, if there are no users added yet you will be given the chance to connect and add one, but only from localhost.
[00:06:45] <GothAlice> An important note: make your first user a full admin. If you don't, after adding the user you'll be required to authenticate… and won't be able to do anything. ;)
[00:07:20] <GothAlice> a la https://gist.github.com/amcgregor/c33da0d76350f7018875#file-cluster-sh-L78
[00:07:37] <wsmoak> I will read up before I try anything. happily typing away on localhost behind a firewall atm.
[00:11:05] <wsmoak> I need query help. From this: https://gist.github.com/wsmoak/679c25268d21fcb8a13a How do I get (for each) the *max* date out of that inside array of documents ?
[00:12:48] <GothAlice> wsmoak: db.coll.aggregate({"$group": {"_id": 1, "maximum": {"$max": "$date"}}})
[00:13:29] <GothAlice> wsmoak: db.coll.aggregate([{"$match": {…}}, {"$group": {"_id": 1, "maximum": {"$max": "$date"}}}])
[00:13:41] <GothAlice> You are free to perform whatever query you need to before finally finding the max.
[00:15:45] <GothAlice> "how do I get the *max* date" was your question, no? The problem with the find().foreach approach is that you will be needlessly transferring data from the server to your application, the majority of which being simply thrown away.
[00:16:47] <wsmoak> the question was … How do I get (for each) the *max* date out of that inside array of documents ?
[00:18:37] <GothAlice> wsmoak: You have fixated on one potential solution, the inferior of two solutions given the questions and examples you have provided. Why, exactly, do you need the results back for iteration using foreach? (Again, this will be hideously inefficient. Is this a homework question?)
[00:19:53] <wsmoak> yes that is ruby. no it’s not homework. More like My First Sinatra and mongoDB App.
[00:21:17] <GothAlice> wsmoak: Then I feel it's doubly important to learn to perform tasks in the "correct" (optimum) way instead of potentially learning bad habits. "How do I 'each' across this?" is a generic Ruby question (and one which I can't actually help with, #ruby may be more useful). "Getting the max" is a MongoDB question, and an aggregate query is the Right Way… even if you're doing other aggregation. That's why the aggregate framework exists.
[00:24:20] <GothAlice> db.coll.aggregate([{"$match": {"type": "breeding"}}, {"$project": {"date": 1}}, {"$group": {"_id": 1, "maximum": {"$max": "$date"}}}]) — this is the mongo shell / JS / Python way of doing it, including your initial query and an optimization limiting processing to just the "date" field.
[00:25:07] <wsmoak> I possibly don’t understand aggregate . (the docs I found started talking about map reduce and such.) But I don’t want the max across all the breedings in the database.
[00:25:47] <GothAlice> wsmoak: Then what is your query? I can certainly help formulate it (and the query will be useful in both find() and aggregate() operations; the syntax is the same).
[00:26:11] <wsmoak> I updated the gist so you can see where it gets used. https://gist.github.com/wsmoak/679c25268d21fcb8a13a
[00:28:01] <GothAlice> wsmoak: If you could ensure new values are only ever $push'ed into that array, your query becomes trivial.
[00:29:56] <giuseppes> weird is that the first time it was working but I realized I had typed the wrong password so I interrupted
[00:30:56] <GothAlice> wsmoak: db.coll.find({"type": "breeding"}, {"exposures": {"$slice": -1}}) — you'll have to translate this into Ruby, though. When you .each across these results, the "exposures" array will only contain the last entry appended.
[00:32:14] <GothAlice> wsmoak: I avoid needing funny projections by adding a "latest" date field to my top-level documents, then having the operation that $push'es a sub-document to the array also $set the top-level "latest" date. Then "latest" is always available.
[00:35:12] <wsmoak> well… it’s already stored inside the array, and it could get out of sync, so yeah, duplicate :)
[00:35:23] <xxtjaxx> Hi! I have a live-ticker-esque app where I want to show data to the customer that just came in. This should(tm) work with capped collection and a tailable cursor and sending off new items to the clients upon request. However I also want to store the data in an uncapped collection for further analysis functions. Is there a way to make this possible without inserting a document twice(in the capped and uncapped collection)
[00:36:43] <GothAlice> wsmoak: If you always $set that value when you $push the more complete record, it won't get out of sync.
[00:37:20] <GothAlice> wsmoak: And it's not really a duplicate. Each child record has its own date… but you have a *separate* value (that you want, likely frequently!) that represents the "latest" one. That's a different thing, no duplication.
[00:39:27] <wsmoak> GothAlice: thank you, very valuable. I did multi-value long ago, and then some SQL in between … it’s been fun lettin g the data model for this app emerge (even if I get stuck sometimes).
[00:40:19] <wsmoak> and *so* much easier to change when I realize I’ve left something out or done something entirely wrong!
[00:43:31] <probablyafish> Hello! Had a question about using data from mongodump that's located on an external harddrive. I don't have enough space for all of the data on my local hd – if I specify the --dbpath to the external hd when I run mongorestore and have set --dbpath to the external hd as well in the mongod process should I be good to go? Do I need to be worried about mongorestore copying data to my local harddrive?
[00:47:28] <GothAlice> xxtjaxx: No. You'll have to issue two inserts. I do this in my distributed RPC system, built on capped collections. It's doubly important for my data, since the data describing a "job" may be large, but the notifications about new jobs being added and old jobs completing is basically just {added: ObjectId(…)} or {finished: ObjectId(…), success=True}.
[00:50:07] <GothAlice> probablyafish: You should be good. Note that there are some other files MongoDB might want to create, for example log files, process ID (if running in the background), etc. You may wish to double-check the locations of these files. (If configured to append to the log, it could potentially grow without restriction.)
[00:51:58] <probablyafish> @GothAlice: Thanks! Also for the --filter option for mongorestore am I able to use query operators? eg something like --filter '{"field": { $regex: ... }'
[00:52:35] <nitestryker> i know this is sort of a php question but i can't get an answer from anyone over in the other chan. i was hoping someone could help me out I need to know what the proper way to create a new collection in php
[00:52:59] <GothAlice> nitestryker: Inserting your first record should automatically create the collection.
[00:53:42] <GothAlice> probablyafish: That's a good question. I'm not sure. The documentation doesn't state if that's a full query descriptor or just a simple == matching.
[00:56:56] <GothAlice> That first chunk of code isn't a thing. createCollection creates a collection, it doesn't insert data.
[00:59:00] <GothAlice> I always recommend using descriptive names, where possible. If this is a collection of data about users, call the collection "users".
[00:59:32] <GothAlice> (And never again be confused.) Note that your first example (using a dynamic collection name) while useful in some circumstances, is not useful here.
[01:01:34] <nitestryker> after you select the db the variable that uses to seect the database will go to the next line
[01:03:46] <GothAlice> (The reason I said no is that you either copied and pasted that from elsewhere, or typed it manually into IRC; it isn't your actual code. Pasting lines that aren't actually in your code leads to confusion.)
[01:06:49] <GothAlice> (In the grand scheme, too, you don't want to createCollection all the time, it's pointless since inserting a record will create the collection anyway. It *may* be worthwhile to check if it exists, and if it doesn't create it and add the indexes. But I'm seeing here the MongoDB class doesn't provide a function to check if a collection exists, other than getCollectionNames.)
[01:07:57] <GothAlice> Don't suppose there's any way to convince you to use anything in the world other than PHP, eh?
[01:09:41] <GothAlice> https://gist.github.com/amcgregor/18e3180215736369b49e is an example of Python declaring a DB signal handler that updates timestamps automatically and a sample document schema that uses it. (MongoEngine package on top of Python).
[01:11:34] <GothAlice> http://stackoverflow.com/questions/1966010/what-does-this-mean-parse-error-syntax-error-unexpected-t-paamayim-nekudotay
[01:14:37] <GothAlice> LouisT: You remember register_globals? (I.e. all GET/POST/SESSION vars get injected into the global scope) It's still there in the handy little http://php.net/parse_str function. It takes an optional second argument to populate an array, cool. Still populates the current scope even if you supply it. They recommend passing it $_SERVER["QUERY_STRING"] — register globals. And, my favourite little PHP oddity, the second variable you pass
[01:42:33] <wsmoak> giuseppes: some options on http://docs.mongodb.org/ecosystem/tools/administration-interfaces/
[01:45:00] <wsmoak> not yet. it’s just me on localhost so far, not complicated enough to need pictures.
[01:46:35] <giuseppes> wsmoak, geez! you don't like gui's, I've been using one from the first day, I need some graphic feedback
[01:47:26] <wsmoak> well… I’m developing an API and basic CRUD webapp, so I get to see the data in that
[01:48:36] <GothAlice> (We use MMS, but for monitoring notifications and backups, not general management.)
[01:49:56] <GothAlice> In my case the ipython shell. (A much improved Python REPL with full syntax highlighting and pretty-printing by default.)
[01:50:41] <GothAlice> bpython is another, lighter-weight alternative that does the ncurses thing to provide a text-mode GUI, and for interactive debugging I use pudb (same ncurses UI).
[01:51:18] <GothAlice> Indeed; when you use a data access layer, it's a good idea to use it for administration, as well.
[01:51:44] <GothAlice> (A lot of my collections have events that are only triggered Python-side, like automatic updating of modification timestamps, that using another tool wouldn't replicate.)
[01:54:13] <wsmoak> what about naming… I (of course) have “mydb” right now. And I have a collection for this app… do I call the collection “nivens” (the app’s name) or “rabbits” (what’s in the collection) ?
[01:55:13] <GothAlice> (It's unusual to only need one collection for an entire app. While possible, mixing multiple types of documents into one collection can be very confusing to develop/debug.
[01:55:50] <wsmoak> oh, I saw examples of documents that used {“type” => “type-of-thing”, … } and went with that
[02:02:23] <GothAlice> wsmoak: Most often having a "type" value is used when you want to have "inheritance" (the equivalent of SQL's concrete single-table inheritance, just way more efficient).
[02:10:33] <GothAlice> Using "type" in your own application to determine record type. Read the tree from bottom to top as: "Bowler is a type of cricketer is a type of player."
[02:13:33] <wsmoak> yeah… I was using it for entirely separate (but related) kinds of things, like players, balls, [some other sports-related thing]
[02:29:40] <wsmoak> GothAlice: thanks for all your help. those changes to the db made the code much cleaner.
[07:54:08] <diegoaguilar> Hello, what's the most simple strategy to have temporal based document properties?
[09:45:08] <dorongutman> I have a document with several dates in it - createdAt, updatedAt, startDate, endDate, approvedDate
[09:45:51] <dorongutman> I will need to create an index on a few of them - mainly startDate and endDate
[09:47:23] <dorongutman> since they are all dates, I thought about grouping them in a sub-json, like {_id: “abc”, “dates”: {startDate: …, endDate:…, createdAt:…, updatedAt: …, approvedDate: …}}
[09:47:53] <dorongutman> so from an application point of view, it’ll be a bit easier to manage them in my mapper
[13:37:30] <giuseppesolinas> now I'm having an issue tho, It looks like I'm not able to save a timestamp in mongo
[13:38:07] <wsmoak> giuseppesolinas: pffft. keep notes on a gist while you’re working. clean up formatting. done.
[13:40:42] <giuseppesolinas> wsmoak, node.js, I create an object with all my post's data then save the post with that as an argument, I usually add info on conditions, but if I create an object with timestamp as an attribute it doesn't get saved to the db
[13:41:40] <giuseppesolinas> If I take a look at the object before it gets saved I can see my timestamp, but looking at the object in the db all attributes are there except the timestamp
[13:44:05] <wsmoak> heh, and I’m trying to get to node.js next. just using sinatra to develop the data model and api because it’s simple.
[13:51:07] <wsmoak> is that supposed to be a utc timestamp? Because Time.at(1415540681506) -> 46826-09-12 12:51:46 -0400
[13:52:19] <giuseppesolinas> wsmoak, maybe I have to update the model first, I just noticed but I actually thought that it was all more flexible :)
[13:53:15] <wsmoak> I’m not sure why it’s not getting inserted at all… unless there is some validation / constraints preventing it ?
[13:56:28] <wsmoak> BSON Date is a 64-bit integer that represents the number of milliseconds since the Unix epoch (Jan 1, 1970).
[13:57:26] <wsmoak> (you actually said “timestamp” earlier though … apparently that’s an _internal_ thing for mongodb)
[13:58:56] <giuseppesolinas> wsmoak, I could as well save it as a string, it should get written anyway
[14:00:26] <giuseppesolinas> wsmoak, also something prevents me from using "createdAt" as an attribute name
[16:16:22] <the_drow> Is there a way to specify multiple db paths if I use the directory per database configuration?
[16:16:39] <the_drow> Someone on the mailing list suggested symlinking but I don't think that's a good idea
[20:13:16] <probablyafish> Hello! Was wondering what the easiest way to convert output from mongodump to valid JSON.. my mongodump .bson is ~95GB. I used bsondump but the file still has a bunch of ObjectId("...") and Date("...") things so I can't parse it. Then I thought I'd just use mongorestore and restore from the .bson file but it's building the index and it's been > 24 hrs. I want to extract some data from the .bson file
[21:00:26] <giuseppesolinas> if your schema has properties a, b, c, call save object with properties a, b
[22:12:02] <ningu> I'm trying a bulk-load some data into mongodb using cayley (a graph database that can use mongodb as a store) but after loading the first ~44GB or so in I get this error in the mongodb logs: SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:41297]
[22:13:30] <ningu> the database is all stored in 20 or so files in /var/lib/mongodb so I don't know about that, for number of open files... but could be something else?
[23:29:09] <bunbury> cheeser : want to read emails from a service mail address and parse to json then store in mongo
[23:30:34] <bunbury> cheese: it makes sense to do that though right ? goolge business bandwidth limit would be exceeded each day doing imap queries
[23:31:25] <bunbury> cheeser: it makes sense to do that though right ? goolge business bandwidth limit would be exceeded each day doing imap queries
[23:36:09] <bunbury> found this https://developers.google.com/resources/api-libraries/documentation/gmail/v1/java/latest/com/google/api/services/gmail/model/MessagePart.html
[23:46:17] <bunbury> be able to find messages that bounced cuz ppl email dont exist. message delayed