Log file Viewer
#mongodb logs for Friday the 13th of March, 2015
[00:01:39] <quan2m> I've taken the horrible step of disabling SELinux, and the file permissions are correct.
[00:44:42] <GothAlice> tsyd: Also, depends on if you consider the data in ObjectIds sensitive or not, in case they are shown to a user (i.e. in a URL).
[00:47:27] <Jonno_FTW> I'm using $where:"this.datetime.getDay()==4", but it just returns all records, not those on 4th day of week
[01:15:01] <mdavid613> hey all, question for you. I'm trying to setup a new replicaSet with x509 user / member authentication. I have a primary setup with a replicaSet enabled, x509 and users setup etc…when I try to rs.add() the other host I'm getting back "exception: need most members up to reconfigure, not ok" from the Primary. From the new secondary I'm seeing this in the logs: Unauthorized not authorized on admin to execute command { replSetHeartbeat:
[01:15:26] <mdavid613> does anyone have any ideas what I may be doing wrong? Also, the new secondary member has a blanked data directory
[01:19:38] <mdavid613> and the x509 for cluster members documentation doesn't mention enterprise only support
[01:24:49] <joannac> Jonno_FTW: http://www.w3resource.com/javascript/object-property-method/date-getDay.php
[01:24:55] <Jonno_FTW> joannac: I think I found the problem, getDay() is the UTC date, but my dates are time zone adjusted
[01:25:22] <joannac> Jonno_FTW: http://www.w3resource.com/javascript/object-property-method/date-getDate.php
[01:31:57] <Jonno_FTW> how come ISODate("2013-06-06T03:05:00.000Z").getDay() == ISODate("2013-06-05T14:30:00.000Z").getDay(), I don't follow
[01:39:02] <joannac> Jonno_FTW: if you're using something other than the mongo shell, I can't comment on what they do with ISODates
[01:58:27] <chanced> would you use mongo for a calendar schema or is there an alternative db with features that could make the process of handling recurring events easier?
[02:56:19] <GrubLord> Quick question about replica sets: sure, the servers vote among themselves as to who’s primary, but how does my app know which one to send the request to?
[02:56:48] <GrubLord> Do I query one of them for the primary, somehow, and then set the request to that one?
[02:56:56] <joannac> create a replica set connection, the driver will poll the replica set and figure out which one is primary
[03:05:14] <ladessa> I;m trying to do a OR query...but it doesn't work..its returning nil..but in RoboMongo the query works...the query is http://mysite:3000/obterOcorrencias?query={"$or": [ { "_id":ObjectId("54ff5ed8d094b1e371fba0a7")} ]} https://www.irccloud.com/pastebin/0UwB1sLy
[03:09:54] <joannac> ladessa: print the query before you send it to mongodb. make sure it looks how you expect
[03:11:05] <ladessa> {"$or": [ { "_id":ObjectId("54ff5ed8d094b1e371fba0a7")}, {"_id":ObjectId("54ffcc00bef7ea3b78d11789")} ]}
[03:12:55] <ladessa> https://usercontent.irccloud-cdn.com/file/29onumzm/Captura+de+Tela+2015-03-13+a%CC%80s+00.11.59.png
[03:40:49] <ladessa> It works..but only 'findOne' 'find' not works https://www.irccloud.com/pastebin/NmFKDUyV
[04:00:02] <ladessa> https://www.irccloud.com/pastebin/IahYSiG5/It+works+great%21+the+only+problem+now+is+use+query+from+the+url+param
[04:33:27] <joannac> so did you do check the mongod is started and there are no firewalls in the way?
[04:36:12] <joannac> dbclk: before you do that, can you log into the primary, and check whether the mongod is running?
[04:44:59] <ladessa> {"$or": [ { "_id":ObjectId("54ff5ed8d094b1e371fba0a7")}, {"_id":ObjectId("54ffcc00bef7ea3b78d11789")} ]} -? Why am I getting SyntaxError: Unexpected token O in ObjectId?
[04:45:24] <joannac> dbclk: hard to say where the problem is other than "somewhere beetween the 2 servers"
[05:23:55] <dbclk> hey joannac is it possible from the secondary I run the mongo command to connect to remote on port?
[10:30:11] <sterichards> Well JSON supports array data, CSV has to be hacked to support it. Can mongo support any kind of hacked CSV structure that’s “supporting” arraays
[11:24:43] <boutell> Morning! How do I ensure read-after-write consistency when my code might be running on a replica set? That is, how do I ensure that I can always read what I’ve just written? And is it possible to set the “default write concern” so that I don’t have to patch every insert() call? I’m using the node driver. Thanks.
[11:26:46] <KekSi> 'morning, is there an alternative to running mongo --eval "" on the commandline now that eval is deprecated?
[11:35:35] <MatheusOl> KekSi: Notice that is advisable to keep the connection where you issued fsyncLock from opened and do fsyncUnlock from the same one, as fsyncLock can in some circumstances block new connections
[12:31:45] <amitprakash> Hi, not as much a mongo question as an application using mongo question, I wish to be able to attach future timestamps to documents which when expire(datetime.now() == future_ts) trigger some sort of callback/service. These future timestamps can change before expiry setting them to a closer/farther point in the future
[12:44:22] <GothAlice> amitprakash: If the callback is for cleanup, a TTL index will do what you want, and lets MongoDB handle the periodic clean-up.
[12:45:13] <GothAlice> I typically have an "expires" diatomite, with a TTL of zero. Any value of None is set to not expire. :)
[12:49:23] <amitprakash> GothAlice, cheeser, I see a small problem with the polling approach, a particular document can have an event expire and then be updated to a future ts between two consecutive polls
[12:50:31] <GothAlice> A slightly more efficient method would be to have your thread with a wake-up queue, and a capped collection to notify when new dates are inserted.
[12:51:24] <amitprakash> GothAlice, that should work except when we are doing a couple of million notifications/events ?
[12:52:08] <GothAlice> Thread A "tails" the capped collection. When you update a future_ts time, push the ID of the record and the new time to the capped collection. Thread A wakes up, sees the new time, and pushes it onto the queue. The "worker" thread wakes up, reads the new expected timestamp and ID, then waits on the waker queue for the time between now and the future time. After it executes a task, it queries for the next task, time-wise, then waits that
[12:52:56] <boutell> How do I ensure read-after-write consistency when my code might be running on a replica set? That is, how do I ensure that I can always read what I’ve just written? And is it possible to set the “default write concern” so that I don’t have to patch every insert() call? I’m using the node driver. Thanks.
[12:54:03] <GothAlice> boutell: Saw your question from earlier, but you left. ;) Two ways: since writes need to go to the primary anyway, for queries where you must have immediate consistency, issue your query on the primary. Replication lag can't bite you, then. Or, if you are willing to accept slower writes, set a write concern of N (for the number of secondaries you have), then even reads from secondaries will be consistent.
[12:55:19] <GothAlice> boutell: And yes, you can set a default write concern (and read concern) when connecting your client app. I.e. the "w" "j" and "async" arguments to MongoClient: http://api.mongodb.org/python/current/api/pymongo/mongo_client.html
[12:57:23] <boutell> GothAlice: interesting. I’m not sure how solution one helps me, in that my read might hit a secondary which hasn’t replicated what I just wrote to the primary yet, right? I see how solution two works.
[12:57:36] <GothAlice> amitprakash: https://gist.github.com/amcgregor/4207375 < this "actual work collection" and "notification queue" method is what I use for my distributed RPC, including scheduled (not just immediate) task execution. A single host with two producers, four consumers benchmarked at 1.9 million RPC calls per second. (So… should be pretty efficient.) The capped collection size in your case could be quite small: only times are being pushed there.
[12:58:00] <GothAlice> boutell: I thought I was clear, if you explicitly read from the primary, and _not_ a secondary. That's solution one.
[12:58:32] <amitprakash> the idea with r/o replicas is to ensure read heavy apps do not affect writes
[12:59:11] <amitprakash> however for mission critical reads you should either stick to primary or have to take a hit on perf via w=N given theres latency b/w replication
[12:59:26] <oznt> hi everyone, I have a db ca 300GB big, I added an empty replica member (I did not preseed it). It started recovery, and the amount of files was growing for about 18h. I saw in the logs it was building indecies, but after a while (I have had already 114Gb copied) all the data was gone from the replica, now I have again ca. 22 Gb, and I don't know what to look for in the logs
[12:59:53] <boutell> So if I’m writing an open source npm module that needs read-after-write behavior, how would I accomplish that without knowing the number of secondaries going in (since npm modules are popped into other people’s projects)? I’m getting complaints that, for instance, the connect-mongo module’s sessions don’t appear to work, presumably due to the replication issue.
[13:00:13] <GothAlice> oznt: Sounds like your oplog isn't large enough to handle all of the activity that happens while a replica is seeding itself.
[13:00:22] <amitprakash> boutell, either read from primary or require user to specify N in some setting/config
[13:01:02] <oznt> GothAlice, yes, my Oplog is small, (only 7GB) But I am afraid to change it without having another replica member or at least a backup...)
[13:01:24] <GothAlice> oznt: You'll either need to increase the oplog size to accommodate possible activity during seeding (i.e. it needs to store everything up while the new secondary populates itself, so that the new secondary can catch up fully) or pre-seed.
[13:02:00] <boutell> interesting. So I can’t determine n at start by querying the server or something? I guess another replica could be added at any time, though. Of course that’s also a problem when you preconfigure n.
[13:06:55] <GothAlice> boutell: http://api.mongodb.org/python/current/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient.nodes (for the Python version of this, should be similar on other languages)
[13:09:13] <oznt> Always use filesystem snapshots to create a copy of a member of the existing replica set. Do not use mongodump and mongorestore to seed a new replica set member." found this here: http://docs.mongodb.org/manual/tutorial/expand-replica-set/
[13:09:45] <GothAlice> oznt: Ah, I was referring to simply having a backup of your data. A task for which mongodump excels.
[13:10:47] <GothAlice> boutell: Testing that out in the Python version, upon first connection (seeding the server list with the primary only) and prior to any secondary-capable queries, my .nodes only include the seed node. Hmm. Further down the pipe, you could also ask the server itself using a command.
[13:13:37] <GothAlice> oznt: At the moment, likely due to a lack of actionable monitoring catching that your oplog is no longer large enough for the amount of data you have, you have no choice but to either increase the oplog size, or pre-seed a node _then_ increase the oplog size to ensure a node going down and needing to be re-seeded won't simply die.
[13:15:24] <dbclk> folks i'm trying to connect to mongo on a remote server using an IP on port 27017
[13:16:02] <boutell> dbclk: in that, if no other security policy is in place, it would be dangerous not to restrict that.
[13:16:38] <GothAlice> dbclk: a) It's not actually running. b) mongod is only bound to localhost, not 0.0.0.0 (all interfaces). c) You have firewall rules preventing access. d) True local access often happens over an on-disk socket, not a network TCP socket, so you may even have TCP disabled.
[13:17:27] <GothAlice> dbclk: "Firewall disabled" often blocks everything, for security sake. SELinux… causes more problems than it solves, for most inexperienced users.
[13:17:47] <GothAlice> (Well, blocks everything except SSH access from the service net on my machines.)
[13:20:20] <GothAlice> dbclk: From within the host running mongod, run and pastebin/gist the result: sudo lost | grep mongod | grep TCP | grep LISTEN
[13:21:12] <GothAlice> This will give you the authoritative answer to "is mongod listening to a TCP port?"
[13:22:04] <GothAlice> sudo lsof — kali: Your IRC client doesn't auto-apply s/search/replace/ corrective regexen automatically? ;)
[13:23:01] <GothAlice> You can also get this information from /proc, but it's a bit more of a PITA to do it that way. ;)
[13:23:57] <GothAlice> I get, on my development machine: mongod 535 amcgregor 6u IPv4 … TCP localhost:27017 (LISTEN) — bound to localhost only, running as me, on the default port.
[13:25:53] <GothAlice> <dbclk> folks i'm trying to connect to mongo on a remote server using an IP on port 27017 — false. You're listening on 28017.
[13:28:23] <GothAlice> dbclk: So your problems don't relate to mongod specifically. It looks like it's doing the right thing.
[13:29:02] <GothAlice> dbclk: I misinterpreted your paste as one long, wrapped line, instead of two. Your monogd is bound correctly to 27017 for client connections, but it's also expecting to be in a replica set, thus 28017 also being bound.
[13:30:49] <GothAlice> ('Cause MongoDB won't actually accept connections until it says as much in the log after initial startup and replication init.) Your mongod process is bound to all IPs.
[13:31:17] <dbclk> 10.1.1.x and 10.1.6.x, I can connect to the server over 10.1.1.x (which is an extremely busy network) but, I can't connect over 10.1.6.x (which is a quiet network)
[13:32:48] <GothAlice> dbclk: Could you paste/gist the output of: sudo iptables -L # (-L = list) On a machine with "no firewall enabled" the INPUT, FORWARD, and OUTPUT "chains" should all be empty.
[13:36:20] <GothAlice> However, it really sounds like you've got a general Linux sysops issue at the moment. mongod is behaving correctly. The IRC channel for your distribution (which I'm not familiar with) may have more helpful advice than the simple diagnostics I'm stepping you through. Or, more generally, #linux. Is the network interface you want to use (the "quiet" one) configured? (ifconfig to check)
[13:37:48] <GothAlice> If ifconfig reports the expected IP address on that interface, next is to check the routing tables to make sure there's a valid route for that interface. (The route command for this.)
[13:38:43] <GothAlice> (replacing ethN with the correct eth1, eth2, etc. network device you want to test, and IP with the IP of another node on that "quiet" network.)
[13:40:54] <GothAlice> If you try to "tail" a capped collection that contains no records, the query will immediately fail instead of waiting for new data to be added. So you have to "prime" the capped collection with a bogus/no-operation record to allow the tailing queries to properly wait.
[13:43:34] <GothAlice> amitprakash: From the full implementation (link at the bottom of the slides gist) https://gist.github.com/amcgregor/52a684854fb77b6e7395#file-worker-py-L79-L81 automatically handles injection of a "no-op" record into any empty capped collection that gets tailed.
[13:46:03] <GothAlice> amitprakash: I'm working on generalizing that code and producing a proper package for it: https://github.com/marrow/task
[13:47:11] <GothAlice> Two of the goals that go beyond the original gist: a) allow remote execution of Python generator functions, streaming the generated data to any listeners, and b) a Futures-compatible API, so it can be a drop-in replacement for multi-processing and multi-threading concurrency approaches.
[14:20:59] <dbclk> question folks..has anyone find the issue with a mongo server that has 2 nics/ 2 IPS (different subnet network) though mongo is configued to listen on all port, you're only able to connect to one?
[14:59:17] <GothAlice> StephenLynx: Those questions are ancillary to him not being able to generally connect. (See the IRC log from earlier today for the sysop diagnostic steps we went through to ensure that mongod is, in fact, behaving correctly in his environment.)
[15:00:18] <StephenLynx> so he is INDEED using every single port for mongo? nothing else is able to listen after mongo is running?
[15:00:49] <GothAlice> dbclk: I run my DB VMs with four interfaces: loopback, internet-facing (MongoDB doesn't listen here), ServiceNet (nor here), and DataNet. ServiceNet bridges all VMs and lets me access SSH and whatnot without exposing SSH to the internet. DataNet is a custom VLAN interface which is common to only my own VMs, which MongoDB communication happens over.
[15:01:27] <GothAlice> StephenLynx: Pedantry doesn't help solve problems. It's listening on all interfaces, again, feel free to check the log to see the other things we examined, including firewall, etc.
[15:01:44] <StephenLynx> i am not being pedant, I am really curious because I have never heard about this kind of thing.
[15:03:21] <GothAlice> (Esp. considering half of them are allocated, by default, to outbound connections, i.e. ports 32767+. :)
[15:07:46] <GothAlice> diegoaguilar: Yeah. It's a performance thing. The kernel is configured to allocate larger-than-normal "pages" of memory to applications. This can lead to virtual memory map fragmentation in MongoDB, I believe.
[15:07:54] <PedroDiogo> i'm new to mongo and I want to test the performance of it when using a chat-kind of app
[15:08:13] <GothAlice> (And active degragmenting, the second option it's warning about, is also a performance thing. You don't want chunks of memory being moved around if you can avoid it.)
[15:09:07] <PedroDiogo> as heroku does not offer local mongodb support, I want to test the performance of other database as a service
[15:10:53] <GothAlice> diegoaguilar: Not really, at the moment. Though certainly asking in #linux or a channel dedicated to your distribution may be useful.
[15:11:09] <PedroDiogo> that is why I first wanted to go with a VPS; however, I'm a bit concerned about backups and scaling... I'm guessing that is something that mongolab or any other service would be helpful. Am i right ?
[15:11:27] <diegoaguilar> a last question, even in order to assist PedroDiogo , is wired tiger enabled by default in mongodb3?
[15:12:26] <GothAlice> (My own benchmarks see ~15% improvement in typical query performance, and a healthy 50% or so reduction in disk usage, but I managed to make a VM kernel panic under load, so…)
[15:14:42] <GothAlice> PedroDiogo: I run a 27 TiB cluster on three physical boxen at home and due to the size of the data utilize filesystem snapshotting trickery to do offsite backups. At work, our cluster is more… reasonably sized. There, we use hidden replicas to stream a backup from production to the office, one of them is delayed by 24h to allow us to undo user mistakes, and we also make regular mongodump snapshots.
[15:15:51] <GothAlice> PedroDiogo: As a note, we've never had a node actually _fail_ in four years, and until a recent security patch that required a reboot, my boxen had 3-year uptimes. ¬_¬
[15:16:27] <GothAlice> Derick: Heck no. Our reliability went through the roof once we migrated away from AWS.
[15:18:03] <GothAlice> So accounting for the tiny amount of time that reboot needed, we've got something like five nines averaged across three years.
[15:18:29] <PedroDiogo> GothAlice: can you share with me some tips to maintain a good DB working, IF i go with a VPS ?
[15:21:36] <GothAlice> PedroDiogo: a) Find a reliable hosting provider. Cheap != good in this regard, esp. as many VPSes over-provision. (I.e. they promise and sell more capacity than they have, hoping everyone uses a fraction of what they're given.) b) Automate, automate, automate. MMS is a good start, but to be truly reliable you need to be able to kill -9 a random DB VM and still have your application survive.
[15:22:46] <GothAlice> (This means potentially automating the spinning up of new VMs, and _automatically_ having them join the existing cluster on startup in a way that won't risk "run on the bank" situations, which can cause cascade failures.)
[15:23:41] <GothAlice> diegoaguilar: No, I actually recommend staying away from them. (I have several multi-year outstanding tickets with public histories with repeated "We're working on it, it'll be done in three months" every six months.)
[15:25:26] <GothAlice> I recently evaluated clever-cloud.com, and they're working with 10gen/Mongodb.com to provide a MongoDB offering, currently in beta. Their application host pricing is _brilliantly_ competitive, and their auto-scaling infrastructure is pretty awesome.
[15:28:43] <GothAlice> I also highly approve of their git-based deployment workflow. (We use this style at work, though we don't use Clever Cloud. Even our VMs themselves have their root filesystem git-managed…)
[15:30:05] <GothAlice> … the site doesn't inspire confidence. Looks like a cheap whitelabeling of AWS.
[15:30:56] <PedroDiogo> its a new Portuguese solution...everybody is talking about them here in Portugal, but I haven't seen any good benchmark
[15:31:33] <PedroDiogo> clever-cloud looks pretty good - its a pitty they do not provide mongodb support yet
[15:32:18] <GothAlice> PedroDiogo: https://www.clever-cloud.com/doc/addons/clever-cloud-addons/#mongodb
[15:33:33] <GothAlice> I asked them for their initial (and subject to change before release) planned pricing for MongoDB: http://cl.ly/image/3H3I1M0U0w0y
[15:33:57] <GothAlice> Notably, if I tried to host my 27 TiB home dataset with compose.io, it'd cost me half a million USD per month.
[15:34:22] <StephenLynx> plus I suggest taking a look into io.js if you haven't heard about it yet. it's node.js but better and faster.
[15:34:42] <GothAlice> (We use Rackspace, but we need SLAs and things like that and deal with big companies that like the name. Linode is awesome people, too.)
[15:34:50] <PedroDiogo> I'm ok with things like mongolab or compose, but I'm a bit concerned with their performance because of the IO lag...BUT I do like to think I can rely on these type of services to scale and backup my DB as it goes...
[15:36:03] <GothAlice> diegoaguilar: It's my personal dataset, hosted three Dell 1U rack-mounts and three Drobo 8-something-i rack-mount iSCSI arrays. It's been growing since 2001, so it's hard for me to estimate the "total cost". (But you could price it out: grab three two-year old servers and three arrays, full of 4TiB WD Green Power drives.)
[15:37:37] <GothAlice> diegoaguilar: It's a transparent proxy and ARP MITM that records every bit of digital information I touch across any of my devices or touches my network.
[15:38:43] <GothAlice> (Stored as a GridFS metadata filesystem, primarily organized using bare tags and key/value tags, with synonym association, natural language processing so I don't have to manually tag everything, and some neural network bits.)
[15:39:22] <GothAlice> diegoaguilar: I plan on uploading, so I thought I'd give myself a head start. Also, experiments. This dataset includes a complete copy of Wikipedia, and 350,000 books, as an example. It's a fun playground for AI research.
[15:39:55] <GothAlice> I could. Can't release it though. I've found a few hundred patents I'm in violation of. ;)
[15:40:29] <StephenLynx> you could make an anonymous release on some god-forsaken server on a god forsaken country
[15:43:32] <GothAlice> Though I do try to open-source what bits I can get away with. For example, years before MongoDB had full-text indexing, I added it: https://github.com/marrow/contentment/blob/develop/web/extras/contentment/components/search/model.py#L72-L112 (This is a parallel Okapi BM-25 ranking algorithm, the same as Sphinx, Lucene, or Yahoo, with boolean pre-filtering that avoids the MS patent.)
[15:45:06] <GothAlice> diegoaguilar: Though I do rely on my own packages: https://github.com/marrow/ See: https://youtu.be/M9V1e-rG7VA?t=11m26s "Easy AI with Python", specifically, the "neural networks for data mining" part—this is what got me started with AI in Python.
[15:47:17] <GothAlice> At work we use Jython to have our Python code orchestrate Hadoop, Mahout, and Weka.
[15:48:31] <diegoaguilar> on music and alcohol, are they related, when any general topic on music is popular, is activity on music increasing?
[15:49:29] <diegoaguilar> so last question to be answered should be "would a rock style advertising work with beers, whiskey or anything else in somewhere"
[15:50:57] <GothAlice> We find MongoDB to be the best at streaming live analytics. (Using pre-aggregation.) For our NLP/ranking/AI-type-stuff we pop into Java and more dedicated tools.
[15:51:35] <GothAlice> In a similar way, I wouldn't attempt to model a graph in MongoDB, there are better tools for the job. (And mongo-connector to keep the datasets in sync.)
[15:52:12] <GothAlice> (Though we haven't been using that, since our Java code has direct execution access to our Python code, so it's all one system.)
[15:53:11] <GothAlice> Jython's performance is "acceptable". Technically it's just compiling Python code for the JVM, with some glue in there to work around the differences.
[15:53:29] <GothAlice> JVM does threading better, but dynamic types on the Python-side slow things down a bit.
[15:54:54] <GothAlice> diegoaguilar: Illico Hodes, an HR consulting firm and job distribution platform provider. :)
[15:56:56] <GothAlice> French is mostly useful, in a world-wide scale, for international treaties and international contracts.
[15:57:24] <GothAlice> Even in Canada, which is hypothetically bilingual, French is mostly relegated to the province of Québec.
[15:58:07] <GothAlice> Parts of Africa (along with Dutch and various other languages held over from colonization), etc.
[15:58:26] <GothAlice> Mostly Québec. Government workers anywhere in Canada need to be fluent in both, though.
[15:59:24] <GothAlice> diegoaguilar: That's mostly a USA thing. Canada is very… diverse. Vancouver is largely chinese, Richmond is largely middle-eastern, etc., etc. (Examples from BC, where I grew up.)
[16:04:14] <GothAlice> (http://lojban.github.io/cll/ I find to be the easiest way to read the reference grammer.)
[16:07:11] <GothAlice> diegoaguilar: My highschool had a two-year elective for it. Hallier-sensei was awesome.
[16:07:45] <GothAlice> Japanese is my second favourite language, after Lojban. (For many of the same reasons I like Lojban, actually. It's a pretty grammatically regular language.)
[16:08:29] <GothAlice> Well, let's just say Lojban's grammar is completely unambiguous and wholly defined by a FLEX/YACC grammar, so it's 100% machine parseable, too.
[16:09:01] <GothAlice> For examlpe: http://www.lojban.org/camxes/?text=le%20mi%20varkiclaflo%27i%20cu%20culno%20lo%20angila (my hovercraft is full of eels)
[16:12:42] <GothAlice> My uni experience was psychology, with my "big project" being on the Sapir-Whorf Hypothesis (how the language you are taught might determine the range of thoughts you are capable of experiencing). Lojban is a direct way to test the hypothesis.
[16:14:03] <GothAlice> Psychology. The professor teaching the CS courses was a former employee of mine… so yeah.
[16:14:38] <GothAlice> As one really neat example: "that's a pretty little girls school". (apostophe on "girls" omitted intentionally, pretend it's not needed in this instance) — what exactly did I mean by that? Is it a school for pretty little girls? Is it a pretty school, for little girls? Is it a little school for pretty girls? Etc.
[16:15:17] <GothAlice> (In Lojban, there is no ambiguous way to represent this statement. It's _always_ one of the specific forms.)
[16:18:41] <GothAlice> (I focused on the biology of psychology as an interest in the eventual goal of uploading.)
[16:21:17] <GothAlice> These days I apply my knowledge of the nervous system to software systems. (I'm going the top-down functional / block diagram approach to creating an ACI instead of the physical simulation bottom-up approach.)
[16:22:17] <GothAlice> Stackless Python (which Pypy supports a subset of) is awesome for modelling processing pipelines in a similar way to the functional data processing in our brain. Parallel everything FTW.
[16:24:07] <GothAlice> So, beyond the fact that I knew I knew more than the IT professor, I didn't really see the point of the expenditure just to get a bit of paper. ;)
[16:53:14] <Petazz> Let's say I have a blog and I want to know as a time series how many distinct users have made a comment as a total sum
[16:54:20] <GothAlice> Petazz: For my forums that's done using a simple aggregate query. I range select the modification time of the threads, unwind on replies, project just the author field, then group on that author field while $sum 1'ing.
[16:59:37] <GothAlice> The number of replies, number of upvotes, etc. are stored against each reply, but at the Thread level there's a pre-aggregated version of all of those stats so I can more efficiently get per-thread stats grouped in various ways without needing to reprocess (unwind) every single reply each time.
[17:00:17] <lmofi> Using MongoDB now i get data of the right row but how do i delete these rows channelTB.messages.forEach(function(v){ console.log(v) }); v output is like {data: data} {data: data} {data: data}
[17:13:31] <blizzow> I installed a copy of 3.0 and added a root user. I also commented out bind_ip=127.0.0.1 in my mongod.conf and restarted mongo services. I'm still only able to authenticate from the host that mongo is running on. Is there something else I have to do to allow remote authentication?
[17:27:16] <Petazz> GothAlice: In your way of unwinding the replies, you miss out the timestamp though?
[17:27:38] <blizzow> GothAlice: I tried giving bind_ip=0.0.0.0 and I'm still unable to authenticate from remote hosts.
[17:28:39] <GothAlice> blizzow: Are you at least able to connect to the host remotely? Telnet to the MongoDB port on it? (27017 I believe.) Ping it?
[17:30:28] <GothAlice> blizzow: See http://docs.mongodb.org/v3.0/release-notes/3.0-compatibility/ and the sections on authentication/security changes.
[17:31:53] <lmofi> Using MongoDB now i get data of the right row but how do i delete these rows channelTB.messages.forEach(function(v){ console.log(v) }); v output is like {data: data} {data: data} {data: data}
[17:43:16] <srajbr> Derick thanks. I was confused as _id Object generation follow some rule dependent on host
[17:46:57] <GothAlice> srajbr: Yes, but that data is collected and generated in the client application, not the server.
[17:47:16] <GothAlice> machine = app server, PID = application process ID, increment = record generated in that process, starting at a random value.
[17:50:02] <GothAlice> Task workers, if you use distributed tasks, would be another example of a mongodb client. (They're not webapps, but they're still clients.)
[17:56:02] <lmofi> Using MongoDB now i get data of the right row but how do i delete these rows channelTB.messages.forEach(function(v){ console.log(v) }); v output is like {data: data} {data: data} {data: data}
[18:04:46] <daidoji> giving error exception:BSON representation of supplied JSON is too large: code FailedToParse perhaps?
[18:05:48] <Boomtime> daidoji: are you saying you get that error when you try to mongoimport a specific doc?
[18:11:10] <daidoji> Boomtime: no, I'm saying I mongoexport a collection and then mongoimport and it throws that error
[18:19:56] <GothAlice> Potentially, shorts are being converted to longs or floats, since all numbers in JSON are floats. This may be increasing the storage requirements of one or more records beyond the 16MB limit. This is, however, unlikely.
[18:23:14] <GothAlice> Arrays of ints, or arrays of arrays of ints, would be the only way I'd think would be feasible to reach the 16MB limit, though. 16MB is a _lot_. Binary data?
[18:25:11] <daidoji> this goes back to my questions the other day about transferring data from my data-warehouse mongo instance into a production mongoinstance
[18:25:53] <GothAlice> That's strange, considering mongodump/restore won't need to deserialize data back into BSON first.
[18:26:10] <daidoji> so its essentially mongoexport -d db -c collection | bzip2 -9 col.bz2; on one end
[18:26:21] <GothAlice> (And can bundle things like oplogs, and index data, further reducing downtime.)
[18:28:29] <daidoji> hmmm, wait a second. maybe its not a mongoimport issue... The second time I run it here it looks like I run into a bzip2 error...
[18:28:55] <GothAlice> You're upserting. You may be upserting too much (too many appends, for example) into a single record, and need to change how you're grouping those.
[18:30:22] <GothAlice> I.e. if you're grouping your records into five minute time slices (I'm doing analytics at work, so… this is the example that comes to me) but have too many events in a single five-minute period, the record may grow beyond the 16MB limit.
[18:31:05] <GothAlice> To "re-group" this data, I'd change it to two minute time intervals, for example, to reduce the individual number of events per time period (at the expense of 2.5x more records to read when performing aggregate queries.)
[18:32:43] <daidoji> hmmm, these are idempotent records and I think the distribution of upserts shouldnt' be too bad, but I'll check to see what it is
[20:05:31] <GothAlice> arussel: Store a normalized-lowercase version of that field, then search lowercase.
[20:06:45] <arussel> yeah, I know that /^foo/ uses the index and I know the work around, but I was hoping /i could use it to 'out of the box'
[20:11:27] <arussel> with an index on {foo:1}, I was expecting: db.mydb.find({}).sort({foo:1}).skip(500).limit(1) to scan only 1 document. Is my index badly done or did I misunderstood something ?
[20:16:08] <StephenLynx> and if you wish to update multiple documents, you need a third block, for options. and then you put multi : true in this third block.
[20:17:14] <lmofi> StephenLynx: the document inside the collection looks like ObjectId("55033asdsa03b193d4319de486")
[20:19:41] <lmofi> StephenLynx: but im getting in the document with ChannelTB.findOne({name: {$regex: '^' + channel_name + '$' , $options: 'i' }}).exec(function(err, channelTB){
[20:34:42] <hmsimha> Hey, I'm reading about the changes to MongoDB between the 2.4 and 2.6 releases. It seems that a big one that might affect compatibility with the application I work on is returning a cursor for aggregations, instead of the full set. Is there a way to ensure aggregations still request the full set instead of a cursor?
[20:38:16] <medmr1> i just went from 2.4 to 2.6 finally. pro-tip: don't let the ops guys install 2.6.3, you want 2.6.8
[20:42:10] <hmsimha> ah, we use compose.io cheeser. The upgrade itself isn't the problem (just select from a dropdown) but I want to make sure it's not going to break anything in the application
[20:47:49] <hmsimha> More compatibility issues to check StephenLynx. The main reason to upgrade is so I can take advantage of bulk inserts
[21:22:15] <lmofi> Omg can somebody help me with a documten i want to empty up a index inside a document, collection > document >1 index value named messages
[22:47:44] <daidoji> has anyone ever run into an issue where mongoexport | bzip2 always causes a corrupt bzip file?
[23:47:11] <cchristie> i'm trying to get the latest version of mongo installed, but i am having difficulty
[23:47:51] <cchristie> when i run `sudo apt-get update` i am presented with W: Failed to fetch http://repo.mongodb.org/apt/ubuntu/dists/utopic/mongodb-org/3.0/multiverse/binary-amd64/Packages 404 Not Found
[23:48:09] <cchristie> W: Failed to fetch http://repo.mongodb.org/apt/ubuntu/dists/utopic/mongodb-org/3.0/multiverse/binary-i386/Packages 404 Not Found
[23:58:35] <sahilsk> Greetings.. I just install MMS agent using ubuntu debian file. I can see the agent entry in mms gui.. I click on "install monitoring agent" and "install backup agent" but notning happen .
[23:59:19] <sahilsk> All 0 Mongo processes are in goal state, Monitoring agent in goal state, Backup agent in goal state