Log file Viewer
#mongodb logs for Monday the 29th of July, 2013
[00:06:01] <tpayne> yeah this isn't working out very well, it does a find instead of an absolute heh
[00:22:49] <tpayne> cheeser: yeah, for example if the value is ["2", "3"] and I insert ["2"] it collides
[00:52:50] <caitp> client's request has that start time URL encoded (coming from an HTML5 time input)
[03:30:10] <pngl> Do people use the projection part of queries a lot or do people get all the records back and sort things out in their language's driver?
[03:45:04] <cheeser> i'll see if i can't shake some trees tomorrow. you could probably file an issue or two on the docs/examples
[03:49:08] <tpayne> cheesed maybe i can ask you how to do something via command line, that way i'll know if it's possible or not
[03:51:37] <tpayne> i want to query mongo to give me all of this data back, in a bucket. OR Map[String, List[String]]
[03:52:19] <tpayne> so that i have Map("hello" -> List("world", "tpayne"), "goodbye" -> List("buddy"))
[03:52:47] <tpayne> that's it. I was looking into using coll. aggregate to returning something like this
[06:00:31] <raar> Hi, I just removed all collections from a 800GB database and I want to re-claim (un-allocate) this space. I'm considering deleting the whole database, but I don't want a global write lock for more than a couple of seconds - any idea how long deleting this DB might take?
[07:27:07] <raar> any ideas on my issue? How to remove allocated space from my now empty database, which is 800gb?
[07:36:49] <rspijker> cmex: you can do it nested, just keep in mind the maximum size of a document is 16MB
[07:37:34] <raar> the database is empty, but still takes up the originally allocated space (which is expected behaviour)
[07:38:08] <cmex> my problem is to work with data like this . i mead find a document inside document update ..
[07:38:27] <rspijker> which will lock it, but if the amount of actual data in there is small, it will be fairly fast
[07:40:20] <raar> I don't have access to the filesystem at present (it's being hosted by mongohq) - repairDatabase() on 800GB is going to take a while I assume, no? I don't want a global lock of more than a couple of seconds
[07:41:06] <rspijker> raar: the problem with repairDatabase is that it needs 800GB to start if your current storage size is 800GB
[07:41:44] <joannac> cmex: The link rspijker gave has a example on how to find a path through the thread to a comment
[07:41:53] <raar> rspijker: The actual data - well it's empty, so I guess the actual data is zero, no?
[07:49:41] <raar> the problem is that we have a whole bunch of other databases that are production-critical on the same setup
[07:50:40] <joannac> Warning This command obtains a global write lock and will block other operations until it has completed.
[08:53:15] <_Heisenberg_> hI! I installed mongodb on ubuntu 12.04 which automatically installs startscripts in /etc/init and /etc/init.d ... If I want to autmatically start a configserver instead of a normal mongod instance, do I have to change the startscripts or just a config file?
[08:56:31] <rspijker> _Heisenberg_: if you want to start both, you need to add another start script for that
[08:56:47] <rspijker> if you want to start config _instead_ of mongos, then you only need to modify the config file it uses
[09:02:23] <_Heisenberg_> Nodex: by default it does not use a config file? so I need to change the intit scripts to tell it to use a config file no?
[09:08:53] <_Heisenberg_> seems like i fucked up the installation. cant even start the mongod by using sudo service mongodb start, ps ax returns no runnig mongod instance
[09:12:46] <_Heisenberg_> yes, there is. but also in /etc/init/ . here there is a symlink to /lib/init/upstart-job and a conf file
[09:17:36] <_Heisenberg_> can't figure out the config file from the init script --> http://pastebin.com/mKEvDW4j
[09:30:44] <rspijker> puppeh: are you using 1 $addToSet to add multiple items? If so, are you using the $each command?
[10:36:47] <aandy> hi, i'm running 2.4.3, and have a weird situation i don't understand. i have a collections, empty, then import a json file with mongoimport which reports ~1k imported objects. from the shell, db.find()/db.findOne() still says null. i then try to insert a doc manually from shell, and it can see only that doc. i export the whole (supposedly 1 object) collection and end up with a ~1k json file again. the only setup for the collection i did was to add db.ensur
[10:37:50] <aandy> it's part of a 1 master, 2 slaves setup if that helps shed some light on it. all actions done to the primary
[10:38:25] <aandy> i did change to the admin db to enable text indices, but i've reverted back since
[11:10:48] <_Heisenberg_> I set up a replicaset in order to deploy a sharded testcluster. am I on the currect way? what makes me suspicious is, that all replicas seem to be "primarys" ---> http://pastebin.com/z6q4yTRB
[12:10:21] <hard_day> i have a problem with cursor , i'm reading the doc and the google group by continuous the problem and i have no a idea http://pastie.org/8186478
[12:10:38] <amitprakash> Hi, is it possible to pass the document name as a variable? i.e. instead of saying db.Document.find, I want to call db.find(documentnName, params)
[12:12:24] <kali> amitprakash: you mean the collection name, right ? if not, you're seriously confused
[12:20:33] <kali> hard_day: http://docs.mongodb.org/manual/reference/limits/#Restriction on Collection Names
[12:25:07] <jimbosimo> hi, question if anyone knows, what will happen if I issue a multi-document update instruction and server crashes before all documents are updated? will it redo the last operation when it goes up again ?
[13:08:07] <_Heisenberg_> if I try adding a replica as a shard I keep getting "couldn't connect to new shard ReplicaSetMonitor no master found for set: rs0" even if the replica set has one primary and two slaves in it
[13:28:06] <_Heisenberg_> kali: sorry just dropped out. my command looks like that; mongos> sh.addShard("rs0/10.1.1.49:27017")
[13:50:19] <cheeser> yeah, i think you have to pick with the host name or the IP and be consistent with that usage everywhere.
[14:10:58] <_Heisenberg_> someone has a tip where to get sample data sets, product catloge would be great :>
[14:23:59] <cheeser> i need that for my javaone presentation actually. i need a ton of "products" in my db to play with.
[14:27:05] <ron> dude, you're just embarrassed admitting I actually gave a good advice. it's okay, I understand.
[14:27:44] <cheeser> it's not so much embarrassed as amazed. i've never seen it happen before. ever. not sure how to cope.
[14:28:24] <remonvv> To be fair I just spit out some tea when someone mentioned ron said something useful.
[14:28:46] <remonvv> I'm so proud. It's...it's hard to put into words really. I need a moment to gather myself.
[14:29:12] <ron> Nodex: cheeser knows me for far longer than you do. and trust me, once he learns you adore php, he would have as little respect towards you as me. probably even moreso.
[14:31:02] <Nodex> yet ron i am still better than you in spite of these things so that really doesn't say a lot about you does it :)
[14:32:00] <Nodex> in all seriousness, apart from making tea and being a fluffer what do you do in your job?
[14:34:17] <remonvv> Indeed. It's rather like a blind guy complimenting you on your painting skills.
[14:35:24] <cheeser> i've just wrapped up everything i'd tagged for 0.102 and was about to cut a release.
[14:35:46] <remonvv> I wrote my own mapping layer after Morphia had some issues. But this was way back when.
[14:36:53] <remonvv> cheeser: It wasn't super active, I think only scott was maintaining it. And I don't like the API. And there were quite a few bugs at the time but I can't comment on that now.
[14:37:24] <cheeser> the API will likely shift quite a bit with the switch to the 3.x driver underneath
[14:38:47] <remonvv> cheeser: 3.0 will change details, maybe. But the general approach of using builder patterns for query composition and whatnot will stay the same.
[14:39:26] <cheeser> the 3.x driver does a lot more than it used so morphia will trim down a bit, i expect.
[14:40:43] <remonvv> cheeser: Okay, but that's moving things around. The gap between JSON/shell syntax and Morphia/driver code is too big in my opinion, and much more verbose.
[14:41:50] <remonvv> cheeser: Wouldn't go that far but our developers pretty much universally wanted to get rid of Morphia in favor of a more..hm...JPA-ish style.
[14:45:03] <remonvv> cheeser: It's hard for me to explain in detail, not a native english speaker ;) Our code basically looks something like this : User u = find("{userId: ?1"}).one(); List<User> users = find("{someStatestate: ?state}"}).all()
[14:48:38] <ron> when cheeser asks things that way, I'm never sure if he's asking sarcastically or not.
[14:48:39] <remonvv> cheeser: Well, JPAQL is a new query language spec, I rather think mongodb's query language should be mirrored as closely as possible in APIs
[14:49:34] <remonvv> cheeser: I've had this discussion with scott as well and I think it basically boils down to type safety and whatnot. Our stuff has to do runtime type checking and whatnot.
[14:49:39] <cheeser> remonvv: right. i've played with doing that in ophelia. it's a bit tricky but I've learned a way to help that out. it just means a server round trip in the easy solution.
[14:50:17] <cheeser> basically morphia is JPA's criteria API for mongo when you want something more like its QL
[14:51:19] <remonvv> cheeser: Right, we need maintainable/readable code and Morphia/driver code doesn't give you that in our experience.
[14:52:04] <remonvv> cheeser: It becomes cluttered, verbose and so forth. This may improve in 3.0/Morphia next gen but at that time we had to decide.
[14:52:05] <cheeser> i wrote critter to help with the maintainability but it's kinda going in the other direction.
[14:52:52] <_Heisenberg_> ron: I dont get it maybe you can help me, what does the two parameters of "repeat" mean in the http://www.json-generator.com/ thing?
[14:52:56] <remonvv> cheeser: And right now it's a 3year old layer that runs on production clusters that routinely host 100,000+ concurrent users so for us switching back to something else that we don't have to maintain is only an option if the functional gap is relatively limited and the robustness equal or better.
[14:54:11] <cheeser> it's mildly experimental. i've used it in a couple of projects and it works well enough so far.
[14:54:50] <remonvv> cheeser: Another thing is that we had to add a ton of other features that were then (and sometimes still) not supported. For example, hash indexes and automatic hashing of shard key values, validation of cluster topology metadata versus our document POJOs (uses similar model to Morphia but with slightly different annotations and sometimes different behaviour).
[14:55:42] <cheeser> i'm adding support for text indexing and queries and also aggregation in the next few weeks.
[14:58:46] <remonvv> cheeser: It's not impossible, but it's hard. And dislike it too. It just doesn't outweigh the benefits.
[15:02:58] <ron> cheeser: Derick is the php guy. so it's kind of a dilemma. you have to be nice to him since he's a 10gen employee, but he does php.
[15:04:37] <cheeser> it's too bad you're not in the office to meet in person. i'd love to be in the UK for a while.
[15:07:27] <cheeser> 3.x has encoders/decoders which do a lot of the work shuttling data that morphia's been doing. morphia will use/build on that functionality.
[15:10:21] <remonvv> Yeah as long as the driver itself doesn't get too feature-creepy. It should be a driver with a low level interface. People (or 10gen) can build seperately packaged magic on top of that.
[15:10:35] <cheeser> Derick: sorry. i had a whitelist running. i need to update my config to not load it by default anymore. try again. :)
[15:23:51] <flaper87> Hey guys, Can you help me figuring out why this query is not using _id's index for sorting documents? http://bpaste.net/show/OX41rlImcg2eHZYe6V4c/
[15:25:25] <Derick> and it uses "active" already for the match, and it can't use that for the _id sort
[15:34:35] <remonvv> At some point someone should write a good blog about composite indexes and sorting and such.
[16:01:30] <ashley_w_> i'm trying to do a query where the timestamp is newer than another timestamp, but i get "$err" : "error on invocation of $where function:\nJS Error: ReferenceError: o is not defined nofile_b:0" http://pastebin.com/RaAf7XfT
[16:04:19] <Nodex> http://docs.mongodb.org/manual/reference/operator/where/ <-- look at the table on the bottom for supported commands to "this"
[16:11:52] <Nodex> http://docs.mongodb.org/manual/reference/operator/where/ <-- look at the table on the bottom for supported commands to "this"
[17:40:47] <rogerthealien> hi, i have a question and was hoping someone here could help... I'm getting "TypeError: if no direction is specified, key_or_list must be an instance of list" when trying to call collection.ensure_index({'attributes.id':1}) through pymongo, and the pymongo documentation doesn't seem to be very helpful on this
[17:42:48] <idank> I have two collections: companies and employees. When choosing between embedding employees or putting it in a separate collection with a company reference, how does one deal with high read throughput (embedding is superior) and thousands of employees per company, surpassing the 16MB limit per document (GridFS sounds like overkill)?
[17:49:29] <idank> DanWilson: but that kills read performance when reading companies and getting their employees
[17:52:47] <DanWilson> are you worried about the disk seeking? or the total amount of data brought back?
[17:53:55] <idank> DanWilson: worried about seeking. I've done the reference thing once and it killed performance
[17:55:40] <bcows> if each employee only has 1 company then it would be a smaller document then embedding all company employees in a company document
[17:56:23] <idank> bcows: yeah but iterating the employees grouped by company will still involve a massive amount of seeks
[17:56:54] <idank> DanWilson: the employees collection has 30 million docs with an int _id, companies is around 3 million
[17:57:03] <Nodex> idank : I think the main issue you're missing is embedd what you need for 1 read and store the rest in a separate collection
[17:57:44] <idank> Nodex: for each company I need all its employees, what would go in a separate collection?
[17:59:14] <Nodex> idank : I am talking about for a quick view..... for example if you only need the name of each employee on a result of a company then drill down further then embedd each name and a reference to the _id of said employee in the other collection
[18:00:25] <Nodex> so store them in thier own collection and embedd them in the employer.employee array also
[18:01:33] <Nodex> if you dont need the data until someone click and drills down then there is no point
[18:03:40] <Nodex> if you're constrained then you have no choice but to reference, in which case you shoud embedd the _id of the employee and do an $in - which may or may not be faster
[18:07:55] <Nodex> well I suggest you use replica sets for the read and reference the _id's if you;'re worried about size
[18:09:06] <ashley_w_> hence my "throw more hardware at it" comment. vague, but i had no idea of current setup
[18:11:32] <peterjohnson> I'm trying to search a message database for any messages with body containing website links (starts with www. , http or contains .com) but for some reason when I try db.messages.find( { type : 0 , body : { $regex: '\.com', $options: 's' } } the period isn't included in the search, so any result with "com" comes back
[18:11:35] <Nodex> not a lot you can do about it tbh, you can't control when an employee is added and thus cannot control the seek
[18:14:48] <Nodex> I didn't mean splitting when embedding and it still doens't get you round your 16mb problem
[18:15:36] <Nodex> how many req/s do you think will kill your app ? is it CRM and if so how large scale is it?
[18:18:30] <Nodex> here is some specs... our CRM has roughly 25M docs with perhaps 50M associated to the docs - contacts/notes/documents (pdf's etc) and I can fire wel over 1000 req/s at it without it blinking
[18:20:15] <idank> but I was in a similar position a month ago, only the two collections were for scraped websites
[18:21:19] <idank> that thing had terrible performance, once I started embedding things worked much better
[18:24:58] <Nodex> mongo adds a certain amount of padding per document incase it grows a little but not a lot
[18:25:33] <Nodex> I understand if you scrape again and find another page you missed that it will be at the end (fragmentation) but that's just the way it is
[18:26:49] <Nodex> scraping is a bad idea to embedd, you don't control the data or it's size so it's not predictable performance
[18:43:31] <bluefoxxx> I have a node on a secure network that I would like to connect to a replica set as a non-voting, hidden member. Ideally it would connect to the MongoDB replica set in the DMZ and keep a running replica of all the data.
[18:43:49] <bluefoxxx> the idea being that this one is on our secure network, and is pulling data for backup.
[18:44:40] <bluefoxxx> we have Symantec back-up software, so the client has to connect to the server rather than a back-up agent supplying a pull service (it does, but they have to have two-way connection initiation--client must connect to the server, server must open connections to the client)
[18:45:28] <slowest> I saw that there's group by and count/sum functions for mongodb. Will this work properly if you have a large data set spread on several servers? I guess itwould be quite slow if it had to fetch from all servers? Even if it's properly indexed it would still have to talk to all servers?
[18:46:16] <cheeser> because unless you're sharding, all of a collection can be found on each machine in the repl set
[18:46:36] <mfletcher> Hi, anyone here with experience in load balancing mongodb servers? Id like to use a load balancer to balance connections between a bunch of mongo db servers that would be a in replica set. Is there any side-effects to load balancing all the nodes in the replica set including the primary node?
[18:48:48] <mfletcher> I was just thinking for the point of view if I have webs pointed at the mongodb servers, and one of them goes down, or out for maintenance, Id still like that web server to serve traffic
[18:49:17] <remonvv> mfletcher: Use local mongos processes, use appropriate read preferences to balance reads
[18:49:48] <slowest> cheeser: oh, I thought it was sharded automatically, or either way I'll have to shard it because of size.
[18:49:50] <cheeser> remonvv: i would say I should probably just get to work but i suppose this is technically work. :)
[18:50:13] <remonvv> mfletcher: What language? Java has a different implementation of read preferences than all the other drivers
[18:52:00] <remonvv> cheeser: No not that one. It's a two-part. I think currently read secondary is conceptually broken in mongos due to a weird sticky connection rule and there's a special Java only implementation of that read preference that makes it different for that environment.
[18:52:44] <mfletcher> We're using python with pymongo. Im not familiar with the implemenation, one of my devs tells me he just maintains a list of of the mongo servers in a config file, but I figured there might be a better way to do it, since we'd be scaling up and down the number of mongo servers in the farm
[18:53:18] <remonvv> cheeser: https://jira.mongodb.org/browse/SERVER-9788, the Java note is is Randolphs comment "Actually, the Java driver is a special case where the pinning behavior must be explicitly requested by the user."
[18:54:33] <remonvv> cheeser: I don't recognize 889. I've done tests with that setup and it works fine.
[18:54:50] <remonvv> cheeser: The issue there is most likely the "closest first" system in the current driver
[18:56:38] <remonvv> cheeser: Not sure if they/you are redoing the io/network thing but another weird issue is that you can improve throughput by having multiple Mongo(Client) instances. Seems to be a locking issue somewhere.
[19:00:06] <remonvv> mfletcher: Either way, the better way to do it is put mongos in front of it and let the config servers deal with cluster topology data. Your app should only have a configurable endpoint for mongos. If you want to connect directly to a repset you will need all mongod addresses.
[19:12:27] <remonvv> mfletcher: yw, the only downside to mongos is that is relatively cpu hungry so it will affect net throughput of your app per box slightly if it's pressed
[19:32:19] <Ontological> I'm having a hard time using simple regex on pymongo. Here is an example of my problem: http://pastie.org/private/1rzmozkankutqc7thbcjng
[19:35:16] <remonvv> Ontological: Also, if at all possible try to reproduce your problem in the shell first
[19:36:17] <Ontological> Well, the shell does it correctly because the shell doesn't encapsulate ''. That's my problem, is that it's being treated as a string
[19:56:18] <konr`> So, dear folks, is there an utility to import data from an .sql file to a collection in mongo?
[19:57:24] <jblack> konr`: Not that I'm aware of, but you should be able to use a csv gem and mongo to do it easily enough.
[19:58:59] <jblack> http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV.html : CSV.foreach("path/to/file.csv") do |row| Mongodb insert row; end
[20:00:45] <pwelch> anyone seen this error when trying to connect to a replicaset with ruby driver: Mongo::ConnectionFailure: Failed to connect to any given member.
[20:02:56] <pwelch> ranman: can you expand on that? If I use the MongoClient.new it connects to a single instance fine
[20:09:03] <pwelch> ranman: reading over it. going to get a co-worker to get a second pair of eyes to see if it's PEBKAC
[20:36:24] <cheeser> i just pushed morphia 0.102 for those interested. https://groups.google.com/forum/#!topic/morphia/TNrA-y5RS8g
[20:41:38] <caitp> for a document with a start date and end date, I need to be able to query for documents where the start date has a time which starts at a certain hour
[20:42:59] <remonvv> caitp: Ignoring $where which you shouldn't use; that's not really possible. The standard way of doing that sort of thing is to store the exact data you want to query on. In this case if you update the start date also write a field for the start data hour of the day.
[20:43:41] <caitp> I am currently storing auxilliary fields which are UTCHours() * 3600 + UTCMinutes() * 60
[20:46:29] <caitp> the aux field thing doesn't really work, because I don't get reasonable values from dates
[20:46:56] <ron> remonvv: dude, it's late for you. why are you on irc? you have a life, not like me.
[20:48:11] <caitp> and then converting to UTC to try and get around the timezone issue screws it up a fair bit more
[20:52:52] <caitp> I'm not sure why this doesn't make sense to you, I think I'm communicating clearly :l
[20:53:24] <remonvv> caitp: I'm not disagreeing. But if you're getting 1am UTC out of a simple Date.getTime() manipulation for a UTC Date that's midnight something's going wrong no?
[20:55:37] <remonvv> caitp: secondsInDay = new Date("2013-01-01 00:00:00Z").getTime() % (24 * 60 * 60 * 1000) / 1000
[20:55:50] <caitp> if I say "var date = new Date(); console.log(date.getHours())" at 1am, it will return '0'
[20:57:21] <caitp> the strategy of storing times as an integer of seconds since midnight in the database, in UTC time, simply is not working
[20:58:11] <caitp> because otherwise, I have to tell someone that a feature they want is not feasible
[21:01:58] <caitp> what I need is literally just hours/minutes, I can't take the date into account. eg, "Show me the events that happen after midnight", not "Show me the events that happen after midnight of august 1st"
[21:02:21] <caitp> but of course, "after midnight" doesn't really give you anytthing meaningful, that would give you everything
[21:06:52] <remonvv> caitp: getTime() returns the number of ms since epoch for the given Date. If you strip the date and millisecond components you will have a seconds in day value.
[21:13:02] <remonvv> Print current UTC date, then calc those seconds, convert them to hours and minute and show me that's a different value than current UTC hours:minutes
[21:15:16] <caitp> so, using a UTC timestamp isn't going to be helpful here, because UTC midnight is 0, and >= 0 is going to return everything, and <= 0 is going to return almost nothing -- Local 6:00 pm might be UTC midnight, for instance
[21:20:51] <remonvv> Right, so seperate the date component from the hours (or seconds, whatever) in your doc, both UTC. Do timezone conversion to UTC, you'll get one or two candidate matches (two if your range runs across two UTC days even if it's single TZ local day) and combine appside as needed.
[21:21:33] <remonvv> It just means you have to query for two documents sometimes if your range intersects UTC midnight. It's not exactly rocket science.
[21:23:32] <remonvv> Not really. Our stuff (amongst other things) tracks daily and hourly leaderboards for gaming services that do roughly that.
[21:23:34] <caitp> "You'll get one or two candidate matches (two if your range runs across two UTC days even if it's asingle TZ local day)" -- I don't follow your logic, one or two candidate matches for what?
[21:24:12] <remonvv> Was your original problem that you need to finds documents that match an certain "hours of the day" range?
[21:24:37] <remonvv> e.g. "Get all foos that were created between start time and end time of day T in timezone Z"
[21:25:38] <caitp> the service is an event management thing, so events have startdates/times and enddates/times -- the UI is supposed to enable you to search for dates/times starting at or after X or ending at or before X
[21:26:01] <remonvv> yes but X as in absolute date or X as in time of the day (so date component removed)
[21:28:02] <remonvv> Say you store your times in your event as {utcDateInDays:221323, utcTimeOfDayInSeconds:8731}
[21:28:26] <remonvv> That format would make it super easy for a UTC based range check but you have timezones
[21:28:46] <remonvv> so in cases where start.getUTCDay() != end.getUTCDay() you need to create two seperate queries
[21:29:28] <remonvv> One that grabs the events from your local TZ start to UTC midnight and one from UTC midnight to your TZ end (your TZ values converted to UTC)
[21:30:26] <remonvv> Just remove the TZ conversion from the problem and create code that allows range queries on UTC start to end where end - start < 24
[21:33:41] <remonvv> So say your local TZ range is from 15:00pm to 23:00pm that might convert to say; UTC 19:00pm to 03:00am (next day), then you'll grab all events that are 19:00-00:00 on utcDateInDays = X and all events 00:00-03:00 on utcDateInDays = X + 1
[21:34:26] <remonvv> or if you need all of them rather than for a specific day ignore utcDateInDays altogether
[21:35:19] <remonvv> So your UI shows all events between 01:00 and 02:00 for all days in the past or something?
[21:36:23] <caitp> filtering by _just_ time cannot use date as a factor (so I can't say, check >= utcTodayX AND <= utcTomorrowY)
[21:41:13] <remonvv> If time T in your local time converts to UTC = 0 then you're assured the end of the range is <24
[21:41:39] <remonvv> At what scenario would you not have an end time if your requirement is "of the day"
[21:44:43] <remonvv> Wait, X as in I want events that ended after the start of the hour and before the half hour mark?
[21:47:48] <remonvv> so in UTC "between 04:30 - 02:00(+1)" = "between 04:30 - 00:00" OR "between 00:00 - 02:00"
[21:49:40] <remonvv> Your simple shifting the window and have the complication of having the UTC midnight as a range seperator
[21:50:34] <remonvv> so if your UTC converted range is still within the same UTC day you can simple TZ convert, if it spans two UTC days you have to use an $or query to grab both ranges.
[21:52:19] <remonvv> caitp : Test it in a spreadsheet or something if you can't wrap your head around it.
[21:54:55] <caitp> in a case where I am only specifying a start time, I need to say "show me events that start between <UTC startTime> and UTC midnight"
[21:55:12] <remonvv> well, only small footnote is that the right side of your ringe has to be *inclusive* on the highest possible value (23 if you do hours) rather than the 0 (00:00) i mentioned to clarify the approach
[22:01:10] <caitp> yeah it didn't seem like it was going to make things simpler :( although this looks like it could turn into a potentially complicated bad query too
[22:02:26] <remonvv> easy case is {end:{$gte:<S>, $lte:<E>}} and worst case is $or:[{end:{$gte:<S>, $lte:"23:59"}}, {end:{$gte:"00:00", $lte:<E>}}]
[22:04:01] <caitp> {$and: [{$gt: {'startTime': startTime}}, {$lte: {'startTime': utcMidnightMinus1}}]},
[22:05:07] <Chammmmm> Hi Guys, I have getting this error message: balancer move failed: { ok: 0.0, errmsg: "" } ... I got this after adding a shard.. everything was going smoothly and now it cannot move chunks.. and I get an empty error message.. (running mongo 2.4.2) I tried to restart all the mongos.. and restart the config servers.. but nothing
[22:09:27] <caitp> constructing the query looks like this http://pastebin.mozilla.org/2739992 -- whether it will work or not, i'll have to see
[22:11:38] <remonvv> Remember that even with just an endTime you can have two seperate clauses in your $or
[22:13:17] <remonvv> actualStartTime = startTime == null ? 00:00 : startTime; actualEndTime = endTime == null ? 23:59 : endTime
[22:13:40] <remonvv> see if there's an UTC midnight in between, if no, single clause, if yes, $or the two ranges that result
[22:15:36] <remonvv> midnight cross check = actualStartTime.toUTC().getDay() != actualEndTime.toUTC().getDay()
[22:16:56] <caitp> er, so with time there's no meaningful concept of day. I think it's just "if endTime < startTime -- assume midnight crossing"
[22:19:19] <remonvv> (although at some point we converted leaderboards to have timezone metadata because we rarely had cross TZ leaderboards and such ;) )