Log file Viewer

[00:24:23] <tomasso> i have a collection of objects that currently do have a lat and lng properties embedded. Im reading about geospatial indexes, and it says that i need to use GeoJson object. The thing is that the GeoJson format pretends to be the central feature of the object, so that it has a definition, and then a properties field in which you can embeed your stuff.. the question is. Should I create a list of GeoJSON objects and in the properties field reference my curren

[00:51:17] <ColKurtz> if i have a resource that could be one of multiple types, how should I model that? for example, I have Posts, and each of those posts could either be an image, video, or text.

[00:53:14] <ColKurtz> should there be a "type" key, with a value that indicates which type?

[01:53:44] <foobar2k> I'm getting a strange error when querying a mongos instance

[01:53:52] <foobar2k> "too many retries of stale version info"

[01:53:54] <foobar2k> any ideas?

[01:54:01] <foobar2k> restarting mongos didnt help

[02:55:34] <rdegges_> Hey all, I'm considering using Mongo for a new project, but have a question I can't really seem to figure out from anywhere else.

[02:55:54] <rdegges_> I've got this project which runs a series of different web scrapers, and collects loosely structured data as JSON.

[02:56:19] <rdegges_> Right now I'm POST'ing all this data into a PostgreSQL database, but it's sort of annoying as I've got to maintain a schema for each scraper source, and the data is non-relational.

[02:56:40] <rdegges_> I'm also having issues getting my data OUT of the postgres database: I've got a simple GET API that allows me to paginate through all my results.

[02:56:55] <rdegges_> And it's really slow to do the whole: SELECT * FROM blah LIMIT 25 OFFSET 25;

[02:57:00] <rdegges_> Here's my question:

[02:57:21] <rdegges_> If I move my data over to mongodb, can I expect my GET API to become quicker when I paginate through large amounts of documents?

[02:57:31] <rdegges_> I'm looking at having around ~300m documents per collection.

[02:57:35] <rdegges_> With rougly 10 collections.

[02:57:43] <rdegges_> *roughly

[02:58:02] <cheeser> "quicker" is probably hard thing to say

[02:58:04] <rdegges_> The main thing I'd like is to have my GET API become a lot quicker, so my client applications can quickly paginate through the entire dataset.

[02:58:09] <cheeser> it's be simpler by far

[02:58:18] <rdegges_> Well right now, for instance --

[02:58:33] <rdegges_> If I get to page 400 of results, I'm looking at over 5 minutes of query time to retrieve 25 documents from Postgres.

[02:58:38] <cheeser> it'd *probably* be quicker given that mongodb tries to keep the working set in memory, etc.

[02:59:01] <cheeser> oh, that should never happen if you index properly.

[02:59:10] <rdegges_> Well, for these queries I'm not hitting any indexes.

[02:59:17] <rdegges_> I'm just paginating through an entire table / collection.

[02:59:34] <rdegges_> the equivalent of: mongo.db.collection.find().limit(100).offset(100) or something similar

[02:59:36] <cheeser> using skip() should be fine either way

[02:59:47] <rdegges_> I just want to paginate through like, 300m documents relatively quickly.

[02:59:53] <cheeser> it'd take a couple of hours to test, really.

[03:00:08] <rdegges_> hm

[03:00:14] <cheeser> importing your data would take the longest.

[03:00:19] <supershabam> does your query need to understand the content in any way... or ONLY ever skips and limits?

[03:01:01] <rdegges_> cheeser: yeah, I mean -- I was going to kinda cheat.

[03:01:14] <rdegges_> I was going to do a pgdump, then use a script to read each CSV row into memory and submit it to mongo.

[03:01:17] <rdegges_> or something like that

[03:01:23] <cheeser> that could work

[03:01:24] <rdegges_> since it's so slow to paginate through the postgres tables =/

[03:01:38] <rdegges_> supershabam: nope!

[03:01:43] <rdegges_> Only skips / limits

[03:01:59] <rdegges_> I mean, it'd be nice to be able to retrieve stuff by _id, on occasion.

[03:02:03] <rdegges_> but that'd be pretty rare

[03:02:08] <rdegges_> our _id is always a phone number (unique)

[03:11:43] <foobar2k> Can you not do sh.shardCollection("db.collection", {project_id: 1, _id: -1});

[03:11:52] <foobar2k> "Unsupported shard key pattern. Pattern must either be a single hashed field, or a list of ascending fields."

[03:12:06] <foobar2k> I cant find anywhere in the docs where it says the fields must be ascending

[03:12:46] <cheeser> i'm not entirely sure why the order would even matter.

[03:17:50] <foobar2k> nor am i

[03:29:44] <foobar2k> also, is there no way to delete a sharded collection?

[03:29:52] <foobar2k> db.mycollection.remove() doesnt do anything

[04:03:03] <SuperH> Hello.

[04:03:18] <SuperH> How good is Mongo support for ARM processors?

[04:04:33] <cheeser> nonexistent i believe

[04:05:42] <SuperH> Too bad. :(

[13:06:17] <durre> I wish to query all documents created after a certain date. is there a site that converts human readable dates -> timestamp -> hex -> mongodb id? :)

[13:09:34] <Nodex> you can't convert a timestamp into an ObjectId

[13:15:41] <durre> Nodex: ehm... ObjectId(Math.floor(new Date()/1000).toString(16) + "0000000000000000")

[13:36:56] <ckrause> This line doesn't work, but it shows what I am attempting to do

[13:36:58] <ckrause> db.symbols.update({"symbol":"spzp------^^^^^"},{"$set":{"symbol" : symbol.toUpperCase()}});

[13:37:28] <ckrause> I am new to mongoDB and I don't know how to reference a property to modify it

[13:37:59] <ckrause> in this case, I want to take document.symbol and update it to the UpperCase version of itself.

[13:42:12] <ckrause> I see there is a $toUpper, so I will use it in this case, however I still would like to know how to access a property to modify it

[13:43:51] <ckrause> well, $toUpper didn't work as I expected. I tried to use it like $inc

[14:08:38] <Nodex> ckrause : you can't use javascript functions like you're trying to

[14:08:52] <Nodex> the shell has no concept of "this"

[14:09:16] <ckrause> so I need to write a javascript script?

[14:09:35] <Nodex> you need to write somehting to loop it in a scripting language

[14:10:02] <ckrause> the shell will execute JavaScript though, correct?

[14:10:08] <ckrause> using load()

[14:10:29] <Nodex> I coulnd't say, I have never tried it

[14:10:59] <Nodex> http://docs.mongodb.org/manual/reference/method/load/

[14:11:02] <Nodex> seems it does

[14:26:52] <ckrause> I am about to try https://gist.github.com/tinjaw/7371691

[14:27:02] <ckrause> does that look about right?

[14:36:09] <the_lord> Hello, is there a way to tell mongo to ignore fields if they are being stored as null?

[14:47:30] <jmar777> the_lord: Have you looked at $exists yet? http://docs.mongodb.org/manual/reference/operator/query/exists/

[15:24:05] <fvs> hi, newbie here, running mongod with auth=true. I can't get records but am able to post without setting auth via localhost. Is this normal?

[15:24:46] <fvs> get = find(), post = insert()

[15:26:58] <fvs> output from get/post here: http://pastie.org/8465657

[15:32:22] <the_lord> jmar777, yeah, but I want it in the store level

[15:32:48] <jmar777> the_lord: not sure I follow, then. You want it to just not store null values at all?

[15:33:06] <the_lord> jmar777, I mean, I know that if you don't _explicitly_ tell mongoid that a field is nil, it won't send field=null to the document

[15:33:27] <the_lord> sorry, wrong channel

[15:34:08] <the_lord> jmar777, in mongodb, I was wondering if the database can be set to ignore the field=null when creating a document

[15:34:37] <the_lord> so if I send a document with some fields in null, mongo just doesn't store those fields

[15:35:14] <Nodex> the_lord : no that's not possible

[15:35:26] <the_lord> Nodex, ok, thanks

[17:33:09] <Juv1228> Hello, is it possible in the C# driver to specify default values for items in an array/list

[17:33:48] <Juv1228> for example, I have a List<double> which may contain many instances of 0, so I would like to set 0 as the default value and have it not serialize the default

[18:08:35] <gen0cide_> Ok mongo experts, here's a question: I've got latest source, latest scons, but I ONLY want the libmongoclient.a installed to my system - I don't seem to be able to enforce that with scons mongoclient --64 --release

[18:08:39] <gen0cide_> any idea / tips / thoughts?

[18:11:15] <gen0cide_> everything I've read online tells me I need to do a full installation which frankly seems stupid

[18:12:49] <Nodex> 42

[18:14:15] <gen0cide_> Nodex: I completely agree

[18:21:05] <gen0cide_> very helpful #mongodb

[18:26:15] <Nodex> it's friday night, not sure what you expect

[18:37:04] <maverick1> Hi, We are planning to setup a Disaster Recovery. Ideally we'd like to sycn the data from the production Replica Set to a node in DR. and from this DR node to 2 other nodes.

[18:37:04] <maverick1> and these 3 DR nodes shouldn't be primary at any time(priority=0 works?). but may participate in election.

[18:37:14] <maverick1> any idea?

[18:38:00] <maverick1> Can someone help?

[18:41:20] <Slacker1> Can some one answer the maverick1's question?

[18:44:09] <cheeser> i think priority=0 should work

[18:47:08] <Slacker1> cheeser: thank you, and as 3 nodes in DR, we want only one node(Let's x ) to sync the data from original replica set and others 2 nodes should sync data from this DR node x.

[18:47:14] <Slacker1> Is that possible?

[18:52:58] <cheeser> ehershey: do you recall if you can force one replica set to only sync from another? I want to say that's possible but I don't recall offhand.

[18:53:04] <cheeser> i'll dig in to it, though.

[19:18:25] <Slacker1> cheeser: thanks

[19:43:31] <drjeats> is using findAndModify to set a sort of application-level write lock on a document a common and relatively safe pattern?

[19:50:02] <cheeser> drjeats: it's probably what i'd use, fwiw

[19:50:53] <drjeats> cheeser: any sort of affirmation helps. thanks!

[19:51:50] <cheeser> sure

[19:52:33] <joannac_> cheeser: very late, but http://docs.mongodb.org/manual/tutorial/configure-replica-set-secondary-sync-target/

[20:03:57] <clamothe> In querying an array, can I use both $in and $all? What about multiple $in's and an $all? I.e. db.coll.find({characteristics: {$all: ["required1", "required2"], $any: ["blue", "green"], $any: ["flat","round"]})

[20:04:19] <clamothe> /r/any/in

[20:07:25] <TylerB> If I were to create an index on "age", then do a query on "age" and "gender", would the "age" index be used and then the "gender" part of the query be done? Would it improve performance? I've been unable to find a concrete answer online.

[20:07:32] <cheeser> joannac_: danke

[20:07:58] <cheeser> of course, Slacker1's gone, but oh well.

[20:08:07] <cheeser> now i know. and knowing is half the battle.

[20:08:26] <cheeser> the other half is pure, unrestrained violence.

[20:10:22] <TylerB> And, as it goes, I've just found it.

[20:10:56] <cheeser> rubber duck debugging++

[20:11:58] <ehershey> cheeser: sorry I missed your q earlier

[20:12:12] <cheeser> no worries

[20:12:24] <cheeser> i only ambushed you specifically because you happened to join about that time. :D

[20:34:20] <LoneSoldier728> hey how do i create two object ids

[20:34:23] <LoneSoldier728> is it possible?

[20:34:28] <LoneSoldier728> on one document

[20:40:39] <joannac_> LoneSoldier728: What's the use case?

[20:41:25] <joannac_> LoneSoldier728: http://pastebin.com/NpxP2Xqa

[20:57:54] <LoneSoldier728> I am trying to make a unique id

[20:58:06] <LoneSoldier728> without having to use a whole auto increment system

[20:59:08] <LoneSoldier728> but yeah I saw that

[20:59:27] <LoneSoldier728> x = ObjectId() in

[20:59:37] <LoneSoldier728> but yeah I saw x = ObjectId ()

[20:59:46] <LoneSoldier728> in mongodb's docs

[21:26:41] <gen0cide_> I've got latest source, latest scons, but I ONLY want the libmongoclient.a installed to my system - I don't seem to be able to enforce that with scons mongoclient --64 --release - all I want is the /usr/lib64/libmongoclient.so to exist :)

[22:01:48] <DaveKS> i'm running 2.2.2 with the mongo driver for ruby and am trying to do a $project after a $group using aggregate and it's not working at all.... https://gist.github.com/davesloan/7378377

[22:13:29] <ctorp> Is there a way to pipe a mongo query to vim?

[22:35:55] <DaveKS> not sure this is the right room for this... http://stackoverflow.com/questions/19869555/the-project-operation-fails-to-work-with-mongoid-aggregate-method-when-used-aft

[23:36:36] <ctorp> Is there a way to pipe a mongo query to vim?

[23:51:23] <joannac_> Number6: ping

[23:52:02] <Number6> pong

[23:58:30] <parashute> I am using Spring Data. What annotation should I use for member objects. For example, for person JSON {name:"", address: {"line1":"", "line2": ""}}. For name I am using @Field annotation. What annotation should I use for address?