« Change the algorithm, not the dataset | Main | Why SMS GupShup is Bigger than Twitter »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

website design

Sweet. Now GupShup just needs to take a page from Yaari and spams the crap out of the address book of every new GupShup users to convince leftover Twitter users to migrate. No, I kid. Plaxo was pulling the same crap long before Yaari. Except now they're playing as Google's [subservient chicken](http://www.subservientchicken.com/) hoping it wins them brownie points with the early adopter crowd.

Robert Lowe

> Twitter has said in the past that sending messages via the SMS gateway is one of their most expensive operations, so the fact that only a small fraction of their users use the SMS option makes their task a lot easier than GupShup's.

I always assumed they meant that it was expensive in terms of financial, rather than computing, resources.

Dhananjay Nene

Twitter numbers seem to be much higher. On peak load days they are almost 130 million (I don't recollect the source but I will post it as soon as I find it).

Having said that, I think the sharding does make sense.

Thejesh GN

I guess in terms of traffic twitter is lot bigger than gupshup. Dont just consider the SMS.. they get more through their api/im/web. But only way of posting to GupShup is by SMS which is expensive.

Thejesh GN

Ok. I agree I had not visited the GupShup recently.. now they have web/im interface. Good. Let revive my old account.

Craig Baker

They run JBoss appserver with C++? That's a little odd.

John thomas

LOL, funyn name but twitter does ROCK!

JT
http://www.ULtimate-Anonymity.com

secretGeek

interesting write up.

i've previously heard this "sharding" referred to as "horizontal partitioning" or "row partitioning".

gregory

is this article going to make techmeme? will anybody learn anything from it connected with twitter? and why have i never heard of gupshup, is it only a metro thing? vacuums everywhere

Super Dude

Ha proof that RoR is a passing fad. I told you so.

Die annoying script kiddies, die!

PJ Brunet

But a web interface is way more complex/demanding than a cell-phone interface, lots more code. And we're pounding Twitter all day long w/ our Twitkits and RSS readers, blogs auto-posting links, there's even a satellite in space hitting Twitter with updates, and then they got spam attacks to deal with, right?

Also you should explain your numbers. If I text something and it goes to 20 people does that count for GupShup as 21 messages? Because if I post something to Twitter I may have 20 followers but that probably only counts as one message.

Ed

Hi Anand,

I read where some companies are using RoR for the front-end but use something else for the backend.

Can you tell me how this is being done?

Are you using just the View, Controllers but not the Model/ActiveRecord?

Rishi Agarwal

Nice post.

As I too belong to an Indian start-up, I am really impressed by the usage numbers of Gupshup and the way they have handled scalability. Kudos for their achievements.

I have used RoR and liked it for its MVC architecture. But in the end, concerns do arise about its scalability.

tc

"We all learned back in the 90's that this is an unscalable model, yet it is the model for most Rails applications. No amount of caching can help a 2-tier read/write application scale."

So those Rails apps that are pushing more traffic than Twitter but seem to be scaling just fine, that's just a fluke?

Rails is 2-tier, Twitter uses Rails, Twitter has scaling issues, therefore 2-tier has scaling issues. Please ignore 2-tier sites that do not have scaling issues, as they contradict the point I'm trying to make. Uh-huh.

Besides, didn't we all learn back in the 2000's that RDBMSs aren't nearly as scalable as distributed databases like BigTable?

Anand Rajaraman

TC: You make 2 different points, which I'll try to address:

1. The 2-tier architecture makes the database the bottleneck for most nontrivial apps. I've seen this at many, many companies since the 1990's. That's one of the reasons app servers were invented, so you could partition the database.

2. BigTable (and it's open source cousin HyperTable) is indeed a more scalable model than an RDBMS. These pieces of software have a built-in middle-tier that abstracts the physical storage location and presents a single-system interface. So with BigTable you ARE building a 3-tier app.

Alberto

Interesting post, never heard of GupShup before. I will definitely try it, but I don't see references to open APIs and XMPP support.

Agree that 3-tier model is the recommended approach to build scalable web applications, but sometimes traditional MVC 2-tier architectures are enough and I believe that was on Twitter devs reasoning when they designed it as a CMS, since Twitter is used more as a messaging utility, they need to re-architect and denormalize.

For a complete comparison on Twitter's "public" estimates. It will be good to know how many connections (group subscriptions) do 7 million users in GupShup have. I assume is way more than the 4 million figure mentioned.

GupShup seems to have solved the scalability issue for Web and SMS but the real scalability challenge Twitter faces is XMPP support. Facebook has a good approach http://highscalability.com/new-facebook-chat-feature-scales-70-million-users-using-erlang towards solving this issue, but how can you scale BOSH-type features like 'track' ? In the meantime, Twitter has disabled it for now.

The Boss

Ef. JBoss - I'm surprised GupShup even saw the light of day.

AI

Hey Anand, are you sure SMS GupShup has 7mill active bloggers? And that their subscribers are really pounding their web-servers with tens of thousands of concurrent feed reads? It seems to me that you're comparing apples with oranges!
(Do you understand the difference in delayed sms-push vs immediate web-pull??)

Anand Rajaraman

AI: I do understand the difference. SMSGupShup posts every message on their website *and* pushes to SMS. It's a superset of the work done by Twitter on most tweets.

AI

Anand: The point isn't on what SMS GupShup is providing. It's on the 'user' behavior. Do you get me? It doesn't matter whether or not GupShup is rss enabled. It's on how their subscribers are receiving their tweets. :-)

However, I must add, supporting such a high volume in group SMS is a pretty decent feat in itself. Kudos to SMS GupShup and their team of engineers.

I am just clarifying on the comparison you have made.

Swaroop

There has been some discussion on this among the twitterati(http://twitter.com/scorpion032/statuses/835728007). The question they ask is whether these numbers are only for SMS input or do they include the API usage?

Greg Tallent

Good read. Thank you. I've just launched a twitter-like dating service, www.TalkAboutADate.com using a LAMP stack and am looking forward to having these scaling issues. Means we're getting used. :)

Sam Freiberg

Anand I like how you completely ignore tc's point about bigger sites that don't have scaling issues. Twitter's scaling problems != Rails scaling problems.

And I can't believe nobody has corrected this before but Rails can be two tier OR three tier. In many, many deployments the first tier is the web server, the second tier is a bunch of mongrels running rails (on different servers) and the third tier is obviously the database. So your whole rant pretty well falls down. Not to mention there is no reason you can't do sharding in Rails. You could either do it through the database itself or you could do it in Rails, but either way this isn't a rails limitation any more than saying that since JDBC drivers don't have this ability automatically means Java can't scale.

If you don't want to use Rails that's fine, but don't bash something you clearly don't understand. The fact that yellow pages and hulu are running rails clearly means nothing since a little startup known as twitter has scaling issues.

Anand Rajaraman

Sam Freiberg: I'm not bashing rails at all. I love RoR, and my own startup uses it to run a very popular website.

My only point is that scalable websites should use a 3-tier rather than a 2-tier architecture. In fact, we do exactly this with RoR at my company Kosmix. So yes, i do know that Rails supports 3-tier; but that's not how most people use it out of the box.

Sam Freiberg

Anand: Fair enough. :) It seems like there's an awful lot of disinformation about scaling and Rails so I apologize for being harsh. Your post does make it sound like you can't do sharding simply because you are using a two tier architecture though which isn't the case at all. Rails might not do this out of the box but it can be done pretty easily, in many cases in the database directly.

The comments to this entry are closed.