India's SMS GupShup Has 3x The Usage Of Twitter And No Downtime

I recently started using Twitter and have become a big fan of the service. I've been appalled by the downtime the service has endured, but sympathetic because I assumed the growth in usage is so fast that much might be excused. Then I read this TechCrunch post on the Twitter usage numbers and sympathy turned to bafflement - because I'm intimately familiar with SMS Gupshup, a startup in India that boasts usage numbers much, much higher than Twitter's, but has scaled without a glitch.

I'll let the numbers speak for themselves:

• Users: Twitter (1+ million), SMS GupShup (7 million)
• Messages per day: Twitter (3 million); SMS GupShup (10+ million)

Actually, these numbers don't even tell the whole story. India is a land of few PCs and many mobile phones. Thus, almost all GupShup messages are posted via mobile phones using SMS. And almost every GupShup message is posted simultaneously to the website and to the mobile phones of followers via SMS. That's why they have the SMS in the name of the service. Contrast with Twitter, where the majority of the posting and reading is done through the web. Twitter has said in the past that sending messages via the SMS gateway is one of their most expensive operations, so the fact that only a small fraction of their users use the SMS option makes their task a lot easier than GupShup's.

So I sat down with Beerud Sheth, co-founder of Webaroo, the company behind GupShup (the other founder Rakesh Mathur is my co-founder from a prior company, Junglee). I wanted to understand why GupShup scaled without a hitch while Twitter is having fits. Beerud tells me that GupShup runs on commodity Linux hardware and uses MySQL, the same as Twitter. But the big difference is in the architecture: right from day 1, they started with a three-tier architecture, with JBoss app servers sitting between the webservers and the database.

GupShup also uses an object architecture (called the "objectpool") which allows each task to be componentized and run separately - this helps immensely with reliability (can automatically handle machine failure) and scalability (can scale dynamically to handle increased load). The objectpool model allows each module to be run as multiple parallel instances - each of them doing a part of the work. They can be run on different machines, can be started/stopped independently, without affecting each other. So the "receiver", the "sender", and the "ad server" all run as multiple instances. As traffic scales, they can just add more hardware -- no re-architecting. If one machine fails, the instance is restarted on a different machine.

In read/write applications, the database is often the bottleneck. To avoid this problem, the GupShup database is sharded. So, the tables are broken into parts. For e.g., users A-F in one instance, G-K in another etc. The shards are periodically rebalanced as the database grows. The JBoss middle-tier contains the logic that hides this detail from the webserver tier.

I'm not familiar with the details of Twitter's architecture, beyond knowing they use Ruby on Rails with MySQL. It appears that the biggest difference between Twitter and GupShup is 3-tier versus 2-tier. RoR is fantastic for turning out applications quickly, but the way Rails works, the out-of-the-box approach leads to a two-tier architecture (webserver talking directly to database). We all learned back in the 90's that this is an unscalable model, yet it is the model for most Rails applications. No amount of caching can help a 2-tier read/write application scale. The middle-tier enables the database to be sharded, and that's what gets you the scalability. I believe Twitter has recently started using message queues as a middle-tier to accomplish the same thing, but they haven't partitioned the database yet -- which is the key step here.

I don't intend this as a knock on RoR, rather on the way it is used by default. At my company Kosmix we use an RoR frontend for a website that serves millions of page views every day; we use a 3-tier model where the bulk of the application logic resides in a middle-tier coded in C++. Three-tier is the way to go to build scalable web applications, regardless of the programming language(s) you use.

Update: VentureBeat has a follow-up guest post by me, with some more details on SMS GupShup. Also my theory on why SMS GupShup is growing faster than Twitter: Microblogging is a nice-to-have in places with high PC penetration, like the US, but a must-have in places with very low PC penetration, like India.

Disclosure: My fund Cambrian Ventures is an investor in Webaroo, the company behind SMS GupShup. But these are my opinions as a database geek, not as an investor.

Comments

[website design]

Sweet. Now GupShup just needs to take a page from Yaari and spams the crap out of the address book of every new GupShup users to convince leftover Twitter users to migrate. No, I kid. Plaxo was pulling the same crap long before Yaari. Except now they're playing as Google's [subservient chicken](http://www.subservientchicken.com/) hoping it wins them brownie points with the early adopter crowd.

[Robert Lowe]

> Twitter has said in the past that sending messages via the SMS gateway is one of their most expensive operations, so the fact that only a small fraction of their users use the SMS option makes their task a lot easier than GupShup's.

I always assumed they meant that it was expensive in terms of financial, rather than computing, resources.

[Dhananjay Nene]

Twitter numbers seem to be much higher. On peak load days they are almost 130 million (I don't recollect the source but I will post it as soon as I find it).

Having said that, I think the sharding does make sense.

[Thejesh GN]

I guess in terms of traffic twitter is lot bigger than gupshup. Dont just consider the SMS.. they get more through their api/im/web. But only way of posting to GupShup is by SMS which is expensive.

[Thejesh GN]

Ok. I agree I had not visited the GupShup recently.. now they have web/im interface. Good. Let revive my old account.

[Craig Baker]

They run JBoss appserver with C++? That's a little odd.

[John thomas]

LOL, funyn name but twitter does ROCK!

JT
http://www.ULtimate-Anonymity.com

[secretGeek]

interesting write up.

i've previously heard this "sharding" referred to as "horizontal partitioning" or "row partitioning".

[gregory]

is this article going to make techmeme? will anybody learn anything from it connected with twitter? and why have i never heard of gupshup, is it only a metro thing? vacuums everywhere

[Super Dude]

Ha proof that RoR is a passing fad. I told you so.

Die annoying script kiddies, die!

[PJ Brunet]

But a web interface is way more complex/demanding than a cell-phone interface, lots more code. And we're pounding Twitter all day long w/ our Twitkits and RSS readers, blogs auto-posting links, there's even a satellite in space hitting Twitter with updates, and then they got spam attacks to deal with, right?

Also you should explain your numbers. If I text something and it goes to 20 people does that count for GupShup as 21 messages? Because if I post something to Twitter I may have 20 followers but that probably only counts as one message.

[Ed]

Hi Anand,

I read where some companies are using RoR for the front-end but use something else for the backend.

Can you tell me how this is being done?

Are you using just the View, Controllers but not the Model/ActiveRecord?

[Rishi Agarwal]

Nice post.

As I too belong to an Indian start-up, I am really impressed by the usage numbers of Gupshup and the way they have handled scalability. Kudos for their achievements.

I have used RoR and liked it for its MVC architecture. But in the end, concerns do arise about its scalability.

[tc]

"We all learned back in the 90's that this is an unscalable model, yet it is the model for most Rails applications. No amount of caching can help a 2-tier read/write application scale."

So those Rails apps that are pushing more traffic than Twitter but seem to be scaling just fine, that's just a fluke?

Rails is 2-tier, Twitter uses Rails, Twitter has scaling issues, therefore 2-tier has scaling issues. Please ignore 2-tier sites that do not have scaling issues, as they contradict the point I'm trying to make. Uh-huh.

Besides, didn't we all learn back in the 2000's that RDBMSs aren't nearly as scalable as distributed databases like BigTable?

[Anand Rajaraman]

TC: You make 2 different points, which I'll try to address:

1. The 2-tier architecture makes the database the bottleneck for most nontrivial apps. I've seen this at many, many companies since the 1990's. That's one of the reasons app servers were invented, so you could partition the database.

2. BigTable (and it's open source cousin HyperTable) is indeed a more scalable model than an RDBMS. These pieces of software have a built-in middle-tier that abstracts the physical storage location and presents a single-system interface. So with BigTable you ARE building a 3-tier app.

[Alberto]

Interesting post, never heard of GupShup before. I will definitely try it, but I don't see references to open APIs and XMPP support.

Agree that 3-tier model is the recommended approach to build scalable web applications, but sometimes traditional MVC 2-tier architectures are enough and I believe that was on Twitter devs reasoning when they designed it as a CMS, since Twitter is used more as a messaging utility, they need to re-architect and denormalize.

For a complete comparison on Twitter's "public" estimates. It will be good to know how many connections (group subscriptions) do 7 million users in GupShup have. I assume is way more than the 4 million figure mentioned.

GupShup seems to have solved the scalability issue for Web and SMS but the real scalability challenge Twitter faces is XMPP support. Facebook has a good approach http://highscalability.com/new-facebook-chat-feature-scales-70-million-users-using-erlang towards solving this issue, but how can you scale BOSH-type features like 'track' ? In the meantime, Twitter has disabled it for now.

[The Boss]

Ef. JBoss - I'm surprised GupShup even saw the light of day.

[AI]

Hey Anand, are you sure SMS GupShup has 7mill active bloggers? And that their subscribers are really pounding their web-servers with tens of thousands of concurrent feed reads? It seems to me that you're comparing apples with oranges!
(Do you understand the difference in delayed sms-push vs immediate web-pull??)

[Anand Rajaraman]

AI: I do understand the difference. SMSGupShup posts every message on their website *and* pushes to SMS. It's a superset of the work done by Twitter on most tweets.

[AI]

Anand: The point isn't on what SMS GupShup is providing. It's on the 'user' behavior. Do you get me? It doesn't matter whether or not GupShup is rss enabled. It's on how their subscribers are receiving their tweets. :-)

However, I must add, supporting such a high volume in group SMS is a pretty decent feat in itself. Kudos to SMS GupShup and their team of engineers.

I am just clarifying on the comparison you have made.

[Swaroop]

There has been some discussion on this among the twitterati(http://twitter.com/scorpion032/statuses/835728007). The question they ask is whether these numbers are only for SMS input or do they include the API usage?

[Greg Tallent]

Good read. Thank you. I've just launched a twitter-like dating service, www.TalkAboutADate.com using a LAMP stack and am looking forward to having these scaling issues. Means we're getting used. :)

[Sam Freiberg]

Anand I like how you completely ignore tc's point about bigger sites that don't have scaling issues. Twitter's scaling problems != Rails scaling problems.

And I can't believe nobody has corrected this before but Rails can be two tier OR three tier. In many, many deployments the first tier is the web server, the second tier is a bunch of mongrels running rails (on different servers) and the third tier is obviously the database. So your whole rant pretty well falls down. Not to mention there is no reason you can't do sharding in Rails. You could either do it through the database itself or you could do it in Rails, but either way this isn't a rails limitation any more than saying that since JDBC drivers don't have this ability automatically means Java can't scale.

If you don't want to use Rails that's fine, but don't bash something you clearly don't understand. The fact that yellow pages and hulu are running rails clearly means nothing since a little startup known as twitter has scaling issues.

[Anand Rajaraman]

Sam Freiberg: I'm not bashing rails at all. I love RoR, and my own startup uses it to run a very popular website.

My only point is that scalable websites should use a 3-tier rather than a 2-tier architecture. In fact, we do exactly this with RoR at my company Kosmix. So yes, i do know that Rails supports 3-tier; but that's not how most people use it out of the box.

[Sam Freiberg]

Anand: Fair enough. :) It seems like there's an awful lot of disinformation about scaling and Rails so I apologize for being harsh. Your post does make it sound like you can't do sharding simply because you are using a two tier architecture though which isn't the case at all. Rails might not do this out of the box but it can be done pretty easily, in many cases in the database directly.