Two days back there was a post in OnSip blog describing a technique to scale stateful SIP Proxy servers. I am interested in this topic and have developed an untested design. So I wanted to study the post in detail. But alas, that post is not accessible, even though the tweets and RTs mentioning the post are still available. I have a vague recollection of the main points of the post:
In the absence of the details of the post, I thought I will document my thoughts on this topic and solicit feedback from you.
Inferred Skype Architecture
Though the full details of the architecture that Skype uses are not known, there is a general understanding on how it operates and scales. There is a central “name server” that Skype clients authenticate themselves at login. Subsequently, the end-point approaches a set of supernodes (one by one, from a list provided by the name server) that has capacity to act as a proxy to the client. It is not known how Skype ensures that the name server scales and is fault tolerant. But with the P2P architecture, Skype can add/remove supernodes at any time and they all will collectively maintain routing information among them. This allows Skype to scale. Even though this architecture allows Skpe to recover from failed supernodes, I think all the clients currently connected to a failed supernode must go through the login procedure to be assigned a new supernode.
Monkey Infers, Monkey Does
So my great idea is to replicate the inferred Skype architecture. So as a first step, we need to have a name server, but something more. Our name server will contain all the users’ information like userid and credentials. It will also contain information about all the deployed Proxy servers - IP address and set of clients associated with them. To ensure scalability and fault tolerance, the name server will be a Casandra system running on commodity servers. Additionally, it will store for each userid, information about registered clients and associated Proxy servers.
The set of Proxy servers are networked using Chord, a distributed hash table indexing. Also these Proxy servers will use Cassandra to store active session information.
When a client authenticates itself with the Name Server, the Name Server will provide the client with a set of Proxy servers. The client can register itself with one of the Proxy servers. If no Proxy server is available then it will request a new set of Proxy servers from the Name Server.
When a user initiates a session from a client, the associated Proxy server will query the Name Server for the list of clients of the called user and the associated Proxy servers. Then the originating Proxy server will use Chord to reach the Proxy servers associated with the called user.
The clients associated with a failed Proxy server will notice the failure. When that happens, the client will try to connect with another Proxy server from the list provided by the Name Server.
Analysis
It is clear that the proposed system is highly scalable. A failed proxy server can be functionally replaced. Furthermore, all the active sessions will either be brought under control or will be forcefully closed by the Proxy server with the session information stored in the Casandra.
Posted by aswath at December 16, 2010 01:24 AM
Once on the "Inter-tubes", always on the "Inter-tubes"! http://bit.ly/fZTBHn
Posted by: Adam Uzelac at December 16, 2010 09:12 AMHi Aswath,
I've used Casandra, and I think it's an excellent choice for this job.
I would add perhaps one further suggestion by extending your point #5 to say that clients could (not to say they should, just an option) connect to 2 different Proxies at once, for a call. Then you have fault tolerance within a call without a need to reconnect on the fly - you just switch to the other one. Having an essentially idle secondary connection will have close to zero load impact, but it's there in reserve. That would speed up the handover a bit.
What you describe at the signalling/nameserver end, if SIP were the protocol, is essentially a non-Stateful SIP Registrar. Kamailio and openSIPs can both do this. They can both run in proxy mode or non-proxy (P2P) mode.
We used to run VoIP User in exactly this configuration using SIP as the protocol, other than we didn't need a hashtable proxy index, as we only had 2 proxies and no requirement to scale beyond that.
We subsequently changed to a stateful SIP proxy model to assist with NAT traversal for the signalling side. No reason this couldn't be included in your Proxy model (so proxy traffics both media and signalling), which is exactly what Skype super-nodes do. Again, if SIP, FreeSwitch can be configured to behave in this way.
You can also do some interesting tricks to look for the mean best proxy/route for the 2 parties traffic and send them to the *best* proxy for their call (either an unloaded route or mean best distance between the 2).
Dean
Posted by: Dean at December 16, 2010 10:18 AMThanks Adam for the image.
Thanks Dean for your detailed comment. I considered the second proxy option and dismissed for the simple reason that the SP must deploy "double" the servers. But you are suggesting that it will not have much impact. Don't the client and standby server need to maintain the TCP connection? Are you thinking the standby server can support lot more end-points than the primary server?
In my design I want to include stateful Proxy servers. Even though I did not explicitly identify SIP to be general. But in the case of SIP, do you think my proposal does not cover stateful Proxies? I am thinking that this design uses stateful Registrar because the name server maintains the list of registered end points and the corresponding Proxy servers.
My proposal does not exclude relay nodes, but they are independent of this.
Posted by: Aswath at December 16, 2010 12:08 PMHi Aswath,
>>Thanks Dean for your detailed comment. I considered the second proxy option and dismissed for the simple reason that the SP must deploy "double" the servers. But you are suggesting that it will not have much impact. Don't the client and standby server need to maintain the TCP connection? Are you thinking the standby server can support lot more end-points than the primary server?>But in the case of SIP, do you think my proposal does not cover stateful Proxies?> I am thinking that this design uses stateful Registrar because the name server maintains the list of registered end points and the corresponding Proxy servers.<<
In which case what you describe is exactly as VoIP User is configured at the moment, and is basically a standard design for a SIP network.
Posted by: Dean at December 16, 2010 01:00 PMSorry, my message got truncated for some reason.
"But you are suggesting that it will not have much impact."
Correct.
"Don't the client and standby server need to maintain the TCP connection?"
Media is almost never sent over TCP. It is UDP, which is stateless and therefore consumes no resource at either side of the socket. There's a small amount of in-memory "AUTH" which is required, but that's minimal (negligible).
Dean
Posted by: dean at December 16, 2010 07:07 PM@Aswath: good post, thanks.
@Dean, @Aswath: I know original intent was to stay generic but for SIP specifically are you saying most SIP clients support ability to register to multiple proxies and that there are widely implemented standards-based mechanisms to govern fail-over, re-balancing after failure cleared etc?
Posted by: gzino at December 29, 2010 10:58 PM