XSockets vs SignalR

Sunday, October 06, 2013 10:45 PM 8

This document contains a comparison table for XSockets vs SignalR. It also includes some metrics based on a single XSockets.NET server running on a standard laptop.

Below you find the comparison table and comments on features we think might need some explanation. This list will have many features that does not exists in SignalR, and we might have missed some features in SignalR that does not exists in XSockets. If so give us a hint and we will add those features to the list.

The source for SignalR features and functions is http://www.asp.net/signalr

Comparison table - XSockets vs. SignalR

FeatureXSocketsSignalR

WebSockets

Yes,Always

Yes, but only if you have .NET 4.5 and Windows 2012 Server or Windows 8 and later.

 

Since XSockets always has websocket support you can run the server on anything that runs Mono/.NET and TCP/IP. So the full XSockets server can run on anything from RaspberryPI to a real server.

The fact that you need not only .NET 4.5 but also 2012 server/Win8 is probably the reason for SignalR´s nice transports. They gracefully fallback to SSE, For ever frame and longpolling.

Fallback

Yes,Longpolling

Yes,
Server-Sent Events (All but IE)
Forever Frame (IE only)
Longpolling

 

XSockets only have Websockets or longpolling since there is no need for SSE when the server always deliver websockets regardless of where it runs. Longpolling using Async MVC Controllers or ASP.NET WebAPI

SignalR’s transports mechanism is nice! They have solved it in a really good way. However, our first thought was why have SSE at all when the browsers supporting SSE also support websockets? Well, since SignalR does not have websockets if you do not fulfill the server side demands (.NET 4.5, 2012 server/Win8) they had to implement SSE to not have just a longpolling framework.

Binary Support

Yes

No

 

XSockets can even combine regular data with binary to describe the binary data being sent. Any model can be attached with the binary data and then extracted at server side.

 

WebRTC

Yes

No

 

XSockets has a controller for webrtc and also a javascript API. All of these are open source on https://github.com/XSockets/WebRTC

 

Peer-To-Peer

Yes

No

 

When using a WebRTC capable browser you can also use XSockets to communicate between clients peer-to-peer sending messages that never touch the XSockets server.

 

Server side state

Yes

No

 

By having state we always know everything about each client connected. The advantage of this is that we easily can use lambda expressions to target clients. The disadvantage is of course that state cost resources on the server side.

Our opinion is that if you do not have state you do not have realtime. A message should NEVER be sent to a client that does not subscribe to it and SignalR broadcasts to all clients as long as you do not use SignalR groups.

SignalR has no state by design. This means that you can't target specific clients when sending data. SignalR has solved this by using something called groups. So that is a kind of state. As far as we know you can add connections to a group, for example “Male” and then send messages to all clients in the group… But what happens if you want to send to all clients within the range of x km from y? Or if you want to target only males between 25 - 30 yrs that has red hair… You can't do it easily.. And if you can do it at all it will be a mess!

The advantage of not having state is that you should get higher speed when sending (broadcasting). But the downside of that is that you might send data to clients not interested, or even worse.. the client is not allowed to see the data. Groups avoids this in SignalR

Publish/Subscribe

Yes

Yes, with groups… Kind of pub/sub

 

The server will know what each and every client subscribes to. You can easily target a subset of subscribers by using lambda expressions since XSockets have state

By putting clients into groups you can target a specific group with a message.

Self-Hosted

Yes

Yes (via Owin, Katana)

Azure-Hosted

Yes

Yes

WindowsService

Yes

Yes

Amazon

Yes

Yes

   

We really don't know this but we suppose that SignalR can be hosted in Amazon

Hosting in general

XSockets full server implementation can run on any device with Mono/.NET and TCP/IP. This means that pretty much anything running Linux, IOS, Windows etc can run the XSockets server with WebSockets as the transport.

The server can certainly be put into many places.. But to use websockets as transport is will need 2012 Server/Win8 and .NET 4.5

Client API (C#/VB.NET)

Yes

Yes

Client side WebSockets

Yes

No, only on Win8 with .NET 4.5

StronglyTypedModelBinding

Yes

Yes

JavaScript Proxy

No

Yes

 

We have talked about adding the possibility to generate javascript proxies to get intellisense matching the controller in xsockets. But at the moment there is no such thing.

SignalR has a nice auto generated proxy. If you go to http://yourapp:port/SignalR/Hubs

you will see the proxy generated for you which is cool

Offline Messages

Yes

No

 

XSockets uses an in-memory storage so that you can store messages that arrive when a client is offline and then push the messages to the client when he/she reconnects

 

Server side storage

Yes

No

 

XSockets has a generic static in-memory storage (Repository<TK,T>) for storing simple data-structures without using a database.

 

Plugin based architecture

Yes

No

 

Almost everything in XSockets is a plugin and you can replace most of them yourself… If you want a custom serializer build one, if you need a new protocol for communication build one… If you want to intercept the xsockets pipeline build a plugin… Build a custom controller… Its a plugin and so on..

In fact you can develop assemblies on you Mac using Xamarin and then just drop it in the server on another machine running windows and it will work.

SignalR does not have plugins but they support IoC and let you configure your container of choice

Custom Serializer

Yes

No

 

As already stated almost everything in XSockets is a plugin and you can replace most of them yourself… If you want a custom serializer build one. By default XSockets uses ServiceStack.Text, but you can replace it with your own favorite by just implementing the interface!

SignalR have a very strong dependency to JSON.NET. David Fowlers (Microsoft/SignalR) own words is "I wouldn't bother trying. We have a deep dependency on JSON.NET and we've even removed this extensibility in the next release. Sorry." in this post on StackOverflow

Scaling

Yes

Yes

 

XSockets-servers can live in a cluster where all other servers are siblings and this will boost the performance and scale in a great way.

SignalR can use Azure, Redis, SqlServer to scale out… http://www.asp.net/signalr/overview/performance-and-scaling/scaleout-in-signalr

Reduce message size

Yes, if you use the default serializer

Yes

  If you use the serializer shipped with XSockets you can use the DataMember attribute on properties to get shorter names in serialization.

SignalR lets you set JsonProperty attributes on properties so that the size of the message will be smaller. On the client you can the remap the short message with a contract. Pretty nice when hunting performance!

Passing state from client to server

Yes

Yes

 

XSockets has state during the connections lifetime and XSockets knows everything about each client. If you want to change a property on the clients connection just do it… XSockets has state!


The big difference here is that with XSockets you send the state once and then the server knows about it… In signalr you send the state-object every time you do a server side call...

In SignalR you can use the state object to attach data on every call to the server.

Text below copied from ASP.NET:
Note: This mechanism for persisting state is not intended for large amounts of data, since everything you put in the state or Clients.Caller property is round-tripped with every method invocation. It's useful for smaller items such as user names or counters.

Longrunning Hub/Controller

Yes

No?

 

You can create longrunning controllers in XSockets. These controllers is singleton plugins that can't be connected to from any client. But the longrunning controller can perform tasks and notify “real” controllers when something happens

We have no idea if there is a similar thing in SignalR?

Intercepting the pipeline

Yes

Yes

SSL/TLS

Yes

Yes

Controls frames

Yes No
  The WebSocket protocol specification defines Ping and Pong frames that can be used for keep-alive, heart-beats, network status probing, latency instrumentation, and so forth. XSockets.NET contains server side API's that lets you constructs and pass such frames from the server to the client. We haven't found any information of such possibilities in SignalR

Cross controller/hub communicaton

Yes ?

Custom protocols

Yes No
 

This is one of the most important things in a real-time framework. If the browsers talks RFC6455 (websockets) and you have a TCP/IP enabled device/unit these should ofcourse be able to talk to each other even though they speak different languages (protocols). In XSockets you can within minutes create a custom protocol and connect anything to talk to anything.
A very simple example of doing so: [Talking Between Putty and Browsers]

We have also added a proof of concept showing how easy it is to add Micro Controllers (such as NetDuino, Arduino) to communicate with XSockets.
Sample API for NetDuino on GitHub


 

 

Conclusion

We found that the biggest and most clear difference frameworks their ability to transport information on different network protocols as well as how and when the data is passed to a client. We all know that AJAX can mean that we have thousands of clients hammering a server asking for information when nothing has happen.

If a framework broadcasts data to all clients connected we have just inverted the AJAX issue and now we have a server hammering clients with information they do not want.

At a first glance XSockets and SignalR look alike, but the differences are huge!

The most important diference is that XSockets can talk cross protocol and you can add custom protocols very easy. This enables communication between all devices that supports TCP/IP.

Metrics

Since XSockets knows who subscribes a call on the “SendToAlll” method will actually verify that each client has a valid subscription on the specific topic. This can be summarised as follows.

We do not broadcast, we send to the clients that are subscribers.

Since we in the tests are using “SendToAll” and all clients in the tests are subscribers you will see the relationship 1 message send will give 10 messages out if 10 clients are connected. This relationship will also be obvious in the graphs.

We use randomized intervals in the tests since this is what reality looks like.

 

Test Scenario 1

The table below shows more of a performance test of XSockets.NET. You can see how the server behaves under heavy load under a short period of time (60 - 240 seconds). The tests with 200 & 300 clients show that we can send more than 40.000 messages per second from the server. If you need to send more than 40.000 messages per second for a longer period of time you will have to wait for our cluster to be released.

Subscribers

Publishers

Publish/sec

AvgOut/Sec

MaxOut/Sec

Duration

10

10

2 000

1 756

2 006

240s

100

10

20 000

19 755

33 371

240s

200

10

40 000

38 675

62 580

240s

300

10

60 000

34 606

65 907

240s

300

30

180 000

42 858

76 758

60s

200

60

240 000

40 831

59 909

60s


Each message (C# object) published is serialized as JSON . XSockets.NET ensures that the clients that gets the object is a REAL subscriber (no broadcasting) to the specific topic.

Test scenario 2

A specific number of clients will during a specified time be connected to the server, listening to one topic (sub). A small subset of the connected clients will during this time publish (pub) on this topic with a random interval. The inbound (I) message will be sent (O) to all subscribers (sub). Average msg/sec is calculated by the Time/O, but it is not that interesting with msg/sec considering that the actual load appears as peaks when we send. This is visible in the diagrams Y-axis as big jumps in messages count.

 

#

Time

Interval

Sub

Pub

I

O

Avg msg/sec

Red

120s

10,2s

1000

1

130

118 800

990

Black

120s

12,3s

3000

1

110

285 000

2376

Green

120s

9,1s

1000

5

300

271 700

2264

Blue

120s

10,7s

3000

5

240

524 000

4367

 


Break down of the metrics

If we break down and look at the result of each interval within the given timeframe (time) we can see that depending on the randomized interval we got between 8-12 pub/sub (activities) to occur ( See graph below). The graph below shows you the number of outbound messages delivered from the server to the client and confirmed.

Activity

Blue

Black

Green

Red

1

39949

29707

24820

9917

2

34332

24998

24915

9895

3

80409

31097

24553

9885

4

74801

24503

22919

9916

5

92705

30185

24851

9970

6

26778

30775

25774

9908

7

67753

26090

24555

9821

8

47608

30630

24967

10023

9

59204

28949

24925

9862

10

-

28220

24823

9972

11

-

-

24668

9468

12

-

-

-

10163

Especially notice interval nr 5 on the blue test that pushed over 92.000 messages within a second.

The graph below gives you a view over the table above.

Test Scenario 3

This scenario will connect a specific number of clients (clients ), during a the specified time ( Time). A large subset of clients (subset) will randomly be given the task to once send a payload (payload) to all connected clients. The randomly given time to send is set to be somewhere within the timeframe of (Time). As we spread the work-load over (time) we will get a few peeks

Clients

Subset

Payloads

I

O

Time

Min m/s

Max m/s

Avg/s

1100

1000

1

5562

5838193

595s

1042

22406

1952

500

500

2

3167

1556682

298s

239

15384

5398

2500

500

1

1779

4238751

595s

0 *

23061

7269

The graphs below shows the number of messages outbound messages sent and received ( confirmed by clients) during Time (X-Axis). The number of messages in the y-axis.



Conclusion

We have chosen to have state by design since it is our belief that real-time systems should have state. By showing you these tests we think that we have proven that state does not make it impossible to have good performance. We have work in progress that hopefully will show you that we can scale out XSockets and have linear performance over hundreds of thousands of connections.

Yours sincerely, Team XSockets.NET October 6th 2013.