Scalability testing of various IRC Services platforms

To contact me, bother me on IRC in various places.

The code used to generate the bursts has been committed to Atheme and will be included in Atheme 6.0.0-alpha7 and later.

Methodology

I created several burst files, a set of 10000, 100000 and 1000000 user bursts in both the legacy TS5 and basic TS6 protocols, then I used netcat to set up a listener that fed the burst into to services instance when it connected.

Data

Follows is the data for both RSS (memory) and burst processing time. The burst processing time can be used to extrapolate the number of transactions that can be processed per second by using the formula where N is the number of users and BT is burst time:

rate(Txn) = (N + 4) / BT

A higher transaction processing rate indicates higher efficiency. There is an overhead of 4 messages in the burst for the server handshake.

Unfortunately, I was unable to test Anope 1.9.2 on TS5 because it kept crashing after 10000 users.

Burst processing time, 10000 - 100000 users

RSS usage, 10000 - 100000 users

Burst processing time, 10000 - 1000000 users

RSS usage, 10000 - 100000 users

Conclusions

Shrike provides comparable performance to Atheme, but uses less memory. This is probably due to it's monolithic design and simplicity. However, due to Shrike's simplicity, it fails to scale effectively up to 1 million users. This is probably due to the use of hash tables instead of trees, which provide consistent scalability but are slower for small workloads than hash tables. Shrike 1.6 changed the hash function used and created larger hashtables which have a higher memory usage penalty due to the way that the hash tables are implemented, however the hash function used is weak and does not provide enough resolution for millions of objects.

Anope 1.9 is roughly four times slower than Atheme. Anope 1.8 is considerably slower than both Atheme 6 and Anope 1.9, but provides a lower memory footprint than both platforms.

Older services platforms with simpler designs use less resources than the modern IRC platforms.

TS6 adds additional overhead in all of the services implementations supporting it. This is expected and should not cause any alarm to anybody.

Atheme has the best scalability of all of the platforms tested, being able to handle a million-user burst in only 11 to 12 seconds across all memory allocation configurations.

Anope 1.8 has the worst scalability of all of the platforms tested. This was originally thought to be the fault of the TS6 implementation, but that has recently improved. This is probably due to the use of hash tables in combination with a weak hash function.

Anope 1.9 and Atheme have similar memory usage levels with Atheme's being slightly less on smaller workloads. Anope 1.9 has an advantage at the largest possible workload - I think this is because the STL allocator is tuned for managing millions of objects.