Amazon Aurora in sysbench benchmarks
In my previous post Amazon Aurora – Looking Deeper, I promised benchmark results on Amazon Aurora.
There are already some results available from Amazon itself: https://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf.
There are also some from Marco Tusa: http://www.tusacentral.net/joomla/index.php/mysql-blogs/175-aws-aurora-benchmarking-blast-or-splash.html.
Amazon used quite a small dataset in their benchmark: 250 tables, with 25000 rows each, which in my calculation corresponds to 4.5GB worth of data. For this datasize, Amazon used r3.8xlarge instances, which provided 32 virtual CPUs and 244GB of memory. So I can’t say their benchmark is particularly illustrative, as all the data fits nicely into the available memory.
In my benchmark, I wanted to try different datasizes, and also compare Amazon Aurora with Percona Server 5.6 in identical cloud instances.
You can find my full report there: http://benchmark-docs.readthedocs.org/en/latest/benchmarks/aurora-sysbench-201511.html
Below is a short description of the benchmark:
Data Sizes
- Initial dataset. 32 sysbench tables, 50 million (mln) rows each. It corresponds to about 400GB of data.
- Testing sizes. For this benchmark, we vary the maximum amount of rows used by sysbench: 1mln, 2.5mln, 5mln, 10mln, 25mln, 50mln.
In the chart, the results are marked in thousands of rows: 1000, 2500, 5000, 10000, 25000, 50000. In other words, “1000” corresponds to 1mln rows.
Instance Sizes
It is actually very complicated to find an equal configuration (in both performance and price aspects) to use as a comparison between Percona Server running on an EC2 instance and Amazon Aurora.
Amazon Aurora:
- db.r3.xlarge instance (4 virtual CPUS + 30GB memory)
- Monthly computing cost (1-YEAR TERM, No Upfront): $277.40
- Monthly storage cost: $0.100 per GB-month * 400 GB = $40
- Extra $0.200 per 1 million IO requests
Total cost (per month, excluding extra per IO requests): $311.40
Percona Server:
- r3.xlarge instance (4 virtual CPUS + 30GB memory)
- Monthly computing cost (1-YEAR TERM, No Upfront): $160.60
For the storage we will use 3 options:
- General purpose SSD volume (marked as “ps” in charts), 500GB size, 1500/3000 ios, cost: $0.10 per GB-month * 500 = $50
- Provisioned IOPS SSD volume (marked as “ps-io3000”), 500GB, 3000 IOP = $0.125 per GB-month * 500 + $0.065 per provisioned IOPS-month * 3000 = $62.5 + $195 = $257.5
- Provisioned IOPS SSD volume (marked as “ps-io2000”), 500GB, 2000 IOP = $0.125 per GB-month * 500 + $0.065 per provisioned IOPS-month * 2000 = $62.5 + $130 = $192.5
So corresponding total costs (per month) for used EC2 instances are: $210.60; $418.10; $353.10
Results
More graphs, including timelines, are available by the link http://benchmark-docs.readthedocs.org/en/latest/benchmarks/aurora-sysbench-201511.html
Summary results of Amazon Aurora vs Percona Server vs different datasizes:
Observations
There are few important points to highlight:
- Even in long runs (2 hours) I didn’t see a fluctuation in results. The throughput is stable.
- I actually made one run for 48 hours. There were still no fluctuations.
- For Percona Server, as expected, better storage gives better throughput. 3000 IOPS is better then Amazon Aurora, especially for IO-heavy cases.
- Amazon Aurora shows worse results with smaller datasizes. Aurora outperforms Percona Server (with general purpose SSD and provisioned SSD 2000IOPS volumes) when it comes to big datasizes.
- It appears that Amazon Aurora does not benefit from adding extra memory – the throughput does not grow much with small datasizes. I think it proves my assumption that Aurora has some kind of write-through cache, which shows better results in IO-heavy workloads.
- Provisioned IO volumes indeed give much better performance compared to general purpose volume, though they are more expensive.
- From a cost consideration (compared to provisioned IO volumes) 3000 IOPS is more cost efficient (for this particular case, but in your workload it might be different) than 2000 IOPS, in the sense that it gives more throughput per dollar.
Related
Vadim TkachenkoVadim leads Percona's development group, which produces the Percona Server, Percona Server for MongoDB, Percona XtraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.
Categories:
MySQL,
Percona Server
Peter Zaitsev
Very Interesting!
Few Observations!
– It is interesting how Amazon Aurora scales much better with workload becoming more IO Bound. You would expect that from major optimizations of IO subsystem!
– Interesting to see how with in memory workloads Amazon Aurora is slower with writes, One would expect that with essentially write-through buffer pool design.
– It would be interesting to see the read-only runs as well, with what it seems to be improved Query Cache this might be explanations for some good results posted by others
– The tests done with 16 connections which is likely to be optimal for Percona Server. It would be interesting to see some of the tests for various connection number to see how different systems scale
– It is not clear to me how HA provided is different in this case. As I understand with single Percona Server node if it is ever to crash the EBS can be used to have poor man’s active-passive fail-over. Is same applicable to Amazon Aurora as well ? It would be interesting of course to see how environments which provide HA compare – PXC ? Couple of Amazon Aurora nodes versus replication ?
Mike
Can you speak to the thread count? Are 16 threads on a t2.medium really sufficient to saturate these machines? In the AWS benchmark, they used 1000 threads spread over 4 r3.8xlarge instances.