Low IOPS on Volumes

Question

Low IOPS on Volumes

Hello,

I am trying to host an Ethereum node for development, and I need 500GB of SSD.

The Volumes IOPS is extremely low, 600 IOPS Write, and 200 IOPS Read.

fiotest: (groupid=0, jobs=1): err= 0: pid=47634: Thu Jul 22 08:08:37 2021
  read: IOPS=605, BW=2422KiB/s (2480kB/s)(688MiB/290734msec)
   bw (  KiB/s): min= 1312, max= 5053, per=100.00%, avg=2421.80, stdev=1391.35, samples=581
   iops        : min=  328, max= 1263, avg=605.43, stdev=347.82, samples=581
  write: IOPS=201, BW=808KiB/s (827kB/s)(229MiB/290734msec); 0 zone resets
   bw (  KiB/s): min=  344, max= 2051, per=100.00%, avg=808.28, stdev=467.97, samples=581
   iops        : min=   86, max=  512, avg=202.02, stdev=117.02, samples=581
  cpu          : usr=0.61%, sys=2.39%, ctx=223296, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=176005,58727,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Meanwhile the OS Disk is higher:

fiotest: (groupid=0, jobs=1): err= 0: pid=45587: Thu Jul 22 08:02:41 2021
  read: IOPS=40.9k, BW=160MiB/s (168MB/s)(6141MiB/38438msec)
   bw (  KiB/s): min=83528, max=239616, per=100.00%, avg=163810.66, stdev=27576.03, samples=76
   iops        : min=20882, max=59904, avg=40952.71, stdev=6894.03, samples=76
  write: IOPS=13.7k, BW=53.4MiB/s (55.9MB/s)(2051MiB/38438msec); 0 zone resets
   bw (  KiB/s): min=27928, max=79440, per=100.00%, avg=54700.62, stdev=9201.17, samples=76
   iops        : min= 6982, max=19860, avg=13675.14, stdev=2300.29, samples=76
  cpu          : usr=7.44%, sys=44.25%, ctx=141481, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Is there a way to make the IOPS faster for Volumes

5 Replies

jdutton · Answer 1 · July 22, 2021, 2:23 p.m.

jdutton 3 years, 5 months ago Linode Staff

It's worth noting that Volume performance is expected to be slower than local performance due to the differences in hardware they run on (HDD VS. SSD). The quickest speeds will always reside on your OS disk, so if you end up needing a large amount of space, your best bet is to go with a higher-end plan that natively includes larger amounts of storage. Furthermore, a dedicated CPU plan (as opposed to a shared plan) would ensure that you don't run into performance contention with neighboring servers.

All that to say, those numbers do seem a bit low; you can usually expect up to 150MB/s and up to 5K IOPS on Block Storage Volumes. If you'd like us to check out the host performance internally, I'd recommend opening up a Support ticket and including the output of the following commands so we can get a better idea of your performance:

# benchmark for write speeds
dd if=/dev/zero of=blockstorage.test bs=4M count=4000

# benchmark for read speeds
dd if=blockstorage.test of=/tmp/blockstorage.test bs=4M

# general system stats
iostat 1 10

I also want to mention that upgrades to Block Storage hardware are on the way. I don't have a timeline to share, but it's something to keep an eye out for moving forward.

mohamedmansour · Answer 2 · July 25, 2021, 7:51 a.m.

mohamedmansour 3 years, 5 months ago

The system OS disk has a good IOPS, but now I am stuck in cross roads with which VM provider to choose.

All I care about is 4 CPU Cores, 8GB Ram, and 500GB NVME/SSD + 1TB HDD. With Linode, it is very limited that I am forced to go with 8 CPU Cores, and 32GB RAM to get that 500GB NVME/SSD. Which is getting more expensive to run my open source hobby.

Is there a way for me to get half the cores and half the memory with same amount of NVME/SSD?

andysh · Answer 3 · July 27, 2021, 9:50 p.m.

andysh 3 years, 4 months ago

All I care about is 4 CPU Cores, 8GB Ram, and 500GB NVME/SSD + 1TB HDD

Even with DigitalOcean, the largest “shared” VM only comes with 320GB disk space. A “storage optimised” VM with 600GB will set you back $250/month. Although their block storage does run on SSD, it’s still not as quick as the local SSD.

I’ve no idea what Linode’s bare metal will offer in terms of pricing/spec, but that may also be an option to look at when available.

I’m excited for the faster disks on the block storage, though :)

mohamedmansour · Answer 4 · July 28, 2021, 6:35 a.m.

mohamedmansour 3 years, 4 months ago

The faster block storage would be a game changer, I just upgraded Linode 16 and I think I need to upgrade again to Linode 32 just because I need a fast IOPS (NVME speed). I wish there was a way to resell my unused CPU/Memory that I won't be using.

Bare Metal might be a better choice, I was checking out OVH which has decent prices. But I wish there was a hybrid solution, Bare Metal + cloud attached drives (Block Storage).

jdjordan · Answer 5 · Jan. 10, 2022, 9:15 p.m.

jdjordan 2 years, 11 months ago

Did you ever find a solution to running an Ethereum node with the data on a block storage volume?

I have been trying to get one running using the new NVMe block storage in the Atlanta datacenter but once the blockchain sync finished all I see in the logs is State heal in progress and Unexpected trienode heal packet and it doesn't seem to be able to catch up. What I have read points to not having enough IOPS for the sync to complete. https://eth-docker.net/docs/Usage/ResourceUsage/#initial-sync-times

Atlanta NVMe block storage tests:
fio test: read: IOPS=8998 write: IOPS=3004

fiotest: (groupid=0, jobs=1): err= 0: pid=797: Mon Jan 10 20:54:06 2022
  read: IOPS=8998, BW=35.1MiB/s (36.9MB/s)(6141MiB/174716msec)
   bw (  KiB/s): min=34648, max=42688, per=100.00%, avg=36029.68, stdev=471.31, samples=349
   iops        : min= 8662, max=10672, avg=9007.41, stdev=117.84, samples=349
  write: IOPS=3004, BW=11.7MiB/s (12.3MB/s)(2051MiB/174716msec); 0 zone resets
   bw (  KiB/s): min=11280, max=14712, per=100.00%, avg=12032.21, stdev=329.04, samples=349
   iops        : min= 2820, max= 3678, avg=3008.03, stdev=82.27, samples=349
  cpu          : usr=4.17%, sys=16.80%, ctx=1496424, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

dd write speed test

dd if=/dev/zero of=blockstorage.test bs=4M count=4000
4000+0 records in
4000+0 records out
16777216000 bytes (17 GB, 16 GiB) copied, 29.8303 s, 562 MB/s

dd read speed test

dd if=blockstorage.test of=/tmp/blockstorage.test bs=4M
4000+0 records in
4000+0 records out
16777216000 bytes (17 GB, 16 GiB) copied, 68.5173 s, 245 MB/s

Compute

Storage

Networking

Databases

Services

Developer Tools

Industries

Pricing

Community

Engage With Us

Low IOPS on Volumes

5 Replies

Reply

Tips: