Performance regression with kernel 2.6.32?

@Azathoth:

I see there is paravirtualized kernel 2.6.32 available. However, I'm worried with the rather serious performance regressions found in that kernel for Postgres + ext4, by Phoronix:
From the git commit it would appear less a regression than a correction to prior behavior, which would not necessarily have guaranteed that an fsync made it all the way to the disks if barriers were enabled. So it's probably more that the prior transaction performance was artificially high, at least when running in the safest mode, which is the default on ext4 (but not ext3).

Ensuring an fsync completes all the way through to the disk can be important for data integrity. PostgreSQL (as one example) depends on this to maintain its own integrity of its WAL archive, to ensure there is no window for data loss/corruption. More importantly, without the fsync/flush guarantee you can risk a corrupted filesystem - for example in a journaled filesystem case like ext3/ext4 - where only the journal commit gets written and not the data blocks to which the commit refers.

If you were to use ext4 and decided you didn't want this, you can disable barriers when you mount the filesystem. Or, since ext3 still mounts with barriers disabled by default (from my reading, largely due to not wanting to deal with user outcry from any performance "regression"), using that works too. If you're not using ext4, this particular change has no immediate impact on you.

It's in your control though. Higher performance but increased risk of data loss, even if small. A classic trade-off. And to be fair, the risk of data loss is generally quite small, so it's a tough trade-off given that barriers are by their nature likely to slow things down, as they interfere with drives optimizing their own I/O, though to what level depends on environment and workload.

What would be interesting is to see if there's any way to find out if the host RAID controller and disks at Linode might ever rearrange writes for performance, which is one of the key problem windows that barriers help prevent. There's also the interaction with the RAID process itself.

You can't quite test the way you can on a standalone machine since it would require interrupting the host. But it would be interesting to know if barriers are adding anything in the VPS environment of a Linode. I suspect (but am not absolutely sure how the Xen disk subsystem works) that the crucial failure point is the host, since it powers the disks themselves, and not an individual Linode halting. But I don't know, for example, if the disk subsystems are battery backed, independent of the host to allow them to complete pending writes. I suspect not.

Bottom line though is that this looks like a proper change for the default mount options of ext4 (with barriers), so nothing I'd see as a reason to avoid the kernel release.

– David

Thanks for your very helpful explanation. The Phoronix article does in fact link to another that mentions barriers as the culprit, but I wasn't sure if that was by design or flaw, after all, ext4 has been in the kernel for quite a while now.

Indeed, it would be interesting to know if host RAID system rearranges the writes, and if it is battery backed.

@Azathoth:

Indeed, it would be interesting to know if host RAID system rearranges the writes, and if it is battery backed.
Yes, on both accounts.

-Chris