MySQL performance and (Wordpress) database size

The database of one of my Wordpress blogs can grow 20 MB in one week because of spam comments. They are all caught by Akismet, but I wonder what the influence on performance is (I have a Linode 512 LEMP)?

Basically I think it should not affect performance for visitors of my site whether the comment table (actually the commentmeta table) is 1 MB or 20 MB

Letting it accumulate does affect time to empty the spam comments, to the point of time-outs ( I let scripts run up to 60s). It does affect the size of the daily and weekly backups (especially when I had them emailed to me).

When does size begins to affect performance? Wordpress uses MyISAM.