Site/server is getting very slow, any help please?

I am looking for some help or guidance as our site seems to be getting slow and I am unsure why.

We did upgrade recently to a new Linode with SSD and more memory but I don't want to say its that just yet.

The waiting time or 'Time To First Byte' is getting to about 5 or 6 seconds on the first load. If I press refresh many times after then its very quick, but wait 10 seconds and try again and there is a 5 second delay again.

CPU and memory load seems low using top and free commands…

PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

20 0 39480 19m 14m S 1 1.0 2:38.77 apache2

20 0 40100 12m 8240 S 1 0.6 0:00.02 apache2

20 0 282m 8308 4184 S 0 0.4 79:50.53 fail2ban-server

20 0 2852 2376 1944 S 0 0.1 0:01.58 init

total used free shared buffers cached

Mem: 1994 1572 421 0 91 1177

Swap: 255 0 255

Any guidance on what to check next would be great, thanks