Running Minecraft Vanilla 1.13 Server

I'm currently using shared hosting to host a vanilla 1.13.2 server with about 20 whitelisted players and on average there are 3-5 players online at the same time, sometimes 7+ players.
And on these peak times, the server will lag causing players to rubberband and breaking/placing blocks or opening inventory will be delayed. And the server console will spit out "server overloaded" warnings.
So I'm on a hunt for a good VPS that could handle vanilla minecraft 1.13. And after searching a bit, I found Linode and decided to give it a try.

So I got the Linode 4GB plan and quickly set up a vanilla Minecraft 1.13.2 server on it.
There was no server lag unless I moved around the world too quickly, but I noticed on the graphs at my Linode Mananger that the CPU usage was at 50% with just one player online and even higher with 2 players.
This was on a brand new world, so there was nothing wrong with the world. And I gave the server 2GB ram to work with, later changed it to 3GB, so I doubt not having enough ram could be a issue.

So I'm worried that if Linode 4GB can barely handle 2 players on a new world(CPU usage spiking up to 80%), then it will definitely not be able to handle a 500MB+ world with 5+ players online at the same time.

Am I worrying for nothing? Anything I could be doing wrong? Anyone else been through something similar? Should I just move on and find another VPS provider?
And also upgrading is out of question as the next tier up plan is a little too expensive for me.

By the way, it has to be vanilla and not Spigot or similar. I know Spigot has better performance, but I'm playing the game for the vanilla mechanics and Spigot messes with those mechanics.