Improve SEO with Prerender.io

In Forbes’ recently published list of top website statistics for 2024, we learned that there are approximately 1.09 billion websites currently live on the internet (!). Out of over a billion websites, how will users (and potential customers) find your website? Most likely, they’ll start with a search engine. However, for the search engine to lead users to your site, you need to do your SEO homework.

SEO makes sure your website ranks well on search engines. Implemented correctly, SEO drives traffic to your site. However, sites built on modern JavaScript frameworks, such as React or Vue.js, rely heavily on client-side rendering, and search engines often struggle to index these types of sites correctly. So, you’re left with poor visibility in search results.

That brings us to prerendering. Prerendering involves generating a static HTML version of your web pages that a search engine can easily crawl and index. This technique can dramatically impact your site’s SEO success.

In this post, we’ll look at how prerendering services from Prerender.io can help you overcome SEO obstacles when using these frameworks. We’ll also point you in the right direction for implementing prerendering at either the web server or the application level.

Let’s jump in.

The SEO Challenge with JavaScript Frameworks

Why is SEO a challenge when it comes to using frameworks like React or Vue.js? Traditional search engine bots are designed to read static HTML. Others might handle some of the JavaScript and render pages partially, but they have their limitations. When your site is heavy on client-side rendering, a search engine will most likely not crawl and render the page all the way through, so it won’t see the complete, actual content for indexing.

As an example, Google provides the following guidance to builders of client-side rendered sites:

Dynamic rendering was a workaround and not a long-term solution for problems with JavaScript-generated content in search engines. Instead, we recommend that you use server-side rendering, static rendering, or hydration as a solution.

What’s the impact on your SEO? Naturally, partial or delayed indexing will impact your website’s search rankings. As a result, your site might experience delays in appearing in search results, which translates to missed opportunities for organic traffic. The incomplete or delayed indexing may mean a search engine needs to perform a second crawl of your site, which will introduce further delays.

Looking again at Google’s tip above, we see the recommendation to use “static rendering”—and, to some extent, that is what prerendering gets you.

How Prerender.io Solves this Problem

Prerendering essentially bridges the gap between the dynamic capabilities of modern JavaScript frameworks and the need for static HTML content that search engine bots can work with. Let’s look at how this works with Prerender.io.

Prerender.io executes the JavaScript on your pages in a headless browser, rendering the complete content. Then, it saves this rendered content as static HTML. When a search engine bot crawls your site and requests a certain page, you have Prerender.io serve the prerendered HTML instead of your JavaScript-heavy page. This way, search engines can quickly and easily access the full content of your pages for indexing—without needing to process complex JavaScript.

What are some of the benefits of using this approach?

• Improved SEO: Because Prerender.io can provide search engines with fully rendered HTML, your content will be indexed correctly and promptly. This improves your search rankings and increases site visibility.
• Faster page loads: When you use Prerender.io to serve static HTML to a search engine bot, you reduce the load on your servers. With crawlers from countless search engines constantly in operation, offloading their requests to Prerender.io can improve page load times and improve user experience for your actual users.
• Better user experience: Faster page loads and more accurate indexing will increase users’ chances of finding your site quickly and enjoying a more positive browsing experience. 

We’ve looked at the what, the how, and the why. Now, let’s get to the nuts and bolts.

Implementing Prerender.io in Your Web Application

To integrate Prerender.io into your web application, the two most straightforward approaches would be to do so at the web server level or at the application code level. Let’s consider each one in more detail.

Approach 1: At the web server

If you’re deploying a Node.js application on Linode, you’re likely using the Node.js one-click marketplace app to help you get up and running quickly. After creating a new Linode, the app will help you spin up Nginx, Node.js, NPM, and PM2.

To use Prerender.io with Nginx, you can follow the integration guide for Nginx. First, you’ll create a backup copy of your system’s nginx.conf file. You will use the ngx_http_map_module module to create some variables based on the values of other variables.

You will edit your nginx.conf file to insert sections from the example Nginx configuration file from Prerender.io. First, you’ll add the following map blocks to your http section.

map $http_user_agent $prerender_ua {         default       0;         "~*Prerender" 0;         "~*googlebot"                               1;         "~*yahoo!\ slurp"                           1;         "~*bingbot"                                 1;         "~*yandex"                                  1;         "~*baiduspider"                             1;         "~*facebookexternalhit"                     1;         "~*twitterbot"                              1;         "~*rogerbot"                                1;         "~*linkedinbot"                             1;         "~*embedly"                                 1;         "~*quora\ link\ preview"                    1;         "~*showyoubot"                              1;         "~*outbrain"                                1;         "~*pinterest\/0\."                          1;         "~*developers.google.com\/\+\/web\/snippet" 1;         "~*slackbot"                                1;         "~*vkshare"                                 1;         "~*w3c_validator"                           1;         "~*redditbot"                               1;         "~*applebot"                                1;         "~*whatsapp"                                1;         "~*flipboard"                               1;         "~*tumblr"                                  1;         "~*bitlybot"                                1;         "~*skypeuripreview"                         1;         "~*nuzzel"                                  1;         "~*discordbot"                              1;         "~*google\ page\ speed"                     1;         "~*qwantify"                                1;         "~*pinterestbot"                            1;         "~*bitrix\ link\ preview"                   1;         "~*xing-contenttabreceiver"                 1;         "~*chrome-lighthouse"                       1;         "~*telegrambot"                             1;         "~*google-inspectiontool"                   1;         "~*petalbot"                                1;     }     map $args $prerender_args {         default $prerender_ua;         "~(^|&)_escaped_fragment_=" 1;     }     map $http_x_prerender $x_prerender {         default $prerender_args;         "1"     0;     }     map $uri $prerender {         default $x_prerender;         "~*\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|woff2|svg|eot)" 0;     }

The first map block above examines the user agent of an HTTP request. If the user agent is in the list of search engine bots, then this sets off various other variables and configurations to prepare for redirection to Prerender.io.

Then, within the main location block, add the following to rewrite the URL to prepend the prerenderio/ path:

if ($prerender = 1) {   rewrite (.*) /prerenderio last; }

Finally, add the location block for the prerenderio/ path:

location /prerenderio {   if ($prerender = 0) {     return 404;   }   proxy_set_header X-Prerender-Token YOUR_TOKEN;   proxy_set_header X-Prerender-Int-Type Nginx;   proxy_hide_header Cache-Control;   add_header Cache-Control "private,max-age=600,must-revalidate";   #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs   resolver 8.8.8.8 8.8.4.4;   set $prerender_host "service.prerender.io";   proxy_pass https://$prerender_host;   rewrite .* /$scheme://$host$request_uri? Break;}

This location block tells Prerender.io to render the intended destination page and serve the static HTML. For authentication, be sure to replace YOUR_TOKEN with your actual Prerender.io account token. In the case that Prerender.io has already rendered this page in the past, it may serve a cached version (depending on how you set up caching in your Prerender.io account).

The Nginx integration guide from Prerender.io is quite comprehensive. If you happen to use Apache instead of Nginx, then you can reference the integration guide for Apache web server.

The four basic steps for integrating Prerender.io

Generally speaking, regardless of approach, integrating Prerender.io into your application involves the following steps:

1. Detect crawlers. Inspect the user agent header in requests. In the Nginx example above, we simply need to look at the $http_user_agent variable.
2. Redirect to prerender.io. Forward bot requests to Prerender.io for rendering.
3. Modify request paths. Ensure that your request paths are correctly formatted for Prerender.io, which expects the full URL as the path.
4. Manage caching. Find the balance between leveraging Prerender.io to serve cached static HTML pages (for speed) while directing Prerender.io to recache updated content on your site (for freshness). See guidance on caching from Prerender.io.

Approach 2: In your application code (Node.js Express)

If you have an application that uses a JavaScript framework (such as React, Vue.js, or Angular) for the front end, then you can configure the application server for your front end to perform the four basic steps listed above.

If you happen to use Express to serve your front end, then you can take advantage of pre-built middleware from Prerender.io, which will do all of this for you. You can see the Node integration guide from Prerender.io for more details. To break it down for you, here’s what is involved:

Install the Node.js package:

npm install prerender-node --save

Add the following code to your main index.js or app.js file where your Express server app is defined:

app.use(require('prerender-node').set('prerenderToken', 'YOUR_TOKEN'));

Again, for authentication, make sure to replace YOUR_TOKEN with your actual Prerender.io account token.

Then, restart your Express server.

That’s all there is to it!

Conclusion

Prerendering your pages can help you solve the SEO challenges when using modern JavaScript frameworks and client-side rendering. Prerender.io renders your pages ahead of time and then delivers that static HTML to search engine crawlers. This way, you can make sure that your content is fully accessible to search engine bots, resulting in accurate indexing and better search result rankings. By solving this SEO problem, your website will see improved visibility and increased organic traffic.

When you combine these prerendering techniques with Akamai’s easy-to-manage cloud computing solutions, you’ll have SEO-optimized sites up and running in no time. For more tips and guides on how to get the most out of Akamai’s cloud computing services as you deploy your applications, check out our rich library of docs.