SEO#

To make your application more accessible for search engines you should provide them with the robots.txt and sitemap.xml files

Robots.txt#

Start by informing robots what they should and shouldn't traverse on your website. This helps avoid unnecessary traffic from them. Create a new file, src/robots.txt, and feed it with data:

User-agent: *

Disallow: /favorites/

Disallow: /login/

Disallow: /assets/

Disallow: /api/

Sitemap: https://my-universal-app.herokuapp.com/sitemap.xml

This configuration will prevent search engine robots from visiting paths that they have no need to traverse:

• /favorites is only available to authenticated users.

• /login doesn't contain any content that is critical from the SEO perspective.

• /assets is where static files reside (this doesn't mean that Google won't index images).

• /api is where the backend API resides.

The last line is to inform robots where they should look for the sitemap of your application.