Robots.txt

A robots.txt file is a simple text file that tells search engine crawlers which parts of your web application they are allowed to access and index.

It’s like a set of instructions that says, “Hey, search engines! Here’s what you can look at and use, and here’s what you should leave alone.”

Introduction

Elegant is configured right out of the box to generate a robots.txt for your web application automatically.

Below is a snippet of our robots.txt for this website.

# *
User-agent: *
Allow: /

# Host
Host: https://www.elegantframework.com

# Sitemaps
Sitemap: https://www.elegantframework.com/sitemap.xml

Development

During local development, a robots.txt file will not generated by your application.

By default, npm run dev will not run certain build steps as sitemap and RSS feed generation, and robots.txt creation.

To override this behavior locally, simply run npm run build and navigate to your /public directory.

Production

Once you have deployed your Elegant application to your domain, you can view your robots.txt file by navigating to /robots.txt at the root of your domain.

For example: elegantframework.com/robots.txt.