Adding RSL to robots.txt

RSL expands the limited Allow and Disallow directives of robots.txt files with more granular controls that define machine-readable licensing terms and usage restrictions.

RSL-compliant robots.txt files must include a new License directive that links to an RSL license file that defines machine-readable licensing terms for the website. The License rule is not tied to any specific user agent and must be followed by all crawlers before accessing or processing website’s content.

Usage: Add a link to an RSL license to robots.txt

License: [absoluteURL]

[absoluteURL] must point to the location of a valid RSL license or feed file. It must be a fully qualified URL, including the protocol and host, and does not have to be URL-encoded. The URL does not have to be on the same host as the robots.txt file.

Example robots.txt File

License: https://your-website.com/license.xml

User-agent: Googlebot
Allow: /