Robots.txt File .

A robots.txt file is a small text file placed in the root directory of your website that gives search engine bots (like Google, Bing, and Yahoo) instructions on which pages or sections of your site they are allowed to crawl and index.

Purpose:

• Controls what search engines can and cannot access

• Prevents indexing of sensitive or duplicate content

• Helps search engines focus on your most important pages

Example:

Benefits:

• Improves SEO efficiency by guiding crawlers

• Protects private sections of your website from being indexed

• Reduces server load from unnecessary crawling