Robots.txt is a file that instructs the search engine crawlers or robots to which page of your site you want to run crawlers or not. With a simple statement, I want to describe that which page of your website content you want to allow or disallow, it means which page you want to show or hide but according to search engine crawlers, they shall crawl the
every part of the website or domain.

You know how many credentials you have scattered over the different websites related to bank details, and secret credentials of bills & files but according to robots exclusion protocol(REP), you may allow which section, page, or content of the website you want to allow or disallow it.

Robot.txt Format:
User-agent: *
Disallow: /our-placed-students
Disallow: /cgi-bin/
Disallow: /VIET-webworkshop/
Disallow: /MenClothing
Allow: /MenClothing/tshirts