Before starting, I will give you a brief idea about robots.txt. You know it is a search engine crawler that merely prevents the web page or content of the website according to your allowances. The website page which you will allow or disallow then the search engine bots will show or hide according to the conditions you have set in your robots.txt file. so, the prominent question is why we need it.
Suppose, you have a lot of secret information, credentials, site passwords, or account details that you want to hide from all the bot crawlers. I know it’s not possible but at certain limits, you can allow or disallow your website pages or content of the site. you will set the robots.txt file according to your own format and get the command over your own website pages which you want to hide or show limited sources of content to the visitors or users.
The other need is to if you want to protect your site from overloading issues with your requests and there is no different mechanism that your site will be out from the google search engine but what content, column, paragraph, and credentials of your website do you want to be out from the Google so, you may easily possible with the robots.txt file.
