All the ideas or information, you will get from the below-mentioned articles where I have generated links. Now, let’s come to the prominent part of the robots.txt file so, after creating and knowing how to create robots.txt file how to deploy this txt file to the website. so, you may follow these two steps for better deployment and test of the robots.txt file:
Deploy robots.txt to the site
- Create a file name with robots.txt: First, you make a file with the aid of notepad++ to the name of robots.txt. Before making any robots.txt file, you will write a basic format of robots.txt file with proper website instructions.
- Add protocol over robots.txt file: With of help of a basic format, you will have to put some site instructions with guidance. Like, which website page do you want to allow or disallow on your website then bot crawler will come and crawl your site with proper robots.txt instructions and the crawler will move or check according to the robots.txt file.
Basic format:
User-agent: *
Allow: /wb-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ - Upload robots.txt file: For file uploading, you want to go to your site c-panel and then move to file manager. Next, In your root domain, you need to upload your txt file with the name robots.txt and after doing this, you will have to check whether your robots.txt file is saved or not.
Test robots.txt file
- Search Robots.txt checker over browser: Open your browser and robots.txt checker and go with the Google open robots.txt tester and put your all robots.txt file over robots.txt tester and check whether it’s working or not. And from there, you may allow or disallow pages also which you want to do so and see live yet.
- Search with a domain name on browser: You will search with a domain name as if you want to check your own robots.txt file then search your domain like example.com/robots.txt. After applying over the browser then you will get all the details of your site pages that you have allowed or disallowed over there with the proper set format.
