When user-agent and allow or disallow are together, it really is regarded as being a solitary entry in a file where in fact the action will simply be employed to the certain individual representative. If you want to direct this to numerous user-agents, list an asterisk (*).
You shall then must make sure that your particular robots.txt file is conserved. Ensure that you perform some after in order for internet crawlers should be able to find and recognize your file.
- Save the robots.txt file being a text file.
- Put the file in the directory that is highest-level of site ( or perhaps in the main of this domain).
- The file needs to be called robots.txt. Continue reading