Robot.txt  is a file tells search engine crawlers which pages or file the crawler can or can't request from your site. This file is used to avoid overloading your site with requests. 

Why Robot.txt Is Use 

robot.txt is used basically used to manage crawler traffic to your site and usually to keep the site and usually to keep a page off google depending on the file type.

If you want to block your page from search results you can try another method such as password protection or non-index directive.

Do-Follow Link: When we give do follow to any link means we are giving a signal to google crawler to pass the authority. If you have given do follow to any popular and prestigious site then it gives a correct signal to google about your site.

No, Follow: When you give no follow to any link it means you don't want to pass authority to that site.