How to remove your site from the Google Index
If for some reason you do not want Googlebot or other robot to index your whole website, or web page, include the following text file call robots.txt in the root directory of your website server.
To remove your site from the Google Index and tells Googlebot not to crawl your site:
User-agent: Googlebot
Disallow: /
To tell all search engine robot from crawling your site:
User-agent: *
Disallow: /
To remove certain pages on your site from Google Index:
User-agent: Googlebot
Disallow: /page.html
To exclude all pages within a directory:
User-agent: Googlebot
Disallow: /directory
To remove your site from the Google Index and tells Googlebot not to crawl your site:
User-agent: Googlebot
Disallow: /
To tell all search engine robot from crawling your site:
User-agent: *
Disallow: /
To remove certain pages on your site from Google Index:
User-agent: Googlebot
Disallow: /page.html
To exclude all pages within a directory:
User-agent: Googlebot
Disallow: /directory