https://www.deepcrawl.com/blog/best-practice/noindex-disallow-nofollow/
https://developers.google.com/search/reference/robots_meta_tag?hl=en#using-the-x-robots-tag-http-header
In October 2019, Google announced a new policy.
You need to include noindex in the meta tags of login, admin, profile, member/home. That way, you won't be exposed to search engines.
<meta name="robots" content="noindex">
Be especially careful with private areas, private pages, and admin pages.
robot.txt google project
https://github.com/google/robotstxt
This post was edited by Lee KiWon at August 9, 2020 5:38 AM EDT