How does an SEO company handle robots.txt?
Carefully, because one wrong line can stop search engines from crawling the pages that earn your traffic. The robots.txt file sits at the root of your site and tells crawlers which paths they may request. A company uses it to keep crawlers away from addresses that waste their time, such as internal search results or endless filter combinations, checks that it never blocks pages or files that matter, and never relies on it to keep a page out of search results, which is a job for noindex.
What it controls, and what it does not
Robots.txt controls crawling, not indexing. Google’s introduction to robots.txt explains that a blocked URL can still appear in search results without its content if other pages link to it, and that the way to keep a page out of results is noindex, with crawling allowed so Google can see it. Blocking a page that carries a noindex tag defeats the tag, because the crawler never reads it. The same introduction warns against blocking images, scripts, or style files that pages need, since Google then cannot analyze those pages properly.
The file is also public: anyone can read it, so listing private sections in it advertises them; private content needs a login, not a robots.txt rule.
How Google reads the file
Some technical details change how a company works with the file. Google’s robots.txt specification states that Google generally caches the file’s contents for up to 24 hours, so a fix does not take effect the moment it is published. It enforces a size limit of 500 kibibytes and ignores anything beyond it. And if the file returns a server error, Google stops crawling the site for the first 12 hours while it keeps trying to fetch the file, then uses the last good version for up to 30 days. A robots.txt that fails to load is therefore a crawling problem for the whole site, not a minor glitch.
The mistakes a company checks for
The first is a Disallow: / line, used on staging sites to keep them out of search, copied to the live site where it blocks everything. Others are wildcard patterns that match more addresses than intended and quietly block product or service pages, rules that block the scripts and style files pages need to render, a page blocked in robots.txt that the company is also trying to remove with noindex, and rules on separate hosts, such as a subdomain, that nobody has checked because each host has its own file.
Testing and monitoring
The company tests each change before it goes live and checks it afterward. Search Console’s robots.txt report shows the robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors, and it lets you request a recrawl of the file in an emergency, for example after removing a rule that blocked the whole site. The report is available only for domain-level properties. For individual pages, URL Inspection shows whether a specific address is blocked. After any change, the company watches the Page indexing report and Crawl Stats for unexpected shifts.