Short answer
Meta's updated Commercial Terms let it crawl the websites of businesses that use its products and use that content to train them. The scope is not limited to advertisers. Meta says this is not about scraping news sites, but robots.txt settings still need a review.
Highlights
- The terms take effect on 30 October; continued business use of Meta's products counts as acceptance.
- Meta says it is not about scraping news sites but about pulling accurate product information.
- The new terms do not mention robots.txt; Meta says content owners can still manage access with it.
2 min readAuthor: UNIT Journal EditorEditor-in-chief: Uğur Deniz İlhan
Meta's updated Commercial Terms take effect on 30 October. According to Search Engine Journal the terms give Meta permission to crawl domains and pages associated with a business's products or services with automated tools, and to use the content to develop and train its products.
What changed in the new terms?
Meta's existing ad terms already allowed it to crawl advertisers' websites. The new clause also covers businesses that do not buy ads: any commercial use, such as managing a Facebook Page or using Business Suite, is tied to the permission. Meta says businesses that disagree with the terms may stop using its products for business purposes.
What does Meta say, and what do lawyers say?
Meta spokesperson Andy Stone said the update is not about scraping news sites but about making sure accurate information about products for sale can be pulled for ad and content recommendations. Three legal professionals who spoke to Nieman Lab think the language could support an argument that Meta may scrape news sites.
Where does robots.txt fit in?
Meta says content owners can still manage access with robots.txt, but the terms themselves do not mention it. According to Meta's crawler documentation, Meta-ExternalAgent crawls for training AI models and Meta-WebIndexer improves Meta AI search results. The two can be managed separately.
What should businesses in Turkey do?
Any business using Meta for business falls under the terms, so these steps make sense as the date approaches. Teams that run Meta ads and Page management should start with management and legal:
- Open your robots.txt and write the rule you want for Meta-ExternalAgent and Meta-WebIndexer separately (for example closed to training, open to Meta AI search).
- If your site has content behind a login or a paywall, review the clause on Meta creating test credentials with your legal team.
- Know that the terms apply even if you use Meta only to manage a Page and run no ads.
- Record your decision in writing: whether you accept, restrict or stop using Meta's products for business.
Frequently asked
- Can Meta's crawling be blocked with robots.txt?
- Meta says content owners can manage access with robots.txt. However, the new terms do not mention robots.txt, and some user-initiated fetches such as Meta-ExternalFetcher may bypass it.
Sources
- Search Engine Journal ·
Follow UNIT Journal
What's new in search, AI and technology, in your feed every day.
