Meta AI Crawler Floods Independent Artist's Website With Over 550,000 Requests
An independent artist's website nearly crashed after being overwhelmed by more than 550,000 requests from Meta's AI training and indexing crawlers in a short span of time. The sudden traffic spike exhausted her hosting server resources, forcing her to implement emergency server-side blocks. This incident highlights the growing financial and technical burden that aggressive AI web scrapers place on small business owners and independent creators. As tech giants aggressively scrape public data for large language models, unmonitored automated traffic can cripple vulnerable hosting infrastructures. The traffic originated from Meta's 'Meta External Agent' and 'Meta Web Indexer' crawlers, which exhaustively browse the web to train AI models and optimize search features. Standard defensive tools like robots.txt and firewalls failed to block the requests instantly, requiring the site owner to manually edit hidden configuration files with her hosting provider, SiteGround.
## BACKGROUND
AI web crawlers are automated bots deployed by tech companies to scan the internet and gather vast amounts of text and imagery for training machine learning models. Standard web protocols like robots.txt allow site administrators to specify which areas crawlers can access, though compliance relies on the crawler respecting these instructions.