Cloudflare’s AI crawler rules can block Googlebot
Cloudflare is updating its method for identifying and blocking AI crawlers, which may result in Googlebot being blocked on sites that prevent AI training. The company announced the update as part of its second Content Independence Day. The new controls allow websites to manage automated traffic based on three behaviors, rather than just using a … Read more
