AI bot traffic architecture for ecommerce: allow, authenticate, rate-limit, cache or block?

Last updated: October 27, 2026

AI bot traffic is not one category. Training crawlers collect content at scale. Search bots index pages. Live-fetch assistants retrieve information for users. Shopping agents may attempt account or purchase actions. Malicious automation targets prices, inventory and credentials.

Network data shows why merchants need explicit policies. Akamai reported that commerce produced 47.9% of AI bot traffic observed on its network from July through December 2025. This category covered broad AI bot activity, not verified demand or market-wide traffic. Fastly reported that AI requests on its platform grew about 30% from January through May 2026. That was 6.5 times the growth rate of human traffic during the same period. (akamai.com)

These figures show infrastructure pressure. They do not prove that allowing a crawler increases discovery or revenue. Merchants should test that hypothesis.

Build policy around identity, behavior and cost

The edge cannot know a bot's true intent. It can evaluate four signals: claimed identity, verified service identity, observed behavior and contractual policy.

A practical policy uses five responses:

Authenticate service identity where supported. User-Agent strings are easy to spoof. Web Bot Auth signs requests using HTTP Message Signatures from RFC 9421. As of September 1, 2026 the bot-specific protocol remains an active IETF Internet-Draft and may change. Shopify currently applies higher limits to signed bot traffic on its Storefront API and Shopify-hosted storefront pages, but it does not publish one universal quota. (datatracker.ietf.org)

A valid signature identifies the signing service. It does not prove who the customer is or what the agent may buy. Action-taking agents need separate user authentication and delegated authorization. Use narrow, short-lived scopes, consent records, transaction limits and step-up approval for sensitive changes. Add idempotency keys plus nonce and timestamp checks to limit duplicate actions and signature replay. OAuth deployments should follow current security guidance such as sender-constrained tokens where practical. (rfc-editor.org)

Rate-limit by cost as well as request count. Use token buckets for bursts and sustained quotas for longer windows. Add limits by verified identity, token, tenant, route and concurrency. Weight expensive searches or GraphQL queries more heavily than cached product requests. Return 429 with Retry-After, then escalate from throttling to challenge or block. Detect identity rotation across IPs and proxies.

Cache by route and data sensitivity. Public product and category responses can often be cached at the CDN. Include market, currency, channel and relevant inventory segments in the cache key. Define TTLs, invalidation and stale-data tolerance. Normalize query strings to prevent cache busting.

Never place accounts, carts or checkout responses in a public shared cache. Use Cache-Control: private or no-store where appropriate. Review cookies, authorization headers and Vary behavior to prevent cache poisoning or customer data leakage.

robots.txt remains a voluntary crawl directive. It is not authentication or access control. Enforce sensitive routes through the CDN, WAF, API gateway and application authorization.

Protect the origin with conditional request flows

There is no single correct processing order. A public cache hit may be served before expensive classification. Identity verification may happen first when a signed bot receives a distinct quota or cache policy.

A typical flow is:

DNS and CDN → basic abuse checks → cache lookup or signature verification → WAF and rate policy → API authorization → origin shield → application

Restrict the origin to trusted edge networks or authenticated connections. Plan for key rotation, clock skew and verification outages. A verified identity can still be compromised, so behavior controls must remain active.

Shopify merchants control less of the managed edge and should focus on documented API policies plus app-level authorization. Shopware deployments can enforce controls at their own CDN, reverse proxy and API gateway. Next.js teams should review route rendering, data-cache behavior and revalidation so one page fetch does not fan out into avoidable origin calls.

Measure value without deleting evidence

Classify machine traffic instead of removing it from analytics. Preserve raw events and add dimensions for claimed bot, verification result and human handoff. Track referral parameters plus lawful server-side conversion joins. Report attribution confidence because assistants may suppress referrers or move users across devices.

Useful operating metrics include cache-hit ratio by bot, origin requests per product fetch, p95 origin cost, signature failures, false-positive blocks and assisted conversion.

The business test is:

Contribution margin from attributable or experimentally measured AI orders, plus weighted referral value, minus infrastructure, fraud, chargebacks, support and operating costs.

Review policies regularly. Allowlisting should be an experiment backed by measurable value, not an assumption.

Bot economics belong in every ecommerce architecture audit, alongside performance optimization and agentic commerce readiness.

Webalize disclosure: This is branded technical content. Webalize's Consulting & Strategy service helps teams implement and test platform-specific edge policies.