A Drupal page can work perfectly in your browser while Google sees a redirect chain, a duplicate address, or a security challenge. When Cloudflare sits in front of the site, the response comes from several layers: DNS and edge rules, the web server, Drupal routing, and page metadata.
The most useful SEO investigation follows one real URL through those layers. Start with an example from Search Console, record what each request returns, and change the component responsible for the mismatch.
Separate normal exclusions from problems
Not every URL should appear independently in search. An old address that redirects to its replacement and an alternate URL pointing to the preferred canonical can be expected exclusions. A current service page returning 404 needs a different response.
Google's Page indexing report documentation explains the individual reasons. For each example, ask whether you want that exact address indexed or whether it is an alternate route to another page. Keep a short list containing the reported URL, intended destination, response status, and Google-selected canonical.
Choose one public hostname
Decide whether the public site uses https://example.com or https://www.example.com. Either can work. The important part is consistency across redirects, page canonicals, internal links, and the sitemap.
If both hostnames return the same article with HTTP 200 and each declares itself canonical, the site sends conflicting signals. Configure a permanent redirect for normal page requests from the alternate host to the chosen HTTPS host. Preserve the path and relevant query string, and avoid unnecessary intermediate hops. Keep form submissions and other non-GET requests in the test plan so a hostname rule does not silently discard submitted data.
Use a fixed destination hostname in server rules. Do not build a redirect destination from an arbitrary user-supplied host header. Cloudflare and origin rules should agree; competing rules can create loops.
Make Drupal's canonical and sitemap agree
Open the final page's HTML source and inspect its canonical link. It should identify the intended public URL, including the correct hostname and alias. Check Metatag defaults and any per-node overrides. A staging hostname copied into an SEO field can survive deployment even when the address bar looks correct.
Inspect the XML sitemap separately. Include the preferred public URLs and remove redirects, unpublished content, and addresses that consistently fail. Use the intended production origin when generating absolute URLs from command-line jobs.
Google treats redirects and canonical annotations as canonicalization signals and sitemap inclusion as a weaker signal; a canonical is not a guarantee that Google will select that URL. See Google's duplicate URL consolidation guidance. Linking consistently to the preferred address makes the site's intent easier to interpret.
Check the response instead of guessing
Use a browser's Network panel or a GET request that saves the headers. For a hostname you control, replace the example domain in these commands:
curl -sS -D - -o /dev/null https://www.example.com/contact
curl -sS -L --max-redirs 5 -D - -o /dev/null https://www.example.com/contact
The first command exposes the initial response; the second follows the chain. Look for the expected permanent redirect and a final HTTP 200. A HEAD request is convenient but can behave differently from GET on some stacks, so confirm the visitor's actual request type.
- Test HTTP and HTTPS, the alternate hostname, and a representative deep article URL.
- Check an old URL that has a replacement and one that genuinely does not.
- Inspect both the HTML robots meta tag and any
X-Robots-Tagresponse header. - Compare anonymous requests with authenticated ones; editors may bypass caches or access restrictions.
Review Cloudflare events when crawlers cannot reach pages
A challenge or rate limit can interrupt crawling before Drupal receives the request. Check Cloudflare security events around the observed time and identify the rule or protection responsible. A 429 from your own rapid audit proves that audit was limited; it does not prove Googlebot received the same response.
Do not identify Googlebot from its user-agent string alone. Cloudflare exposes a known-bot field for rule evaluation. Review which controls apply to verified crawlers and make a narrowly scoped adjustment to the responsible rule where supported. Available controls and exceptions vary by plan and product; a custom-rule exception does not necessarily bypass every bot protection feature.
Repeated server errors and rate-limit responses can reduce crawling. Google's HTTP status guidance explains how these responses are handled. Retest at a reasonable rate after a change.
Repair missing URLs without creating soft errors
If /contact-us was an older contact address and /contact is its replacement, redirect that specific old URL. Keep a real 404 for unrelated scanner requests and genuinely missing resources. Sending every unknown path to the homepage confuses users and can create soft-404 signals.
After deployment, clear stale CDN responses for the changed URLs, regenerate the sitemap if needed, and use Search Console URL Inspection to test the preferred page. Request indexing for important corrected pages and monitor subsequent crawls. Validation is a follow-up check, not an immediate indexing promise.
For a broader review, read our Drupal SEO foundations guide or arrange ongoing Drupal support.