ActiveCampaignAEMBeehiivBigCommerceBloggerBrevoClickFunnelsConcrete CMSContentfulConvertKitCraft CMSCustomDev.toDrupalExpressionEngineGetResponseGhostGravHashnodeHubSpotInstapageJoomlaKenticoKlaviyoLeadpagesMagentoMailchimpMediumOctober CMSPrestaShopSanityShopifySilverstripeSitecoreSquarespaceStrapiSubstackTYPO3UmbracoUnbounceWebflowWixWordPressWP.com

Robots.txt

5-15 minIntermediate

AEM generates robots.txt often via OSGi configuration or a dedicated asset.

Prerequisites

  • Admin Access
Advanced Recommended

OSGi Config

System level.

1

Configuration

1

Go to system/console/configMgr

2

Search for 'Day CQ Link Checker Transformer'

3

Or ensure a static robots.txt exists in the DAM and is routed via Dispatcher

Verification Checklist

  • Curl URL