External search engines (Google, Bing) are unable to index pages on a Sitefinity site. The site returns an X-Robots-Tag: noindex, nofollow HTTP response header globally on all pages, which instructs search engines not to index the content and not to follow any links. The site does not appear in search engine results despite pages being configured to allow indexing in Sitefinity.

The X-Robots-Tag HTTP response header is distinct from:

• The robots.txt file, which controls crawling behavior (whether search engines can access pages) but does not control indexing (whether pages appear in search results).
• The <meta name="robots" content="noindex, nofollow"> HTML tag, which is set per-page. In Sitefinity, this tag is controlled by the page-level setting "Allow external search engines to index this page and include in Sitemap" under Title & Properties → Advanced options. Sitefinity adds the noindex meta tag automatically when this checkbox is unchecked.
• The Sitefinity WebSecurityConfig (WebSecurityConfig.config), which manages Content-Security-Policy and other security headers but does not control the X-Robots-Tag header.

When the X-Robots-Tag: noindex, nofollow header is added globally in the web.config custom headers, it overrides any page-level indexing settings in Sitefinity, because the HTTP response header applies to every response served by IIS regardless of Sitefinity configuration.

A custom X-Robots-Tag: noindex, nofollow header has been added to the project's web.config file under the <system.webServer> → <httpProtocol> → <customHeaders> section:

<system.webServer>
  <httpProtocol>
    <customHeaders>
      <add name="x-robots-tag" value="noindex, nofollow" />
    </customHeaders>
  </httpProtocol>
</system.webServer>

This is not a default Sitefinity configuration. The header was added as a customization at the project level. It is applied globally to every HTTP response served by IIS, instructing all search engine crawlers to not index any page on the site and not follow any links.

This type of customization is commonly added during pre-launch or development phases to prevent premature indexing and needs to be removed once the site is ready for public search engine visibility.

Alternatively, the header may have been added in the Global.asax.cs file via the Application_BeginRequest method:

 HttpContext.Current.Response.AddHeader("X-Robots-Tag", "noindex, nofollow");

Step 1: Remove the X-Robots-Tag header

Locate and remove the x-robots-tag entry from the project.

Option A — web.config

Open the web.config file in the root of the project and locate the following section:

<system.webServer>
  <httpProtocol>
    <customHeaders>
      <add name="x-robots-tag" value="noindex, nofollow" />
    </customHeaders>
  </httpProtocol>
</system.webServer>

Remove the line:

<add name="x-robots-tag" value="noindex, nofollow" />

If the <customHeaders> section has no other entries after removal, the entire <httpProtocol> block can also be removed:

<!-- Remove this entire block if no other custom headers remain -->
<httpProtocol>
  <customHeaders>
  </customHeaders>
</httpProtocol>

Save the file.

Option B — Global.asax.cs

Open Global.asax.cs and look for code similar to the following in the Application_BeginRequest method:

HttpContext.Current.Response.AddHeader("X-Robots-Tag", "noindex, nofollow");

Remove the line and save the file.

Step 2: Verify the header is removed

After deployment, verify the header is no longer present:

1. Open the site in the browser.
2. Open the developer tools (F12) → Network tab.
3. Reload the page and select the main document request.
4. Inspect the Response Headers section.
5. Confirm that X-Robots-Tag is no longer listed.

Step 3: Verify Sitefinity page-level settings

Ensure individual pages are configured to allow search engine indexing:

1. In the Sitefinity backend, navigate to Pages.
2. For each page, click Actions → Title & Properties.
3. Expand Advanced options.
4. Verify that "Allow external search engines to index this page and include in Sitemap" is checked.

This checkbox is enabled by default. When unchecked, Sitefinity adds a <meta name="robots" content="noindex"> tag to the page, which independently blocks indexing for that specific page.

Step 4: Verify robots.txt

Ensure the robots.txt file in the project root allows crawling. A permissive configuration:

User-agent: *
Allow: /

If a robots.txt file does not exist, search engines are allowed to crawl all pages by default.

For more information, see: https://community.progress.com/s/article/how-to-create-a-robots-txt-file

Step 5: Regenerate the sitemap

After removing the header, regenerate the Sitefinity sitemap to provide search engines with an up-to-date list of indexable URLs:

1. In the Sitefinity backend, navigate to Administration → Sitemap.
2. Click Generate Now.
3. Once generated, verify the sitemap is accessible (e.g., ~/sitemap/sitemap-index.xml).

Step 6: Submit the sitemap to Google Search Console

1. Go to Google Search Console: https://search.google.com/search-console
2. Add and verify the site property if not already done.
3. Navigate to Sitemaps in the left menu.
4. Enter the sitemap URL (e.g., sitemap/sitemap-index.xml) and click Submit.
5. Optionally, use the URL Inspection tool to request indexing for important pages.

After completing these steps, Google typically begins recrawling and indexing within a few days to a couple of weeks, depending on the site's crawl frequency and history.

If the X-Robots-Tag: noindex, nofollow header was present for an extended period, search engines may have cached the noindex directive. Submitting the sitemap and requesting reindexing through Google Search Console will help accelerate the reindexing process.

Progress Article(s):

1. How to control what content is made available to google, bing, etc...
2. Cloud: Apply HTTP response headers