Managing Store Anti-Crawler Settings

The Anti-crawler settings allow you to restrict crawler access for selected storefront page paths. This feature can help reduce unwanted scraping behavior and limit crawler access to specific storefront content.

This guide explains how to configure anti-crawler settings and manage crawler access behavior for storefront pages.

Understanding Store Anti-Crawler Settings

Anti-crawler settings help restrict crawler access to selected storefront page paths.

This feature is commonly used for:

• Limiting unwanted scraping behavior
• Restricting crawler access to temporary pages
• Reducing excessive crawler traffic
• Preventing selected content from being indexed

When anti-crawler protection is enabled for a page path, crawlers used by search engines may also be unable to access or index the protected content.

Adding an Anti-Crawler Protected Page

1. From your SHOPLINE admin, go to Online Store > Preferences.

2. Locate the Anti-crawler settings section, then click Add an anti-crawler page

   

3. In the Add an anti-crawler page window:

   • Enter the page path you want to restrict from crawler access
   • Only the URL path segment is required

   For example:

   • Full URL: https://yourstore.com/products/example-product
   • Entered path: products/example-product

   

4. Click Add.

The configured page path will then be added to the anti-crawler protection list.

Finding a Storefront Page Path

To find the page path:

1. Open the storefront page you want to restrict from crawler access.
2. Locate the URL in the browser address bar.
3. Copy only the path portion of the URL.
4. Paste the copied path into the Page path field.

Managing Existing Anti-Crawler Protection Rules

In the Anti-crawler settings list, you can manage existing anti-crawler rules through the Actions column.

Editing an Anti-Crawler Protection Rule

To edit an anti-crawler rule, locate the configured page path entry, then click the pencil icon in the Actions column. Update the page path as needed, then save the changes.

Removing an Anti-Crawler Protection Rule

To remove an anti-crawler rule, locate the configured page path entry, then click the trash icon in the Actions column.

Important Notes and Best Practices for Anti-Crawler Settings

• Anti-crawler settings may affect storefront SEO visibility and search engine indexing.
• Search engine crawlers may be unable to access protected page paths.
• Avoid enabling anti-crawler protection for important storefront pages unless necessary.
• Changes to crawler restriction settings may require several minutes to take effect.
• Review anti-crawler configurations carefully before enabling them on production storefronts.

Frequently Asked Questions About Anti-Crawler Settings

Q1: Why isn’t my page appearing in search engine results?

The page path may be protected by anti-crawler settings, preventing search engine crawlers from accessing or indexing the content.

Review your settings in: Online Store > Preferences > Anti-crawler settings

Q2: Does anti-crawler protection block search engines?

Possibly. Search engine crawlers may also be affected by anti-crawler protection settings.

Q3: Can customers still access anti-crawler protected pages?

Yes. Anti-crawler settings are intended to restrict crawler access rather than normal customer browsing behavior.

Q4: Why is my protected page still appearing in search results?

Search engines may take time to update indexing results after crawler restrictions are applied. Existing indexed content may also remain visible temporarily.