Disallow a directory in robots.txt | Community
Skip to main content

Disallow a directory in robots.txt

  • October 24, 2022
  • 3 replies
  • 0 views
F

Hi,

Is there a way to access robots.txt to be able to disallow a specific directory?

for example /cdn-cgi/

If it's not possible to disallow a specific directory, is it possible to disallow the entire support site?

note:
adding noindex won't solve my issue, as google will still crawl the pages I want to block

Thanks

    3 replies

    G
    • Gabriel32
    • October 25, 2022

    Hi Felix,
     
    The robots.txt file is a shared file used by all Zendesk accounts. It is not account-specific and cannot be edited. Currently, the only option to block a page from being indexed is to add a meta tag to your Help Center.

    <meta name="robots" content="noindex" />

    Or, to temporarily exclude a specific page from being indexed by Google, see How do I exclude a Help Center page from being indexed by Google?

    I hope this answers your question.

    J
    • JL67
    • January 11, 2024

    is there any meta tag we can add to document_head.hbs to prevent chatgpt? We're trying to set the following.

    User-agent: GPTBot
Disallow: /
    Tipene
    • Tipene
    • January 19, 2024

    Hi John,

    As mentioned by Gabriel above, the robots.txt file is not able to be edited. Unfortunately, it doesn't look like OpenAI provide any other method to prevent their GPTBot from crawling pages, at least as far as I can see in their docs. You could try using meta tags in the document head, but I'm not entirely sure if that will be effective.

    Terms & ConditionsCookie settingsAccessibility statement