Finding dead links

Hi!

We use Zendesk Guide to document tutorials and common questions/problems for our customers: https://helpdesk.publiq.be/hc/nl

We sometimes link to these guides from other websites (owned by us). Some of these websites have "dead link" checks set up, so they warn us when a link does not work anymore.

Recently we've been getting a lot of warnings that our Zendesk Guide links are not working anymore. After further inspection it seems like the links do work when using a browser, but when trying to check programmatically if the URLs still work, using curl for example, we get a 403 response.

I think this is caused by some kind of bot protection from Cloudflare.

Is there a way to check if a link still works or not programmatically? Ideally without setting up some kind of API integration, because we cannot set up an API integration for every possible website that we link to. All we want to do is follow the URLs used on our websites' pages and check if they work (i.e. return a status code in the 200 range).

For now we have just disabled the link checks for any URL starting with "https://helpdesk.publiq.be" but that's of course not an ideal long-term solution.

Thanks for the help!