Troubleshooting 401 Unauthorized Error in Broken Link Checker

A 401 Unauthorized result in Broken Link Checker almost never means the link is broken. It means the server on the other end asked the crawler to log in, and the crawler could not. The fix depends on which side is asking for credentials: the site you linked to, your own site’s password protection, or a firewall that treats the crawler as a bot. This guide walks through how to tell the three apart and what to do about each.

We maintain Broken Link Checker on a couple of dozen client sites, and 401s come up mostly after someone turns on a staging password, a security plugin or Cloudflare bot protection. The good news is that all of those are fixable in a few minutes once you know where to look.

What a 401 actually means, and how it differs from 403 and 404

HTTP status codes in the 4xx range mean the request reached the server and the server refused it. The three you will see most often in a link report behave differently:

Code	Meaning	Typical cause in a link report	Is the link broken for visitors?
401 Unauthorized	Credentials required and not supplied (or wrong)	HTTP Basic Auth, staging password, private API, member-only page	Usually not, if the reader has the login
403 Forbidden	Server understood the request and refuses it	WAF or bot protection blocking the crawler’s user agent or IP	Usually not
404 Not Found	Nothing exists at that URL	Page deleted, slug changed, typo in the link	Yes

The practical rule: a 404 is a content problem and you fix the link. A 401 or 403 is an access problem and you fix the access (or decide the link is fine and dismiss the warning). Treating a 401 as broken and unlinking it is the most common mistake we see, and it quietly removes working links from your content.

Know which engine you are running

Broken Link Checker (the WordPress.org plugin, maintained by WPMU DEV, currently version 2.4.x with 500,000+ active installs) ships with two scanning engines, and they fail in different ways.

• Cloud Engine. Links are checked from WPMU DEV’s servers. Requests arrive at the target site from WPMU DEV’s IP addresses with the user agent WPMU DEV Broken Link Checker Spider. Results show under Broken Link Checker → Cloud in your dashboard.
• Local Engine. Links are checked from your own server using WP-Cron. Requests come from your hosting IP. Results show under Broken Link Checker → Local (older installs may also have the classic Tools → Broken Links screen).

You switch between them from the plugin’s main screen, and the plugin keeps both sets of settings so you can flip back without losing anything. Why it matters: a 401 from the Cloud Engine against your own site is often a firewall or password issue that the Local Engine would never hit, because the Local Engine is already inside. Switching engines is a quick diagnostic in itself.

Step 1: reproduce the 401 outside the plugin

Before touching settings, check whether the crawler is right. Open the flagged URL in a private browser window (so no cookies from a logged-in session interfere). If you get a login prompt or a “401” page, the target genuinely requires authentication and the plugin is reporting accurately.

For a closer match to what the crawler sees, run a request from the terminal with the plugin’s user agent:

curl -I -A "WPMU DEV Broken Link Checker Spider" https://example.com/the-flagged-page/

Compare the result with a normal browser user agent:

curl -I -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/126.0 Safari/537.36" https://example.com/the-flagged-page/

If the browser UA gets 200 and the spider UA gets 401 or 403, a firewall or bot rule is filtering by user agent. If both get 401, the page is password protected for everyone. If both get 200, the difference is the IP address, which points at an IP-based block. Each scenario has its own fix below.

Scenario A: the external site requires a login

You linked to a members-only forum thread, a client portal, a paywalled article or a private Google Doc. The link works for people who are logged in and returns 401 to everyone else, including the crawler.

There is nothing to fix on your side. Your options in the plugin are:

1. Hover the link in the report and choose Dismiss. It stays in your content and drops out of the broken list. Dismissed links are still rechecked on the next scan but no longer raise a warning.
2. Add the domain to the exclusion list so future scans skip it entirely. In the Local Engine this is under Settings → Link Checker → Which Links To Check → Exclusion list, one domain or URL fragment per line. The Cloud Engine’s exclusion support has been added more recently, so check your plugin version’s Cloud settings.
3. If the page has a public landing URL (many paywalled sites do), swap the link to that instead. Readers get a page that explains what they are clicking into rather than a raw login wall.

Do not unlink these. A link to a gated resource that your readers can access is still a good link.

Scenario B: your own site is password protected

This is the usual cause when every internal link on a site shows 401 at once. Common triggers:

• A staging or development site behind HTTP Basic Auth (the browser username and password dialog). Hosts such as WP Engine, Kinsta and Cloudways offer one-click password protection for staging, and it applies to every request, including the link checker’s.
• A “coming soon” or maintenance plugin that returns 401 or 403 to logged-out visitors.
• A membership plugin that protects the whole site rather than specific pages.
• A .htaccess rule left over from a migration.

The Local Engine has a partial workaround. Under Settings → Link Checker → Advanced you can supply a site-wide username and password that the checker sends with each request, which handles Basic Auth. The Cloud Engine cannot log in to your site, so on a password-protected staging site you should either use the Local Engine or wait until the site goes live to scan it. Scanning staging is rarely worth the effort anyway, since the links you care about are the ones in production.

If the site is live and should be public but still returns 401, test with curl from outside your network. A WordPress site returning 401 to anonymous requests is almost always a plugin or server rule, not WordPress core.

Scenario C: a firewall or bot protection is blocking the crawler

This is the scenario behind most “everything is suddenly broken” reports after a site adds Cloudflare, Wordfence, Sucuri or a host-level WAF. Bot protection sees an unfamiliar user agent making many requests in quick succession and blocks it. Depending on the rule you get a 401, 403, 429 or a JavaScript challenge page that the crawler reads as an error.

Allow the crawler in Cloudflare

1. Log in to Cloudflare and pick the domain.
2. Go to Security → WAF → Custom rules (older dashboards: Firewall Rules) and create a rule.
3. Set Field to User Agent, Operator to contains, Value to WPMU DEV Broken Link Checker.
4. Set the action to Skip and tick the protections to skip (at minimum Bot Fight Mode and the managed rulesets), or Allow on older plans.
5. Deploy, then run a new scan.

If you also use the WPMU DEV Hub to run scans, add a second rule for the user agent WPMU DEV Hub/2.0. If user-agent rules are not enough (Super Bot Fight Mode on Pro plans ignores them), add an IP allowlist under Security → WAF → Tools → IP Access Rules for the crawler IPs listed in WPMU DEV’s documentation. Those IPs change occasionally, so copy them from the docs rather than from a blog post, including this one.

Allow the crawler in Wordfence or a host firewall

In Wordfence, go to Wordfence → Firewall → Blocking and check whether the crawler’s IP or user agent appears in recent blocks. Remove the block and add the IP to Wordfence → All Options → Allowlisted IP addresses. Rate limiting lives under Wordfence → Firewall → Rate Limiting; the “requests per minute” thresholds there are what typically trip on a scan. If your host runs its own WAF (SiteGround, Kinsta and WP Engine all do), open a ticket and give them the user agent and the IPs.

Make sure robots.txt is not the culprit

The Cloud Engine respects robots.txt, including Crawl-delay. A blanket Disallow: / on a staging site, or a rule that blocks unknown agents, will make the crawler skip or fail pages. Check https://yoursite.com/robots.txt and, if needed, add:

User-agent: WPMU DEV Broken Link Checker Spider
Allow: /

Tuning the plugin to reduce false positives

Once access is sorted, a few settings reduce the chance of spurious 401s and 403s in the future.

• Timeout. In the Local Engine, Settings → Link Checker → Advanced → Timeout defaults to 30 seconds. Slow external hosts sometimes fail before answering; raising it to 60 cuts noise on sites with many outbound links to heavy pages.
• Server load limit and parallel checks. The Local Engine throttles itself based on your server’s load average. If scans stall or time out, lower the parallel request count rather than raising limits; fewer, gentler requests are less likely to trigger rate-limiting on the other end.
• Scan schedule. Both engines support scheduled scans (daily, weekly, monthly). Weekly is enough for most content sites and keeps you under third-party rate limits.
• Recheck before you act. Any link can be rechecked from its row in the report. A 401 that clears on recheck was a transient block and can be ignored.
• Email alerts. Turn on notifications so you see a sudden spike (a sign of a firewall change) rather than discovering it months later.

For WooCommerce stores specifically, expect a cluster of 401s around /my-account/, /checkout/ and any downloadable product URLs. Those are meant to require a session. Exclude /my-account and /checkout from scanning rather than chasing them each week. The same applies to LearnDash lesson URLs behind enrolment and BuddyPress private group pages; if you run a community on BuddyX or a similar theme, exclude /members/ and /groups/ paths that are private by design.

When the link really is broken

Occasionally a 401 is a real problem: a partner changed an API endpoint to require a key, a public document was made private, or a vendor’s site moved behind a login. In those cases use the plugin’s inline tools. Edit URL lets you point the link at a replacement without opening the post, and Unlink removes the anchor while keeping the text. Both actions update every post that contains the URL, which is much faster than editing posts one by one.

If you are cleaning up a large site, pair the link checker with a 404 monitor and redirect manager so that any internal URL you change gets a 301 at the same time. The broader SEO side of that work is covered in our WordPress SEO best practices guide.

FAQ

Why does Broken Link Checker show 401 for links that open fine in my browser?

Your browser carries cookies from sites you are logged into and sends a familiar user agent. The crawler has neither. Test in a private window and with curl using the spider’s user agent to see what the crawler sees.

Should I switch from the Cloud Engine to the Local Engine to avoid 401s?

Only if the 401s are coming from your own password-protected site. For external links the two engines behave the same, and the Cloud Engine is considerably faster and puts no load on your server. For most production sites we leave Cloud on and fix the firewall rule.

Can I tell the plugin to ignore all 401 results?

Not by status code. You can dismiss individual links or exclude domains and URL fragments. If a specific service always requires login, exclusion is the cleaner option.

Does a 401 on an outbound link hurt my SEO?

Not directly. Search engines do not penalise you for linking to gated content. A 404 is different: enough dead links signal a poorly maintained page. Focus cleanup effort on 404s and on internal links first.

Where to start

Run curl with the spider user agent against one flagged URL. That single test tells you whether you are dealing with genuine authentication (dismiss or exclude), your own site’s protection (use the Local Engine with credentials, or wait for launch), or a firewall (add the allow rule). Fix the category, rerun the scan, and you should see the 401s disappear together. If they persist on a site we host or maintain, our security hardening team handles the WAF side daily and can usually pin it down in one pass.

Official references: the Broken Link Checker plugin page on WordPress.org and WPMU DEV’s plugin documentation, which lists the current crawler IP addresses.