Frankfurt studio for multilingual digital presence +49 69 95209894 [email protected] Mon–Fri 9 AM–5 PM Client Area →
EnglishEN

2026-07-29 · Baduno Editorial Team · 26 Min. reading time · Blog & Knowledge

Crawling and Audits for Multilingual Websites: How to Uncover Errors in 24 Markets

Crawling and audits are essential for multilingual websites. Learn how to systematically check hreflang tags, sitemaps, and language signals across up to 24 markets. Our guide provides practical methods for error detection and prioritization – from tool selection to automation.

A visualization of a crawler bot traveling through web pages.

Fundamentals of Multilingual Crawling: Why Technical Audits for 24 Markets Are Indispensable

Operators of multilingual websites targeting 24 EU markets face the challenge of reliably detecting technical errors in all language variants. Manual verification of each page is not efficient at this scale. Automated crawling makes it possible to systematically traverse all URLs and capture deviations per market. In practice, experienced teams use crawlers to analyze hreflang attributes, sitemaps, and language signals in parallel. This allows identifying issues such as missing backlinks, incorrect language tags, or broken internal links before they negatively impact indexing.

The benefit is clear: a crawler reliably checks whether each language version correctly references its alternatives. For example, a German page targeting a Swiss audience must also point to the Swiss version. If this reference is missing, users from Switzerland may see the wrong language variant. The same applies to sitemaps: if each market has its own sitemap, it must contain all relevant URLs. A crawler can automatically verify the sitemap structure and report missing subpages. It also uncovers unnecessary redirects or unreachable resources that affect loading times.

A crawler can also simulate different Accept-Language headers to test whether the website correctly redirects to the preferred language. This allows you to detect misconfigurations in the content negotiation process. Additionally, it can check whether each page has a correct hreflang tag and whether the language designation in the HTML lang attribute matches the actual language. If such signals are not consistently set, you risk search engines delivering the wrong language version—a risk that can be minimized with regular audits.

Integrating regular crawling into the workflow reduces the risk of technical errors going unnoticed for long periods. In practice, a monthly or per-release audit has proven effective. Ensure the crawler respects search engine crawl guidelines to avoid negative consequences. Note: the legal framework for crawling your own websites varies by country. We therefore recommend coordinating implementation with a specialized legal advisor. A well-thought-out crawling concept is the foundation for consistent technical quality across all markets.

Common Sources of Error in hreflang, Sitemaps, and Language Signals

When reviewing multilingual websites, the same error sources keep recurring. Among the most frequent hreflang errors are missing alternative references, incorrect language codes (e.g., 'de' instead of 'de-DE'), and inconsistent backlinks between language variants. In practice, we often observe that only one direction is maintained: a French page links to the German one, but the German page forgets the backlink. Equally problematic are pages that reference themselves with hreflang without specifying alternatives. This leads to incomplete signaling to search engines and can impair market indexing.

Specific errors also occur with sitemaps. Some projects consolidate all language versions into a single sitemap, reducing crawling efficiency. It is optimal to create a separate sitemap per market and reference it correctly in robots.txt. A common mistake is omitting certain subpages from the sitemap, leaving them undiscovered by search engines. Additionally, sitemaps should include the lastmod date to signal freshness. A crawler can automatically detect such gaps by comparing the sitemap with the actual page structure.

Language signals such as the HTML lang attribute, hreflang, Content-Language header, and visible text language must be consistent. A typical error source is a contradiction between the HTML lang attribute and the hreflang declaration. For example, a page might have lang="de" but include hreflang="en". Search engines interpret such signals as uncertain. Furthermore, you should verify that each language version is actually written in the declared language. Mixed-language text (e.g., German navigation on what is supposed to be English content) confuses both users and search engines. Regular crawls help uncover these inconsistencies.

To systematically identify these errors, it is advisable to create a checklist with all criteria to be checked. Crawling tools offer filtering functions that allow you to list all pages without correct hreflang tags, for example. Also pay attention to subdomain handling: if you use a separate subdomain for each market (e.g., de.example.com, fr.example.com), hreflang must be correctly set across domain boundaries. With a well-configured crawler, you can check all these aspects in a single pass, significantly reducing maintenance effort.

A screen displays the XML structure of a sitemap file.

Selecting the Right Crawling Tool for Your Requirements

The choice of the right crawling tool depends largely on your project scope, budget, and your team's technical expertise. First, check how many URLs the website comprises in total and how many crawls per month are needed. For a website with 24 markets, several hundred thousand URLs quickly add up. Tools designed for large data volumes offer advantages here. Ensure the crawler supports your specific configurations, such as adjusting the user agent, accept-language headers, or cookie settings. Only then can you simulate realistic scenarios from each market.

Another important criterion is support for multilingual structures. The tool should be able to parse hreflang tags and check for consistency. Ideally, it offers predefined checks for common errors or the ability to define custom rules via regular expressions. The export of results is also crucial: you need clear reports that you can share with your team—whether as CSV, Excel, or via an API. In practice, it has proven beneficial to choose a tool that can be used both desktop-based and cloud-based to flexibly respond to different use cases.

The scalability of the tool plays a central role. A desktop tool may suffice for smaller projects but hits its limits with millions of URLs. Cloud-based solutions distribute the load across multiple servers and significantly speed up the crawling process. Also consider the runtime: a complete crawl of all 24 markets can take several hours or days, depending on size. Therefore, plan sufficient time or use incremental crawls that only check changed pages. Ultimately, weigh the costs against the benefits: a more expensive tool often offers deeper analysis functions, while a cheaper one may still meet your requirements.

Before making a final decision, we recommend using a trial version of the candidate tools. Check whether the user interface is intuitive and whether support responds quickly to questions. Also pay attention to compliance with data protection regulations: the crawler should not collect or store personal data externally unless you have legally compliantly arranged this. Note: The legal permissibility of crawling may vary by country; consult legal counsel if in doubt. With the right tool, you create a reliable basis for the continuous quality assurance of your multilingual website.

Preparation: Define sitemaps, language variants, and test URLs

Before you start automated crawling, you need to create a solid test basis. First, define all relevant language variants of your website. List all countries and languages you want to cover—for the EU, these are 24 official languages. For each variant, note the correct URL structure, for example domain.de, domain.at, or domain.com/de/. Then create a representative list of test URLs that covers all language versions and important page types (homepage, product pages, category pages, legal pages). Select at least five to ten pages per language variant, ideally with different hreflang configurations.

At the same time, you must review and, if necessary, clean up the XML sitemaps. Each language variant should have its own sitemap or clearly separated entries in a common sitemap. Ensure that the sitemaps only point to the official URLs and contain no redirects. Export the sitemaps as a reference so you can later match the crawler results with the expected entries. Avoid including URLs from other language versions in the wrong sitemap—a common mistake that leads to inconsistent signals.

Also determine your crawling parameters: Which tools will you use? Set the maximum crawl depth, user agent settings, and speed limits to avoid overloading the server. Note the expected hreflang values for each test URL in a table. This preparation will save you extensive rework later. In practice, a systematic test definition significantly increases the error detection rate because you are not crawling blindly but can specifically search for deviations.

Also consider the legal framework: When testing within the EU, you must comply with the General Data Protection Regulation. Do not use any personal data in your test URLs, and ensure that your crawling activities do not trigger unwanted access. If in doubt, consult a legal advisor to ensure your audits comply with applicable regulations.

Automated checking of hreflang tags for correctness and consistency

After preparation, start the automated crawling with a focus on hreflang tags. Modern crawling tools can analyze a website's hreflang implementation and identify typical errors such as missing tags, incorrect language codes, or inconsistent links. Configure your tool to read the hreflang elements in the source code or HTTP header for each crawled page. Pay attention to the following test criteria:

Check whether each language variant is assigned a valid language code. Use the ISO 639-1 form (e.g., de, fr, es) and for country variants, the underscore (e.g., en-GB, de-AT). Ensure that the hreflang values are consistent – if page A links to B, B must link back to A (bidirectional consistency). Have missing backlinks or incorrect codes reported as errors. In practice, problems often arise with the use of x-default: this value should only be used for pages without specific language targeting, not as a placeholder for non-existent translations.

After crawling, create an overview of all detected hreflang sets. Each set should contain all language versions of a logical page. If individual variants are missing or duplicate entries exist, mark them as errors. Example: A product page exists in German and French, but the hreflang set only references the German page – then the French entry is missing. Also check the consistency of URLs within the sets: For different paths (e.g., /de/produkt vs. /produkt?lang=de), all variants must be correctly specified.

Document all deviations found and prioritize corrections. In multilingual projects with 24 markets, it is recommended to group errors by language variant and crawl again at regular intervals. Automate this process by comparing the crawling results with your expected hreflang matrix. This ensures that the hreflang tags remain correct in the long term – especially after content updates or page restructurings.

Analysis of XML Sitemaps: Coverage and Correct Language Assignment

XML sitemaps form the backbone of your multilingual website structure. After the hreflang check, turn to the analysis of the sitemaps. Crawl the sitemap files specifically and check whether all language variants are fully covered. A common mistake is that new translations are not included in the sitemap or that outdated pages are still listed. Therefore, check the number of entries per language variant: Expect a similar number of pages for each language (provided your content is consistently translated). Large deviations indicate missing or superfluous entries.

Ensure that the URL specifications in the sitemap match the actual language assignment. Each URL should have a clear language context – either through the domain, directory, or file name. Crawl all sitemap URLs and check whether they point to the correct language version. Use your hreflang knowledge from the previous step: The URLs listed in the sitemap must be consistent with the hreflang tags on the page itself. If the sitemap contains a German URL but the page has no hreflang tag for German, that is a contradiction.

Also check the sitemap index files (if present). Often a parent sitemap is used that links to individual language sitemaps. Ensure that each sub-sitemap is correctly referenced and contains no broken links. A tool like Screaming Frog or Sitebulb can automate this analysis and provide you with a list of all sitemap entries with status codes. Watch out for 404 errors or redirects – these should not occur in the sitemap, as they send unnecessary signals to search engines.

Document all deviations and create an action plan. It is recommended to include the sitemap check in your regular monitoring – ideally after every major content update. This keeps the sitemaps clean and ensures that all 24 markets can be fully indexed. Remember: Legal data protection requirements also apply here; do not use personal data in the sitemaps.

A dashboard showing clear audit results and error indicators.

Detection and Correction of Broken Links and Redirect Errors

Broken links and faulty redirects are common stumbling blocks in multilingual websites. A single broken link in one language version can cost trust and interrupt user flow. Additionally, redirect chains or 404 errors signal to search engines that the site is not optimally maintained – which can affect visibility.

To systematically uncover these errors, rely on automated crawlers that go through all 24 language variants. Tools like Screaming Frog or Sitebulb allow you to configure a crawl with the start URLs of all language versions. Make sure the crawler follows alternative language URLs (e.g., via hreflang) to get a complete picture. Then filter the results by status code: 4xx and 5xx errors, as well as 3xx redirects that do not point to the final target URL.

A proven approach is to create a list of all URLs from the sitemaps of all languages. Have the crawler process this list and log every failed request. Note that redirects are not automatically negative: a temporary redirect (302) during maintenance is acceptable, but permanent ones (301) should only lead to the correct target URL in the same language. In particular, check whether language versions redirect to the wrong language – for example, from /de/ to /en/. That confuses users and search engines.

To fix these issues, proceed systematically: correct broken internal links directly in the CMS by updating the target URL. For external links that are no longer reachable, decide whether to remove them or replace them with an alternative. For redirects, shorten chains to a maximum of one step and ensure language consistency. Schedule regular audits – at least quarterly – because new broken links can appear with every content update. This keeps your multilingual website technically clean and user-friendly.

Checking Meta Tags, Title Tags, and Language Declarations

Meta tags, title tags, and language declarations form the backbone of search-engine-friendly communication of your content. In a multilingual setup, these elements must not only be correct per language version but also consistent across all 24 markets. Errors such as missing or incorrect language specifications in the HTML 'lang' attribute or inconsistent title tags can impair indexing and user understanding.

Use your crawler to extract all relevant meta data. Create a table with the columns: URL, language version, title tag, meta description, HTML lang attribute, and optionally Open Graph tags. Then filter for anomalies: empty title tags or those shorter than 30 characters should be revised. Ensure that title tags use the respective local language and do not contain an English title for the German page. Meta descriptions should also be written in the target language and accurately summarize the content.

The language declaration in the HTML element (<html lang="de">) must match the actual language used. A common mistake is that the lang attribute is set to 'en' even though the content is in French. Also check the 'xml:lang' attribute for XHTML pages. Use a crawler that reads these attributes and cross-reference them with the language version from your sitemap or URL structure. If you use hreflang tags, these must also harmonize with the lang attribute.

To ensure long-term consistency, establish clear editorial guidelines: each language version receives its own translated meta tags, never machine translations without post-editing. Perform an automatic check with each release of new content or a translation – for example, using a CI tool that compares the crawler's output with your specifications. This prevents errors from creeping in and ensures that all 24 markets are equipped with correct, search-engine-optimized meta information.

Content Duplicates and Canonical URLs in Multilingual Setups

In multilingual websites, duplicates often arise not from malicious intent but from technical circumstances: identical product descriptions in different countries, similar landing pages, or missing canonical URLs. Search engines view duplicate content critically because they cannot determine which version is relevant. This can dilute rankings — particularly frustrating when you are present in 24 markets.

A crawling tool helps you systematically identify duplicates. Configure the crawler to capture the content (e.g., the text body) of each page and compare it using a fingerprint (hash). Pages with identical content are flagged — regardless of language. Keep in mind: true duplicates exist when the content appears multiple times in the same language. Content translated into different languages is not considered duplicate. However, it can happen that an English page for the US market and one for the UK market are largely identical — in that case, you should decide whether to set one version as canonical or differentiate the content.

For language versions that overlap (e.g., German in Germany, Austria, and Switzerland), it is advisable to use canonical URLs strategically. If you deliver exactly identical content, set a canonical URL to the preferred variant. Otherwise, use hreflang to indicate alternatives — but ensure both signals harmonize. A common mistake is that hreflang points to a page that specifies a different page as canonical. This creates contradictions.

To resolve duplicates, approach each case individually: For pages with closely related content, differentiate them through language-specific adjustments (e.g., local units of measurement, cultural references). If adaptation is not sensible, consolidate the versions and redirect the others via 301. Set a self-referencing canonical link for each language version — unless you have an explicit reason otherwise. Document your decisions and re-check after every major update to see if new duplicates have emerged. This keeps your multilingual website clean and search-engine friendly.

Crawling and audits are essential for multilingual websites. Learn how to systematically check hreflang tags, sitemaps, and language signals across up to 24 markets. Our guide provides practical methods for error detection and prioritization – from tool selection to automation.

Performance and Load Time Measurement for Every Language Version

Website load time directly impacts user experience and search engine rankings. For multilingual websites, you must conduct separate measurements for each language version, as server locations, CDN configurations, and the size of localized resources vary. Use tools such as Google PageSpeed Insights, Lighthouse, or GTmetrix to capture load time, First Contentful Paint (FCP), and Largest Contentful Paint (LCP) for each URL. Ideally, perform tests from geographically distributed locations to reflect realistic load times — WebPageTest offers multiple test locations for this purpose.

Create a list of all language variants of your homepage and key subpages (e.g., product or category pages). Measure each URL multiple times, preferably at different times of day, and record the average values. Pay special attention to the LCP threshold of 2.5 seconds; beyond 4 seconds, bounce rates typically increase significantly. Also check whether language resources such as fonts or translation files are loaded asynchronously and whether compression (Brotli or Gzip) is enabled.

A common mistake: Language versions served on a different server or via a different CDN configuration show varying load times. Note the values per language variant and compare them. If a particular language version is noticeably slow, check the server location, caching settings, and number of HTTP requests. Optimize images and scripts for the respective language, as localized content (e.g., different image formats or longer texts) can affect load time.

Recommendation: Set up regular monitoring that automatically measures load times for all language versions. Tools like Sitebulb or Screaming Frog can also incorporate performance metrics into the crawl using appropriate scripts. Establish thresholds that trigger manual review. This ensures your multilingual website provides a consistently fast user experience across all markets.

A laptop with SEO tools open and a small globe icon.

Documentation of Results and Error Logging

After crawling and performance measurements, you must document the results in a structured way to track and prioritize errors. Create a central error log, ideally in a spreadsheet (e.g., Google Sheets or Excel) or a ticketing system. For each error, record the affected URL, language version, date, error type (e.g., incorrect hreflang, broken link, slow load time), and status (open, in progress, fixed). Include screenshots or log excerpts so developers can quickly reproduce the error.

Document not only individual errors but also patterns: Are there frequent issues in a specific language version? Which pages (homepage, product pages, blog) show the most errors? Categorizing by error type (technical, content, configuration) facilitates later prioritization. Use consistent naming for logging – e.g., "hreflang target language incorrect" or "meta title missing". Link errors to the corresponding test URLs and, if available, to IDs from your crawling tool.

A proven approach is to create a weekly or monthly audit report showing the trend in error counts. This helps you see if your optimizations are working. Use the export functions of crawling tools like Screaming Frog or Sitebulb, which provide CSV files with all errors found. Combine these with performance measurements into a comprehensive report. Ensure results are sorted by market (language version) so you can quickly identify which countries are most affected.

Recommendation: Implement a clear label indicating whether an error can be fixed automatically (by tool) or manually (by editor). Include brief solution descriptions directly in the log. Schedule regular review meetings where the team discusses open items. Proper documentation is the foundation for efficient error resolution and prevents issues from being processed multiple times.

Prioritizing Error Fixes by Market Importance and Impact

Not every error has the same impact on your multilingual website. You must prioritize error fixes based on the importance of the market and the potential impact on user experience. First, define the market importance of your 24 EU language versions: Higher-revenue countries or strategically important markets receive higher priority. Create a ranking of languages by traffic, conversions, or revenue. Errors in these markets should be fixed faster than in smaller or less revenue-generating variants.

Assess the impact of an error: Does it prevent search engines from indexing the page (e.g., incorrect hreflang or faulty sitemap)? Does it lead to poor user experience (e.g., broken link, very slow load time)? Or does it affect content quality (e.g., missing title tag)? Errors with high impact on discoverability (crawl budget, indexing) should be fixed immediately, as well as those with a direct negative effect on conversion-relevant pages like checkout.

Use a simple matrix to prioritize errors: X-axis = market importance (low to high), Y-axis = error impact (low to high). Errors in the "high/high" quadrant have the highest urgency. Practical approach: Sort your error log by these two criteria and assign a priority level (1 = immediate, 2 = next week, 3 = next month) to each error. Discuss the prioritization with the team to ensure everyone applies the same weighting.

Recommendation: For each error, note the expected effort (in hours) and weigh effort against benefit. Errors that are quick to fix and have a large impact should be done first. For extensive technical issues (e.g., incorrect hreflang setup for all languages), create a project plan with milestones. After fixing, verify results with a recrawl. Consistent prioritization ensures your resources are used optimally and the most important markets benefit first.

Regular Crawling Routines: Intervals and Automation Options

A one-time audit is not enough to keep 24 language versions permanently error-free. Content changes, new pages are added, and technical configurations can be inadvertently altered. Therefore, it is advisable to establish recurring crawling routines. The intervals depend on your website's update frequency and market dynamics. For static pages with rare changes, a monthly crawl may suffice. For daily new content, such as in online shops or news portals, a weekly or even daily run makes sense.

For automation, crawling tools like Screaming Frog, Sitebulb, or DeepCrawl offer APIs and CLI interfaces. You can trigger the crawl via a cron job on your server or through CI/CD pipelines. A practical setup: export the crawl configuration as a project file, create a shell script that calls the tool, and integrate it into your scheduler. Ensure that the output—ideally as a CSV or JSON report—is automatically transferred to a central dashboard or an issue tracking system like Jira. This keeps all stakeholders informed without manual effort.

A central point: adjust the crawl settings for multilingual audits. Each language crawl should only scan the corresponding URLs to reduce runtime. For tools that crawl the entire domain, filter by path or subdirectory. Use regular expressions to exclude irrelevant areas (e.g., '/en/', '/fr/', etc.). If your website serves language variants via subdomains, you need to configure separate crawls for each subdomain and later merge the results. This requires some initial work but prevents you from listing pages in the wrong language.

Regularly check whether your crawling tool correctly interprets current hreflang rules. For this, a monthly comparison of hreflang references with your sitemap is recommended. Also automate validation: a script can check whether each language version contains a backlink in the opposite direction. This avoids inconsistencies. Document your routine in an internal wiki so that colleagues can understand what to do in case of failures. In practice, it has proven useful to perform a complete manual audit once per quarter and compare the results with automatically generated reports—this eliminates time delays.

Legal notice: The intervals and automation options described here are merely general guidance. The specific design should always be coordinated with your legal department, especially if personal data is processed during crawling.

Checklist for Completion: Comprehensive Audit Report and Next Steps

A thorough audit report summarizes all results in a clear manner and serves as a basis for prioritizing corrections. The following checklist helps you ensure nothing is overlooked:

• All 24 language versions have been fully crawled—including all subpages listed in the sitemap. • hreflang tags are present on every page and consistently reference all language variants (including x-default). • The XML sitemaps contain all relevant URLs, are correctly linked per language, and are indexed by search engines. • No page returns a 404 error or leads through a redirect chain—especially after language switching. • Meta tags (title, description) and language declarations (lang attribute) are consistent. • No significant content duplicates between language versions—canonical URLs are properly set. • Load times are under 2 seconds for each language version (measured with crawling tool or external services like PageSpeed Insights). • All errors are categorized by severity: critical (faulty hreflang, 404s), medium (missing titles, redirects), and low (cosmetic meta errors).

After the audit, create a central issue tracking document—for example, as a shared spreadsheet (Google Sheets, Airtable) and assign each error to a responsible person. Note estimated effort and deadline. Example: "hreflang circular reference on /de/produkt and /en/product: Max Müller, effort 2 h, by 15.03." Link the table with your project management tool to track progress.

The next steps should be prioritized—by market relevance and technical impact. Start with errors that prevent search engines from correctly indexing your content (e.g., faulty hreflang). Then fix technical issues that affect user experience (broken links, slow pages). Lowest priority goes to metadata optimizations. After completing all corrections, schedule a new crawl to verify effectiveness. Ensure that all team members understand the results and that the next steps are clearly communicated.

Finally: Keep the audit report ready as a reference for the next quarter. Compare error rates over time to identify trends. In practice, repeated audits gradually reduce the number of errors—provided that root causes are not only superficially fixed. A good tracking system helps identify recurring issues. Remember: the report is not an end in itself but a tool for continuous improvement.

Legal notice: The prioritization suggestions do not replace legal advice. For questions regarding compliance (e.g., GDPR, imprint obligations), consult your legal department.

Pitfalls and Common Misconceptions in Multilingual Crawling Audits

Even experienced teams overlook typical error sources when auditing multilingual websites. Example: hreflang tags with x-default correctly set, but the referenced URL uses a different domain or wrong protocol (HTTP vs. HTTPS). The crawler shows no warning because the tag is syntactically correct – but the target URLs do not exist. Therefore, always verify the resolution of each hreflang URL. Another pitfall: language variants of a page reside on different subdomains, and the sitemap contains only one of them. The crawler does not find the others because no internal linking exists. Solve this by explicitly including all variants in the sitemap and ensuring each language version is linked from at least one other page. Redirect errors are also tricky: A redirect from /de/artikel to /de-seite?lang=de creates a redirect chain that destroys hreflang signals. Crawl your start URLs with redirect tracking enabled and check whether each language version is served directly. A common misconception concerns the canonical URL: When content is identical across languages, some set the same canonical URL for all variants. This contradicts the purpose of language alternatives. Each language variant should self-reference unless it is a true duplicate (e.g., DE and AT for the same content). Also note that Google detects a page's language not only from hreflang but also from the content itself. A crawler that only checks HTML structure will not flag errors here. Therefore, integrate a language detector for the text body to uncover false language declarations. Avoid pitfalls like missing language codes in URL structure (e.g., using only parameters), as these are often ignored by crawlers. Document every anomaly found with a screenshot and source code excerpt to prevent misinterpretation within the team.

Practical example: Step-by-step audit of a multilingual website with 24 markets

Let's take a fictional website offered in 24 EU languages, with URL structure example.com/{languagecode}/ (e.g., /de/, /fr/). Step 1: Collect all language variants of the homepage and check if each contains an hreflang tag with 24 alternatives plus x-default. Crawl each start URL manually with a tool like Screaming Frog and extract the hreflang tags. Step 2: Validate the sitemap. Often, individual language variants are missing or incorrectly assigned. An Excel export of sitemap URLs with language code breakdown helps identify gaps. Step 3: Perform a full crawl of all 24 start URLs (limit: 10,000 URLs). Ensure the crawler treats each language version as a separate host or at least path. Note all 4xx and 5xx errors and redirect chains. Step 4: Analyze internal links: Does the German homepage link to the French one? If the link is missing, Google may not discover the French page even if the sitemap is correct. Tools like DeepCrawl or Sitebulb's link analysis reveal such gaps. Step 5: Check canonical URLs. Visit each language version and inspect the source code to see if the canonical URL points to its own version. Step 6: Measure loading time for each language using a headless browser. Differences exceeding 2 seconds indicate inefficient resources per market. Step 7: Create an error report by priority: High priority (e.g., incorrect hreflang, missing language versions), medium (e.g., redirect chain, missing internal links), low (e.g., performance optimization). In this specific case, we found a hreflang typo in the Spanish version: 'es-ES' instead of 'es'. Such typos are often overlooked because the crawler accepts the tag syntactically. Document each error with the exact URL and recommended correction. After fixing, repeat the audit to confirm correctness. Regular monthly crawls prevent new errors from going unnoticed.

FAQs

Which crawling tools are suitable for multilingual audits?

The selection depends on your requirements. Free tools like Screaming Frog SEO Spider support multiple languages but require manual configuration. For large setups with 24 markets, enterprise solutions such as DeepCrawl or Sitebulb are recommended, offering automated hreflang checks and scalable reports. Look for features like language detection and export options for different market crawls.

How do I automatically test hreflang tags for correctness?

Use tools with built-in hreflang validation that check mutual references and missing backlinks. Alternatively, you can use your own scripts: crawl all language versions, extract hreflang annotations from the HTML, and compare them with the data from XML sitemaps. Also check the consistency of language codes (ISO 639-1) and the alignment of href attributes with the actual URLs.

What intervals do you recommend for regular crawling routines?

The frequency depends on the update rate of your content. For weekly content updates, a weekly crawl is sensible; for monthly changes, a monthly crawl. For large, dynamic shops, a daily crawl of the most important pages is recommended. Also plan ad-hoc audits after major changes such as market launches or CMS updates. Automate the routines via cron jobs or tools like CloudCrawler.

Request a non-binding quote

Response within 24 hours on business days.

German GmbHLocal Court Frankfurt am Main · HRB 111727
D-U-N-S® registered315030052
GDPR-compliant processingHosting in Germany
Fixed prices with written delivery guarantee