Next.js 14 - Create a Broken Link Checker based on the sitemap.xml

Table of Contents

Used Stack
No new NPM Packages needed
New API Route for reading the sitemap.xml
New SitemapChecker Component
Main parts of the Broken Link Checker
Integrating Component in Page.tsx
Final result
Cloudapp-dev, and before you leave us

A broken link checker is crucial for maintaining website health and user experience. It helps identify and fix dead links, which can frustrate users and negatively impact SEO rankings. Regularly checking for broken links ensures all pages are accessible, preserving the website’s credibility and functionality. Additionally, it aids in identifying outdated or moved content that needs updating or redirecting. Ultimately, it supports overall site performance and user satisfaction.

Used Stack

I will start with my default stack:

Contentful as CMS and the foundation for the sitemap.xml creation
Next.js 14 as the web framework, and I will use the provided API Routes
Vercel for hosting

Here is the GitHub repo with the full code.

Example page hosted on Vercel -> https://nextjs14-azureb2c-prisma.vercel.app/

No new NPM Packages needed

Since we rely on the previous story, where I implemented a Sitemap Counter, there is no need to install new packages to make the “Broken Link Checker” work as expected.

New API Route for reading the sitemap.xml

Let’s start with the new API Route for reading the sitemap.xml and then test each URL.

//src/app/api/sitemaplinkcheck/route.ts
import { NextRequest, NextResponse } from "next/server";
import { parseStringPromise } from "xml2js";

export async function GET(request: NextRequest) {
  const { searchParams } = new URL(request.url);
  const url = searchParams.get("url");

  if (!url) {
    return NextResponse.json({ error: "URL is required" }, { status: 400 });
  }

  const broken: string[] = [];

  try {
    const response = await fetch(url as string);
    const xmlData = await response.text();
    const result = await parseStringPromise(xmlData);
    const urls = result.urlset.url.map((url: any) => url.loc[0]);

    let data: Response;

    for (const url of urls) {
      try {
        data = await fetch(url);
        // If needed you can check the status code of the response

        // if (data.status !== 200) {
        //   broken.push(url as string);
        // }
      } catch (error) {
        console.error("Error fetching the sitemap:", error);
        broken.push(url as string);
      }
    }
  } catch (error) {
    console.error("Error fetching the sitemap:", error);
  } finally {
  }

  if (broken !== null) {
    return NextResponse.json({ entries: broken || 0 });
  } else {
    return NextResponse.json(
      { error: "Failed to fetch or parse sitemap" },
      { status: 500 }
    );
  }

  // Alternatively, you could return a 202 Accepted status code

  return new NextResponse(null, { status: 202 });
}

New SitemapChecker Component

This time, the complete logic sits within one new component.

//src/components/tools/sitemapchecker.component.tsx
"use client";
import { useState } from "react";
import { Button } from "@tremor/react";
import type { LocaleTypes } from "@/app/i18n/settings";
import { useTranslation } from "@/app/i18n/client";
import { useParams } from "next/navigation";

const SitemapChecker: React.FC = () => {
  const [brokenLinks, setBrokenLinks] = useState<string[]>([]);
  const [loading, setLoading] = useState(false);
  const [url, setUrl] = useState<string>("");

  const locale = useParams()?.locale as LocaleTypes;
  const { t } = useTranslation(locale, "common");

  const checkLinks = async () => {
    setLoading(true);

    try {
      const res = await fetch(
        `/api/sitemaplinkcheck?url=${encodeURIComponent(url)}`
      );
      const data = await res.json();
      if (res.ok) {
        setBrokenLinks(data.entries);
      } else {
        setBrokenLinks(data.error);
      }
    } catch (err) {
      console.error("An unexpected error occurred", err);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="rounded-md mt-4">
      <h2 className="text-2xl font-bold mb-4">Broken Link Checker</h2>
      <input
        type="text"
        value={url}
        name="search"
        id="search"
        className="block w-full h-10 max-w-md border border-gray-200 rounded-md pl-2 focus:border-indigo-500 focus:ring-indigo-500 sm:text-base"
        placeholder={t("sitemapcounter.placeholder")}
        spellCheck={false}
        onChange={(e) => setUrl(e.target.value)}
      />
      <Button
        size="lg"
        className="mt-4 sm:flex sm:max-w-md"
        onClick={checkLinks}
        disabled={!url}
      >
        {loading ? "Checking..." : "Check Links"}
      </Button>
      {brokenLinks.length > 0 && (
        <div className="mt-4">
          <h3 className="text-xl font-semibold mb-2">Broken Links:</h3>
          <ul className="list-disc list-inside">
            {brokenLinks.map((link, index) => (
              <li key={index} className="text-red-500">
                {link}
              </li>
            ))}
          </ul>
        </div>
      )}

      {brokenLinks.length === 0 && url !== "" && !loading && (
        <p className="mt-4 text-green-500">No broken links found!</p>
      )}
    </div>
  );
};

export default SitemapChecker;

Main parts of the Broken Link Checker

We have a simple input field for the Sitemap URL and a Button to trigger the Check.

Below, you see the logic for fetching the sitemap.xml, checking the entries, and pushing errors to the corresponding variable.

const checkLinks = async () => {
    setLoading(true);

    try {
      const response = await fetch(url);
      const xmlData = await response.text();
      const result = await parseStringPromise(xmlData);
      const urls = result.urlset.url.map((url: any) => url.loc[0]);

      const broken: string[] = [];

      for (const url of urls) {
        try {
          await fetch(url as string);
        } catch (error) {
          console.error("Error fetching the sitemap:", error);
          broken.push(url as string);
        }
      }

      setBrokenLinks(broken);
    } catch (error) {
      console.error("Error fetching the sitemap:", error);
    } finally {
      setLoading(false);
    }
  };

The check is quite basic, but it’s a good foundation for further enhancements (status, timings, etc.)

Integrating Component in Page.tsx

As the last step, I added the new component to my page.tsx under src/app/[locale]

import { client } from "@/lib/client";
import { notFound } from "next/navigation";
import { ArticleHero } from "@/components/contentful/ArticleHero";
import { ArticleTileGrid } from "@/components/contentful/ArticleTileGrid";
import { Container } from "@/components/contentful/container/Container";
import { draftMode } from "next/headers";
// Internationalization
import { locales, LocaleTypes } from "@/app/i18n/settings";
import { createTranslation } from "@/app/i18n/server";
//SEO - JSON-LD
import { Article, WithContext } from "schema-dts";
import path from "path";
import Script from "next/script";
import { Metadata, ResolvingMetadata } from "next";
// New Fields Part 9 of tutorial
import { PageBlogPostOrder } from "@/lib/__generated/sdk";
import { TextHighLight } from "@/components/contentful/TextHighLight";
import { revalidateDuration } from "@/utils/constants";
import { TagCloudSimpleHome } from "@/components/search/tagcloudsimpleHome.component";
import Link from "next/link";
import { LandingContent } from "@/components/contentful/ArticleContentLanding";

import Sitemapcounter from "@/components/tools/sitemapcounter/counter.component";
import SitemapChecker from "@/components/tools/sitemapchecker/sitemapchecker.component";

export const revalidate = revalidateDuration; // revalidate at most every hour

interface PageParams {
  slug: string;
  locale: string;
}

interface PageProps {
  params: PageParams;
}

const generateUrl = (locale: string, slug: string) => {
  if (locale === "en-US") {
    return new URL(slug, process.env.NEXT_PUBLIC_BASE_URL!).toString();
  } else {
    return new URL(locale, process.env.NEXT_PUBLIC_BASE_URL!).toString();
  }
};

const WebUrl = process.env.NEXT_PUBLIC_BASE_URL as string;

export async function generateMetadata(
  { params }: PageProps,
  parent: ResolvingMetadata
): Promise<Metadata> {
  const [PagedataSeo] = await Promise.all([
    client.pageLanding({
      slug: "/",
      locale: params.locale.toString(),
      preview: draftMode().isEnabled,
    }),
  ]);

  const landingPage = PagedataSeo.pageLandingCollection?.items[0];

  if (!landingPage) {
    return notFound();
  }

  const url = generateUrl(params.locale || "", "");

  return {
    title: landingPage.seoFields?.pageTitle,
    description: landingPage.seoFields?.pageDescription,
    metadataBase: new URL(WebUrl),
    alternates: {
      canonical: url,
      languages: {
        "en-US": "/",
        "de-DE": "/de-DE",
        "x-default": "/",
      },
    },
    openGraph: {
      type: "website",
      siteName: "Example.dev - Free Tutorials and Resources for Developers",
      locale: params.locale,
      url: url || "",

      title: landingPage.seoFields?.pageTitle || undefined,
      description: landingPage.seoFields?.pageDescription || undefined,
      images: landingPage.seoFields?.shareImagesCollection?.items.map(
        (item) => ({
          url: item?.url || "",
          width: item?.width || 0,
          height: item?.height || 0,
          alt: item?.description || "",
          type: item?.contentType || "",
        })
      ),
    },
    robots: {
      follow: landingPage.seoFields?.follow || false,
      index: landingPage.seoFields?.index || false,
      googleBot: {
        index: true,
        follow: true,
        "max-image-preview": "large",
      },
    },
  };
}

async function Home({ params }: PageProps) {
  const { isEnabled } = draftMode();
  //declare JSON-LD schema
  let jsonLd: WithContext<Article> = {} as WithContext<Article>;
  const [landingPageData] = await Promise.all([
    client.pageLanding({
      slug: "/",
      locale: params.locale.toString(),
      preview: isEnabled,
    }),
  ]);

  const page = landingPageData.pageLandingCollection?.items[0];

  if (!page) {
    // If a blog post can't be found,
    // tell Next.js to render a 404 page.
    return notFound();
  }

  // TagCloud
  const showTagCloud = page.showTagCloud === "Yes";

  let { datanew, minSize, maxSize } = {
    datanew: [],
    minSize: 0,
    maxSize: 0,
  };

  if (showTagCloud) {
    const searchFacets = await fetch(
      `${process.env.NEXT_PUBLIC_BASE_URL}/api/search/facets`,
      {}
    )
      .then((res) => res.json())
      .catch((error) => {
        console.log("No data found");
      });

    if (searchFacets) {
      maxSize = searchFacets.maxSize;
      minSize = searchFacets.minSize;
      datanew = searchFacets.datanew;
    }
  }

  // Getting BlogPosts
  const blogPostsData = await client.pageBlogPostCollection({
    limit: 10,
    locale: params.locale.toString(),
    skip: 0,
    preview: isEnabled,
    order: PageBlogPostOrder.PublishedDateDesc,
    where: {
      slug_not: page?.featuredBlogPost?.slug,
    },
  });
  const posts = blogPostsData.pageBlogPostCollection?.items;

  // Create JSON-LD schema only if blogPost is available
  if (page) {
    jsonLd = {
      "@context": "https://schema.org",
      "@type": "Article",
      headline: page?.seoFields?.pageTitle || undefined,
      author: {
        "@type": "Person",
        name: page?.featuredBlogPost?.author?.name || undefined,
        // The full URL must be provided, including the website's domain.
        url: new URL(
          path.join(
            params.locale.toString() || "",
            params.slug?.toString() || ""
          ),
          process.env.NEXT_PUBLIC_BASE_URL!
        ).toString(),
      },
      publisher: {
        "@type": "Organization",
        name: "Example.dev - Free Tutorials and Resources for Developers",
        logo: {
          "@type": "ImageObject",
          url: "https://www.example.dev/favicons/icon-192x192.png",
        },
      },
      image: page?.featuredBlogPost?.featuredImage?.url || undefined,
      datePublished: page.sys.firstPublishedAt,
      dateModified: page.sys.publishedAt,
    };
  }

  // Internationalization, get the translation function
  const { t } = await createTranslation(params.locale as LocaleTypes, "common");

  const highLightHeadings: any = page.textHighlightCollection?.items[0];

  if (!page?.featuredBlogPost || !posts) return;

  return (
    <>
      {page && (
        <Script
          id="article-schema"
          type="application/ld+json"
          dangerouslySetInnerHTML={{
            __html: JSON.stringify(jsonLd),
          }}
        />
      )}
      <Container className="mt-5">
        {highLightHeadings && <TextHighLight headings={highLightHeadings} />}

        {showTagCloud && datanew.length > 0 && (
          <TagCloudSimpleHome
            datanew={datanew}
            minSize={minSize * 10}
            maxSize={maxSize * 5}
            locale={params.locale.toString()}
            source={"homepage"}
          />
        )}

        <Link
          href={`/${params.locale.toString()}/${page.featuredBlogPost.slug}`}
        >
          <ArticleHero article={page.featuredBlogPost} isHomePage={true} />
        </Link>
        <div className="md:mx-24 md:my-24 sm:mx-16 sm:my-16">
          <LandingContent landing={page} />
        </div>
      </Container>

      <Container className="my-8 md:mb-10 lg:mb-16">
        {posts.length > 0 && (
          <h2 className="mb-4 md:mb-6">{t("landingPage.latestArticles")}</h2>
        )}
        <ArticleTileGrid
          className="grid-cols-1 md:grid-cols-2 lg:grid-cols-4"
          articles={posts}
          slug={page.featuredBlogPost.slug}
          source="loadmore"
          locale={params.locale.toString()}
        />
        {/* SitemapCounter */}
        <Sitemapcounter />
        {/* SitemapChecker */}
        <SitemapChecker />      
      </Container>
    </>
  );
}

export default Home;

Final result

And here we go. Below, you can see the new Sitemap Counter from the previous story and below the new Component for the Broken Link Check on the Homepage.