How to Create a Sitemap in 2026: XML Sitemaps Step by Step, With Real Crawler Data and a Bug in Our Own

Learning how to create a sitemap is barely work anymore. Almost every CMS and framework generates the file for you, and most guides stop right there: install a plugin, paste the URL into Search Console, done. For this article we wanted to know what happens next. So we took apart the sitemap of getmind.io and went through 17 days of server logs (151,296 requests) to see who actually fetches it.

What we found was a bug we had built ourselves. Our sitemap listed 234 URLs and not a single modification date. The file was valid, returned status 200, and Google, Bing and several other crawlers fetched it regularly. But it told nobody which of those 234 pages were new. For Google, the lastmod field is the only value in a sitemap besides the URL that matters at all. We fixed it this morning, and the code is further down.

In this guide you’ll learn how an XML sitemap is structured, how to create a sitemap in WordPress, Shopware, Astro, Next.js or with no CMS at all, how to submit it to Google and Bing, and what a sitemap does and doesn’t do for you. Every number comes from our own logs from October 3 to 20, 2026 and from tests run on October 20, 2026.

This video by AshWebDesigns (March 2026) walks through finding and submitting a sitemap in Google Search Console. It covers the clicks; this article covers what happens after them.

What is a sitemap?

A sitemap is a file that lists every page on your website that you want to show up in search engines. Think of it as a table of contents for crawlers, the programs Google, Bing and others use to discover the web.

When people say “sitemap” today they almost always mean an XML sitemap following the sitemaps.org protocol. Google, Bing, Yahoo and Ask introduced it together in 2006, and it has barely changed since. Two other things go by a similar name:

• HTML sitemap: a regular web page for humans that links to every section of a site, usually from the footer. It helps with navigation; to search engines it’s just another page with lots of links.
• Other sitemap formats: Google also accepts an RSS or Atom feed and a plain text file with one URL per line. Both are allowed, but less flexible than XML.

An XML sitemap doesn’t replace good internal linking. Google still discovers pages mostly through links. The sitemap is the second path, and it matters most for pages that links don’t reach well.

Does my website even need a sitemap?

The honest answer: not every site does. Google’s own documentation says small sites with up to roughly 500 pages that are well linked internally don’t necessarily need one. If you have ten pages and all of them are in your menu, a sitemap won’t give you a measurable advantage.

A sitemap is useful, or even important, if

• your site is large, with hundreds or thousands of pages, such as a store with many products,
• your pages are poorly linked, for example archive pages only reachable via pagination or internal search,
• your site is new and has few third-party links,
• you publish new content regularly, like a blog or news site,
• you have images, videos or news you want to submit to Google specifically,
• you run multiple language versions and want an extra safety net for how they map to each other.

Several of these apply to getmind.io: a new article every day, two languages, 234 URLs. A correct sitemap never hurts. So our advice is: if your system generates one anyway, keep it on and make sure it’s right.

How is an XML sitemap structured?

A minimal sitemap looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/first-post/</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
</urlset>

That’s all you need. The building blocks:

The last two rows surprise a lot of people. Older guides still tell you to give your homepage priority 1.0 and blog posts changefreq daily. Google has said openly for years that it ignores both. Bing treats them similarly. You can drop them and stop spending time fine-tuning them.

lastmod: the only field that really counts

lastmod says when the content of a page last changed significantly. The W3C date format is allowed, so 2026-10-20 or with a time, 2026-10-20T07:00:00+00:00.

Google uses the value to decide which pages to recrawl. But only as long as the value is believable. If a site sets lastmod to “now” on every build or every request, Google notices and stops trusting the field. A changed copyright line in the footer or a new sidebar widget isn’t a significant change. A new version of the article, corrected facts or a new section are.

Two rules follow from that:

1. Never use the build date or file modification time. On static sites it changes for every page at once with each deploy.
2. No lastmod beats a wrong lastmod. A missing field costs you one hint; a wrong one costs you trust in all the others.

Limits: 50,000 URLs and 50 MB

A single sitemap can contain at most 50,000 URLs and be at most 50 MB uncompressed. Beyond that you split it into several files and tie them together with a sitemap index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-products-1.xml</loc>
    <lastmod>2026-10-20</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
  </sitemap>
</sitemapindex>

Our own file shows how much room that is in practice: 234 URLs with lastmod take 23,620 bytes, about 100 bytes per entry. Gzipped, the file shrinks to 2,577 bytes. Extrapolated, 50,000 entries come to roughly 5 MB. Practically nobody hits the 50 MB limit; large stores hit the 50,000-URL limit quickly.

Sitemaps may also be served gzipped as sitemap.xml.gz. That’s rarely necessary. Our web server compresses the file on the fly anyway; in our test only 2,049 bytes went over the wire.

What belongs in a sitemap and what doesn’t?

Only URLs you want to see in Google belong in your sitemap. In practice every URL must

• return status 200 (no redirects, no 404s),
• be canonical, meaning its canonical tag points to itself,
• be indexable (no noindex),
• not be blocked by robots.txt.

Leave out redirects, filter and search result pages, login and cart pages, test pages, parameter duplicates and anything set to noindex.

We checked our own sitemap against exactly that on October 20, 2026: all 234 of 234 URLs returned 200. In a random sample of 40 pages, the canonical pointed to the exact sitemap URL every single time, and none of the 234 pages carried noindex. We had already excluded two addresses on purpose because they only redirect now (an old tool path and a retired article). That’s exactly the kind of upkeep a sitemap needs: as soon as a page redirects, it comes out.

How to create a sitemap: the options at a glance

How you create a sitemap depends on what your site runs on. Almost every common system has a built-in solution or a standard plugin.

WordPress

Since version 5.5 (August 2020), WordPress generates its own sitemap at /wp-sitemap.xml. Nothing to install. We verified that on a WordPress test container:

The last row stands out. WordPress core also lists author archive pages, and their URL contains the username. On our test system it read /author/admin/. So anyone who opens the sitemap learns the administrator’s login name. It’s not really a secret, since WordPress leaks usernames through other routes too. But if you only have one author you don’t need that child sitemap anyway. You can switch it off in code:

add_filter( 'wp_sitemaps_add_provider', function ( $provider, $name ) {
    return ( 'users' === $name ) ? false : $provider;
}, 10, 2 );

If you install an SEO plugin such as Yoast SEO, Rank Math or SEOPress, its sitemap replaces the built-in one. Yoast serves it at /sitemap_index.xml and automatically leaves out anything set to noindex. Which plugin we recommend for which case is in our comparison of WordPress SEO plugins. The only thing that matters here: use exactly one sitemap. Two sitemaps running in parallel aren’t an error, but they confuse anyone who later tries to figure out which one is actually submitted.

Shopware 6

Shopware 6 generates the sitemap itself, per sales channel and domain. You’ll find the settings under Settings → Shop → Sitemap. There you choose whether the sitemap is generated live on request, via a scheduled task, or manually. For large catalogs the scheduled task is the best choice; manually it works from the console:

bin/console sitemap:generate

The finished file lives at https://your-store.com/sitemap.xml. With many products Shopware automatically splits it into several compressed files. More Shopware-specific points, from product variants to filter pages, are collected in our article on Shopware SEO.

Astro

Our own site runs on Astro. The official @astrojs/sitemap integration creates an index (sitemap-index.xml) and the actual files (sitemap-0.xml and so on) at build time. The basic setup is short:

// astro.config.mjs
import { defineConfig } from 'astro/config';
import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://example.com',
  integrations: [sitemap()],
});

This is exactly where our bug was. Out of the box the integration writes no lastmod. It only knows the final URLs, not your content. There is a lastmod option, but it sets the same date on every page, which is precisely the kind of lastmod Google ignores. Our fix reads the date from each article’s frontmatter and attaches it only to the matching URLs:

import fs from 'node:fs';
import path from 'node:path';

// Date from frontmatter: updatedDate, else pubDate.
// Deliberately NOT the build date.
function readArticleDates() {
  const map = new Map();
  const dirs = [['src/content/blog', '/blog/'], ['src/content/blog-de', '/de/blog/']];
  for (const [dir, prefix] of dirs) {
    for (const f of fs.readdirSync(dir)) {
      if (!f.endsWith('.mdx')) continue;
      const head = fs.readFileSync(path.join(dir, f), 'utf8').split('\n---')[0];
      const pick = (k) => (head.match(new RegExp('^' + k + ':\\s*["\']?(\\d{4}-\\d{2}-\\d{2})', 'm')) || [])[1];
      const d = pick('updatedDate') || pick('pubDate');
      if (d) map.set(prefix + f.replace(/\.mdx$/, '') + '/', d);
    }
  }
  return map;
}
const DATES = readArticleDates();

// inside integrations:
sitemap({
  serialize(item) {
    const d = DATES.get(new URL(item.url).pathname);
    if (d) item.lastmod = d;
    return item;
  },
}),

After the build, 182 of the 234 URLs had a lastmod: every article in both languages. The remaining 52 are home, category, tool and legal pages. We have no honest modification date for those, so they don’t get one. We also added a new optional updatedDate field to the frontmatter. When an article gets a substantial rewrite, we enter the date there and the sitemap picks it up.

Next.js

In the Next.js App Router you create a file app/sitemap.ts, and Next.js turns it into /sitemap.xml:

// app/sitemap.ts
import type { MetadataRoute } from 'next';
import { getAllPosts } from '@/lib/posts';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts();
  return [
    { url: 'https://example.com/', lastModified: new Date('2026-10-01') },
    ...posts.map((p) => ({
      url: `https://example.com/blog/${p.slug}/`,
      lastModified: new Date(p.updatedAt ?? p.publishedAt),
    })),
  ];
}

The most common trap here is ours in reverse: lastModified: new Date() looks harmless but marks every page as “just changed” on every request. For more than 50,000 URLs there’s generateSitemaps(), which creates multiple files automatically. We cover dynamic routes in more detail in our article on Next.js dynamic routes. Which of the two approaches suits a blog better is shown in our benchmark Astro vs Next.js.

Without a CMS: by hand or with a script

For a small static site with no CMS you can write the sitemap by hand. Beyond a handful of pages a small script is safer. This example reads every HTML file in a folder and takes the modification date from Git rather than the file system:

#!/usr/bin/env bash
# sitemap.sh — builds sitemap.xml from every index.html in ./public
BASE="https://example.com"
{
  echo '<?xml version="1.0" encoding="UTF-8"?>'
  echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
  find public -name index.html | sort | while read -r f; do
    path="${f#public}"; path="${path%index.html}"
    date=$(git log -1 --format=%cs -- "$f")
    echo "  <url><loc>${BASE}${path}</loc>${date:+<lastmod>$date</lastmod>}</url>"
  done
  echo '</urlset>'
} > public/sitemap.xml

The Git date (%cs = date of the last commit that touched the file) beats mtime by a mile, because a fresh checkout on the server would otherwise stamp every file with today’s date. If Git is still new to you, start with our introduction What is Git?.

Online generators

Tools like xml-sitemaps.com or the Screaming Frog SEO Spider crawl your site and produce a downloadable sitemap. That’s handy for a one-off inventory. As a permanent solution it doesn’t work. The file is a snapshot and goes stale with every new page. Worse, a crawler can’t find exactly the pages that are poorly linked, which are the ones you need a sitemap for in the first place. A sitemap should be generated from your data, not from a crawl.

Check your sitemap before submitting it

Before you submit your sitemap anywhere, check it yourself once. Three commands do the job:

# 1. Is the file reachable and valid XML?
curl -s https://example.com/sitemap.xml | xmllint --noout - && echo "XML ok"

# 2. Do all URLs return 200?
curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' \
  | xargs -P 16 -I{} curl -s -o /dev/null -w "%{http_code} {}\n" {} \
  | sort | grep -v '^200' || echo "all 200"

# 3. How many entries have a lastmod?
curl -s https://example.com/sitemap.xml | grep -o '<lastmod>' | wc -l

We briefly fell for the third one ourselves. Our first attempt was grep -c '<lastmod>'. It returned 1, even though 182 entries were there. grep -c counts lines, not matches, and our sitemap is one single long line. Only grep -o … | wc -l counts each match. Had we believed the first number, we’d have declared a working fix broken.

Also check the Content-Type. It should be application/xml or text/xml. Ours comes back as text/xml; charset=utf-8. And make sure the URLs in your sitemap match your canonical domain exactly: with or without www, with or without a trailing slash. A sitemap full of http:// addresses on an https:// site is nothing but redirects.

How to submit a sitemap: Google, Bing and robots.txt

A sitemap nobody knows about does nothing. There are three ways to announce it, and you should use all three.

1. Reference it in robots.txt

The simplest route works for every search engine at once, including the ones you don’t have an account with. One line in robots.txt is enough:

Sitemap: https://example.com/sitemap.xml

The URL must be absolute, with https:// and the domain. Multiple sitemap lines are allowed. What the rest of your robots.txt should look like and where the traps are is in our guide on how to create a robots.txt. Our own robots.txt points to the sitemap index, not the individual files. That’s enough; crawlers follow the index on their own.

2. Submit it in Google Search Console

1. Open Google Search Console and select your property.
2. Click Indexing → Sitemaps in the left menu.
3. Under “Add a new sitemap”, enter the path, e.g. sitemap.xml or sitemap-index.xml.
4. Click Submit.

Google then shows a status. “Success” only means Google could read the file. It doesn’t mean all pages are indexed. Whether and how many pages make it into the index is shown in the Page indexing report. You only submit once, by the way. After that Google fetches the sitemap on its own.

Being George’s video (July 2026) shows the Search Console submission click by click, if you prefer to watch it once before doing it.

3. Submit it in Bing Webmaster Tools

Bing works almost the same way: in Bing Webmaster Tools under Sitemaps, click “Submit sitemap” and enter the URL. If Search Console is already set up, you can import your site into Bing straight from there, sitemap included. Why Bing is worth more to a small site than most people think is covered in our article on Bing SEO, not least because Copilot and other AI assistants draw on the Bing index.

The sitemap ping is dead

Many older guides and plugins still mention a fourth route: calling a URL like https://www.google.com/ping?sitemap=… to make Google reload the sitemap immediately. Google deprecated that ping endpoint in June 2023 and shut it down at the end of 2023. Today it only returns an error. Bing dropped its ping as well and points to IndexNow instead. If a plugin or deploy script still pings, you can delete that line.

Who actually reads our sitemap?

This is where it gets interesting, because almost no guide covers it. We analyzed every request to getmind.io from October 3, 13:19 to October 20, 07:02 (UTC), 151,296 requests in total. Crawlers were identified by user agent. Anyone can fake that, so treat the numbers as an approximation. For Google we also looked at IP addresses: they come from the 66.249.x.x and 192.178.x.x ranges Google uses for Googlebot.

Three things stand out.

First: Google fetches the sitemap about once a day. 20 fetches in just under 17 days, mostly at night, and almost always the sitemap-0.xml file directly, not the index. Once Google has read the index, it remembers the actual file.

Second: ClaudeBot only reads the table of contents. Anthropic’s crawler fetched the sitemap index 177 times, but the file with the actual URLs not once. From our sitemap it only ever learns that there’s another file, and it never looks inside. It still finds our articles, just via links. If you optimize your sitemap for AI crawlers, with this one you’re optimizing for a reader that never opens the content.

Third: many crawlers ignore the sitemap entirely. PerplexityBot, Applebot and OAI-SearchBot fetched robots.txt 182 times between them, but never the sitemap. They apparently rely entirely on links.

There were also 30 requests for /sitemap.xml, an address that doesn’t exist on our site because Astro names the file differently. 25 of them got a 404. On top of that came requests for /wp-sitemap.xml, /sitemap_index.xml, /sitemap.php and even /sitemaps/.env. Those aren’t search engines but scanners probing known paths of WordPress and other systems. So a sitemap with an unusual name is no problem as long as it’s listed in robots.txt. A crawler that only tries /sitemap.xml usually isn’t one you need anyway.

How fast do crawlers find a new article?

For the six article pairs published October 14 to 19, we measured how many hours after publication each crawler first fetched the new page:

A dash means no fetch before October 20, 07:02.

Bing arrived within minutes for all twelve pages. That’s not the sitemap, it’s IndexNow. We ping every new article via the IndexNow protocol right after deploying, and bingbot shows up immediately. Bing does fetch the sitemap regularly too, but by then the new pages have long been crawled.

Google took between 2.6 and 19.4 hours, and for five of the twelve pages Googlebot didn’t show up at all during the measurement window. That’s despite all twelve URLs being in the sitemap and Google fetching it daily. This is where the missing lastmod shows: our sitemap didn’t tell Google there were new pages; they were just sitting somewhere among 233 others. Whether the fix changes that is something we’ll measure over the coming weeks.

So a sitemap is no guarantee of fast crawling, let alone indexing. It’s a hint Google weighs. For fresh articles, the fastest way to Google remains the URL Inspection tool in Search Console with “Request indexing”. For Bing, Yandex, Seznam and Naver it’s IndexNow.

Special sitemaps: images, video, news and languages

The sitemap protocol is extensible. The four most important extensions:

Image sitemap. With the image namespace you can list the images that appear on each URL. Worth it if images load via JavaScript and Google wouldn’t otherwise find them. With normal <img> tags Google finds them without help.

Video sitemap. For self-hosted videos you can add title, description, thumbnail and duration. For embedded YouTube videos it adds little, since Google already knows the video from YouTube.

News sitemap. Only for sites that appear in Google News. It contains only articles from the last two days.

Language versions (hreflang). If your site exists in several languages, you can declare the translation mapping in the sitemap too:

<url>
  <loc>https://example.com/blog/create-sitemap/</loc>
  <xhtml:link rel="alternate" hreflang="en" href="https://example.com/blog/create-sitemap/"/>
  <xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/blog/sitemap-erstellen/"/>
</url>

That’s an alternative to hreflang tags in the HTML head, not an extra requirement. We set hreflang in the HTML and check on every build that each German article finds its English counterpart and vice versa. Declaring it twice does no harm as long as both agree. If they contradict each other, things get messy.

The most common mistakes when you create a sitemap

1. lastmod missing or lying. Either no date at all (our bug) or every page stamped with the last build date. Both take away Google’s only useful hint.
2. Redirects in the sitemap. Typical after moving from http to https or renaming URLs. Every entry should return 200 directly.
3. noindex pages in the sitemap. You’re telling Google “please look” and “please don’t include” at the same time. Search Console flags it.
4. Blocked URLs. Pages blocked in robots.txt have no business in the sitemap.
5. Wrong domain variant. www instead of none, http instead of https, trailing slash or not. Only the canonical variant goes in.
6. Two sitemaps side by side. WordPress’s built-in one plus a plugin’s, or an old static file next to the new dynamic one.
7. Stale sitemap. A one-off export that hasn’t been updated in months while new pages kept appearing.
8. Ping code calling into the void. Harmless, but a sign nobody’s checking anymore.
9. Too many entries in one file. Past 50,000 URLs or 50 MB you must split, or the file is rejected.
10. Counting wrong. Like us with grep -c: a check that counts the wrong unit can report a working state as broken, or a broken one as healthy.

Conclusion: creating a sitemap is easy, keeping a good one is the work

You can create a sitemap in five minutes, and on most systems it happens on its own. The difference between a sitemap that merely exists and one that helps comes down to three things: it only contains canonical, indexable URLs returning 200, every page has an honest lastmod, and the file is referenced in robots.txt and submitted to both Search Console and Bing Webmaster Tools.

Our logs also show where the limits are. Google reads the sitemap daily and still crawls new pages hours later, or not at all. AI crawlers like ClaudeBot, PerplexityBot and OAI-SearchBot practically never open it. Bing is fastest, but because of IndexNow, not the sitemap. If you want new content found quickly, you need the sitemap as a foundation, solid internal links, and for fresh pages the URL Inspection tool in Search Console plus IndexNow on top.

And run the three commands above against your own sitemap once. We wouldn’t have noticed without writing this article.

Frequently asked questions about creating a sitemap

How do I create a sitemap?

The easiest way is through your CMS: WordPress has generated /wp-sitemap.xml automatically since version 5.5, and SEO plugins like Yoast or Rank Math replace it with their own. Shopware, Astro (@astrojs/sitemap) and Next.js (app/sitemap.ts) have built-in solutions. Without a CMS you write an XML file with <urlset>, one <url> entry per page and a <loc> holding the full URL, ideally generated by a script.

What should an XML sitemap contain?

Only <urlset>, <url> and <loc> are required. <lastmod> with the date of the last significant content change is recommended. Google ignores <changefreq> and <priority>, so you can leave them out. Only include canonical URLs that return status 200, aren’t set to noindex and aren’t blocked by robots.txt.

Where do I find my sitemap?

Usually at /sitemap.xml, /sitemap_index.xml (Yoast), /wp-sitemap.xml (WordPress without a plugin) or /sitemap-index.xml (Astro). The most reliable way is to look at your robots.txt at https://your-domain.com/robots.txt: there should be a Sitemap: line.

How do I submit a sitemap to Google?

In Google Search Console under Indexing → Sitemaps, enter the path and click Submit. Also add the line Sitemap: https://your-domain.com/sitemap.xml to your robots.txt. Submitting once is enough; after that Google fetches the file regularly on its own. The old ping endpoint has been gone since the end of 2023.

How many URLs can a sitemap contain?

At most 50,000 URLs and at most 50 MB uncompressed per file. With more pages, split them across several files and tie them together with a sitemap index, which itself may list up to 50,000 sitemaps. In practice an entry takes about 100 bytes, so the URL limit is almost always the first one you hit.

Does every website need a sitemap?

No. According to Google, small, well-linked sites with up to roughly 500 pages are fine without one. It becomes important for large sites, stores, new domains without third-party links, sites with poorly linked sections and sites that publish new content regularly. A correct sitemap never hurts.

How often should I update my sitemap?

Whenever pages change, ideally automatically. The sitemap should be generated by your system rather than sitting around as a one-off export. You don’t need to resubmit it in Search Console; Google fetches it on its own. On our site that was 20 fetches in 17 days, about one a day.

What’s the difference between an HTML and an XML sitemap?

The XML sitemap is a machine-readable file for search engines that visitors never see. The HTML sitemap is an ordinary web page linking to every section, meant for humans. For SEO the XML sitemap is the tool; an HTML sitemap is at most a navigation aid.

Does lastmod in the sitemap matter?

Yes. It’s the only optional field Google actually uses, but only if it’s reliably accurate. If you set it to the current date on every build, Google learns not to trust it. Use the date of the last real content change, for example from frontmatter, your database or Git history. No lastmod beats a wrong one.

Why aren’t pages from my sitemap getting indexed?

A sitemap is a recommendation, not a command. Google decides for itself what to crawl and index. Common reasons pages stay out are thin or duplicate content, a differing canonical, noindex, robots.txt blocks, or simply low priority on new sites. The Page indexing report in Search Console shows the exact reason.

Does a sitemap help with AI search like ChatGPT or Perplexity?

Based on our logs, barely. PerplexityBot and OAI-SearchBot never fetched our sitemap in 17 days, and ClaudeBot only fetched the index, never the actual URLs. These crawlers find pages through links. Indirectly the sitemap still helps: Copilot and other assistants draw on the Bing index, and Bing does use the sitemap. More in our analysis of AI crawlers in our log files.