Improve WordPress SEO with XML sitemaps and robots.txt without a plugin

Home » Optimization » Improve WordPress SEO with XML sitemaps and robots.txt without a plugin
March 1, 2022 Optimization, SEO
Improve WordPress SEO with XML sitemaps and robots.txt without a plugin

An XML sitemap tells search engines which URLs on your site deserve to be crawled and indexed, while robots.txt controls which paths crawlers are allowed to visit in the first place. Together they form the foundation of technical SEO, yet many WordPress developers reach for a heavy SEO plugin just to get these two files right. WordPress 5.5 introduced a built-in sitemap at /wp-sitemap.xml, but it is generic and cannot be tuned without hooking into its API. Writing your own sitemap endpoint takes fewer than sixty lines of PHP and gives you full control over which post types and taxonomies are included, what the <lastmod> date reports, and how many URLs appear per sitemap page. The robots.txt file in WordPress is generated dynamically by wp_robots_txt(); you can modify it via the robots_txt filter without creating a physical file on disk. Keeping both files plugin-free means one fewer dependency to update, audit, and troubleshoot. This article shows how to register a custom rewrite rule that serves your own XML sitemap, how to populate it from WP_Query, and how to extend robots.txt with custom directives. You will also learn how to verify that Google Search Console can fetch both files after deployment. The approach works on any shared host or VPS without shell access. All output is properly escaped and served with the correct Content-Type header. You can extend the same pattern to generate news sitemaps, image sitemaps, or video sitemaps by adjusting the query and adding the relevant XML namespace. Performance-sensitive sites should cache the generated sitemap output using the Transients API.

Problem: You want search engines to index all published content on your WordPress site and restrict crawler access to sensitive paths, without installing an SEO plugin.

Solution: Register a custom /sitemap.xml rewrite endpoint that outputs your posts and pages in XML, and extend WordPress’s dynamic robots.txt output via filter:

<?php
// Register /sitemap.xml rewrite rule
add_action( 'init', function () {
    add_rewrite_rule( '^sitemap\.xml$', 'index.php?custom_sitemap=1', 'top' );
    add_rewrite_tag( '%custom_sitemap%', '([0-9]+)' );
} );

// Output the sitemap when the rewrite fires
add_action( 'template_redirect', function () {
    if ( ! get_query_var( 'custom_sitemap' ) ) {
        return;
    }

    $posts = get_posts( [
        'post_type'      => [ 'post', 'page' ],
        'post_status'    => 'publish',
        'posts_per_page' => 500,
        'no_found_rows'  => true,
    ] );

    header( 'Content-Type: application/xml; charset=UTF-8' );
    echo '<?xml version="1.0" encoding="UTF-8"?>' . "
";
    echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "
";

    // Homepage
    echo '<url><loc>' . esc_url( home_url( '/' ) ) . '</loc><changefreq>daily</changefreq><priority>1.0</priority></url>' . "
";

    foreach ( $posts as $p ) {
        $modified = mysql2date( 'c', $p->post_modified_gmt . ' GMT' );
        echo '<url>' . "
";
        echo '  <loc>' . esc_url( get_permalink( $p ) ) . '</loc>' . "
";
        echo '  <lastmod>' . esc_html( $modified ) . '</lastmod>' . "
";
        echo '  <changefreq>weekly</changefreq>' . "
";
        echo '  <priority>0.8</priority>' . "
";
        echo '</url>' . "
";
    }

    echo '</urlset>';
    exit;
} );

// Extend robots.txt via filter
add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "
Sitemap: " . esc_url( home_url( '/sitemap.xml' ) ) . "
";
    $output .= "Disallow: /wp-admin/
";
    $output .= "Disallow: /xmlrpc.php
";
    return $output;
}, 10, 2 );

NOTE: After adding the rewrite rule flush your permalinks once by visiting Settings → Permalinks and clicking Save. Without the flush WordPress will not recognise the sitemap.xml rewrite and you will see a 404. Also disable the built-in WordPress 5.5 sitemap if you are replacing it: add_filter( ‘wp_sitemaps_enabled’, ‘__return_false’ );. Pair this with the JSON-LD schema guide to complete your on-page technical SEO setup.

Related posts