External crawling and sitemap contents

Sitemap Generator Pro uses a developer-operated external service to crawl your WordPress site like a normal visitor and generate an XML sitemap. The service receives your site URL and the fact that it uses WordPress. Your web server still responds to the crawler’s requests, while sitemap generation runs externally. This can fit shared hosting; a dedicated server may already have enough capacity to generate a sitemap locally. The crawler cannot reach a site available only on your local network.

The generator is designed to work with WordPress plugins and can include nearly all indexable file types, such as PDF, XLS, and DOC files. It respects robots.txt and its crawl-delay directive, and excludes pages marked with a robots noindex meta element. Use the MB-SiteCrawler visitor-agent group in robots.txt to control which pages the crawler parses.

Image and video sitemaps

• Images: Pro generates image sitemaps for pictures found in your site content.
• Video: Pro works with video sitemaps for HTML5 video elements and embedded YouTube videos.

Generate and check the sitemap

Install and activate the extension, open Sitemap Generator in the WordPress sidebar, then select “Generate your sitemap.” When the external crawl finishes, the sitemap is saved as sitemap.xml in your WordPress root directory. Select “Show the sitemap” to download it and review its contents. Generation overwrites an existing sitemap.xml without asking, so back up that file first.

Hosting requirements and version limits

The hosting environment needs cURL 7.18.1 or higher and OpenSSL 0.9.8f or higher. Current PHP versions supported by WordPress should provide a compatible cURL version in most hosting environments. Windows webspace has not been tested.

The free version supports sites with up to 500 URLs and does not include image or video sitemap support. Pro adds those media sitemap capabilities; a separate URL limit for Pro is not specified.