Good free site map generator for large web sites

[pintobuck]

Using Joomla 3.6.5 on a web site with 2,000+ pages.

I have been looking for a good, free site map generator. I tried some of the "free" Joomla site map extensions, but the ones I tried turned out not to be free for medium to large web sites. For example, some offered an on-line service that would make free site maps for websites with 500 urls or less. And, some Joomla site map generators would make free site maps for Joomla Core (basically your content articles only), but required paid plugins to work for your forum, social, community, or store extensions. (Of course, it is only fair that programmers should be able to make some money off of their work.)

All that made me long for the old style, free site map generators like gSiteMap or gSiteCrawler, which ran on my local computer, scanned every link on my website, identified broken links, and generated site maps. The problem is that those programs have not been updated and they do not work with Windows 10.

Most recently, I found Xenu Link Sleuth site map generator, which runs on my Windows 10 computer. Xenu scans every link on your web site, identifies broken links (so you can fix or exclude them), and it generates site maps for very large sites. Xenu is totally free.

I found Xenu Link Sleuth site map generator on Cnet.com, so I felt safe downloading it. I have been using it for a couple months with good results. (I was not put off by the icon of a space alien. Space geeks can be good programmers, too.)

Purpose of this post is to share my find, and to get suggestions.

[Jaypatel5]

You have to try a website that generates a sitemap for you. I am SEO expert and I have been using this https://www.xml-sitemaps.com website to generate XML sitemap. I think you should try this.

[james_mc]

Try to use Aimy Sitemap https://extensions.joomla.org/extension/aimy-sitemap/

They state "although larger websites are supported, we recommend using Aimy Sitemap on small to medium websites with a couple of hundred pages only."

However, we had experience with really large websites.

[Webdongle]

https://extensions.joomla.org/extension ... map/osmap/ is good

[KeywordLove]

For a free desktop tool handling 2000+ pages I think you found the best. I assume you need it regularly and not just for a limited time?

[pintobuck]

Thanks for the feedback.

New info: When I activated SSL on my site, then "Xenu Link Sleuth" could not crawl my site anymore.

I found two more sitemap generators that run on Windows 10 on your local PC and crawl almost unlimited pages:
"WonderWebWare Sitemap Generator" and "AnalogX LinkExaminer". But, neither of them would crawl my SSL site.

My solution, so far, has been to backup my site to a non-SSL domain, and then crawl that site. After the crawler generates a site map, I use Excel to search and replace the domain name. Then, I have an updated site map for my SSL site.

[pintobuck]

Additional info: In order to get the above mentioned site map web crawlers to consistently work on my site, even the non-SSL site, I found it necessary to disable "JU BlockIP" security plugin while they are crawling my site.

[alex29]

You can use — https://extensions.joomla.org/extension ... p/sitemap/
or as says @Webdongle — https://extensions.joomla.org/extension ... map/osmap/

It is a good sitemap generators

[pintobuck]

Thank you for the suggestions. The free OSMap seems to crawl only "core" Joomla content, and it would not crawl my forum, so the forum was not be included in the resulting sitemap. To crawl and sitemap the forum, or other noncore Joomla content, it seems that we need the premium OSMap, which costs $37 for a 6 months subscription.

[TP-John]

Try to use Aimy Sitemap https://extensions.joomla.org/extension/aimy-sitemap/

They state "although larger websites are supported, we recommend using Aimy Sitemap on small to medium websites with a couple of hundred pages only."

However, we had experience with really large websites.

A word of caution on Amy Sitemap: HTTPS-only sites require the paid/Pro version