Good free site map generator for large web sites

Relax and enjoy The Lounge. For all Non-Joomla! topics or ones that don't fit anywhere else. Normal forum rules apply.
pintobuck
Joomla! Explorer
Joomla! Explorer
Posts: 286
Joined: Tue Mar 06, 2007 9:39 am

Good free site map generator for large web sites

Postby pintobuck » Thu Apr 13, 2017 12:11 pm

Using Joomla 3.6.5 on a web site with 2,000+ pages.

I have been looking for a good, free site map generator. I tried some of the "free" Joomla site map extensions, but the ones I tried turned out not to be free for medium to large web sites. For example, some offered an on-line service that would make free site maps for websites with 500 urls or less. And, some Joomla site map generators would make free site maps for Joomla Core (basically your content articles only), but required paid plugins to work for your forum, social, community, or store extensions. (Of course, it is only fair that programmers should be able to make some money off of their work.)

All that made me long for the old style, free site map generators like gSiteMap or gSiteCrawler, which ran on my local computer, scanned every link on my website, identified broken links, and generated site maps. The problem is that those programs have not been updated and they do not work with Windows 10.

Most recently, I found Xenu Link Sleuth site map generator, which runs on my Windows 10 computer. Xenu scans every link on your web site, identifies broken links (so you can fix or exclude them), and it generates site maps for very large sites. Xenu is totally free.

I found Xenu Link Sleuth site map generator on Cnet.com, so I felt safe downloading it. I have been using it for a couple months with good results. (I was not put off by the icon of a space alien. Space geeks can be good programmers, too.)

Purpose of this post is to share my find, and to get suggestions.
Last edited by toivo on Wed Nov 29, 2017 5:18 pm, edited 1 time in total.
Reason: mod note: moved to The Lounge

Jaypatel5
Joomla! Apprentice
Joomla! Apprentice
Posts: 20
Joined: Mon Mar 20, 2017 6:19 am

Re: Good free site map generator for large web sites

Postby Jaypatel5 » Fri Apr 14, 2017 8:38 am

You have to try a website that generates a sitemap for you. I am SEO expert and I have been using this https://www.xml-sitemaps.com website to generate XML sitemap. I think you should try this.

james_mc
Joomla! Apprentice
Joomla! Apprentice
Posts: 34
Joined: Wed Mar 23, 2016 12:45 pm
Contact:

Re: Good free site map generator for large web sites

Postby james_mc » Fri Apr 14, 2017 12:06 pm

Try to use Aimy Sitemap https://extensions.joomla.org/extension/aimy-sitemap/

They state "although larger websites are supported, we recommend using Aimy Sitemap on small to medium websites with a couple of hundred pages only."

However, we had experience with really large websites.
James,
software developer
https://hqsoftwarelab.com


User avatar
KeywordLove
Joomla! Apprentice
Joomla! Apprentice
Posts: 14
Joined: Fri Dec 05, 2008 6:12 pm
Contact:

Re: Good free site map generator for large web sites

Postby KeywordLove » Tue May 23, 2017 6:53 pm

For a free desktop tool handling 2000+ pages I think you found the best. I assume you need it regularly and not just for a limited time?
Use built-in scan preset for Joomla if 403/forbidden/Sh404SEF issues when using A1 Sitemap Generator
http://www.microsystools.com/products/sitemap-generator/help/sitemap-generator-joomla-website/

pintobuck
Joomla! Explorer
Joomla! Explorer
Posts: 286
Joined: Tue Mar 06, 2007 9:39 am

Re: Good free site map generator for large web sites

Postby pintobuck » Wed Nov 29, 2017 4:39 pm

Thanks for the feedback.

New info: When I activated SSL on my site, then "Xenu Link Sleuth" could not crawl my site anymore.

I found two more sitemap generators that run on Windows 10 on your local PC and crawl almost unlimited pages:
"WonderWebWare Sitemap Generator" and "AnalogX LinkExaminer". But, neither of them would crawl my SSL site.

My solution, so far, has been to backup my site to a non-SSL domain, and then crawl that site. After the crawler generates a site map, I use Excel to search and replace the domain name. Then, I have an updated site map for my SSL site.

pintobuck
Joomla! Explorer
Joomla! Explorer
Posts: 286
Joined: Tue Mar 06, 2007 9:39 am

Re: Good free site map generator for large web sites

Postby pintobuck » Mon Dec 11, 2017 4:04 pm

Additional info: In order to get the above mentioned site map web crawlers to consistently work on my site, even the non-SSL site, I found it necessary to disable "JU BlockIP" security plugin while they are crawling my site.

alex29
Joomla! Apprentice
Joomla! Apprentice
Posts: 30
Joined: Tue Dec 19, 2017 11:25 am

Re: Good free site map generator for large web sites

Postby alex29 » Tue Dec 19, 2017 3:25 pm

You can use — https://extensions.joomla.org/extension ... p/sitemap/
or as says @Webdongle — https://extensions.joomla.org/extension ... map/osmap/

It is a good sitemap generators

pintobuck
Joomla! Explorer
Joomla! Explorer
Posts: 286
Joined: Tue Mar 06, 2007 9:39 am

Re: Good free site map generator for large web sites

Postby pintobuck » Sun Dec 24, 2017 10:54 pm

Thank you for the suggestions. The free OSMap seems to crawl only "core" Joomla content, and it would not crawl my forum, so the forum was not be included in the resulting sitemap. To crawl and sitemap the forum, or other noncore Joomla content, it seems that we need the premium OSMap, which costs $37 for a 6 months subscription.

sam amaar
Joomla! Fledgling
Joomla! Fledgling
Posts: 1
Joined: Fri Jan 26, 2018 2:44 pm

Re: Good free site map generator for large web sites

Postby sam amaar » Fri Jan 26, 2018 2:53 pm

Jaypatel5 wrote:You have to try a website that generates a sitemap for you. I am SEO expert and I have been using this https://www.xml-sitemaps.com website to generate XML sitemap. I think you should try this.


I used to use this service too often, but in my opinion Built-in sitemap plugins are better for SEO and they're auto upsate and gets faster indexing by Google crawlers.


Return to “The Lounge”

Who is online

Users browsing this forum: No registered users and 6 guests