SEO: 404 Pages

User avatar
alledia
Joomla! Ace
Joomla! Ace
Posts: 1067
Joined: Tue Jul 18, 2006 3:55 pm
Contact:

SEO: 404 Pages

Post by alledia » Tue Mar 25, 2008 5:00 pm

Short description

Ability to create custom 404 pages as content items

Expected benefits/ Problem solved

1 - Provide better user experience to visitors, by showing them resources available on the site

2 - May help keep a visitor on the site, by providing him with similar url actually existing on the instead. Appropriate url can be identified by doing a search on keywords from the requested url which resulted in a 404. This is facilitated if allowing regular content to be used, as any plugin can be run over such a page

Also, worth considering whether users will be able to define other error pages such as 500.
Joomla extensions and templates: http://Joomlashack.com

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Tue Mar 25, 2008 8:44 pm

Yes on this one. This could be used for site migrations, old J! urls, too.

I'm creating an extension (almost done) that has a new table with old URLs and new J! URLs in it.

In the header for the error.php file, I use the URL to see if there is a new URL available. If so, I issue an 301 header so that Google gets the links updated and redirect to the new URL.

Otherwise, I issue a 404 in the headers (since Joomla! does not do that) and redirect to a 404 page (which you can obviously add the search module on - or decorate with an image, etc.)

This is a good core function. Love to see it in 1.6.

Thanks!
Amy :)

pasamio
Joomla! Ace
Joomla! Ace
Posts: 1318
Joined: Thu Aug 18, 2005 9:27 am
Location: San Jose, CA, USA
Contact:

Re: SEO: 404 Pages

Post by pasamio » Fri Apr 11, 2008 3:54 am

@amy: check out the migration_backlinks table and the backlink migration plugin.

Technically this would be possible right now with template overrides though it would be nicer to have something easily configurable in the admin.
Sam Moffatt
Updater, Installer and Authentication Systems
JoomlaCode Backend Systems
Pie.

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Fri Apr 11, 2008 5:52 am

K - I will look at it, thanks Sam.

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Sat Apr 19, 2008 9:30 am

Steve, would you be prepared to lead a bit of a review of 404's, 403's and 401's also? We've come across a few instances where 403's are being used but maybe they should be 401's (eg, bookmarking a registered menu item, then returning when you are logged out issues a 403 - maybe should be a 401).
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
alledia
Joomla! Ace
Joomla! Ace
Posts: 1067
Joined: Tue Jul 18, 2006 3:55 pm
Contact:

Re: SEO: 404 Pages

Post by alledia » Sat Apr 19, 2008 12:09 pm

I'd be happy to. :) Let me know whats needed.
Joomla extensions and templates: http://Joomlashack.com

User avatar
Jinx
Joomla! Champion
Joomla! Champion
Posts: 6508
Joined: Fri Aug 12, 2005 12:47 am
Contact:

Re: SEO: 404 Pages

Post by Jinx » Sun Apr 20, 2008 9:58 pm

alledia wrote:Short description
Ability to create custom 404 pages as content items
Personaly I don't think this is a good idea. Granted you will add more flexibility, but at the same time you will also add more complexity. I think the first question should not be can we do this, but instead : 'is this really something we want to bother the user with ?'

My answer, I don't think we should. I'll try to explain.

When designing the architecture and UI for 1.5 we divided our users into 4 groups :
  • users
  • administrators
  • designers
  • developers
Each group has it's specific needs and skills and each uses the system in a different way. Users work with the frontend of the site, administrators have access to the backend and perform management tasks, designers work on a template level and developers work on a code level.

Why is this important ? I believe that the user experience should be specifically targetted towards each group. This also means that the experience for each group should not be mixed with that of another. For example, the system shouldn't offer the possibility for users to make design decisions. Like setting css classes for example, or like being able to change css files, or the actual template. Why ? Because doing so would mean the user needs to understand what css is. Those responsabilities belong to the designer and his preferred tools will be an ftp/ssh client and a decent css and html editor.

This is also why I don't believe that one should be able to edit css files or the template form within the backend But .. I'll hear you think this can be usefull sometimes. I'm not going to argue that. Does it pass the 80/20 test ? I would argue not.

In the case of error pages I would argue that it's not the user/administrators responsability to change them. Bascially a user or administrator shouldn't even need to know what an error page is ... and most users and administrators don't. Try asking one of your friends who know and use Joomla! what a 500, 404, 403 is and what the difference is and see what they say ...

Throwing error pages is the responsability of the system and of the developer designing the code. It's his responsabilty to decide if a 500, 404, 403, ... needs to be thrown.

But what about the look and feel of the error page ? Well, look and feel === design === the designer. So, it's the designer who should match the error page to fit the design of the website.

Conclusion, should we allow the ability to create custom 404 pages as content items ? No, we should not. Should we allow designers to customise the error pages ? Yes ! Can Joomla! 1.5 do that, yes it can.

I rest my case ;)
Johan Janssens - Joomla Co-Founder, Lead Developer of Joomla 1.5

http://www.joomlatools.com - Joomla extensions that just work

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Sun Apr 20, 2008 11:11 pm

More complexity? I'm not certain the extent of this problem is appreciated.

Today, Joomla! returns an incorrect HTTP status code in response to a 404 or 403. It returns a '200 - the server successfully returned the page' for error pages. This is problematic for those working with Google Webmaster Services and trying to get a sitemap resolved. Joomla! should issue header("HTTP/1.0 404 Not Found");

I think it's too complex to expect every single site developer (or designer if they have one) to update the error.php file to add a set of PHP IF statements to trap the appropriate errors and issue header statements like header("HTTP/1.0 404 Not Found"). That is beyond the skillset most designers require. And, unnecessary, IMO, since adding such code once to core should help everyone.

Taking a step back, let's recognize that this problem is common across for all CMS's. Apache can't handle the 404's since the pages are not static and Apache has no idea if it's a valid URL, or not. So, it becomes the job of the CMS and traffic is handed to the CMS. WordPress and Drupal have elegant solutions with defaults that no one has to bother with if they don't need to tweak the logic. The correct HTTP status code is issued, then an automated redirect brings the site visitor to a nice, soft landing on a page where a search option or menus can help them find what is needed. (Whatever the site developer deems appropriate.) (And, yes, the designer, if they have one, can make the page look like whatever is desired.)

I think your point is a good one that end users should not be bothered with 404s and 403's and 500 errors. But, on the vast majority of Joomla! v 1.5 today, most of them are. Getting dumped off at a dead end with a big red box and cryptic codes is bothersome.

Here's the real plus to moving forward with this project --> doing so sets up URL migrations for all those people who are switching over to Joomla! and have a list of old URLs that need to redirect to new Joomla! URLs. Smart.

Of course, the real question will be: is there a free software developer who thinks this is interesting enough to pick up the challenge and run with it. I think this area holds such promise for all Joomla! Web sites that we will see one of our devs run with this one.

Nice to see you around, Johan,
Amy :)

User avatar
alledia
Joomla! Ace
Joomla! Ace
Posts: 1067
Joined: Tue Jul 18, 2006 3:55 pm
Contact:

Re: SEO: 404 Pages

Post by alledia » Sun Apr 20, 2008 11:25 pm

I've talked through the possibilities with Johan and came to the conclusion that there are two separate options (I've excluded the possibility of including the error pages in the template folder for the reasons that Amy mentions above)

1) Use Joomla articles as the error pages. Through an option (in Global Configuration?), people could choose which article corresponded to which error code.

Pros: Great flexibility to design and manipulate the error pages.
Cons: Potentially confusing overlap between articles / error pages.

2) The errors are handled by the template and the language for the error page is included in the site's language file.

Pros: Non-technical site admins have less to worry about. No confusing overlap between articles / error pages.
Cons: No ability to create error pages of any complexity (search boxes etc.) because of the limited code that can put in the language file.

Either way its an important issue as Amy points out ... 1.0 has no error codes and last time I checked they didn't work correctly in 1.5.

I'd love to hear people's feedback on the two alternatives or whether there may in fact be a third option ...
Joomla extensions and templates: http://Joomlashack.com

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Mon Apr 21, 2008 12:00 am

I think a third alternative is to create a new core component that allows you to configure how each of these error codes are handled.

The Bug Squad has had extensive discussion on 403's alternatives, already. A Site Developer might want a 403 treated like a 404 so as to not confirm the presence of a page that they do not wish confirmed. Or, a site developer might want a 403 to redirect to a logon page. Or, a 403 might go to the default 403 page which has [insert message here where a common message is already there at default.]

For 404's, you might want to a) first see if the requested page is in this list (a new table), and if it is in that table, then grab the new URL, issue a 301 header so Google knows the page has been permanently moved to the corresponding URL's location and redirect to the new page (that's how site migrations could easily be slipped into this solution) or b) go to the default 404 page which has [insert message here where a common message is already there at default.]

And so, on.

A new component would enable normal Joomla! configuration where modules could be shown or not shown on the page. Template designers could eliminate extra work of copying the error.php file in and completely styling it since the pages are already part of the normal application.

Sam has logic already for legacy URL migrations and I wonder how that might be leveraged here. It will take a developer to consider the best approach, but I do agree that there is lots of benefit to considering friendlier approaches.

Thanks for raising this, Steve,
Amy :)

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Mon Apr 21, 2008 1:27 am

alledia wrote:I'd be happy to. :) Let me know whats needed.
I'll be in touch :)
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
Jinx
Joomla! Champion
Joomla! Champion
Posts: 6508
Joined: Fri Aug 12, 2005 12:47 am
Contact:

Re: SEO: 404 Pages

Post by Jinx » Mon Apr 21, 2008 3:03 pm

AmyStephen wrote: Today, Joomla! returns an incorrect HTTP status code in response to a 404 or 403. It returns a '200 - the server successfully returned the page' for error pages. This is problematic for those working with Google Webmaster Services and trying to get a sitemap resolved. Joomla! should issue header("HTTP/1.0 404 Not Found");
I have tested a local 1.5.2 install fresh from SVN and I cannot replicate this behavior. I'm getting correct 404, 403 or 500 headers as expected.
AmyStephen wrote: I think your point is a good one that end users should not be bothered with 404s and 403's and 500 errors. But, on the vast majority of Joomla! v 1.5 today, most of them are. Getting dumped off at a dead end with a big red box and cryptic codes is bothersome.
I don't argue that, styling these pages for consistency with the rest of the site is the task of the designer. The flexibility to do so is there. A first step to get designers to pick this up might be more advocacy. Seems most people don't realise these pages can be easily overridden.
AmyStephen wrote: Nice to see you around, Johan,
Amy :)
Your welcome, I'm just sticking my head in. Steve asked me to comment on this. Still enjoying some R&R.
Johan Janssens - Joomla Co-Founder, Lead Developer of Joomla 1.5

http://www.joomlatools.com - Joomla extensions that just work

User avatar
Jinx
Joomla! Champion
Joomla! Champion
Posts: 6508
Joined: Fri Aug 12, 2005 12:47 am
Contact:

Re: SEO: 404 Pages

Post by Jinx » Mon Apr 21, 2008 3:11 pm

AmyStephen wrote:I think a third alternative is to create a new core component that allows you to configure how each of these error codes are handled.
You serious ? That would mean throwing in even more complexity both on a UI and code level. Let's try to come up with KISS solutions.
The Bug Squad has had extensive discussion on 403's alternatives, already. A Site Developer might want a 403 treated like a 404 so as to not confirm the presence of a page that they do not wish confirmed. Or, a site developer might want a 403 to redirect to a logon page. Or, a 403 might go to the default 403 page which has [insert message here where a common message is already there at default.
As you correctly point out 'a developer might want ...' to me this means a code solution not a UI setting. Adapting the way the way Joomla! handles error codes can be done by developing a system plugin to extends or change the core behavior. The plugin could contain conditional logic to decide how it needs to deal with error codes. This is how Sam's backlink plugin work for example.

I consider this off-topic to the question raised by Steve.
Johan Janssens - Joomla Co-Founder, Lead Developer of Joomla 1.5

http://www.joomlatools.com - Joomla extensions that just work

User avatar
Jinx
Joomla! Champion
Joomla! Champion
Posts: 6508
Joined: Fri Aug 12, 2005 12:47 am
Contact:

Re: SEO: 404 Pages

Post by Jinx » Mon Apr 21, 2008 3:39 pm

alledia wrote: I'd love to hear people's feedback on the two alternatives or whether there may in fact be a third option ...
Here is one :

What about implementing a meta-refresh redirect in the error page ? This would allow full flexibility and limit the complexity to a minimum ?

Advantages :

- It would not only allow to redirect to an article but also to any other part in the system, for example after a 403 you could redirect to a login screen, storing the initial url in the session and after the redirect return to the originally requested page.

- The administrator will not need to understand a the meaning of the error codes. He just needs to know that the system can throw errors and that he can direct the system after an error to another page. Which will improve the users experience on the site.

- The meta-refresh allows us to keep the existing behavior, this is important for developers since the error pages are also used to output extra debug information. In case debug is enabled the redirect would not be outputted in the error template and no redirection would happen allowing the developer to examine the debug output.

I personally feel this approach gives a lot of benefits and allows us to keep things KISS. If this approach is chosen, the next decision would be what extra settings we want to expose to the UI and where to put them. My proposal :

1. Put in a simple setting in the Global configuration/site settings and choose the default redirect for the meta-refresh. Don't split this up by error code. Just allow people to enter a URL.

2. Add an error plugin that is capable of extending the error refreshing behavior. This plugin would basically by a router plugin that can decide based on it's settings how to rewrite the meta-refresh redirect. The plugin could expose error codes and allow for more fine-tuned behavior.

Why choose this double approach ? Well, you keep the UI simple, super admins can further tweak things by the settings in the plugin and developers could extend the plugin for their client installations based on their needs. Roles are separated, everyone is happy and Jinx can go drink some more Rum and enjoy his R&R ;)

Johan
Johan Janssens - Joomla Co-Founder, Lead Developer of Joomla 1.5

http://www.joomlatools.com - Joomla extensions that just work

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Tue Apr 22, 2008 3:42 am

Jinx wrote:I consider this off-topic to the question raised by Steve.
I'm expanding the topic to include other error codes so this is not a problem.
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
Jinx
Joomla! Champion
Joomla! Champion
Posts: 6508
Joined: Fri Aug 12, 2005 12:47 am
Contact:

Re: SEO: 404 Pages

Post by Jinx » Tue Apr 22, 2008 6:53 pm

I was not refering to other error codes, but to the part of Amy's reply I quoted where she was talking about advanced error routing. Could we please correctly define what we are talking about here. Personal I'm going from the assumption that the topic at hand is :

Finding a solution to allow for the creation of custom error pages.
Johan Janssens - Joomla Co-Founder, Lead Developer of Joomla 1.5

http://www.joomlatools.com - Joomla extensions that just work

User avatar
brad
Joomla! Master
Joomla! Master
Posts: 13272
Joined: Fri Aug 12, 2005 12:38 am
Location: Australia
Contact:

Re: SEO: 404 Pages

Post by brad » Tue Apr 22, 2008 10:07 pm

I've been following this thread since the start and want to chime in now.

Background:

When we moved from Mambo to Joomla is was the first time we (Core Team) had control of the servers that ran our sites. One thing that we noticed pretty soon was how badly Joomla handled 404's. As noted above, the correct error code was not being returned to the browser either, despite the page title being 404.

.. anyway.. this was rectified when we (devs) switched to using apache to supply the valid 404. ie see: http://www.joomla.org/testing404.html <-- ugly, but for a server guy, beautiful as there is no load on the server to generate this page. Nothing even goes via php.

Bottom line, from a server resource usage POV Joomla 1.5 should be able to handle 404's but giving the correct error code to the browser. We run a few large Joomla sites of clients, and this is the reason many of them do not use 3rd party SEF as the piping of the 404's via php adds to the loads.

Perhaps we might consider a switch to enable the larger sites to just fall back to apache/webserver to handle the 404's? It's not that hard to specify in your .htaccess file an alternative styled 404 page either. In any case, I hope you get where I am coming from and take this into consideration. I think it is vital to consider. Oh and finally: http://www.google.com/testing404.html even Google uses the default apache handling of their 404's, not that we should not be addressing this issue.. ;)

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Tue Apr 22, 2008 10:43 pm

Brad, in order for the 404 to be generated, Joomla! actually does have to fire up and send the header back. Will the web server be able to intercept this or will that require special configuration?
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
brad
Joomla! Master
Joomla! Master
Posts: 13272
Joined: Fri Aug 12, 2005 12:38 am
Location: Australia
Contact:

Re: SEO: 404 Pages

Post by brad » Tue Apr 22, 2008 11:25 pm

What I mean is in 1.5, when you choose to add the extension to the url, there should be an optional line to add/comment out in the htaccess file. Otherwise the server could be parsing url's that you could avoid.

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Tue Apr 22, 2008 11:32 pm

Ah, ok. That's a good point. That's worth looking at for sure.
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
pvh123
Joomla! Ace
Joomla! Ace
Posts: 1156
Joined: Wed Oct 05, 2005 7:25 am
Location: Emmen

Re: SEO: 404 Pages

Post by pvh123 » Wed Apr 23, 2008 11:52 am

Hi All,
Reading through this post, I was missing something. The red thread is how to cover the 403's and 404's. The importance to come to a good and easy to build and accepted solution is paramount, obviously.
What I am missing is the fact that when we have covered the 403 and 404 (and indeed any error page) in a nice way, it would be very important for the Administrator to know which part of the website is throwing the 403 and 404's, etc.
So, we need some sort of communication to the Administrator for him to solve the problem at the base. Because , me as a user , gets very annoyed if I go back to a website after a period and find that the webmaster/administrator is still giving me his nice alternative page but not having solved the problem.

So, my view in compressed mode:

- Based on the above a " standard" possibility for the layman J! website builder to cover the 403/404.
- A expert interface giving the possibility to build a more sophisticated 403/404 error page containing an alternative into the website again; we do not want to loose our visitors ;)
- Logging, either via a logfile or underwater email to the administrator, the 403/404's that have occurred.

All this toward a, towards the visitors, self repairing and never failing website .
Note: Sending me private messages for personal support, without this been requested, would lead to ignoring any of your posts in future!

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Wed Apr 23, 2008 10:00 pm

Logging is a good suggestion pieter. Because these don't get logged in the web server error log there is no way to find out what's going wrong (or if someone is trying something funny).
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Wed Apr 23, 2008 11:27 pm

Just want to let everyone know that I've asked Steve to help facilitate this discussion. He's going to manage the "brainstorming" for a while to get a good view of what the big picture issues are. Then we'll need to hone those ideas into what are potential deliverables for 1.6 (or maybe even a longer term plan spanning multiple versions).

I'm targetting a Beta for 1.6 at the end of July so there are a few weeks that this topic can bounce around, but after that we need to make sure we have enough time to actually plan and implement what's needed.

Anyway, thankyou Steve for doing this ... and over to you :)
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.

User avatar
mcsmom
Joomla! Exemplar
Joomla! Exemplar
Posts: 7897
Joined: Thu Aug 18, 2005 8:43 pm
Location: New York
Contact:

Re: SEO: 404 Pages

Post by mcsmom » Thu Apr 24, 2008 2:51 pm

First, what are the goals? In my view, they should be for those errors that end users and searchbots see
1. Give the user appropriate useful information in a visually attractive format
2. Send the appropriate header to both the server log and the searchbots.

Second,
What are we talking about? The 404s (and 403s and even 401s):
1. That come from apache (such as when the user types mydomain.com/cat.html and there is no such page)?
2. That come from Joomla! and are the intentional ones that are in the code and generally related to user rights being checked in the code?
3. The ones that happen when someone types mydomain.com/index.php/cat.hml and there is no such page?
Of course I think all of the above should be dealt with, but they really are almost completely separate issues.

Also, this should be about both with and without SEF since things are currently handled somewhat differently for them.
For example
/index.php/cat.hml
with SEf on returns the system error page and the 404 header but with it off it returns the index.php and a 200.

There are a lot of issues to think about here, such as how important is it that the proper 40x header be thrown for a given situation (such as do we want to continue the practice of using 403 for not authorized when properly that is a 401 and 404 for forbidden when what is properly 403).

Also, there are places in the code where redirects are being used and thus improper 30x headers are being sent. This is an especially bad idea if you are redirecting to the index.php page since if you do it a lot that is probably going to cause loss of page rank. If you must do it, do it to the login page or some other page that you don't care about but better to keep that page out of the google index.

Also, while I think it would be lovely for every site to have a staff of three, we need to understand that most of the time that is really one person who is also the same person who chose Joomla! and who probably also reboots the router when necessary

Interesting reading:
http://www.stonetemple.com/articles/int ... utts.shtml (see the part about 301s versus meta refresh)
http://www.mmt.org/special_features/new ... or_not.php
So we must fix our vision not merely on the negative expulsion of war, but upon the positive affirmation of peace. MLK 1964.
http://officialjoomlabook.com Get it at http://www.joomla.org/joomla-press-official-books.html Buy a book, support Joomla!.

User avatar
alledia
Joomla! Ace
Joomla! Ace
Posts: 1067
Joined: Tue Jul 18, 2006 3:55 pm
Contact:

Re: SEO: 404 Pages

Post by alledia » Fri Apr 25, 2008 3:24 pm

mcsmom wrote:First, what are the goals? In my view, they should be for those errors that end users and searchbots see
1. Give the user appropriate useful information in a visually attractive format
2. Send the appropriate header to both the server log and the searchbots.
A very good summing up. I understand Brad's point that sometimes the default error page is needed to reduce server load.

However, large professional sites have error pages that maintain the look and feel of the main site. Any visitor sent to an apache error page is a lost visitor because they don't have any navigation to send them to a valid page. Some examples of how big sites will keep their navigation:

http://www.washingtonpost.com/3f3gdfgfg
http://www.latimes.com/3f3gdfgfg
http://sportsillustrated.cnn.com/3f3gdfgfg
http://www.facebook.com/3f3gdfgfg
http://www.collegehumor.com/3f3gdfgfg
http://mashable.com/3f3gdfgfg
http://www.techcrunch.com/3f3gdfgfg
Joomla extensions and templates: http://Joomlashack.com

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Fri Apr 25, 2008 4:08 pm

With dynamic Web pages, the Web server cannot determine whether or not a page exists, or not. The application has to figure it out and, unfortunately, that does mean server resources. There are some cases where one will get an Apache error message (for example, try navigating to an .asp page). But, for the most part, Apache (or IIS) can't help, anyway.

User avatar
alledia
Joomla! Ace
Joomla! Ace
Posts: 1067
Joined: Tue Jul 18, 2006 3:55 pm
Contact:

Re: SEO: 404 Pages

Post by alledia » Fri Apr 25, 2008 4:21 pm

Given that the extra use of server resources is inevitable, it sounds like we should definitely plan on including an "off" switch for the 404 pages.

Maybe the 404 option could be included as a plugin?

The parameters could be set inside the plugin (keeping the Global Configuration simple). Even the text for the error page could be inside the plugin parameters (thus not leading to any confusion in the Article Manager). The plugin could then be unpublished if needed.
Joomla extensions and templates: http://Joomlashack.com

AmyStephen
Joomla! Champion
Joomla! Champion
Posts: 7018
Joined: Wed Nov 22, 2006 3:35 pm
Location: Nebraska
Contact:

Re: SEO: 404 Pages

Post by AmyStephen » Fri Apr 25, 2008 5:51 pm

Steve -

My point is we cannot avoid using additional resources.

With static Web sites, a Web server can tell by the names of the files on the Web site whether or not a page can be found. The files are there - it checks - if it doesn't find the file - it issues a 404. It's lower in resources since no database call is issued and it's pretty straight forward logic.

With a dynamic Web site, the Web page does not physically reside on a file server. Determining 404's is beyond the ability of Web server in a dynamic Web site. So, the Web server has to turn this job over to the application (in this case, PHP) to determine whether or not the request is a real page, or not. Only the application knows if it is a 404. It takes logic, database calls and additional processing to figure it out.

The resources are burned by the application making that determination. Presenting the 404 page, itself, is not the resource intensive piece.

For site migrations, it could be lower server usage (?) to have Apache use an .htaccess file with a list of 301 redirects. That would slow every page request down, though, since the list would be checked first, before the application was even involved. It's possible that would result in lower server overhead, though. I don't know.

If that is still confusing, I'll look for a graphic or maybe someone else can explain it more clearly.

Thanks for helping lead this effort, Steve, you are an obvious perfect choice.
Amy :)

User avatar
pvh123
Joomla! Ace
Joomla! Ace
Posts: 1156
Joined: Wed Oct 05, 2005 7:25 am
Location: Emmen

Re: SEO: 404 Pages

Post by pvh123 » Fri Apr 25, 2008 8:20 pm

Although you are very "dynamic" as usual inexplaining why dynamic websites do not know things and static websites do, what about this thought:
I appreciate that a webserver only knows what he gets, right? I think we all agree :)
Its the app, in this case J!, that is supposed to make sure that the webserver gets the correct thing. I do not believe that our discussion is about what the webserver should do or should have intelligence about.
If a 40n page is issued to the webserver, decision time about how that page looks has been passed and therefore the way an error page looks, is handled in the app.
It would be different if the webserver starts throwing 404/403 error pages on its own account, that would be the last time that this (thinking) webserver would have a sayso in my book.
Just tell me when I am wrong, but the decision to send an error page to the webserver is taken by the Joomla code because it cannot completely handle the page request for some reason. And I believe the reason is known, otherwise it would not make that decision. And so on, and so on.
Note: Sending me private messages for personal support, without this been requested, would lead to ignoring any of your posts in future!

User avatar
masterchief
Joomla! Hero
Joomla! Hero
Posts: 2247
Joined: Fri Aug 12, 2005 2:45 am
Location: Brisbane, Australia
Contact:

Re: SEO: 404 Pages

Post by masterchief » Fri Apr 25, 2008 10:57 pm

AmyStephen wrote: My point is we cannot avoid using additional resources.
Actually you can make some savings. If you do not turn on the "Add Extension" option, you can tune your htaccess directives to let the web server handle any page with an extension, and Joomla! handle any page without. If you have "Add Extension" on, then you definitely have to parse almost every URL (.html, .pdf, etc). That was Brad's point.
Andrew Eddie - Tweet @AndrewEddie
<><
http://eddify.me
http://www.kiva.org/team/joomla - Got Joomla for free? Pay it forward and help fight poverty.


Locked

Return to “Accepted - Archived”