UTF-8 support in Joomla 1.0.2 -- some characters screwed up

For Joomla! 1.0 Coding related discussions.
Locked
tijs
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 106
Joined: Mon Aug 29, 2005 7:59 pm

UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by tijs » Mon Oct 03, 2005 9:28 pm

Hi,

in the changelog I read that UTF-8 support has been added
30-Sep-2005 Levis Bisson
+ Added UTF-8 support
# Fixed tooltips empty links
# Fixed artf1265 : url in 'edit-menue-item' of submenues is wrong
# Fixed artf1277 : News Feed Display Bad Accent character
I suspect this is creating some problems for me since I had the same problem with another site after I manually changed the english.php file of a Mambo 4.5.2.3 site to set UTF-8 as the character set. Compare http://www.handsoffvenezuela.org/espanol.htm , a Mambo 4.5.2.3 (live) site and http://www.handsoffvenezuela.org/joomla/espanol.htm (Joomla 1.0.2)

My site has several languages such as Spanish, French, Persian, Greek, etc, all of which seem to be screwed up in one way or another. I am not sure why some "foreign" characters turn up OK and others dont't. Greek, for example, is no problem (http://www.handsoffvenezuela.org/joomla ... ika_greek/) but Portuguese (http://www.handsoffvenezuela.org/joomla ... ortuguese/) and Spanish have question marks in the ê, ç, ã, é and è characters (plus some others).

I thought of bringing this to your attention, and I hope to find a solution for this too since this stops me from converting to Joomla.

Terenzusum
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 189
Joined: Thu Aug 18, 2005 4:07 pm
Contact:

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by Terenzusum » Mon Oct 03, 2005 10:28 pm

yah UTF-8 support is the moust important for me to use joomla in production environment

so is it ok now?

or just first stage of UTF-8 implementation in joomla! ?

regards
Poll [ Homepage: http://www.bertran.uni.cc/ORION ]
Why there are no updates on the current donation and add revenue status? helping out Joomla! users, "giving back" to the community...
Testing Joomla! 1.5 beta 2 SVN on http://bertran.uni.cc/j15b2

tijs
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 106
Joined: Mon Aug 29, 2005 7:59 pm

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by tijs » Mon Oct 03, 2005 11:16 pm

Terenzusum wrote: yah UTF-8 support is the moust important for me to use joomla in production environment

so is it ok now?

or just first stage of UTF-8 implementation in joomla! ?
I don't know, but probably it's just a start. I guess from a fresh database it would work OK, allowing you to have UTF-8 support in Joomla (so finally getting rid of problems with multiple languages), but starting from an existing database, it seems to screw some characters up. If I resave content then the actual article comes up OK. So now the job is to recopy and paste all articles in the database. If somebody has a more elegant and especially faster way of fixing these characters, I'd be very grateful not to have to go through the pain of copying and pasting about a 1000 articles  :o

t.

tijs
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 106
Joined: Mon Aug 29, 2005 7:59 pm

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by tijs » Mon Oct 03, 2005 11:31 pm

I stand corrected... resaving content does not seem to solve the issue. However, if you manually switch to ISO-8859-1 encoding then it works fine. In other words, the text in the actual database is OK and displays OK, with all the ñ and ê signs, it is only in the actual rendering in the frontend that things go wrong. But where oh where?

By the way, why is it that http://www.joomla.org/ has the ISO-8859-1 encoding? Is Unicode then not enabled by default after all?

Terenzusum
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 189
Joined: Thu Aug 18, 2005 4:07 pm
Contact:

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by Terenzusum » Mon Oct 03, 2005 11:50 pm

tijs wrote: I stand corrected... resaving content does not seem to solve the issue. However, if you manually switch to ISO-8859-1 encoding then it works fine. In other words, the text in the actual database is OK and displays OK, with all the ñ and ê signs, it is only in the actual rendering in the frontend that things go wrong. But where oh where?

By the way, why is it that http://www.joomla.org/ has the ISO-8859-1 encoding? Is Unicode then not enabled by default after all?

as far as i know their site isn't upgraded to 1.0.2 yet
Poll [ Homepage: http://www.bertran.uni.cc/ORION ]
Why there are no updates on the current donation and add revenue status? helping out Joomla! users, "giving back" to the community...
Testing Joomla! 1.5 beta 2 SVN on http://bertran.uni.cc/j15b2

User avatar
silverfish
Joomla! Intern
Joomla! Intern
Posts: 56
Joined: Wed Aug 24, 2005 6:27 am

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by silverfish » Tue Oct 04, 2005 4:35 pm

I have the same problem. After upgrading from Mambo to Joomla some characters are showing as "?". How do I fix this?

Silver

tijs
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 106
Joined: Mon Aug 29, 2005 7:59 pm

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by tijs » Tue Oct 04, 2005 9:33 pm

Silverfish, if you don't need Unicode you might consider switching back to ISO-8859-1 character encoding, see http://forum.joomla.org/index.php/topic ... l#msg70332 on how to do this.
But as I said in that other thread, that is no real fix for the problem, it is merely switching back to ISO-8859-1, whereas UTF-8 is far superior as a character set since it comprises almost all characters. Hence for multilanguage sites UTF-8 is the way to go!

User avatar
akede
Joomla! Ace
Joomla! Ace
Posts: 1636
Joined: Fri Aug 12, 2005 6:15 am
Location: Munich, Germany
Contact:

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by akede » Wed Oct 05, 2005 4:14 am

Hey all,

the UTF-8 support started with the 1.0.2 code base. It's not a complete version yet, this will come with 1.1.

The reason is simple. If you start switching to UTF-8 in the database as well, you will encounter several addtional problems (sizes of fields and stuff like this). But all this is important for a real UTF-8 support in an environment that you describe.

We are working on this full support and will introduce it soon.

Alex
Joom!Fish 2.0 your free multilingual solution for Joomla! 1.5 i - http://www.joomfish.net - follow us on twitter @joomfish
Meet us at J and Beyond, 30.05 - 1.06.10, Wiesbaden, Germany - http://jandbeyond.org

xender
Joomla! Apprentice
Joomla! Apprentice
Posts: 26
Joined: Wed Sep 28, 2005 10:15 am

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by xender » Thu Oct 06, 2005 1:46 pm

Well, ok, let me try to give you a tip, based on how I deal with switching MySQL databases from ISO to UTF8, both in Joomla and any other CMS. It is probably a very inefficient procedure, but always works for me.

I use for the process an UltraEdit / UEStudio:

1. Dump the database, leaving the orginal encoding
2. Load the file into UEdit, go to File->Convert-> ASCII to UTF-8.
3. See my other post (http://forum.joomla.org/index.php/topic,9052.0.html) on necessary fix to the joomla database. Update the sql.
4. Save the file. Import data back to MySQL. I use phpmyadmin, telling it to use UTF-8 on import (You must have UTF8 enabled in PhpMyAdmin - some old versions don't!)

Now, two issues while importing:
a) If you have a problem with the file saying there are no SQL commands inside, it can be because of the BOM character at the beginning of file. I use Notepad++ to strip the BOM (Format->Ascii Encoding, & Format->Display as UTF8).

Update: Uedit/UEstudio now also can save UTF files with or without BOM (the choice is available in the save dialog).

b) make sure that tables are imported as UTF! If your database doesn't have UTF set as default, you need to force UTF for each table, like that:

Code: Select all

CREATE TABLE `name` (
...
) DEFAULT CHARSET=utf8; 
And voila, you're all set  ;)

Let me know if it was of any help!

Grzegorz

P.S. I just reviewed the 1.0.2 release, and (1) full setup does NOT force the UTF-8 charset in tables, and has the UTF-8 hack disabled in the includes/database.php file, (2) the patch, as far as I can see, does nothing to update to the database to UTF-8, so no surprise your websites crash on non-iso characters.
Last edited by xender on Fri Oct 07, 2005 2:50 pm, edited 1 time in total.

DataHellas
Joomla! Apprentice
Joomla! Apprentice
Posts: 16
Joined: Mon Aug 22, 2005 10:14 am
Location: Greece
Contact:

Re: UTF-8 support in Joomla 1.0.2 -- some characters screwed up

Post by DataHellas » Tue Oct 18, 2005 4:56 am

tijs in the link you gave with a page in Greek, the Greek characters are not displayed correctly! There are displayed as html entites (&#xxx;&#yyy; etc) this is why you see them ok. But there are not! Do a "view source" at your page to see it yourself. Displaying characters as html entites creates other problems: you can not use search and it takes much more space in the database. Joomla has problem with utf-8...

the link you gave:

http://www.handsoffvenezuela.org/joomla ... ika_greek/
Elxis Team


Locked

Return to “Joomla! 1.0 Coding”