Joomla! Discussion Forums

thanks! so its utf-8.

i will push through my plan to translate joomla in our local dialect (Bangsamoro) soon.

Dear info graf

i found new error in save data into mysql 5.0.x , when i set my encoding utf-8 in our page the data was sucsess add to data base but when i do edit the data in mysql with phpmyadmin 2.6 all data is not set and charcter have an eror .
the data when save into data base use latin1_general_ci not utf8_persian_ci.
could you tell me 

I am assuming that you are referring to utf-8 in Joomla 1.0.x

The issue is quite simple - Joomla 1.0.x does not support any multibyte character encodings such as utf-8. This support will come in version 1.1.

However, one can configure Joomla 1.0.x in a reasonable way to support utf-8. The database needs to be made utf-8 compliant. Existing data needs to be converted to utf-8. Connection to the database needs to be encoded utf-8. Language files need to be converted to utf-8. Even after all this most of the components are not utf-8 compliant and php string functions are not utf-8 compliant which could cause some artifacts and logical errors.

A quick guideline to getting Joomla 1.0.x to work with utf-8 is as follows:
- use MySQL version 4.1.2 or newer (older versions don't support utf-8).
- create an empty database manually before installing Joomla. Set the character set to utf8 with some collation (utf8_general_ci is the default and should be OK. If a specific utf8 collation exists - it can be used).
- convert the language files to utf-8 (all language files including for editors, components etc.)
- Install Joomla using the pre-existing database
- set 'charset=utf-8' in the _ISO define in the language file
- You should uncomment one line of code in the includes/database.php file at about line 102 (second line below)


Please note that the above does not make Joomla 1.0.x fully utf-8 compatible. All string functions will still be using singlebyte character functions. This works well in most cases. There could be some instances of garbage characters especially with diacritic Latin characters.

Is it working with mozila firefox.
Working only with IE

Welcome on the Joomla forums.
As far as I am aware, this not related to the browser.
Please browse my sig url with either browser: you should get the same results.

What about the integration  inside joomla of any script (ie forum/gallery)
that does not fully support utf-8 ?
Are you sure it's wise to force every user to use  utf-8 ? 
Also I have a joomla website mixing arabic and french, and it is  perfectely displayed with charset=iso-8859-1
(I've made some quick fix though  )
So why should I absolutely switch to  utf-8 ?
Looks boring: will have to convert forum, gallery , any component using a langage file ....

By the way, infograf768, I took a look to your sig... What is freeway ?

Nobody forces anyone to use utf-8 for Joomla 1.0.series. 
It will be used as default for 1.5 though.
And, yes, quite a few Extensions wil have to be rewritten to accomodate this vey needed feature for Mutiple language sites.
BTW, some are already utf-8 aware.
Your Arabic-French site will really be different in 1.5, and without any hack

OT: Freeway is a WYSIWYG Macintosh commercial app which lets users create static sites almost as if they were using QuarkXpress.
It includes a lot of "Actions" (JS) which lets customize the output and lets one integrate in a very friendly way any type of media.
One of its great advantage is the creation on the fly of gif text, which remains editable in the app.
Some aspects of xhtml content are very difficult to integarte in a CMS. This where Freeway comes handy.

Mehdi hello,

Going for utf-8 was a strategic decision as this is where the web is going and it is the only way to truly suppport complete localisations and multilingual support. The price to be paid is that extensions will also need to support utf-8. However utf-8 does provide backward compatibilty to anything using iso-8859-1 provided that there are no extended ascii characters (in other words English only).

It is interesting to note that you have both Arabic and French on a site using iso-8859-1. The implication is that all Arabic texts are most likely stored as NCR codes (similar to HTMLentities) which is extremely inefficient and can result in data truncation. Have you tried using 'windows-1256' instead?

Cheers,

yes, I use NCR for Arabic. What's wrong with this ?
I've fixed almost anything  in Joomla while using NCR.
The hardest problem I got to fix was with title not displaying properly just after article was stored (but fixed by me here)
The only remaining  problem  is that  big Arabic titles are truncated ....
I think that's  because data corresponding to title in database  is limited to a certain size (and with NCR it takes more place)

Concerning windows-1256... I didn't try before launching my website. Is it a mistake ?
Anyway I don't know how I would convert my database to support this.
Just for Coppermine, it was a true pain to learn how to convert my database into utf-8 (after upgrading my gallery, now the standard for Coppermine is utf-8). Although I've finaly succeeded to perform conversion, I 've finally decided to stick  again to iso-8859-1  (for integration with Joomla) ; at least until Joomla 1.1 is released. OOOps I meant 1.5. ( I forget from times to times)

NCR codes use 6-7 bytes for each character that would otherwise be represented by one byte in encodings such as iso-8859-1 or windows-1256.

This has two effects as content in Arabic now uses 6 to 7 time more space:
1. Data truncation will for sure occur in the cases that the database uses fixed length character fields (of type varchar). If you fill a 20 character field (i.e one of the contact fields) to the limit with 20 Arabic characters - after saving and reloading the string will be truncated.
2. In the cases that variable length character fields are use (i.e. content items body) there is no limit and truncation will not occur. However 7 time more space is used in the database (which slows it down) and time to transmit the page is also increased accordingly.

Using NCR codes is only recommended when short amounts of data (in Arabic for example) are needed in a site that pedominantly uses another encoding. Care must be taken on fixed length fields.

If you have large amounts of data already in Arabic it would require converting the content in order to change to windows-1256. There would be no need to touch the database and it is perfectly OK to have it using charset 'latin1'.

Cheers,

Hi thanks for your explanations...
But I've just realized something.
Even if I convert my whole website into utf-8,
all articles written in NCR will stay in NCR.
The fact is, that to perform the conversion I just do
a sql dump of the whole website and then with my favorite editor
I  do the conversion ASCII ---> UTF8  (+ collation, + type of created tables)
before restoring the database.
Unfortunately all articles written in NCR will stay in NCR.

So if  I want to have the  full benefit of Utf-8 I should perform two conversions:
iso-8859-1 (with NCR) --> windows-1256 (get rid of NCR) ---> utf-8
Isn't it ?

Problem is that I performed a google search and didn't find any editor/tool that would allow
to convert  easily and quickly all these NCR characters into single byte characters...

Any hint ? Thanks...

I suggest using a freeware editor named "BabelPad"  http://www.babelstone.co.uk/Software/BabelPad.html

If you take the entire sql dump, select it all and use the NCR to utf-8 function it will translate all codes and not mess with the rest.
It's quite an amazing tool. Play with it a bit to get to know what is what.

Thanks davidgal,
what I could do with BabelPad
is selecting the whole sql dump
and then do NCR --> to Unicode.
And then I don't know if it would  be wise to do
also HTML Entities --> to Unicode.
Unfortunately Babelpad doesn't handle ASCII --> utf-8 conversion.
This mean I would have to play with two tools.

It's quite surprising , I didn't find any small tool that allow  same functionalities as Babelpad.
I will contact Ultraedit developers and ask them  if they can make an effort 

Hi,

Quite true! There is a need for one tool for the ASCII to utf-8 conversion and then the BabelPad can do the translation of the HTML and NCR codes.

I'm glad you found the tool useful

I played a bit with Babelpad ...
Babel support conversion ASCII ----> UTF-8
It's  just  hidden in  the "save as" menu....  :laugh:
If enconding is not unicode(i.e utf-8 or utf-16 LE or utf-16 BE) , and that you press directly save, your text is automatically saved in utf-8.
Sometimes it refuse to directely save, and open the save as dialog
(should I understand this by a warning: "take care this operation
might not be harmless ?"). I don't how reliable Babelpad  is for a big sql dump ...
I say this because I've tested before the tool iconv and I got problems after conversion with it.

BTW  using  windows-1256 is a bad idea. Seems that the conversion windows-1256 ---> utf-8 not obvious.
Iconv allows this, but I don't rely on that tool.
So if plan to switch later to utf-8,  I'd better stick with iso-8859-1 and my NCR characters.

Hello All!

is there a release of joomla 1.1 in UTF? or is it abandoned?

Thanks!

Peleg.

1.1 was the original version number and was never released.

1.5 it is and still beta for now. 

thanks!