Problem with greek and cyrillic characters in Fireboard

ivo.apostolov · Post by **ivo.apostolov** » Wed May 30, 2007 11:49 pm

It is pretty strange, I work on 1.5 Joomla with latin collation and Cyrillic and have no problems, where is the difference between the armenian and the rest?

aravot · Post by **aravot** » Thu May 31, 2007 12:02 am

Our alphabet is different http://en.wikipedia.org/wiki/Armenian_alphabet

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 12:06 am

Yes, every alphabet is different, but it is not a reason something to not work. Can you give me an example of 1.5 installed on MySQL with collation latin for example?

Ivo

aravot · Post by **aravot** » Thu May 31, 2007 12:23 am

Let me install it on a live site, will PM you the link once done (won't be today)

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 12:27 am

OK, I am performing a test on that right now. Can you PM me any page that has massive Armenian text on it.

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 1:32 am

Ahhhh, now I get the issue.

1. During installation Joomla! attempts to change the database collation to utf8_general_ci, but from my past experience (pre Beta2) this was not always successful (need to perform additional tests on that).
2. If Joomla! makes it to change the database collation to utf8_general_ci, there are no issues at all.
3. If Joomla! doesn't makes it, problems start to appear with 3PD extensions, because in their queries usually there is no collation set. This is pretty normal as in 1.0.x series we expect the component to work with any collation and to take the database default collation (which is supposed to be correct). In the Joomla! 1.5 SQL files you can find the following code:

Code: Select all

TYPE=MyISAM CHARACTER SET `utf8`

which makes everything to work excellent with the core staff, but is missing in 1.0.x extensions and causes problems.

I performed a test regarding the Armenian staff and the report of it can be found at:
http://armenian.yantra.biz/TestReport.pdf

In this particular case, Joomla! was able to change the database collation from latin1_swedish_ci to utf8_general_ci, but I am not really sure if it works all the time and.

So if the questions is: Does Joomla! require the database collation to be utf8_general_ci?
And the answer: Joomla! 1.5 itself doesn't require that, but its current extensions DO.

Edit: Typos

aravot · Post by **aravot** » Thu May 31, 2007 1:50 am

Ivo thanks for testing this.

yvolk · Post by **yvolk** » Thu May 31, 2007 6:26 am

ivo.apostolov wrote: There is no issue with it (if else and someone encounters issues with it please post details here). What I mean is that the database collation could be latin, windows Cyrillic or anything, but Joomla! API saves and gets the information from there as UTF.

Ivo, it is VERY easy to reproduce the problem:
1. Install Joomla 1.5 beta 2, creating new database with "latin1-general-ci" collation (this is what users get by default!)
2. Open the site
3. Login as Admin, go to the menu "Menus"->"Main Menu"
4. Click "The News" Menu Item.
5. Change "Title:" field from "The News" to "Новини"

6. Press "Save" button.
7. See, what you get in the menu item's title ("? ? ? ? ? ?")
8. Reopen The bug tracker Item

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 11:09 am

Yuri, I did that already, I pretty fine get the issue at the moment, but the idea was how to explain it to the dev team.

yvolk · Post by **yvolk** » Thu May 31, 2007 1:27 pm

ivo.apostolov wrote: Yuri, I did that already, I pretty fine get the issue at the moment, but the idea was how to explain it to the dev team.

Maybe, my above posted sequence of steps to reproduce the bug is enough?

I don't know exactly, what "collation" in MySQL Server means (and I don't think I need to know in this case), but the "collation" may be checked and changed through common tools (MySqlAdmin...) and "wrong collation" leads to the error, that may be easily reproduced.

Jinx · Post by **Jinx** » Thu May 31, 2007 1:43 pm

Guys,

Hold your horses here, this is not a collation problem it's a character set problem. Collation only defines sorting order it does not define the actual database data. What happens with 1.0 extensions is that they need to change their installation script and add a install.utf.sql (for MySQL 4.0.17 and up) and install.nonutf.sql file (for lower then MySQL 4.0.17). The only difference between both is that for the install.utf.sql you need tp set the table charset to utf8 using :

Code: Select all

CHARSET=utf8

This is all carefully explained on the wiki : http://dev.joomla.org/component/option, ... nes:utf-8/

Hope this helps to shed a light on the issue.

Johan

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 1:51 pm

Johan,
two questions came up here:
1. If it is not a collation, but encoding, why it is written as collation in phpMyAdmin? Is this some kind of misunderstanding there? Just don't get it. You say it's encoding, but phpMyAdmin says it is collation.

2. What you basically say is that whatever the encoding of the database (which is mentioned in the above posts and understood as "collation") is not utf8, the 1.0 extensions all need re-write even to work in legacy mode?

3. Is Joomla! changing the database encoding (or what we understand as "collation") to utf8 during installation?

Thanks

Edit: Attaching two screenshots, where in phpMyAdmin, the value that is actually the one controlling how the data is stored is named collation

Jinx · Post by **Jinx** » Thu May 31, 2007 2:25 pm

Oke I'll try to explain, we are talking about two diferent things here :

1. Character set = the character set of a database/table defines the encoding for that database/table, for Joomla! 1.5 to work with all languages this needs to be set to utf8.

2. Collation = the collation of the database/table defines the sorting order when sort queries are preformed, this is because in different languages letters are sorted differently.

Both are two different settings and should not be confused. The problem we are talking about here is character set not collation. If your collation isn't set correctly it does not have any effect on the output of the data, it can only have an effect on sorting.

The problem with the character set is that Joomla! 1.5 needs it to be utf8, and to make sure it is utf8 it forces the charcater set to utf8 on a per table basis. Joomla! 1.0 on the other hand does not care about the character set of a table and relies on the global character set for the whole database.

Joomma! 1.0 extensions will work in 1.5 as expected. Meaning that they will not support all langauges, this is the same situation as in 1.0. If a extension developer want to support full internationalisation he needs to make a small change to his installation script as explained in the link i provided above. This change is BC and does not have any effect on Joomla! 1.0.

So to answer your questions :

1. It's not collation, but character set. phpMyAdmin mentions only the collation for each table, the character set can only be set on a per table basis since MySQL 4.0.17. I'm unsure if the newer version of phpMyAdmin already support this.

2. As explained collation and character set are two different settings with different usage. If the character set of the table is not set to utf8 then problem will occur with international langauges. To make sure it is set to utf8 a extension needs to force it's tables to utf8 using a special sql script as explained in the link. If it doesn't the situation will be the same as in Joomla! 1.0, meaning it could work depending on the setting of the database or it could fail.

3. It's a bit more complicated then that since 1.5 supports all MySQL versions. But simply said Joomla! 1.5 is forcing the character set for the database to utf8, this happens dynamically by the database driver.

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 2:37 pm

Everything seems to be pretty fine and understandable, I read careful the wiki entry as well.

So, if encoding is set on per table basis, I can see that when in the SQL queries there is no encoding defined (ie charset=utf8), MySQL is creating the table with encoding that is the same to the database collation. So here comes the question, is Joomla! able to change the collation of the database during the installation? My test above show that it does ..., just not pretty sure if it works all the time.

So, if you say that the collation has nothing to do with how data is stored, I would agree definitely, but it has impact on the encoding of new created tables where there is no charset in the query.

So it turns now more like a feature request: Can Joomla! change the database collation during installation and in that way we will void approx. 10,000 posts in the forum like "Pleasee heeelp, strange chars appear"

Edit: 10 typos

yvolk · Post by **yvolk** » Thu May 31, 2007 2:53 pm

Hi, Johan!
Let's start from empty list

1. I think we shall return from terminology to the problem, visible to the users:
"Greek and Cyrillic characters, stored in Joomla! database, appear as boxes or question marks"
(Note, that we don't mention any extension here!)

2. This problem is simple to reproduce, because it affects Joomla! core functions, so there is no need to install any extention.
1. Install Joomla 1.5 beta 2, creating new database
2. Open the site
3. Login as Admin, go to the menu "Menus"->"Main Menu"
4. Click "The News" Menu Item.
5. Change "Title:" field from "The News" to "Новини"
6. Press "Save" button.
7. See, what you get in the menu item's title ("? ? ? ? ? ?")

Note: It was observed, that if the above bug reveals itself, User may see, using "phpMyAdmin" tool,
that all tables in Joomla's database have "latin1-general-ci" value in the column named "collation".

To reproduce the bug, if it doesn't appear during above scenario, do the following:
1. Create database manually before installation with "collation"="latin1-general-ci" (phpMyAdmin v.2.9.1.1 has this "collation" listbox in the "Create new database" form).
2. Use this existing database in the above scenario.

May you tell us simple thing:
1. Does Dev team know about this bug and confirms it?

Jinx · Post by **Jinx** » Thu May 31, 2007 2:54 pm

ivo.apostolov wrote: So, if encoding is set on per table basis, I can see that when in the SQL queries there is no encoding defined (ie charset=utf8), MySQL is creating the table with encoding that is the same to the database collation. So here comes the question, is Joomla! able to change the collation of the database during the installation? My test above show that it does ..., just not pretty sure if it works all the time.

As explained this is where the install.utf8.sql scripts come in for the extensions. For the core this is done by the installer.

So, if you say that the collation has nothing to do with how data is stored, I would agree definitely, but it has impact on the encoding of new created tables where there is no charset in the query.

Collation should not have any impact on the encoding of newly created tables. Collation and character set are two complete seperate settings.

So it turns now more like a feature request: Can Joomla! change the database collation during installation and in that way we will void approx. 10,000 posts in the forum like "Pleasee heeelp, strange chars appear"

Again you are mixing up collation with character set, Joomla! 1.5 forces the character set to utf8 for all core tables, for extensions this needs to happen by the extension itself when it is installed.

Joomla! cannot set the collation of the database itself, it relies on the predefined collation. If you need a different collation you can always changes this by hand. Again collation does not have any major effects on your system it only defines sorting order and it will not corrupt data.

Jinx · Post by **Jinx** » Thu May 31, 2007 2:59 pm

yvolk wrote: ".May you tell us simple thing:
1. Does Dev team know about this bug and confirms it?

I have tried to replicate this on two different local installation and it all works fine here. Can you start by giving me your MySQL version and the Joomla! 1.5 version you are ruhnning ?

ivo.apostolov · Post by **ivo.apostolov** » Thu May 31, 2007 3:01 pm

Johan,
believe me or not, but new tables except in the cases when the charset is specified in the SQL query are created with the same encoding as the database collation is. I need to sit down for a couple of hours and re-think the whole process.

yvolk · Post by **yvolk** » Thu May 31, 2007 3:15 pm

Jinx wrote:
yvolk wrote: ".May you tell us simple thing:
1. Does Dev team know about this bug and confirms it?
I have tried to replicate this on two different local installation and it all works fine here. Can you start by giving me your MySQL version and the Joomla! 1.5 version you are running ?

Of cause: all information is written in The bug tracker Item.
(BTW, I've corrected that MySql tool name in above message, it is called "phpMyAdmin").

To reproduce the bug, if it doesn't appear during above scenario, do the following:
1. Create database manually before installation with "collation"="latin1-general-ci" (phpMyAdmin v.2.9.1.1 has this "collation" listbox in the "Create new database" form).
2. Use this existing database in the above scenario.

Jinx · Post by **Jinx** » Thu May 31, 2007 3:32 pm

I'll investiagte.

Post by **infograf768** » Wed Sep 12, 2007 6:44 am

Anything new on this?

Jinx · Post by **Jinx** » Wed Sep 12, 2007 1:35 pm

Yes, after some investigation we have discovered that this is a phpMyAdmin issue. phpMyAdmin is coupling character set and collaction, which makes sense.

Post by **infograf768** » Wed Sep 12, 2007 3:16 pm

BTW, is'nt 2.1 in http://dev.joomla.org/component/option, ... tf-8/  obsolete?
As of now, we use

Code: Select all

) TYPE=MyISAM CHARACTER SET `utf8`;

in joomla.sql

soulgreater · Post by **soulgreater** » Sun Oct 26, 2008 5:31 am

hi all
I have a problem when install Extensions for joomla 1.5.7 like (fireboard)
all Tables in database take latin1_swedish_ci (Collation) when I write anything in arabic language I see it like this

??????????
MySQL charset is : UTF-8 Unicode (utf8)
can someone help me ?

kmaliagkas · Post by **kmaliagkas** » Sat Dec 20, 2008 1:31 am

Hi Guys,

I've read all the above but non of those solve my problem. The correct steps to have correct UTF-8 support is as follows:

Step 1.

Create the database from phpmyadmin interface with collation utf8_general_ci

Step 2.

Procced to Joomla installation

Step 3.

Open the file globals.php and at the end of it, right after the last

}

and right before

?>

put the following code:

header("Content-Type: text/html; charset=UTF-8");

Step 4.

Open the file language/english.php and go on line 421. Change the following code from:

DEFINE('_ISO','charset=iso-8859-1');

to:

DEFINE('_ISO','charset=UTF-8');

Step 5. ( and most importand for me )

Open the file includes/database.php and change the following code from:

// @mysql_query("SET NAMES 'utf8'", $this->_resource);

to:

@mysql_query("SET NAMES 'utf8'", $this->_resource);

THE END.

The last one solve my problem to transfer my site to a different server.

I Agree with the friend below, but this is the guide to make a sucssefull migration from Joomla 1.0.15 to Joomla 1.5.x. And also if you want to transfer site from on Web Hosting Server to another.

Regards.

Post by **infograf768** » Sat Dec 20, 2008 7:48 am

You are mixing Joomla 1.0 and Joomla 1.5
There is no language/english.php file in Joomla 1.5

kmaliagkas · Post by **kmaliagkas** » Tue Dec 23, 2008 1:00 pm

Hi,

I Agree with the friend above, but this is the guide to make a sucssefull migration from Joomla 1.0.15 to Joomla 1.5.x. And also if you want to transfer site from on Web Hosting Server to another.
In my little experience on Joomla 1.5 if you convert the tables in UTF-8, everithing work fine. The only think you may have to do is everithing you see in Symbols to write it again in your language.

Regards.

The Joomla! Forum™

Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard

Re: Problem with greek and cyrillic characters in Fireboard