Upper case accented or special like Ñ characters in sef translator extension are not recognized Topic is solved

General questions regarding the use of languages in Joomla! 3.x.

Moderators: infograf768, General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Post Reply
joomleb
Joomla! Guru
Joomla! Guru
Posts: 983
Joined: Sat Aug 20, 2005 8:53 am
Location: Panamá

Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by joomleb » Sun Jun 30, 2019 3:47 pm

Hi guys,
I need help, suggestions, to solve a bug:
- I'm using SEF Translator for automatic translations;
- the language list is managed by it's .../components/com_seftranslate/languages.ini file where:
AFRIKAANS = af
ALBANIAN = sq
ARABIC = ar
ARMENIAN = hy
AZERBAIJANI = az
BASQUE = eu
...etc.
- Where the Uppercase "ARABIC" in the Front end into the SEF Translate module dropdown menu is automatically shown and tranformed in lowercase "Arabic" - take a look here
- My default Front end language is Spanish, so, I transformed the languages list in Spanish.
- As you know, a lot of languages words in Spanish have the accent, like GERMAN > ALEMÁN, or ARABIC > ÁRABE or special characters like ESPAÑOL
- inside the SEF Translate Module selector as you can see here all special Ñ or Á letters are still upper case because are not recognized. In case of ÁRABE, it cause that the word is moved at the end of the list (because a reordering alphabetical rule is added)

- Also inside the SEF Translate Component (I attach an image for you) seem that special characters are not recognized.
2019-06-18-accent-upper-case.jpg

Please,
1 - Do you think it can be a server settings issue ?
2 - What can I do to exclude it is a Joomla issue ?
3 - Do you have any type of suggestion ? Tests ? Ideas ? Etc. ...

Any type of help is welcome !
You do not have the required permissions to view the files attached to this post.
Last edited by toivo on Sun Jun 30, 2019 4:08 pm, edited 1 time in total.

User avatar
infograf768
Joomla! Master
Joomla! Master
Posts: 18871
Joined: Fri Aug 12, 2005 3:47 pm
Location: **Translation Matters**

Re: Bug - Upper case accented or special like Ñ characters are not recognized

Post by infograf768 » Mon Jul 01, 2019 7:55 am

You can totally exclude a Joomla issue.

strtolower and strtoupper are not UTF-8 compatible!! specially for Uppercase letters.
Modify your languages.ini file and use lowercase when necessary, except for the First letter

Español = es
and NOT
EspaÑol = es

It should now display correctly
Screen Shot 2019-07-01 at 09.02.56.png
Note: this extension is very badly coded...

Another solution is to correct the code and use `mb_strtolower($str, 'UTF-8')` if mb_strtolower is available on your server.
There are other solutions for 3.x by using StringHelper::strtolower() after namespacing: adding `use Joomla\String\StringHelper;`
You do not have the required permissions to view the files attached to this post.
Jean-Marie Simonet / infograf · http://www.info-graf.fr
---------------------------------
ex-Joomla Translation Coordination Team • ex-Joomla! Production Working Group

joomleb
Joomla! Guru
Joomla! Guru
Posts: 983
Joined: Sat Aug 20, 2005 8:53 am
Location: Panamá

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by joomleb » Mon Jul 01, 2019 10:24 am

Hi infograf768,
many thanks for your suggestions, in the meantime you were answering I made some more tests on the languages.ini file. Please, before going any further, I ask you if, in light of these new tests, Do you confirm your suggestions?:

- if I wrote them all Upper case, like "ÁRABE = ar", "ALEMÁN = de", "COREANO = ko", into the dropdown menu are shown as "Árabe", "AlemÁn", "Coreano".
- if I wrote them with the initial capital letter, like "Árabe = ar", "Alemán = de", "Coreano = ko", into the dropdown menu are shown as "Árabe", "Alemán", "Coreano".
- if I wrote them all Lower case, like "árabe = ar", "alemán = de", "coreano = ko", into the dropdown menu are shown as "Árabe", "Alemán", "Coreano".
Always, the "Árabe" is moved automatically at the end of the list.

This mean that:
a - Doesn't matter if I wrote it "ÁRABE = ar", or "Árabe = ar", "árabe = ar", the initial capital letter is always transformed in Upper case.
b - Doesn't matter if I wrote it "ÁRABE = ar", or "Árabe = ar", "árabe = ar", the rest of the letters are always transformed in Lower case.
c - Just the Upper case special characters like "Á" and "Ñ" are not recognized. While if Lower case like "á" and "ñ" are well recognized
d - The Upper case character Á of "Árabe" is not recognized and is always moved automatically at the end of the list by the alphabetical reorder rule.

And, Please, in your opinion, What should be the road to take between the alternatives (considering the Joomla point of view)? Here you can see my considerations...

Many Thanks for help on understand here...

User avatar
infograf768
Joomla! Master
Joomla! Master
Posts: 18871
Joined: Fri Aug 12, 2005 3:47 pm
Location: **Translation Matters**

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by infograf768 » Mon Jul 01, 2019 3:44 pm

The code in this extension is basically, everywhere it is used:

Code: Select all

$t->title=ucfirst(strtolower($key));
'strtolower($key)' makes all letters lower case (but, as I said above it does not work for some UTF8 characters).
'ucfirst' sets the first letter as uppercase

Therefore it makes no difference if you use `À` or `à` as first letter as anyway it will be changed to Uppercase.

Then the usual ordering in the db (or in a list) for latin characters is alphabetical, but it makes a difference if the first letter is uppercase or not, and if it is accented with simple accents or diacritics or not.

For the database Joomla uses "utf8mb4_unicode_ci" collation and therefore the order you get is the default UTF8 ordering for that collation. There is a specific utf8mb4 collation for Spanish.

For lists made from items which are not in the db as for example .ini (this case) or .txt files I guess the order or is the default English one or depends on the "locale" set by your browser. For French or English (and I also guess for Spanish), accented will come after non accented.

This is what happens if I change the string for Akeeba in 'en-GB.com_akeeba.sys.ini' and I display components the dropdown Component menus (my locale is

Normally it is
COM_AKEEBA="Akeeba Backup"

I get
Screen Shot 2019-07-01 at 16.53.41.png
After I change to
COM_AKEEBA="Àkeeba Backup"
I get
Screen Shot 2019-07-01 at 17.00.43.png
The rule is that a lowercase first letter will be displayed after its uppercase.
Second rule is that accented characters will be displayed after the non-accented characters.

To modify this "locale order", one can use some javascript
See https://stackoverflow.com/questions/690 ... th-unicode

Hope it helped.
You do not have the required permissions to view the files attached to this post.
Jean-Marie Simonet / infograf · http://www.info-graf.fr
---------------------------------
ex-Joomla Translation Coordination Team • ex-Joomla! Production Working Group

joomleb
Joomla! Guru
Joomla! Guru
Posts: 983
Joined: Sat Aug 20, 2005 8:53 am
Location: Panamá

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by joomleb » Mon Jul 01, 2019 6:55 pm

Hey infograf768,
you really helped a lot! Many Thanks!
When you go to Panama or Italy / Spain you are welcome, I owe you at least a pizza and a good glass of wine :)

The only thing I still don't understand is:
- if I wrote them all Upper case, like "ÁRABE = ar", "ALEMÁN = de", "COREANO = ko", into the dropdown menu are shown as "Árabe", "AlemÁn", "Coreano".
Why the "Á" is not transformed lowercase (when it is not the initial letter) ?

I'd like to know your opinion about my considerations on SEF Translate languages.ini file (Please, take a look at the link):
What should be the road to take between "your alternatives" (considering the Joomla point of view)?

User avatar
infograf768
Joomla! Master
Joomla! Master
Posts: 18871
Joined: Fri Aug 12, 2005 3:47 pm
Location: **Translation Matters**

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by infograf768 » Tue Jul 02, 2019 5:37 am

Why the "Á" is not transformed lowercase (when it is not the initial letter) ?
As I explained above it is because the php native method strtolower() does not work with UTF8.

Just explain the developer of the extension that he has to use
StringHelper::strtolower() after namespacing by adding `use Joomla\String\StringHelper;` when J version is 3.9+
It will solve the issue nicely EVERYWHERE a similar code exists as it is totally UTF8 compatible.

NOTE: Concerning the ordering, it is not as simple as I thought first.
This because I had opened and saved the component itself (with the free version) after choosing the site language as French.

The order now is the order of the ini file and not the locale. I.e. Àrabe comes always just after Albanian.
In frontend, the list of languages MAY BE translated by Bing WHEN a language is selected and keeps the order.
In backend, obviously, it reflects the existing .ini as it it not bing translated.

The translation of the select and language is not reliable. It depends on the backend default site language defined in the component. And the resulting translation of the page is also unreliable...
Jean-Marie Simonet / infograf · http://www.info-graf.fr
---------------------------------
ex-Joomla Translation Coordination Team • ex-Joomla! Production Working Group

joomleb
Joomla! Guru
Joomla! Guru
Posts: 983
Joined: Sat Aug 20, 2005 8:53 am
Location: Panamá

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by joomleb » Tue Jul 02, 2019 9:06 am

Hi inforgraf768,
always many thanks for your help!

The order now is the order of the ini file and not the locale. I.e. Àrabe comes always just after Albanian.
In frontend, the list of languages MAY BE translated by Bing WHEN a language is selected and keeps the order.
In backend, obviously, it reflects the existing .ini as it it not bing translated
These are the info I have:
- With "Instant" Method (the only one available with free version) only Google Translated can be used. You can select Bing, but your are always using the Google Translator.
- Yes, the default order was defined by the languages.ini.
- With last release, thanks to developer, the default order would be defined by reorder rule applied to the languages.ini. This is why "I discovered" the "Á" issue, because the "Árabe" was moved at the end of the list.
- Anyway, in my opinion, this "reorder" rule would be useful ONLY if applied after each time a language has been selected and the list translated... Otherwise, there is no sense to "reorder", we can just edit the languages.ini as we want...

An alternative solution in a Joomla point of view could be:
- Right now the SEF Translate module "Languages Dropdown List" is established from the english file: /components/com_seftranslate/languages.ini
- SEF Translate knows which language has been selected into the SEF Translate module Languages Dropdown List
- We'll have to transform the generic languages.ini into:
en-EN.languages.ini
fr-FR.languages.ini
it-IT.languages.ini ...etc.
- And tell to SEF Translate:
when English is selected the en-EN.languages.ini file is loaded
when French is selected the fr-FR.languages.ini file is loaded ...etc.
- In this way all the languages ini files will be yet translated and the alphabetical ordered (per language)...
- Moving all the .ini language files into the respective frontend .../Languages/en-EN/ ...etc. folders we’ll could use also the default Joomla Languages Overrides feature if needed (or preferred)
What do you think about ? (I'd like to know your opinion)


The translation of the select and language is not reliable. It depends on the backend default site language defined in the component. And the resulting translation of the page is also unreliable...
Sorry, I don't understand what do you mean, Please, Can you explain me ?

User avatar
infograf768
Joomla! Master
Joomla! Master
Posts: 18871
Joined: Fri Aug 12, 2005 3:47 pm
Location: **Translation Matters**

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by infograf768 » Tue Jul 02, 2019 10:32 am

Honestly, as I am not very much concerned by that type of component (not fan of automatic translations...) and also think the code was written originally for 1.6 and would need a full refactoring, I will leave this discussion here.
Jean-Marie Simonet / infograf · http://www.info-graf.fr
---------------------------------
ex-Joomla Translation Coordination Team • ex-Joomla! Production Working Group

joomleb
Joomla! Guru
Joomla! Guru
Posts: 983
Joined: Sat Aug 20, 2005 8:53 am
Location: Panamá

Re: Upper case accented or special like Ñ characters in sef translator extension are not recognized

Post by joomleb » Fri Jul 05, 2019 8:08 am

@infograf768
Okay, message received. Really Many thanks for your help :)

PS About Automatic Translations
I'm not a fan too, but with experience I realized that is quite impossible manage portals with manual translations for a lot of reasons.
The only way is to run istant Automatic Translations (AI help too much) and leave to manual only the review, correcting, editing the content... = Automatic Translations as a must to support Manual Translations
As you know, I also discussed it in past here


Post Reply

Return to “Language - Joomla! 3.x”