The Joomla! Forum ™

I have one of the most important french joomla websites, the migration sql takes a few days but it works I have about 50MB data base I put on the installer side with ftp. no errors with the migration, but after that...
- no modules at all (not even the basic ones)
- sections/categories have disappeared as well
- some article are nice, but most of them are cut erratically (I guess it's not erratic, but I don't see the pattern)
since it's a rc3, I a a little worried. have you tested migrator with foreign website, at least 2000+ articles inside ?
as for now, too many errors to correct them after the migration. I weed extensive but correction, and if possible another procecude : not 100% by hand, but not 100% automatic either. some middleground in order to limit errors must be found.
imho, such a so drastic migration was an error from the very beginning.
on the other hand, the impex utilitie from vubellin works perfectly with huge boards, so I guess it was possible to do it from joomla to joomla, but right now, i bet huge websites like mine won't migrate, and huge websites are your best adv for joomla.

Hi lafactory,

Would you be able to send me a copy of the sql file and what character set that the file/database is in. Additionally, could you do a small examination of the SQL file and see if the inconsistencies exist in there as well. Also did you use the iconv conversion or convert before by hand?

With regards testing, no I haven't personally done a large foreign site because I don't have access to one nor do I speak any other language with great skill to adequately test those sites. We're trying to get things tested by the community as much as possible because its hard to build up a large foreign language on our own so any help you can send me I'll try to debug it and get things fixed This is why we're working on release candidates to find bugs from a large range of sites and get it fixed. It is hard to build a system to migrate large sites when by default PHP's limitation is towards 30 seconds of time and 8MB of memory -- not the friendliest to bulk data transfers but we're working on getting there with everyone's help

Sam

(my email: sam DOT moffatt AT joomla DOT org)

_________________
Sam Moffatt
Dev WG: Update and Install Technologies, Authentication Systems
S&I WG: Package Builder Maintainer, JoomlaCode Backend Systems
Simplest way to update your site: http://extensions.joomla.org/extensions ... llers/9332

I sent you an email. if you could add directly migrator and some documentation to joomla rc itself it would be very useful : so far the problem is no to get new joomla users, the problem is to migrate as clean as possible. I'm not even sure the "foreign" thing is relevant : I have accentuation in my articles after the migration. another thing I don't understand is why the migrator script must run within the browser : if the browser is off, the migration script stops. sounds crazy to me : why not a background script with perl ? it take many days to create the sql migration script. I don't understand that. imho you should have put more horsepower to the migration part, even at the cost of some features in joomla 1.5.

Hi lafactory,

The reason we do it in the browser is simple because its the lowest common denominator across all platforms (Windows/Unix) that we know will work regardless. I can't in any easy way set a perl script to run under Windows (or PHP CLI), nor might I be able to do it on some jailed Unix systems (e.g. with safe mode enabled in PHP), so its better to put the time into the system that will work across all systems than doing a lot of work for one. This is why it has been set up with the browser web interface, and its been that way for a rather long period of time (the original migrator written by David Gal before this one was also a web interface).

Sam

_________________
Sam Moffatt
Dev WG: Update and Install Technologies, Authentication Systems
S&I WG: Package Builder Maintainer, JoomlaCode Backend Systems
Simplest way to update your site: http://extensions.joomla.org/extensions ... llers/9332

I have same remarks as lafactory

It could be a good tool for American people but not for us, French

- No problems with articles

- But troncatures with category, section, tittle of an article if this word includes some french specific characters as à é è ç

Can you help us ?

Thanks

_________________
Didier L

I've been working on an RC4. I'm not quite sure where the truncation came from because the RC4 release of the migrator didn't change any of the export code (or the import code for that matter) but using LaFactory's site an import into the latest 1.5 release seemed to work fine. I'm not quite sure why but it appears at least to be fixed as I see multiple sections, categories and content titles with accents in their name without issues. Attached image is a sample taken from the categories list.

What version of MySQL are you running? I'm not sure what the export site's MySQL was but the local version I am using is MySQL 4.1.22 which may also impact on the behaviour as well.

_________________
Sam Moffatt
Dev WG: Update and Install Technologies, Authentication Systems
S&I WG: Package Builder Maintainer, JoomlaCode Backend Systems
Simplest way to update your site: http://extensions.joomla.org/extensions ... llers/9332

I work on it with Sam. with 2000+ articles on lafactory when the import will be bug free on my website I am pretty confident it will work for any other website, at least french.

Hi Sam and lafactory,
I've been struggling with this truncation problem in my own site. I'm using Joomla! 1.5 RC4 with mySQL 4.1.22-standard on an Apache Unix server. I had the same problem because in Portuguese we have some extended characters too.
All the migration process went by as a breeze but all the items titles and menus got truncated exactly where the firs special character is. Though the articles content were not truncated at all.
I gave a look into the SQL dump generated by the migration tool and spotted the problem. While in the articles body all the special characters appear as their HTML code equivalents, it doesn't happen with the titles and menu names.
What I did was a search for all the special characters in the SQL dump and replaced them with their HTML code (i.e. á -> á  à -> à  etc) and now I'll execute the migration again... Hope that solves the problem. I'll let you know the results as soon as I manage to have some time to do the migration.

UPDATE: OK, I've made the migration with the altered SQL dump and, yes, it worked (kind of).
The only problem now is with the titles. They're not broken at all, but instead of showing the actual character they show the HTML code for them.
The menus are OK. If you look to the titles in a list style (section or category) they're also OK. Only when showing the top title of the article they show differently. I think now it's just a matter of changing the code that actually shows the title to force it to convert the HTML codes into the actual characters.
For the com_migration development team: it would be very nice if the component could take care of the changes itself. It's just a matter of searching the SQL dump for the foreign characters and replacing them with their HTML code.
For the core development team: could you please make the mods in the code to correct the issue with the title and put it in the SVN?
For the sitemasters who want to make the changes in the SQL dump from the 1.0.X database, don't forget to save it with UTF-8 encoding after finishing. Otherwise the migration will fail. 

Hope this can help lots of people who like lafactory, didier L and myself want to migrate their foreign language site(s) to 1.5!

ANOTHER UPDATE: The title issue isn't a core issue at all! It's a template issue!
It happens with rhuk_milkyway and beez the same way, but I've installed JA_purity and the problem is gone!

Regards

Airton.

_________________
Airton Torres
Joomla Bug Squad http://groups.google.com/group/joomlabugsquad
Community website - http://community.joomla.org
Unsolicited support PMs will be deleted and the user added to the foes list.

Sorry but it's not a problem of RC4  or migrator tool

I have checked data on MySQL database (with PHPMyAdmin) :

-  in jos_content table, format is correct :
An example :
" Le mot dignité, comme le mot économie, a plusieurs sens. Juridiquement, l'homme est digne du simple fait qu'il est né homme. L'homme, fin en soi, a une valeur, et non un prix. L'homme est déchu de sa dignité"

-  in other tables, it's not utf8 format but ISO 8859-1 format

Some examples from jos_menu table :

Before migration (Joomla 1.01)

id = 30
menutype = Notre engagement citoyen
name = La laïcité
link = index.php?option=com_content&task=blogcategory&id=... 


id = 31
menutype = Notre engagement citoyen
name = La république

...
After migration (after using migrator tool) :

id =30 
menutype = Notre engagement citoyen
name = La la
alias  = la-latotre-engagement-citoyen
link = index.php?option=com_content&view=category&layout=...



id = 31
menutype = Notre engagement citoyen
name = La r
alias = la-rblique-notre-engagement-citoyen

An explanation  :
- for content, I use editor provided with Joomla (Tiny MCE) who converts characters in good format
- for others items, I type directly characters using joomla interface and navigator (Firefox) and in this case, selected format is not good (ISO 8859 instead of utf8)


So it obvious that migrator converts utf8 but rejects incorrect strings coded in ISO format ... 


The new question : how to type directly items in the good format  ? 

I have checked Firefox configuration (in option menu - content, i had selected utf8 format but it's not sufficient !

Thanks for your help ! 

Didier

_________________
Didier L

Well, I've just found out that the problem isn't totally solved with the template. There are some comps that still show the titles in a wrong way. In may case, using JA_Purity template, it persists in the breadcrumbs and some other places. 

I noticed also that my site in J! 1.0.13 uses ISO-8859-1 encoding (Western European (Windows) in IE6) and my new site in J! 1.5 uses UTF-8. 

I'm sure it's an issue with the migrator regarding the character encoding switching from ISO to UTF-8 but I don't know how to solve this. I hope somebody comes up with a solution soon. I have one hundred something articles in my site right now and don't want to go into the trouble of copying/pasting everything from 1.0.13 to 1.5 to get thing right.

Thanks for helping.

Airton

_________________
Airton Torres
Joomla Bug Squad http://groups.google.com/group/joomlabugsquad
Community website - http://community.joomla.org
Unsolicited support PMs will be deleted and the user added to the foes list.

For the initial web site - built with Joomla 1.0 -, I have defined a database with utf8 format, as for the new database. So migrator converts the old database without modifying format.

I have checked that, when I type a title (category title for example), coding is not correct (ISO 8859 and not utf8 as for articles in database), both with the older version (1.0.12) and the new (1.5 RC4)

I have always the same question : how to type a title coding in utf8 format ?

Didier L

_________________
Didier L