[33]Library Installer

[gencom]

multi extension (component-plugin-mambot-module) installer.

An example

com_fireboard
mod_fireboard
bot_fireboard

This package's all must become at archives and the xml file must become at the shape.

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<install version="1.5" type="extensionpack">

METADATA
METADATA

<installcomponent="com_fireboard">

	METADATA
	METADATA

		<files>
			<filename>com_directory/docs/doc.js</filename>
		</files>

		<params>
		</params>

	<images>
		<filename>com_directory/images/image.gif</filename>
	</images>
	
<installcomponent>

<installmodule="mod_fireboard">

	METADATA
	METADATA

		<files>
			<filename>mod_directory/docs/doc.js</filename>
		</files>

		<params>
		</params>

	<images>
		<filename>mod_directory/images/image.gif</filename>
	</images>

<installmodule>

<installmambot="bot_fireboard">

	METADATA
	METADATA

		<files>
			<filename>bot_directory/docs/doc.js</filename>
		</files>

		<params>
		</params>

	<images>
		<filename>bot_directory/images/image.gif</filename>
	</images>

<installmambot>

</mosinstall>

The I this name one click joomla!

1.5 with this installer can become elastic more much.

Can she be make become? At next demands.

note; see which has know the me much. The I do not understand a code more at aller write. Very only idea is coming white spotted.

http://forum.joomla.org/viewtopic.php?f ... &sk=t&sd=a

[pasamio]

A white paper for this has already been written, please search before you post. Additionally what you have written isn't a white paper it is more a request for a feature. Below is the previously posted whitepaper for the topic.

http://forum.joomla.org/viewtopic.php?f=500&t=265962

[tcp]

Some insteresting thing about <install method="upgrade" ...>

It works great and can be used in 1.5 but there is someting what is wrong for me. If user upgrades this way and has SEF enabled, every menu link to the component will be invalid ( index.php?Itemid=20&option= ). If he goes to menu link and save this link, it will be repaired...

Yes, in J1.5 we rebuild the menu, from scratch. We don't keep the previous extension IDs and we don't repair the menus. This is really a bug in my opinion, and something that we can fix in 1.5 . We can also implement a check for an upgrade.php script in 1.5 if we wanted to.

So I think there is a lot of agreement here. As Wilco points out there was a substantial amount of work put into a packaging system for the SOC 2006 and I think its implementation is a good thing for 1.6 . +1

tcp

[infograf768]

The issue:
As the basic joomla! package, to be light enough, is limited to provide English language.
As even the localized community packages are limited to a few languages, mostly English and another one.
It would be a useful addition to let user install non included language packages from Joomla UI, corresponding to the Installation languages at least.

1. This feature was discussed as far ago as 1.1 and was rightfully postponed at the time.
2. It is related to extensions installation from the internet. Since then, JPackage is on the backburner.
3. There is a difference though with other kind of installations/upgrades: we would not here have to implement it from any place on the net but just from a central Joomla repository containing Accredited Translations.

It is also to note that no core code or MySqL change would be necessary (different in that respect from some of our 3pds libraries/add-ons as tcpdf, TinyMCE, others or extensions updating).
4. To be of use, the feature should be implemented in 2 places of the UI
— At time of installation
— As a specific dropdown of available languages in the Install/Uninstall Manager.
5. A check concerning Joomla! version installed as well as a check concerning language package corresponding version would be necessary.

[tfuller]

I support this proposal but would like it extended to the com_uninstall as well. In my case I wish users to choose in the uninstall routine to do things like keep all tables, backup tables, or remove tables.

[Stasys]

Hannes, you should update your list:

TCPDF is already upgraded to 2.6 version in joomla core and TCPDF 2.7 is Released


I also hope we'll soon upgrade TinyMCE to 3.0.5 version!

[torkil]

An upgrade to TinyMCE 3.0 would be most welcome

Also: It would be nice to leave a door open for people who want to install the extra commercial plugins for TinyMCE (image- and fileeditor). By "open door" I mean a way to add more plugins to the editor without having to "hack" files that are part of the Joomla distribution, so that I won't have to manually hack these files every time I upgrade Joomla.

[willebil]

The proposal is a bit to big to post here, for the full proposal, please download the PDF:

http://joomlacode.org/gf/download/frsre ... .v.1.0.pdf

This is the first concept of the paper, feel free to post feedback.

[Chris Davenport]

This paper proposes a number of mostly quite small changes to the Joomla! extension installer. The changes are mostly independent of one another but as they are all fairly small they are grouped together into this one white paper for convenience.

Introduction
The principal goal here is to enable all XML installer files to be validated against an appropriate XML Document Type Definition (DTD) or XML schema. This would allow extension developers to validate their XML files using one of the readily available validation tools. This in turn would make it easier to track down certain classes of installer bug, some of which can be hard to find. DTDs were in existence even for the 1.0.x series, but they have not been kept up-to-date and were not amended for the 1.5.0 release. Some work has already been done to create new DTDs for 1.5.0, particularly in regard to templates with the template DTD being constructed by one of our GHOP students and this works well. Similar work was attempted for the other extension types and this needs to be finished, but there is a problem with the component DTD which will require changes to the installer itself. This paper details the changes required and also looks at how the DTDs should be maintained in the future.

In addition to this important change, a number of other changes are proposed that incrementally improve the installer without requiring significant development work.

Quick summary of changes proposed

1. Ensure that installer XML DTDs are up-to-date and accurate and that all core installer XML files will validate against these DTDs.
   a) Place all XML DTDs and XML schemas under SVN version control.
   b) Add <dbinstall> tag as synonym of the non-root <install> tag. Non-root <install> to be deprecated in 1.6 and removed in 1.7.
   c) Add <dbuninstall> tag as synonym of the <uninstall> tag. <uninstall> to be deprecated in 1.6 and removed in 1.7.
   d) Finish creating DTDs for all extension types.
   e) Update core installer XML files and ensure that they validate.
2. Automatically add index.html to folders.
3. <licence> tag implemented as a synonym of <license>.
4. Translate the <description> tag.
5. Better error detection and diagnosis.

Change 1
Unfortunately, there is a clash of XML tag names in the 1.5.0 release of the component installer which means that the installer actually requires the XML to be invalid! The problem is that the root node is <install>, but the <install> tag is also used deeper in the XML tree for a quite different purpose. This violates one of the XML validity constraints (Unique Element Type Declaration: http://www.w3.org/TR/REC-xml/#EDUnique). Because the XML is inherently invalid it is not possible to write a DTD to validate it.

The solution proposed here is for the non-root XML tag to be renamed to <dbinstall>. For backwards compatibility the installer should accept the old <install> tag too, but the component DTD will insist on the new form only (it could not do otherwise). For no better reason than that I like the symmetry, I would recommend that the <uninstall> tag be renamed to <dbuninstall> at the same time. Again, for backwards compatibility the old form should continue to be supported by the installer for some time, but only the new form will be included in the DTD.

In addition to fixing this obvious problem the general organisation and use of DTDs (and any future XML schemas) should be considered. The following action plan is proposed to correct the validation problem and to properly implement installer XML DTDs:-

1. Although XML DTDs are hosted on http://dev.joomla.org/xml they should also be added to the SVN repository for proper version control. My thinking is that http://dev.joomla.org/xml should be an exported copy of the appropriate location in the SVN repository. Decide where in the SVN repository to keep XML DTDs and XML schemas and commit the current DTDs to this location.
2. Some documentation on the template DTD and how to use it has already been created (by Witchakorn Kamolpornwijit (chalet16)). This needs to be brought into the official documentation and updated. Documentation Working Group will take care of this.
3. Modify the component installer to allow <dbinstall> as a synonym for the non-root <install> tag. The non-root <install> tag will be marked as deprecated in 1.6 and support for it will be removed in 1.7.
4. Modify the component installer to allow <dbuninstall> as a synonym for the <uninstall> tag. The <uninstall> tag will be marked as deprecated in 1.6 and support for it will be removed in 1.7.
5. Finish the task of creating DTDs for each of the extension types.
6. Having amended the installer as described above, all the core extension XML files should be modified to include the appropriate DOCTYPE and to fully validate against the relevant DTD.
7. Ensure that documentation covering all the installer XML files and how to make use of them is up-to-date.

See the Appendix to this white paper for a proposed DOCTYPE syntax specification.

Change 2
For security reasons it is a good idea to include an index.html file in all directories so that the web server does not return a list of files present in a directory if the user does not specify one. However, it is a common oversight to omit one of these files during development. The suggestion is made that the extension installer should automatically include an index.html file by default in all directories that it creates.

The change suggested is for

Code: Select all

<folder source="sourcefolder">dest-folder</folder> 

to automatically create an empty index.html in dest-folder (and any directories on the path to dest-folder that do not already have an index.html, except the Joomla! root directory).

If for any reason this behaviour is not required then

Code: Select all

<folder source="sourcefolder" noindex="true">dest-folder</folder> 

would suppress the creation of new index.html files.

For backwards compatibility, extensions that include <filename>index.html</filename> would cause the automatically-created file to be silently overwritten.

Reference: http://forum.joomla.org/viewtopic.php?t=252107

Change 3
The official language of the Joomla! project is British/Australian English yet the <license> tag in the installer XML insists on the American spelling. This trivial change would permit <licence> to be used as a synonym for <license>. Closing tags would still need to match opening tags of course, so <license>...</licence> would not be permitted. The DTDs would permit either form as being equally valid.

Change 4
The <description> tag in extension installer XML files is not currently translated. This should be fixed.

Change 5
Some of the difficulties most frequently encountered by users are to do with installing Joomla! extensions. However, the error messages produced by the installer are often not very helpful as they tend to describe the immediate problem from the point of view of the installer rather than being related to the users' goals or indeed sometimes without giving any indication of possible remedial actions.

For example, it is not uncommon to get a JFTP error when installing an extension. Firstly, most users will probably not know what JFTP is and so will not know where to begin to try to fix it. Secondly, the fact that the installer is only attempting to FTP into the server because permissions or ownership of files or directories does not permit direct access is not even hinted at. The solution to the users problem may be a matter of changing permissions on a directory, but the installer does not say which directory or what permissions would be required. Or it may be a matter of changing ownership of a directory, but again which directory and what ownership would be required? Having reached the fallback position of trying to use FTP the installer could help the user by checking obvious problems such as a missing username or password. I believe all this information is available to the installer and so it should be possible to help the user by showing this information together with an explanation of why this might be a problem.

For example, instead of an error message like this:

JFTP error: Cannot login

the user might instead see:

Cannot write to directory /whatever as server does not have write permission for this directory.
Current permissions on directory /whatever are set to 644 (-rw-r—r--), recommend permissions are set to 755 (-rwxr-xr-x).
Directory /whatever is owned by user somebody. Web server is running as user www-data.
Attempting to access server using FTP instead of direct access to filesystem.
Attempting to login to http://www.domain.com using FTP: Username is chrisdavenport, password is not empty.
FTP login failed: FTP server did not respond. Check that the FTP server is running, that it is listening on port 21 and that port 21 is not being blocked by a firewall.

This is not a single error message and most are only warnings since only the last would actually cause the installer to abort. None of them would be visible if the installer did not encounter a problem. This suggests that these messages are accumulated in a buffer during the installation process and only output to the user if the installation process cannot be completed. The messages shown above should not be construed as being the actual proposed messages. They are shown only to give a flavour of the proposed changes.

In another white paper I have proposed that error messages be viewed as entry points into the documentation. Some or all of these messages could link to further information on the documentation server.

Reference: http://forum.joomla.org/viewtopic.php?f=500&t=268156

Appendix: DOCTYPE syntax specification
A typical DOCTYPE statement looks like this:

Code: Select all

<!DOCTYPE install PUBLIC "-//Joomla! 1.5//DTD template 1.0//EN" "http://dev.joomla.org/xml/1.5/template-install.dtd"> 

This one happens to be the one recommended for use in Joomla! 1.5 template installer XML files and is currently in use.

The production rule for the DOCTYPE declaration is included here for reference:

Code: Select all

doctypedecl ::= '<!DOCTYPE' S Name (S ExternalID)? S? ('[' intSubset ']' S?)? '>'

For Joomla! installer XML files the Name must be 'install' and we will not generally include an intSubset. The full production rule for the ExternalID is given as:

Code: Select all

ExternalID ::= 'SYSTEM' S SystemLiteral | 'PUBLIC' S PubidLiteral S SystemLiteral 

For Joomla! installer XML files we will generally use the 'PUBLIC' variant and so for our purposes we can define the ExternalID production rule as:

Code: Select all

ExternalID ::= 'PUBLIC' S PubidLiteral S SystemLiteral

The XML specification defines the public and system identifiers only in terms of their allowed characters. For our purposes it will be useful to imbue them with their own syntax and semantics, described here.
It is proposed that the public identifier will be given a definite structure as follows:

Code: Select all

PubidLiteral ::= '”-//Joomla!' S JVer '//DTD' S Ext S DVer '//' LangCode '”'
JVer ::= JMajor . JMinor | JMajor . JMinor . Jmaintenance
JMajor ::= ( Digit )+ 
JMinor ::= ( Digit )+ 
JMaintenance ::= ( Digit )+ 
Ext ::= 'template' | 'component' | 'module' | 'plugin'
DVer ::= DMajor . DMinor | DMajor . DMinor . DMaintenance
DMajor ::= ( Digit )+ 
DMinor ::= ( Digit )+ 
DMaintenance ::= ( Digit )+ 
LangCode ::= Name  

Note that JVer is the earliest version of Joomla! to which this DTD applies, whereas DVer is an independent version number referring to the DTD itself.

The production rule for Ext may be extended to support other extension types, or even non-installer XML files, in line with continuing Joomla! development.

LangCode is the ISO 639 language code, folded to upper case. Since the Joomla! extension installer does not accept multi-language tags the only valid value for LangCode is 'EN'.

The system identifier is a URI which will, for our purposes, be defined by the production rules:

Code: Select all

SystemLiteral ::= 'http://dev.joomla.org/xml/' JVer '/' Filename
Filename ::= Ext '-' Descriptor '.dtd'
Descriptor ::= Name   

The Descriptor is an identifier which hopefully conveys an idea of the purpose of the DTD. For example, 'install' could be used to indicate that the XML file is intended for use by the Joomla! extension installer. As an example, the Filename 'template-install.dtd' readily suggests that the DTD is for use during template installation. Future Descriptors should be chosen to give similarly meaningful hints as to the purpose of the DTD. The possible values of Descriptor are not constrained further here.

Note that JVer and Ext in the system identifier must match those given for the public identifier.

A list of the officially defined public and system identifiers with links to supporting documentation can be found at http://docs.joomla.org/Official_DTDs

Further Reading and References

1. Google Highly Open Participation Contest task 45: “Write a DTD for template XML installer files (templateDetails.xml)”:-
   a) Forum discussion: http://forum.joomla.org/index.php/topic,238762.0.html
   b) Output: http://help.joomla.org/ghop/feb2008/task045/
2. Extensible Markup Language (XML) 1.0 (Fourth Edition): http://www.w3.org/TR/REC-xml/
3. ISO 639: (International Organization for Standardization). ISO 639:1988 (E). Code for the representation of names of languages. [Geneva]: International Organization for Standardization, 1988.

[Hackwar]

I like this whitepaper. Its extremely well formulated and gets to the point. I would like to add one little thing, that is not entirely related to DTDs, but still... I'd like to see a consistent naming convention for extension installer XMLs. This would also help to keep some sort of backwards compatibility. If the new way would rely on an install.xml file and maybe use another root tag or something, another XML could be used for legacy installation.

[dynedain]

I apologize in advance if this is formatted improperly.

I propose that additional compatibility testing is done during the Joomla installation process, preferably as the very first step before any of the wizard begins.

The current system requirements (http://help.joomla.org/content/view/1938/302/) detail that certain versions of PHP have problems with the Joomla installation (notably 4.4.2). These are exceptions to the normal rule of at least v 4.3.10. A 4.4.2 installation can result in the first page of the Joomla installer not even displaying, leaving the person performing the install without any indication of what the problem might be.

It would be preferable for the very first step of the Joomla installation to be a quick check of server software versions and settings so that critical problems can immediately be identified before anything else happens. Any software installer as a matter of good practice, should perform a compatibility check to make sure the installer itself can run, let alone the final installed product.

Ideally this should be a relatively small change, pre-empting the first step of the installer (language choice). If Joomla installer does not pass the critical test of PHP and Apache versions, there should be an immediate message indicating as much, and should include links to more information on the Joomla help pages. Optional requirements (such as zlib support or certain PHP settings) should remain where they currently are in the installer process.

[Hackwar]

I'd like to expand this a little bit not only to libraries, but to files in general. I've proposed a few things here, that will work without any other extension, but will only add a few files to an existing extension.

[skOre]

I second that this should not just concern libraries. Just a spark of thought: It would be nice if there was a central point for components to communicate with each other, so instead of libraries, they would put APIs in there for the other components to use.

[pasamio]

@hackwar: Expanding it to general components is a cool idea, though it can get messy with uninstallation procedures as well. At the moment we don't support any thing for libraries, so I'd personally like to see this resolved. We can move to bigger and better things as we go.

@Skore: This is already possible in 1.5 if you want it in a way. Check out JLoader::register($class, $file). What it will allow you to do is link to given files. In PHP4 its a big messy (force loads the file) but in PHP5 it uses the autoloader to handle things. You could put this into a system plugin to run onAfterInitialise() to register a component file. However once we get more things shunted into libraries and out of components, libraries become more powerful as part of building smaller component sets and not crossing the extension boundaries.

[skOre]

pasamio: Yeah, good call. I was just thinking how much "integration" I have already accumulated in the AEC and its always standard stuff like CB profile manipulation... something that I'm sure other components might want to do as well. Will check out the 1.5 routines for this then!

[Jonathan Cameron]

Improvements in installation of language packs for extensions.

Along the same lines as my "Simplify installation of multi-part extensions" proposal above, some improvements would be welcome in constructing language packs for extensions.

The current system works but has a few limitations

• It requires separate files for the front end files and for the back end files. Even with only two files per language, if the extension has more than a few languages, it can become inefficient to require two files per language.
• The installed packs are not tracked in the install/uninstall system anywhere and cannot be uninstalled using the normal Joomla uninstaller.
• It is not obvious how to install the language-specific help files for a component using the current system.

I would like to see the improvements that are developed for the installation system be flexible enough to support language packs for extensions that have the following properties:

• Consolidates all langauge files for a multi-part extension into one language pack file (per language)
• Is tracked by the install/uninstall system so that the files can easily be uninstalled with the normal Joomla uninstaller.
• Includes the capability to install the language-specific help files in the appropriate directory (for instance: administrator/component/com_mycomponent/help/qq-QQ/help.html)

Thanks!

-Jonathan

[Hackwar]

I earlier tried to implement this function in 1.5, but we had to take it out due to performance reasons. I'd like to discuss this again:
This is for JFile.

Code: Select all

	/**
	 * Checks a list of files against a list of MD5 hashes
	 * The hashes have to be in a file with
	 * one <hash> <file> per line. The path of the file
	 * has to be relative to the Joomla root
	 *
	 * @param string $file File path
	 * @return mixed array when files found, true when all ok, false when md5 file not found or not in the right format
	 * @since 1.5
	 */
	function checkMD5sum( $md5file, $basepath = '' ) {

		$md5file = JPATH_ROOT . JPath::clean( $md5file, false );
		if (!is_readable($md5file)) {
			return false;
		}

		// Reading the file into an array
		$md5files = file( $md5file );
		$i = 0;
		foreach($md5files as $md5file) {
			// Getting filename and path from line
			$file = JPATH_ROOT . DS . $basepath . JPath::clean( substr( $md5file, 33, strlen( $md5file ) - 35 ), 0 );
			$md5data = substr( $md5file, 0, 32);
			if (is_readable($file)) {
				$currentmd5 = md5( JFile::read( $file ) );
				if (!($md5data == $currentmd5)) {
					$files[$i] = array( 'filename' => $file, 'exists' => true );
					$i++;
				}
			} else {
				$files[$i] = array( 'filename' => $file, 'exists' => false );
				$i++;
			}
		}

		if (isset($files)) {
			return $files;
		} else {
			return true;
		}
	}

[carsten888]

try jce editor. its got all that!

[torkil]

But not in the 1.5 series, at least not yet?

[H13]

I miss the option to add the 'Use Global' value like as it works with 'Select box' or 'Radio button' into parameters displayed (rendered) as 'Text' or 'Textarea'.

Example:
(Select box) In Parameters in Component's Global Configuration user selects e.g. 'display=1' and in Parameters Component in Menu link user select 'use global' and the 'display' value is set to '1' for the specifed menu link

(Textarea) In Parameters in Component's Global Configuration user selects e.g. 'display='value'' and in Parameters Component in Menu link there is no option 'use global', so in this parameter the value from e.g. config.xml is displayed (e.g. ''display='novalue'') and this value will be used for specifed menu link (User cannot use the value from Global Configuration)


Jan

[mcsmom]

For Tinymce upgrade when it happens please take a look at this issue.

http://joomlacode.org/gf/project/joomla ... em_id=9617

[metux]

Hi folks,

I *REALLY* like to see that external libraries really *EXTERNAL*.
In other words: should NOT be contained within Joomla. The installer should only check/warn
about the dependencies and nothing more.

cu

[pasamio]

So you want to put libraries outside of the /libraries folder? What is in aide of specifically?

[metux]

So you want to put libraries outside of the /libraries folder? What is in aide of specifically?

Because separate packages should be maintained separately. Updating library packages should be done by the host's package management.

If you just copy-in an external package and maintain it by your own, that's the opposite of
modularization and cooperation !

cu

[skOre]

Well... you can always make the folder a symlink to your library folder?

[metux]

Well... you can always make the folder a symlink to your library folder?

Requires to have all libs residing in one directory, so you loose the ability to work on php's include path.

[pasamio]

The problem that I find is that updating external libraries has a habit of breaking their dependencies. Look at a project like Debian, updating libraries out of release is taken as a rather serious situation. Randomly updating dependent libraries is an issue, and even when we've put it into Core we've had issues with things, TCPDF is a good example of this where the upgrade ended up breaking more than it fixed, though it did fix the original problem. Additionally a lot of the libraries are kept there more for backwards compatibility as well, such as domit or pattemplate, which will then be removed in the next upgrade. I could perhaps see a shared libraries folder but you'd have to keep everything consistent. Finally the support issues would be a nightmare trying to track down issues against different sets of libraries, it would be well and truly unsupportable. PHP/MySQL combinations already raise enough strange combinations to test on without complicating things even further. Its something that you're welcome to work on for your own projects however I don't see it ever being put into the Core due to the large amount of complications it'd introduce. We will be shipping a per instance library installer and updater however this will be managed within the instance itself and we'll be tying it to dependencies as well to avoid breaking things where possible and mark incompatibilities, modelled similar to the way Debian handles items.

[metux]

The problem that I find is that updating external libraries has a habit of breaking their dependencies.

That's the job of the host's package management and the distro's QM team behind.
For now, Joomla devs have do duplicate work which is already done by distros.

I hope, you don't want J to become OO ;-o

Look at a project like Debian, updating libraries out of release is taken as a rather serious situation.

Because they don't do their homework properly. The fact that there things like separate
Distro releases is a major problem. It's not as bad as in M$ world, but already bad enough.

Randomly updating dependent libraries is an issue, and even when we've put it into Core
we've had issues with things,

Not randomly. Wisely planned.
Maybe just get more packages into PEAR.

TCPDF is a good example of this where the upgrade ended up breaking more than it fixed,
though it did fix the original problem.

If upstream makes crap, do a fork or choose a more sane library.

I could perhaps see a shared libraries folder but you'd have to keep everything consistent.

Yeah, like the dll crap on Windoze ? ;-o
Either you have a proper package management or you'll someday end up in version hell.

The current status is completely unsatisfying:
Let's imagine two components or modules which import some external package libfoo.
For now they have to ship everything on their own (assuming it's neither in core nor
host-provided). Now they might ship different versions or even different (incompatible)
branches. Booom! Evrything goes to hell.

Finally the support issues would be a nightmare trying to track down issues against different sets of libraries, it would be well and truly unsupportable.

Design by contract ?
QM constraints ?

PHP/MySQL combinations already raise enough strange combinations to test on without complicating things even further.

Take out mysql and you'll get rid of 90% ;-P


cu

[metux]

PS: it would be nice if we had another extension type: "library" and a clear and strict dependency handling. The installer should not allow to install anything which dependencies are not completely satisfied. Ah, and dependencies should be made on interfaces, not packages.

[Hackwar]

a lot more interesting would be a file installer, that copies a bunch of files to some other place in the system. I'm thinking about adding new views, new layouts, new elements for JParameter, etc. and of course libraries. This could also help with updating Joomla. With a bunch of checksums, we could make sure that we don't overwrite modified files.