The Joomla! Forum ™



Forum rules


Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.



Post new topic Reply to topic  [ 11 posts ] 
Author Message
PostPosted: Fri Oct 07, 2005 7:01 pm 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
Hi folks

I have about 2,000 HTML pages (thankfully in a flat directory) with alot of relative links in it, and I would like to import the whole lot into Joomla as a set of items in one category.

I would like to convert all the links to SEF compliant links and automate their addition to menus (initially just one menu for the whole lot, because the html already contains its own tocs and so on).

I would like to include the items in searches, with results rescricted depending on user acls.

Then of course I would like to share this with others as a Joomla component. Any one who'd like to join in this development is most welcome.

While I can write code, I have only just started learning PHP. Could any one advise, what's the best way to do the hml parsing and link transformation? 


Top
 Profile  
 
PostPosted: Sat Oct 08, 2005 9:38 am 
User avatar
Joomla! Intern
Joomla! Intern
Offline

Joined: Sat Aug 20, 2005 11:28 am
Posts: 60
Hi,

The staticXT-component is built to integrate HTML pages into joomla, it parses the pages and converts relative links, src-tags etc... but it doesn't "import" the pages, hence they are not available for searching etc...

regards,

Oli

_________________
JoomlaXT!


Top
 Profile  
 
PostPosted: Sat Oct 08, 2005 7:50 pm 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
I'm finding staticXt very helpful. I tried subscribing but the spam software is rejecting the confirmaiton emails, otherwise I would have thanked the fellow.

I'm not finding it handles relative links inside the content. Perhaps I need to fiddle with SEF some more.

I've got a single import database working, and I'm in the process of looping it. PHP is an odd language, sometimes there's arrays when you don't know it and file paths are very perplexing!


Top
 Profile  
 
PostPosted: Sun Oct 09, 2005 3:57 am 
User avatar
Joomla! Hero
Joomla! Hero
Offline

Joined: Fri Aug 19, 2005 2:26 am
Posts: 2337
Location: Lancaster, Lancashire, United Kingdom
Given that you are already talking with him - MamboXT Oli Merten - I would think the forum would be a suitable conduit for your thanks.

Dean.

_________________
Dean Marshall Consultancy - team of six Joomla experts
http://www.deanmarshall.co.uk/joomla-se ... rvice.html
Joomla Jargon: http://www.deanmarshall.co.uk/joomla-gl ... ons-a.html


Top
 Profile  
 
PostPosted: Sun Oct 09, 2005 7:34 am 
User avatar
Joomla! Intern
Joomla! Intern
Offline

Joined: Sat Aug 20, 2005 11:28 am
Posts: 60
DeanMarshall wrote:
Given that you are already talking with him - MamboXT Oli Merten - I would think the forum would be a suitable conduit for your thanks.


Actually it's JoomlaT in the meantime  ;)

emeyer wrote:
I'm not finding it handles relative links inside the content. Perhaps I need to fiddle with SEF some more.


Just let me know (ideally in the JoomlaXT-Forum) which problems you have exactly and I'll have a look...

regards,

Oli

_________________
JoomlaXT!


Top
 Profile  
 
PostPosted: Sun Oct 09, 2005 12:45 pm 
User avatar
Joomla! Hero
Joomla! Hero
Offline

Joined: Fri Aug 19, 2005 2:26 am
Posts: 2337
Location: Lancaster, Lancashire, United Kingdom
omerten wrote:
DeanMarshall wrote:
Given that you are already talking with him - MamboXT Oli Merten - I would think the forum would be a suitable conduit for your thanks.

Actually it's JoomlaT in the meantime  ;)

Quite so Oli - my brain has now switched off to the difference in the two names - I read the two words as synonyms.

Dean.

_________________
Dean Marshall Consultancy - team of six Joomla experts
http://www.deanmarshall.co.uk/joomla-se ... rvice.html
Joomla Jargon: http://www.deanmarshall.co.uk/joomla-gl ... ons-a.html


Top
 Profile  
 
PostPosted: Sun Oct 09, 2005 4:16 pm 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
I cant post a message in thed forums. I tried subscribing several times, but our virus software has quarantined the replies, so I cant get a login!


Top
 Profile  
 
PostPosted: Sun Oct 09, 2005 4:24 pm 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
Here is the code so far--I have hacked the files:

...\administrator\components\com_typedcontent.php
...\administrator\components\com_typedcontent.html.php

The "Edit" or "new" buttons let you select a file in a directory, upon which it attempts to upload all the files in the directory and put them in the database. By using the existing static component admin interface, I got It working for one file, but when I added the directory iteration, upon each iterationit forgets the path and can't find the files. On checking around this is a common problem with PHP. Have to fix this problem first...any suggestions on the best way to do it? Ideally I'd like it to load files from any location, as, since they are being put in the database, there is no security risk.


You do not have the required permissions to view the files attached to this post.


Top
 Profile  
 
PostPosted: Tue Oct 11, 2005 8:46 am 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
The file recursion is working, there is still a problem with SEF, so I can'r fix the internal links yet. I've asked on opensef.org for help...here is the code which replaces the save function in administrator/components/com_typedcontent/admin.typedcontent.php (there is some issue with this board duplicating strings occasionally, hopefuly this wont mess it up:

function save( $option, $task ) {
global $database, $my, $mainframe;
// global $mosConfig_sef;
$ordering=1;
$mosConfig_sef = 0;
$nullDate = $database->getNullDate();
$menu = mosGetParam( $_POST, 'menu', 'mainmenu' );
$menuid = mosGetParam( $_POST, 'menuid', 0 );
$row = new mosContent( $database );
if (!$row->bind( $_POST )) {
echo "\n";
exit();
}
if (!$row->introtext) {
echo "\n";
}
echo $row->introtext.'
';
$dir = dirname($row->introtext );
$arrfiles = scan_Dir($dir);
foreach ($arrfiles as $file) {
// SETUP DATA
$row = new mosContent( $database );
if (!$row->bind( $_POST )) {
echo "\n";
exit();
}
// ID AND DATES
if ( $row->id ) {
$row->modified = date( 'Y-m-d H:i:s' );
$row->modified_by = $my->id;
} else {
$row->created = date( 'Y-m-d H:i:s' );
$row->created_by = $row->created_by ? $row->created_by : $my->id;
}
if (trim( $row->publish_down ) == 'Never') {
$row->publish_down = $nullDate;
}
// PARAMS
$params = mosGetParam( $_POST, 'params', '' );
if (is_array( $params )) {
$txt = array();
foreach ( $params as $k=>$v) {
$txt[] = "$k=$v";
}
}
$row->attribs = implode( "\n", $txt );
// TITLE
echo '

Loading file: '.$file.'

';
$page = file_get_contents($file);
$page = str_replace("\n", '', $page);
$page = str_replace("\r", '', $page);
$title = ExtractString($page,'','');
if ($mytitle = '') $mytitle = $file;
$row->title = $title;
$row->title = ampReplace( $row->title );
$row->title_alias = $title;
$mainframe->appendPathWay($row->title);
// TEXT
echo '

title: '.$title.'

';
if ( $mosConfig_sef ) {
$path = sefRelToAbs("component/option,com_typedContent/".$title);
//$com_staticxt_url = "component/option,com_staticxt/".$title;
/* $pattern = "/".preg_quote($com_staticxt_url, "/")."([^\"\s]*=)/i";
preg_match_all($pattern, $pageContent ,$matches, PREG_OFFSET_CAPTURE);
$i=0;
$offset = intval($matches[1][$i][1]);
if ($offset > 0) {
$tmp = $matches[1][$i][0];
$tmp = str_replace('=', ',', $tmp);
$tmp = preg_replace('/(\?|&)/', '/', $tmp);
$pageContent = str_replace($matches[1][$i][0], $tmp, $pageContent);
$i++;
$offset = intval($matches[1][$i][1]);
}
*/
} else {
$path = "index.php?option=com_typedContent&id=".$row->id;
}
$page = preg_replace('/(.*)(?i)()/', '
'.'\2', $page);
$page = preg_replace('/(?i)(<\/body>)(.*)/', '
', $page);
preg_replace( '/(?i)(\ssrc([\s\"=\']*))/', '\1'.$path, $page);
preg_replace( '/(?i)(<(table|th|td).*)(\sbackground([\s\"=\']*))/', '\1'.'\3'.$dir, $page);
preg_replace( '/(?i)(movie.*)(\svalue([\s\"=\']*))/', '\1'.'\2'.$path, $page);
$pattern = "/(?i)([^=\"\s]*?(\.css))(?=((?!(href=)))*?)(?=((?!(<(link).)))>?)/";
preg_replace( $pattern, $dir.'\1', $page);
$pattern = "/(?i)([^=\"\s]*?(\.(jp?g|gif|png)))(?=((?!(background=)))*?)(?=((?!(<(table|th|td).)))>?)/";
preg_replace( $pattern, $path.'\1', $page);
// code cleaner for xhtml transitional compliance
$page = str_replace( '
', '
', $page);
/*
if ($pathwayPatch == "X") {
if ( $mosConfig_sef ) {
$xtItem = "Itemid,".$Itemid."/xt_item,1/"; }
else {
$xtItem = "&Itemid=".$Itemid."&xt_item=1"; }
} else $xtItem = "";
*/
//$page = strip_tags($page);
// $page = html_entity_decode($page, ENT_NOQUOTES);
// echo $page;
$row->introtext = $page;
      if (!$row->check()) {
echo "\n";
closedir($dir);
exit();
}
if (!$row->store()) {
echo "\n";
closedir($dir);
exit();
}
$row->checkin();
$query = "UPDATE #__content"
. "\n SET state = 1"
. "\n WHERE id = $row->id "
;
$database->setQuery( $query );
if (!$database->query()) {
echo "\n";
exit();
}
$ordering++;
AddToMenu($path, $title,$row->id, $ordering);
}
exit;
switch($task) {
case 'go2menu':
mosRedirect( 'index2.php?option=com_menus&menutype='. $menu );
break;
case 'go2menuitem':
mosRedirect( 'index2.php?option=com_menus&menutype='. $menu .'&task=edit&hidemainmenu=1&id='. $menuid );
break;
case 'menulink':
menuLink( $option, $row->id );
break;
case 'resethits':
resethits( $option, $row->id );
break;
case 'save':
$msg = 'Typed Content Item saved';
mosRedirect( 'index2.php?option='. $option, $msg );
break;
case 'apply':
default:
$msg = 'Changes to Typed Content Item saved';
mosRedirect( 'index2.php?option='. $option .'&task=edit&hidemainmenu=1&id='. $row->id, $msg );
break;
}
}

function scan_Dir($dir) {
$arrfiles = array();
$exts = array("htm", "html"); //"jpg$|\\.jpeg","png","bmp");
if (is_dir($dir)) {
if ($handle = opendir($dir)) {
chdir($dir);
while (false !== ($file = readdir($handle))) {
if ($file != "." && $file != "..") {
if (is_dir($file)) {
$arr = scan_Dir($file);
foreach ($arr as $value) {
$arrfiles[] = $dir."\\".$value;
}
} else {
foreach ($exts as $value) {
$testExt = "\.".$value."$";
if (eregi($testExt, $file)){
$arrfiles[] = $dir."\\".$file;
}
}
}
}
}
chdir("../");
}
closedir($handle);
}
return $arrfiles;
}

function ExtractString($str, $start, $end){
$str_low = strtolower($str);
$pos_start = strpos($str_low, $start);
$posoffset = $pos_start + strlen($start);
$pos_end = strpos($str_low, $end, (string) $posoffset);
if ( ($pos_start !== false) && ($pos_end !== false) ){
$pos1 = $pos_start + strlen($start);
$pos2 = $pos_end - $pos1;
return substr($str, $pos1, $pos2);
}
}

function AddToMenu($url, $title,$cid,$ordering){
global $database;
$myDebug = 0;
// $id = 10;
$menutype =  'other';
$menulink = 'index.php?option=com_content&task=view&id='.$cid;
$type = 'content_typed';
$browserNav = 0;

$title = substr($title, 0, 20) ; // min length?
$title = substr(addslashes($title), 0, 50);
if (strlen($title) == 20) {$title[18]='.'; $title[19]='.'; };
/*
if ($level+1>4) {
$componentid = 0;
$browserNav = 1;
}$content_id =  $content_id++;
*/
$query = 'INSERT INTO `jos_menu` ( `id` , `menutype` , `name` , `link` , `type` , `published` , `parent` , `componentid` , `sublevel` , `ordering` , `checked_out` , `checked_out_time` , `pollid` , `browserNav` , `access` , `utaccess` , `params` ) '
          . " VALUES (  'NULL', '$menutype',  '$title',  '$menulink',  'content_typed',  '1',  '0',  '$cid',  '0',  '$ordering',  '0',  '0000-00-00 00:00:00',  '0',  '$browserNav',  '0',  '3',  '' );"
          . ' ';
if ($myDebug) echo "query: $query
";
$database->setQuery( $query );
if (!$database->query()) {
echo "\n";
exit();
}
}
?>


Top
 Profile  
 
PostPosted: Thu Nov 03, 2005 12:14 pm 
Joomla! Enthusiast
Joomla! Enthusiast
Offline

Joined: Thu Sep 29, 2005 2:37 am
Posts: 165
This is now available as the "site import" component in developer's area, the SEF issue is also fixed:

http://developer.joomla.org/sf/projects/site_import


Top
 Profile  
 
PostPosted: Thu Mar 30, 2006 7:48 pm 
User avatar
Joomla! Apprentice
Joomla! Apprentice
Offline

Joined: Wed Mar 22, 2006 9:11 am
Posts: 18
Ernest,
I have need for such a function, but can't get yours to work with version 1.0.8 of Joomla. Are you still supporting this?
Ideally I would like to import a site I have on file (http://www.hsvwik.com dutch) as content, after which comes the task of assigning Categories and Sections.

Any insight would be helpfull.

Rimsky

_________________
TANSTAAFL
http://www.hsvwik.com


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 



Who is online

Users browsing this forum: MSN [Bot] and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
Powered by phpBB® Forum Software © phpBB Group