AI Semantic Search Integration with Joomla

[nexai]

Hi,

This is similar to my previous post: viewtopic.php?f=844&t=1008777

I created a React Component UI for AI Chat and AI Semantic Search.

For the Semantic Search, Does anyone know of a way to index the Joomla website or does it provide a way to integrate with so you can get a list of all the pages and index it?

Currently the Semantic Search requires you to upload documents or give it all your webpage URLs which is tedious.

With a CMS there should be an API to get all the content list automatically? If anyone has experience with developing search components for has a git repository with example code for Joomla please let me know.

For an example see the search at the top of: [ redacted ]

I'd like to integrate this with an open source version for Joomla so it can have an AI backed semantic search.

[toivo]

Welcome to the Joomla forum!

Have you seen this presentation from the recent JoomlaDay USA: Adding AI-Powered Semantic Search To Your Joomla Site?

[nexai]

Hi,

Thanks for the welcome and the link.

The actual AI semantic search and indexing is not the crucial part of the integration since we already have this platform in place.

What I'm looking for is information on way to integrate a 3rd party service with Joomla from an extension perspective.

Where do you plug into Joomla to get all the Joomla content automatically to add into your index search engine index? Do you use a plugin for this?

For example, let's say someone added a new page to a Joomla site, may it be a component or Joomla content page. Is there a way to keep track of this either via events, or some Joomla API that lists all joomla pages.

My current investigation is leading to getting all the menu items from the database. This however does not reveal the actual content of components which create menu items.

... initialization code...

Code: Select all

$nexai = new Nexai(apiKey);
$query = $dbo->getQuery(true);
$query->select($dbo->quoteName(array('id', 'title', 'alias')));
$query->from($dbo->quoteName('#__menu'));
$query->where($dbo->quoteName('published') . ' = 1'); // Assuming you only want published pages

$dbo->setQuery($query);
$results = $dbo->loadObjectList();

if ($results) {
    echo "<ul>";
    foreach ($results as $page) {
        $url = JRoute::_('index.php?Itemid=' . $page->id);
        $nexai->add($url, ['title' => $title]);
    }
    echo "</ul>";
}

This requires the URL to be retrieved using HTTP, downloading the HTML and indexing it.

Is there a way to get the actual content of the URL with some metadata or keywords, semantic information without resorting to HTTP.