Joomla 4 Smart search: so disk space consuming? Topic is solved

[fabio42]

Hello !
I've just migrated from Joomla 3 to Joomla 4. Read all the articles concerning smart search and deleted the old Jooma 3 search package. I followed the stepas concerning the indexation of the contents with the smart search component, and suddenly, my database exceeded the limit because of a rise in 97 Mb for the indexation of more than 300 articles...
Is there a way to dodge this or am I "condemned" to buy more disk space from my provider?
Because of course, without indexing the contents, the search box does not return any results...
Thanks for your answer(s)!
Fabrice

[SharkyKZ]

Did you try optimizing tables? There is Optimise button in the component. Do you have "Search for Phrases" indexing option enabled? And can you clarify if you mean 97 Mb or MB? 97 Mb (about 12 MB) sounds possible for 300 articles.

[fabio42]

Hello SharkyKZ and thanks a lot for your quick answer!
Indeed, you were right. Optimizing the tables did the trick. And yes, the first time I indexed the tables, the finder_links_terms weighed about 97 Mb (I guess something must have gone wrong anyway) !! While now it is only 16 Mb. Which is lighter. While the finder_terms table is 22 Mb big.
Now, I enabled the "search for phrases" indexing option. And everything seems to go plain-sailing.
Thanks so much for your advice.
Regards
Fabrice

[toivo]

And can you clarify if you mean 97 Mb or MB?

@fabio42, as requested by @SharkyKZ, can you please clarify this. There is a big difference between bits and bytes.

[fabio42]

Seems like I'm discovering lots of things right here! Actually I must have been wrong again, not being tech-savvy.
I checked and it's all about Mib so Mebibytes then. Am I right now ?

[fabio42]

Well, here I am again. Without doing anything, the tables have grown again... no idea why.
The whole finder thing now weighs more than 100 Mib.


Is it normal?
Lost me

[toivo]

I checked and it's all about Mib so Mebibytes then. Am I right now ?

The abbreviation for mebibyte is MiB. There are mebibits, too: Mibit.

This area can be confusing, even for us who started programming in early 1970s in octal and hex code, with just a few KB to play with

Ref. 1 Multiple-byte units
Ref. 2 Multiple-bit units

[fabio42]

Still stuck with more than 160 Mib in the finder tables. I'm clueless. Any idea?

[Nidzo2203]

Don't use Smart search on big sites.

[fabio42]

I'm not sure it's possible to disable Smart Search. If I don't index my contents, then, the search result box displays "no results found" for any word search.
There's no alternative for the smart search component actually. The old search component will be removed in the next versions, so not using smart search is not a satisfying solution...
I'm looking for a reasonable solution that will help me not consume so much disk space. Perhaps that has to do with some tweaking in the component but the options seem limited, or else I'm missing something.

[Nidzo2203]

I use Google Custom Search for site with over 20.000 articles.

Joomla Smart search is data consuming since was introduced.

Check this Joomla documentation https://docs.joomla.org/Smart_Search_on_large_sites

[fabio42]

Thanks for this documentation which explains clearly what smart search implies. I will try out Google custom search, which might be a workaround! (which is quite disappointing for a component presented as a revolution)

[pe7er]

I have worked on large Joomla websites were we have used external services like Elastic Search for indexing.

There are 3rd party extensions available that integrate Elastic Search into your Joomla website. In that case the search index data is not stored in your website but on an Elastic Search server. Elastic Search stores it data in files using key/value which is much faster than using a database.

Your visitors still use Smart Search for searching. The Smart Search component will look up the data on the Elastic Search server.

[fabio42]

Sounds really interesting and worth trying! Thanks @pe7er!

[fabio42]

Hey there!
Coming back after some timeand after trying to see if there was any alternative to the Smart search plugin. Alas, after investigating the different alternative options (Elastic search, Google Search), I must say that even if they sound interesting, they remain pay options.
I just came to wonder that, after clearly admitting that the smart search included in Joomla 4 is really disk consuming for large websites (https://docs.joomla.org/Smart_Search_on_large_sites) - though my website only contains 533 articles (is that supposed to be huge?) - why on earth is nothing done to improve the plugin? (as it seems that Joomla does not benefit from the Smart search on this point, to my mind). The solution suggested in the above article (see link) mentions Joomla Solr as the solution for "larger webistes" but the extension has been unpublished!
So users are left with no other solution since the advent of Joomla 4 but to either disable it or find a pay replacement for it. Not sure whether the "old" search plugin is still working... but if it does, for how long? (I had once read it was going to be deprecated as I said in my previous posts, so it was recommended to remove it on a Joomla 4 install, which I sadly did... but perhaps I can try to install it again?).
Well, as you can see, there's not much we can do about it for now, but wait as the article said that this point will be addressed in future versions of Smart search. Can't wait for that.
Meanwhile, which feature(s) would you advise me to tweak in the Smart search parameters to allow a decrease in my finder_links table (which is 100 Mio heavy for the moment)?

Thanks for your help. Any advice is useful for now.
Fabrice

[DV-admin]

I have some experience regarding this because I have a much bigger site. I turned off search phrase and keywords. It created too many results. Do not cache the search results either if you have a very busy website. I introduced Jfilters a whole back and don't cache the results. This basically works pretty well. You can think about using sitesearch360 it works with elasticsearch which is useful if you can pay for it.

[Nidzo2203]

Hey there!
Coming back after some timeand after trying to see if there was any alternative to the Smart search plugin. Alas, after investigating the different alternative options (Elastic search, Google Search), I must say that even if they sound interesting, they remain pay options.
Thanks for your help. Any advice is useful for now.
Fabrice

Google Custom Search is free.

[fabio42]

Thank you @DV-admin for your answer. I did what you suggested but unfortunately, the finder_links table is still 100 Mio, even after optimizing Smart Search. Measn I'll have to explore the Google custom search option, though I'm not a fan of Google's intrusive tools. Better than nothing anyway! (thanks @Nitzo2203!)

[factum]

Having a reliable search engine, requires text parsing and indexing.
This has to be done for any information passed (i.e. title, text, custom fields, categories, tags, meta-tags).
It is not an easy task and requires a lot of data to be reliable.
So there is a correlation between accuracy and data size.

You have some options in the Smart Search config > Index tab. Do not forget to re-index your content if you change that.
Also you can define the Content Maps in the Smart Search.

[fabio42]

@factum
Thanks for your answer. Yet, the parameters do not seem to influence the size of the database. They rather influence the efficiency of the search in itself. The documentation mentions one paramater called "Memory table limit", yet, this parameter is absent in the latest version of Joomla 4.
As for the contents map, could you elaborate please?