How-to: FTS (full text search) in dovecot using solr

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,792
Location
LT, EU
Install java 8 or greater, check with:
Code:
java -version

Install solr:
Code:
wget -O solr-7.7.0.tgz http://apache.mirror.vu.lt/apache/lucene/solr/7.7.0/solr-7.7.0.tgz
tar xzf solr-7.7.0.tgz solr-7.7.0/bin/install_solr_service.sh --strip-components=2
bash install_solr_service.sh solr-7.7.0.tgz
su -m solr -c "/opt/solr/bin/solr create_core -c dovecot"
curl http://localhost:8983/solr/dovecot/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
rm -f /var/solr/data/dovecot/conf/managed-schema

Download /var/solr/data/dovecot/conf/solrconfig.xml:
Code:
wget -O /var/solr/data/dovecot/conf/solrconfig.xml https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0.xml

Download /var/solr/data/dovecot/conf/schema.xml:
Code:
wget -O /var/solr/data/dovecot/conf/schema.xml https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml

Re-start solr, enable it in chkconfig:
Code:
service solr restart
chkconfig solr on

Configure dovecot for solr:
Code:
cd /usr/local/directadmin/custombuild
mkdir -p custom/dovecot/
cp -p configure/dovecot/configure.dovecot custom/dovecot/

Edit custom/dovecot/configure.dovecot and add:
Code:
--with-solr

Run:
Code:
./build dovecot

Enable plugin in dovecot:
Code:
mkdir -p custom/dovecot/conf/
echo 'mail_plugins = $mail_plugins quota zlib fts fts_solr' > custom/dovecot/conf/mail_plugins.conf

Create /etc/dovecot/conf.d/90-solr.conf:
Code:
plugin {
  fts = solr
  fts_solr = url=http://localhost:8983/solr/dovecot/
}

Rewrite dovecot confs:
Code:
./build dovecot_conf

Optional: ​Don't track user solr processes in lfd:
Code:
echo 'user:solr' >> /etc/csf/csf.pignore
service lfd restart

Logs are stored /var/solr/logs/solr.log, /var/log/maillog shouldn't show errors regarding fts_solr too.
 
Last edited:

kevinwalter

Verified User
Joined
Jun 1, 2011
Messages
48
Change it to this and it's working again. Why not activate this by default? Searching is now much faster. :)

Code:
wget -O solr-7.3.1.tgz [url]http://apache.mirror.vu.lt/apache/lucene/solr/7.3.1/solr-7.3.1.tgz[/url]
tar xzf solr-7.3.1.tgz solr-7.3.1/bin/install_solr_service.sh --strip-components=2
bash install_solr_service.sh solr-7.3.1.tgz
su -m solr -c "/opt/solr/bin/solr create_core -c dovecot"
curl [url]http://localhost:8983/solr/dovecot/config[/url] -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
rm -f /var/solr/data/dovecot/conf/managed-schema
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
It might be worth add this in the installation:

Code:
for USER in `cat /etc/virtual/domainowners  | cut -d: -f2 | cut -d\  -f2`
        do
        doveadm fts rescan -u $USER
done
To create the indexes for all existing users.

Also I would suggest (if possible) in CB to modify the default dovecot.conf to instead of single loading conf/mail_plugins.conf) to run conf.plugins/* where all the plugins (single file per plugin as per imap quota, solr, sieve, etc) will be stored (and managed by CB in custom/conf.plugins).

Would that be possible?

Thanks for the very useful guide :)

Best regards
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
8,792
Location
LT, EU
Why not activate this by default? Searching is now much faster. :)
I didn't personally like Java/Solr service part and complexity of it there :) CustomBuild 2.0 rev. 2333 includes fts-xapian, enabling it would be as easy as:
Code:
cd /usr/local/directadmin/custombuild
./build update
./build dovecot
./build fts-xapian
As initial indexing might take some CPU power until it's done, we're not enabling it by default yet :)
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
Should this be used instead of SOLR?
Any specific reason to switch from SOLR to that?
Any way to have it installed on a central node and have all servers connect to that instead of each server having a local instance?

Thanks :)
 

bdacus01

Verified User
Joined
Jul 22, 2017
Messages
672
Location
Murfreesboro
I didn't personally like Java/Solr service part and complexity of it there :) CustomBuild 2.0 rev. 2333 includes fts-xapian, enabling it would be as easy as:
Code:
cd /usr/local/directadmin/custombuild
./build update
./build dovecot
./build fts-xapian
As initial indexing might take some CPU power until it's done, we're not enabling it by default yet :)
Maybe make a seperate post and Guide for this?
 

buzzard

New member
Joined
Mar 19, 2011
Messages
4
I have successfully installed Solr, but I was wondering how secure this is? Is the webapp by default not exposed to the internet? How can one secure this through CSF+LFD while not tracking the solr processes in lfd?
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,793
Location
A Coruña, Spain
I can't remember what port SOLR listen to by default, but if that port is not open on the firewall you can be assured that access to it cannot be done.

If you add the solr process to the csf.pignore, csf will not complain about it's execution time, that doesn't mean will allow access to the TCP Port that SOLR opens.

Hope this clarify
 
Top