How-to: FTS (full text search) in dovecot using solr

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,628
Location
LT, EU
Install java 8 or greater, check with:
Code:
java -version


Install solr:
Code:
wget -O solr-7.7.0.tgz http://apache.mirror.vu.lt/apache/lucene/solr/7.7.0/solr-7.7.0.tgz
tar xzf solr-7.7.0.tgz solr-7.7.0/bin/install_solr_service.sh --strip-components=2
bash install_solr_service.sh solr-7.7.0.tgz
su -m solr -c "/opt/solr/bin/solr create_core -c dovecot"
curl http://localhost:8983/solr/dovecot/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
rm -f /var/solr/data/dovecot/conf/managed-schema


Download /var/solr/data/dovecot/conf/solrconfig.xml:
Code:
wget -O /var/solr/data/dovecot/conf/solrconfig.xml https://raw.githubusercontent.com/dovecot/core/master/doc/solr-config-7.7.0.xml


Download /var/solr/data/dovecot/conf/schema.xml:
Code:
wget -O /var/solr/data/dovecot/conf/schema.xml https://raw.githubusercontent.com/dovecot/core/master/doc/solr-schema-7.7.0.xml


Re-start solr, enable it in chkconfig:
Code:
service solr restart
chkconfig solr on


Configure dovecot for solr:
Code:
cd /usr/local/directadmin/custombuild
mkdir -p custom/dovecot/
cp -p configure/dovecot/configure.dovecot custom/dovecot/


Edit custom/dovecot/configure.dovecot and add:
Code:
--with-solr


Run:
Code:
./build dovecot


Enable plugin in dovecot:
Code:
mkdir -p custom/dovecot/conf/
echo 'mail_plugins = $mail_plugins quota zlib fts fts_solr' > custom/dovecot/conf/mail_plugins.conf


Create /etc/dovecot/conf.d/90-solr.conf:
Code:
plugin {
  fts = solr
  fts_solr = url=http://localhost:8983/solr/dovecot/
}


Rewrite dovecot confs:
Code:
./build dovecot_conf


Optional: ​Don't track user solr processes in lfd:
Code:
echo 'user:solr' >> /etc/csf/csf.pignore
service lfd restart


Logs are stored /var/solr/logs/solr.log, /var/log/maillog shouldn't show errors regarding fts_solr too.
 
Last edited:
Change it to this and it's working again. Why not activate this by default? Searching is now much faster. :)

Code:
wget -O solr-7.3.1.tgz [url]http://apache.mirror.vu.lt/apache/lucene/solr/7.3.1/solr-7.3.1.tgz[/url]
tar xzf solr-7.3.1.tgz solr-7.3.1/bin/install_solr_service.sh --strip-components=2
bash install_solr_service.sh solr-7.3.1.tgz
su -m solr -c "/opt/solr/bin/solr create_core -c dovecot"
curl [url]http://localhost:8983/solr/dovecot/config[/url] -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
rm -f /var/solr/data/dovecot/conf/managed-schema
 
It might be worth add this in the installation:

Code:
for USER in `cat /etc/virtual/domainowners  | cut -d: -f2 | cut -d\  -f2`
        do
        doveadm fts rescan -u $USER
done

To create the indexes for all existing users.

Also I would suggest (if possible) in CB to modify the default dovecot.conf to instead of single loading conf/mail_plugins.conf) to run conf.plugins/* where all the plugins (single file per plugin as per imap quota, solr, sieve, etc) will be stored (and managed by CB in custom/conf.plugins).

Would that be possible?

Thanks for the very useful guide :)

Best regards
 
Why not activate this by default? Searching is now much faster. :)

I didn't personally like Java/Solr service part and complexity of it there :) CustomBuild 2.0 rev. 2333 includes fts-xapian, enabling it would be as easy as:
Code:
cd /usr/local/directadmin/custombuild
./build update
./build dovecot
./build fts-xapian

As initial indexing might take some CPU power until it's done, we're not enabling it by default yet :)
 
Should this be used instead of SOLR?
Any specific reason to switch from SOLR to that?
Any way to have it installed on a central node and have all servers connect to that instead of each server having a local instance?

Thanks :)
 
I didn't personally like Java/Solr service part and complexity of it there :) CustomBuild 2.0 rev. 2333 includes fts-xapian, enabling it would be as easy as:
Code:
cd /usr/local/directadmin/custombuild
./build update
./build dovecot
./build fts-xapian

As initial indexing might take some CPU power until it's done, we're not enabling it by default yet :)
Maybe make a seperate post and Guide for this?
 
I have successfully installed Solr, but I was wondering how secure this is? Is the webapp by default not exposed to the internet? How can one secure this through CSF+LFD while not tracking the solr processes in lfd?
 
I can't remember what port SOLR listen to by default, but if that port is not open on the firewall you can be assured that access to it cannot be done.

If you add the solr process to the csf.pignore, csf will not complain about it's execution time, that doesn't mean will allow access to the TCP Port that SOLR opens.

Hope this clarify
 
cd /usr/local/directadmin/custombuild
./build
+-----------------------------------------------------------+
| DirectAdmin WebServices Installer |
| Written by Martynas Bendorius and DirectAdmin |
| Version: 2.0.0 (rev: 2491) |
+-----------------------------------------------------------+

./build update
./build dovecot
./build fts-xapian

Problem ....


checking for SQLITE3... no
configure: error: SQLite3 missing
Trying to make fts-xapian 1.3.1
make: *** No targets specified and no makefile found. Stop.


# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668
 
cd /usr/local/directadmin/custombuild
./build
+-----------------------------------------------------------+
| DirectAdmin WebServices Installer |
| Written by Martynas Bendorius and DirectAdmin |
| Version: 2.0.0 (rev: 2491) |
+-----------------------------------------------------------+

./build update
./build dovecot
./build fts-xapian

Problem ....


checking for SQLITE3... no
configure: error: SQLite3 missing
Trying to make fts-xapian 1.3.1
make: *** No targets specified and no makefile found. Stop.


# sqlite3 --version
3.7.17 2013-05-20 00:56:22 118a3b35693b134d56ebd780123b7fd6f1497668

The fix is as simple as:
Code:
yum -y install sqlite-devel

Fixed in CB 2.0 rev. 2492 as well.

Thank you for the report.
 
Hi

cd /usr/local/directadmin/custombuild
rm -f /usr/lib64/dovecot/lib21_fts_xapian_plugin*

not found
rm -f : cannot access /usr/lib64/dovecot/lib21_fts_xapian_plugin*: No such file or directory

M.
 
Hi

cd /usr/local/directadmin/custombuild
rm -f /usr/lib64/dovecot/lib21_fts_xapian_plugin*

not found
rm -f : cannot access /usr/lib64/dovecot/lib21_fts_xapian_plugin*: No such file or directory

M.

Is it debian? Try lib instead of lib64.
 
uname -a
Linux shared1.provector.pl 3.10.0-962.3.2.lve1.5.26.3.el7.x86_64 #1 SMP Wed Aug 14 08:29:59 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

/usr/lib/dovecot/lib21_fts_xapian_plugin* - this is the correct path for me

thank you for your help and best regards
Mariusz
 
When I follow: https://help.directadmin.com/item.php?id=2128 and run ./build fts-xapian. I got the following error

configure: error: *** A compiler with support for C++11 language features is required.
Trying to make xapian-core
make: *** No targets specified and no makefile found. Stop.

yum install gcc-c++ does not fix anything
Package gcc-c++-4.4.7-23.el6.x86_64 already installed and latest version

CentOS version: 6
 
Last edited:
When I follow: https://help.directadmin.com/item.php?id=2128 and run ./build fts-xapian. I got the following error

configure: error: *** A compiler with support for C++11 language features is required.
Trying to make xapian-core
make: *** No targets specified and no makefile found. Stop.

yum install gcc-c++ does not fix anything
Package gcc-c++-4.4.7-23.el6.x86_64 already installed and latest version

CentOS version: 6
May you try CB 2.0 rev. 2561 ?
 
I've ran the following but I'm still on version 2560

1005 ./build version
1006 ./build update_versions
1007 ./build version
1008 cd /usr/local/directadmin/custombuild
1009 ./build update
1010 ./build versions
 
I've ran the following but I'm still on version 2560

1005 ./build version
1006 ./build update_versions
1007 ./build version
1008 cd /usr/local/directadmin/custombuild
1009 ./build update
1010 ./build versions
What's the mirror used?
 
I'm using Dutch mirrors if I'm not mistaken I'll check tomorrow too see if it'll fix the issues thanks for your fast response!
 
Back
Top