My Solr FTS problem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Gao
Reply | Threaded
Open this post in threaded view
|

My Solr FTS problem

Gao
I am testing the solr FTS following the guide here:
http://things.m31.ch/?p=379

Now I am having problem when I try to test:

# doveadm -v  index -u [hidden email] Inbox
doveadm([hidden email]): Error: fts_solr: Invalid XML input at
1:0: not well-formed (invalid token) (near: {
   "responseHeader":{
     "status":0,
     "QTime":0,
     "params":{
       "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa)
doveadm([hidden email]): Error: Mailbox INBOX: Status lookup
failed: Internal error occurred. Refer to server log for more
information. [2017-11-30 13:05:24]

My system is CentOS 7, Dovecot 3.2.4 (installed from source with solr
support), Java OpenJDK-1.8.0, Solr v7.1.0.

Could some one help me on this please?

Here is detailed debug out put:

[root@mail ~]# doveadm -vD  index -u [hidden email] Inbox
Debug: Loading modules from directory: /usr/lib64/dovecot
Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so
Debug: Module loaded: /usr/lib64/dovecot/lib20_fts_plugin.so
Debug: Module loaded: /usr/lib64/dovecot/lib21_fts_solr_plugin.so
Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined
symbol: acl_user_module (this is usually intentional, so just ignore
this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined
symbol: expire_set_deinit (this is usually intentional, so just ignore
this message)
Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so
Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so:
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
intentional, so just ignore this message)
doveadm([hidden email]): Debug: Added userdb setting:
mail=maildir:/home/vmail/mydomain.com/fail2ban/
doveadm([hidden email]): Debug: Added userdb setting:
plugin/quota_rule=*:bytes=0
doveadm([hidden email]): Debug: Effective uid=5000, gid=5000,
home=/home/vmail/mydomain.com/fail2ban/
doveadm([hidden email]): Debug: Quota root: name=User quota
backend=maildir args=
doveadm([hidden email]): Debug: Quota rule: root=User quota
mailbox=* bytes=0 messages=0
doveadm([hidden email]): Debug: Quota rule: root=User quota
mailbox=Trash bytes=+104857600 messages=0
doveadm([hidden email]): Debug: Quota warning: bytes=0 (90%)
messages=0 reverse=no command=quota-warning 90 [hidden email]
mydomain.com
doveadm([hidden email]): Debug: Quota warning: bytes=0 (80%)
messages=0 reverse=no command=quota-warning 80 [hidden email]
mydomain.com
doveadm([hidden email]): Debug: Quota grace: root=User quota
bytes=0 (10%)
doveadm([hidden email]): Debug: Namespace inbox: type=private,
prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:/home/vmail/mydomain.com/fail2ban/
doveadm([hidden email]): Debug: maildir++:
root=/home/vmail/mydomain.com/fail2ban, index=, indexpvt=, control=,
inbox=/home/vmail/mydomain.com/fail2ban, alt=
doveadm([hidden email]): Debug: quota: quota_over_flag check:
quota_over_script unset - skipping
doveadm([hidden email]): Debug: INBOX: Mailbox opened because: index
doveadm([hidden email]): Debug: http-client: host 127.0.0.1: Host
created
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Peer created
doveadm([hidden email]): Debug: http-client: queue
<a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Setting up connection to 127.0.0.1:8983 (1
requests pending)
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Linked queue http://127.0.0.1:8983 (1 queues linked)
doveadm([hidden email]): Debug: http-client: queue
<a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Started new connection to 127.0.0.1:8983
doveadm([hidden email]): Debug: http-client: request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
Submitted
doveadm([hidden email]): Debug: http-client: Waiting for 1
requests to finish
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Creating 1 new connections to handle requests (already 0 usable,
connecting to 0, closing 0)
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Making new connection 1 of 1
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: HTTP connection created (1 parallel connections exist)
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Connected
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Ready for requests
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Successfully connected (connections=1)
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Using 1 idle connections to handle 1 requests (1 total connections ready)
doveadm([hidden email]): Debug: http-client: queue
<a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Connection to peer 127.0.0.1:8983 claimed request
[Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]

doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Claimed request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
doveadm([hidden email]): Debug: http-client: request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
Sent header
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
No more requests to service for this peer (1 connections exist)
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Got 200 response for request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
(took 3 ms + 2 ms in queue)
doveadm([hidden email]): Error: fts_solr: Invalid XML input at
1:0: not well-formed (invalid token) (near: {
   "responseHeader":{
     "status":0,
     "QTime":1,
     "params":{
       "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa)
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Response payload stream destroyed (0 ms after initial response)
doveadm([hidden email]): Debug: http-client: request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
Finished
doveadm([hidden email]): Debug: http-client: queue
<a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Dropping request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
doveadm([hidden email]): Debug: http-client: host 127.0.0.1: Host
is idle (timeout = 100 msecs)
doveadm([hidden email]): Debug: http-client: request [Req1: GET
http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
Free (requests left=1)
doveadm([hidden email]): Debug: http-client: All requests finished
doveadm([hidden email]): Error: Mailbox INBOX: Status lookup
failed: Internal error occurred. Refer to server log for more
information. [2017-11-30 13:13:57]
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Peer close
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Peer disconnect
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Peer closed
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Connection disconnect
doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
[0]: Connection destroy
doveadm([hidden email]): Debug: http-client: peer 127.0.0.1:8983:
Peer destroy
doveadm([hidden email]): Debug: http-client: host 127.0.0.1: Host
destroy
doveadm([hidden email]): Debug: http-client: queue
<a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Destroy

Reply | Threaded
Open this post in threaded view
|

Re: My Solr FTS problem

Christian Kivalo-2


Am 30. November 2017 22:14:59 MEZ schrieb Gao <[hidden email]>:

>I am testing the solr FTS following the guide here:
>http://things.m31.ch/?p=379
>
>Now I am having problem when I try to test:
>
># doveadm -v  index -u [hidden email] Inbox
>doveadm([hidden email]): Error: fts_solr: Invalid XML input at
>1:0: not well-formed (invalid token) (near: {
>   "responseHeader":{
>     "status":0,
>     "QTime":0,
>     "params":{
>       "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa)
>doveadm([hidden email]): Error: Mailbox INBOX: Status lookup
>failed: Internal error occurred. Refer to server log for more
>information. [2017-11-30 13:05:24]
>
>My system is CentOS 7, Dovecot 3.2.4 (installed from source with solr
>support), Java OpenJDK-1.8.0, Solr v7.1.0.

The link you posted is for solr 6.5.0 and you are using solr 7.1.0 and solr needs to be configured to accept XML queries instead of json as the default starting with version 7.
Search the list archives there have been postings regarding solr 7 and dovecot in the past.

>Could some one help me on this please?
>
>Here is detailed debug out put:
>
>[root@mail ~]# doveadm -vD  index -u [hidden email] Inbox
>Debug: Loading modules from directory: /usr/lib64/dovecot
>Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so
>Debug: Module loaded: /usr/lib64/dovecot/lib20_fts_plugin.so
>Debug: Module loaded: /usr/lib64/dovecot/lib21_fts_solr_plugin.so
>Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
>Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
>/usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined
>symbol: acl_user_module (this is usually intentional, so just ignore
>this message)
>Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
>/usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined
>symbol: expire_set_deinit (this is usually intentional, so just ignore
>this message)
>Debug: Module loaded:
>/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so
>Debug: Module loaded:
>/usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
>Debug: Module loaded:
>/usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so
>Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
>failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so:
>undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
>intentional, so just ignore this message)
>doveadm([hidden email]): Debug: Added userdb setting:
>mail=maildir:/home/vmail/mydomain.com/fail2ban/
>doveadm([hidden email]): Debug: Added userdb setting:
>plugin/quota_rule=*:bytes=0
>doveadm([hidden email]): Debug: Effective uid=5000, gid=5000,
>home=/home/vmail/mydomain.com/fail2ban/
>doveadm([hidden email]): Debug: Quota root: name=User quota
>backend=maildir args=
>doveadm([hidden email]): Debug: Quota rule: root=User quota
>mailbox=* bytes=0 messages=0
>doveadm([hidden email]): Debug: Quota rule: root=User quota
>mailbox=Trash bytes=+104857600 messages=0
>doveadm([hidden email]): Debug: Quota warning: bytes=0 (90%)
>messages=0 reverse=no command=quota-warning 90 [hidden email]
>mydomain.com
>doveadm([hidden email]): Debug: Quota warning: bytes=0 (80%)
>messages=0 reverse=no command=quota-warning 80 [hidden email]
>mydomain.com
>doveadm([hidden email]): Debug: Quota grace: root=User quota
>bytes=0 (10%)
>doveadm([hidden email]): Debug: Namespace inbox: type=private,
>prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes
>location=maildir:/home/vmail/mydomain.com/fail2ban/
>doveadm([hidden email]): Debug: maildir++:
>root=/home/vmail/mydomain.com/fail2ban, index=, indexpvt=, control=,
>inbox=/home/vmail/mydomain.com/fail2ban, alt=
>doveadm([hidden email]): Debug: quota: quota_over_flag check:
>quota_over_script unset - skipping
>doveadm([hidden email]): Debug: INBOX: Mailbox opened because:
>index
>doveadm([hidden email]): Debug: http-client: host 127.0.0.1:
>Host
>created
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Peer created
>doveadm([hidden email]): Debug: http-client: queue
><a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Setting up connection to 127.0.0.1:8983 (1
>requests pending)
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Linked queue http://127.0.0.1:8983 (1 queues linked)
>doveadm([hidden email]): Debug: http-client: queue
><a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Started new connection to 127.0.0.1:8983
>doveadm([hidden email]): Debug: http-client: request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
>
>Submitted
>doveadm([hidden email]): Debug: http-client: Waiting for 1
>requests to finish
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Creating 1 new connections to handle requests (already 0 usable,
>connecting to 0, closing 0)
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Making new connection 1 of 1
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: HTTP connection created (1 parallel connections exist)
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Connected
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Ready for requests
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Successfully connected (connections=1)
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Using 1 idle connections to handle 1 requests (1 total connections
>ready)
>doveadm([hidden email]): Debug: http-client: queue
><a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Connection to peer 127.0.0.1:8983 claimed
>request
>[Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
>
>
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Claimed request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
>doveadm([hidden email]): Debug: http-client: request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
>
>Sent header
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>No more requests to service for this peer (1 connections exist)
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Got 200 response for request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
>
>(took 3 ms + 2 ms in queue)
>doveadm([hidden email]): Error: fts_solr: Invalid XML input at
>1:0: not well-formed (invalid token) (near: {
>   "responseHeader":{
>     "status":0,
>     "QTime":1,
>     "params":{
>       "q":"box:8864fa1d51ea1d5a7b12000096a1aaf8 AND user:fa)
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Response payload stream destroyed (0 ms after initial response)
>doveadm([hidden email]): Debug: http-client: request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
>
>Finished
>doveadm([hidden email]): Debug: http-client: queue
><a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Dropping request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]
>doveadm([hidden email]): Debug: http-client: host 127.0.0.1:
>Host
>is idle (timeout = 100 msecs)
>doveadm([hidden email]): Debug: http-client: request [Req1: GET
>http://127.0.0.1:8983/solr/dovecot/select?fl=uid&rows=1&sort=uid+desc&q=box:8864fa1d51ea1d5a7b12000096a1aaf8+AND+user:fail2ban@...]:
>
>Free (requests left=1)
>doveadm([hidden email]): Debug: http-client: All requests
>finished
>doveadm([hidden email]): Error: Mailbox INBOX: Status lookup
>failed: Internal error occurred. Refer to server log for more
>information. [2017-11-30 13:13:57]
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Peer close
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Peer disconnect
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Peer closed
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Connection disconnect
>doveadm([hidden email]): Debug: http-client: conn 127.0.0.1:8983
>
>[0]: Connection destroy
>doveadm([hidden email]): Debug: http-client: peer
>127.0.0.1:8983:
>Peer destroy
>doveadm([hidden email]): Debug: http-client: host 127.0.0.1:
>Host
>destroy
>doveadm([hidden email]): Debug: http-client: queue
><a href="http://127.0.0.1:8983:">http://127.0.0.1:8983: Destroy

--
Christian Kivalo
Gao
Reply | Threaded
Open this post in threaded view
|

Re: My Solr FTS problem

Gao
Thanks a lot for the help.

You are right about the JSON as the default in solr v7.1.0. I now
switched it back to XML and now it works.

# doveadm -v index -u [hidden email] Inbox
doveadm([hidden email]): Info: INBOX: Caching mails seq=1..388
388/388

Gao

On 2017-11-30 02:00 PM, Christian Kivalo wrote:
> solr 7 and dovecot