Maildir filename has wrong S value

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Maildir filename has wrong S value

Wouter de Geus-3
Hello folks,

This morning I tried to open an old archive mail folder using Mutt.
However, while fetching headers it aborted.
Checking the dovecot log gave me this:

@400000004fb21996267d37d4 imap(benv): Error: Cached message size smaller than expected (9115 < 9420)
@400000004fb21996267e8bac imap(benv): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S to /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9420:2,S
@400000004fb21996267e937c imap(benv): Error: Corrupted index cache file /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/dovecot.index.cache: Broken physical size for mail UID 294
@400000004fb21996267eaaec imap(benv): Error: read(/home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S) failed: Input/output error (uid=294)
@400000004fb219962680683c imap(benv): Info: Disconnected: Internal error occurred. Refer to server log for more information. [2012-05-15 10:53:32] in=6503 out=192718

I've retried opening this folder several times, but each time Dovecot only fixes 1 file and then aborts.
(the folder apparently has a lot more files with incorrect sizes).
My question here is: Why does dovecot abort? Isn't the issue fixed after the rename?

Also: The files are not corrupted or unreadable, reading them through the commandline works fine.
I've looked through my backups and saw that at least half a year ago this mismatch in size was already present.

I've been running dovecot for quite a while, so the cause of this size mismatch might very well be a issue in an older version of dovecot / my configuration.
Right now I'm running dovecot 2.1.6, but I've been running the v1.2 version for quite a while before that.

Is this something that can be improved in dovecot so it doesn't abort after 1 rename?
(of course I could script a rename operation, but that seems like a workaround to me)

Thanks for reading.

Wouter.




My current configuration:
=========================
# 2.1.6: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32.32-g3d14ce7 x86_64 Slackware 11.0.0
base_dir = /var/run/dovecot2/
disable_plaintext_auth = no
first_valid_uid = 89
info_log_path = /dev/stderr
last_valid_uid = 89
log_path = /dev/stderr
log_timestamp =
mail_debug = yes
mail_location = maildir:~/Maildir
mail_max_userip_connections = 50
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify e
nvironment mailbox date
namespace {
  inbox = yes
  location =
  prefix = INBOX.
  separator = .
  type = private
}
passdb {
  driver = vpopmail
}
plugin {
  autocreate = INBOX.Spam
  quota = maildir
  sieve = ~/.sieve/dovecot.sieve
  sieve_dir = ~/.sieve
  sieve_global_dir = /etc/dovecot/sieve/
  sieve_subaddress_sep = -+
}
protocols = imap pop3 sieve
service auth {
  unix_listener auth-master {
    group = vchkpw
    mode = 0660
  }
  unix_listener auth-userdb {
    group = vchkpw
    mode = 0660
  }
}
service imap-login {
  inet_listener imap {
    address = [::] *
    port = 143
  }
  inet_listener imaps {
    address = [::] *
    port = 993
  }
  process_limit = 256
  process_min_avail = 3
  service_count = 1
  user = dovecot
}
service managesieve-login {
  process_limit = 256
  process_min_avail = 3
  service_count = 1
  user = dovecot
}
service pop3-login {
  inet_listener pop3 {
    address = [::] *
    port = 110
  }
  inet_listener pop3s {
    address = [::] *
    port = 995
  }
  process_limit = 256
  process_min_avail = 3
  service_count = 1
  user = dovecot
}
ssl_cert = </var/qmail/control/servercert.pem
ssl_cipher_list = ALL:!LOW
ssl_key = </var/qmail/control/servercert.pem
userdb {
  driver = vpopmail
}
verbose_proctitle = yes
protocol lda {
  hostname = mail.benv.junerules.com
  info_log_path = /var/log/dovecot/dovecot2-deliver.log
  log_path = /var/log/dovecot/dovecot2-deliver-errors.log
  mail_plugins = sieve
  postmaster_address = [hidden email]
  sendmail_path = /var/qmail/bin/sendmail
}
protocol imap {
  imap_max_line_length = 64 k
  mail_plugins = quota imap_quota autocreate
}
protocol pop3 {
  mail_plugins = quota autocreate
  pop3_no_flag_updates = no
  pop3_uidl_format = %v-%u
}
=================================
Reply | Threaded
Open this post in threaded view
|

Re: Maildir filename has wrong S value

Timo Sirainen
On Tue, 2012-05-15 at 12:23 +0200, Wouter de Geus wrote:

> Hello folks,
>
> This morning I tried to open an old archive mail folder using Mutt.
> However, while fetching headers it aborted.
> Checking the dovecot log gave me this:
>
> @400000004fb21996267d37d4 imap(benv): Error: Cached message size smaller than expected (9115 < 9420)
> @400000004fb21996267e8bac imap(benv): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S to /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9420:2,S
> @400000004fb21996267e937c imap(benv): Error: Corrupted index cache file /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/dovecot.index.cache: Broken physical size for mail UID 294
> @400000004fb21996267eaaec imap(benv): Error: read(/home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S) failed: Input/output error (uid=294)
> @400000004fb219962680683c imap(benv): Info: Disconnected: Internal error occurred. Refer to server log for more information. [2012-05-15 10:53:32] in=6503 out=192718
>
> I've retried opening this folder several times, but each time Dovecot only fixes 1 file and then aborts.
> (the folder apparently has a lot more files with incorrect sizes).
> My question here is: Why does dovecot abort? Isn't the issue fixed after the rename?

That file was fixed, but you probably have tons of them and it aborts
after each one. With IMAP protocol there's really no other good way to
do this. I guess if it notices one file being wrong it could just decide
to go into error recovery check and read through all of the maildir
files, but that's annoying extra piece of code for something that
shouldn't happen anyway..

You should be able to manually fix them by reading them via doveadm,
something like:

doveadm fetch -u benv text all > /dev/null

> Also: The files are not corrupted or unreadable, reading them through the commandline works fine.
> I've looked through my backups and saw that at least half a year ago this mismatch in size was already present.
>
> I've been running dovecot for quite a while, so the cause of this size mismatch might very well be a issue in an older version of dovecot / my configuration.
> Right now I'm running dovecot 2.1.6, but I've been running the v1.2 version for quite a while before that.

Old Dovecot versions didn't notice the problem. You can make v2.1 also
just ignore the wrong sizes with maildir_broken_filename_sizes=yes.


Reply | Threaded
Open this post in threaded view
|

Re: Maildir filename has wrong S value

Wouter de Geus-3
* Timo Sirainen <[hidden email]> [2012-05-19 22:17:01 +0300]:

> > My question here is: Why does dovecot abort? Isn't the issue fixed after the rename?
>
> That file was fixed, but you probably have tons of them and it aborts
> after each one. With IMAP protocol there's really no other good way to
> do this. I guess if it notices one file being wrong it could just decide
> to go into error recovery check and read through all of the maildir
> files, but that's annoying extra piece of code for something that
> shouldn't happen anyway..

Ah, good point on the protocol. However, as far as I'm concerned this automatic
recovery would be a good thing (under the "self healing" statement from the dovecot site?)
Just a suggestion though, I understand that it's a lot of work for this specific issue.

> You should be able to manually fix them by reading them via doveadm,
> something like:
>
> doveadm fetch -u benv text all > /dev/null

Thanks for that one, worked great for my maildir. (and saved me from writing a shell script) :)

> Old Dovecot versions didn't notice the problem. You can make v2.1 also
> just ignore the wrong sizes with maildir_broken_filename_sizes=yes.

Ah, that explains.

Well, thanks for the support :)

Regards,

Wouter.
Reply | Threaded
Open this post in threaded view
|

Re: Maildir filename has wrong S value

Ralf Hildebrandt
In reply to this post by Timo Sirainen
* Timo Sirainen <[hidden email]>:

> That file was fixed, but you probably have tons of them and it aborts
> after each one. With IMAP protocol there's really no other good way to
> do this. I guess if it notices one file being wrong it could just decide
> to go into error recovery check and read through all of the maildir
> files, but that's annoying extra piece of code for something that
> shouldn't happen anyway..

Well, it happened for me :)

--
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  CharitĂ© - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  [hidden email] | http://www.charite.de
           
Reply | Threaded
Open this post in threaded view
|

Re: Maildir filename has wrong S value

Patrick Domack
Quoting Ralf Hildebrandt <[hidden email]>:

> * Timo Sirainen <[hidden email]>:
>
>> That file was fixed, but you probably have tons of them and it aborts
>> after each one. With IMAP protocol there's really no other good way to
>> do this. I guess if it notices one file being wrong it could just decide
>> to go into error recovery check and read through all of the maildir
>> files, but that's annoying extra piece of code for something that
>> shouldn't happen anyway..
>
> Well, it happened for me :)
>

Upgrading to 2.1.4, and using a script to go through the whole  
mailstore and correct/fix them did it for me.
I have sense not seen any more of the errors in my logs.


Reply | Threaded
Open this post in threaded view
|

Re: Maildir filename has wrong S value

Michael Gmelin-2
In reply to this post by Wouter de Geus-3
On 15.5.12 12:23, Wouter de Geus wrote:

> Hello folks,
>
> This morning I tried to open an old archive mail folder using Mutt.
> However, while fetching headers it aborted.
> Checking the dovecot log gave me this:
>
> @400000004fb21996267d37d4 imap(benv): Error: Cached message size smaller than expected (9115 < 9420)
> @400000004fb21996267e8bac imap(benv): Error: Maildir filename has wrong S value, renamed the file from /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S to /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9420:2,S
> @400000004fb21996267e937c imap(benv): Error: Corrupted index cache file /home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/dovecot.index.cache: Broken physical size for mail UID 294
> @400000004fb21996267eaaec imap(benv): Error: read(/home/vpopmail/domains/benv.junerules.com/benv/Maildir/.Old.2009/cur/1260395566.28175.black,S=9115:2,S) failed: Input/output error (uid=294)
> @400000004fb219962680683c imap(benv): Info: Disconnected: Internal error occurred. Refer to server log for more information. [2012-05-15 10:53:32] in=6503 out=192718
>
> I've retried opening this folder several times, but each time Dovecot only fixes 1 file and then aborts.
> (the folder apparently has a lot more files with incorrect sizes).
> My question here is: Why does dovecot abort? Isn't the issue fixed after the rename?
>
> Also: The files are not corrupted or unreadable, reading them through the commandline works fine.
> I've looked through my backups and saw that at least half a year ago this mismatch in size was already present.
>
> I've been running dovecot for quite a while, so the cause of this size mismatch might very well be a issue in an older version of dovecot / my configuration.
> Right now I'm running dovecot 2.1.6, but I've been running the v1.2 version for quite a while before that.
>
> Is this something that can be improved in dovecot so it doesn't abort after 1 rename?
> (of course I could script a rename operation, but that seems like a workaround to me)
>
> Thanks for reading.
>
> Wouter.
>
>
>
>
> My current configuration:
> =========================
> # 2.1.6: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32.32-g3d14ce7 x86_64 Slackware 11.0.0
> base_dir = /var/run/dovecot2/
> disable_plaintext_auth = no
> first_valid_uid = 89
> info_log_path = /dev/stderr
> last_valid_uid = 89
> log_path = /dev/stderr
> log_timestamp =
> mail_debug = yes
> mail_location = maildir:~/Maildir
> mail_max_userip_connections = 50
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify e
> nvironment mailbox date
> namespace {
>   inbox = yes
>   location =
>   prefix = INBOX.
>   separator = .
>   type = private
> }
> passdb {
>   driver = vpopmail
> }
> plugin {
>   autocreate = INBOX.Spam
>   quota = maildir
>   sieve = ~/.sieve/dovecot.sieve
>   sieve_dir = ~/.sieve
>   sieve_global_dir = /etc/dovecot/sieve/
>   sieve_subaddress_sep = -+
> }
> protocols = imap pop3 sieve
> service auth {
>   unix_listener auth-master {
>     group = vchkpw
>     mode = 0660
>   }
>   unix_listener auth-userdb {
>     group = vchkpw
>     mode = 0660
>   }
> }
> service imap-login {
>   inet_listener imap {
>     address = [::] *
>     port = 143
>   }
>   inet_listener imaps {
>     address = [::] *
>     port = 993
>   }
>   process_limit = 256
>   process_min_avail = 3
>   service_count = 1
>   user = dovecot
> }
> service managesieve-login {
>   process_limit = 256
>   process_min_avail = 3
>   service_count = 1
>   user = dovecot
> }
> service pop3-login {
>   inet_listener pop3 {
>     address = [::] *
>     port = 110
>   }
>   inet_listener pop3s {
>     address = [::] *
>     port = 995
>   }
>   process_limit = 256
>   process_min_avail = 3
>   service_count = 1
>   user = dovecot
> }
> ssl_cert = </var/qmail/control/servercert.pem
> ssl_cipher_list = ALL:!LOW
> ssl_key = </var/qmail/control/servercert.pem
> userdb {
>   driver = vpopmail
> }
> verbose_proctitle = yes
> protocol lda {
>   hostname = mail.benv.junerules.com
>   info_log_path = /var/log/dovecot/dovecot2-deliver.log
>   log_path = /var/log/dovecot/dovecot2-deliver-errors.log
>   mail_plugins = sieve
>   postmaster_address = [hidden email]
>   sendmail_path = /var/qmail/bin/sendmail
> }
> protocol imap {
>   imap_max_line_length = 64 k
>   mail_plugins = quota imap_quota autocreate
> }
> protocol pop3 {
>   mail_plugins = quota autocreate
>   pop3_no_flag_updates = no
>   pop3_uidl_format = %v-%u
> }
> =================================
>
I've been facing the same issue today in a similar setup (qmail +
vpopmail + dovecot) and figured, that the qmail maildir++ patch I'm
using miscalculates the size of mail, since it ignores Delivered-To and
Return-Path in the size calculation. Fixing this and reinstalling
qmail-local ( plus fixing existing mail using doveadm fetch -u username
text all > /dev/null ) solved the problem for me.

The problematic line in qmail-local.c is:

s += fmt_ulong(s,st.st_size); *s++ = 0;

which should be:

s += fmt_ulong(s,st.st_size+rpline.len+dtline.len); *s++ = 0;

(for FreeBSD users: I submitted a patch to the qmail port, fixing this)
http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/168870

Fixing this also fixed fts_squat for me (dovecot 2.1.7)

--
Michael