dsync deleting too many emails (sdbox)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

dsync deleting too many emails (sdbox)

Asheesh Laroia-14
I'm using dsync to synchronize emails on a laptop where wifi connectivity
sometimes fails in the middle of a sync. I have a shell script that runs
dsync, and here is one line of it including the output of dsync:

+ dsync -f -m realmail mirror /home/paulproteus/projects/ssh-attach/run ssh rose.makesad.us dsync
dsync-local(paulproteus): Error: dbox /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Unexpectedly lost uid=337460
dsync-local(paulproteus): Error: msg guid lookup failed: Internal error occurred. Refer to server log for more information. [2012-02-02 11:02:12]
dsync-local(paulproteus): Warning: sdbox /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Rebuilding index
dsync-local(paulproteus): Warning: sdbox /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid filename 130608.broken
dsync-local(paulproteus): Warning: sdbox /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid filename 66159.broken
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error: link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159, /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159.broken) failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file: /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.66159
dsync-local(paulproteus): Warning: sdbox /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails: Ignoring invalid filename 336269.broken
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error: link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608, /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608.broken) failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file: /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.130608
dsync-remote(paulproteus): Error: proxy server timed out
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Error: link(/home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269, /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269.broken) failed: File exists
dsync-local(paulproteus): Error: Corrupted dbox file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269 (around offset=16): EOF reading msg header (got 0/30 bytes)
dsync-local(paulproteus): Warning: sdbox: Skipping unfixable file: /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/u.336269
dsync-local(paulproteus): Warning: Transaction log file /home/paulproteus/Maildir/dbox/mailboxes/realmail/dbox-Mails/dovecot.index.log was locked for 1528 seconds

That seemed problematic, but not dangerous.

Then I ran a fresh sync, and found 3000 (of 60,000) messages had been deleted and expunged
from the "realmail" box.

I'm guessing this is some bad interaction with sdbox and partial file downloads?

I haven't read the code for this, but I would guess the dsync process isn't being atomic
about file transfers, so it is leaving half-completed transfers in place, which results
in corrupt files when they're next examined.

dovecot -n output:

# 2.0.15: /etc/dovecot/dovecot.conf
# OS: Linux 3.0.0-2-amd64 x86_64 Debian wheezy/sid
mail_location = sdbox:~/Maildir/dbox
passdb {
  driver = pam
}
protocols = " imap"
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
  driver = passwd
}

-- Asheesh.
Reply | Threaded
Open this post in threaded view
|

Re: dsync deleting too many emails (sdbox)

Timo Sirainen
On Thu, 2012-02-02 at 14:59 -0500, Asheesh Laroia wrote:
> I'm guessing this is some bad interaction with sdbox and partial file downloads?
>
> I haven't read the code for this, but I would guess the dsync process isn't being atomic
> about file transfers, so it is leaving half-completed transfers in place, which results
> in corrupt files when they're next examined.

There were some problems related to this in dbox, although in your case
it seems to be worse than what it should..

I've anyway done several fixes in v2.1. Can you try if these problems
happen with it too? And in any case cleanup the dbox from the *.broken
files, so that "doveadm force-resync" won't give any errors.


Reply | Threaded
Open this post in threaded view
|

Re: dsync deleting too many emails (sdbox)

Timo Sirainen
On 9.2.2012, at 21.47, Timo Sirainen wrote:

> I've anyway done several fixes in v2.1. Can you try if these problems
> happen with it too? And in any case cleanup the dbox from the *.broken
> files, so that "doveadm force-resync" won't give any errors.

A bit more specifically: The last such dbox bug was fixed only today, so you'd need v2.1 hg version or wait for v2.1.rc6 which should happen this week.

And in general: It would be helpful to have a clean fully working dbox, and then know the *first* error(s) that gets printed about dsync corrupting it. Otherwise it's difficult to guess what are some old problems and what are new ones and which problems happens only because of another problem.

Reply | Threaded
Open this post in threaded view
|

Re: dsync deleting too many emails (sdbox)

Asheesh Laroia
On Thu, 9 Feb 2012, Timo Sirainen wrote:

> On 9.2.2012, at 21.47, Timo Sirainen wrote:
>
>> I've anyway done several fixes in v2.1. Can you try if these problems
>> happen with it too? And in any case cleanup the dbox from the *.broken
>> files, so that "doveadm force-resync" won't give any errors.
>
> A bit more specifically: The last such dbox bug was fixed only today, so
> you'd need v2.1 hg version or wait for v2.1.rc6 which should happen this
> week.
>
> And in general: It would be helpful to have a clean fully working dbox,
> and then know the *first* error(s) that gets printed about dsync
> corrupting it. Otherwise it's difficult to guess what are some old
> problems and what are new ones and which problems happens only because
> of another problem.

Good to know. This weekend I can try to set up something of a 'lab' for
testing dsync + (s)dbox, both to see if I can reproduce the errors with
the old versions, and to see if the new versions fix them. I'll keep in
mind the consideration of knowing the first error that gets printed!

-- Asheesh.