Convert from mbox to Maildir

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Convert from mbox to Maildir

Josef Wolf
Hello everybody,

I am in the process to install dovecot as an IMAP server.

Currently, this system runs postfix and uses procmail for local (mbox
style) delivery.

Procmail ist started by postfix like this:

    /etc/postfix/main.cf:
        mailbox_command = /usr/bin/procmail

    ~/.forward:
        "|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 # jw"

Before going into IMAP details, I decided to convert the system to
Maildir as a first step. So I created a procmail-entryrecipe to populate
a test folder:

    ~/.procmailrc:
        :0
        * ^Subject:.*dovecot-test
        {
          ORGMAIL="${HOME}/Maildir/"
          DEFAULT=${ORGMAIL}
          DELIVER="/usr/lib/dovecot/deliver -o mail_location=maildir:~/Maildir -o lda_mailbox_autocreate=yes"
          :0
          | $DELIVER -m dovecot-test
        }

This setup seems to work.

But there is one point that bugs me: in the first tests, dovecot was
mis-configured (the two -o options were missing) and the test mails
were silently lost. No errors in the procmail logfile. Only dovecot
reports errors in its logfile. But the error seems not to be noticed
by procmail. Therefore procmail assumes deliver could store the mail
successfully and won't keep a backup of it.

I am sure, I'm doing something stupid here. I have never seen such
behavior from procmail before. Normally procmail would report failure
of sub-commands.

Now I hesitate to go forward in this conversion process. I don't want to
loose mails if something is wrong with the dovcot setup.

Besides: I wonder whether the setting

          DEFAULT=${ORGMAIL}

is a good idea. Wouldn't it be better to have

          DEFAULT=/var/spool/mail/$USER

so there would be a fallback in case there are delivery problems to
${ORGMAIL}

Any thougths?

--
Josef Wolf
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

David.M.Clark
Hi Josef,

Just went through this one recently using CentOS with SendMail, Procmail
and IMAP conversion from mbox to Maildir - I needed to make sure I could
run both formats while I migrated users one-by-one.

Not sure if what I did helps, but I created a 'how-to' page on it on my
website rather than post an essay on the e-mail forum:

http://dmc1961.id.au/howto_17101101.php

In essence I am using /etc/procmailrc and the users_db approach to get
e-mail delivered correctly after I have converted an account to Maildir
format. I also never trust anything in my IT life, so I backup the mail
directories under $HOME to mail_old.

Not sure if this is any use to you but works for me now.

On 13/10/17 07:52, Josef Wolf wrote:

> Hello everybody,
>
> I am in the process to install dovecot as an IMAP server.
>
> Currently, this system runs postfix and uses procmail for local (mbox
> style) delivery.
>
> Procmail ist started by postfix like this:
>
>      /etc/postfix/main.cf:
>          mailbox_command = /usr/bin/procmail
>
>      ~/.forward:
>          "|IFS=' ' && exec /usr/bin/procmail -f- || exit 75 # jw"
>
> Before going into IMAP details, I decided to convert the system to
> Maildir as a first step. So I created a procmail-entryrecipe to populate
> a test folder:
>
>      ~/.procmailrc:
>          :0
>          * ^Subject:.*dovecot-test
>          {
>            ORGMAIL="${HOME}/Maildir/"
>            DEFAULT=${ORGMAIL}
>            DELIVER="/usr/lib/dovecot/deliver -o mail_location=maildir:~/Maildir -o lda_mailbox_autocreate=yes"
>            :0
>            | $DELIVER -m dovecot-test
>          }
>
> This setup seems to work.
>
> But there is one point that bugs me: in the first tests, dovecot was
> mis-configured (the two -o options were missing) and the test mails
> were silently lost. No errors in the procmail logfile. Only dovecot
> reports errors in its logfile. But the error seems not to be noticed
> by procmail. Therefore procmail assumes deliver could store the mail
> successfully and won't keep a backup of it.
>
> I am sure, I'm doing something stupid here. I have never seen such
> behavior from procmail before. Normally procmail would report failure
> of sub-commands.
>
> Now I hesitate to go forward in this conversion process. I don't want to
> loose mails if something is wrong with the dovcot setup.
>
> Besides: I wonder whether the setting
>
>            DEFAULT=${ORGMAIL}
>
> is a good idea. Wouldn't it be better to have
>
>            DEFAULT=/var/spool/mail/$USER
>
> so there would be a fallback in case there are delivery problems to
> ${ORGMAIL}
>
> Any thougths?
>

--

As always, I remain at your service.

Kindest Regards,
David.M.Clark (Director - Senior Linux/UNIX Consultant)
=--------------------------------------------------------------------------=
  Davrom Consulting Pty Ltd            Mobile: 0418763124
  PO Box 1644, Sunnybank Hills, 4109   E-mail (Work): [hidden email]
  ABN: 81 096 990 804                  E-mail (Priv): [hidden email]
  Website: http://davrom.com           Skype: dmc1961
  Podcast: http://ldup.com.au          Google: [hidden email]
=--------------------------------------------------------------------------=
  Specialising in: Linux (Fedora/RedHat/CentOS), UNIX, SCO, MikroTik,
                   Networking/Internet, E-mail/Web Technologies
=--------------------------------------------------------------------------=
  Please note: Any e-mail communication bearing this signature is for
  the exclusive purpose of the sender and is not for publication
  without the expressed permission of the sender or respective
  sender's organisation.
=--------------------------------------------------------------------------=
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

Josef Wolf
On Fri, Oct 13, 2017 at 11:34:22AM +1000, David.M.Clark wrote:
> Just went through this one recently using CentOS with SendMail, Procmail and
> IMAP conversion from mbox to Maildir - I needed to make sure I could run
> both formats while I migrated users one-by-one.

Thanks for your suggestion, David!

My approach is similar. But I want to migrate not only users one-by-one but
also their mailboxes one-by-one.

I'm going the route that procmail will deliver to mbox folders directly. So
dovecot-lda won't even see mails for mbox folders.

But that's not the point that bothers me.

The problem is that in case of problems (e.g. broken/bad dovecot
configuration) mail is silently dropped without passing back the error
indication back to procmail. Therefore procmail assumes the delivery was
successful and won't bother to store it into some safe place. This effectively
results in lost mail.

--
Josef Wolf
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

Nikolai Lusan
In reply to this post by David.M.Clark
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Greetings,

On Fri, 2017-10-13 at 11:34 +1000, David.M.Clark wrote:

> Just went through this one recently using CentOS with SendMail, Procmail
> and IMAP conversion from mbox to Maildir - I needed to make sure I could
> run both formats while I migrated users one-by-one.

When I have migrated solutions from mbox to maildir (I have done more than
one) I found it easier to get a mail client to do the hard work for me (if
you're on a LAN with the server it's faster). I created new usernames with
maildir as their base, and copied the folder over. This approach could be
automated, probably easier than going via a script to open each mbox and
push each message through an LDA. There are a heap of examples around for
connecting to IMAP via PERL (https://www.example-code.com/perl/imap.asp is
good). If you write your perl script properly you could migrate multiple
users at once from the cli ... xargs is a wonderful tool :). Unless you are
hosting a mail service that needs to be up 24/7 you should be able to
migrate large amounts of mail out of hours. If you have users that have
been using POP to get their email and want to move them to IMAP it's also
easier to have the MUA do the work (I have migrated users from using POP
and local storage, to remote IMAP this way - especially users of MS
Outlook).


> In essence I am using /etc/procmailrc and the users_db approach to get
> e-mail delivered correctly after I have converted an account to Maildir
> format. I also never trust anything in my IT life, so I backup the mail
> directories under $HOME to mail_old.

Yeah, safe bet there. Creating copies before starting the migration process
 is the best way (depending on the underlying filesystem also probably
faster if you need to restore something). Even using a second copy as the
base to push things from mbox into a new maildir.

While I like procmail (it has some nice features), I also migrated things
to sieve a while back. There are a couple of things I can do with procmail
that I have found it hard to match with sieve, but properly nested if
statements mean I can traverse my rules faster. Most people will end up
using MUA filters on an inbox rather than a procmail/sieve setup (the
number of MUA's that support either of these server side filters is
minimal, and the number that support them properly is even smaller - I did
have a webmail client destroy my nested "if" statements). Using procmail or
sieve as an intermediate step for initial transition from mbox to maildir
is a safe bet, then it comes down to having dovecot setup properly to
deliver into maildir.

- --
Nikolai Lusan <[hidden email]>
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAlngpycACgkQ4ZaDRV2V
L6SDsxAAn65oBEhOb4HxOzL5oAg+GREziMEcav0iOkXwjkGi6ithX3xc5T9vgA36
dJWDgkTdTVN82vaZKu3Eg7V8baqld5l6alFRJv5nhU3G37m287SapxCG3smwT8OA
2y3SAkPJO905cOcINYGESIMLtBtRSympDcVtM8+gHjy7+xSWiiaxEt3wSojbmrAT
NqunbuBCrDQXK5GeWYBcOPTHj+7Qb3CEGCEo3mTDtgIBsNRpw07PIc+px0KbRJT2
4h5Gft1IzHUC9b63yRXUK7O00SAxxjAQ2fJAmjq5kBxPuwmW70pbcQMPrTQd4HAL
X8QtzoqoLPCLHu3Ptx5eRLpw3IUSiHWFeBZGG/5RWx/GPyzKFMMFmSURKjxmVbkG
imo+2vR8LgBFsE1KXc7cRPjkHaTJNus9N7LvWepR/uSDthXJGWpYttCilAm9wmRG
usflda8Qqq+AJxiUOYP4n+Qu130yNaW0K5T4LJRVJ1DHiQcRRrk6D+r8wpmG6nBi
jrJR7/ge2mk+axl1PBJklAycTxdjEFwdXA3/NmBA0nN1NLxMMdVMneipGuAYQVP7
JJA6mMySMahj1c9DSDj8O8x3cbc2uSXSB7AzKogRnnWeKewh1DoEIeVSiMwJysvS
0lNEWGIzE/cXG72JQniDGrKLZZN5yJEuKZPiSmEbJB/rYOyGpGE=
=JJgP
-----END PGP SIGNATURE-----
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

Joseph Tam-2
In reply to this post by Josef Wolf

Nikolai Lusan <[hidden email]> writes:

>> Just went through this one recently using CentOS with SendMail, Procmail
>> and IMAP conversion from mbox to Maildir - I needed to make sure I could
>> run both formats while I migrated users one-by-one.
>
> When I have migrated solutions from mbox to maildir (I have done more than
> one) I found it easier to get a mail client to do the hard work for me (if
> you're on a LAN with the server it's faster). I created new usernames with
> maildir as their base, and copied the folder over.

Reading over David's and various schemes, the tricky part is consistency:
avoiding race conditions where you have the user manipulating their
mailboxes and your MTA/LDA delivering mail, while simultaneously doing
mbox->Maildir conversion.

To guarantee consistency, it's necessary to lock out other avenues of
mbox changes while the conversion takes place.  Otherwise, you'll have the
situation where mail can get duplicated, or worse, deleted.  For example,
think of this sequence: convert B, user moves message A->B, convert A.

I think the safest is method is

  1) Deny the user access to IMAP/POP: deny
  userdb/passdb, or lock their password
  and flush their auth cache entry.
  2) TEMPFAIL MTA delivery to user during conversion
  3) Kick the user
  4) Do the conversion
  5) Undo 1+2

If you have a smart conversion script (one that recognizes changes and
only do delta updates) you can greatly speed up step 4 and minimize
outage time.

  0) Do an initial conversion that will do the bulk of the work
  but does not guranatee consistency.

Joseph Tam <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

Marc Stürmer-2
In reply to this post by Josef Wolf
Zitat von Josef Wolf <[hidden email]>:

> Any thougths?

Deliver IMHO should be run directly by the MTA, if possible, because  
being run by procmail you loose the capability of using SIEVE.

The dovecot way to convert mailbox formats is the dsync (doveadm sync)  
command, which is the Swiss army knife for this kind of purpose.

And if you are in need of converting a bunch of inboxes, one after  
another, keep in mind that we got the mail_location = auto: command,  
where dovecot decides on runtime by the name of the underlyding  
subfolder about the mail format to be used.

So for example, dovecot can distinguish between ~/Maildir, ~/Mail,  
~/mdbox and others. This gives you the tools at hand to migrate  
literally tens of thousands of mailboxes, while your services are  
running normally, one after another.
Reply | Threaded
Open this post in threaded view
|

Re: Convert from mbox to Maildir

Josef Wolf
On Sat, Oct 14, 2017 at 03:56:17PM +0200, Marc Stürmer wrote:
> Zitat von Josef Wolf <[hidden email]>:

> Deliver IMHO should be run directly by the MTA, if possible, because being
> run by procmail you loose the capability of using SIEVE.

I have not looked into SIEVE yet. From some ramblings on the mailing list,
SIEVE seems not to be as powerful as procmail?

> And if you are in need of converting a bunch of inboxes, one after another,
> keep in mind that we got the mail_location = auto: command, where dovecot
> decides on runtime by the name of the underlyding subfolder about the mail
> format to be used.
>
> So for example, dovecot can distinguish between ~/Maildir, ~/Mail, ~/mdbox
> and others. This gives you the tools at hand to migrate literally tens of
> thousands of mailboxes, while your services are running normally, one after
> another.

My problem is NOT the conversion process (yet). For conversion, I have created
procmail recipes like this:

  :0
  * ^TOgimp-user
  {
     :0:
     * ? test -f $HOME/Mail/ML.gimp-user
     ML.gimp-user
     :0
     {
       MAILDIR=$HOME/Maildir/
       DEFAULT=$HOME/Maildir/
       :0
       | $DELIVER -m ML.gimp-user
     }
  }

With this, procmail will file into mbox when an mbox folder exists, and into
Maildir if no mbox folder exists. That way I can simply remove mbox files one
by one. No changes to any config files are needed when folders are converted.


But as I stated before, my REAL concern is that mail is lost when dovecot
fails to store the mail. Reporting the failure from dovecot-lda back to
procmail doesn't work properly. Any thougths on THIS topic?


--
Josef Wolf
[hidden email]