Recommended tool for migrating IMAP servers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Recommended tool for migrating IMAP servers

Davide Marchi
Hi Friends,
I would like to ask you a suggestion:
I need to migrate a imap server to a new one and then dismiss the old
one.
Reading from relative Dovecot documentation page
(https://wiki.dovecot.org/Migration), more tools are shown:

UW-IMAP's mailutil, imapsync, YippieMove and Larch.

The each mail servers are Linux based, one of this (mine) is Dovecot.
Based on your experience which of these tools would be preferable to
use?


Thank you very much

Davide
x9p
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

x9p
Hi,

I vouch for imapsync. Have used it in the past with quite a big amount of
emails.

cheers.

x0p

> Hi Friends,
> I would like to ask you a suggestion:
> I need to migrate a imap server to a new one and then dismiss the old
> one.
> Reading from relative Dovecot documentation page
> (https://wiki.dovecot.org/Migration), more tools are shown:
>
> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>
> The each mail servers are Linux based, one of this (mine) is Dovecot.
> Based on your experience which of these tools would be preferable to
> use?
>
>
> Thank you very much
>
> Davide
>


Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Harondel J. Sibble-2
Imapsync for sure. Have used it for both IMAP to IMAP and IMAP to Exchange migrations. Works great.


> On Dec 3, 2017, at 2:08 PM, x9p <[hidden email]> wrote:
>
> Hi,
>
> I vouch for imapsync. Have used it in the past with quite a big amount of
> emails.
>
> cheers.
>
> x0p
>
>> Hi Friends,
>> I would like to ask you a suggestion:
>> I need to migrate a imap server to a new one and then dismiss the old
>> one.
>> Reading from relative Dovecot documentation page
>> (https://wiki.dovecot.org/Migration), more tools are shown:
>>
>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>>
>> The each mail servers are Linux based, one of this (mine) is Dovecot.
>> Based on your experience which of these tools would be preferable to
>> use?
>>
>>
>> Thank you very much
>>
>> Davide
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Aki Tuomi-2
Also if you have fs access on both servers, and you are using maildir,
plain rsync works just as well.

Aki


On 04.12.2017 00:17, Harondel J. Sibble wrote:

> Imapsync for sure. Have used it for both IMAP to IMAP and IMAP to Exchange migrations. Works great.
>
>
>> On Dec 3, 2017, at 2:08 PM, x9p <[hidden email]> wrote:
>>
>> Hi,
>>
>> I vouch for imapsync. Have used it in the past with quite a big amount of
>> emails.
>>
>> cheers.
>>
>> x0p
>>
>>> Hi Friends,
>>> I would like to ask you a suggestion:
>>> I need to migrate a imap server to a new one and then dismiss the old
>>> one.
>>> Reading from relative Dovecot documentation page
>>> (https://wiki.dovecot.org/Migration), more tools are shown:
>>>
>>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>>>
>>> The each mail servers are Linux based, one of this (mine) is Dovecot.
>>> Based on your experience which of these tools would be preferable to
>>> use?
>>>
>>>
>>> Thank you very much
>>>
>>> Davide
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Sami Ketola
In reply to this post by Davide Marchi

> On 3 Dec 2017, at 23.23, Davide Marchi <[hidden email]> wrote:
>
> Hi Friends,
> I would like to ask you a suggestion:
> I need to migrate a imap server to a new one and then dismiss the old one.
> Reading from relative Dovecot documentation page (https://wiki.dovecot.org/Migration), more tools are shown:
>
> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>
> The each mail servers are Linux based, one of this (mine) is Dovecot.
> Based on your experience which of these tools would be preferable to use?

If you want to preserve IMAP UID:s and possibly also POP3 UIDL:s then dovecot internal
dsync is the only tool that can do it.

With every other tool you will face end users needing to  invalidate their local caches and
redownloading all headers if not also all mail bodies.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Davide Marchi
In reply to this post by Davide Marchi
[..]
> Hi,
>
> I vouch for imapsync. Have used it in the past with quite a big amount
> of
> emails.
>
> cheers.
>
> x0p

Ah, thanks  x0!

> Also if you have fs access on both servers, and you are using maildir,
> plain rsync works just as well.
>
> Aki

no, I've not fs access on both servers! :-/

> If you want to preserve IMAP UID:s and possibly also POP3 UIDL:s then
> dovecot internal
> dsync is the only tool that can do it.
>
> With every other tool you will face end users needing to  invalidate
> their local caches and
> redownloading all headers if not also all mail bodies.
>
> Sami
>

Can I use this tool even if I do not know the other remote server
typology?

Many thanks to all!!

Davide



--
firma

cosmogoniA <https://www.cosmogonia.org/>
n o p r o v a r e n o f a r e o n o n f a r e n o n c e p r o v a r e
x9p
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

x9p
>> Hi,
>>
>> I vouch for imapsync. Have used it in the past with quite a big amount
>> of
>> emails.
>>
>> cheers.
>>
>> x0p
>
> Ah, thanks  x0!
>

welcome!

>
> Can I use this tool even if I do not know the other remote server
> typology?
>

sure. just need both IMAP ports reachable and valid user/pass for both
servers.

cheers.

x9p



Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Paolo-2
Il 04/12/2017 14:33, x9p ha scritto:
>
>> Can I use this tool even if I do not know the other remote server
>> typology?
>>
> sure. just need both IMAP ports reachable and valid user/pass for both
> servers.
I think Davide was asking about dsync. If so, the answer is no: dsync
works only when both servers are Dovecot and needs some additional
configuration to work through the network (see
https://wiki2.dovecot.org/Replication).
I don't know about imapsync but I suppose it is a generic IMAP tool that
replicates mailboxes using IMAP protocol as a client between two servers.

Cheers
Paolo

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Michael Slusarz-2
> On December 4, 2017 at 8:46 AM Paolo <[hidden email]> wrote:
>
>
> Il 04/12/2017 14:33, x9p ha scritto:
> >
> >> Can I use this tool even if I do not know the other remote server
> >> typology?
> >>
> > sure. just need both IMAP ports reachable and valid user/pass for both
> > servers.
> I think Davide was asking about dsync. If so, the answer is no: dsync
> works only when both servers are Dovecot and needs some additional
> configuration to work through the network (see
> https://wiki2.dovecot.org/Replication).

This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP server.

The recommended tool for migrating into Dovecot is dsync.  You don't need any other tool, and other tools aren't going to preserve state so they are pretty much worthless for a real-world in-place migration.

michael
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Webert de Souza Lima
In reply to this post by Sami Ketola
On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola <[hidden email]> wrote:

>
> With every other tool you will face end users needing to  invalidate their
> local caches and
> redownloading all headers if not also all mail bodies.
>
> Sami
>
>
I don't think so. Been using imapsync for large scale migrations from
external servers to our dovecot setup. Users don't even see it when the key
is switched (DNS changes).
Go for it.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Joseph Tam-2
In reply to this post by Davide Marchi
"Davide Marchi" <[hidden email]> writes:

>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.

Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
of time to kill.  It is dreadfully slow -- I used it to export some of
my users' mailboxes to Gmail or other remote mail servers, and I could
almost cut&paste the messages faster.

Like Aki said, if you have the same mailbox format and FS access on both
sides, rsync is much simpler.  You can also try exporting the
old mailboxes via NFS, and with some artful symlinks, march through your user
mailboxes replacing the symlinks with the instantiated local copies and
have almost zero downtime.

Joseph Tam <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Steve Litt
On Mon, 4 Dec 2017 12:53:15 -0800 (PST)
Joseph Tam <[hidden email]> wrote:

> "Davide Marchi" <[hidden email]> writes:
>
> >> UW-IMAP's mailutil, imapsync, YippieMove and Larch.  
>
> Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
> of time to kill.  It is dreadfully slow -- I used it to export some of
> my users' mailboxes to Gmail or other remote mail servers, and I could
> almost cut&paste the messages faster.
>
> Like Aki said, if you have the same mailbox format and FS access on
> both sides, rsync is much simpler.  You can also try exporting the
> old mailboxes via NFS, and with some artful symlinks, march through
> your user mailboxes replacing the symlinks with the instantiated
> local copies and have almost zero downtime.
>
> Joseph Tam <[hidden email]>

Another possibility is to use an email client like Claws-Mail, which is
very fast, create two accounts: One for the old IMAP, one for the new
one, and just copy trees.

SteveT

Steve Litt
November 2017 featured book: Troubleshooting: Just the Facts
http://www.troubleshooters.com/tjust
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Sami Ketola
In reply to this post by Webert de Souza Lima

> On 4 Dec 2017, at 19.59, Webert de Souza Lima <[hidden email]> wrote:
>
> On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola <[hidden email]> wrote:
>
>>
>> With every other tool you will face end users needing to  invalidate their
>> local caches and
>> redownloading all headers if not also all mail bodies.
>>
>> Sami
>>
>>
> I don't think so. Been using imapsync for large scale migrations from
> external servers to our dovecot setup. Users don't even see it when the key
> is switched (DNS changes).
> Go for it.

You are wrong. There is no way to assign IMAP UID:s over IMAP protocol. It simply does not support it.
With imapsync there is absolutely no way to preserve them and you will face problems with IMAP UID:s
not matching the cached mail anymore.

Trust us. We have run multiple migrations at scale of 10+ million users.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Paolo-2
In reply to this post by Michael Slusarz-2
Il 04/12/2017 17:37, Michael Slusarz ha scritto:
>> I think Davide was asking about dsync. If so, the answer is no: dsync
>> works only when both servers are Dovecot and needs some additional
>> configuration to work through the network (see
>> https://wiki2.dovecot.org/Replication).
> This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP server.
>
> The recommended tool for migrating into Dovecot is dsync.  You don't need any other tool, and other tools aren't going to preserve state so they are pretty much worthless for a real-world in-place migration.
Entirely?! At most half incorrect. My apologies, I didn't know about the
dsync's IMAP feature (I suppose it wasn't there from the beginning).
This fact still remain: you can use dsync only when dovecot is involved.
Even so (IMHO) the sentence "You don't need any other tool" is a bit too
much. Who ever know all possible use cases in the world?
Cheers
Paolo
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Sami Ketola

> On 5 Dec 2017, at 10.20, Paolo <[hidden email]> wrote:
>
> Il 04/12/2017 17:37, Michael Slusarz ha scritto:
>>> I think Davide was asking about dsync. If so, the answer is no: dsync
>>> works only when both servers are Dovecot and needs some additional
>>> configuration to work through the network (see
>>> https://wiki2.dovecot.org/Replication).
>> This is entirely incorrect.  The source platform for dsync can be ANY IMAP/POP server.
>>
>> The recommended tool for migrating into Dovecot is dsync.  You don't need any other tool, and other tools aren't going to preserve state so they are pretty much worthless for a real-world in-place migration.
> Entirely?! At most half incorrect. My apologies, I didn't know about the dsync's IMAP feature (I suppose it wasn't there from the beginning).
> This fact still remain: you can use dsync only when dovecot is involved.
> Even so (IMHO) the sentence "You don't need any other tool" is a bit too much. Who ever know all possible use cases in the world?
> Cheers
> Paolo

IMAPC storage driver has been part of dovecot since January 2011:

https://github.com/dovecot/core/commit/9fb018dea4e2073639249ea8a14ae27cab2c0aac <https://github.com/dovecot/core/commit/9fb018dea4e2073639249ea8a14ae27cab2c0aac>

Sami


Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Webert de Souza Lima
In reply to this post by Sami Ketola
On Tue, Dec 5, 2017 at 4:16 AM, Sami Ketola <[hidden email]> wrote:

> On Mon, Dec 4, 2017 at 9:16 AM, Sami Ketola <[hidden email]>
>  wrote:
>
>>
>> With every other tool you will face end users needing to  invalidate
>> their local caches and
>> redownloading all headers if not also all mail bodies.
>>
>> Sami
>>
>> > On 4 Dec 2017, at 19.59, Webert de Souza Lima <[hidden email]>
> wrote:
> >
> > I don't think so. Been using imapsync for large scale migrations from
> > external servers to our dovecot setup. Users don't even see it when the
> key
> > is switched (DNS changes).
> > Go for it.
>
> You are wrong. There is no way to assign IMAP UID:s over IMAP protocol. It
> simply does not support it.
> With imapsync there is absolutely no way to preserve them and you will
> face problems with IMAP UID:s
> not matching the cached mail anymore.
>
> Trust us. We have run multiple migrations at scale of 10+ million users.
>
> Sami


Sorry, I might be wrong about cache invalidation, indeed. What I'm sure is
that users will hardly notice any server change.
We've never had user complaints about mailboxes resyncing or anything like
that after imapsync'ing to a new server, that's why I'd recommend using
imapsync with no worries.

It's easy enough to test this on a single user account first and see how is
imap client's behavior after the DNS change.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Sami Ketola

> On 5 Dec 2017, at 12.46, Webert de Souza Lima <[hidden email]> wrote:
>
> Sorry, I might be wrong about cache invalidation, indeed. What I'm sure is
> that users will hardly notice any server change.
> We've never had user complaints about mailboxes resyncing or anything like
> that after imapsync'ing to a new server, that's why I'd recommend using
> imapsync with no worries.
>
> It's easy enough to test this on a single user account first and see how is
> imap client's behavior after the DNS change.
>

Can't really do DNS change to test with one user as DNS change affects all users. What we usually
do is put layer of dovecot proxies in front of the legacy and destination platform and then have database
with information if user has been migrated already and forward the connection to legacy or dovecot
platform based on that info. That enables us to migrate user-by-user.

And both users and mail server admins do notice if there is IMAP UID or POP3 UIDL changes. Imagine
what happens when millions of users suddenly start to redownload terabytes of data.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Sebastian Arcus-2
In reply to this post by Steve Litt
On 04/12/17 23:15, Steve Litt wrote:

> On Mon, 4 Dec 2017 12:53:15 -0800 (PST)
> Joseph Tam <[hidden email]> wrote:
>
>> "Davide Marchi" <[hidden email]> writes:
>>
>>>> UW-IMAP's mailutil, imapsync, YippieMove and Larch.
>>
>> Whatever you use, *don't* use UW-IMAP's mailutil unless you got lots
>> of time to kill.  It is dreadfully slow -- I used it to export some of
>> my users' mailboxes to Gmail or other remote mail servers, and I could
>> almost cut&paste the messages faster.
>>
>> Like Aki said, if you have the same mailbox format and FS access on
>> both sides, rsync is much simpler.  You can also try exporting the
>> old mailboxes via NFS, and with some artful symlinks, march through
>> your user mailboxes replacing the symlinks with the instantiated
>> local copies and have almost zero downtime.
>>
>> Joseph Tam <[hidden email]>
>
> Another possibility is to use an email client like Claws-Mail, which is
> very fast, create two accounts: One for the old IMAP, one for the new
> one, and just copy trees.

I've never used Claws-Mail - so I don't know how well it does. But I've
done a lot of data migrations between imap servers and accounts - and on
many occasions I had to use Outlook or Thunderbird to do it. It is
extremely inefficient, specially if the user has a large number of
subfolders - as neither software seem to be able to copy subfolders in
block. The other problem I stumbled on is that you can initiate a large
transfer (5000 emails at once), the connection, the client or even the
server chokes at some point for various reasons - and more often than
not it restarts the transfer from the beginning - and you end up with
lots of duplications. Specially if transferring through a non-local,
slower link (ADSL or something similar), I had to end up moving blocks
of a few hundred emails at a time - then rinse and repeat. Not very
happy memories when you are dealing with 30,000 emails mailboxes with
100 subfolders!

To answer the OP, if you have full access to the destination server, and
the two servers are not Maildir or mbox storage (so copying directly is
not an option), another possibility might be to use getmail. I am about
to test this in a few weeks time. Getmail has an option to download
email from all folders of the originating account. I think I would
script it with the following options:

1. Download 100 emails at a time - so that if something goes wrong it
doesn't risk screwing up a large batch (getmail has this option).
2. Download emails from all folders of the account.
3. Delete emails from source after downloading - so that if things go
wrong, or you have to interrupt the process, you know what has already
been transferred.
4. Check for success exit code, and repeat until everything is copied.

I haven't tried the above yet, but I think it should be quite robust and
efficient, and able to run unattended once started.
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

María Arrea
     We migrated 36 TB of mail (65K users) from Sun Messaging to Dovecot
using Imapsync, worked like a charm.

     Regards

     María
Reply | Threaded
Open this post in threaded view
|

Re: Recommended tool for migrating IMAP servers

Webert de Souza Lima
In reply to this post by Sami Ketola
On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola <[hidden email]> wrote:

>
> Can't really do DNS change to test with one user as DNS change affects all
> users.


Yes, of course. For test purposes one could justchange the system's host
file and simulate a DNS change, after imapsync is complete
I meant a simple test just to see what happens at the client size, to see
what the user itself could notice.

On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola <[hidden email]> wrote:

>
> What we usually do is put layer of dovecot proxies in front of the legacy
> and destination platform and then have database
> with information if user has been migrated already and forward the
> connection to legacy or dovecot
> platform based on that info. That enables us to migrate user-by-user.


Cool, we do that too :D. We might have learned that from you even. In that
way users can use our Webmail from day 1 when migration starts.


On Tue, Dec 5, 2017 at 8:54 AM, Sami Ketola <[hidden email]> wrote:

>
> And both users and mail server admins do notice if there is IMAP UID or
> POP3 UIDL changes. Imagine
> what happens when millions of users suddenly start to redownload terabytes
> of data.
>
> Sami
>

That would just kill my cluster. Happened once due to NFS sharing hang.
Terrible day to be alive.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
12