Best way to move mail from one server to another

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Best way to move mail from one server to another

James Brown
I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:

IMAP-Sync
Rsync
Doveadmin backup and
dsync

What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.

Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.

Does anyone have any advice or examples?

Thanks, James.
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Aki Tuomi-2
You probably can do this with rsync, if you do not need to convert mail storage format. Safest way would be to stop both servers, do rsync, start both servers.

If your users have already written to the new server, you will have to use doveadm sync, otherwise data loss will occur.

Aki

> On 04 September 2018 at 10:20 James Brown <[hidden email]> wrote:
>
>
> I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:
>
> IMAP-Sync
> Rsync
> Doveadmin backup and
> dsync
>
> What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.
>
> Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.
>
> Does anyone have any advice or examples?
>
> Thanks, James.
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola
In reply to this post by James Brown


> On 4 Sep 2018, at 10.20, James Brown <[hidden email]> wrote:
>
> I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:
>
> IMAP-Sync
> Rsync
> Doveadmin backup and
> dsync
>
> What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.
>
> Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.
>
> Does anyone have any advice or examples?


It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server.
If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.

imapsync always loses data.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

James Brown
Thanks Sam

Users not using new mail server yet.

Both old and new using maildir.

Sounds like rsync is the way to go.

Thanks for your help.

James.

Sent from my iPhone XI

> On 4 Sep 2018, at 5:41 pm, Sami Ketola <[hidden email]> wrote:
>
>
>
>> On 4 Sep 2018, at 10.20, James Brown <[hidden email]> wrote:
>>
>> I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:
>>
>> IMAP-Sync
>> Rsync
>> Doveadmin backup and
>> dsync
>>
>> What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.
>>
>> Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.
>>
>> Does anyone have any advice or examples?
>
>
> It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server.
> If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.
>
> imapsync always loses data.
>
> Sami
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Reio Remma
If you have lots of data, you can make an initial pass with rsync whilst the old server is running and another rsync pass (with --delete, use dry run to test) when you've shut down the old server to sync the changes since the initial pass.

Good luck!
Reio


On 04/09/2018 11:40, James Brown wrote:
Thanks Sam

Users not using new mail server yet. 

Both old and new using maildir. 

Sounds like rsync is the way to go. 

Thanks for your help. 

James. 

Sent from my iPhone XI

On 4 Sep 2018, at 5:41 pm, Sami Ketola [hidden email] wrote:



On 4 Sep 2018, at 10.20, James Brown [hidden email] wrote:

I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:

IMAP-Sync
Rsync
Doveadmin backup and
dsync

What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.

Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.

Does anyone have any advice or examples?

It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server.
If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.

imapsync always loses data.

Sami




--
Tervitades
Reio Remma


MR
        Stuudio 25 aastat

MR Stuudio OÜ
Tondi 17b, 11316, Tallinn
Tel +372 650 4808
Mob +372 56 22 00 33
[hidden email]
www.mrstuudio.ee



Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Jeff Abrahamson

How do you test the new server?

(Sorry, this seems like it must be obvious.  The server I set up is the first one handling this domain, so testing was easy in that it didn't have history.)

Jeff Abrahamson
+33 6 24 40 01 57
+44 7920 594 255

http://p27.eu/jeff/
http://transport-nantes.com/

On 04/09/18 10:49, Reio Remma wrote:
If you have lots of data, you can make an initial pass with rsync whilst the old server is running and another rsync pass (with --delete, use dry run to test) when you've shut down the old server to sync the changes since the initial pass.

Good luck!
Reio


On 04/09/2018 11:40, James Brown wrote:
Thanks Sam

Users not using new mail server yet. 

Both old and new using maildir. 

Sounds like rsync is the way to go. 

Thanks for your help. 

James. 

Sent from my iPhone XI

On 4 Sep 2018, at 5:41 pm, Sami Ketola [hidden email] wrote:



On 4 Sep 2018, at 10.20, James Brown [hidden email] wrote:

I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:

IMAP-Sync
Rsync
Doveadmin backup and
dsync

What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.

Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.

Does anyone have any advice or examples?
It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server.
If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.

imapsync always loses data.

Sami



--
Tervitades
Reio Remma


MR Stuudio 25
          aastat

MR Stuudio OÜ
Tondi 17b, 11316, Tallinn
Tel +372 650 4808
Mob +372 56 22 00 33
[hidden email]
www.mrstuudio.ee




-- 

Jeff Abrahamson
+33 6 24 40 01 57
+44 7920 594 255

http://p27.eu/jeff/
http://transport-nantes.com/
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Robert Schetterer-2
In reply to this post by Sami Ketola
Am 04.09.2018 um 09:41 schrieb Sami Ketola:
> imapsync always loses data

never saw this, be carefull by anounce such myths


Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola


> On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:
>
> Am 04.09.2018 um 09:41 schrieb Sami Ketola:
>> imapsync always loses data
>
> never saw this, be carefull by anounce such myths


It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.

Sami
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Robert Schetterer-2
Am 04.09.2018 um 16:52 schrieb Sami Ketola:

>
>
>> On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:
>>
>> Am 04.09.2018 um 09:41 schrieb Sami Ketola:
>>> imapsync always loses data
>>
>> never saw this, be carefull by anounce such myths
>
>
> It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.

but using "looses data" might others think it also may fail with imap,
so be more detailed next time
>
> Sami
>



Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola


> On 4 Sep 2018, at 18.00, Robert Schetterer <[hidden email]> wrote:
>
> Am 04.09.2018 um 16:52 schrieb Sami Ketola:
>>
>>
>>> On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:
>>>
>>> Am 04.09.2018 um 09:41 schrieb Sami Ketola:
>>>> imapsync always loses data
>>>
>>> never saw this, be carefull by anounce such myths
>>
>>
>> It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.
>
> but using "looses data" might others think it also may fail with imap,
> so be more detailed next time


UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

Sami
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Robert Schetterer-2
Am 04.09.2018 um 17:18 schrieb Sami Ketola:

>
>
>> On 4 Sep 2018, at 18.00, Robert Schetterer <[hidden email]> wrote:
>>
>> Am 04.09.2018 um 16:52 schrieb Sami Ketola:
>>>
>>>
>>>> On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:
>>>>
>>>> Am 04.09.2018 um 09:41 schrieb Sami Ketola:
>>>>> imapsync always loses data
>>>>
>>>> never saw this, be carefull by anounce such myths
>>>
>>>
>>> It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.
>>
>> but using "looses data" might others think it also may fail with imap,
>> so be more detailed next time
>
>
> UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".
>
> Sami
>

Sorry i migrated terrabytes of mail with imapsync and never had a
problem, it works as designed, also with maildir rsync did a good job,
what never worked as it should was dsync ,cause of bugs ,that may
changed now

so this is my answer to topic

 "Best way to move mail from one server to another"

after all there is no "best way", there are a few ways
choose the one which fits best to your needs goals and starting points (
kind of servers etc )

Best Regards
MfG Robert Schetterer

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Rick Romero-2

Quoting Robert Schetterer <[hidden email]>:

Am 04.09.2018 um 17:18 schrieb Sami Ketola:

On 4 Sep 2018, at 18.00, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

imapsync always loses data

never saw this, be carefull by anounce such myths
It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.
but using "looses data" might others think it also may fail with imap,
so be more detailed next time
UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

Sami
Sorry i migrated terrabytes of mail with imapsync and never had a
problem, it works as designed, also with maildir rsync did a good job,
what never worked as it should was dsync ,cause of bugs ,that may
changed now

so this is my answer to topic

"Best way to move mail from one server to another"

after all there is no "best way", there are a few ways
choose the one which fits best to your needs goals and starting points (
kind of servers etc )


Seems like there's an easy solution to the UID problem

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Rick Romero-2
In reply to this post by Robert Schetterer-2

Quoting Robert Schetterer <[hidden email]>:

Am 04.09.2018 um 17:18 schrieb Sami Ketola:

On 4 Sep 2018, at 18.00, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

imapsync always loses data

never saw this, be carefull by anounce such myths
It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.
but using "looses data" might others think it also may fail with imap,
so be more detailed next time
UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

Sami
Sorry i migrated terrabytes of mail with imapsync and never had a
problem, it works as designed, also with maildir rsync did a good job,
what never worked as it should was dsync ,cause of bugs ,that may
changed now

so this is my answer to topic

"Best way to move mail from one server to another"
 

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola


On 4 Sep 2018, at 18.45, Rick Romero <[hidden email]> wrote:

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.

No it does not. This is different thing. This is about managing duplicates on multiple syncs.
Imapsync seems to keep track of UID -> MAIL mapping for server A and UID -> MAIL mapping for server B so that on multiple runs it does not sync the same messages again. However it does not use the same UID -> MAIL for server B as is originally on server A. And that is the problem.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Michael Slusarz-2
In reply to this post by Rick Romero-2
Sami is correct.  imapsync loses data.

Source -> imapsync -> destination : the mailboxes are different to an IMAP client
Source -> dsync -> destination (running Dovecot w/doveadm): the mailboxes are the same

You may not care that your users w/100,000 message mailboxes that have been cached on their client now have to invalidate that entire cache (and most likely re-download all of those messages).

So there is zero argument that, due to IMAP metadata loss, this is something that doesn't happen if dsync (or rsync) is used.

michael

On September 4, 2018 at 9:45 AM Rick Romero <[hidden email]> wrote:

Quoting Robert Schetterer <[hidden email]>:

Am 04.09.2018 um 17:18 schrieb Sami Ketola:

On 4 Sep 2018, at 18.00, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

On 4 Sep 2018, at 17.47, Robert Schetterer <[hidden email]> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

imapsync always loses data

never saw this, be carefull by anounce such myths
It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.
but using "looses data" might others think it also may fail with imap,
so be more detailed next time
UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

Sami
Sorry i migrated terrabytes of mail with imapsync and never had a
problem, it works as designed, also with maildir rsync did a good job,
what never worked as it should was dsync ,cause of bugs ,that may
changed now

so this is my answer to topic

"Best way to move mail from one server to another"
 

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.


 
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola
In reply to this post by Robert Schetterer-2


> On 4 Sep 2018, at 18.38, Robert Schetterer <[hidden email]> wrote:
>
> Sorry i migrated terrabytes of mail with imapsync and never had a
> problem, it works as designed, also with maildir rsync did a good job,
> what never worked as it should was dsync ,cause of bugs ,that may
> changed now
>

I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers.
Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.

Sami


Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Rick Romero-2
In reply to this post by Sami Ketola

Quoting Sami Ketola <[hidden email]>:



On 4 Sep 2018, at 18.45, Rick Romero <[hidden email]> wrote:

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.
 

No it does not. This is different thing. This is about managing duplicates on multiple syncs.
Imapsync seems to keep track of UID -> MAIL mapping for server A and UID -> MAIL mapping for server B so that on multiple runs it does not sync the same messages again. However it does not use the same UID -> MAIL for server B as is originally on server A. And that is the problem.

That's not what you said, you said there was a client sync issue.  That's definitely an issue (especially if you run Mac Mail).

Ignoring duplicates on multiples runs is a non-issue, even with the 2011 version.  Recording UIDs is not necessary for that.

Rick

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola


On 4 Sep 2018, at 19.32, Rick Romero <[hidden email]> wrote:
No it does not. This is different thing. This is about managing duplicates on multiple syncs.
Imapsync seems to keep track of UID -> MAIL mapping for server A and UID -> MAIL mapping for server B so that on multiple runs it does not sync the same messages again. However it does not use the same UID -> MAIL for server B as is originally on server A. And that is the problem.

That's not what you said, you said there was a client sync issue.  That's definitely an issue (especially if you run Mac Mail).

Ignoring duplicates on multiples runs is a non-issue, even with the 2011 version.  Recording UIDs is not necessary for that.


Nope. I said that with imapsync there is no way to preserve the same UID for a mail on the destination. Data is lost.

If the UID -> MAIL pairs are not kept then client caches need to be invalidated and redownloaded.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Ruben Safir
In reply to this post by Sami Ketola
On 9/4/18 12:27 PM, Sami Ketola wrote:

>
>
>> On 4 Sep 2018, at 18.38, Robert Schetterer <[hidden email]> wrote:
>>
>> Sorry i migrated terrabytes of mail with imapsync and never had a
>> problem, it works as designed, also with maildir rsync did a good job,
>> what never worked as it should was dsync ,cause of bugs ,that may
>> changed now
>>
>
> I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers.
> Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.
>
> Sami
>
>


why does not rsync work?

Ruben

--
So many immigrant groups have swept through our town
that Brooklyn, like Atlantis, reaches mythological
proportions in the mind of the world - RI Safir 1998
http://www.mrbrklyn.com
DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002

http://www.nylxs.com - Leadership Development in Free Software
http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps,
but incompatible with living as a free human being. -RI Safir 2013
Reply | Threaded
Open this post in threaded view
|

Re: Best way to move mail from one server to another

Sami Ketola


> On 4 Sep 2018, at 21.24, Ruben Safir <[hidden email]> wrote:
>
> On 9/4/18 12:27 PM, Sami Ketola wrote:
>>
>>
>>> On 4 Sep 2018, at 18.38, Robert Schetterer <[hidden email]> wrote:
>>>
>>> Sorry i migrated terrabytes of mail with imapsync and never had a
>>> problem, it works as designed, also with maildir rsync did a good job,
>>> what never worked as it should was dsync ,cause of bugs ,that may
>>> changed now
>>>
>>
>> I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers.
>> Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.
>>
>> Sami
>>
>>
>
>
> why does not rsync work?

Nobody said it does not work. But it can be used only if the mail storage format does not change.

Sami
12