Dovecot UIDs and POP.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Dovecot UIDs and POP.

Dovecot mailing list
Hi,

I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another.

This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine.

So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.

Here's what I understand. The list of dovecot files in an account looks like this:
-rw-r-----   1 admin mail    0 Jul 11  2018 dovecot-acl-list
-rw-r-----   1 admin mail  14K Sep 20 08:36 dovecot.index
-rw-r-----   1 admin mail 216K Sep 20 17:24 dovecot.index.cache
-rw-r-----   1 admin mail  29K Sep 20 17:24 dovecot.index.log
-rw-r-----   1 admin mail  388 Dec 17  2017 dovecot.list.index
-rw-r-----   1 admin mail 1.6K Jul 11  2018 dovecot.list.index.log
-rw-r-----   1 admin mail   24 Jul 11  2018 dovecot.mailbox.log
-rw-r-----   1 admin mail   53 Sep 12 09:42 dovecot-quota
-rw-r-----   1 admin mail  52K Sep 20 17:24 dovecot-uidlist
-rw-r-----   1 admin mail    8 Jul 11  2018 dovecot-uidvalidity
-rw-r-----   1 admin mail    0 Jul  4  2016 dovecot-uidvalidity.5779bbeb

It seems that the dovecot-uidlist is an index of all the mail files in the account. As far as I can gather, this should be the same on both servers after I've rsynced them over ssh: The files and the list of them in dovecot-uidlist should correspond.
So what is it that the POP client looks at and decides to re-download all the emails?
And how could I have stopped it from doing that, and get it to realize that the list of emails it holds internally is the same as the list on the server? What would I need to change in the dovecot files?

Thanks for any insight into the process.

As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues?

P.
Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
On Sep 20, 2019, at 10:38 PM, Plutocrat <[hidden email]> wrote:
> I recently performed a mail server migration for a client under fairly serious time constraints (ahem, yes one of those jobs). I would normally use imapsync to get all the mail copied to the new server, but under the circumstances, I had to write a script to rsync each of the mailboxes over ssh, from one server to another.
>
> This all worked fine, except for the fact that if a client was using a POP email client, when they connected to the new server, they re-downloaded all their mail, creating duplicates of everything in their Outlook client. Of course they weren't happy about this. IMAP email clients were OK. No duplicates, and everything was fine.

Dealing with weird POP3 things h is why I disabled it on my server more than a decade ago; users have to beg for POP access and promise they will ONLY use it to get their mail into gmail.

> So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.

Did you check <https://wiki.dovecot.org/Migration>? It has a lot of info on this.

Seems dsync would hav been the best way to do this?

<https://wiki.dovecot.org/Migration/Dsync>

> As a secondary question -- and perhaps I should put this in a separate message -- I did notice that "doveadm sync" would apparently have helped me with this, but I wasn't able to get that to work either. I believe it was something to do with the fact that all the mailboxes were under the same linux user account on the target server, and I couldn't figure out all the paths and permissions in time. Would 'doveadm sync' have fixed all the UID and duplicate POP email issues?

Don’t know, but dsync says it does this:

"The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.”

Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync?


--
@mdhughes: One of the few regrets I have about lawn-less apartments:
Shallow graves are so much harder to come by.

Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
On 22/09/2019 4:08 AM, @lbutlr via dovecot wrote:
>> So while I was migrating the mail, I did try for a while to understand the format of the UID files, but failed to do so in the available time, so the client just had to deal with duplicate emails. But now the smoke has cleared, I'd like to understand the problem a little better, and I was hoping someone on this forum could explain it to me, and the changes I'd need to make to the files so that the POP client DIDN'T download the duplicate emails.
>
> Did you check <https://wiki.dovecot.org/Migration>? It has a lot of info on this.

I did see that page, but was unable to figure out which of the three UID related files I needed to keep, and which to edit in order to 'trick' the email client into not re-downloading the mails.

The migration was from dovecot 2.x to 2.x, so I don't think there was any translation issue between servers. The POP clients were Outlook (my old nemesis). The only information I could find is that the dovecot UIDL format is
pop3_uidl_format = %v.%u
But that didn't really take me anywhere. So its all a bit of a mystery. I was hoping there would be a UID guru on this list who could walk me through it, or point me to some reference.

> Don’t know, but dsync says it does this:
> "The pop3-migration plugin is used to preserve POP3 UIDLs. When dsync is handling IMAP INBOX and requests a POP3 UIDL, the plugin connects to the POP3 server and figures out which IMAP messages match which POP3 messages and then returns the appropriate POP3 UIDL.”
> Trouble is, if you are migrating POP and the server is not up, I am not sure what you can do with dsync?

I did try to use dsync (aka doveadm sync) for a few hours, but couldn't get a test account to migrate across. It seems like this would have been the solution but I was unable to confirm. At times it seemed to login and complete without error, but I could never find where it had put the mail! Other times it refused to login or gave permission errors. I looked around for real life examples, but wasn't able to find any.

Had I got it to work, I could maybe have figured out the UID problem. Maybe that will be a project for me in the future, when I have a spare few hours!

Maybe if I gave a few details of the directory structure that would help. I was migrating from a Cpanel installation to a standalone mail server.
In Cpanel, there was the admin account login, and under that
~/mail/domain1.com/mailbox1
~/mail/domain1.com/mailbox2
~/mail/domain2.com/mailbox1
~/mail/domain2.com/mailbox2

On the target server the same structure existed. One login, admin, and under that account
~/mail/domain1.com/mailbox1
~/mail/domain1.com/mailbox2
~/mail/domain2.com/mailbox1
~/mail/domain2.com/mailbox2
All files and directories were chown admin:mail

The command I used was
doveadm sync -u [hidden email] ssh -i id_rsa -o "StrictHostKeyChecking=no" admin@100.110.120.130

I tried a few options to tell it where to put the mail, but dovecot on the target server didn't seem to know where each account was located.

P.
Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list


On 23 Sep 2019, at 8.20, Plutocrat via dovecot <[hidden email]> wrote:

The command I used was
doveadm sync -u [hidden email] ssh -i id_rsa -o "StrictHostKeyChecking=no" [hidden email]

I tried a few options to tell it where to put the mail, but dovecot on the target server didn't seem to know where each account was located.


doveadm backup -u [hidden email] -R ssh [hidden email] "sudo /usr/bin/doveadm dsync-server -u [hidden email]"

run this on the target system to which you are migrating to.

HTH

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
In reply to this post by Dovecot mailing list
On Sep 22, 2019, at 11:20 PM, Plutocrat <[hidden email]> wrote:
> doveadm sync -u

Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.



--
"Back off, man. I'm a scientist.”

Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list


> On 24 Sep 2019, at 17.14, @lbutlr via dovecot <[hidden email]> wrote:
>
> On Sep 22, 2019, at 11:20 PM, Plutocrat <[hidden email]> wrote:
>> doveadm sync -u
>
> Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.

Yes. Target machine should be configured and tested before starting any migration. That is the thing with *all* migrations.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
In reply to this post by Dovecot mailing list
On 23/09/2019 5:01 PM, Sami Ketola wrote:
> doveadm backup -u [hidden email] <mailto:[hidden email]> -R ssh [hidden email] <mailto:[hidden email]> "sudo /usr/bin/doveadm dsync-server -u [hidden email] <mailto:[hidden email]>"
> run this on the target system to which you are migrating to.

Seems counter-intuitive to run the command on the target system! Pull, rather than push. However that wasn't an option for me at the time. I could only make a connection outbound from the source server, as ssh was locked down, and I was forced to use Cpanel's 'Terminal' App.

So maybe that was the problem them. I could only push and not pull.

P.


Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
In reply to this post by Dovecot mailing list
On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
> Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.

Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.

I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.

P.
Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list


> On 25 Sep 2019, at 4.52, Plutocrat via dovecot <[hidden email]> wrote:
>
> On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
>> Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
>
> Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
>
> I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.

If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.

imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list


On 25.09.19 12:29, Sami Ketola via dovecot wrote:

>
>
>> On 25 Sep 2019, at 4.52, Plutocrat via dovecot <[hidden email]> wrote:
>>
>> On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
>>> Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
>>
>> Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
>>
>> I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
>
> If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.
>
> imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.
>
> Sami
>
Very interesting thread, since I was about to perform the migration in
the next few days. I was operating under the assumption that imapsync
would retain both imap uids and pop3 uidls. According to Sami's post
this is not the case. This begs the question which path to take. The
source server is dovecot, albeit a very old version with no dsync (we're
talking dovecot 1.x.x).

Any ideas?

Thanks
Ionel
Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list


> On 25 Sep 2019, at 15.14, Ionel via dovecot <[hidden email]> wrote:
>
>
>
> On 25.09.19 12:29, Sami Ketola via dovecot wrote:
>>
>>
>>> On 25 Sep 2019, at 4.52, Plutocrat via dovecot <[hidden email]> wrote:
>>>
>>> On 24/09/2019 10:14 PM, @lbutlr via dovecot wrote:
>>>> Did the target machine already have the user setup? I think dsync wants to sync mailboxes between configured and working servers with users already defined.
>>>
>>> Yes, of course. And the directory structure was pretty much identical between the two. Which was how I was able to write and run the rsync script.
>>>
>>> I think at this point, I'm past the dsync problems. That window has closed, and the migration is done, albeit with this irritating re-downloading of email with POP problem. However I'm still keen to get to the bottom of how this whole UID/UIDL/POP3 transaction takes place, so I can understand it for the future, and prevent it happening again. I'm guessing it would be a useful addition to the dovecot documentation as well.
>>
>> If you want to keep the data complete then the only proper way to do the migration is dsync. Specially since you now were migrating dovecot to dovecot.
>>
>> imapsync does not retain imap uids or pop3 uidls. all users would need to clear their local caches.
>>
>> Sami
>>
> Very interesting thread, since I was about to perform the migration in
> the next few days. I was operating under the assumption that imapsync
> would retain both imap uids and pop3 uidls. According to Sami's post
> this is not the case. This begs the question which path to take. The
> source server is dovecot, albeit a very old version with no dsync (we're
> talking dovecot 1.x.x).
>
> Any ideas?
>


Native doveadm protocol between two major versions might not be compatible so I would not use that.
What I would do in your case is to use imapc connector to pull the emails with dovecot dsync and pop3_migration -plugin
to fetch the POP3 UIDLs from the old server over pop3 connection.

https://wiki2.dovecot.org/Migration/Dsync

Sami


Reply | Threaded
Open this post in threaded view
|

Re: Dovecot UIDs and POP.

Dovecot mailing list
On Sep 25, 2019, at 07:16, Sami Ketola <[hidden email]> wrote:
>
> What I would do in your case is to use imapc connector to pull the emails with dovecot dsync and pop3_migration -plugin to fetch the POP3 UIDLs from the old server over pop3 connection.

Also, consider not using POP3 at all. It was designed for an entirely different world than the one we live in.

--
My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.