dsync and altpath on shared storage.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

dsync and altpath on shared storage.

Dovecot mailing list
Hi,

I was wondering...

If one had mdbox ALT path set to a shared storage mount (say, on NFS)
and one wanted to move a mailbox to a different host... I guess it in
principle wouldn't be necessary to copy all the ALT storage through
dsync, when the volume could just be mounted on the new host.

Is there anyway for dsync to avoid moving Gigabytes of data for could
just be "moved" by moving the mount?

/Peter
Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list


> On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot <[hidden email]> wrote:
>
> Hi,
>
> I was wondering...
>
> If one had mdbox ALT path set to a shared storage mount (say, on NFS)
> and one wanted to move a mailbox to a different host... I guess it in
> principle wouldn't be necessary to copy all the ALT storage through
> dsync, when the volume could just be mounted on the new host.
>
> Is there anyway for dsync to avoid moving Gigabytes of data for could
> just be "moved" by moving the mount?


Not tested but you can probably do something like this in the target server:

doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"

just leave ALT storage path from the setting.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list


On 9/2/19 3:03 PM, Sami Ketola wrote:
>> On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot <[hidden email]> wrote:
...
>> Is there anyway for dsync to avoid moving Gigabytes of data for could
>> just be "moved" by moving the mount?
>
>
> Not tested but you can probably do something like this in the target server:
>
> doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
>
> just leave ALT storage path from the setting.


I'll have to test this... but my initial guess would be that doveadm
would then think the mails has disappeared. Would it then copy the index
metadata for those mails to the target host anyway?

/Peter
Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list


> On 3 Sep 2019, at 15.34, Peter Mogensen via dovecot <[hidden email]> wrote:
>
>
>
> On 9/2/19 3:03 PM, Sami Ketola wrote:
>>> On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot <[hidden email]> wrote:
> ...
>>> Is there anyway for dsync to avoid moving Gigabytes of data for could
>>> just be "moved" by moving the mount?
>>
>>
>> Not tested but you can probably do something like this in the target server:
>>
>> doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
>>
>> just leave ALT storage path from the setting.
>
>
> I'll have to test this... but my initial guess would be that doveadm
> would then think the mails has disappeared. Would it then copy the index
> metadata for those mails to the target host anyway?


Hmm. That is true. It will probably not work after all then.

Now I'm out of ideas how to do this efficiently.

Sami

Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list


On 9/3/19 2:38 PM, Sami Ketola wrote:

>
>
>> On 3 Sep 2019, at 15.34, Peter Mogensen via dovecot <[hidden email]> wrote:
>>
>>
>>
>> On 9/2/19 3:03 PM, Sami Ketola wrote:
>>>> On 2 Sep 2019, at 15.25, Peter Mogensen via dovecot <[hidden email]> wrote:
>> ...
>>>> Is there anyway for dsync to avoid moving Gigabytes of data for could
>>>> just be "moved" by moving the mount?
>>>
>>>
>>> Not tested but you can probably do something like this in the target server:
>>>
>>> doveadm backup -u victim -R ssh sudouser@old-server "sudo doveadm dsync-server -o mail_location=sdbox:/location-to-your-sdbox/ -u victim"
>>>
>>> just leave ALT storage path from the setting.
>>
>>
>> I'll have to test this... but my initial guess would be that doveadm
>> would then think the mails has disappeared. Would it then copy the index
>> metadata for those mails to the target host anyway?
>
>
> Hmm. That is true. It will probably not work after all then.
>
> Now I'm out of ideas how to do this efficiently.

I assume it won't even work to just premount the shared storage
read-only on the target side, so the mails are already there.
... since I suppose the receiving dsync reserves the right to re-pack
the m.* storage files?

/Peter

Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list
In reply to this post by Dovecot mailing list

So... I've done some testing.

One method which seemed to work - at least for primitive cases - was to:

* Mount the ALT storage on the destination.
* Run "doveadm force-resync \*" on the destination.
  (putting all the mails in ALT storage into the dovecot.map.index)
* Run dsync from source to destination.

Of course... if there was some way to avoid step 2...

/Peter
Reply | Threaded
Open this post in threaded view
|

Re: dsync and altpath on shared storage.

Dovecot mailing list


On 9/4/19 2:12 PM, Peter Mogensen wrote:

>
> So... I've done some testing.
>
> One method which seemed to work - at least for primitive cases - was to:
>
> * Mount the ALT storage on the destination.
> * Run "doveadm force-resync \*" on the destination.
>   (putting all the mails in ALT storage into the dovecot.map.index)
> * Run dsync from source to destination.
>
> Of course... if there was some way to avoid step 2...

So ... I have an idea.

Assuming users mail_location is:

mdbox:~/mdbox:ALT=/alt:INDEX=~/idx

And /alt is a shard mounted storage.

Then, it suspect the following steps would make dsync avoid transfering
mails on shared storage:

1) Create a rudimentary mdbox on the target side (just containing the
dbox-alt-root link)

2) Mount /alt on the target host

3) Copy all dovecot.index and dovecot.map.index in ~/idx from source to
target. That is: not the transaction (*.log) files or cache files.
I suppose this needs to be done under appropriate read locking.

4) doveadm sync -u source doveadm dsync-server -u target
  ... to get the rest of the mails in primary storage and all updates
sine the index files where snapshot.



It would be nice if there was a way to force dovecot*index.log files to
be snapshot to index files.

If the aim is not to sync two different accounts but to simply move one
account from one host to a new host where it doesn't exist in advance,
are there any caveats with this?

... apart from a few missing tools.

/Peter