dovecot-lda without starting dovecot?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
Hello,

we have a setup where SMTP/LDA and POP3/IMAP are on different physical hosts.
They share the mail data via an external storage.
Now we would like to use dovecot-lda on the smtp host, so we wonder if the
lda binary works without starting dovecot from init. As there will be no
POP3/IMAP usage on this host it seems unnecessary. Nevertheless we cannot
judge if it is still needed for lda&sieve to work.
Your opinion?

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Aki Tuomi-2

> On November 3, 2017 at 1:50 PM Stephan von Krawczynski <[hidden email]> wrote:
>
>
> Hello,
>
> we have a setup where SMTP/LDA and POP3/IMAP are on different physical hosts.
> They share the mail data via an external storage.
> Now we would like to use dovecot-lda on the smtp host, so we wonder if the
> lda binary works without starting dovecot from init. As there will be no
> POP3/IMAP usage on this host it seems unnecessary. Nevertheless we cannot
> judge if it is still needed for lda&sieve to work.
> Your opinion?
>
> --
> Regards,
> Stephan

dovecot-lda does not work without dovecot unless you have physical users and you run the binary as target user. with virtual users it's virtually impossible to achieve.

Aki
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
On Fri, 3 Nov 2017 17:53:47 +0200 (EET)
Aki Tuomi <[hidden email]> wrote:

> > On November 3, 2017 at 1:50 PM Stephan von Krawczynski
> > <[hidden email]> wrote:
> >
> >
> > Hello,
> >
> > we have a setup where SMTP/LDA and POP3/IMAP are on different physical
> > hosts. They share the mail data via an external storage.
> > Now we would like to use dovecot-lda on the smtp host, so we wonder if the
> > lda binary works without starting dovecot from init. As there will be no
> > POP3/IMAP usage on this host it seems unnecessary. Nevertheless we cannot
> > judge if it is still needed for lda&sieve to work.
> > Your opinion?
> >
> > --
> > Regards,
> > Stephan  
>
> dovecot-lda does not work without dovecot unless you have physical users and
> you run the binary as target user. with virtual users it's virtually
> impossible to achieve.
>
> Aki

Hello Aki,

let me explain this a bit more. We do not intend to use only some copied
binary. Of course we would do a full installation of dovecot and pidgeonhole,
only we question if it is necessary to start the dovecot init-file bringing up
the dovecot imap/imap-login/pop/pop-login/auth and other processes.
Indeed we have virtual users.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Sami Ketola

> On 3 Nov 2017, at 18.23, Stephan von Krawczynski <[hidden email]> wrote:
> Hello Aki,
>
> let me explain this a bit more. We do not intend to use only some copied
> binary. Of course we would do a full installation of dovecot and pidgeonhole,
> only we question if it is necessary to start the dovecot init-file bringing up
> the dovecot imap/imap-login/pop/pop-login/auth and other processes.
> Indeed we have virtual users.


While it might be possible to disable all the other services except master I must ask why?
How would the users be accessing their mails then?

Sami
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
On Fri, 3 Nov 2017 19:30:22 +0200
Sami Ketola <[hidden email]> wrote:

> > On 3 Nov 2017, at 18.23, Stephan von Krawczynski <[hidden email]>
> > wrote: Hello Aki,
> >
> > let me explain this a bit more. We do not intend to use only some copied
> > binary. Of course we would do a full installation of dovecot and
> > pidgeonhole, only we question if it is necessary to start the dovecot
> > init-file bringing up the dovecot imap/imap-login/pop/pop-login/auth and
> > other processes. Indeed we have virtual users.  
>
>
> While it might be possible to disable all the other services except master I
> must ask why? How would the users be accessing their mails then?
>
> Sami

Hello Sami,

you did not read my first post. We are talking about a multihost installation
where one host does SMTP and LDA, and the other does POP and IMAP (with
dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP
host_. So there is no need for open POP or IMAP ports and the corresponding
running services.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Aki Tuomi-2
In reply to this post by Stephan von Krawczynski-2
you could try setting
protocols =
in config file to disable (most) listeners.
---Aki TuomiDovecot oy
-------- Original message --------From: Stephan von Krawczynski <[hidden email]> Date: 03/11/2017  19:39  (GMT+02:00) To: Dovecot Mailing List <[hidden email]> Subject: Re: dovecot-lda without starting dovecot?
On Fri, 3 Nov 2017 19:30:22 +0200
Sami Ketola <[hidden email]> wrote:

> > On 3 Nov 2017, at 18.23, Stephan von Krawczynski <[hidden email]>
> > wrote: Hello Aki,
> >
> > let me explain this a bit more. We do not intend to use only some copied
> > binary. Of course we would do a full installation of dovecot and
> > pidgeonhole, only we question if it is necessary to start the dovecot
> > init-file bringing up the dovecot imap/imap-login/pop/pop-login/auth and
> > other processes. Indeed we have virtual users. 
>
>
> While it might be possible to disable all the other services except master I
> must ask why? How would the users be accessing their mails then?
>
> Sami

Hello Sami,

you did not read my first post. We are talking about a multihost installation
where one host does SMTP and LDA, and the other does POP and IMAP (with
dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP
host_. So there is no need for open POP or IMAP ports and the corresponding
running services.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Sami Ketola
>>
>> While it might be possible to disable all the other services except master I
>> must ask why? How would the users be accessing their mails then?
>>
>> Sami
>
> Hello Sami,
>
> you did not read my first post. We are talking about a multihost installation
> where one host does SMTP and LDA, and the other does POP and IMAP (with
> dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP
> host_. So there is no need for open POP or IMAP ports and the corresponding
> running services.

Again that does not answer my question why? Why do you want all the locking problems
and multi-access problems that come with setup like that? What is the actual problem that
you are trying to solve?

Sami
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan Herker
On 11/3/2017 4:57 PM, Sami Ketola wrote:

>>> While it might be possible to disable all the other services except master I
>>> must ask why? How would the users be accessing their mails then?
>>>
>>> Sami
>> Hello Sami,
>>
>> you did not read my first post. We are talking about a multihost installation
>> where one host does SMTP and LDA, and the other does POP and IMAP (with
>> dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP
>> host_. So there is no need for open POP or IMAP ports and the corresponding
>> running services.
> Again that does not answer my question why? Why do you want all the locking problems
> and multi-access problems that come with setup like that? What is the actual problem that
> you are trying to solve?
>
> Sami
>
I think you have to run dovecot on the server receiving mails, just
don't open imap/pop3 ports so users can't get there.  Then publish the
other server as your imap/pop3 server.  Both servers probably should
have the same setup for postfix and dovecot, but just kill the ports on
the one that doesn't need it.  To run lda you're going to have to run
dovecot on your mail relay.
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
In reply to this post by Sami Ketola
On Sat, 4 Nov 2017 01:57:31 +0200
Sami Ketola <[hidden email]> wrote:

> >>
> >> While it might be possible to disable all the other services except
> >> master I must ask why? How would the users be accessing their mails then?
> >>
> >> Sami  
> >
> > Hello Sami,
> >
> > you did not read my first post. We are talking about a multihost
> > installation where one host does SMTP and LDA, and the other does POP and
> > IMAP (with dovecot). Now we want to use dovecot-lda for the local delivery
> > _on the SMTP host_. So there is no need for open POP or IMAP ports and the
> > corresponding running services.  
>
> Again that does not answer my question why? Why do you want all the locking
> problems and multi-access problems that come with setup like that? What is
> the actual problem that you are trying to solve?
>
> Sami

Really, I can hardly believe you don't now large loadbalancing ISP setups with
multiple nodes per single service ...?
The simple problem: massive numbers of emails

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Sami Ketola

> On 4 Nov 2017, at 10.31, Stephan von Krawczynski <[hidden email]> wrote:
>
> On Sat, 4 Nov 2017 01:57:31 +0200
> Sami Ketola <[hidden email]> wrote:
>> Again that does not answer my question why? Why do you want all the locking
>> problems and multi-access problems that come with setup like that? What is
>> the actual problem that you are trying to solve?
>>
>> Sami
>
> Really, I can hardly believe you don't now large loadbalancing ISP setups with
> multiple nodes per single service ...?
> The simple problem: massive numbers of emails

Nope. Has never been done. Has never been recommended way. You will get more problems
with that setup that you are seeking to solve.

Use multiple dovecot backends with director ring in front and switch to lmtp delivery via the director
ring if you have scalability problems. Then you can just increase number of backends in case
they are overloaded.

Sami
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
On Sun, 5 Nov 2017 10:44:25 +0200
Sami Ketola <[hidden email]> wrote:

> > On 4 Nov 2017, at 10.31, Stephan von Krawczynski <[hidden email]>
> > wrote:
> >
> > On Sat, 4 Nov 2017 01:57:31 +0200
> > Sami Ketola <[hidden email]> wrote:  
> >> Again that does not answer my question why? Why do you want all the
> >> locking problems and multi-access problems that come with setup like
> >> that? What is the actual problem that you are trying to solve?
> >>
> >> Sami  
> >
> > Really, I can hardly believe you don't now large loadbalancing ISP setups
> > with multiple nodes per single service ...?
> > The simple problem: massive numbers of emails  
>
> Nope. Has never been done. Has never been recommended way. You will get more
> problems with that setup that you are seeking to solve.
>
> Use multiple dovecot backends with director ring in front and switch to lmtp
> delivery via the director ring if you have scalability problems. Then you
> can just increase number of backends in case they are overloaded.
>
> Sami

Sorry to say this setup works flawlessly for years. The only addition we
will make now is to do the delivery with dovecot-lda. Everything else
(including multiple dovecot pop/imap servers) will stay as is.
Hopefully dovecot-lda does not fiddle around with the indexes too much, as we
then would have to delete this part of the code out. It is not needed as we
found out during the last 10 years of delivering mails into the maildirs by
atomic rename action while dovecot is presenting them over imap.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Sami Ketola

> On 5 Nov 2017, at 12.55, Stephan von Krawczynski <[hidden email]> wrote:
> Sorry to say this setup works flawlessly for years. The only addition we
> will make now is to do the delivery with dovecot-lda. Everything else
> (including multiple dovecot pop/imap servers) will stay as is.
> Hopefully dovecot-lda does not fiddle around with the indexes too much, as we
> then would have to delete this part of the code out. It is not needed as we
> found out during the last 10 years of delivering mails into the maildirs by
> atomic rename action while dovecot is presenting them over imap.


Feel free to do anything you like. I'm just going to mention to people later reading these
from the achives not to take this kind of strange hack as an example of recommended
dovecot clustering. Instead consider it as an opposite of any best practices cluster setup.

Sami
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
On Mon, 6 Nov 2017 08:37:11 +0200
Sami Ketola <[hidden email]> wrote:

> > On 5 Nov 2017, at 12.55, Stephan von Krawczynski <[hidden email]>
> > wrote: Sorry to say this setup works flawlessly for years. The only
> > addition we will make now is to do the delivery with dovecot-lda.
> > Everything else (including multiple dovecot pop/imap servers) will stay as
> > is. Hopefully dovecot-lda does not fiddle around with the indexes too
> > much, as we then would have to delete this part of the code out. It is not
> > needed as we found out during the last 10 years of delivering mails into
> > the maildirs by atomic rename action while dovecot is presenting them over
> > imap.  
>
>
> Feel free to do anything you like. I'm just going to mention to people later
> reading these from the achives not to take this kind of strange hack as an
> example of recommended dovecot clustering. Instead consider it as an
> opposite of any best practices cluster setup.
>
> Sami

Maybe they are interested that it runs like charm ;-)
Of course it is overly complex to let dovecot-lda look up the mail_location
since we already know that when starting it, but at least it shows a nice
output in the logs.
Still we are not content with it touching/locking dovecot.index.log. If
someone pointed at one location in the code where this could be disabled we
would implement a new param for switching that off.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Tanstaafl
On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <[hidden email]>
wrote:
> Still we are not content with it touching/locking dovecot.index.log. If
> someone pointed at one location in the code where this could be disabled we
> would implement a new param for switching that off.

?????????????????????????

Dovecot's indexing is one of its main features, and WHY it is so much
faster than others.

And you want to just turn it off? Good luck...
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
On Mon, 6 Nov 2017 09:50:16 -0500
Tanstaafl <[hidden email]> wrote:

> On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <[hidden email]>
> wrote:
> > Still we are not content with it touching/locking dovecot.index.log. If
> > someone pointed at one location in the code where this could be disabled we
> > would implement a new param for switching that off.  
>
> ?????????????????????????
>
> Dovecot's indexing is one of its main features, and WHY it is so much
> faster than others.
>
> And you want to just turn it off? Good luck...

It seems you have not understood what I am talking about. Our pre-dovecot lda
did not touch the index either. And it did not harm the imap/pop procedure in
any way. So we know there is no need to fiddle with the index in the process
of delivery into the maildirs to keep our performance as it was before.

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Noel Butler
On 07/11/2017 09:18, Stephan von Krawczynski wrote:

> On Mon, 6 Nov 2017 09:50:16 -0500
> Tanstaafl <[hidden email]> wrote:
>
> On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <[hidden email]>
> wrote: Still we are not content with it touching/locking dovecot.index.log. If
> someone pointed at one location in the code where this could be disabled we
> would implement a new param for switching that off.  
> ?????????????????????????
>
> Dovecot's indexing is one of its main features, and WHY it is so much
> faster than others.
>
> And you want to just turn it off? Good luck...

It seems you have not understood what I am talking about. Our
pre-dovecot lda
did not touch the index either. And it did not harm the imap/pop
procedure in
any way. So we know there is no need to fiddle with the index in the
process
of delivery into the maildirs to keep our performance as it was before.

mail_location   Optionally disable indexes using   :INDEX=MEMORY  

don't use this on IMAP boxes, but is safe to use on SMTP and POP3's
boxes though

eg:

mail_location =
maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY

--
Kind Regards,

Noel Butler

  This Email, including any attachments, may contain legally privileged
information, therefore remains confidential and subject to copyright
protected under international law. You may not disseminate, discuss, or
reveal, any part, to anyone, without the authors express written
authority to do so. If you are not the intended recipient, please notify
the sender then delete all copies of this message including attachments,
immediately. Confidentiality, copyright, and legal privilege are not
waived or lost by reason of the mistaken delivery of this message. Only
PDF [1] and ODF [2] documents accepted, please do not send proprietary
formatted documents

 

Links:
------
[1] http://www.adobe.com/
[2] http://en.wikipedia.org/wiki/OpenDocument
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Angel L. Mateo
In reply to this post by Stephan von Krawczynski-2
El 03/11/17 a las 12:50, Stephan von Krawczynski escribió:
> Hello,
>
> we have a setup where SMTP/LDA and POP3/IMAP are on different physical hosts.
> They share the mail data via an external storage.
> Now we would like to use dovecot-lda on the smtp host, so we wonder if the
> lda binary works without starting dovecot from init. As there will be no
> POP3/IMAP usage on this host it seems unnecessary. Nevertheless we cannot
> judge if it is still needed for lda&sieve to work.
> Your opinion?

        You could configure dovecot in the smtp box without pop3 and imap
daemons without any problem (I guess). All you have to do is disable
these services in master configuration.

        But I wouldn't do it that way. I think is easier to configure LDA in
the pop3/imap server and configure SMTP to deliver mail via LMTP to the
dovecot server. This way you have to deal with dovecot configuration
just in the mailbox server and you don't have to share storage.

--
Angel L. Mateo Martínez
Sección de Telemática
Área de Tecnologías de la Información
y las Comunicaciones Aplicadas (ATICA)
http://www.um.es/atica
Tfo: 868889150
Fax: 868888337
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Stephan von Krawczynski-2
In reply to this post by Noel Butler
On Tue, 07 Nov 2017 13:19:12 +1000
Noel Butler <[hidden email]> wrote:
 

> mail_location   Optionally disable indexes using   :INDEX=MEMORY  
>
> don't use this on IMAP boxes, but is safe to use on SMTP and POP3's
> boxes though
>
> eg:
>
> mail_location =
> maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
>
> --
> Kind Regards,
>
> Noel Butler

Hello Noel,

this sounds interesting. Can you please elaborate why you think this is no
good idea for IMAP?
We used a different LDA-scheme before (simply created the mail in maildir/tmp,
then renamed it to maildir/new, just like Bernstein suggests to do) and it
worked very well, no matter if the box was used whith IMAP or POP3.
Why should there be any difference in using dovecot-lda without indexes?
Does dovecot-lda "create" the new mail by atomic rename from tmp, too?

--
Regards,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Tanstaafl
In reply to this post by Stephan von Krawczynski-2
On 11/6/2017, 6:18:43 PM, Stephan von Krawczynski <[hidden email]>
wrote:
> On Mon, 6 Nov 2017 09:50:16 -0500 Tanstaafl <[hidden email]> wrote:
>> Dovecot's indexing is one of its main features, and WHY it is so much
>> faster than others.
>>
>> And you want to just turn it off? Good luck...

> It seems you have not understood what I am talking about. Our pre-dovecot lda
> did not touch the index either. And it did not harm the imap/pop procedure in
> any way. So we know there is no need to fiddle with the index in the process
> of delivery into the maildirs to keep our performance as it was before.

Apparently I'm still missing something...

Sure, you may be keeping your performance at pre-dovecot levels - but
why on earth would that even be a goal?

One of the main reasons I switched from Courier to Dovecot a very long
time ago was for the expected performance boost from the use of the
indexes, which were automatically updated at delivery time (if you used
the LDA), and the boost was huge, I was extremely pleased with the results.

There is no 'fiddling', it just works.
Reply | Threaded
Open this post in threaded view
|

Re: dovecot-lda without starting dovecot?

Timo Sirainen
In reply to this post by Stephan von Krawczynski-2
On 7 Nov 2017, at 9.15, Stephan von Krawczynski <[hidden email]> wrote:

>
> On Tue, 07 Nov 2017 13:19:12 +1000
> Noel Butler <[hidden email]> wrote:
>
>> mail_location   Optionally disable indexes using   :INDEX=MEMORY  
>>
>> don't use this on IMAP boxes, but is safe to use on SMTP and POP3's
>> boxes though
>>
>> eg:
>>
>> mail_location =
>> maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
>>
> Hello Noel,
>
> this sounds interesting. Can you please elaborate why you think this is no
> good idea for IMAP?
> We used a different LDA-scheme before (simply created the mail in maildir/tmp,
> then renamed it to maildir/new, just like Bernstein suggests to do) and it
> worked very well, no matter if the box was used whith IMAP or POP3.
> Why should there be any difference in using dovecot-lda without indexes?
> Does dovecot-lda "create" the new mail by atomic rename from tmp, too?

Although the above disables updating indexes, there's still dovecot-uidlist file that is always updated by dovecot-lda. It might also cause corruption problems when multiple servers are accessing it at the same time. There's some old code that attempts to avoid updating the uidlist when it's not necessary (MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS==0), but I don't know if that works. There's also .dovecot.lda-dupes file that is written. And the Sieve scripts' compiled versions that you wanted to start using. In short: This might work, but I have a feeling you'll run into random corruption problems. I gave up trying to support simultaneous access in multiple servers via NFS long time ago, because no matter what I did kernel always cached too much and caused corruption (or alternatively it didn't cache enough and caused performance problems).
12