dovecot as gmail imap proxy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

dovecot as gmail imap proxy

Farkas Levente-2
hi,
may be this is a bit of topic, but i hope not. it's always a big
question which is the better to choose a hosting provider to keep and
manage you mail or setup your own mail server, hosting, virus and spam
filter etc.. now as more and more people and company move to gmail or
google apps mail service it seems google can't handle the load and it's
getting more and more slower. what's more not just the imap interface
but nowadays the web interface used to be hang or even stop working.
it's getting more and more annoying that the speed is worst than in a 10
years ago. i thing about how can this be solved without totally give up
gmail as a mail service provider (and may be they can solve it in a few
months/years:-).
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he can
use gmail in vacation or any other time, but when the email is real
critical apps (not acceptable to wait 1 minutes for an email to load or
to open a folder/label) than he can use a proxy server eg. dovecot which
act as a imap proxy to gmail and can be hosted on closer, faster server
and this speed problem never happend. what's more it can be used as a
mail backup server too.
anybody has any such setup, idea? would it be possible with dovecot.
any other suggestion?
yours.

--
   Levente                               "Si vis pacem para bellum!"
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Alexander Prinsier-4
To have a speed increase, dovecot needs direct access to the emails. So
you'll need them stored locally. You can't use it as a proxy, just to
speed up things, because dovecot would just need to wait for gmail too,
to load an email (except if dovecot would cache the emails, but then
again, it's just a cache, and you're talking about 'critical applications').

I'm not sure, but I don't think there's even a way to run dovecot as a
proxy.

Why don't you forward mail from your gmail account to your own server?
That's so much easier and more common. You get the benefit of good
filtering, and you have your own server to access your mail. If your own
server goes down, you still have gmail as backup.

Alexander

Farkas Levente wrote:

> hi,
> may be this is a bit of topic, but i hope not. it's always a big
> question which is the better to choose a hosting provider to keep and
> manage you mail or setup your own mail server, hosting, virus and spam
> filter etc.. now as more and more people and company move to gmail or
> google apps mail service it seems google can't handle the load and it's
> getting more and more slower. what's more not just the imap interface
> but nowadays the web interface used to be hang or even stop working.
> it's getting more and more annoying that the speed is worst than in a 10
> years ago. i thing about how can this be solved without totally give up
> gmail as a mail service provider (and may be they can solve it in a few
> months/years:-).
> would it be possible to create a dovecot server as an imap proxy for
> gmail and google apps?
> imho many people/company would be love to use such a setup where he can
> use gmail in vacation or any other time, but when the email is real
> critical apps (not acceptable to wait 1 minutes for an email to load or
> to open a folder/label) than he can use a proxy server eg. dovecot which
> act as a imap proxy to gmail and can be hosted on closer, faster server
> and this speed problem never happend. what's more it can be used as a
> mail backup server too.
> anybody has any such setup, idea? would it be possible with dovecot.
> any other suggestion?
> yours.
>

Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Timo Sirainen
In reply to this post by Farkas Levente-2
On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote:

> would it be possible to create a dovecot server as an imap proxy for  
> gmail and google apps?
> imho many people/company would be love to use such a setup where he  
> can use gmail in vacation or any other time, but when the email is  
> real critical apps (not acceptable to wait 1 minutes for an email to  
> load or to open a folder/label) than he can use a proxy server eg.  
> dovecot which act as a imap proxy to gmail and can be hosted on  
> closer, faster server and this speed problem never happend. what's  
> more it can be used as a mail backup server too.

I've thought about a caching IMAP mail storage backend for Dovecot and  
even started implementing it once, but it didn't get very far. If  
someone wants to try to continue it, I can send what I've written so  
far. Although it's mostly a copy of cydir backend with a tiny bit of  
it replaced with IMAP code.


PGP.sig (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Timo Sirainen
On Jun 17, 2008, at 6:14 PM, Timo Sirainen wrote:

> On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote:
>
>> would it be possible to create a dovecot server as an imap proxy  
>> for gmail and google apps?
>> imho many people/company would be love to use such a setup where he  
>> can use gmail in vacation or any other time, but when the email is  
>> real critical apps (not acceptable to wait 1 minutes for an email  
>> to load or to open a folder/label) than he can use a proxy server  
>> eg. dovecot which act as a imap proxy to gmail and can be hosted on  
>> closer, faster server and this speed problem never happend. what's  
>> more it can be used as a mail backup server too.
>
> I've thought about a caching IMAP mail storage backend for Dovecot  
> and even started implementing it once, but it didn't get very far.  
> If someone wants to try to continue it, I can send what I've written  
> so far. Although it's mostly a copy of cydir backend with a tiny bit  
> of it replaced with IMAP code.
Now that I think of it, the IMAP backend should do only direct IMAP  
backend accessing and if caching is wanted it could be a generic  
middle layer. Might be also useful for other things like caching mails  
from NFS server. Although Linux already has a cachefs for that.

PGP.sig (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Farkas Levente-2
Timo Sirainen wrote:

> On Jun 17, 2008, at 6:14 PM, Timo Sirainen wrote:
>
>> On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote:
>>
>>> would it be possible to create a dovecot server as an imap proxy for
>>> gmail and google apps?
>>> imho many people/company would be love to use such a setup where he
>>> can use gmail in vacation or any other time, but when the email is
>>> real critical apps (not acceptable to wait 1 minutes for an email to
>>> load or to open a folder/label) than he can use a proxy server eg.
>>> dovecot which act as a imap proxy to gmail and can be hosted on
>>> closer, faster server and this speed problem never happend. what's
>>> more it can be used as a mail backup server too.
>>
>> I've thought about a caching IMAP mail storage backend for Dovecot and
>> even started implementing it once, but it didn't get very far. If
>> someone wants to try to continue it, I can send what I've written so
>> far. Although it's mostly a copy of cydir backend with a tiny bit of
>> it replaced with IMAP code.
>
> Now that I think of it, the IMAP backend should do only direct IMAP
> backend accessing and if caching is wanted it could be a generic middle
> layer. Might be also useful for other things like caching mails from NFS
> server. Although Linux already has a cachefs for that.

yes, that's exactly what i'm thinking about. all imap command which
issued by the client to the middle layer should have to propagate back
to the backed (in this case gmail). but imho the hard part that we've to
assume that the backend (in this case gmail) is slow, so the middle
layer have to cache the imap commands to be able to propagate all
command to the backend (and command queue for all mailbox). so if the
connection between dovecot and gmail are fast then both has the same
messages and flags, etc. if the connection is slow than gmail may have a
small delay, but this delay probably won't be visible to the user (if he
not use the web interface at the same time too).


--
   Levente                               "Si vis pacem para bellum!"
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Peter Hessler
In reply to this post by Alexander Prinsier-4
I do this exact thing, and it is faster than gmail itself.  With IDLE, I
get the msg alert on my dovecot server before the web interface shows a
new message.

Also, far more reliable, with a superior UI.


On 2008 Jun 17 (Tue) at 17:13:32 +0200 (+0200), Alexander Prinsier wrote:
:Why don't you forward mail from your gmail account to your own server?
:That's so much easier and more common. You get the benefit of good
:filtering, and you have your own server to access your mail. If your own
:server goes down, you still have gmail as backup.

--
New Hampshire law forbids you to tap your feet, nod your head, or in
any way keep time to the music in a tavern, restaurant, or cafe.
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Farkas Levente-2
Peter Hessler wrote:
> I do this exact thing, and it is faster than gmail itself.  With IDLE, I
> get the msg alert on my dovecot server before the web interface shows a
> new message.

which exact thing?
and what does the "with idle" means?

> Also, far more reliable, with a superior UI.
>
>
> On 2008 Jun 17 (Tue) at 17:13:32 +0200 (+0200), Alexander Prinsier wrote:
> :Why don't you forward mail from your gmail account to your own server?
> :That's so much easier and more common. You get the benefit of good
> :filtering, and you have your own server to access your mail. If your own
> :server goes down, you still have gmail as backup.
>
       

--
   Levente                               "Si vis pacem para bellum!"
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Farkas Levente-2
In reply to this post by Timo Sirainen
Timo Sirainen wrote:

> On Jun 17, 2008, at 6:14 PM, Timo Sirainen wrote:
>
>> On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote:
>>
>>> would it be possible to create a dovecot server as an imap proxy for
>>> gmail and google apps?
>>> imho many people/company would be love to use such a setup where he
>>> can use gmail in vacation or any other time, but when the email is
>>> real critical apps (not acceptable to wait 1 minutes for an email to
>>> load or to open a folder/label) than he can use a proxy server eg.
>>> dovecot which act as a imap proxy to gmail and can be hosted on
>>> closer, faster server and this speed problem never happend. what's
>>> more it can be used as a mail backup server too.
>>
>> I've thought about a caching IMAP mail storage backend for Dovecot and
>> even started implementing it once, but it didn't get very far. If
>> someone wants to try to continue it, I can send what I've written so
>> far. Although it's mostly a copy of cydir backend with a tiny bit of
>> it replaced with IMAP code.
>
> Now that I think of it, the IMAP backend should do only direct IMAP
> backend accessing and if caching is wanted it could be a generic middle
> layer. Might be also useful for other things like caching mails from NFS
> server. Although Linux already has a cachefs for that.

it seems other people also thing about this:
http://community.livejournal.com/linux/1748995.html
so it'd be useful thing for many of us:-)

--
   Levente                               "Si vis pacem para bellum!"
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Asheesh Laroia
In reply to this post by Peter Hessler
On Tue, 17 Jun 2008, Peter Hessler wrote:

> I do this exact thing, and it is faster than gmail itself.  With IDLE, I
> get the msg alert on my dovecot server before the web interface shows a
> new message.
>
> Also, far more reliable, with a superior UI.

Which is the superior UI?  Honest question - I'm shopping around for nice
IMAP interfaces, if that's what you mean.  I'd love to know what you're
using.

-- Asheesh.

--
Marriage is the triumph of imagination over intelligence.  Second marriage is
the triumph of hope over experience.
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Asheesh Laroia
In reply to this post by Farkas Levente-2
On Tue, 17 Jun 2008, Farkas Levente wrote:

> hi,
> may be this is a bit of topic, but i hope not. it's always a big question
> which is the better to choose a hosting provider to keep and manage you mail
> or setup your own mail server, hosting, virus and spam filter etc.. now as
> more and more people and company move to gmail or google apps mail service it
> seems google can't handle the load and it's getting more and more slower.
> what's more not just the imap interface but nowadays the web interface used
> to be hang or even stop working. it's getting more and more annoying that the
> speed is worst than in a 10 years ago. i thing about how can this be solved
> without totally give up gmail as a mail service provider (and may be they can
> solve it in a few months/years:-).
> would it be possible to create a dovecot server as an imap proxy for gmail
> and google apps?

What I have is:

* rose, my primary server (but far away)
* supercore, another server at my parents' house

rose is the primary MX.

I run offlineimap on supercore to sync a local Maildir on supercore with
rose's IMAP - that way, my parents can use a server with <1ms ping, and as
they use Dovecot to modify the local Maildir on supercore, a minute
later offlineimap will wake up and create IMAP operations to synchronize
rose with the state on supercore.

FWIW.

-- Asheesh.

--
Real Users find the one combination of bizarre input values that shuts
down the system for days.
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Peter Hessler
In reply to this post by Asheesh Laroia
On 2008 Jun 17 (Tue) at 17:18:20 -0400 (-0400), Asheesh Laroia wrote:

> On Tue, 17 Jun 2008, Peter Hessler wrote:
>
>> I do this exact thing, and it is faster than gmail itself.  With IDLE,
>> I get the msg alert on my dovecot server before the web interface shows
>> a new message.
>>
>> Also, far more reliable, with a superior UI.
>
> Which is the superior UI?  Honest question - I'm shopping around for nice
> IMAP interfaces, if that's what you mean.  I'd love to know what you're  
> using.
>
> -- Asheesh.
>
> --
> Marriage is the triumph of imagination over intelligence.  Second marriage is
> the triumph of hope over experience.
>

I <3 mutt.  Easy enough, but lots of power is available.

I haven't found any webmail interfaces that I like, but then again, I have yet
to find *any* web interface that I like.


--
Marriage is the only adventure open to the cowardly.
                -- Voltaire
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Geert Hendrickx
On Tue, Jun 17, 2008 at 03:37:30PM -0700, Peter Hessler wrote:
> I <3 mutt.  Easy enough, but lots of power is available.


Yes, mutt is great! :-)

Btw, you can use mutt with dovecot without running dovecot as a daemon/server,
by putting this in your .muttrc:

set tunnel="MAIL=/home/geert/mbox /usr/pkg/libexec/dovecot/imap"

(adapt paths for your own system)

This way mutt will talk to the dovecot imap binary via stdin/stdout.  This
will also work over ssh.  This way you don't need to run dovecot as a server
and you still benefit from eg. the cache etc.

        Geert


Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Timo Sirainen
On Wed, 2008-06-18 at 10:32 +0200, Geert Hendrickx wrote:

> On Tue, Jun 17, 2008 at 03:37:30PM -0700, Peter Hessler wrote:
> > I <3 mutt.  Easy enough, but lots of power is available.
>
>
> Yes, mutt is great! :-)
>
> Btw, you can use mutt with dovecot without running dovecot as a daemon/server,
> by putting this in your .muttrc:
>
> set tunnel="MAIL=/home/geert/mbox /usr/pkg/libexec/dovecot/imap"
"dovecot --exec-mail imap" is better. Then it uses the settings from
dovecot.conf as well as the proper default settings. With the above
you're for example using mbox_lazy_writes=no


signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Farkas Levente-2
In reply to this post by Asheesh Laroia
Asheesh Laroia wrote:

> On Tue, 17 Jun 2008, Farkas Levente wrote:
>
>> hi,
>> may be this is a bit of topic, but i hope not. it's always a big
>> question which is the better to choose a hosting provider to keep and
>> manage you mail or setup your own mail server, hosting, virus and spam
>> filter etc.. now as more and more people and company move to gmail or
>> google apps mail service it seems google can't handle the load and
>> it's getting more and more slower. what's more not just the imap
>> interface but nowadays the web interface used to be hang or even stop
>> working. it's getting more and more annoying that the speed is worst
>> than in a 10 years ago. i thing about how can this be solved without
>> totally give up gmail as a mail service provider (and may be they can
>> solve it in a few months/years:-).
>> would it be possible to create a dovecot server as an imap proxy for
>> gmail and google apps?
>
> What I have is:
>
> * rose, my primary server (but far away)
> * supercore, another server at my parents' house
>
> rose is the primary MX.
>
> I run offlineimap on supercore to sync a local Maildir on supercore with
> rose's IMAP - that way, my parents can use a server with <1ms ping, and
> as they use Dovecot to modify the local Maildir on supercore, a minute
> later offlineimap will wake up and create IMAP operations to synchronize
> rose with the state on supercore.

i'm just look into offlineimap. half of the code for ui, another part to
implement maildir and imap. so it seems for me there is only a little
code which do the synchronization.
Timo what do you think how much work would be add such a middleware imap
server capabilities (ie. imap backend) to dovecot?

--
   Levente                               "Si vis pacem para bellum!"
Reply | Threaded
Open this post in threaded view
|

Re: dovecot as gmail imap proxy

Timo Sirainen
On Jun 18, 2008, at 2:59 PM, Farkas Levente wrote:

> Timo what do you think how much work would be add such a middleware  
> imap server capabilities (ie. imap backend) to dovecot?

I think it would be quite a lot of work. Definitely a lot more than I  
have time for currently. The most difficult thing is probably writing  
IMAP parsing and event handling for it. I've written such code for my  
imaptest tool already though, maybe at some point they could be more  
or less merged to a common lib-imap-client.


PGP.sig (201 bytes) Download Attachment