Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Stefan Neben
Hello,

i have a problem with a very huge mail account.

The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
from time to time the index cache make problems and i got the
following in the logs:

--- Snip ---
Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
failed: (offset < 0x40000000)
--- Snap ---

I skip the raw backtrace to keep the post "small", but can hand it in
later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...

The size of the file dovecot.index.cache of INBOX was in the last seen
problem situations between 1.1GB to 2.3GB.

To solve the problem the last times was to reduce the amount of mails
to under 1.7 million (seems to be the limit to trigger the problem),
delete the index cache and let him rebuild with a controled request.
But this is a odd situation (because it confused 3rd party tooling,
which access this mailbox) and i cannot find any helpful informations
to solve this problem.

Sharding it into several folders to keep the amount of mails in INBOX
small is not an option currently.

If further informations are needed, just let me know. Thanks for any help!


Kind regards
Stefan
Reply | Threaded
Open this post in threaded view
|

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Stefan Neben
Sorry, i forgot to mention, that the storage format is mdbox and the
version is 2.2.27-3~bpo8+1 from jessie backports.

Kind regards
Stefan

2018-01-10 21:00 GMT+01:00 Stefan Neben <[hidden email]>:

> Hello,
>
> i have a problem with a very huge mail account.
>
> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
> from time to time the index cache make problems and i got the
> following in the logs:
>
> --- Snip ---
> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
> failed: (offset < 0x40000000)
> --- Snap ---
>
> I skip the raw backtrace to keep the post "small", but can hand it in
> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>
> The size of the file dovecot.index.cache of INBOX was in the last seen
> problem situations between 1.1GB to 2.3GB.
>
> To solve the problem the last times was to reduce the amount of mails
> to under 1.7 million (seems to be the limit to trigger the problem),
> delete the index cache and let him rebuild with a controled request.
> But this is a odd situation (because it confused 3rd party tooling,
> which access this mailbox) and i cannot find any helpful informations
> to solve this problem.
>
> Sharding it into several folders to keep the amount of mails in INBOX
> small is not an option currently.
>
> If further informations are needed, just let me know. Thanks for any help!
>
>
> Kind regards
> Stefan
Reply | Threaded
Open this post in threaded view
|

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Rupert Gallagher
You are storing 1.7 million e-mails in a single mbox file. I would rather store the archive using one file per e-mail. 
Reply | Threaded
Open this post in threaded view
|

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Stefan Neben
Hi,

> You are storing 1.7 million e-mails in a single mbox file.

no, of course not:

mdbox_rotate_size = 64 M

> I would rather store the archive using one file per e-mail.

I think one file per mail would be not good for this setup. One
keypoint is that the INBOX is searchable (in a acceptable amount of
time). Further this probably could slow down the backup process (i
think).

2018-01-11 0:46 GMT+01:00 Rupert Gallagher <[hidden email]>:
> You are storing 1.7 million e-mails in a single mbox file. I would rather
> store the archive using one file per e-mail.
Reply | Threaded
Open this post in threaded view
|

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Aki Tuomi-2
In reply to this post by Stefan Neben


On 10.01.2018 22:00, Stefan Neben wrote:

> Hello,
>
> i have a problem with a very huge mail account.
>
> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
> from time to time the index cache make problems and i got the
> following in the logs:
>
> --- Snip ---
> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
> failed: (offset < 0x40000000)
> --- Snap ---
>
> I skip the raw backtrace to keep the post "small", but can hand it in
> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>
> The size of the file dovecot.index.cache of INBOX was in the last seen
> problem situations between 1.1GB to 2.3GB.
>
> To solve the problem the last times was to reduce the amount of mails
> to under 1.7 million (seems to be the limit to trigger the problem),
> delete the index cache and let him rebuild with a controled request.
> But this is a odd situation (because it confused 3rd party tooling,
> which access this mailbox) and i cannot find any helpful informations
> to solve this problem.
>
> Sharding it into several folders to keep the amount of mails in INBOX
> small is not an option currently.
>
> If further informations are needed, just let me know. Thanks for any help!
>
>
> Kind regards
> Stefan
Hi!

Unfortunately dovecot has an in-built limit of 2GB for the index cache
file. Your only solution is to move some of the mails to some other folder.

Aki
Reply | Threaded
Open this post in threaded view
|

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

Stefan Neben
Hi,

> Unfortunately dovecot has an in-built limit of
> 2GB for the index cache file

okay. I assumed there is a limit but not that low, thought about 4GB
because of this:

https://www.dovecot.org/list/dovecot/2015-September/101863.html

And the strange thing is, i had this problem yesterday twice. The
first time the index cache was 1.2GB and the second time it was 1.1GB.
Is there a second factor, which could limit this?

> Your only solution is to move some of the mails to some other folder.

Yes, this was my fallback solution, but this means a lot of work to
adapt the 3rd party tooling around it :-(

2018-01-11 12:33 GMT+01:00 Aki Tuomi <[hidden email]>:

>
>
> On 10.01.2018 22:00, Stefan Neben wrote:
>> Hello,
>>
>> i have a problem with a very huge mail account.
>>
>> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
>> from time to time the index cache make problems and i got the
>> following in the logs:
>>
>> --- Snip ---
>> Jan 10 20:43:04 XXXXX dovecot: imap(XXXXX): Panic: file
>> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
>> failed: (offset < 0x40000000)
>> --- Snap ---
>>
>> I skip the raw backtrace to keep the post "small", but can hand it in
>> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>>
>> The size of the file dovecot.index.cache of INBOX was in the last seen
>> problem situations between 1.1GB to 2.3GB.
>>
>> To solve the problem the last times was to reduce the amount of mails
>> to under 1.7 million (seems to be the limit to trigger the problem),
>> delete the index cache and let him rebuild with a controled request.
>> But this is a odd situation (because it confused 3rd party tooling,
>> which access this mailbox) and i cannot find any helpful informations
>> to solve this problem.
>>
>> Sharding it into several folders to keep the amount of mails in INBOX
>> small is not an option currently.
>>
>> If further informations are needed, just let me know. Thanks for any help!
>>
>>
>> Kind regards
>> Stefan
> Hi!
>
> Unfortunately dovecot has an in-built limit of 2GB for the index cache
> file. Your only solution is to move some of the mails to some other folder.
>
> Aki