Mu4e and the meaning of the "Deleted" IMAP flag

[Dima Kogan]

Hi.

I use mu/mu4e 0.9.9.5-dev6. When I delete a message in mu4e, two things
happen:

1. The message is moved to the Trash folder
2. The message gets a "Deleted" IMAP flag (T maildir flag)

Most of the time, this works, except sometimes I'm seeing that the
messages are expunged from the Trash, and thus disappear. After a few
months of living with this intermittent behavior (and keeping my mail in
git to fight these purges), I figured out the cause: my mail server
deletes all mail with the Deleted flag set, but only actually does this
when I visit their webmail interface, which I do on occasion. The server
is fastmail.fm, by the way.

I asked them about it, and they claim that the IMAP spec says that the
"Deleted" flag means that the mail should be deleted, thus they're right
to do it. Further, they said that it's uncommon and wrong to set the
Deleted flag on the mail in the Trash folder that is intended to be
kept.

I just looked at the IMAP spec. It says that any message with that flag
is deleted when an Expunge command. So I don't know who's right here.
Questions:

1. Does anybody know if mail servers are supposed to be sending the IMAP
Expunge command? Do most of them do it? If so, mu users should be
seeing disappearing mail in the Trash. I see at least one mailing
list post about mysteriously disappearing mail:

http://groups.google.com/group/mu-discuss/browse_frm/thread/cba4fe32958a5e1e/1cafd86fc81e9e53?lnk=gst&q=trashed#1cafd86fc81e9e53

I do know about the 'expunge' offlineimap setting, but I'd rather not
use it.

2. Is mu4e wrong to set the Deleted flag? Currently there's no setting
to prevent it from being added. Should there be? The flag is added
with the "+T" in mu4e-mark-execute-all() in mu4e-mark.el.

Thanks much.

dima

[Dirk-Jan C. Binnema]

Hi,

Post by Dima Kogan
Hi.
I use mu/mu4e 0.9.9.5-dev6. When I delete a message in mu4e, two things
1. The message is moved to the Trash folder
2. The message gets a "Deleted" IMAP flag (T maildir flag)
Most of the time, this works, except sometimes I'm seeing that the
messages are expunged from the Trash, and thus disappear. After a few
months of living with this intermittent behavior (and keeping my mail in
git to fight these purges), I figured out the cause: my mail server
deletes all mail with the Deleted flag set, but only actually does this
when I visit their webmail interface, which I do on occasion. The server
is fastmail.fm, by the way.
I asked them about it, and they claim that the IMAP spec says that the
"Deleted" flag means that the mail should be deleted, thus they're right
to do it. Further, they said that it's uncommon and wrong to set the
Deleted flag on the mail in the Trash folder that is intended to be
kept.
I just looked at the IMAP spec. It says that any message with that flag
is deleted when an Expunge command. So I don't know who's right here.
1. Does anybody know if mail servers are supposed to be sending the IMAP
Expunge command? Do most of them do it? If so, mu users should be
seeing disappearing mail in the Trash. I see at least one mailing
http://groups.google.com/group/mu-discuss/browse_frm/thread/cba4fe32958a5e1e/1cafd86fc81e9e53?lnk=gst&q=trashed#1cafd86fc81e9e53
I do know about the 'expunge' offlineimap setting, but I'd rather not
use it.
2. Is mu4e wrong to set the Deleted flag? Currently there's no setting
to prevent it from being added. Should there be? The flag is added
with the "+T" in mu4e-mark-execute-all() in mu4e-mark.el.

mu4e is not setting the 'deleted' flag, it's setting the 'trashed' flag;
as the maildir spec[1] says:

"Flag "T" (trashed): the user has moved this message to the trash;
the trash will be emptied by a later user action."

So, the issue here is that offlineimap interprets this as the "Deleted"
flag, and then the IMAP-server just uses that.

Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.

Best wishes,
Dirk.


[1] http://cr.yp.to/proto/maildir.html

[Dima Kogan]

Post by Dirk-Jan C. Binnema
Hi,

Post by Dima Kogan
Hi.
I use mu/mu4e 0.9.9.5-dev6. When I delete a message in mu4e, two things
1. The message is moved to the Trash folder
2. The message gets a "Deleted" IMAP flag (T maildir flag)
Most of the time, this works, except sometimes I'm seeing that the
messages are expunged from the Trash, and thus disappear. After a few
months of living with this intermittent behavior (and keeping my mail in
git to fight these purges), I figured out the cause: my mail server
deletes all mail with the Deleted flag set, but only actually does this
when I visit their webmail interface, which I do on occasion. The server
is fastmail.fm, by the way.
I asked them about it, and they claim that the IMAP spec says that the
"Deleted" flag means that the mail should be deleted, thus they're right
to do it. Further, they said that it's uncommon and wrong to set the
Deleted flag on the mail in the Trash folder that is intended to be
kept.
I just looked at the IMAP spec. It says that any message with that flag
is deleted when an Expunge command. So I don't know who's right here.
1. Does anybody know if mail servers are supposed to be sending the IMAP
Expunge command? Do most of them do it? If so, mu users should be
seeing disappearing mail in the Trash. I see at least one mailing
http://groups.google.com/group/mu-discuss/browse_frm/thread/cba4fe32958a5e1e/1cafd86fc81e9e53?lnk=gst&q=trashed#1cafd86fc81e9e53
I do know about the 'expunge' offlineimap setting, but I'd rather not
use it.
2. Is mu4e wrong to set the Deleted flag? Currently there's no setting
to prevent it from being added. Should there be? The flag is added
with the "+T" in mu4e-mark-execute-all() in mu4e-mark.el.

mu4e is not setting the 'deleted' flag, it's setting the 'trashed' flag;
"Flag "T" (trashed): the user has moved this message to the trash;
the trash will be emptied by a later user action."
So, the issue here is that offlineimap interprets this as the "Deleted"
flag, and then the IMAP-server just uses that.
Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.
Best wishes,
Dirk.
[1] http://cr.yp.to/proto/maildir.html

Hi Dirk.

The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Thanks

dima

[Dirk-Jan C. Binnema]

<snip>

Post by Dima Kogan

Post by Dirk-Jan C. Binnema
mu4e is not setting the 'deleted' flag, it's setting the 'trashed' flag;
"Flag "T" (trashed): the user has moved this message to the trash;
the trash will be emptied by a later user action."
So, the issue here is that offlineimap interprets this as the "Deleted"
flag, and then the IMAP-server just uses that.
Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.
[1] http://cr.yp.to/proto/maildir.html

Hi Dirk.
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

Cheers,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:djcb-***@public.gmane.org w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Dima Kogan]

Post by Dirk-Jan C. Binnema

Post by Dima Kogan
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?
From an interface viewpoint, currently I use the 'd' key to move mail to

the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.

I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

Thanks

[Joost Kremers]

Post by Dirk-Jan C. Binnema

Post by Dirk-Jan C. Binnema

Post by Dima Kogan
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.
I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

I use fastmail.fm and I don't have a problem with the way that trashed mail is handled. Specifically, I'm wondering why you move mail to the trash and expect it to remain there. I mean, the trash mail folder is really what its name says: trash. Stuff you throw away. If you want to keep a mail, you shouldn't be putting it in the trash, right?

Or am I missing something?

[Dima Kogan]

Post by Joost Kremers

Post by Dirk-Jan C. Binnema

Post by Dirk-Jan C. Binnema

Post by Dima Kogan
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.
I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

I use fastmail.fm and I don't have a problem with the way that trashed
mail is handled. Specifically, I'm wondering why you move mail to the
trash and expect it to remain there. I mean, the trash mail folder is
really what its name says: trash. Stuff you throw away. If you want to
keep a mail, you shouldn't be putting it in the trash, right?
Or am I missing something?

Maybe I've been doing it wrong all these years. My mail workflow is to
move all mail that I'm done with into the Trash, but to never actually
delete anything. I treat the "Trash" as an "archive", so to speak, and I
don't have any actual "Trash".

This has been working fine, in general. Some mail servers like to purge
your Trash for you, so I disable that.

Is this workflow very different from what others use? Do most people
have an "archive" and a "Trash"?

As for actually sending Expunge, I just tested two other mail servers I
have access to. Zimbra doesn't Expunge Deleted mail, at least not
immediately. Gmail apparently does, like fastmail.fm. So any gmail IMAP
users that use stock offlineimap and stock mu4e would see mail they
place into the Trash disappear.

dima

[Mandar Mitra]

Post by Dima Kogan
Is this workflow very different from what others use? Do most people
have an "archive" and a "Trash"?

I can't speak for most people, but yes, "archive" and "trash" are
different for me (I use Gmail). Gmail keeps archived email in "All
Mail" and deleted mail in Trash. "Trash" seems an odd choice of name
for archive, no?

Post by Dima Kogan
As for actually sending Expunge, I just tested two other mail servers I
have access to. Zimbra doesn't Expunge Deleted mail, at least not
immediately. Gmail apparently does, like fastmail.fm. So any gmail IMAP
users that use stock offlineimap and stock mu4e would see mail they
place into the Trash disappear.

I use mutt + offlineimap and have bound 'd' (normally delete) to save
a message to Gmail's Trash. This way, I get behaviour that is consistent
with the web interface: "deleted" mail goes to Trash and remains there
for some days. Gmail takes care of actually removing it all together.

mandar.

[James Ladan]

Post by Joost Kremers
I use fastmail.fm and I don't have a problem with the way that trashed
mail is handled. Specifically, I'm wondering why you move mail to the trash
and expect it to remain there. I mean, the trash mail folder is really what
its name says: trash. Stuff you throw away. If you want to keep a mail, you
shouldn't be putting it in the trash, right?
Or am I missing something?

In the Gmail case, it automatically flushes things out after ~30 days, so I
can trash to my heart's content and if I make a brain-dead move to trash, I
can always recover it within that window. With the interaction between mu4e
and offlineimap, the email is immediately purged and gone for good. Usually
not so bad, but not what's expected for the normal Gmail workflow.

[James Ladan]

Post by Dirk-Jan C. Binnema
From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.
I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

Gmail had the same issue for me. I had to move to trash instead of using
'd'.

[Dirk-Jan C. Binnema]

Post by Dirk-Jan C. Binnema

Post by Dirk-Jan C. Binnema

Post by Dima Kogan
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.
I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

It's not that I don't want to help you, but adding extra complexity here
doesn't seem warranted since mu4e allows you to work-around the
offlineimap behavior -- I'd be happy to include a note in the
documentation, and you can change the keybindings etc.

However, I think the best place to change this would be in offlineimap.

Best wishes,
Dirk.

[Steve Dowe]

Post by Dirk-Jan C. Binnema
However, I think the best place to change this would be in offlineimap.

I am inclined to agree with this. The deletion behaviour is completely
mail server specific. The Dovecot IMAP server, for example, doesn't
automatically expunge trashed mail after a set period, whereas the
Courier IMAP server apparently does.

Mu sits "down the line" from IMAP behaviour. My experience with Mu,
offlineimap and Dovecot has been one of harmony - /almost/ no
inconsistencies at all... :D
--
Steve Dowe warp2.me/sd
Warp Universal warpuni.com

[Dima Kogan]

Post by Dirk-Jan C. Binnema

Post by Dirk-Jan C. Binnema

Post by Dirk-Jan C. Binnema

Post by Dima Kogan
The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.
I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

It's not that I don't want to help you, but adding extra complexity here
doesn't seem warranted since mu4e allows you to work-around the
offlineimap behavior -- I'd be happy to include a note in the
documentation, and you can change the keybindings etc.
However, I think the best place to change this would be in offlineimap.

OK.

If you could include a documentation note that'd be great.

Thanks again.

dima

[Alan Schmitt]

Post by Dirk-Jan C. Binnema
Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.

I was having trouble with this a while back, so I put this in my
configuration:

,----
| (setq mu4e-maildir-shortcuts
| '( ...
| ("/Trash" . ?T)
| ...)
|
| (fset 'my-move-to-trash "mT")
| (define-key mu4e-headers-mode-map (kbd "d") 'my-move-to-trash)
| (define-key mu4e-view-mode-map (kbd "d") 'my-move-to-trash)
`----

Alan

[Paul Rankin]

Post by Alan Schmitt

Post by Dirk-Jan C. Binnema
Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.

I was having trouble with this a while back, so I put this in my
,----
| (setq mu4e-maildir-shortcuts
| '( ...
| ("/Trash" . ?T)
| ...)
|
| (fset 'my-move-to-trash "mT")
| (define-key mu4e-headers-mode-map (kbd "d") 'my-move-to-trash)
| (define-key mu4e-view-mode-map (kbd "d") 'my-move-to-trash)
`----
Alan

Thank you so much. Between the offlineimap expunge settings and this I was
going a little nuts.

I think the FastMail guys are at fault here, but an option to prevent the
flag within mu4e would be nice too.