Internet-Draft Structured and Dynamic Email January 2023
Happel Expires 31 July 2023 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-happel-structured-dynamic-email-00
Published:
Intended Status:
Informational
Expires:
Author:
H.-J. Happel
audriga GmbH

Structured and Dynamic Email

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 31 July 2023.

Table of Contents

1. Introduction

Email can be considered a successful technology, based on its broad adoption and based on the fact dozens of email-related RFCs have been written over time [MailRFCs].

Despite of these various RFCs, which are dealing with a multitude of small improvements, the major inner workings of email remain widely unchanged since their inception.

In particular, email is still mostly a text-based medium, which requires a lot of human attention, even though more and more so called "transactional" emails are authored in an automated fashion. Accordingly, the task of dealing with large amounts of email is still a major challenge for many users.

Most existing attempts that have been made to improve this situation focus on particular use cases. The main point of this paper is to suggest that it might be worthwhile to consider if there exists a generalizable core underneath those use cases. Such a more streamlined approach might help to condense specifications and to increase adoption by email servers and clients.

Furthermore, we argue that such an approach might enable novel use cases by leveraging the specific and partially unique properties of email technology, which include:

Essentially, this entails that a consciously designed approach for the automated processing of formally structured information in emails could be considered a "private API" of email users. While this certainly raises security and trust concerns, the opportunities behind such unifying approach seem worthwhile discussing.

2. Problem statement

2.1. Lack of structured data

A large share of today's emails is of transactional nature, i.e., sent by automated agents or processes to human users. While those emails often have relatively clear semantics (such as Invoice or Reservation), the medium of those emails is still human-readable text. Users and their email software cannot leverage knowledge inside and about those emails to make their processing a more efficient and enjoyable task.

2.2. Lack of structured interaction

Using informal HTML email conventions, such emails can be considered as small websites, adapted and personalized for a certain use case. However, they are mostly a "dead end" for interaction. Due to the lack of explicit semantics in the underlying email, particular actions afforded by transactional mail (such as Approval or Rating) cannot be easily made actionable for users. Also, many interactions require to switch from email to a regular web browser, resulting in a process disruption which is coined Medienbruch in German language.

2.3. Current solutions

Several attempts have been made to improve this situation. We discuss some of them in the following.

Within the IETF, various RFCs have been devised to structure certain types of email interaction. Probably most notable, iMIP [RFC2447] allows users to deal with meeting workflows in a structured manner. Further examples structure interaction with message delivery notifications [RFC8098], mailinglist subscriptions [RFC8058] or specify header-based semantic indicators for certain domains [RFC6477] or Emoji-based semantics of approval/disapproval in mailinglist discussions [RFC9078].

In more administrative use cases, special kinds of email body parts are used for abuse reporting [RFC6650] or administrative reporting [RFC6522]. In a USENET context [RFC1036], so-called "control messages" are defined for what can be considered certain API calls, such as for creating a USENET group.

Looking outside the IETF, the "Email Markup" approach [EMarkup] by Google allows allows senders to integrate certain Schema.org [SchemaOrg] annotations (such as for hotel reservations or parcel delivery) in their email, such that email clients can provide customized support, including certain easy to use response actions.

Email markup is still in use, but has a number of limitations:

  • Being a proprietary standard owned by Google
  • It does not support to annotate standard email elements such as headers or attachments
  • It is supported by only few providers
  • It is limited to a few fixed use cases
  • It is only available for senders that went through an approval process
  • It is not designed for human end users to send structured emails

More recently, AMP email [AMPemail] (also initiated by Google) and Microsoft Actionable Messages [AM] have been specified. Compared to Email Markup, their focus is less on semantically annotating email content, but on more rich interaction, e.g., allowing the submission of simple forms. In addition, both approaches can include dynamic elements, by retrieving certain data, or even replacing the complete email body at reading time.

Both approaches however suffer from similar limitations as those already pointed out for Email Markup. There also seems to be a lack of consensus, since Microsoft Outlook is reported to have removed initial support for AMP email [OutAMP].

2.4. Goals

To summarize, we see a need for a vendor neutral standard for structured and dynamic email. It should enable email senders to provide structured information about the semantics of their message and allow recipients (may it be human users or their agents) for rich and structured interaction with those emails.

Such standard should also take into account the various RFCs specifying email semantics for specific use cases and ideally provide a more generalized framework for such use cases, which can be more consistently and easily implemented by vendors. For example, there exist various "informal" standards for certain types of email messages, that have not yet been standardized in dedicated RFCs, such as out-of-office (aka vacation notice) emails.

3. Design principles

3.1. Protocol-agnostic implementation

Structured email should work with existing protocols such as IMAP [RFC9051], SMTP [RFC5321], and novel ones such as JMAP [RFC8620]. Accordingly, most extensions would probably realized in the scope of the Internet Message Format [RFC5322] or in areas not yet addressed by standardization (see, e.g., Section 4.3, Section 4.5, Section 4.6, Section 4.7).

3.2. Co-existence of approaches

A major advantage of email technology is the fact, that email is backwards compatible. For example, most HTML emails are accompanied by a plain text version to be used as a fallback in case an email client cannot deal with HTML. In a similar fashion, structured email could be provided as an additional option for clients able and willing to deal with it, thus avoiding a classic "chicken/egg" problem for novel technology.

3.3. Decentral semantics

With respect to structured markup, an incremental and decentral approach is proposed. Instead of an authoritative fixed set of schemas, as in the case of Email Markup, users should be allows to create and extend schemas on their own.

4. Solution approach

This draft can and will not yet provide a full specification for the problems stated. In this section, we will single out certain aspects that might be addressed by such specifications.

4.1. Email content (body)

As described before, an (optional and alternate) structured description of the semantics of email messages is supposed be in the core of this proposal. While existing approaches such as Email Markup offer different options for how to include structured data, an alternative, machine-readable presentation of the message content in addition to human readable text and HTML representations, might be the most natural approach.

4.2. Email metadata (header)

For the sake of efficient processing and perhaps also for reasons of data privacy, certain structured data about an email might also be captured in its headers.

Header fields of certain message parts (such as attachments) might also be enriched by semantic annotation.

4.3. Email storage

Email messages stored in IMAP are currently immutable. Receivers are not supposed to modify their actual content. Hence, the main means to add and store additional data on the receiving side are:

  • IMAP flags
  • IMAP mailboxes (sorting into folders, which can be considered as some sort of tagging)
  • Custom data storage on server or email client side

As of today, this already yields certain data portability problems - e.g., IMAP flags are often not considered when exporting raw email message files. Accordingly, we see a need for the standardized persistent storage of data which is added by algorithmic processing or by users and their email clients.

4.4. Email filtering

Once emails contain structured data about their content, this might create a demand for related extensions to the Sieve email filtering language [RFC5228].

4.5. Email context

Email lives in tight symbiosis with strongly related data such as address books, calendars, or tasks. However, besides specific use cases such as meeting organization, an integration between such data is mainly subject to non-standardized functionality of client software.

Notably, email also operates in a wider context such as services which send transactional emails and also applications running on the same Desktop or Mobile device as the email client.

Transactional emails are sent by services and organizations which are in a relation to the user. Further related data may be managed in outside applications or would be a candidate for being managed within email client software.

From a broader architectural perspective, email can be considered a technology particularly suited to support novel trends in data sovereignty and decentralized data storage, since by design it bridges the public Internet data space with the private data spaces of mailbox users.

Structured email will open new opportunities for exchanging data in all those cases. Accordingly, standards may help to enable and organize interoperability.

4.6. Discovery

In order to allow users to send structured mails, they need to know if and which kind of structured emails a recipient can process. There might be multiple ways of conveying such information:

  • Headers in emails received from users, which a receiver can store/look up
  • A DNS/HTTP-based discovery service, which returns structured email capability for a domain or particular sender

An example for the latter could be:

  • User selects "john@doe.com" as recipient
  • Email client checks if discovery service is offered by "doe.com"
  • Email client queries "doe.com" for structured email capabilities
  • Discovery service at "doe.com" returns a document which states a number of structured data types or actions that are supported by the "john@doe.com" account
  • The email client will allow the user to optionally select from those structured interaction options and provide an appropriate user experience

4.7. HTTP services

For dynamic email features, an endpoint must be available to provide updated data for an email currently read by a user. Similar, if a structured email allows for the execution of a certain action or the submission of form data, there needs to be a receiving endpoint in place. For interoperability reasons, those endpoint APIs should be standardized.

4.8. Trust

Structured and automated interaction certainly raises various security concerns (see Section 5).

While not the focus of this current draft, we assume that parts of the proposed approach can also be used on a special "trust layer". E.g., certain structured message exchanges, in combination with signing or encryption could help to establish trust necessary for further structured communication.

5. Security considerations

While it is clear that trust, security, and anti-spam considerations are core to the technology suggested in this document, this aspect is not discussed in depth at this stage in order to reduce complexity.

While existing approaches such as Email Markup or Actionable Messages enforce strict control by registration processes, we think that a more open process is better suited for the decentralized character of email.

Various trust mechanisms such as DKIM [RFC6376] validation, challenge-response emails, or trusted receivers in address books/white lists might be considered as a prerequisite for structured email processing.

6. IANA Considerations

This document has no IANA actions at this time.

7. Informative References

[AM]
Microsoft Inc., "Actionable Messages", <https://learn.microsoft.com/en-us/outlook/actionable-messages/>.
[AMPemail]
OpenJS Foundation, "AMP email", <https://amp.dev/about/email/>.
[EMarkup]
Google Inc., "Email Markup", <https://developers.google.com/gmail/markup>.
[MailRFCs]
Takizawa, T., "MAIL RFCs", <https://emaillab.jp/mail/mail-rfc/>.
[OutAMP]
Steinbrinck, K., "Outlook is Turning Off AMP for Email", <https://www.emailonacid.com/blog/article/industry-news/outlook-turning-off-amp-for-email/>.
[RFC1036]
Horton, M. and R. Adams, "Standard for interchange of USENET messages", RFC 1036, DOI 10.17487/RFC1036, , <https://www.rfc-editor.org/info/rfc1036>.
[RFC2447]
Dawson, F., Mansour, S., and S. Silverberg, "iCalendar Message-Based Interoperability Protocol (iMIP)", RFC 2447, DOI 10.17487/RFC2447, , <https://www.rfc-editor.org/info/rfc2447>.
[RFC5228]
Guenther, P., Ed. and T. Showalter, Ed., "Sieve: An Email Filtering Language", RFC 5228, DOI 10.17487/RFC5228, , <https://www.rfc-editor.org/info/rfc5228>.
[RFC5321]
Klensin, J., "Simple Mail Transfer Protocol", RFC 5321, DOI 10.17487/RFC5321, , <https://www.rfc-editor.org/info/rfc5321>.
[RFC5322]
Resnick, P., Ed., "Internet Message Format", RFC 5322, DOI 10.17487/RFC5322, , <https://www.rfc-editor.org/info/rfc5322>.
[RFC6376]
Crocker, D., Ed., Hansen, T., Ed., and M. Kucherawy, Ed., "DomainKeys Identified Mail (DKIM) Signatures", STD 76, RFC 6376, DOI 10.17487/RFC6376, , <https://www.rfc-editor.org/info/rfc6376>.
[RFC6477]
Melnikov, A. and G. Lunt, "Registration of Military Message Handling System (MMHS) Header Fields for Use in Internet Mail", RFC 6477, DOI 10.17487/RFC6477, , <https://www.rfc-editor.org/info/rfc6477>.
[RFC6522]
Kucherawy, M., Ed., "The Multipart/Report Media Type for the Reporting of Mail System Administrative Messages", STD 73, RFC 6522, DOI 10.17487/RFC6522, , <https://www.rfc-editor.org/info/rfc6522>.
[RFC6650]
Falk, J. and M. Kucherawy, Ed., "Creation and Use of Email Feedback Reports: An Applicability Statement for the Abuse Reporting Format (ARF)", RFC 6650, DOI 10.17487/RFC6650, , <https://www.rfc-editor.org/info/rfc6650>.
[RFC8058]
Levine, J. and T. Herkula, "Signaling One-Click Functionality for List Email Headers", RFC 8058, DOI 10.17487/RFC8058, , <https://www.rfc-editor.org/info/rfc8058>.
[RFC8098]
Hansen, T., Ed. and A. Melnikov, Ed., "Message Disposition Notification", STD 85, RFC 8098, DOI 10.17487/RFC8098, , <https://www.rfc-editor.org/info/rfc8098>.
[RFC8620]
Jenkins, N. and C. Newman, "The JSON Meta Application Protocol (JMAP)", RFC 8620, DOI 10.17487/RFC8620, , <https://www.rfc-editor.org/info/rfc8620>.
[RFC9051]
Melnikov, A., Ed. and B. Leiba, Ed., "Internet Message Access Protocol (IMAP) - Version 4rev2", RFC 9051, DOI 10.17487/RFC9051, , <https://www.rfc-editor.org/info/rfc9051>.
[RFC9078]
Crocker, D., Signes, R., and N. Freed, "Reaction: Indicating Summary Reaction to a Message", RFC 9078, DOI 10.17487/RFC9078, , <https://www.rfc-editor.org/info/rfc9078>.
[SchemaOrg]
W3C Schema.org Community Group, "Schema.org", <https://schema.org/>.

Author's Address

Hans-Joerg Happel
audriga GmbH