A Secure Protocol for Desktop Web Servers


Abstract

This specification describes a method for applications to receive event notifications using HTTP. An example application of this specification is a news wire service that notifies news publishers when stories have become available.

In terms of transactions, the traditional request/response pair is extended with a third element, a notification. The notification requires a role reversal between client and server, in that it is emitted by the same party that received the original request and received by the same party that emitted the original request. This is done by having the client run a server.

Having clients run servers is a large change in existing HTTP semantics. HTTP expressions are about exposed resources on the server; they leave resources on the client completely opaque. The notification technique attempts to stay close to those semantics -- that is, to avoid exposing resources on the client -- by limiting exposed resources on the client to meanings arranged in a previous request/response. A notification, as described here, is nothing but a flag that an event defined during a previous request/response transaction has occurred.

A callback signifies that an event has occurred which the consumer has previously expressed interest in. There is exactly one such callback notification from server to client for each request for a notification from client to server. The notification does not say what happened, it does not carry any information about the server address to call back to, it does not say whether the event was an error or a success.

Everything of interest in the system is a HTTP resource -- there is a subscription resource, a watched resource, and a notification resource. Events are defined as changes in state of a watched resource. A subscription resource knows how to detect a state change and how to alert interested parties. A notification resource is a URL on the client that understands alerts to mean that the watched resource has a new state.

The use of this technique allows us to rely on existing web technology to the greatest extent possible, and as a result minimizes the security, design and implementation challenges of making HTTP relationships newly symmetrical.

Purpose

This document is part of a broad effort to develop asynchronous protocols and decentralized architectures. It describes a technique to handle just one problem -- event notifications -- in the hope that a really good job on a relatively approachable task will provide reusable solutions to harder problems such as store-and-forward and updates pushed from server to client. A solution to event notifications can, hopefully, be used as a wedge to answer the two broad questions:

  1. How can HTTP be used for asynchronous messaging?
  2. How can HTTP clients and servers be combined?
Most attempts to do the first use HTTP as a transport for a new, non-HTTP, protocol like SOAP or XML-RPC. Most attempts to do the second do an incomplete job, and end up creating as many problems as they solve. The approach taken here is of strict compliance with HTTP on both a syntactic and semantic level, and of restricting the functionality to a small enough subset to do a thorough job.

It is not a purpose of this document to provide normative requirements. The language of standards is used only because it is a good tool for describing protocols. This memo defines an experimental protocol for the Internet community. This memo does not specify an Internet standard of any kind. Discussion and suggestions for improvement are requested. Distribution of this memo is unlimited.

Format

This document is structured as a recipe -- a clear, unambiguous and carefully debugged set of instructions for building notification services. In developing concise instructions a number of more abstract issues came up, such as the security implication of bundling servers with clients; these are developed in prose subsections.

A number of useful features were left out because they required subscriptions that persisted beyond a single transaction. This version of this document does not support any kind of subscription management, not because there is no need, but because subscription management capabilities of any kind dramatically increase the level of complexity. The technique described in this document does not support any ongoing relationship between sink and source beyond a single subscription and any following notification.

The technique described in this document uses mainly HTTP header fields to communicate input and output parameters, as well as HTTP status codes. The primary benefit is to leave the entity body open for future extension and to avoid creating a set of standardized URIs (for example the use of {$root_url}/request_parameter_name to communicate success or failure).


Terminology

Application Status Code: A status code pertaining to application-level status. Distinct from HTTP status code.

Checkpoint: A specific point in the stream of state changes of the watched resource.

Event: A change in the state of the watched resource. A stream of events is a stream of distinguishable states.

MUST / SHOULD / MAY / MUST NOT / SHOULD NOT / MAY NOT: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Notification Resource: The resource identified by the Notification-URI at the sink.

Resource: A network data object or service that can be identified by a URI, as defined in section 3.2. Resources may be available in multiple representations (e.g. multiple languages, data formats, size, and resolutions) or vary in other ways. See RFC 2396.

Sink: When a server performs a callback to a client that has expressed interest in an event, the client is acting as an event sink. This language is used because having servers and clients use callbacks from server to client causes a role reversal, and hence the term "client" is ambiguous. In the case of this specification the term "sink" is equivalent to "the original client."

Source: When a server performs a callback to a client that has expressed interest in an event, the server is acting as an event source. This language is used because having servers and clients use callbacks from server to client causes a role reversal, and hence the term "server" is ambiguous. In the case of this specification the term "source" is equivalent to "the original server."

Subscription Resource: The server resource for creating new subscriptions.

Subscription Management Resource: The server resource for managing existing subscriptions.

Watched Resource: The sink is interested in changes in the watched resource; the source tracks changes in the watched resource. The resource itself is known by its URI. The watched resource may or may not have a direct relationship with the subscription resource.


Usecases

  1. Push notifications improve on POP3 and IMAP by reducing the lag time between polling cycles and at the same time reduce the server overhead of processing unnecessary polls.
  2. Push notifications improve on spidering search engines by reducing the lag time between polling cycles and at the same time reduce the server overhead of processing unnecessary polls.
  3. Push notifications allow a client to watch many more resources than possible with polling loops, because all the polling loops that would not have resulted in finding a new state are eliminated.
  4. Push notifications improve the scalability of messaging servers such as Microsoft Exchange by granting the server control over the scheduling of client requests. While polling clients decide for themselves when they will contact the messaging server, and can accidentally flood the server by polling at the same time, a notifying server can perform load balancing by spreading notifications over time.
  5. Push notifications enable peer-to-peer applications to implement semi-symetrical relationships, as happens naturally when clients and servers are joined, but retain the benefits of the HTTP protocol.
  6. Idempotent notifications as defined here allow the client greater security than with push updates, and allow the client to continue using existing web infrastructure to fetch updates.

Transaction Outline

General Outline

  1. Client to server: request a subscription to changes in the watched resource. (Subscription).
  2. Client to watched resource: request the current state of the watched resource. (Fetch).
  3. Time passes: the server tracks the state of the watched resource until there is a change.
  4. Source to sink: send notification that the resource has changed. (Notification).
  5. Repeat.

Detailed Outline

Step 1. Client to server: request a subscription to changes in the watched resource.
Step 2. Client to watched resource: request the current state of the resource.
Step 3. Time passes.
Step 4. Source to sink: send notification that the watched resource has changed.
Step 5. Repeat.

Design Notes

Race Conditions

Events that happen between the time a client initiates a fetch and the time the fetch is complete might fail to cause a usable notification to be received on the client. This is only a problem for a client which intends to continue tracking changes in the watched resource after the initial notification; other clients can ignore the following.

After reception of a Step 4 (Notification), the client MUST perform a Step 1 (Subscription) before the Step 2 (Fetch). The purpose is to allow the server to checkpoint the stream of state changes of the watched resource, and therefore to know whether there are any states which have not caused a Step 4 (Notification).

Third Party Watchers

It is permissible for an event source that receives Step 1 (Subscription) requests and emits Step 4 (Notification) requests to be a third party with regard to the watched resource. One example is a spider that notices state changes and emits notifications to interested parties. Another example is a sink which receives notifications directly from the owner of the watched resource and passes them on, switching roles to act as a source. Another example is a CGI script on the same host and under the same ownership as the watched resource, but which has no privileged relationship with the watched resource.

Though the Step 1 (Subscription) MAY have side effects on the Step 2 (Fetch), the client MUST NOT assume a relationship between the subscription resource and the watched resource. As far as the client is concerned, the subscription resource and the watched resource are always third parties.

Polling for document changes is not a primary usecase of the technique described in this document. It is accounted for only because it is inevitable.

Multistate Resources

Watched resources will pass through a series of states, and watchers that use the technique described in this document will often be interested in all of those states. However, accumulating a collection of states into a single coherent resource is application defined. There is no way to do this generically, and the technique described here does not attempt it. Rather, this document takes the position that existing tools for accumulating states merely need to be enabled.

CVS, for example, is a tool for accumulating states. It manages successive states of a collection of source files. It presents a view of the current state of the collection, recordings of previous states of the collection, and the ability to branch states. Using the language of this document, the collection is a watched resource and the entire version history is a multistate resource. It is evident from the complexity of CVS that the business of managing versions is a difficult and ambitious task that deserves dedicated tools to be done adequately.

Email infrastructure, including SMTP, POP3, and mailbox files, is another tool for accumulating and managing stateful resources. An email message is a new resource. A mailbox file is a container for resources. POP3 is a tool for navigating a resource container. The relevant similarity between email, CVS and the concerns of this document is that every email message is a write-once resource, every version of a file in CVS is a write-once resource, and every state of a watched resource is a write-once resource. In each case there is an application-defined wrapper around a collection of write-once states.

The technique described in this document is for managing a single state resource. The relevant state of the resource is the "current" one. Step 1 (Subscription) marks the current state of the resource. That mark in the current state of the resource is a "checkpoint". A checkpoint remains the definition of "current" until the next instance of a Step 1 (Subscription). Step 2 (Fetch) may be relative to that checkpoint (if there are side effects of the POST on the watched resource), which is a supporting reason that Step 1 (Subscription) must always precede, not follow, Step 2 (Fetch).

The ability to accumulate a bunch of states in a coherent way is left to specialist applications like CVS and email. This document aims to enable, not replace, applications like CVS and email.


Error Handling

Offline Error Handling

Errors may occur as a result of Step 1 (Subscription) that are not foreseeable while it is ongoing. However, managing them would require a form of subscription management, because the source needs to cache them for later pickup. Therefore the technique described in this document simply lets all such errors fall on the floor. If the sink detects a problem with the watched resource, it MUST consider it a state change like any other and emit a Step 4 (Notification) according to normal procedure.

The worst such offline error is that a notification resource would be invalid, for example if it and the subscription resource were separated by a firewall. It is expected that the sink will guarantee the integrity of the callback URI before creating the subscription. This may be handled by having the sink call a web service that tests the sink IP. If the web service were hosted by the notification source then it would be able to determine with certainty whether the sink callback was reachable from its perspective. Such a web service is fairly trivial to imagine but is not within the scope of this document.

For the source, the integrity of the callback must be assumed. Because it is not possible for a source to know whether an unreachable sink has gone missing permanently or temporarily, a source is obligated to maintain the subscription until the expiration date stated in the Step 1 (Subscription) Subscription-Expiration response header.

Application Status Codes

It is not yet clear whether the subscription resource should return application-specific status codes as extensions to the existing RFC 2616 codes or as separate response entity elements. The following is a strawman definition using a separate header in the response.

[PROPOSED]

An application can return status codes in response headers titled "Resource-Status-Code." There can be 0 or more such response headers. The value of a Resource-Status-Code field MUST contain a numeric status code, a space, and a textual explanation. Whether the subscription request was a success or failure depends on the specific Resource-Status-Code; if a failure the sink MUST NOT create the notification resource.

Example response message:

HTTP/1.1 400 Bad Request
Resource-Status-Code: 1.0 CALLBACK URI SYNTAX

Resource-Status-Code values:

Numeric code Reason phrase Explanation
1.0 CALLBACK URI SYNTAX The server knows in advance that it cannot parse the callback URI, for example it might be a malformed HTTP URL. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.
1.1 CALLBACK URI UNREACHABLE The server knows in advance that it cannot establish a network connection to the callback URI, for example because of a firewall. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.
1.2 CALLBACK URI UNSUPPORTED The server can parse the callback URI but not support it fully, for example it recognizes a mailto URI but does not know how to support it. This would be thrown by the subscription resource to flag an error in the ongoing request. The sink MUST NOT create the notification resource.


Security Considerations

Threats to the Client/Sink

There are a large number of potential security holes created by having clients become servers. The design used in this specification attempts to meet the following bar: a sink should be no less secure than if it were strictly a client. This is done by implementing as little sink functionality as possible within the callback, the Step 4 (Notification). Instead, as much as possible is done during the Step 1 (Subscription), where the client is no less secure than it ever was.

Attackers are prevented from altering the meaning of a Step 4 (Notification) by the following means: the only data transmitted in the Step 4 (Notification) is the fact that a pre-arranged callback URI has been activated. The meaning of that URI is left entirely up to the sink, which generates it, and to any information exchanged during a Step 1 (Subscription).

Attackers are prevented from tricking a sink into believing that a Step 4 (Notification) has happened by the following means: the callback URI itself is a secret until the source invokes it in the Step 4 (Notification), the callback URI MUST be hard to guess (for example, by being a 50 character random string), and the callback URI is never used more than once.

Attackers are prevented from creating buffer overflows by the following means: the only HTTP server functionality which a sink MUST implement is the ability to read a PUT request at a valid URI. A sink MAY disconnect a request to any invalid URI, which can be detected within the first line of a request, and a sink MAY limit the maximum length of the Request-line. As a result attackers must detect and block Step 4 (Notification) requests before they can attempt a buffer overflow attack, and they must do so before any retry following an alternate path reaches the client.

(A fringe benefit of this approach is that it is presently easier to use client/server interactions, which are well-known and mature, than source/sink interactions, which are brand new and immature .)

Threats to the Server/Source

A unique threat to the server caused by the technique described within this document is that it may be tricked into accessing resources that are off limits. For example a watched resource may have posted legal notices that it intends to sue anybody that accesses it, regardless of whether such a suit is right or winnable. This is only a best-guess means for the server to protect itself. This is not legal advice. The author is a programmer, not a lawyer. Comments from legal professionals would be welcome. The server is protected by:

Threats to the Watched Resource

If the subscription resource executes the watch using a polling loop, it may generate more traffic at the watched resource than it would otherwise incur. This is less of a problem than you might expect, though. When clients use the subscription resource instead of performing a poll themselves, the subscription resource is aggregating their requests and thus reducing the total number of pollers.

Enabling Bot Exclusion

Polling is not a primary usecase for the technique described in the document, but it is inevitable, and some resources should not polled. A watched resource may be a GET query that improperly has side effects, it may have business reasons to block polling, it may be a very large query that was inadvertently exposed to the public.

A Standard for Robot Exclusion exists for similar reasons, but is not quite applicable to watchers. It exists to block bots that retrieve the same files repeatedly, as a watcher is supposed to do, or that block parts of web servers that aren't suitable for spidering (such as very deep virtual trees and duplicated information) but that are suitable for watching.

If threats to watched resources create a sufficient need, a file with the same format as the Robot Exclusion Standard, but with a new name, is a simple solution. An appropropriate new name is "watchers.txt."

To reduce load on the watcher caused by polling, a watcher which performs a polling loop MUST use the HEAD method rather than the GET method.

Enabling Bot Inclusion

A watched resource may not object to being watched, but may prefer to use a specific watcher with which it enjoys a privileged relationship. For example, a local watcher may do polling via a filesystem stat(...) call to avoid the overhead of web transactions, or it may be attached to a database trigger. Potential approaches to this extension are:

  1. Watched resources MAY return a "Subscription-Resource" header to identify a resource capable of acting as a server to a Step 1 (Subscription) as defined in this document. Advantages: may be implemented on either a server-wide basis as a server extension or on a very granular basis as a per-resource extension.
  2. Watched resources MAY implement a file that follows the syntax of robots.txt but has a new name, for example "watchers.txt", and that implements the following extension: "Subscription-Resource: URI". A watcher that finds such an entry in the watchers.txt file MUST chain to it as a client to a Step 1 (Subscription).
Finally, a watcher MUST respect the value of any Retry-After header returned by the watched resource, regardless of whether the watched resource returns a 503 or 3xx status code.

The discussion in this design note is somewhat irrelevant until the rest of the mechanisms work well and have reached some amount of adoption, and thus can be safely left to a late state of development.

Threats to Third Parties

Since the technique described in this document induces bots at the source and sink to emit queries, it is conceivable that bots could be manipulated into participating in a distributed denial of service (DDoS) attack against a third party.

An attacker could modify any of the URIs transmitted and later invoked, such as the sink callback URI or the watched resource URI. However, the one-shot nature of event transmissions as described in this document means that the attacker would have to intercept and modify one message for every message emitted in the DDoS attack, which nullifies the attacker's advantage. (Thanks to Paul Prscod (private communication) for this point.)

An attacker could initiate a large number of watches against a target. If the attacker used a single watcher, then the watcher would actually aggregate all the queries into a single one and in effect reduce the number of queries received by the target.


Appendix: Email Binding

The format of the Step 4 (Notification) is simple enough to be easily adapted to email, and difficulties in establishing HTTP back channels to edge nodes make this a very tempting approach. Though email is a highly flawed medium for application messaging it is likely to be inevitable, and so a simple binding outlined here may help keep implementions in the best possible condition.

A mailto URI MAY be passed to the Step 1 (Subscription) resource as the sink callback URI. Such a URI MUST be compliant with RFC 2368 (The mailto URL scheme). A source MUST assume that a sent Step 4 (Notification) message emitted via email has not been received unless it receives a return receipt compliant with RFC 2298 (Message Disposition Notifications).

The sink is responsible for generating a mailto URI that is compliant with the other requirements of this document, for example by creating a unique and difficult to guess identifier for each request to the Step 1 (Subscription) resource.

It is expected that a mailto sink URI will be received by a bot listener, not a human user agent.

DSNs and MDNs are defined in the following IETF RFCs:

Acknowledgments and Related Work

The architectural style underlying this document is a combination of Event-Based Integration and Representational State Transfer.

This document has benefitted from conversation on the rest-discuss  mailing list, specifically including comments by Jeff Bone, Mark Baker, Paul Prescod, and Mike Dierken. Related background includes the formal documents RFC 2616, Roy Fielding's dissertation, and Notification for Proxy Caches; as well as the informal documents aREST -- Asynchronous REST, Yet Another HTTP Event Architecture (YAHEA), Http Events, the REST Wiki, HttpEvents as currently contemplated are extraneous?, and a proposal for secure notifications.

The rssCloud interface is very similar to the protocol described here. On a superficial level it is different in that it uses XML-RPC and only allows notifications for RSS files. On a design level the difference is that the notification carries metadata about which of a group of watched resources has changed state. In the language used in this document, rssCloud is an update protocol. The notification protocol described here is intended as a basis for a yet to be defined update protocol.


Adminstrative Information

Author: Lucas Gonze.
Contact:
lucas@gonze.com
Document created: February 8, 2002.
Last update: September 26, 2003.
Address of this document on the web: http://www.gonze.com/http-notifications.html