Path: utzoo!attcan!uunet!lll-winken!lll-tis!helios.ee.lbl.gov!nosc!ucsd!brian From: brian@ucsd.EDU (Brian Kantor) Newsgroups: comp.doc Subject: RFC1056 - PCMAIL: Distributed Mail for Personal Computers (2 of 2) Message-ID: <1141@ucsd.EDU> Date: 10 Sep 88 18:43:45 GMT Distribution: na Organization: The Avant-Garde of the Now, Ltd. Lines: 1119 Approved: brian@cyberpunk.ucsd.edu RFC 1056 PCMAIL June 1988 incoming mail, or in the form of changes from other clients. A client that is not always connected to the net cannot immediately receive the global changes. In addition, the client's user can make his own changes on the local mail state. Pcmail's architecture allows fast synchronization between client local mail states and the repository's global mail state. Each client is identified in the repository by a client object attached to the user. This object forms the basis for synchronization between local and global mail states. Some of the less common state changes include the adding and deleting of user mailboxes and the adding and deleting of address objects. Synchronization of these changes is performed via DMSP list operations, which allow clients to compare their local versions of mailbox and address object lists with the repository's global version and make any appropriate changes. The majority of possible changes to a user's mail state are in the form of changed descriptors. Since most users will have a large number of messages, and message states will change relatively often, special attention needs to be paid to message synchronization. An existing descriptor can be changed in one of three ways: first, one of its sixteen flag values can be changed (this encompasses the user's reading an unseen message, deleting a message, printing a message, etc). Second, a descriptor can be created, either by the delivery of a new message or by the copying of a message from one mailbox to another. Finally, a descriptor can be destroyed, via an "expunge-mailbox" operation. In the above cases, synchronization is required between the repository and every client that has not previously noted the change. To keep track of which clients have noticed a global mail state change and changed their local states accordingly, each mailbox has associated with it a list of active clients. Each client has a (potentially empty) "update list" of messages which have changed since that client last synchronized. When a client connects to the repository, it executes a DMSP "fetch- changed-descriptors" operation. This causes the repository to return a list of all descriptors on that client's update list. When the client receives the changed descriptors, it may do one of two things: if the descriptor is marked "expunged", it can remove the corresponding message from the local mailbox. If the descriptor is not expunged, the client can store the descriptor, thus updating the local mail state. After a changed descriptor has been recorded, the client uses the DMSP "reset-descriptors" operation to remove descriptors from its update list. Those descriptors will now not be sent to the client unless (1) it is explicitly requested via a "fetch-descriptors" operation, or (2) it changes again. Lambert [Page 19] RFC 1056 PCMAIL June 1988 In this manner, a client can run through its user's mailboxes, getting all changes, incorporating them into the local mail state, and marking the changes as recorded. 5.3. Batch operation versus interactive operation Because of the portable nature of some workstations, they may not always be connected to a network (and able to communicate with the repository). Since each client maintains a local mail state, Pcmail users can manipulate the local state while not connected to the repository. This is known as "batch" operation, since all changes are recorded by the client and made to the repository's global state in a batch, when the client next connects to the repository. Interactive operation occurs when a client is always connected to the repository. In interactive mode, changes made to the local mail state are also immediately made to the global state via DMSP operations. In batch mode, interaction between client and repository takes the following form: the client connects to the repository and sends over all the changes made by the user to the local mail state. The repository changes its global mail state accordingly. When all changes have been processed, the client begins synchronization; this incorporates newly-arrived mail, as well as mail state changes by other clients, into the local state. In interactive mode, since local changes are immediately propagated to the repository, the first part of batch-type operation is eliminated. The synchronization process also changes; although one synchronization is required when the client first opens a connection to the repository, subsequent synchronizations can be performed either at the user's request or automatically every so often by the client. 5.4. Message summaries Smaller workstations may have little in the way of disk storage. Clients running on these workstations may never have enough room for a complete local copy of a user's global mail state. This means that Pcmail's client architecture must allow user's to obtain a clear picture of their mail state without having all their messages present. Descriptors provide message information without taking up large amounts of storage. Each descriptor contains a summary of information on a message. This information includes the message UID, its length in bytes and lines, its status (contained in the eight system-defined and eight user-defined flags), and portions of its Lambert [Page 20] RFC 1056 PCMAIL June 1988 RFC-822 header (the "from:", "to:", "date:" and "subject:" fields). All of this information can be encoded in a small (around 100 bytes) data structure whose length is independent of the size of the message it describes. Most clients should be able to store a complete list of message descriptors with little problem. This allows a user to get a complete picture of his mail state without having all his messages present locally. If a client has extremely limited amounts of disk storage, it is also possible to get a subset of the descriptors from the repository. Short messages can reside on the client, along with the descriptors, and long messages can either be printed via the DMSP print-message operation, or specially pulled over via the fetch- message operation. 6. Typical interactive-style client-repository interaction The following example describes a typical communication session between the repository and a client mail reader. The client is one of three belonging to user "Fred". Its name is "office-client", and since Fred has used the client within the last week, it is marked as "active". Fred has two mailboxes: "fred" is where all of his current mail is stored; "archive" is where messages of lasting importance are kept. The example will run through a simple synchronization operation. Typically, the synchronization will be performed by a mail reader as part of a "get new mail" operation. First Fred's mail reader connects to the repository and receives the following banner: 200 Pcmail repository version 3.0.0 ready In order to access his global mail state, the mail reader must authenticate Fred to the repository; this is done via the DMSP login operation: login fred fred-password office-client 0 0 This tells the repository that Fred is logging in via "office- client", and that "office-client" is identified by an existing client object in Fred's mail state. The first argument to the login operation is Fred's repository user name. The second argument is Fred's password. The third argument is the name of the client communicating with the repository. The fourth argument tells the repository not to create "office-client" even if it cannot find its client object. The final argument tells the repository that Fred's client is not operating in batch mode but rather in interactive mode. Lambert [Page 21] RFC 1056 PCMAIL June 1988 Fred's authentication checks out, so the repository logs him in. 200 command OK Now that Fred is logged in, the mail reader performs an initial synchronization. This process starts with the mail reader's asking for an up-to-date list of mailboxes: list-mailboxes The repository replies with: 230 mailbox list follows: fred 2313 10 1 archive 101 100 0 . This tells the mail reader that there are two mailboxes, "fred" and "archive". "Fred" has 10 messages, one of which is unseen. The next incoming message will be assigned a UID of 2313. "Archive", on the other hand, has 100 messages, none of which are unseen. The next message sent to "archive" will be assigned the UID 101. There are no new mailboxes in the list (if there were, the mail reader would create them. On the other hand, if some mailboxes in the mail reader's local list were not in the repository's list, the program would assume them deleted by another client and delete them locally as well). To synchronize, the mail reader need only look at each mailbox's contents to see if (1) any new mail has arrived, or (2) if Fred changed any messages on one of his other two clients subsequent to "office-client"'s last connection to the repository. The mail reader asks for any changed descriptors via the "fetch- changed-descriptors" operation. It requests at most ten changed descriptors since storage is very limited on Fred's workstation. fetch-changed-descriptors fred 10 The repository responds with: 250 descriptor list follows: expunged Lambert [Page 22] RFC 1056 PCMAIL June 1988 2101 expunged 2104 descriptor 2107 1100011100000010 1400 30 foo@bar.edu (Foo Jones) fred@PTT.LCS.MIT.EDU Wed, 9 Dec 87 10:43:52 EST A typical subject line descriptor 2312 0000000000000000 12232 320 joe@athena.mit.edu fred@PTT.LCS.MIT.EDU Thu, 17 Dec 87 18:24:09 PST Another typical subject line . If a descriptor changed because it was expunged, it is transmitted as two lines: the word "expunged" on one line, followed by the message UID on the next line. If one of its flags changed state, or it is a new message, it is transmitted as six lines: the word "descriptor" on one line, followed by a line containing the message UID, flags, and length in bytes and lines, followed by the to, from, date, and subject fields, each on one line. The flags are transmitted as a single string of ones and zeroes, a one if the flag is on and a zero if the flag is off. All 16 flags are always transmitted. Flag zero's state is the first character in the flag string; flag fifteen's is the last character in the flag string. The first two descriptors in the list have been expunged, presumably by Fred's expunging his mailbox on another client. The mail reader removes messages 2101 and 2104 from its local copy of mailbox "fred". The next descriptor in the list is one which Fred marked for deletion on another client yesterday. The mail reader marks the local version of the message as deleted. The last descriptor in the list is a new one. The mail reader adds the descriptor to its local list. Since all changes to mailbox "fred" have now been recorded locally, the update list can be reset: reset-descriptors fred 1 2312 The repository responds with: 200 command OK indicating that it has removed from "office-client"'s update list all Lambert [Page 23] RFC 1056 PCMAIL June 1988 messages in mailbox "fred" with UIDs between 1 and 2312 inclusive (in this case just two messages). "Fred" has now been synchronized. The mail reader now turns to Fred's "archive" mailbox and asks for the first ten changed descriptors. fetch-changed-descriptors archive 10 The repository responds with: 250 descriptor list follows: . The zero-length list tells the mail reader that no descriptors have been changed in "archive" since its last synchronization. No new synchronization needs to be performed. Fred's mail reader is now ready to pull over the new message. The message is 320 lines long; there might not be sufficient storage on "office-client" to hold the new message. The mail reader tries anyway: fetch-message fred 2312 The repository begins transmitting the message: 251 message follows: UID: 2312 From: joe@bar.mit.edu To: fred@PTT.LCS.MIT.EDU Date: Thu, 17 Dec 87 18:24:09 PST Subject: Another typical subject line Fred, ... Halfway through the message transmission, Fred's workstation runs out of disk space. Because all DMSP operations are defined to be failure-atomic, the portion of the message already transmitted is destroyed locally and the operation fails. The mail reader informs Fred that the message cannot be pulled over because of a lack of disk space. The synchronization process is now finished and Fred can start reading his mail. The new message that was too big to fit on "office-client" will be marked "off line"; Fred can use the mail Lambert [Page 24] RFC 1056 PCMAIL June 1988 reader to either remote-print it or delete and expunge other messages until he has enough space to store the new message. Since Fred is running in interactive mode, changes he makes to any messages will immediately be transmitted into DMSP operations and sent to the repository. Depending on the mail reader implementation, Fred will either have to execute a "synchronize" command periodically or the client will synchronize for him automatically every so often. 7. A current Pcmail implementation The following section briefly describes a current Pcmail system that services a small community of users. The Pcmail repository runs under UNIX on a DEC Microvax-II connected to the Internet. The clients run on IBM PCs, XTs, and ATs, as well as Sun workstations, Microvaxes, and VAX-750s. 7.1. IBM PC client code Client code for the IBM machines operates only in batch mode. Users make local state changes in a mail reader; the changes are queued until the user runs a network client program. The program connects to the repository, performs the queued changes, and synchronizes local and global mail states. The network client program then disconnects from the repository. The IBM PC client code has gone through several revisions since the first Pcmail RFC was published. What was once a fairly primitive and cumbersome system has evolved into a system that makes excellent use of the PC's limited resources and provides a fairly powerful, easy- to-use mail reader. Users access and modify their local mail state via a mail reader written in the Epsilon text editor's EEL extension language. Users are given a variety of commands to operate on individual messages and mailboxes, as well as to compose outgoing mail. Synchronization and the processing of queued changes is performed by a separate program, which the user runs as desired. The program takes any actions queued while operating the mail reader, and converts them into DMSP operations. All queued changes are made before any synchronization is performed. The program can be invoked directly from the mail reader, without having to exit and restart. The limitation of IBM PC client operation to batch mode was made because of development environment limitations. The mail reader cannot work with the network code inside it because of the network program architecture. The only solution was to provide a two-part Lambert [Page 25] RFC 1056 PCMAIL June 1988 client, one part of which read the mail and one part of which interacted with the repository. Although slightly cumbersome, the two-program setup works quite well. 7.2. UNIX client code Client code for the Suns, Microvaxes, and VAX-750s runs on 4.2/4.3BSD UNIX. It is fully interactive, with a powerful mail reader inside Richard Stallman's GNU-EMACS editor. Since UNIX-based workstations have a good deal of main memory and disk storage, no effort was made to lower local mail state size by keeping message descriptors rather than message text. The local mail state consists of a number of BABYL-format mailboxes. The interface is very similar to the RMAIL mail reader already present in GNU-EMACS. The mail reader communicates with the repository through network code implemented in EMACS-LISP. Changes to the local mail state are immediately made on the repository; although the repository is fast, there is a small noticeable delay in performing operations over the network. There is no provision for automatic synchronization whenever new mail arrives or old mail is changed by another client. Instead, users must get any new mail explicitly. A simple "notification" program runs in the background and wakes up every minute to check for new mail; when mail arrives, the user executes a command to get the new mail, synchronizing the mailbox at the same time. 7.3. Repository code The repository is implemented in C on 4.2/4.3BSD UNIX. Currently it runs on DEC VAX-750s and Microvaxes, although other repositories will soon be running on IBM RT machines and Sun workstations. The repository code is designed to allow several clients belonging to a particular user to "concurrently" modify the user's state. A locking scheme prevents one client from modifying mail state while another client is modifying the same state. 8. Conclusions Pcmail is now used by a small community of people at the MIT Laboratory for Computer Science. The repository design works well, providing an efficient means of storing and maintaining mail state for several users. Its performance is quite good when up to ten users are connected; it remains to be seen whether or not the repository will be efficient at managing the state of ten or a Lambert [Page 26] RFC 1056 PCMAIL June 1988 hundred times that many users. Given sufficient disk storage, it should be able to, since communication between different users' clients and the repository is likely to be very asynchronous and likely to occur in short bursts with long "quiet intervals" in between as users are busy doing other things. Members of another research group at LCS are currently working on a replicated, scalable version of the repository designed to support a very large community of users with high availability. This repository also uses DMSP and has successfully communicated with clients that use the current repository implementation. DMSP therefore seems to be usable over several flavors of repository design. The IBM PC clients are very limited in the way of resources. The mail reader/editor combination is quite powerful, making local mail state manipulation fairly easy. Obviously a big performance enhancement would be to provide a fully interactive client. As it is, batch-style synchronization is relatively time consuming due to the low performance of the PCs. The "batch-mode" that the PCs use tends to be good for those PCs that spend a large percentage of their time unplugged and away from a network. It is somewhat inconvenient for those PCs that are always connected to a network and could make good use of an "interactive-mode" state manipulation. The UNIX-based clients are more powerful and easier to use than their PC counterparts. Synchronization is much faster, and there is far more functionality in the mail reader (having an interface that runs within GNU-EMACS helps a lot in this respect). Most of those people using the Pcmail system use the UNIX-based client code. Lambert [Page 27] RFC 1056 PCMAIL June 1988 I. DMSP Protocol Specification Following are a list of DMSP operations by object type, together with syntax, and possible responses. Some responses may be followed by zero or more lines of text, terminated by a single period plus CR-LF pair. Only success responses and common error responses are listed; a complete list of possible responses follows this appendix. Expressions in angle brackets (i.e. ) are metalinguistic variables indicating a general request or response form. Operations with arguments have a sample invocation following the operation syntax and response. General operations: HELP 100 Repository version xxx. Following are supported: HELP SEND-VERSION SEND-MESSAGE LOGIN LOGOUT ... FETCH-MESSAGE COPY-MESSAGE . SEND-VERSION 200 Command OK 500 version skew! i.e. SEND-VERSION 230 SEND-MESSAGE 350 enter message; end with "." To: markl From: markl Subject: a test message this is a test message . Lambert [Page 28] RFC 1056 PCMAIL June 1988 Repository responds: 200 Command OK 403 message syntax error User operations: LOGIN 200 Command OK 221 Client out of date by > 1 week 404 Bad password 405 Client is locked 411 No user named 421 Client not found i.e. LOGIN markl foo random-client-name 1 0 LOGOUT 200 Command OK SET-PASSWORD 200 Command OK 404 Incorrect old password i.e. SET-PASSWORD foo bar Client operations: LIST-CLIENTS 220 Client list follows: client-1 active client-2 inactive client-3 active ... client-foobar active . Each line of the list contains a client name, followed by whitespace, followed by the word "active" or the word "inactive", indicating whether or not the client has connected to the repository within the last week. Lambert [Page 29] RFC 1056 PCMAIL June 1988 CREATE-CLIENT 200 Command OK 403 is an illegal name 420 Client exists i.e. CREATE-CLIENT new-client DELETE-CLIENT 200 Command OK 421 Client not found 405 Client is locked i.e. DELETE-CLIENT old-client RESET-CLIENT 200 Command OK 421 Client not found 405 Client is locked i.e. RESET-CLIENT any-old-client Mailbox operations: LIST-MAILBOXES 230 Mbox list <#msgs> <#new> follows: mailbox-1 2338 8 1 mailbox-2 59 44 0 ... mailbox-foobar 19 9 0 . Each line of the list contains a mailbox name, followed by the mailbox's next available unique identifier, followed by the number of messages in the mailbox, followed finally by the number of unseen messages in the mailbox. Unseen messages are those whose descriptors have flag #1 ("message has been seen") set to zero. CREATE-MAILBOX 200 Command OK 403 is an illegal name 430 already exists 440 exists as a bboard subscription Lambert [Page 30] RFC 1056 PCMAIL June 1988 i.e. CREATE-MAILBOX current-events DELETE-MAILBOX 200 Command OK 431 mailbox not found 440 is a bboard; use delete-bboard-mailbox i.e. DELETE-MAILBOX income-tax-information CREATE-BBOARD-MAILBOX 200 Command OK 430 a mailbox named already exists. 430 a bboard mailbox named already exists. 403 is an illegal name i.e. CREATE-BBOARD-MAILBOX sf-lovers DELETE-BBOARD-MAILBOX 200 Command OK 404 not owner of 431 no bboard mailbox named i.e. DELETE-BBOARD-MAILBOX rec.autos RESET-MAILBOX 200 Command OK 431 mailbox not found i.e. RESET-MAILBOX british-cars EXPUNGE-MAILBOX 200 Command OK 431 mailbox not found EXPUNGE-MAILBOX british-cars Address operations: LIST-ADDRESSES 260 Address list for follows: address-1 Lambert [Page 31] RFC 1056 PCMAIL June 1988 address-2 ... address-6 . or 431 mailbox not found i.e. LIST-ADDRESSES archive Each line of the list consists solely of one address. CREATE-ADDRESS 200 Command OK 403 is an illegal name 431 mailbox not found 460 already exists i.e. CREATE-ADDRESS markl markl-bug-pcmail DELETE-ADDRESS 200 Command OK 431 mailbox not found 461 address not found i.e. DELETE-ADDRESS markl markl-info-cobol Subscription operations: LIST-SUBSCRIPTIONS 240 subscription list follows: bboard-1 2573 33 2606 bboard-2 541 4 545 ... bboard-6 1530 43 1573 . Each line of the list consists of a bulletin-board name, followed by the UID of the first message which the user has not yet looked at, followed by the number of messages in the bulletin-board that the user has not yet looked at, followed by the bulletin-board's next available unique message identifier. Lambert [Page 32] RFC 1056 PCMAIL June 1988 CREATE-SUBSCRIPTION 200 Command OK 403 is an illegal name 430 A mailbox named already exists 431 Bboard mailbox not found 440 Already subscribing to i.e. CREATE-SUBSCRIPTION sf-lovers DELETE-SUBSCRIPTION 200 Command OK 441 Subscription not found i.e. DELETE-SUBSCRIPTION rec.music RESET-SUBSCRIPTION 200 Command OK 441 Subscription not found i.e. RESET-SUBSCRIPTION rec.music.gdead 1210 LIST-AVAILABLE-SUBSCRIPTIONS 241 All available bboards follow: mod.politics sfl tcp-ip forum ... comp.emacs . Each line of the list consists solely of one bulletin-board name. Message operations: FETCH-CHANGED-DESCRIPTORS 250 Descriptor list follows: expunged 2333 expunged 2334 Lambert [Page 33] RFC 1056 PCMAIL June 1988 descriptor 2337 0001000001110000 481 14 croaker@ptt.lcs.mit.edu fred@anymachine.mit.edu Tue, 19 Jan 88 11:10:03 EST a typical subject line descriptor 2339 0000000000000000 1457 40 bob@lcs.mit.edu csr-people@ptt.lcs.mit.edu Mon, 18 Jan 88 13:08:17 +0000 another typical subject line expunged 2340 . or 431 mailbox not found i.e. FETCH-CHANGED-DESCRIPTORS markl 100 Each element of the descriptor list is either two or six lines long. Descriptors which have been expunged are transmitted as two lines: the word "expunged" on one line, followed by the message unique identifier on the next line. Descriptors which still exist are transmitted as six lines: the word "descriptor" on one line, followed by a line containing the message unique identifier, flag states (sixteen characters either one or zero depending on the associated flag value), followed by the message length in characters, followed by the message length in lines. The next four lines contain the message's "from:", "to:", "date:", and "subject:" fields, respectively. Flag zero's state is the first character in the flag string; flag fifteen's is the last character in the flag string. FETCH-DESCRIPTORS 250 Descriptor list follows: descriptor 2337 0001000001110000 481 14 croaker@ptt.lcs.mit.edu fred@anymachine.mit.edu Tue, 19 Jan 88 11:10:03 EST a typical subject line descriptor 2339 0000000000000000 1457 40 bob@lcs.mit.edu csr-people@ptt.lcs.mit.edu Lambert [Page 34] RFC 1056 PCMAIL June 1988 Mon, 18 Jan 88 13:08:17 +0000 another typical subject line . or 431 mailbox not found i.e. FETCH-DESCRIPTORS british-cars 12 31 COPY-MESSAGE 250 Descriptor list follows: descriptor 2339 0000000000000000 1457 40 bob@lcs.mit.edu csr-people@ptt.lcs.mit.edu Mon, 18 Jan 88 13:08:17 +0000 another typical subject line . or 400 cannot copy message onto itself 431 target mailbox not found 431 source mailbox not found 451 message not found i.e. COPY-MESSAGE markl british-cars 2338 RESET-DESCRIPTORS 200 Command OK 431 mailbox not found i.e. RESET-DESCRIPTORS markl 1 10000 PRINT-MESSAGE 200 Command OK 401 printer not found 431 mailbox not found 451 message not found i.e. PRINT-MESSAGE markl 2433 pravda Lambert [Page 35] RFC 1056 PCMAIL June 1988 SET-MESSAGE-FLAG 200 Command OK 431 mailbox not found 451 message not found 500 flag number out of range i.e. SET-MESSAGE-FLAG british-cars 23 0 1 FETCH-MESSAGE 251 message follows: From: markl@ptt.lcs.mit.edu To: markl@ptt.lcs.mit.edu Date: Sun, 17 Jan 88 11:11:11 EST Subject: anything this is a sample of some message text . or 431 Mailbox not found 451 message not found i.e. FETCH-MESSAGE current-events 495 Lambert [Page 36] RFC 1056 PCMAIL June 1988 II. Operations by name copy-message create-address create-bboard-mailbox create-client create-mailbox create-subscription delete-address delete-bboard-mailbox delete-client delete-mailbox delete-subscription expunge-mailbox fetch-changed-descriptors fetch-descriptors fetch-message help list-addresses list-available-subscriptions list-clients list-mailboxes list-subscriptions login logout print-message reset-client reset-descriptors reset-mailbox reset-subscription send-message send-version set-message-flag set-password Lambert [Page 37] RFC 1056 PCMAIL June 1988 III. Responses by number 100 Pcmail repository version XXX; following are supported 200 Command OK 220 Client list follows: 221 Client out of date by > 1 week 230 Mailbox list <#msgs> <#new> follows: 240 Subscription list follows: 250 Descriptor list follows: 251 Message follows: 260 Address list follows: 350 enter message; end with "." 400 cannot copy message onto itself 410 already logged in 420 client already exists 430 mailbox already exists 430 bboard mailbox already exists 440 subscription already exists 460 address already exists 411 no user named 421 client not found 431 mailbox not found 441 subscription not found 451 message not found 461 address not found 402 internal error message 403 syntax error in outbound message 404 bad password or permission denied 405 mail state is temporarily in use by another client 406 please log in 500 operation syntax error or illegal argument Lambert [Page 38]