REST as practiced usually arrives wrapped in HTTP — verbs, URLs, status codes. Strip all of that away and a more general principle is left underneath: for any object, we should be able to create it, read it, update it every valid way, and delete it, one at a time or as a member of a collection.

The last post found one source of a complete API in mathematics: when a type has algebraic structure, the structure hands us the operations. This post is about another reliable source, and it covers a far larger category: ordinary objects with a lifecycle, and the collections they belong to. Its name is CRUD (create, read, update, delete). No HTTP, no status codes; just the operations objects and collections owe their callers, prescribed by the plain fact that an object gets born, used, changed, and retired, always as a member of some collection, named or not. Stripped of HTTP, REST teaches us how to deal with containers in general. I talked about building CRUD interfaces years ago, in CRUD with Dojo and RAD CRUD.

REST and CRUD are often conflated. REST, short for Representational State Transfer, is an architectural style of general principles, and HTTP is its best-known application. Its central feature, in Fielding’s words, is “a uniform interface between components”, and one constraint of that interface is the one many “REST APIs” drop: links. Their clients know out of band that gizmo 1 lives at /gizmos/1 and build that URL themselves, instead of receiving /gizmos/1 as the reference to the object. What APIs commonly practice under the name is CRUD over resources and their collections, and in this post REST means that practice.

CRUD prescribes the operations

An object’s life has four phases, and each one names a part of the API:

• Create. Constructors and factory methods. Every valid initial object should be constructible: if some legitimate starting state can’t be built, the create side is incomplete.
• Read. Accessors in some form, including subsetting: asking for only the fields or parts we need instead of the whole object.
• Update. Every valid modification. This is the one everything depends on. After any modification the object has to stay well-formed. The operations that guarantee that belong to the API from the start. It is the previous posts’ “reach every valid state” applied to a single object: from any valid state, some sequence of modifications has to reach any other. If one cannot, the update side has a hole.
• Delete. Removal from the store, or disposal of an object in memory: destructors in C++, using with Symbol.dispose in JavaScript. Either way, release resources and hold the invariants on the way out, so teardown leaves the rest intact.

Some domains narrow the set on purpose. A log creates immutable records, so updating them is out of the question. Even a log needs a delete, or we drown in records, but it is a mass operation, such as removing every record in a file, run in a separate workflow, often outside the application that writes the log.

And the set recurses: a sub-object gets the same four. An object made of objects is complete only when its parts are too. How to expose a sub-object is a question for a later post.

A leaked key

The set is easiest to see when one of its operations is missing. A service of mine had a simple user-control system, where each user has a unique access key. One key leaked, and we disabled that user. But the key could not be changed, because it is set when the user is created. And there was no way to delete a user, so we could not create the user again with a different key. The account was stuck: compromised, disabled, and no operation in the API could fix it. We fixed it by editing the data by hand. The irony is that we still needed the full set of operations; they were not coded, so we performed them manually.

The minimal set falls straight out of that. We need to create and delete users. To use them at all, we have to reach them: list the users, and get one by its name. Is that enough? To change one small property, the access key, we can read the user, delete it, and create it again with the new properties. That reaches every combination of properties, but not every state. A re-created user is a new object: its history and everything that referred to the old one stay behind. Changing the key of the same user takes an update, so it belongs to the minimal set:

// enough to reach every state
users.create({name, accessKey});
users.list();
users.get(name);
users.update(name, {accessKey});
users.delete(name);

// the second layer: intent
users.rotate(name);

In memory, in a single thread, update() could be a getter and a setter. Over a wire it earns a method of its own. Marshaling a whole object is expensive, so a read asks for a subset and a write sends a patch, which makes an update more than a wholesale replacement, even when it travels as a PUT. And in a distributed system the update has to be atomic, so ACID (atomicity, consistency, isolation, durability) and its variants become central.

That is not the end, and the reason is the second layer from the last post. If keys are rotated periodically, that names one more operation: rotate(). Reachability first, then efficiency and intent on top.

The users are a collection, and the question we asked of them, whether every valid state can be reached from any other, is the one the last post asked of a list. There is no firm wall between math and CRUD. The split between the two posts is a convenience of presentation, and the test is the same in both.

When the storage prescribes it instead

CRUD is one candidate for what dictates the operation set. Storage is a rival, and it wins by default when the choice goes unmade.

Picture an online store whose database pages by cursor: each request returns a batch and a token that marks where it stopped. Shoppers browse the catalog in numbered pages, “page 7 of 20”, and jump to the last one. The domain asks for exactly that: reach any page, and know the page count. Pass the database’s paging straight through, and the API grows a shape nobody would choose for a catalog. Fetch a batch, get a token, pass it back to continue. Ask for page 7 and there is no answer. Ask how many pages there are and it has to count. Every one of those quirks is a fact about the storage, invisible to the caller.

The leak outlives its cause. Migrate to a database with offsets tomorrow and the API still has the token, because callers built on it. Page numbers and a count can be added later, but every caller has already worked around their absence. The implementation can change, and the interface keeps its quirks. And this is the series’ claim running backwards. The domain hands us the operation set. Skip that step and something else hands it to us: usually whatever we happened to build first.

None of which forbids an API shaped like that — cursors are the right answer for a feed. The question is whether the shape came from the domain or from the schema, and we should be able to say which.

The opposite case is legitimate. An API often has to expose machinery the domain never names: indices, caching, compaction. A subject-matter expert would not ask for them, and they belong in the API anyway, on purpose. The failure above is storage leaking in by accident.

Collections are abstract indices

An object always lives in a collection, named or not. Objects are kept somewhere, usually in a database, which is a collection itself. A collection is a set of objects, ordered or not, and usually linear. When objects are known by their keys, a collection is conceptually a list of keys we operate on. It can be live, defined by properties and built on demand: a search or a filter creates one, virtual and often ephemeral, and every page of results is one more. It can be generated, by walking a tree or by calculating values. Or it can be arbitrary, like a shopping cart: only the items we picked, in the order we chose or in no order at all. Whatever the kind, the API is the same: the collection works as an index, a list of members by key, with mass operations that act over many of them at once. Its members matter to it very little: at most it knows a key to sort by.

Three distinctions make the collection side tractable.

First, mass operations split by what they do to identity. Update preserves identity: change the members in place and they are the same objects afterward. Copy, rename, and move create new ones in a keyed store, where the key is the identity: rename and move change it, so the result is a new object even when its content is the same. Settling that split early keeps the API honest about when an operation is a mutation and when it is a minting.

Second, choosing which objects and choosing which fields are orthogonal axes. Filtering, sorting, and paging are index operations: they decide which members appear and in what order. Subsetting is a read operation: it decides which fields of each member come back. They compose freely, and a complete collection API treats them as the separate things they are instead of tangling them together.

Third, removal splits by ownership. A store is a container with an index of its own, such as a primary key, and it owns its objects: deleting one destroys it. Whether a record exists outside its container depends on the substrate. In a database it does not: no row, no record. C++ can move an object into a container that then owns it. JavaScript leaves it to the garbage collector. Other indices only refer to objects, so removing a member leaves the object alive. The shopping cart is an index over the catalog:

cart.remove(sku);    // the product is still for sale
catalog.delete(sku); // the product is gone

Such an index is an ordinary object, with its own interface and its own lifecycle: dropping a database index deletes no rows. A live collection stores no members, so it has nothing to remove. We change its query, or a filter applied after it, which is usually a query too: adding id !== 123 excludes one item by its id. A collection API has to say whether a removed member survives; two verbs, as above, say it at every call.

Stepanov brought generalized containers to the masses. REST taught us how to access them. It cares less about how a container is implemented and more about how it is reached: paging, sorting, filtering and search, subsetting and projection, all forced into the open because the data was remote. They carry over to local code as principles: iterators, for-of loops, and any kind of streaming use them the same way. Their implementation can leak, and sorting shows it. In memory, a sort has to read the whole sequence before it returns the first item, in place with sort() or into a copy with toSorted(): a delay, and for the copy, extra memory. In a database, the order can be an index: an ordered sequence we can stream, filter, and page like any other collection.

The same access works on collections that are computed instead of stored. A Fibonacci sequence framed as an iterator is an infinite collection, and the iterator helpers filter and page it:

function* fib() {
  let [a, b] = [0n, 1n];
  for (;;) {
    yield a;
    [a, b] = [b, a + b];
  }
}

fib().filter(x => x % 2n === 0n).take(5).toArray();
// [0n, 2n, 8n, 34n, 144n]
fib().drop(10).take(3).toArray();
// [55n, 89n, 144n]

For a computed sequence, “remote” means expensive and unbounded, so paging becomes drop() and take(). A string gets the same view from replace() with a global regular expression and a function, which iterates over the matches; a post of its own covers it.

No HTTP required

None of that mentioned HTTP. The same CRUD-plus-index shape maps onto any substrate: a function library, an in-memory object model, a database layer, a message protocol. HTTP-based REST is one realization of it, a very successful one. It has verbs for the operations, status codes for the outcomes, headers for caching and conditional writes, and conventions for bulk and long-running work. That realization deserves a treatment of its own. But the principle underneath (create, read, update every valid way, delete, over objects and the indices they belong to) is substrate-independent. “REST” is the famous instance; completeness over an object’s lifecycle is the idea.

The converse holds too. A local API can front remote resources, and then everything learned from HTTP endpoints applies directly, for performance: paging, projection, and filtering on the server side, because the cost is remote whatever the call looks like.

In local code GET maps onto a getter and a PUT of one property onto a setter, as long as the operation is cheap and touches nothing else. When it is not, a method says so: getSummary() for a summary that takes real computation, setX() for a replacement that notifies other objects.

Worked example: dynamodb-toolkit

dynamodb-toolkit, a library of mine, is such an API: local calls in front of a remote store, with the shape made concrete. It gives a collection uniform list operations (filter, sort, page) and batch operations addressed by a set of keys. It keeps the identity split explicit: identity-preserving mass updates on one side, identity-creating copies on the other.

Paging shows the domain and the storage side by side. DynamoDB pages by cursor: a token that marks where the last page stopped. A catalog wants numbered pages with a total. The toolkit offers both shapes:

import {paginateList, cursorList}
  from 'dynamodb-toolkit/mass';

// numbered pages: page 7 of N
const page = await paginateList(client, params,
  {offset: 60, limit: 10});
// {data, offset: 60, limit: 10, total}

// the storage's own shape
let next = await cursorList(client, params,
  {limit: 10});
next = await cursorList(client, params,
  {limit: 10, cursor: next.cursor});
// {data, limit, cursor}

The cursor costs the same for every page. The offset and the total cost more: to reach page 7 the toolkit counts its way past the 60 items before it, and the total takes another counting pass over the rest. That walk is the catalog’s quirk from before, moved behind the shape the domain asks for, with its price stated. Both shapes are there, and the caller picks one knowing the price.

Being a toolkit, it is meant to be drawn from selectively. That is the procedure of the first post in practice: work out the complete set first, then let each application cut what it does not need.

Summary

Three things to take away:

• Strip HTTP out of REST as practiced, and the residue is CRUD as a completeness principle: create, read, update every valid way, delete, recursing into sub-objects, with efficiency and intent layered on top. Skip that step and the storage prescribes the API instead.
• Collections aren’t special, and often they aren’t physical: a search result or a page is one. They are indices with mass operations: identity separates an update from a copy or a move, which objects is independent of which fields, and ownership decides whether removing a member destroys it.
• REST’s access operations (paging, sorting, filtering, projection) work without HTTP, and a local API over remote data should adopt them. Their implementation can leak: a sort in memory reads the whole sequence before it returns anything, while a database index streams the order.

Math prescribed the API for mathematical types; CRUD prescribes it for objects and their collections — the same discipline over two domains. One more post closes this half of the series: it looks at what it costs when a language ships the specialized operation without the general one. After that the series turns to how to express a complete API: what makes it easy to use, the techniques and tactics, and how APIs fit together.