By Sophia Ciocca and Jeff Sisson

It takes a lot of people to publish an article at The New York Times, and sometimes things get messy. Several reporters and editors may participate in the writing, revising and publication of a single story, and that collaboration is often nonlinear, which can lead to people stepping on each other’s toes.

This is especially true in breaking news situations when many types of collaborators (photo editors, copy editors, reporters, producers) need to make edits to a document at the same time. When we first created Oak, The Times’s next-generation article editing interface, only one person could work on a document at a time. Oak used “field locking” to facilitate some simultaneous collaboration features, like letting different people edit separate metadata fields, but only one person could edit the body of an article at a time.

It was clear to us that a more perfect article editor would allow collaborators to work in a document concurrently and seamlessly. Oak needed a collaborative editing feature.

But collaborative editing is a hard problem to solve. It presents many new puzzles, including how to incorporate live updates, what to do about conflicting edits, and how to handle errors resulting from spotty wifi. A product like Google Docs has huge teams and resources devoted to solving these kinds of problems at scale. Our team only had a couple of months to solve these puzzles in a way that suited The Times’s editorial workflow. Luckily, support for these types of technical challenges was one of the original reasons we chose ProseMirror, an open-source text editing library, as the technical substrate for Oak.

Everyone needs the steps

For an Oak article to be collaborative, every person with access to the article needs to have the most up-to-date version in real time. That means that every time an edit is made, it needs to be visible from every collaborator’s computer. Translating individual edits into a form that can be sent over the internet is something ProseMirror does at a low level; it calls each of these edits “steps.” When a user types a letter, that’s a step. Pressing the enter key to go to a new line? Also a step. Anything that manipulates the document — adding an image, editing the headline, deleting a paragraph — counts as a step.

What does the data of a step look like, exactly? We portray steps as JSON objects that act as directions of sorts for how exactly the step changes the document. Here’s what it looks like to place the letter “H” in the second position in a document:

{
  “stepId”: 1,
  “step”: {
    “stepType”:”replace”,
    ”from”:2,
    ”to”:2,
    ”slice”: {
      “content”:[{
        “type”:”text”,
        ”text”:”h”
      }]
    }
  },
  “clientID”: “1111–42783748297342”,
  “timestamp”: “Sun Feb 10 2019 12:31:32 GMT-0500 (Eastern Standard Time)”
}

When a user types words, changes an image or adds a comment, the steps of those actions are added to the user’s local document, but those steps haven’t yet been sent to anyone else. In order for the steps to show up on other people’s computers, they need to be sent to and confirmed by a remote server, which we call the “authority server.”

The authority server acts like a bouncer at a club: when it receives steps from a user, it has to decide whether to let those steps in to its database. If it sees that no steps have been saved since the user’s browser last talked to the authority server, it will save them to its database. Otherwise, it will tell that user’s browser to fetch any steps it hasn’t seen yet.

A typical conversation between the local document and the remote authority server might look like this:

1. Local: “Hi, I have steps one, two and three.”
2. Remote: “Great, I’ve stored steps one, two and three.”
3. Local: “Hello again… I have steps four, five and six.”
4. Remote: “Ah! Someone else has inserted steps four and five. Please fetch them.”
5. *Local fetches steps four and five, rebases its previous steps four, five and six as six, seven and eight to reflect the confirmed steps four and five.*
6. Local: “Hey… I now have steps six, seven and eight.”
7. Remote: “Cool, I’ve stored steps six, seven and eight.”

Notice how in line four the local document tries to send steps four, five and six, but the remote server tells the browser that steps already exist there. The browser needs to fetch the steps, rebase and try to send the steps again. What’s going on there?

Rebasing is the process that happens when two users have inserted steps around the same time and the code needs to negotiate how they fit together. It involves one of the most important parts of ProseMirror’s collaborative editing support: an algorithm called Operational Transformation, which determines how to place a user’s steps on top of a newer version of the document.

Home is where the steps are

Though ProseMirror provides great building blocks (and a fully working demo) to implement a collaborative editing server, it doesn’t specify how to store steps in a database. We had to architect and implement our own solution that catered to our specific needs. Some of the questions we considered in our prototyping phase included: How can we scale the authority server to allow for lots of edits — up to 25,000 per article? How might we preserve the editing history of a collaborative document? And finally, how can we support a host of features, like pushing the latest document to our print production systems, that are unique to our newsroom’s needs?

After sketching out a few different prototypes, we landed on using the real-time database Firestore to store our steps and act as the bouncer for incoming steps. Firestore supports database transactions, which means that steps submitted from multiple people will be retried by the Firestore library until one person’s insertion “wins” and is written to the database. Ordering multiple peoples’ steps in this way makes Firestore a good fit as an authority server.

When a write operation to the database has succeeded, Firestore will notify remote users that new steps have been added to the database, which causes the app to fetch any newly-confirmed steps and bring their copy of the document up-to-date.

Several people are typing

Keeping the collaborative document up-to-date is important, but so is knowing who else is lurking in a document. We wanted everyone in a document to be able to see where their collaborators’ cursors might be, or if their collaborators highlight a certain passage with a mouse. Leveraging APIs from ProseMirror and Firebase (a sibling of Firestore), we added a feature to Oak that shows who is in the document and where they’ve left their cursor.

Any time a user changes their selection within the Oak editor by typing, clicking or dragging, ProseMirror observes the change to the current state of their browser selection and creates a Selection data structure with that information. This data reflects information about where a user’s cursor is in the text, and contains the start position (or the head) and the end position (the anchor).

If a user makes a selection in a document, the selection gets rendered for every computer that has the document open and displays the name of the user next to it. When the user changes their selection in their browser, that change needs to be mirrored in the document, so we push the updated selection data (head + anchor positions) into Firebase.

To handle these selections, we sometimes have to predict the future. Because selection data and changes to the document are being sent to separate servers, the Oak app might receive selection changes for versions of the document that the browser hasn’t yet received. If the Oak app receives an edit to the document before a corresponding change to a selection, it has to simulate how it thinks that selection would change. Similarly, if it receives a change to a selection that’s tied to an edit that hasn’t arrived yet, the app waits until that edit arrives to render the change to the selection. To accomplish this, we use Redux Sagas to wait for and then apply these selection updates from the future.

It’s worth noting that we chose to use Firebase to store these selections, as opposed to Firestore (which we use to store steps), because Firebase makes an onDisconnect hook available, which allows us to perform database modifications that are guaranteed to run even if the user closes the tab. This helps to ensure that when a user leaves the Oak editor, their cursor will be cleaned up after them.

Publish or perish

Once all of the article edits, or steps, have been inserted into Firestore, every browser window open to that article is updated with the latest steps. When an article is ready for publication, the backend service that handles publishing also gets up-to-date article data, which it sends to the publishing pipeline that our website and apps pull from to render articles on the user end.

Before we implemented collaborative editing, this flow was simpler: The backend service could go straight to our MySQL database where the article’s contents and metadata lived. For new collaboratively edited articles, however, we created an App Engine service (which we nicknamed the “collab service”) that pushes the contents of an article from Firestore to our publishing service. When a user hits the publish button, the browser makes a request to this collab service, which copies the data from Firestore onto the publishing pipeline.

Once a collaboratively edited article has been published digitally, it may need a tune-up for print. Before an article can be sent to the printing plant, editors in the newsroom need to remove hyperlinks and embedded content, and may need to shorten the article for print.

The interface for making these print edits relies on the existing MySQL database. Because collaborative articles are stored in Firestore and therefore disconnected from this print database, we developed a system that copies the collaborative state of an article to our primary MySQL database at regular intervals. This system consists of a set of Google Cloud Functions and Google Cloud Tasks, and allows up-to-the-second collaborative changes to be previewed as they will be printed in the newspaper — a helpful feature for our print editors.

Once a collaboratively-edited Oak article has been published and sent to print, it has finally made it to the finish line! Some articles might be updated with corrections after publication, and the steps will remain stored in Firestore for change-tracking purposes, but digital publication is where the story ends for our articles and their steps. We now have a working collaborative editor that seamlessly works with our existing news workflow, providing a massively improved user experience for collaborators across the newsroom.

This has been a huge, multi-team effort. Many people on the Oak and CMS teams past and present had a hand in getting us to a place where articles can be edited collaboratively. The Oak team in particular — Minerva Archer, Sophia Ciocca, Tom Holcolmb, Dmitriy Matveev, Shane Moore, Dylan Nelson, Thomas Rhiel, Alexandra Shaheen, Jeff Sisson and Matthew Stake — has done an amazing job chipping away at this since 2016.

Thank you also to Travis Rich of PubPub, who provided our first insights into using ProseMirror with Firebase, and to Marijn Haverbeke, the author of ProseMirror, who has given helpful guidance on this project.

Sophia Ciocca is a Software Engineer on the Publishing team at The New York Times, working on Oak. Follow her on Medium.

Jeff Sisson is a Lead Software Engineer on the Publishing team at The New York Times, working on Oak. Follow him on Twitter or his personal homepage.