WABBIT · ESSAYS
Silent data loss in Payload: when the API says success and nothing saved
Payload's write path is forgiving. It drops what it doesn't recognize, fills in what you left out, and validates things you didn't touch, and in most of those cases it still answers 200. Here are the ten ways we've seen it go wrong, where each one lives in the Payload source, and how we stop them now.
- Tested on
- Payload 3.90.2 (source and a live probe), 3.92.1 source, and the 4.0 beta checked for each cause
- Database
- MongoDB (two causes are adapter-specific, and the SQL adapters have their own)
- Applies to
- any Payload 3 project that writes through the Local API, REST or a script
- Causes
- 10, about 26 min read
- Files you'll touch
- one read-back helper, a row helper, a config walker, a publish guard, a paging helper, one field pattern
Every Payload write ends the same way: a 200 or a 201, and the document as Payload now sees it. The trouble is how much can happen between your data object and that response without an error. Payload's own docs are candid about one case: when a field's update access returns false, "the operation will not throw an error however the field will be omitted from the update operation" (field access docs). A row in a blocks field whose blockType matches nothing is dropped with a 200 (issue #16980, open). A field inside a nested array with access.update: () => false can be reset to its default by a status-only publish (issue #18415, open). And payload.update() validates the whole document, including fields you never sent, so a stale value in a field you didn't touch can reject the field you did (discussion #18375).
Most of these are not bugs in the sense of a crash. They're the write path doing what it was built to do: sanitize incoming data against the field config, refill what's missing from the stored document, apply defaults, and save. Each of those steps is right for the admin panel, where the form sends the whole document every time. Each becomes a trap the moment you write from somewhere else: a server action, a webhook handler, a migration script, an agent. Two of the ten are filed upstream as bugs, and one is an adapter inconsistency with no ruling either way; the sections say which.
We've hit most of these. A notifications collection where every row had a blank pointer, because a subfield was named id. A published page that went 404 after a copy fix, because the script merged onto a stale autosave. A paid deal whose follow-up steps stopped halfway, because a select option had been retired. A backfill that reported success after walking half its rows, because its own writes moved the pages under it. In each case the write returned 200, the test passed, and the gap showed up later as a support email or a missing page.
This walkthrough goes through all ten in the order the write path meets them. Eight are incidents we traced to the line. Two (4 and 5) are incidents we logged on older releases whose mechanism we could not reproduce on the current source, and those sections say so and give the current behavior from a live probe instead. For each one: what you see, the line in Payload that does it, how we fixed it, and how to detect it before production does. Every cause was rechecked against the 3.92.1 source and the 4.0 beta; where upstream has changed something, the section says so. The code is written for a generic Payload 3 project and typechecks against a real one.
Section 1: A denied field rule drops the field and answers 200
You write { listing: otherListingId } to a review as a member. The response is 200 and the document comes back with listing unchanged. Nothing in the response says the key was refused. A lesson-progress route that calls payload.update() as the student, on fields with admin-only access.update, returns 200 on every "complete" click and saves nothing.
Why. Field access runs inside the field beforeValidate traversal (fields/hooks/beforeValidate/promise.ts). When field.access.update returns false, Payload does delete siblingData[field.name], then, because the key is now undefined, refills it from the stored document through getFallbackValue. The rest of the request is saved. The same code is in 3.92.1 and in the 4.0 beta, which adds a validate operation to the same branch. The docs describe it as intended.
There's a sharper edge on the same mechanism. Issue #18415 (open, with a fix proposed in PR #18426) shows that for a field inside a nested array (array > array > field), a partial update from a user who fails the field check resets the stored value to the field's defaultValue instead of keeping it. The reporter, on 3.83.0, had about 2,000 stable identifiers reset to their default by list-view bulk publishes and scripted status-only publishes, which broke roughly 40% of the references pointing at them. If you rely on access.update: () => false to make a field immutable inside nested arrays, read that issue before you trust it.
The fix. Our access-control walkthrough covers the pattern in full under "Field rules fail quietly": keep the field rule, and add a wrapper on the collection's update rule that refuses the whole request when an identity field changes, so the attempt is loud and nothing is half-applied. We won't repeat the code here. For a server route that writes on a user's behalf, the fix is the other way around: do your own authority check first, then write with overrideAccess: true and a comment saying why.
How to detect it. The general tool for this whole post is a read-back: after a write, fetch the document at depth 0 with access off and compare the keys you sent to what's stored. We use one helper for it everywhere, in tests, seed scripts and any patch script that writes to production:
Reads the document back and compares only the keys you wrote, as values rather than as text, so key order can't fail it. Row ids are ignored because Payload adds them.
import { isDeepStrictEqual } from 'node:util'
import type { CollectionSlug, Payload } from 'payload'
type Doc = Record<string, unknown>
// Payload adds `id` to every array and block row,
// so comparisons ignore that key at every depth.
const strip = (v: unknown): unknown =>
Array.isArray(v)
? v.map(strip)
: v && typeof v === 'object'
? Object.fromEntries(
Object.entries(v as Doc)
.filter(([k]) => k !== 'id')
.map(([k, x]) => [k, strip(x)]),
)
: v
const same = (a: unknown, b: unknown) =>
isDeepStrictEqual(strip(a), strip(b))
export async function readBack(
payload: Payload,
collection: CollectionSlug,
id: number | string,
): Promise<Doc> {
const doc = await payload.findByID({
collection,
id,
depth: 0,
overrideAccess: true,
})
return doc as unknown as Doc
}
// Throws when a key you wrote is stored differently.
export async function assertPersisted(
payload: Payload,
collection: CollectionSlug,
id: number | string,
expected: Doc,
): Promise<Doc> {
const doc = await readBack(payload, collection, id)
const wrong = Object.keys(expected).filter(
(key) => !same(doc[key], expected[key]),
)
if (wrong.length > 0) {
throw new Error(
`${collection}/${id} not stored as written: ` +
wrong.join(', '),
)
}
return doc
}For cause 1 the test is a write as a member followed by assertPersisted on the key they weren't allowed to change, expecting the old value, plus an assertion that the collection rule rejected the request. The helper is deliberately strict about null versus a missing key, because the rest of this post is about that difference. One thing it can't see: findByID runs afterRead, which writes a declared field's defaultValue onto any undefined value, so a dropped field that has a default reads back as the default. Expect the default in that case, and treat "reads back as the default" as its own warning sign (section 7).
Section 2: Keys that aren't fields never reach the database
A seed writes { blockType: 'quiz', question, options } flat on a lesson row. The row is stored as { blockType, id }. A migration sends a block as { blockName: 'hero' } instead of blockType. The page saves with an empty layout. A server action writes attemptNumber to a collection whose installed package doesn't define it yet. The write returns 200 and the column doesn't exist.
Why. Three layers agree on this. The field hooks walk your config, not your object: beforeValidate and beforeChange each take the collection's fields and visit siblingData[field.name] for every field, so a key with no field is never looked at. For blocks, the blocks case in beforeValidate/promise.ts finds the block by row.blockType (or, when the row's id matches a stored row, the stored row's blockType) and only traverses rows where a block matched; a row with an unknown blockType, or none and no stored row to inherit one from, is skipped without a warning, which is issue #16980 (open at the time of writing). The row is never validated. What reaches the database is a stub: on 3.90 we watched Mongo keep { blockType: 'noSuchBlock' } and nothing else from the row, and the API never returns it, because no block matches on read either. The undeclared top-level key in the same write was not stored at all. And the Mongo adapter builds a Mongoose schema per collection with strict mode at its default, which Mongoose documents as: values "that were not specified in our schema do not get saved to the db" (Mongoose guide), and the same page says the option applies to updates. The adapter's allowAdditionalKeys option only governs what it strips on read, so it isn't an escape hatch for writes. A block from a pack that isn't installed is the same thing seen from the other side: its slug isn't in the field's blocks, so the row goes.
The fix. Author in the declared shape, and read the collection's fields before you write to it rather than the shape you remember. When a package defines the collection, a key your code writes before the installed version declares it is dropped. Upgrade the package or add the field. A seed that wrote flat rows gets rewritten to the real row shape, with a comment saying a flat row persists as { blockType, id }.
How to detect it. assertPersisted from section 1 catches every one of these, because a key with no field config reads back undefined (a declared field with a default reads back as the default, which is the section 1 caveat). For blocks, add a check before the write that every blockType is one the field accepts:
Checks row slugs against the blocks a specific top-level field accepts, so an unknown slug fails before the write instead of vanishing in it.
import type {
CollectionSlug,
FlattenedField,
Payload,
} from 'payload'
type Row = { blockType?: unknown }
function blockSlugs(
field: FlattenedField | undefined,
): string[] {
if (!field || field.type !== 'blocks') return []
const refs = field.blockReferences ?? field.blocks
return refs.map((b) =>
typeof b === 'string' ? b : b.slug,
)
}
export function assertKnownBlocks(
payload: Payload,
collection: CollectionSlug,
fieldName: string,
rows: Row[],
): void {
const field = payload.collections[collection]
?.config.flattenedFields.find(
(f) => f.name === fieldName,
)
const known = new Set(blockSlugs(field))
const bad = rows
.map((row) => String(row.blockType ?? ''))
.filter((slug) => !known.has(slug))
if (bad.length > 0) {
throw new Error(
`${collection}.${fieldName}: ` +
`unknown block types ${bad.join(', ')}`,
)
}
}Run it on seeds and imports. It reads top-level fields only; a blocks field inside a group needs its own lookup. In 4.0 the beta source still skips unmatched rows the same way, so the check stays useful after an upgrade.
Section 3: A group subfield named id is stripped on save
A notifications collection has a group entityRef with two text subfields, collection and id. Every write returns 200. Every stored document has entityRef: { collection } and nothing else. The first visible symptom is usually a button that reads the pointer, finds nothing and shows a "missing reference" error. Query the database and every notification ever written has the pointer missing.
Why. This one is in the Mongo adapter. buildSchema in @payloadcms/db-mongodb is called once for the collection and again for every group, named tab and array to build the subschema. Unless the caller passes allowIDField: true, it looks for a field named id, maps it to the schema's _id and filters the id field out of the fields it builds. The array builder passes allowIDField: true, because array rows have ids. The group and tabs builders don't. So a group's id subfield is removed from the Mongoose subschema, and strict mode drops the value on write. Payload's config layer treats a nested id as an ordinary field (that's PR #9245, which stopped a group's id from being mistaken for the collection's custom id), and its docs list the forbidden field names as __v, salt, hash, file and status, without id. The adapter still eats it. The same code is at v3.92.1 and in the 4.0 beta, and we found none filed for it.
The fix. Rename the subfield and backfill. Something like entityId works. If the same id survives in another field, have the reader fall back to it until the backfill runs:
A pointer group with a subfield that isn't id. The old name is kept out on purpose.
import type { Field } from 'payload'
// Never name a subfield `id`. The Mongo adapter
// treats it as the row's own id and drops it.
export const entityRef: Field = {
name: 'entityRef',
type: 'group',
fields: [
{ name: 'collection', type: 'text' },
{ name: 'entityId', type: 'text' },
],
}The backfill reads the value from wherever else it survived and writes it into the new key. The loop below is shaped around two rules that matter for every backfill. When the update changes the thing the query matches on, the pages shift under you, so after --apply always read the first page again. And a row that can't be sourced never leaves the query, so remember which ids you've tried, or page one fills with them and the loop never ends. A backfill that breaks the first rule processes about half its rows and reports success. One that keeps the first rule and skips the second can loop forever on rows it can't source.
Dry run pages through and counts. With --apply it always reads page 1 and excludes the ids it has already tried, so fixed rows leave the query and unfixable rows can't hold it.
import type {
CollectionSlug,
Payload,
Where,
} from 'payload'
type Doc = Record<string, unknown>
export async function backfillKey(
payload: Payload,
args: {
collection: CollectionSlug
missing: Where
group: string
key: string
source: (doc: Doc) => unknown
apply: boolean
},
): Promise<number> {
let fixed = 0
const attempted = new Set<string>()
for (let page = 1; ; page += 1) {
const skip = [...attempted]
const where: Where =
args.apply && skip.length > 0
? {
and: [
args.missing,
{ id: { not_in: skip } },
],
}
: args.missing
const res = await payload.find({
collection: args.collection,
where,
limit: 200,
page: args.apply ? 1 : page,
depth: 0,
})
if (res.docs.length === 0) break
for (const raw of res.docs) {
const doc = raw as unknown as Doc
attempted.add(String(doc.id))
const value = args.source(doc)
if (value == null) continue
fixed += 1
if (!args.apply) continue
const stored = (doc[args.group] ?? {}) as Doc
const next = { ...stored, [args.key]: value }
await payload.update({
collection: args.collection,
id: doc.id as string,
data: {
[args.group]: next,
} as unknown as never,
})
}
if (!res.hasNextPage) break
}
return fixed
}The as unknown as never on data is there only because the slug is a variable; with a literal slug, Payload types the data for you and the cast goes. Run it without --apply first, read the count, then run it with the flag and read the count again. The second number is what changed.
How to detect it. A lint over your collection configs: any field named id inside a group or named tab is a bug (array rows carry an id on purpose, and the collection's own custom id lives at the top level). The config walker in section 7 is easy to extend for this. And assertPersisted after the first write of any new group catches it in a test before the collection fills up with blanks.
Section 4: Partial group updates: what refills and what wipes
A server action updates one subfield of a group, onboarding.engagementStatus. Afterwards the sibling onboarding.onboardingStatus is blank. On another site, a patch script deleted a key from a block before updating, expecting it cleared. The key came back with its old value every run.
These two look opposite and come from the same rule, so it's worth stating precisely what the write path does with a key on an update, read from beforeValidate/promise.ts and getFallbackValue.ts and checked on a test database running 3.90:
| What you send for a subfield | What's stored |
|---|---|
The key is absent, or undefined | The stored value (on update). On create, the field's defaultValue, or nothing |
null | null. The field is cleared |
'' | An empty string on text-like fields. Payload rewrites '' to false on a checkbox, null on a number, and null ([] for hasMany) on a relationship or upload |
| A nested group object | Traversed subfield by subfield with the same rules, so a partial group object refills its missing siblings from the stored group |
What you send for a subfield: The key is absent, or
undefinedWhat's stored: The stored value (on update). On create, the field's
defaultValue, or nothingWhat you send for a subfield:
nullWhat's stored:
null. The field is clearedWhat you send for a subfield:
''What's stored: An empty string on text-like fields. Payload rewrites
''tofalseon a checkbox,nullon a number, andnull([]forhasMany) on a relationship or uploadWhat you send for a subfield: A nested group object
What's stored: Traversed subfield by subfield with the same rules, so a partial group object refills its missing siblings from the stored group
That last row is the part that surprised us. For a named group, beforeValidate sets siblingDoc[field.name] to the stored group and traverses the subfields with it, and a subfield that's undefined in your data gets getFallbackValue, which on update returns the stored value. So a partial group object alone doesn't wipe siblings in 3.90 or 3.92.1, and the 4.0 beta has the same branch. On the test database, an update that sent only meta.title kept meta.description; one that sent description: null cleared it; one that sent '' stored an empty string. The wipe we logged was on 3.78, with no reproduction recorded, and we couldn't reproduce it from the current source. What does wipe on every version is sending null for the siblings, which is exactly what a form state object does when it serializes every field it knows about. If you build data from a client form, untouched fields arrive as null or '', and those are values. That's the version of this bug we believe we hit, and the rule we wrote for it still holds.
The "delete doesn't clear" case is the same table from the other end. Deleting a key makes it absent, and absent means "keep what's stored." Write null.
The fix. Build the group explicitly: read the stored group, spread it, apply the change, write the whole group. It's one extra read, it's self-describing, and it's immune to whatever your form layer does with untouched fields. Pass req through when you call it from inside a hook or a transaction, so both calls share it.
Reads the stored group, overlays the change and writes the whole group. Pass null for anything you want cleared, and req from inside a hook.
import type {
CollectionSlug,
Payload,
PayloadRequest,
} from 'payload'
type Doc = Record<string, unknown>
export async function patchGroup(
payload: Payload,
collection: CollectionSlug,
id: number | string,
group: string,
patch: Doc,
req?: PayloadRequest,
): Promise<Doc> {
const doc = (await payload.findByID({
collection,
id,
depth: 0,
req,
})) as unknown as Doc
const stored = (doc[group] ?? {}) as Doc
const next = { ...stored, ...patch }
await payload.update({
collection,
id,
data: { [group]: next } as unknown as never,
req,
})
return next
}How to detect it. When a server action takes an object from a client, log the keys whose value is null or '' before the write, at least in development. Any key the user didn't touch that shows up there is a wipe waiting to happen. Then assertPersisted with the full group you intended, siblings included.
Section 5: Writing null to a relationship: prove it landed
A position record has a currentHolder relationship. The action to vacate it wrote { currentHolder: null }. The response was 200 and the holder was still there, and an afterChange hook that syncs access off the holder kept syncing the old person. The fix at the time bypassed Payload and ran a Mongo updateOne with $set: { currentHolder: null }.
Why. We logged this on 3.65 and couldn't find the mechanism in the current source. On a test database running 3.90 we wrote null to a single relationship and read null back from the update's return value, from findByID and from the raw document, so on current versions the write lands. On 3.90 and 3.92.1, beforeValidate normalizes '', 'none', 'null' and null on a relationship to null for a single relation and to [] for hasMany. The relationship validator accepts null when the field isn't required. The Mongo write transform skips relationship conversion when the value is falsy and leaves the key in data, and updateOne passes data to findOneAndUpdate, whose top-level keys Mongoose treats as $set, so the null is stored. We found none open, and the oldest issue in this area (issue #3013, about undefined not clearing a field) was closed in the 1.x era, with a Payload team member explaining on the community thread that undefined "will not work because we merge existing data with incoming data." So treat the $set workaround as the thing you reach for when you've proven the Local API write didn't land, after a read-back.
Two related facts are current and do bite:
- A
hasManyrelationship can't benull. It becomes[]. If your code checks=== nullto mean "unassigned," it never matches. - Payload doesn't check that a relationship id points at a real document, or at the right collection (the validator checks
required, row counts andfilterOptions, nothing else). A subscription handler that writes a member id where a user id belongs succeeds, the row exists, and every lookup keyed on the user id misses it, which looks exactly like "nothing was written."
The fix. Write null through the Local API, then read it back. If it didn't clear, the raw write is available through the adapter. It skips every hook and updatedAt, and on a collection with versions it leaves the newest version row untouched, so the next Payload update rebuilds from that version (section 9) and the old holder comes back. It's acceptable on a collection without versions, or as a one-off followed by a Payload write that puts a version row in place; otherwise fix the version row too.
Writes null through Payload and proves it. The raw fallback runs no hooks and leaves version rows alone, which is why it's the second choice.
import type {
MongooseAdapter,
} from '@payloadcms/db-mongodb'
import type { CollectionSlug, Payload } from 'payload'
import { assertPersisted } from './assertPersisted'
export async function clearRelation(
payload: Payload,
collection: CollectionSlug,
id: string,
field: string,
): Promise<void> {
await payload.update({
collection,
id,
data: { [field]: null } as unknown as never,
})
await assertPersisted(payload, collection, id, {
[field]: null,
})
}
// Last resort: no hooks, no versions, no access.
// On a versioned collection the next Payload update
// rebuilds from the newest version row (section 9).
export async function clearRelationRaw(
payload: Payload,
collection: CollectionSlug,
id: string,
field: string,
): Promise<void> {
const db = payload.db as MongooseAdapter
const model = db.collections[collection]
if (!model) {
throw new Error(`no model: ${collection}`)
}
await model.updateOne(
{ _id: id },
{ $set: { [field]: null } },
)
}How to detect it. assertPersisted with { field: null }. For the wrong-collection id, query the related collection for the stored id once in a test; a relationship that resolves to nothing at depth 1 is the tell.
Section 6: Array rows merge by id, so what you drop comes back
A grading route rebuilds a completedModules array and writes it. A row that was supposed to lose its cooldownExpiry keeps it. A content patch rewrites a block's items and a removed subfield reappears on the next run.
Why. In beforeChange/promise.ts, every array and block row is traversed with siblingDoc: getExistingRowDoc(row, siblingDoc[field.name]). That helper finds the stored row with the same id and hands it in as the row's stored document. From there the section 4 table applies per row: a subfield you left out is refilled from the matched stored row. The same function is at v3.92.1 and in the 4.0 beta, unchanged. Rows without an id match nothing, so they're taken exactly as written and Payload mints a new id for each one (the array's hidden id field has a defaultValue that generates one).
A common rule is "strip the ids before saving to force replacement," sometimes credited with fixing a reorder that didn't stick. We couldn't find the reorder in the source: the stored order is the order of the incoming array, and the id match only decides which stored row supplies missing subfields. On 3.90 two rows reordered with their ids kept came back in the new order, so the id match does not pin position. If you see a reorder fail, look at the client first, and read the rows back before blaming the adapter. The ghost-value case is real and is what the rule fixes: on the same test database, a row sent by id without its label came back with the stored label.
Stripping ids has a cost of its own. Anything that points at a row by its id (a map marker, a photo reference, an anchor link) now points at nothing, because every row got a new id. So we keep two helpers and pick on purpose:
Two ways to replace rows. The first gets fresh ids; the second keeps ids by position for in-place edits, so references survive.
type Row = Record<string, unknown>
// Full replacement. Nothing is refilled from the
// stored rows, and every row gets a new id.
export const withoutIds = (rows: Row[]): Row[] =>
rows.map(({ id: _id, ...rest }) => rest)
// In-place edits only: the id follows the slot, so
// use it when the order and count are unchanged.
export const keepIdsByPosition = (
next: Row[],
stored: Row[],
): Row[] =>
next.map((row, i) => {
const { id: _drop, ...rest } = row
const id = stored[i]?.id
return typeof id === 'string'
? { id, ...rest }
: rest
})keepIdsByPosition is for editing rows where they stand, which is how our content patches use it ("unchanged rows keep their identity"). If a row moved or one was inserted, the id follows the slot, and a marker that pointed at row X now points at whatever sits in X's old position. With kept ids the refill rule is back in play, so send every subfield you care about, with null for the ones that should clear. withoutIds is the simpler choice when nothing references rows.
How to detect it. assertPersisted on the array compares rows with ids stripped, so a subfield that came back from the stored row fails the check. In a review, grep for payload.update calls that write an array built from a client object without going through one of the two helpers.
Section 7: defaultValue is written into the document, and an unsaved global reads as its defaults
A block has a text field with defaultValue: 'Start your project', written for one page. The same block is seeded on a second page without that field. The second page ships the first page's heading. Separately, a theme global that nobody has saved yet returns a full palette from findGlobal, and the site renders it.
Why. Defaults aren't a render-time fallback. On create, and on update when the stored document has no value, getFallbackValue returns field.defaultValue and it's saved with the rest of the document. The fields docs say defaults "populate missing or undefined field values during the create or update operations," and that static defaults are also applied by the database adapters to their schemas. On the test database, a page created with no hero was stored with four hero subfields at their config defaults. The read side has its own rule: afterRead/promise.ts sets field.defaultValue on any field whose stored value is undefined, with a comment that this is "for globals being returned without being first created or collection documents created prior to having a default." For a global that has never been saved, the read returned {} before 3.64 (issue #14476, reported on Mongo; fixed in the global findOne operation by PR #14556, v3.64.0, which also stopped defaults leaking when access was denied); since then an unsaved global reads as its schema defaults. The 4.0 beta keeps both behaviors.
Two consequences follow. A default you add to a shared block later is already persisted as the old default on every existing document, so "the component resolves per-variant defaults" stops being true the day the first document is saved. And a read can't tell you whether a value was chosen or defaulted.
The fix. For blocks reused across pages, seed an explicit null for every defaulted text field the new page doesn't want, and keep defaults on shared blocks neutral. For globals, decide whether "unsaved" should render the defaults (we wanted that for the theme palette) and make that decision in the component that reads it, because the API won't distinguish the cases for you.
How to detect it. Walk the config and list every defaulted path, then review that list whenever a block is reused or a default changes:
Lists every field path with a defaultValue, through groups, tabs, arrays and inline blocks. Blocks referenced by slug string live in config.blocks and need their own walk.
import type { Field } from 'payload'
function walk(
fields: Field[],
prefix: string,
out: string[],
): void {
for (const field of fields) {
const name =
'name' in field ? field.name : undefined
const path = name ? prefix + name : prefix
if (
'defaultValue' in field &&
field.defaultValue !== undefined
) {
out.push(path)
}
const next = name ? path + '.' : prefix
if ('fields' in field) {
walk(field.fields, next, out)
}
if (field.type === 'tabs') {
for (const tab of field.tabs) {
const tabPrefix =
'name' in tab ? next + tab.name + '.' : next
walk(tab.fields, tabPrefix, out)
}
}
if (field.type === 'blocks') {
const refs =
field.blockReferences ?? field.blocks
for (const block of refs) {
if (typeof block === 'string') continue
const p = `${next}[${block.slug}].`
walk(block.fields, p, out)
}
}
}
}
export function defaultedPaths(
fields: Field[],
): string[] {
const out: string[] = []
walk(fields, '', out)
return out.sort()
}Feed it payload.collections.pages.config.fields and you get lines like layout[intakeForm].formHeading. Blocks registered in config.blocks and referenced by slug are skipped here; walk payload.blocks for those. Commit the list the way we commit the access-defaults list; a new line is a decision someone should make.
Section 8: A retired select option rejects every later write to that document
A deal's plan select had a value that was later removed from the options. A payment webhook then advanced a deal still holding it to paid. The deal's main row was written as paid, the save threw while inserting the version row, and the cascade died between the two. Every retry saw "already paid" and skipped it, and because the handler caught the error, nobody saw one.
Why. Removing an option breaks later writes in two places. Payload's select validator (fields/validations.ts, select) rejects any value not in the options, and update() validates the whole document, including fields that aren't in data, against their stored values. The 3.90 source has a narrower mode (shouldLimitValidationToSubmittedFields in collections/operations/utilities/update.ts) that validates only the submitted top-level fields, and it's enabled only for trash and unpublishAllLocales; a plain update doesn't get it (discussion #18375 asks for the option, and its author found the same gate). So when the field is validated, the write fails with a 400 before anything is stored, naming a field you didn't send.
When it isn't validated, the second layer catches it. Field validation is skipped when the field's admin.condition is false for that document (beforeChange/promise.ts, passesCondition), and a select that only shows on some deals has a condition that's false on the rest. The Mongo adapter still has the option list as an enum on the schema path for every select (buildSchema.ts, the select builder). The version collection's model is built from the same fields, and a version row is inserted with Model.create, where Mongoose runs its validators, while the main row is written with findOneAndUpdate, where Mongoose's update validators "are off by default" (Mongoose validation docs). So the main row went to paid, the version insert threw a Mongoose enum error, and the operation failed after a half-write, with no 400 and no rollback.
The "vanishing side effect" is the second half. A write that fails after another write has landed is an inconsistent state unless both ran in one transaction. Payload carries a transaction on req (transactions docs), so pass the same req through every Local API call in a handler; on MongoDB that needs a replica set, and even then a hook you don't await can return success on a request that is rolled back. In the case above the half-write was inside a single update (main row written, version row rejected), so the thing that saves you there is the handler failing loud enough that a human reconciles it.
The fix. Never remove a select value. Retire it: keep it as a valid option with a retired label, map it to the live value in a beforeValidate hook so the next save normalizes it, and resolve it with a fallback wherever the code reads it.
Retired values stay valid options. The hook rewrites them to the live value on the next save of any document holding them.
import type { Field } from 'payload'
export const PLANS = ['basic', 'pro', 'team'] as const
// Add to this list. Never remove from it.
export const RETIRED_PLANS = [
'starter',
'legacy',
] as const
const retired: readonly string[] = RETIRED_PLANS
export const planField: Field = {
name: 'plan',
type: 'select',
defaultValue: 'basic',
options: [
...PLANS.map((v) => ({ label: v, value: v })),
...RETIRED_PLANS.map((v) => ({
label: `${v} (retired)`,
value: v,
})),
],
hooks: {
beforeValidate: [
({ value }) =>
typeof value === 'string' &&
retired.includes(value)
? 'basic'
: value,
],
},
}For multi-document handlers, run the writes in one transaction by passing the same req to each Local API call, and let the handler fail loudly when a later write fails, so the retry runs again instead of skipping.
How to detect it. A unit test that pins the option list to include every retired value, so a cleanup commit can't drop one. And before removing any option, count the documents holding it with payload.count({ collection, where: { plan: { equals: 'legacy' } } }), and the version rows, which validate too, with payload.countVersions({ collection, where: { 'version.plan': { equals: 'legacy' } } }).
Section 9: update on a draft-enabled collection builds on the newest draft, and that can unpublish the page
A patch script fixes two sentences in a published essay with payload.update({ id, data, draft: false }). The essay goes 404. Its _status is now draft. The editor had opened the post in the admin weeks earlier and autosave had left a draft version behind.
Why. updateByID loads the base document through getLatestCollectionVersion, which queries the versions collection for parent = id and latest = true, sorted by updatedAt descending, and returns that version's content as the document to merge onto. It doesn't look for the published one. If the newest version is a draft, its _status: 'draft' and its content are the base, and since your data didn't include _status, the section 4 rule refills it from that base. draft: false only means the main collection row is written; the drafts docs say the flag "does NOT control whether a document is published or not." The same lookup is at v3.92.1 and in the 4.0 beta. It bit us three times before we wrote a guard: once by a layout patch that put a page into draft, once by an essay fix, and once the other way, where a status-only write left a doc in draft because _status was never sent. We reproduced it in three calls on a test database running 3.90: create a page as published, save a draft with draft: true that changes the title, then update an unrelated field with no draft flag. The update returned the draft's title, and the main document's _status was draft afterwards. The page was offline.
The fix. Before writing, check whether the newest version is an unpublished draft and stop if it is. Write with _status: 'published' explicitly when publishing is the intent. After the write, read the document back with draft: false and the newest version, and fail if either isn't published.
Refuses to write over a pending draft, and proves the page is still published after the write.
import type { CollectionSlug, Payload } from 'payload'
type Target = {
collection: CollectionSlug
id: string
}
type Versioned = { _status?: string }
async function newestVersion(
payload: Payload,
{ collection, id }: Target,
) {
const { docs } = await payload.findVersions({
collection,
where: { parent: { equals: id } },
sort: '-updatedAt',
limit: 1,
depth: 0,
})
const latest = docs[0]
if (!latest) return null
return {
savedAt: latest.updatedAt,
status: (latest.version as Versioned)._status,
}
}
// A message when a newer draft would be the base.
export async function pendingDraft(
payload: Payload,
target: Target,
): Promise<string | null> {
const latest = await newestVersion(payload, target)
if (latest?.status !== 'draft') return null
return (
`${target.collection}/${target.id} has a draft ` +
`(saved ${latest.savedAt}) newer than the ` +
'published version. Publish or discard it in ' +
'the admin, then re-run.'
)
}
export async function assertPublished(
payload: Payload,
target: Target,
): Promise<void> {
const doc = (await payload.findByID({
...target,
depth: 0,
draft: false,
})) as Versioned
const latest = await newestVersion(payload, target)
if (
doc._status !== 'published' ||
(latest && latest.status !== 'published')
) {
throw new Error(
`${target.collection}/${target.id} is not ` +
'published after the write. Republish it.',
)
}
}The three calls in order. The status is explicit, so nothing is inherited from the base version.
// in the patch script, per document
const stop = await pendingDraft(payload, target)
if (stop) throw new Error(stop)
await payload.update({
...target,
data: { ...changes, _status: 'published' },
draft: false,
})
await assertPublished(payload, target)How to detect it. A throwaway-database rehearsal only finds this if it copies the _<collection>_versions rows, or creates a pending draft on purpose. A rehearsal that loads only the main documents never exercises the draft case. Add the guard's "pending draft" branch to the rehearsal and make sure it fires once.
Section 10: A limit still applies with pagination: false
A sweep written as find({ limit: 5000, pagination: false }) would report success on a collection past 5,000 rows and never read the rest. That shape is why the platform's paging helper exists (see How Tome does it): on a billing run it would mean customers who are never charged, and on an expiry sweep records that never expire. The incidents we did log are the two neighbors of this one. A count was done with find({ limit: 0 }) and .totalDocs, which loaded every matching row into memory on every save to produce one number. And a backfill paged a query that its own writes were shrinking, processed about half its rows and reported success (section 3).
Why. In collections/operations/find.ts, usePagination = pagination && limit !== 0 and sanitizedLimit = limit ?? (usePagination ? 10 : 0). The Mongo adapter then sets options.limit = limit whenever limit >= 0, with a comment that the limit "must also be set here, it's ignored when pagination is false," and turns pagination off itself when limit === 0. So pagination: false only removes the count query and the page math; your limit is still a limit, and with no limit at all the default is 10 unless pagination is off. On the test database, find({ pagination: false }) with no limit returned all 20 documents. The pagination docs now say this in so many words: "If you specify a positive limit, Payload still applies that limit." The same lines are at v3.92.1 and in the 4.0 beta. This one is documented behavior, so the question is only whether your sweeps know it.
The fix. Page explicitly with a ceiling, and return the fact that the ceiling was hit instead of logging it. For counts, use payload.count(), which reads only the number.
Pages with a declared ceiling and reports when it stopped early. A truncated scan is only safe for a job that runs again.
import type {
CollectionSlug,
Payload,
Where,
} from 'payload'
export async function findAll<T>(
payload: Payload,
args: {
collection: CollectionSlug
where: Where
maxPages?: number
pageSize?: number
},
): Promise<{ docs: T[]; truncated: boolean }> {
const pageSize = args.pageSize ?? 200
const maxPages = args.maxPages ?? 50
const docs: T[] = []
let truncated = false
for (let page = 1; ; page += 1) {
const res = await payload.find({
collection: args.collection,
where: args.where,
limit: pageSize,
page,
depth: 0,
})
docs.push(...(res.docs as T[]))
if (!res.hasNextPage) break
if (page >= maxPages) {
truncated = true
break
}
}
return { docs, truncated }
}A job that converges (the remainder is picked up next run) can tolerate truncated: true as long as it reports it. A billing run or an expiry sweep can't, so those raise the ceiling or fail. Also keep in mind that paging a query whose results your own update removes from the match skips half the rows; section 3's backfill shows the shape that avoids it.
How to detect it. Grep for pagination: false and limit: 0 in jobs and route handlers, and for .totalDocs used as a count. Add a test that seeds one more row than the page size and asserts the sweep saw it. The join case has its own history (issue #11187, closed, fixed in 3.26), where a collection with a join field ignored the limit on Mongo; if your sweep touches a collection with joins, test it with a limit smaller than the row count.
Common mistakes
Treating a 200 as proof the data is stored. Most of this post is a list of ways it isn't. Read the row back, at depth 0, with access off, and compare the keys you sent.
Building data from a client form object. Untouched fields arrive as null or '', and both are values. Send the keys you changed, with null only where you mean "clear."
Deleting a key to clear it. Absent means "keep what's stored." Write null.
Naming anything id inside a group or named tab. The Mongo adapter removes it from the subschema and the value is dropped on write.
Removing a select option. Every document still holding it, and every version row, fails on its next write, including writes to other fields: Payload's validator when the field is validated, the Mongo schema's enum when it isn't.
Writing to a drafts collection without checking for a pending draft. The newest version is the base, draft or published, and draft: false doesn't change that.
Using pagination: false as "all rows." It means "no count query." The limit still applies.
Rehearsing scripts on a throwaway database without version rows. The draft case can't exist in a test that didn't copy _<collection>_versions.
Trusting a hook trace instead of a read-back. The hooks walk the config, not your object; the only way to know what a nested write stored is to read it.
How Tome does it
Tome's core package carries the paging helper from section 10 (findPaged, with a page ceiling and a truncated flag), and its layers count with payload.count() rather than find({ limit: 0 }).totalDocs. Relationship ids are read through one relationId reader, so a populated object and a bare id resolve the same way.
The code in this walkthrough was written for this post. You can use it in your own project.
Checklist
- Every scripted or server-side write is followed by a read-back at depth 0 that compares the keys you sent.
- Identity fields are guarded at the collection level, so a refused field change fails the request; routes that write for a user check authority first and pass
overrideAccess: true. - Seeds and imports check every
blockTypeagainst the field's blocks before writing. - No group or named tab has a subfield named
id. - Server actions send only the keys they changed, and
nullonly where "clear" is meant. - Array writes go through a helper that either strips row ids or keeps them by position for in-place edits, chosen on purpose.
- Every
defaultValuepath is listed and reviewed when a block is reused across pages. - Select options are retired with a label and a
beforeValidaterewrite, never removed. - Scripts on draft-enabled collections refuse a pending draft, write
_statusexplicitly and assert the document is still published. - Sweeps page with a ceiling and report truncation; backfills exclude the ids they've tried; counts use
payload.count().
FAQ
Why does a Payload update return 200 without saving my field?
The most common reasons are a field-level access.update rule that returned false (Payload drops the field and keeps the stored value), a key that doesn't match a field in the config, a block row whose blockType isn't registered, or a subfield named id inside a group. Read the document back at depth 0 and compare it to what you sent.
How do I clear a field with the Payload Local API?
Write null. A key that's absent or undefined is refilled from the stored document on update. A hasMany relationship can't be null; it becomes an empty array.
Does pagination: false return all documents in Payload?
Only when you don't pass a limit. With a positive limit, Payload still applies it; pagination: false just skips the count query and page math. Page explicitly with a ceiling for sweeps, and use payload.count() for counts.
Why did payload.update turn my published document into a draft?
On a drafts-enabled collection, update starts from the newest version, which may be an unpublished autosave, and its _status: 'draft' carries into the write unless you send _status: 'published'. Check for a pending draft first and set the status explicitly.
Is defaultValue stored in the Payload database?
Yes. Defaults are written into the document on create, and on update when the stored field has no value. They're also applied on read to fields that are undefined, which is why an unsaved global returns its schema defaults.
Can I remove a select option in Payload?
Not safely while any document or version row still holds it. Validation runs on the whole document, so the next write to that document fails, and on MongoDB the option list is also an enum on the schema, so even a write Payload doesn't validate fails when its version row is inserted. Keep the value as a retired option and rewrite it to the live value in a beforeValidate hook.
Why is a field named id inside a group not saved?
The MongoDB adapter builds a subschema for each group and treats a field named id as the row's own id, removing it from the schema, so the value is dropped on write. Rename the field and backfill.