Dbvisit Zero Data Loss internals

Zero Data Loss is quite a name. Put those three words next to an Oracle database and I immediately want to know where the bytes go and who is checking them. Preferably before I need those bytes back :)

Dbvisit keeps copies of online redo log files on the standby host. Sounds simple enough. Except Oracle keeps writing to the source files, switches logs and eventually reuses the very same files for another sequence. So our nice, innocent file copy has a moving target. Of course it does.

Let’s follow it from the first header read to the patch written on the standby. Then we can get to the part I like most: how much work Dbvisit can leave to Oracle itself.

A quick note before we start poking at offsets: this walk-through follows the Linux x86-64 binaries inspected in Ghidra and the extracted dbvctl 12.4.0 Perl modules. I’ve shortened the function names to their relevant package and receiver. The pseudocode below reconstructs the control flow; it isn’t the original vendor source.

First, who does what?

The documented ZDL architecture puts online redo replication alongside the usual archive synchronization. The archives keep doing their job. Meanwhile, the mirrored online redo files sit outside the standby database, ready to be used during activation. Keep that last bit in mind; we’ll come back to it.

On the primary side, most of the action is in dbvhelper. Its internal/redo/redolog package reads the redo layout and remembers how far it has got through each file. internal/redo/redosvc selects members, coordinates bootstrap and turns the changes into transfers. Across the network, the godbvnet receiver in dbvagentmanager creates or patches the mirror files.

PRIMARY
  selected filesystem redo member
    -> redolog.Tracker.GetChanges
    -> redosvc.replicator
    -> file transport
                         |
                         v
STANDBY HOST
  dbvagentmanager / godbvnet
    -> receiveFile   -> full mirror file
    -> receivePatch  -> overwrite a byte range

ACTIVATION
  dbvctl / Standby::Oracle
    -> catalog the mirror files
    -> ask Oracle to recover from them

There are two different jobs hiding in that diagram. Writing a patch into a redo copy is the Go receiver’s job. Applying that redo to database blocks belongs to Oracle recovery. The distinction matters quite a lot once we get to “apply”.

Find a redo member we can actually read

Before reading anything interesting, Dbvisit needs a file it can open. redosvc.SelectMembers takes the redo members reported by the database, accepts log statuses CURRENT, ACTIVE and INACTIVE, groups the candidates by thread and redo group, then calls selectOneValidMemberPerGroup.

When Dbvisit manages the replication, selection prefers the Local provider and needs a suitable member from every group. Candidates are sorted by their member path. Then isRedoMemberReadable does a rather modest preflight: open read-only, attempt a read, close. We haven’t earned the right to trust the redo yet; the detailed checks come later.

This explains the filesystem member requirement for native ZDL replication. The product requirements describe adding a filesystem member when the database’s redo is on ASM.

OK, now we can poke at the headers

Each selected file gets its own redolog.Tracker. Every time we ask Tracker.GetChanges for new bytes, it starts with redoLog.loadHeader. The helper opens the source read-only when first needed and caches its physical size. Subsequent calls reload the header through redoLog.readHeader; they don’t reopen and measure the whole file every time.

The first read requests 8192 bytes because we don’t know the redo block size yet. Once we do, two redo blocks are enough for the header read. Physical block 0 supplies the format information used here; block 1 carries the redo-file header, including the sequence and SCN/time fields. The data scan starts at block 2. Yes, we’re counting from zero.

I’ll use B for the redo block size. Here’s what the header reader looks at, with offsets measured from the beginning of the file:

And, because this is Oracle, “read the SCN” comes with a small detour. redoLogScnInterpreter.readSCN picks little- or big-endian decoding and handles both the legacy six-byte and extended eight-byte encodings. Grabbing an ordinary unsigned 64-bit integer from that slot would miss the interpretation.

An open CURRENT file can still have a next-SCN sentinel, and the reader is fine with that. The scan uses the physical file size and validated block contents to find how far it can go.

A valid checksum can still belong to the wrong redo

With the headers sorted out, redoLog.readRedoLogBlocks starts at block 2, or just after the last block the tracker has already emitted. It reads chunks of up to 1 MiB, and readBlocks examines the complete B-byte blocks inside them. A few leftover bytes at the end don’t qualify as another block.

For each block, validateBlockHeader checks its identity, then validateBlockChecksum checks its contents:

This is where file reuse gets interesting. An old block can have a perfectly valid checksum and still be completely wrong for the sequence we’re copying. Oracle hasn’t politely erased the old tail for us. So the block’s sequence has to match the file header exactly. An unexpected newer sequence fails the check too.

calcChSumWithEndian XORs 64-bit words across the block, folds the accumulator down by 32 and then 16 bits, removes the stored checksum’s contribution and compares the resulting 16-bit value with the stored field. A stored checksum of zero does not bypass the check in this implementation.

As soon as a header or checksum fails, the scan stops. It keeps the valid prefix before that block and leaves the failing block out. Another tracking pass can try again after Oracle has written more data. No jumping over the awkward block and hoping for the best.

start = max(2, previous_frontier + 1)

for each whole block in chunks of at most 1 MiB:
    require nonempty block header
    require block.number == expected physical block number
    require block.sequence == file.sequence
    require valid block checksum
    collect timestamp metadata
    extend the valid block range

stop at the first block that fails validation
return only the preceding valid range

One small detail before we move on: readAt makes a single File.ReadAt call. A short read with some bytes and EOF can still be useful, provided we have complete blocks to examine. There’s no “read it twice and compare” stability check inside this helper.

Same filename, different sequence

Remembering an offset alone would get us into trouble here. The tracker also keeps the file sequence, a cached copy of the second header block, the last emitted block and the state needed to parse records containing timestamps.

On the next call, it reloads the header. A sequence change resets the frontier and record-parsing state. With an unchanged sequence, it compares the newly read header block with its cached bytes.

getChangeSet can therefore return a header range independently of a data range:

The data range follows the validated scan. After successful change-set construction, the tracker saves the new sequence, header bytes, parsing state and greatest emitted block number.

So the filename can stay exactly the same while its redo sequence changes underneath us. The reader resets its state and avoids treating the old tail as fresh redo. That tracker reset is separate from restarting the full-file bootstrap. I’m pointing this out because it’s very easy to turn “new sequence” into “copy the whole file again” in your head.

The record helpers, including extractTimestampsFromBlock and spanningRecord.extractTimestamp, also handle timestamp extraction across record boundaries. That supports measurement of redo progress. The inspected tracking path publishes validated block ranges.

Copy the file. Oracle keeps writing, naturally.

We need something on the standby before we can patch it. This part starts with primaryMonitor.checkMembers building the selected-member state. In the native replication path, startReplicator calls replicator.start, which sets up the sender and reporter and invokes replicator.bootstrap.

Bootstrap announces its start, gathers file information for progress reporting and calls bootstrapper.run. That function iterates the selected paths and invokes sender.sendPlainFile for each whole source file. The destination is derived from the mirror directory and the source file’s basename.

Of course, Oracle can keep writing while we copy the file. It has transactions to deal with. So after the full copy, Dbvisit creates fresh trackers, collects their initial change ranges and sends them through replicator.processInitialChanges. That’s the catch-up phase.

select members
    -> start sender and reporter
    -> announce bootstrap
    -> send the whole selected files
    -> create trackers and collect initial ranges
    -> send catch-up patches
    -> enter Replicating state
    -> register file watches and queue initial events

Those initial patches can include both header blocks and the validated data prefix. Remember, online redo uses an existing file; looking for bytes appended beyond the end of our copy wouldn’t do the job. The full copy gives us a baseline, and the range transfers catch up with the active redo inside it.

If the selected-member map changes, the monitor cleans up and rebuilds its replication state, including bootstrap. The requirements guide also describes reinitializing with full copies after agent restarts or sufficiently long interruptions. Ordinary sequence changes use the tracker-reset path we’ve just looked at.

Something changed. Go and have a look.

Once catch-up is done, startDetector creates a filesystem watcher and a ticker. It registers the selected paths and immediately queues an event for each file. After that, filesystem notifications put the affected files back on the queue.

There’s also a ten-second ticker that queues all selected files. That gives the detector a periodic fallback alongside the filesystem notifications. The notifications can trigger scans between those ticks.

primaryMonitor.processFileEvent obtains the tracker for each affected path, calls Tracker.GetChanges and passes the resulting per-file ranges to replicator.processChanges. The surveyor also receives those changes for progress reporting.

I like this bit: the tracker already knows where it got to. It reads forward, checks the blocks and emits ranges, without comparing two complete redo files byte by byte.

From blocks to bytes, including that last block

Both processInitialChanges and processChanges translate inclusive block ranges into byte offsets and lengths:

offset = firstBlock * blockSize
length = (lastBlock - firstBlock + 1) * blockSize

Example: blockSize = 512, range = [100, 103]
offset = 51200
length = 2048

The last block is included. Forgetting the +1 would quietly leave one block behind. Oracle has enough ways to make recovery interesting without our help.

sender.patchFile delegates the range to the file-transport layer, where an io.SectionReader reads the requested interval from the source file. The receiving side decodes a patchParams structure with these fields:

// Reconstructed field layout; not a complete protocol implementation.
type patchParams struct {
    Path        string
    Compression int
    Offset      int64
    Length      int64
}

There are a couple of checks on the way in. The patch handler checks that the decompressed payload has the declared length. The generic stream receiver calculates a digest over those decompressed bytes and compares it with the digest supplied by the sender.

The digest in this binary is SHA-1. A compression value of zero means the receiver uses the incoming stream directly; the nonzero branch goes through a Zstandard reader. These transport checks have their own job. We still need the Oracle redo-block checks back on the source.

If the network gets involved in making our day worse, the sender’s wrapper can retry after a network error or EOF, using dialWithRetry within a timeout context. That retry belongs to the transport. The reader coming back later to a block that wasn’t valid yet is a separate thing.

Also, keep an eye on what crosses the boundary between tracker and sender: range metadata. The sender then reads that interval from the source again.

Meanwhile, on the standby...

For a whole bootstrap file, godbvnet.receiveFile writes to a temporary destination with a .part suffix. Once receipt and close succeed, it renames the file to its final destination.

A patch takes another route. First, godbvnet.receivePatch checks the parameters: offset and length cannot be negative, any configured path restriction must be satisfied, and the destination file must already exist. Then it acknowledges readiness and receives the payload into a bytes.Buffer.

Once the complete stream has arrived successfully and its byte count matches Length, we finally get to godbvnet.patchFile. After all that preparation, the actual write is pleasantly boring:

receive and verify the complete payload
require received_bytes == patch.Length

file = OpenFile(patch.Path, O_WRONLY)
file.Seek(patch.Offset, beginning)
CopyN(file, payload, patch.Length)
if not skipSync:
    file.Sync()
file.Close()

report the write result through the transfer handshake

The file is opened without O_CREATE or O_TRUNC. We overwrite the requested interval, and the bytes around it stay where they were. Including the old sequence contents beyond the new valid prefix. There’s that old tail again.

After the write path, the receiver reports the patch result and takes part in the final acknowledgement exchange. Keep the condition around File.Sync in mind: whether it runs depends on skipSync.

And finally, let Oracle do its job

We’ve now followed the bytes all the way to ordinary files on the standby host. During activation, the Perl code in Standby::Oracle brings Oracle into the picture to recover from those files.

First, _ora_catalog_online_redo catalogs the mirror-directory files using RMAN CATALOG ARCHIVELOG. Then _ora_query_latest_redos queries V$ARCHIVED_LOG: available entries, the relevant thread, sequences after the existing archive point and the current database’s RESETLOGS_CHANGE#. It orders them by sequence and returns the next SCN, next timestamp and filename for each entry.

Finally, _apply_online_redo_files builds the recovery command with the selected activation SCN, supplies the filenames and checks for Media recovery complete as well as Oracle errors. If new datafiles need to be added, it can repeat the recovery command. The short version looks like this:

_ora_catalog_online_redo
    -> RMAN: CATALOG ARCHIVELOG '<mirror file>'

_ora_query_latest_redos
    -> available V$ARCHIVED_LOG entries
    -> matching thread and resetlogs incarnation
    -> sequence order, next SCN/time, filenames

_apply_online_redo_files
    -> RECOVER STANDBY DATABASE UNTIL CHANGE <activation SCN> NOPARALLEL
    -> supply the selected redo files
    -> check recovery completion and Oracle errors

All in all, this is a pretty awesome mechanism. Dbvisit still needs to understand the headers, validate blocks and keep track of sequences, but it doesn’t have to reimplement Oracle recovery or dig into every bit and byte of LGWR and LWNs. It “patches” the redo copies with the original block contents and lets Oracle do the recovery work when the standby is activated. That’s when the mirrored online redo files get used for database recovery.

I’m really impressed! Clever, simple... 🍻