Derby Stallion 64: Derby Stallion 64 Prototype Corrective Patch

This patch corrects periodic and predictable corruption errors in the prototype of Derby Stallion 64 released June 25, 2017. It will now run on console under its original checksum, and damaged compressed files have been restored when possible.

There is NO replacement for a good dump. As many errors as could be detected have been corrected, but it is expected vastly more remain. The frequency of corruption was much higher than usual in this prototype, demanding ~460 to correct the boot sector and hundreds of corrections following.

FLASHram is required in order to save data. Failing to provide it causes some menus to permanetly loop during read or write operations.


Apply the xdelta patch using a current version of the xdelta patcher from the predictably named xdelta.org. The patch will only apply to a ROM in native (big-endian) byteorder.
At time of writing only one prototype has been publicly released. If more should follow, only apply to the June 25, 2017 release.


This is part of a series of corrective patches for prototypes exhibiting what’s known as “address corruption” while dumping. In short, when the values being read from the cart meet certain conditions, those values will be replaced with others at a mask of the current address. Both the data and its source are predictable, unlike more common signal noise errors.
This is NOT a criticism of any particular method or person. These patches are only a stopgap to raise awareness that an issue exists, can be detected, and therefore prevented.

Derby Stallion 64 Prototype Corrective Patch

.: Introduction :.
This patch corrects periodic and predictable corruption errors in the prototype of Derby Stallion 64 released June 25, 2017. It will now run on console under its original checksum, and damaged compressed files have been restored when possible.

There is NO replacement for a good dump. As many errors as could be detected have been corrected, but it is expected vastly more remain. The frequency of corruption was much higher than usual in this prototype, demanding ~460 to correct the boot sector and hundreds of corrections following.

FLASHram is required in order to save data. Failing to provide it causes some menus to permanetly loop during read or write operations.


.: Patching :.
Apply the xdelta patch using a current version of the xdelta patcher from the predictably named xdelta.org. The patch will only apply to a ROM in native (big-endian) byteorder.
At time of writing only one prototype has been publicly released. If more should follow, only apply to the June 25, 2017 release.

*Common Names
Derby Stallion 64 (Prototype)
Derby Stallion 64 (2001)(Media Factory)(JP)(beta).bin
*MD5
1051E1402EE110F3C5E372C9E1C5B338
*SHA512
6CAF61D0D8344F17587743ECD8D8970B89D77A22248998ECB1A74168C8B473178F8A5BB55C91F3C642D01C8B7E889FE54D775E335B9A5F6A53FBAF21DE37DC25

*Patched MD5
17D9D80229F59887A71ECA5FAF7F138F
*Patched SHA512
C635832CFDCFB252302114A60588C8551710CE644B6C78F9CC226D2B61A5ED2DDFE1D055A37267C8C42C297E046915EC9F2FD27F2E5AF3ED9A618319286AA5C6


.: Reliability of Data :.
In this project's case, with so many corrections made and so much potential for errors remaining, it's important to mention the methods used and reliability of data.

Address corruption itself is predictable and leaves a clear indicator, but simply searching for all possible addresses meeting the qualifications will produce hundreds of thousands of false-positives. (For more information see ".: About the Project :.") It's best used over select regions, when you already know an error is likely and need a list of condidates.
Direct comparsion is used whenever possible. Microcode was provided to developers and any specific version can be assumed immutable. Most variants also have patterns that meet the qualifications: runs of 0000FFFF, 00007FFF, etc. IPL3, also known as the bootstrap, is shared between all software using the same CIC chip. Anything shared with the retail release can also be directly compared.
This prototype is followed by a second copy of itself starting at 0x1B676C0. Although also subject to address corruption, because of the offset in the base address it repeats at a different rate and there's always potential corruption doesn't occur in one of the two cases. So long as both samples aren't wrong in the same way, an error in one can be detected. In this project, only 9 of the shared 581 errors detected were missed when comparing against overdump.
This title's internal checksum covers the first 1MB of code, starting at 0x1000. It is required this value is correct in order to boot on hardware; a bad checksum causes a permaloop in IPL3, and changes to IPL3 cause a permaloop in IPL2. Although not as reliable as, say, a SHA512 hash it is sufficient to verify that region of code and data has been corrected.

Data verification is the primary issue. Effectively, you need a guide or standard. Any random binary can only be verified by reverse-engineering what it is, how it is used, and what the data is expected to be. Regular data--tables, headers, structures--follow a pattern. Images need to look like something, and near colors can be differentiated looking at the binary.
A bad audio wave sample though? It would need to be remarkably bad for a human to notice a single sample is off at several thousand Hz. Likewise for predictors. The lower half of a float value is unlikely to have impact on whatever it's used for, though there are exceptional cases where 1/10000th of rounding error causes a missed jump. For anything else, the file's use needs to be reverse-engineered. The only clue it may be wrong is an occational glitch--in a prototype not yet bugtested.
In general code isn't subject to address corruption but there are exceptions like permaloops (written 1000FFFF). ASM can be sorted out for invalid instructions or nonsense code automatically and quickly. Standard libraries can be verified if you built signatures from samples in the SDK. Beyond that requires making a table of all used addresses and testing if functions in fact start where they claim. It can also act as a guide when looking at memory, but this can't be automated.
Errors in compressed files can be found when they affect the output size or fail to decompress. For some varieties not even a single byte can be wrong. A few also provide checksums to test against, but this is exceptional.

This title uses two LZ variants and only store output length. Output will fail or produce the wrong length when either the control byte/HW or any 16/24bit dictionary lookup is affected. Literals will only produce the wrong data. In the worst case, the literal rate is 16 out of every 18 bytes.
There are 2715 compressed files total. A subset of these are identical or near-identical to retail (file indicies may differ, for instance). The remaining majority were only checked when decompression failed. *Data verification was not performed and uncaught errors are presumed.
A more thorough test would involve painting which parts of the file were verifiable (ctrl/cmd/dict) and overlaying which are potential candidates for address corruption. From there, specialized decompressors would log where in the decompressed output the values are used (and reused) to aid in verifying the output. Presuming none fall within obviously patterned data this still requires reversing where and how the file is used, but in a vastly more directed way. *It's also a project for somebody else ;*)

Note all of this presumes no outliers are present. There's no rule the universe can only throw one curveball at a time. For that reason, it's important to remember there's no cure for a bad dump other than a good dump.


.: About the Project :.
This is part of a series of corrective patches for prototypes exhibiting what's known as "address corruption" while dumping. In short, when the values being read from the cart meet certain conditions, those values will be replaced with others at a mask of the current address. Both the data and its source are predictable, unlike more common signal noise errors. A longer (winded) explanation can be found here:
https://pastebin.com/UpzXVu3m

These patches try to correct as much of this damage as possible. The goals are:
*) the file is playable on hardware, in case the original data is lost before redumping,
*) act as a guide when a new dump is taken to help confirm it's accurate,
*) determine which methods or setups are subject to this problem, to either detect or avoid future problems,
*) and making a clear case a redump is necessary, as some people take this personally.
This is NOT a criticism of any particular method or person. These patches are only a stopgap to raise awareness that an issue exists, can be detected, and therefore prevented.

-Zoinkity


*Common Names
Derby Stallion 64 (Prototype)
Derby Stallion 64 (2001)(Media Factory)(JP)(beta).bin
*MD5
1051E1402EE110F3C5E372C9E1C5B338
*SHA512
6CAF61D0D8344F17587743ECD8D8970B89D77A22248998ECB1A74168C8B473178F8A5BB55C91F3C642D01C8B7E889FE54D775E335B9A5F6A53FBAF21DE37DC25