milksad

Custom fork of rust-bitcoin with unsafe modifications for higher speed. Unsuitable for production.

Go to file

Andrew Poelstra 1b3a9d3580 Merge rust-bitcoin/rust-bitcoin#1990 : Introduce the `small-hash` feature for `bitcoin_hashes` `f2c5f19557` Introduce the `small-hash` feature for `bitcoin_hashes` (Alekos Filini) Pull request description: When enabled this feature swaps the hash implementation of sha512, sha256 and ripemd160 for a smaller (but also slower) one. On embedded processors (Cortex-M4) it can lead to up to a 52% size reduction, from around 37KiB for just the `process_block` methods of the three hash functions to 17.8KiB. The following numbers were collected on `aarch64-unknown-linux-gnu` with `cargo 1.72.0-nightly`. ## Original ``` RUSTFLAGS='--cfg=bench -C opt-level=z' cargo bench ``` ``` test hash160::benches::hash160_10 ... bench: 33 ns/iter (+/- 1) = 303 MB/s test hash160::benches::hash160_1k ... bench: 2,953 ns/iter (+/- 187) = 346 MB/s test hash160::benches::hash160_64k ... bench: 188,480 ns/iter (+/- 11,595) = 347 MB/s test hmac::benches::hmac_sha256_10 ... bench: 33 ns/iter (+/- 2) = 303 MB/s test hmac::benches::hmac_sha256_1k ... bench: 2,957 ns/iter (+/- 104) = 346 MB/s test hmac::benches::hmac_sha256_64k ... bench: 192,022 ns/iter (+/- 6,407) = 341 MB/s test ripemd160::benches::ripemd160_10 ... bench: 25 ns/iter (+/- 1) = 400 MB/s test ripemd160::benches::ripemd160_1k ... bench: 2,288 ns/iter (+/- 93) = 447 MB/s test ripemd160::benches::ripemd160_64k ... bench: 146,823 ns/iter (+/- 1,102) = 446 MB/s test sha1::benches::sha1_10 ... bench: 41 ns/iter (+/- 0) = 243 MB/s test sha1::benches::sha1_1k ... bench: 3,844 ns/iter (+/- 70) = 266 MB/s test sha1::benches::sha1_64k ... bench: 245,854 ns/iter (+/- 10,158) = 266 MB/s test sha256::benches::sha256_10 ... bench: 35 ns/iter (+/- 0) = 285 MB/s test sha256::benches::sha256_1k ... bench: 3,063 ns/iter (+/- 15) = 334 MB/s test sha256::benches::sha256_64k ... bench: 195,729 ns/iter (+/- 2,880) = 334 MB/s test sha256d::benches::sha256d_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s test sha256d::benches::sha256d_1k ... bench: 3,071 ns/iter (+/- 107) = 333 MB/s test sha256d::benches::sha256d_64k ... bench: 188,614 ns/iter (+/- 8,101) = 347 MB/s test sha512::benches::sha512_10 ... bench: 21 ns/iter (+/- 0) = 476 MB/s test sha512::benches::sha512_1k ... bench: 1,714 ns/iter (+/- 36) = 597 MB/s test sha512::benches::sha512_64k ... bench: 110,084 ns/iter (+/- 3,637) = 595 MB/s test sha512_256::benches::sha512_256_10 ... bench: 22 ns/iter (+/- 1) = 454 MB/s test sha512_256::benches::sha512_256_1k ... bench: 1,822 ns/iter (+/- 70) = 562 MB/s test sha512_256::benches::sha512_256_64k ... bench: 116,231 ns/iter (+/- 4,745) = 563 MB/s test siphash24::benches::siphash24_1ki ... bench: 1,072 ns/iter (+/- 41) = 955 MB/s test siphash24::benches::siphash24_1ki_hash ... bench: 1,102 ns/iter (+/- 42) = 929 MB/s test siphash24::benches::siphash24_1ki_hash_u64 ... bench: 1,064 ns/iter (+/- 41) = 962 MB/s test siphash24::benches::siphash24_64ki ... bench: 69,957 ns/iter (+/- 2,712) = 936 MB/ ``` ``` 0000000000005872 t _ZN84_$LT$bitcoin_hashes..ripemd160..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hc4800746a9da7ff4E 0000000000007956 t _ZN81_$LT$bitcoin_hashes..sha256..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hf49345f65130ce9bE 0000000000008024 t _ZN14bitcoin_hashes6sha2568Midstate10const_hash17h57317bc8012004b4E.llvm.441255102889972912 0000000000010528 t _ZN81_$LT$bitcoin_hashes..sha512..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17h9bc868d4392bd9acE ``` Total size: 32380 bytes ## With `small-hash` enabled ``` RUSTFLAGS='--cfg=bench -C opt-level=z' cargo bench --features small-hash ``` ``` test hash160::benches::hash160_10 ... bench: 52 ns/iter (+/- 3) = 192 MB/s test hash160::benches::hash160_1k ... bench: 4,817 ns/iter (+/- 286) = 212 MB/s test hash160::benches::hash160_64k ... bench: 319,572 ns/iter (+/- 11,031) = 205 MB/s test hmac::benches::hmac_sha256_10 ... bench: 54 ns/iter (+/- 2) = 185 MB/s test hmac::benches::hmac_sha256_1k ... bench: 4,846 ns/iter (+/- 204) = 211 MB/s test hmac::benches::hmac_sha256_64k ... bench: 319,114 ns/iter (+/- 4,451) = 205 MB/s test ripemd160::benches::ripemd160_10 ... bench: 27 ns/iter (+/- 0) = 370 MB/s test ripemd160::benches::ripemd160_1k ... bench: 2,358 ns/iter (+/- 150) = 434 MB/s test ripemd160::benches::ripemd160_64k ... bench: 154,573 ns/iter (+/- 3,954) = 423 MB/s test sha1::benches::sha1_10 ... bench: 41 ns/iter (+/- 1) = 243 MB/s test sha1::benches::sha1_1k ... bench: 3,700 ns/iter (+/- 243) = 276 MB/s test sha1::benches::sha1_64k ... bench: 231,039 ns/iter (+/- 13,989) = 283 MB/s test sha256::benches::sha256_10 ... bench: 51 ns/iter (+/- 3) = 196 MB/s test sha256::benches::sha256_1k ... bench: 4,823 ns/iter (+/- 182) = 212 MB/s test sha256::benches::sha256_64k ... bench: 299,960 ns/iter (+/- 17,545) = 218 MB/s test sha256d::benches::sha256d_10 ... bench: 52 ns/iter (+/- 2) = 192 MB/s test sha256d::benches::sha256d_1k ... bench: 4,827 ns/iter (+/- 323) = 212 MB/s test sha256d::benches::sha256d_64k ... bench: 302,844 ns/iter (+/- 15,796) = 216 MB/s test sha512::benches::sha512_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s test sha512::benches::sha512_1k ... bench: 3,002 ns/iter (+/- 123) = 341 MB/s test sha512::benches::sha512_64k ... bench: 189,767 ns/iter (+/- 10,396) = 345 MB/s test sha512_256::benches::sha512_256_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s test sha512_256::benches::sha512_256_1k ... bench: 2,996 ns/iter (+/- 198) = 341 MB/s test sha512_256::benches::sha512_256_64k ... bench: 192,024 ns/iter (+/- 8,181) = 341 MB/s test siphash24::benches::siphash24_1ki ... bench: 1,081 ns/iter (+/- 65) = 947 MB/s test siphash24::benches::siphash24_1ki_hash ... bench: 1,083 ns/iter (+/- 63) = 945 MB/s test siphash24::benches::siphash24_1ki_hash_u64 ... bench: 1,084 ns/iter (+/- 63) = 944 MB/s test siphash24::benches::siphash24_64ki ... bench: 67,237 ns/iter (+/- 4,185) = 974 MB/s ``` ``` 0000000000005384 t _ZN81_$LT$bitcoin_hashes..sha256..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hae341658cf9b880bE 0000000000005608 t _ZN14bitcoin_hashes9ripemd16010HashEngine13process_block17h3276b13f1e9feef8E.llvm.13618235596061801146 0000000000005616 t _ZN14bitcoin_hashes6sha2568Midstate10const_hash17h3e6fbef64c15ee00E.llvm.7326223909590351031 0000000000005944 t _ZN81_$LT$bitcoin_hashes..sha512..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17h321a237bfbe5c0bbE ``` Total size: 22552 bytes ## Conclusion On `aarch64` there's overall a ~30% improvement in size, although ripemd160 doesn't really shrink that much (and its performance also aren't impacted much with only a 6% slowdown). sha512 and sha256 instead are almost 40% slower with `small-hash` enabled. I don't have performance numbers for other architectures, but in terms of size there was an even larger improvements on `thumbv7em-none-eabihf`, with a 52% size reduction overall: ``` Size Crate Name 25.3KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::sha512::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input 6.9KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::sha256::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input 4.8KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::ripemd160::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input ``` vs ``` Size Crate Name 9.5KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::sha512::HashEngine as bitcoin_hashes[974bb476ef905797]::HashEngine>::input 4.5KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::ripemd160::HashEngine>::process_block 3.8KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::sha256::HashEngine as bitcoin_hashes[974bb476ef905797]::HashEngine>::input ``` I'm assuming this is because on more limited architectures the compiler needs to use more instructions to move data in and out of registers (especially for sha512 which ideally would benefit from 64-bit registers), so reusing the code by moving it into functions saves a lot of those instructions. Also note that the `const_hash` method on `sha256` causes the compiler to emit two independent implementations. I haven't looked into the code yet, maybe there's a way to merge them so that the non-const `process_block` calls into the const fn. ----- Note: commits are unverified right now because I don't have the keys available, I will sign them after addressing the review comments. ACKs for top commit: apoelstra: ACK `f2c5f19557` tcharding: ACK `f2c5f19557` Tree-SHA512: 1d5eb56324c458660e2571e8cf59895dc31dae9c5427c7ed36f8a0e81ca2e9a0f39026f56b6803df03635cc8b66aee3bf5182d51ab8972d169d56bcfec33771c		2023-08-17 16:34:10 +00:00
.github/workflows	ci: fix run syntax in fuzz job	2023-05-08 22:11:40 +00:00
bitcoin	Merge rust-bitcoin/rust-bitcoin#1744 : Expose XOnlyPublicKey in crate root	2023-08-14 14:56:38 +00:00
contrib	Use spaces instead of tabs	2023-08-12 17:34:08 +02:00
fuzz	Move p2p::constants::Network to crate root	2023-08-01 16:46:59 +10:00
githooks	Add cargo fmt to pre-commit githook	2023-03-07 08:58:13 +11:00
hashes	Merge rust-bitcoin/rust-bitcoin#1990 : Introduce the `small-hash` feature for `bitcoin_hashes`	2023-08-17 16:34:10 +00:00
internals	internals: Remove hex module	2023-07-21 10:59:47 +10:00
logo	Add Rust-Bitcoin logo and project header.	2022-03-22 20:30:33 -06:00
.actrc	feat: Support running CI locally with `act`	2022-02-17 21:11:30 +01:00
.gitignore	fix fuzz.sh and cycle.sh to use generated lists of targets	2023-04-27 00:24:52 +00:00
CONTRIBUTING.md	policy: Add refactor carve out	2023-07-18 09:58:55 +10:00
Cargo-minimal.lock	Depend on hex-conservative	2023-07-21 10:59:46 +10:00
Cargo-recent.lock	Depend on hex-conservative	2023-07-21 10:59:46 +10:00
Cargo.toml	Remove path field from internals dependency	2023-07-18 10:27:46 +10:00
LICENSE	Add LICENSE file with CC0 in it	2014-07-18 17:37:13 -07:00
README.md	Refer to the location where the deps are pinned	2023-08-12 17:26:58 +02:00
SECURITY.md	rename bitcoin-private to bitcoin-internals	2023-05-26 19:49:52 +00:00
clippy.toml	Introduce the `small-hash` feature for `bitcoin_hashes`	2023-08-16 14:19:17 +02:00
rustfmt.toml	Enable formatting for hashes	2023-03-21 08:33:24 +11:00

README.md

Rust Bitcoin

Library with support for de/serialization, parsing and executing on data-structures and network messages related to Bitcoin.

Documentation

Supports (or should support)

De/serialization of Bitcoin protocol network messages
De/serialization of blocks and transactions
Script de/serialization
Private keys and address creation, de/serialization and validation (including full BIP32 support)
PSBT v0 de/serialization and all but the Input Finalizer role. Use rust-miniscript to finalize.

For JSONRPC interaction with Bitcoin Core, it is recommended to use rust-bitcoincore-rpc.

It is recommended to always use cargo-crev to verify the trustworthiness of each of your dependencies, including this one.

Known limitations

Consensus

This library must not be used for consensus code (i.e. fully validating blockchain data). It technically supports doing this, but doing so is very ill-advised because there are many deviations, known and unknown, between this library and the Bitcoin Core reference implementation. In a consensus based cryptocurrency such as Bitcoin it is critical that all parties are using the same rules to validate data, and this library is simply unable to implement the same rules as Core.

Given the complexity of both C++ and Rust, it is unlikely that this will ever be fixed, and there are no plans to do so. Of course, patches to fix specific consensus incompatibilities are welcome.

Support for 16-bit pointer sizes

16-bit pointer sizes are not supported and we can't promise they will be. If you care about them please let us know, so we can know how large the interest is and possibly decide to support them.

Documentation

Currently can be found on docs.rs/bitcoin. Patches to add usage examples and to expand on existing docs would be extremely appreciated.

Contributing

Contributions are generally welcome. If you intend to make larger changes please discuss them in an issue before PRing them to avoid duplicate work and architectural mismatches. If you have any questions or ideas you want to discuss please join us in #bitcoin-rust on libera.chat.

For more information please see ./CONTRIBUTING.md.

Minimum Supported Rust Version (MSRV)

This library should always compile with any combination of features on Rust 1.48.0.

To build with the MSRV you will need to pin serde (if you have the feature enabled)

# serde 1.0.157 uses syn 2.0 which requires edition 2021
cargo update -p serde_json --precise 1.0.99
cargo update -p serde --precise 1.0.156
cargo update -p quote --precise 1.0.30
cargo update -p proc-macro2 --precise 1.0.63
cargo update -p serde_test --precise 1.0.175

The above commands are sourced from ./contrib/test.sh.

External dependencies

We integrate with a few external libraries, most notably serde. These are available via feature flags. To ensure compatibility and MSRV stability we provide two lock files as a means of inspecting compatible versions: Cargo-minimal.lock containing minimal versions of dependencies and Cargo-recent.lock containing recent versions of dependencies tested in our CI.

We do not provide any guarantees about the content of these lock files outside of "our CI didn't fail with these versions". Specifically, we do not guarantee that the committed hashes are free from malware. It is your responsibility to review them.

Installing Rust

Rust can be installed using your package manager of choice or rustup.rs. The former way is considered more secure since it typically doesn't involve trust in the CA system. But you should be aware that the version of Rust shipped by your distribution might be out of date. Generally this isn't a problem for rust-bitcoin since we support much older versions than the current stable one (see MSRV section).

Building

The cargo feature std is enabled by default. At least one of the features std or no-std or both must be enabled.

Enabling the no-std feature does not disable std. To disable the std feature you must disable default features. The no-std feature only enables additional features required for this crate to be usable without std. Both can be enabled without conflict.

The library can be built and tested using cargo:

git clone git@github.com:rust-bitcoin/rust-bitcoin.git
cd rust-bitcoin
cargo build

You can run tests with:

cargo test

Please refer to the cargo documentation for more detailed instructions.

Building the docs

We build docs with the nightly toolchain, you may wish to use the following shell alias to check your documentation changes build correctly.

alias build-docs='RUSTDOCFLAGS="--cfg docsrs" cargo +nightly rustdoc --features="$FEATURES" -- -D rustdoc::broken-intra-doc-links'

Testing

Unit and integration tests are available for those interested, along with benchmarks. For project developers, especially new contributors looking for something to work on, we do:

Fuzz testing with Hongfuzz
Mutation testing with Mutagen
Code verification with Kani

There are always more tests to write and more bugs to find, contributions to our testing efforts extremely welcomed. Please consider testing code a first class citizen, we definitely do take PRs improving and cleaning up test code.

Unit/Integration tests

Run as for any other Rust project cargo test --all-features.

Benchmarks

We use a custom Rust compiler configuration conditional to guard the bench mark code. To run the bench marks use: RUSTFLAGS='--cfg=bench' cargo +nightly bench.

Mutation tests

We have started doing mutation testing with mutagen. To run these tests first install the latest dev version with cargo +nightly install --git https://github.com/llogiq/mutagen then run with RUSTFLAGS='--cfg=mutate' cargo +nightly mutagen.

Code verification

We have started using kani, install with cargo install --locked kani-verifier (no need to run cargo kani setup). Run the tests with cargo kani.

Pull Requests

Every PR needs at least two reviews to get merged. During the review phase maintainers and contributors are likely to leave comments and request changes. Please try to address them, otherwise your PR might get closed without merging after a longer time of inactivity. If your PR isn't ready for review yet please mark it by prefixing the title with WIP: .

CI Pipeline

The CI pipeline requires approval before being run on each MR.

In order to speed up the review process the CI pipeline can be run locally using act. The fuzz and Cross jobs will be skipped when using act due to caching being unsupported at this time. We do not actively support act but will merge PRs fixing act issues.

Githooks

To assist devs in catching errors before running CI we provide some githooks. If you do not already have locally configured githooks you can use the ones in this repository by running, in the root directory of the repository:

git config --local core.hooksPath githooks/

Alternatively add symlinks in your .git/hooks directory to any of the githooks we provide.

Policy on Altcoins/Altchains

Since the altcoin landscape includes projects which frequently appear and disappear, and are poorly designed anyway we do not support any altcoins. Supporting Bitcoin properly is already difficult enough and we do not want to increase the maintenance burden and decrease API stability by adding support for other coins.

Our code is public domain so by all means fork it and go wild :)

Release Notes

Release notes are done per crate, see:

Licensing

The code in this project is licensed under the Creative Commons CC0 1.0 Universal license. We use the SPDX license list and SPDX IDs.