f2c5f19557 Introduce the `small-hash` feature for `bitcoin_hashes` (Alekos Filini)
Pull request description:
When enabled this feature swaps the hash implementation of sha512, sha256 and ripemd160 for a smaller (but also slower) one.
On embedded processors (Cortex-M4) it can lead to up to a 52% size reduction, from around 37KiB for just the `process_block` methods of the three hash functions to 17.8KiB.
The following numbers were collected on `aarch64-unknown-linux-gnu` with `cargo 1.72.0-nightly`.
## Original
```
RUSTFLAGS='--cfg=bench -C opt-level=z' cargo bench
```
```
test hash160::benches::hash160_10 ... bench: 33 ns/iter (+/- 1) = 303 MB/s
test hash160::benches::hash160_1k ... bench: 2,953 ns/iter (+/- 187) = 346 MB/s
test hash160::benches::hash160_64k ... bench: 188,480 ns/iter (+/- 11,595) = 347 MB/s
test hmac::benches::hmac_sha256_10 ... bench: 33 ns/iter (+/- 2) = 303 MB/s
test hmac::benches::hmac_sha256_1k ... bench: 2,957 ns/iter (+/- 104) = 346 MB/s
test hmac::benches::hmac_sha256_64k ... bench: 192,022 ns/iter (+/- 6,407) = 341 MB/s
test ripemd160::benches::ripemd160_10 ... bench: 25 ns/iter (+/- 1) = 400 MB/s
test ripemd160::benches::ripemd160_1k ... bench: 2,288 ns/iter (+/- 93) = 447 MB/s
test ripemd160::benches::ripemd160_64k ... bench: 146,823 ns/iter (+/- 1,102) = 446 MB/s
test sha1::benches::sha1_10 ... bench: 41 ns/iter (+/- 0) = 243 MB/s
test sha1::benches::sha1_1k ... bench: 3,844 ns/iter (+/- 70) = 266 MB/s
test sha1::benches::sha1_64k ... bench: 245,854 ns/iter (+/- 10,158) = 266 MB/s
test sha256::benches::sha256_10 ... bench: 35 ns/iter (+/- 0) = 285 MB/s
test sha256::benches::sha256_1k ... bench: 3,063 ns/iter (+/- 15) = 334 MB/s
test sha256::benches::sha256_64k ... bench: 195,729 ns/iter (+/- 2,880) = 334 MB/s
test sha256d::benches::sha256d_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s
test sha256d::benches::sha256d_1k ... bench: 3,071 ns/iter (+/- 107) = 333 MB/s
test sha256d::benches::sha256d_64k ... bench: 188,614 ns/iter (+/- 8,101) = 347 MB/s
test sha512::benches::sha512_10 ... bench: 21 ns/iter (+/- 0) = 476 MB/s
test sha512::benches::sha512_1k ... bench: 1,714 ns/iter (+/- 36) = 597 MB/s
test sha512::benches::sha512_64k ... bench: 110,084 ns/iter (+/- 3,637) = 595 MB/s
test sha512_256::benches::sha512_256_10 ... bench: 22 ns/iter (+/- 1) = 454 MB/s
test sha512_256::benches::sha512_256_1k ... bench: 1,822 ns/iter (+/- 70) = 562 MB/s
test sha512_256::benches::sha512_256_64k ... bench: 116,231 ns/iter (+/- 4,745) = 563 MB/s
test siphash24::benches::siphash24_1ki ... bench: 1,072 ns/iter (+/- 41) = 955 MB/s
test siphash24::benches::siphash24_1ki_hash ... bench: 1,102 ns/iter (+/- 42) = 929 MB/s
test siphash24::benches::siphash24_1ki_hash_u64 ... bench: 1,064 ns/iter (+/- 41) = 962 MB/s
test siphash24::benches::siphash24_64ki ... bench: 69,957 ns/iter (+/- 2,712) = 936 MB/
```
```
0000000000005872 t _ZN84_$LT$bitcoin_hashes..ripemd160..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hc4800746a9da7ff4E
0000000000007956 t _ZN81_$LT$bitcoin_hashes..sha256..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hf49345f65130ce9bE
0000000000008024 t _ZN14bitcoin_hashes6sha2568Midstate10const_hash17h57317bc8012004b4E.llvm.441255102889972912
0000000000010528 t _ZN81_$LT$bitcoin_hashes..sha512..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17h9bc868d4392bd9acE
```
Total size: 32380 bytes
## With `small-hash` enabled
```
RUSTFLAGS='--cfg=bench -C opt-level=z' cargo bench --features small-hash
```
```
test hash160::benches::hash160_10 ... bench: 52 ns/iter (+/- 3) = 192 MB/s
test hash160::benches::hash160_1k ... bench: 4,817 ns/iter (+/- 286) = 212 MB/s
test hash160::benches::hash160_64k ... bench: 319,572 ns/iter (+/- 11,031) = 205 MB/s
test hmac::benches::hmac_sha256_10 ... bench: 54 ns/iter (+/- 2) = 185 MB/s
test hmac::benches::hmac_sha256_1k ... bench: 4,846 ns/iter (+/- 204) = 211 MB/s
test hmac::benches::hmac_sha256_64k ... bench: 319,114 ns/iter (+/- 4,451) = 205 MB/s
test ripemd160::benches::ripemd160_10 ... bench: 27 ns/iter (+/- 0) = 370 MB/s
test ripemd160::benches::ripemd160_1k ... bench: 2,358 ns/iter (+/- 150) = 434 MB/s
test ripemd160::benches::ripemd160_64k ... bench: 154,573 ns/iter (+/- 3,954) = 423 MB/s
test sha1::benches::sha1_10 ... bench: 41 ns/iter (+/- 1) = 243 MB/s
test sha1::benches::sha1_1k ... bench: 3,700 ns/iter (+/- 243) = 276 MB/s
test sha1::benches::sha1_64k ... bench: 231,039 ns/iter (+/- 13,989) = 283 MB/s
test sha256::benches::sha256_10 ... bench: 51 ns/iter (+/- 3) = 196 MB/s
test sha256::benches::sha256_1k ... bench: 4,823 ns/iter (+/- 182) = 212 MB/s
test sha256::benches::sha256_64k ... bench: 299,960 ns/iter (+/- 17,545) = 218 MB/s
test sha256d::benches::sha256d_10 ... bench: 52 ns/iter (+/- 2) = 192 MB/s
test sha256d::benches::sha256d_1k ... bench: 4,827 ns/iter (+/- 323) = 212 MB/s
test sha256d::benches::sha256d_64k ... bench: 302,844 ns/iter (+/- 15,796) = 216 MB/s
test sha512::benches::sha512_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s
test sha512::benches::sha512_1k ... bench: 3,002 ns/iter (+/- 123) = 341 MB/s
test sha512::benches::sha512_64k ... bench: 189,767 ns/iter (+/- 10,396) = 345 MB/s
test sha512_256::benches::sha512_256_10 ... bench: 34 ns/iter (+/- 1) = 294 MB/s
test sha512_256::benches::sha512_256_1k ... bench: 2,996 ns/iter (+/- 198) = 341 MB/s
test sha512_256::benches::sha512_256_64k ... bench: 192,024 ns/iter (+/- 8,181) = 341 MB/s
test siphash24::benches::siphash24_1ki ... bench: 1,081 ns/iter (+/- 65) = 947 MB/s
test siphash24::benches::siphash24_1ki_hash ... bench: 1,083 ns/iter (+/- 63) = 945 MB/s
test siphash24::benches::siphash24_1ki_hash_u64 ... bench: 1,084 ns/iter (+/- 63) = 944 MB/s
test siphash24::benches::siphash24_64ki ... bench: 67,237 ns/iter (+/- 4,185) = 974 MB/s
```
```
0000000000005384 t _ZN81_$LT$bitcoin_hashes..sha256..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17hae341658cf9b880bE
0000000000005608 t _ZN14bitcoin_hashes9ripemd16010HashEngine13process_block17h3276b13f1e9feef8E.llvm.13618235596061801146
0000000000005616 t _ZN14bitcoin_hashes6sha2568Midstate10const_hash17h3e6fbef64c15ee00E.llvm.7326223909590351031
0000000000005944 t _ZN81_$LT$bitcoin_hashes..sha512..HashEngine$u20$as$u20$bitcoin_hashes..HashEngine$GT$5input17h321a237bfbe5c0bbE
```
Total size: 22552 bytes
## Conclusion
On `aarch64` there's overall a ~30% improvement in size, although ripemd160 doesn't really shrink that much (and its performance also aren't impacted much with only a 6% slowdown). sha512 and sha256 instead are almost 40% slower with `small-hash` enabled.
I don't have performance numbers for other architectures, but in terms of size there was an even larger improvements on `thumbv7em-none-eabihf`, with a 52% size reduction overall:
```
Size Crate Name
25.3KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::sha512::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input
6.9KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::sha256::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input
4.8KiB bitcoin_hashes <bitcoin_hashes[fe467ef2aa3a1470]::ripemd160::HashEngine as bitcoin_hashes[fe467ef2aa3a1470]::HashEngine>::input
```
vs
```
Size Crate Name
9.5KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::sha512::HashEngine as bitcoin_hashes[974bb476ef905797]::HashEngine>::input
4.5KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::ripemd160::HashEngine>::process_block
3.8KiB bitcoin_hashes <bitcoin_hashes[974bb476ef905797]::sha256::HashEngine as bitcoin_hashes[974bb476ef905797]::HashEngine>::input
```
I'm assuming this is because on more limited architectures the compiler needs to use more instructions to move data in and out of registers (especially for sha512 which ideally would benefit from 64-bit registers), so reusing the code by moving it into functions saves a lot of those instructions.
Also note that the `const_hash` method on `sha256` causes the compiler to emit two independent implementations. I haven't looked into the code yet, maybe there's a way to merge them so that the non-const `process_block` calls into the const fn.
-----
Note: commits are unverified right now because I don't have the keys available, I will sign them after addressing the review comments.
ACKs for top commit:
apoelstra:
ACK f2c5f19557
tcharding:
ACK f2c5f19557
Tree-SHA512: 1d5eb56324c458660e2571e8cf59895dc31dae9c5427c7ed36f8a0e81ca2e9a0f39026f56b6803df03635cc8b66aee3bf5182d51ab8972d169d56bcfec33771c
546c0122d7 Add simd sha256 intrinsics for x86 machines (sanket1729)
Pull request description:
This is my first time dabbling into architecture specific code and simd. The algorithm is a word to word translation of the C code from 4899efc81d/sha256-x86.c .
Some benchmarks:
With simd
```
test sha256::benches::sha256_10 ... bench: 11 ns/iter (+/- 0) = 909 MB/s
test sha256::benches::sha256_1k ... bench: 712 ns/iter (+/- 2) = 1438 MB/s
test sha256::benches::sha256_64k ... bench: 45,597 ns/iter (+/- 189) = 1437 MB/s
```
Without simd
```
test sha256::benches::sha256_10 ... bench: 47 ns/iter (+/- 0) = 212 MB/s
test sha256::benches::sha256_1k ... bench: 4,243 ns/iter (+/- 17) = 241 MB/s
test sha256::benches::sha256_64k ... bench: 271,263 ns/iter (+/- 1,610) = 241 MB/s
```
ACKs for top commit:
apoelstra:
ACK 546c0122d7
tcharding:
ACK 546c0122d7
Tree-SHA512: 7167c900b77e63cf38135a3960cf9ac2615f73b2ef7020a12b5cc3f4c047910063ba9045217b9ecfa70f7de1eb0f02f2674f291bd023a853bad2b9162fae831e
When enabled this feature swaps the hash implementation of sha512,
sha256 and ripemd160 for a smaller (but also slower) one.
On embedded processors (Cortex-M4) it can lead to up to a 52% size
reduction, from around 37KiB for just the `process_block` methods of the
three hash functions to 17.8KiB.
02890d3fba Use spaces instead of tabs (yancy)
a640f0f034 Refer to the location where the deps are pinned (yancy)
Pull request description:
A rollup of two simple commits. 1) add a line to point developers to the CI test where the deps are tested and 2) replace tabs with spaces (spaces are used everywhere else) in `contrib/test.sh`.
ACKs for top commit:
apoelstra:
ACK 02890d3fba
tcharding:
ACK 02890d3fba
Tree-SHA512: 29b02e6d91a169429fe931a97ff7f86ac99016c6356f85d887a84141b3eec57527f85a43aeb3905a4acc73730e33a505c265b6a22ffbca1b102b63adf8599e4f
fa10668a35 Eliminate a heap allocation from PartialMerkleTree encoding & decoding (Steven Roose)
Pull request description:
Just came across this and felt like doing this.
ACKs for top commit:
apoelstra:
ACK fa10668a35
tcharding:
ACK fa10668a35
Tree-SHA512: 7167c63077851c4c461b33292948d9b09fe21eb45d52ed278ecf884ce15bc8b21c14040fa1eb5f50bfe51c5cde10abc133d0c59be502de139408c0d107ffa7eb
63d0fa0164 Rename All to Opcode (Tobin C. Harding)
881d1b1a9d Move opcode aliases to the all module (Tobin C. Harding)
f3e8dbc392 Remove floating code comment (Tobin C. Harding)
dfd9384d14 opcodes: Fix whitespace (Tobin C. Harding)
Pull request description:
I may be missing something about the `All` type but it seems it can be re-named to `Opcode` with no loss of clarity.
The first few patches do some other clean ups, the last patch is the meat and potatoes.
ACKs for top commit:
apoelstra:
ACK 63d0fa0164
Tree-SHA512: c1b8be9e0c090d015c2bbfcb7ed6abe998ccada5a1eb0f6966c2a7dc462f8bbc45e6b99bf1387c736d2fca21a993a2c7fd5844c18b97a2e37ec43da517566ab1
894aa130b6 Add CI test for pinned versions using MSRV (yancy)
Pull request description:
Add a test for pinned version with current MSRV. I'm not sure how to test this in CI before opening this PR (will probably fail).
ACKs for top commit:
apoelstra:
ACK 894aa130b6
tcharding:
ACK 894aa130b6
Tree-SHA512: ab6a68628d486f8a3aa47ad6921fd90a60df62f99f3bf017d4aac092510b226b3acb8d1ee3b80e9884006bfc746f4ec0d7f150df104ec30a2e0da67886ebd3cd
2e3006a729 Add max standard tx weight constant to transaction (yancy)
Pull request description:
Add a constant for the max transaction weight. Similar to [max block weight](1b009b809b/bitcoin/src/blockdata/weight.rs (L35)). This value is pulled from core [here](44b05bf3fe/src/policy/policy.h (L27))
ACKs for top commit:
apoelstra:
ACK 2e3006a729
sanket1729:
ACK 2e3006a729
Tree-SHA512: 1583695f43387538f948be85ded7ff9a4bf9778169acb958debcbe1572a6dc8bfcd26ddfb8dbe0c030c98ab1f8a66d239a5bc663bf65ec3376a46d5f71e90894
9eff2f2f5e fee_rate: Add public absolute weight convenience functions (Tobin C. Harding)
f00e93bdcd Fix typos in rustdoc (Tobin C. Harding)
f3412325ea weight: Make docs uniform and terse (Tobin C. Harding)
Pull request description:
- Patch 1 is docs cleanup
- Patch 2 adds two functions to `FeeRate`
From the commit log of patch 2:
Calculating the absolute fee from a fee rate can currently be achieved
by creating a `Weight` object and using
`FeeRate::checked_mul_by_weight`. This is kind of hard to discover, we
can add two public convenience functions that make discovery of the
functionality easier.
Add two functions for calculating the absolute fee by multiplying by
weight units (`Weight`) and virtual bytes (by first converting to weight
units).
This seems like an obvious thing so I'm inclined to think that Kixunil left it out for a reason. (Mentioning you here Kix so even if this merges you'll see it in notifications later on.)
ACKs for top commit:
sanket1729:
utACK 9eff2f2f5e
apoelstra:
ACK 9eff2f2f5e
Tree-SHA512: 5b3997b721b7d7225bcf0e8a4a3efb6f207393541dbbcef533135af06a4d9c95210d450bb2322fd65a727e4560b29f0b20fb96c154d019fd4e745506213abc1c
154552e334 docs: Do not link to std::option::Option (Tobin C. Harding)
24843468c3 Remove rustdocs links to serde (Tobin C. Harding)
Pull request description:
Two minor patches to fix up docs links. These were originally done as part of #1880 but are unrelated so pushing them up separately.
ACKs for top commit:
apoelstra:
ACK 154552e334
RCasatta:
utACK 154552e334
Tree-SHA512: e45e1538c66b59d63a66898896927bb6c1336fb4c8515bb9e2204c8035870ef8e4a6fd32dfc83db2938afda67feb27c48989e382410f9e7ea7a967132941c720
27b3c1e0e6 Improve the ScriptHash and WScriptHash types (Tobin C. Harding)
2197f1377f Improve PubkeyHash and WPubkeyHash (Tobin C. Harding)
Pull request description:
Total re-write since review. Now this PR moves the hash type definitions out of `hash_types`. Please see https://github.com/rust-bitcoin/rust-bitcoin/issues/1909#issuecomment-1603634440 for more.
No longer adds unit tests.
Fix: #1909
ACKs for top commit:
apoelstra:
ACK 27b3c1e0e6
Tree-SHA512: 216b9bed05d1a4a4fc493262664ceb5d60f9c30685b63d6f6675d21a7bf811053320a002165487b29599c52f345057d9c92babb0fc1ccd4628671ec468c804f9
50ada8298f Move EncodeSigningDataResult to sighash module (Tobin C. Harding)
1b7dc51ccb Remove deprecated code (Tobin C. Harding)
Pull request description:
We only keep deprecated code around for one release so we can now remove code deprecated in v0.30.0
Done in preparation as we gear up for v0.31.0 release.
ACKs for top commit:
apoelstra:
ACK 50ada8298f
sanket1729:
ACK 50ada8298f
Tree-SHA512: 40769258605563e2e12a6118306655fc9a012ae1f86509fca757ca411f0cef74480b7bb7b0db147f30a7d362b8494a077d5ec04f719351661ceb5a0697a5369d
3c0bb63423 Do trivial rustdoc improvements (Tobin C. Harding)
3225aa9556 Use defensive documentation (Tobin C. Harding)
80d5d6665a crypto: key: Move error code to the bottom of the file (Tobin C. Harding)
fe3b1e1140 Move From for Error impl (Tobin C. Harding)
5f8e0ad67e Fix docs on error type (Tobin C. Harding)
f23155aa16 Do not capitalize error messages (Tobin C. Harding)
ae07786c27 Add InvalidSighashTypeError (Tobin C. Harding)
baba0fde57 Put NonStandardSighashTypeError inside ecdsa::Error variant (Tobin C. Harding)
6c9d9d9c36 Improve error display imlps (Tobin C. Harding)
22c7aa8808 Rename non standard sighash error type (Tobin C. Harding)
Pull request description:
EDIT: The commit hashes below are stale but the text is valid still.
In an effort to "perfect" our error handling, overhaul the error handling in the `crypto` module.
The aim is to do a small chunk so we can bikeshed on it then I can apply the learnings to the rest of the codebase.
Its all pretty trivial except:
- commit `4c180277 Put NonStandardSighashTypeError inside ecdsa::Error variant`
- comimt `5a196535 Add InvalidSighashTypeError`
- commit `05772ade Use defensive documentation`
Particularly the last one might be incorrect/controversial.
Also, please take the time to check the overall state of error code in the `crypto` module on this branch in case there is anything else we want to do.
Thanks
ACKs for top commit:
apoelstra:
ACK 3c0bb63423
Tree-SHA512: 7e5f8590aec5826098d4d8d33351a41b10c42b6379ff86e5b889e73271b71921fc3ca9525baa5da53e07fa2e961e710393694e04658a8243799950b4604caf43
Improve the script hash types by doing:
- Define the types in the `crypto::script` module
- Put the From impls directly below the type definitions
Keep the current crate level re-export so this does not impact the
public API _if_ people are using the re-export but is still a breaking
change.
Improve the pubkey hash types by doing:
- Define the types in the `crypto::key` module
- Add From<&PublicKey> impl for `PubkeyHash`
Keep the current crate level re-export so this does not impact the
public API _if_ people are using the re-export but is still a breaking
change.
Calculating the absolute fee from a fee rate can currently be achieved
by creating a `Weight` object and using
`FeeRate::checked_mul_by_weight`. This is kind of hard to discover, we
can add two public convenience functions that make discovery of the
functionality easier.
Add two functions for calculating the absolute fee by multiplying by
weight units (`Weight`) and virtual bytes (by first converting to weight
units).
d4e8f49fc3 Move p2p::constants::Network to crate root (Tobin C. Harding)
0f78943ef0 Move p2p::constants::Magic to p2p module (Tobin C. Harding)
d9d5a4ed4f Move p2p::constants::ServiceFlags to p2p module (Tobin C. Harding)
99d8ae1173 Improve rustdocs on PROTOCOL_VERSION (Tobin C. Harding)
4330722d62 Move p2p::constants::PROTOCOL_VERSION to p2p module (Tobin C. Harding)
1bac1fd518 Rename the network module to p2p (Tobin C. Harding)
dac97072a5 Add BorrowMut to prelude (Tobin C. Harding)
Pull request description:
This PR has grown.
- Rename the `network` module to `p2p` (fix: #1855)
- add `BorrowMut` to prelude
- Move the `Network`, `ChainHash`, and `Magic` structs to a newly create `p2p::network` module
- Do a few cleanups
ACKs for top commit:
apoelstra:
ACK d4e8f49fc3
Tree-SHA512: 7cee93d36363b8acbece6c92c6afc34bb3abe6d5801373d291db1323c934e7c384f3b5881cefb8b0d102dea53da6ed8da1bf60c324bf60d5dab3af598e0310da
55be538dac policy: Add refactor carve out (Tobin C. Harding)
Pull request description:
I have managed to burn out or bore our reviewers/maintainers. Getting two acks is becoming increasingly difficult. I've pestered everyone to the limit that I feel socially comfortable doing so, and as such, am requesting a carve out to the 2-ACK before merge rule.
The primary justification is that I feel we should have a bit more of BDFL and a bit less total consensus if we are to push forwards.
### Example PRs where this change would apply
- https://github.com/rust-bitcoin/rust-bitcoin/pull/1925
- https://github.com/rust-bitcoin/rust-bitcoin/pull/1854
- https://github.com/rust-bitcoin/rust-bitcoin/pull/1862
ACKs for top commit:
elichai:
I agree this makes sense for refactors ACK 55be538dac
apoelstra:
ACK 55be538dac
sanket1729:
ACK 55be538dac. Same reasons as apoelstra. And this is only for re-factors that are not adding any new features.
RCasatta:
ACK 55be538dac
Tree-SHA512: a5e206252015f49245ed282a3be7a35760d16f94dc6e60f31edf589a41ef642eba52a3bd7d1375b6033f3cf0128f47beee4f03e59cad151c64eedd71ac98baac
The `ServiceFlags` type is used by the p2p layer. It can live in the
`mod.rs` file of the `p2p` module. Done in preparation for removing the
`p2p::constants` module.
This is a straight code move, the `ServiceFlags` replaces the
current re-export.
The `PROTOCOL_VERSION` const is a p2p layer constant. It can live in the
`mod.rs` file of the `p2p` module.
This is a straight code move, the `PROTOCOL_VERSION` replaces the
current re-export.
The `network` module deals with data types and logic related to
internetworking bitcoind nodes, this is commonly referred to as the p2p
layer.
Rename the `network` module to `p2p` and fix all the paths.
Only commit in the docs and error messages to what we _really_ know.
In an attempt to reduce the likelyhood of the code going stale only
commit to what is guaranteed - that we have an error from a module.
This does arguably reduce the amount of context around the error.
As we do for `NonStandardSighashErrorType` add an error struct for
invalid sighash type, used by the `taproot` module instead of returning
a generic error enum with loads of unused variants.