Create opinionated tool for multi-stage and multi-machine Icepick workflows

ryan commented

2024-12-12 19:51:57 +00:00

Owner

Currently, the Solana and SPL Token MVP will be done with a bash script, but the flow should be generic enough that some tool can be created to parse opinionated configuration files and create a workflow for reading inputs and managing mounting SD cards in multi-machine workflows.

For example, in SPL Token:

icepick workflow sol-transfer-token

would ask for the inputs of from-address, to-address, token, and amount. It would then have steps to get a field from icepick sol get-token-address that exports the token-address and token-decimals (TODO: implement token-decimals, it's currently hardcoded for IPDBG). It then generates the transaction, and waits for a blob from the SD card, from icepick workflow sol-get-blockhash-and-broadcast. The blockhash is then read, along with the other arguments, and passed into icepick sol transfer-token, which is then passed into icepick sign, generating a blob that can be returned to the online machine which is waiting for the transaction to broadcast.

The configuration file would likely look something like:

[[module]]
name = "sol"
derivation_prefix = "m/44'/501'/0'"
algorithm = "Ed25519"

[[module.workflow]]
# The name of the workflow, which can be called by `icepick workflow transfer-token`.
name = "transfer-token"

# These values are used as inputs for other workflows, acquired from the CLI.
# These values can only be strings, but other values can be any value that can
# be serialized by serde_json::Value.
# These values can also be loaded using "internal-load-file", using some form
# of later-defined signature validation.
inputs = ["from-address", "to-address", "token-name", "token-amount"]

[[module.workflow.step]]
# Load the Blockhash from the SD card
type = "internal-load-file"

# Pre-defined values to be passed to the module
values = {
	"filename": "blockhash.json",
}

outputs = {
	# TODO: blockhash is the `.blob` value, not a field of a map.
	"blockhash": "blockhash",
}

[[module.workflow.step]]
# Get the token address and token decimals for the given token
type = "sol-token-info"
inputs = {
	# The key is the key that is passed to the program in the
	# `values` field. The value is the item in storage. In this case,
	# they are the same, because we read a `token-name` from our input,
	# store it in our storage as `token-name`, and `sol-token-info` will
	# expect a `token-name`.
	"token-name": "token-name",
}
outputs = {
	# Same as above, because the two fields are unused in our storage, we
	# can 
	"token-address": "token-address",
	"token-decimals": "token-decimals",
}

[[module.workflow.step]]
# Generate an unsigned Transaction
type = "sol-transfer-token"
inputs = {
	"amount": "token-amount",
	"token-address": "token-address",
	"to-address": "to-address",
	"from-address": "from-address",
}

# Store the unsigned transaction
outputs = {
	"transaction": "unsigned-transaction",
}

[[module.workflow.step]]
# Sign the transaction
type = "sol-sign"

inputs = {
	"transaction": "unsigned-transaction",
	"blockhash": "blockhash",
}

outputs = {
	"transaction": "signed-transaction",
}

[[module.workflow.step]]
# Write the transaction to a file
type = "internal-save-file"

values = {
	"filename": "transaction.json",
}

inputs = {
	"transaction": "signed-transaction",
}

Currently, the Solana and SPL Token MVP will be done with a bash script, but the flow should be generic enough that some tool can be created to parse opinionated configuration files and create a workflow for reading inputs and managing mounting SD cards in multi-machine workflows. For example, in SPL Token: ```sh icepick workflow sol-transfer-token ``` would ask for the inputs of `from-address`, `to-address`, `token`, and `amount`. It would then have steps to get a field from `icepick sol get-token-address` that exports the `token-address` and `token-decimals` (TODO: implement `token-decimals`, it's currently hardcoded for IPDBG). It then generates the transaction, and waits for a blob from the SD card, from `icepick workflow sol-get-blockhash-and-broadcast`. The blockhash is then read, along with the other arguments, and passed into `icepick sol transfer-token`, which is then passed into `icepick sign`, generating a blob that can be returned to the online machine which is waiting for the transaction to broadcast. The configuration file would likely look something like: ```toml [[module]] name = "sol" derivation_prefix = "m/44'/501'/0'" algorithm = "Ed25519" [[module.workflow]] # The name of the workflow, which can be called by `icepick workflow transfer-token`. name = "transfer-token" # These values are used as inputs for other workflows, acquired from the CLI. # These values can only be strings, but other values can be any value that can # be serialized by serde_json::Value. # These values can also be loaded using "internal-load-file", using some form # of later-defined signature validation. inputs = ["from-address", "to-address", "token-name", "token-amount"] [[module.workflow.step]] # Load the Blockhash from the SD card type = "internal-load-file" # Pre-defined values to be passed to the module values = { "filename": "blockhash.json", } outputs = { # TODO: blockhash is the `.blob` value, not a field of a map. "blockhash": "blockhash", } [[module.workflow.step]] # Get the token address and token decimals for the given token type = "sol-token-info" inputs = { # The key is the key that is passed to the program in the # `values` field. The value is the item in storage. In this case, # they are the same, because we read a `token-name` from our input, # store it in our storage as `token-name`, and `sol-token-info` will # expect a `token-name`. "token-name": "token-name", } outputs = { # Same as above, because the two fields are unused in our storage, we # can "token-address": "token-address", "token-decimals": "token-decimals", } [[module.workflow.step]] # Generate an unsigned Transaction type = "sol-transfer-token" inputs = { "amount": "token-amount", "token-address": "token-address", "to-address": "to-address", "from-address": "from-address", } # Store the unsigned transaction outputs = { "transaction": "unsigned-transaction", } [[module.workflow.step]] # Sign the transaction type = "sol-sign" inputs = { "transaction": "unsigned-transaction", "blockhash": "blockhash", } outputs = { "transaction": "signed-transaction", } [[module.workflow.step]] # Write the transaction to a file type = "internal-save-file" values = { "filename": "transaction.json", } inputs = { "transaction": "signed-transaction", } ```

ryan added this to the Custody Framework project 2024-12-12 20:25:09 +00:00

ryan added this to the Icepick v0.1.0 milestone 2024-12-12 20:27:50 +00:00

ryan modified the milestone from Icepick v0.1.0 to Icepick v0.2.0

2024-12-12 20:27:52 +00:00

ryan referenced this issue

2024-12-12 20:30:02 +00:00

Tracker: Icepick configuration file #15

ryan commented

2024-12-19 04:11:55 +00:00

Author

Owner

Thinking on this, I think it is reasonable to implement a one-machine system with no internal module so it can run entirely on an online system, then once that's done, write the internal module and make it work across multiple systems.

Thinking on this, I think it is reasonable to implement a one-machine system with no `internal` module so it can run entirely on an online system, then once that's done, write the `internal` module and make it work across multiple systems.

ryan commented

2024-12-19 21:04:23 +00:00

Author

Owner

a18282d107 implements improvements that will help transfer data between modules

a18282d1075328119ea3ad2b328ee9dfca518cc3 implements improvements that will help transfer data between modules

ryan commented

2024-12-24 00:37:08 +00:00

Author

Owner

End to end tests now use icepick workflow with save-to-disk functionality. They still pass in the values manually, so OpenPGP signature validation is not yet implemented.

End to end tests now use `icepick workflow` with save-to-disk functionality. They still pass in the values manually, so OpenPGP signature validation is not yet implemented.

Create opinionated tool for multi-stage and multi-machine Icepick workflows #22