Child requests

Many wallets don't allow you to spend from pending transactions. Besides, most wallets don't allow spending from pending transactions that signal for replace-by-fee (RBF) - the type of transactions managed by Bitpost. This is because these transactions are likely to be replaced and not mined (like any of its children). Fortunately, because Bitpost stores many replacements transactions, it is possible to create children of transactions managed by Bitpost safely. As we did in the previous article, it's possible to build these transactions so that only one can ever be mined.

Overview

In this article, we will see how you can access past and future change UTXOs to be used in a new request for fee adjustment, even if you didn't create any kind of authentification token (what we call wallettoken). Next, we will discuss how you can use these change UTXOs to select your inputs so that all your transactions are mutually exclusive. Finally, we will discuss the scalability of this approach and its tradeoffs.

Setting a wallettoken

The wallettoken is an optional parameter during the creation of a fee adjustment request, and that is randomly generated if it's missing (can be retrieved later). This token is used to fetch information about pending requests for fee adjustment so that child requests can be built.

Recommended default

Generating the token with this method provides easy interoperability between wallets that may integrate our API but the disadvantage of losing privacy towards Bitpost. Although we don't share your data with anyone, if you are very conscious about privacy, you might want to skip this and go to the next section.

This method consists of taking the HASH160 of the master public key, like shown:

import bit
from binascii import hexlify
import hashlib

key = bit.Key.from_bytes(b'REPLACE_WITH_YOUR_RANDOM_STRING')
master_pubkey = key.public_key  # in a HD(BIP32) wallet, use your master public key

pubkey_hash = hashlib.new('ripemd160', hashlib.sha256(master_pubkey).digest())  # aka HASH160
wallettoken = hexlify(pubkey_hash.digest()).decode()

import bit
from binascii import hexlify
import hashlib

key = bit.PrivateKeyTestnet.from_bytes(b'REPLACE_WITH_YOUR_RANDOM_STRING')
master_pubkey = key.public_key  # in a HD(BIP32) wallet, use your master public key

pubkey_hash = hashlib.new('ripemd160', hashlib.sha256(master_pubkey).digest())  # aka HASH160
wallettoken = hexlify(pubkey_hash.digest()).decode()

Generating a new token

You can generate a new token to identify your wallet and possibly change it later to a newer one. Remember that if you forget your most recent token under use, you might accidentally try to override a request and therefore get an error. To avoid this, we recommend only changing your wallet token if there are no pending RBF transactions to any of your wallet addresses.

The wallettoken is a unique identifier, temporary or not, of your wallet and therefore, should be at least as strong as a typical password. It should have between 20 and 64 characters, and there are some checks to make sure there is enough entropy. Only ASCII characters are allowed.

Retrieving a token

If you sent a fee adjustment request to Bitpost but forgot to set the wallettoken, you can obtain the random one created by proving ownership of a public key spent in the request. This will then allow you to retrieve the change UTXOs to build the child request. An example of how the wallettoken can be obtained is shown below:

from bitpost.interface_for_bit import BitpostInterfaceForBit
from bitcoin.signmessage import SignMessage, BitcoinMessage # python-bitcoinlib: https://github.com/petertodd/python-bitcoinlib
from bitcoin.wallet import CBitcoinSecret
from bitcoin.core import x
import bit
import time

bitkey = bit.Key.from_bytes(b'REPLACE_WITH_YOUR_RANDOM_STRING')
key = CBitcoinSecret.from_secret_bytes(x(bitkey.to_hex()))
pub_key_hex = str(key.pub.hex())

message = "bitpost" + str(round(time.time()/1000)) # we add a timestamp to make the proof valid for only ~ 1h
sig = SignMessage(key, BitcoinMessage(message))

bitpost_interface = BitpostInterfaceForBit()
wallettoken = bitpost_interface.get_wallettoken(pub_key_hex, sig)

from bitpost.interface_for_bit import BitpostInterfaceForBit
from bitcoin.signmessage import SignMessage, BitcoinMessage # python-bitcoinlib: https://github.com/petertodd/python-bitcoinlib
from bitcoin.wallet import CBitcoinSecret
from bitcoin.core import x
import bit
import time

bitkey = bit.Key.from_bytes(b'REPLACE_WITH_YOUR_RANDOM_STRING')
key = CBitcoinSecret.from_secret_bytes(x(bitkey.to_hex()))
pub_key_hex = str(key.pub.hex())

message = "bitpost" + str(round(time.time()/1000)) # we add a timestamp to make the proof valid for only ~ 1h
sig = SignMessage(key, BitcoinMessage(message))

bitpost_interface = BitpostInterfaceForBit(testnet=True)
wallettoken = bitpost_interface.get_wallettoken(pub_key_hex, sig)

WARNING

Some libraries like bit DON'T format the to-be-signed message like bitcoin core does. That's why, in this example, Peter Todd's python library instead. Bitcoin core formats the message before signing the following way: the magic string "Bitcoin Signed Message:\n" is prepended to the message, encoded in UTF-8 with a VarInt for both the magic and the message. This byte array is then hashed twice with SHA256, and the digest is what is actually signed by the private key.

Obtaining change UTXOs

Using your wallettoken, you can obtain information about any ongoing request assigned to it. Useful information includes the used UTXOs, which should be locked before coin selection (if not transaction is in the mempool) and available change UTXOs that can be used to make child transactions.

Parent transaction selection

The diagram below represents the transaction structure of two possible fee adjustment requests: a parent request, consisting of green, red and orange transactions; and a child request, consisting of black transactions. The transactions of the parent request spend from the same UTXO(s), and therefore only one parent transaction can ever be mined. Because we don't know which parent transaction will be mined, we need to sign children for every transaction that could be mined: previously broadcasted transactions (green) and transactions that could bump the fee of the parent request (orange).

Notice that, in this example, we didn't make child transactions of the transaction that pays 5 sat/B and 7 sat/B. The reason for this is that for the parent transaction to be replaced, it will need to pay for the fee of both itself and the child transaction and, therefore, minimum feerate bump will be approx. 2 sat/B instead of 1 sat/B. Also, in this example, it's unnecessary to make child transactions for the transactions that pay 2 sat/B and 3 sat/B. This is because, these transactions were skipped when the 4 sat/B was broadcasted.

Change output

Bitpost assumes by default that the transaction fee is discounted from the change, and therefore it can guess, in most scenarios, which output is the change. You can override this behavior by passing the parameter changepos when sending the request to Bitpost. Alternatively, if there are multiple change outputs in each transaction, you can use the raw transactions themselves to build you children.

Coin selection

When using Bitpost, the first transaction broadcast is not always done immediately after the request is made. If the delay parameter is set to higher than zero, and the target for confirmation is many blocks away, the first broadcast may not occur for hours or days. In this situation, your wallet is at a high risk of accidentally creating a double-spend with a scheduled transaction. To avoid this, we fetch the UTXOs reserved for future transactions from Bitpost, and lock (or exclude) these UTXOs before coin selection. All pending RBF transactions should also be locked before coin selection as they are unsafe to spend individually.

from bitpost.interface_for_bit import BitpostInterfaceForBit

# using the wallettoken previously obtained
bitpost_interface = BitpostInterfaceForBit(wallettoken=wallettoken)

unspents = key.get_unspents()
used_utxos = bitpost_interface.get_utxos_used_by_bitpost()
# for convenience we exclude all pending txs, instead of just RBF pending txs
unspents = [utxo for utxo in unspents if
            utxo not in used_utxos and utxo.confirmations > 0]

stripped_balance = 0
for unspent in unspents:
    stripped_balance += unspent.amount

from bitpost.interface_for_bit import BitpostInterfaceForBit

# using the wallettoken previously obtained
bitpost_interface = BitpostInterfaceForBit(wallettoken=wallettoken, testnet=True)

unspents = key.get_unspents()
used_utxos = bitpost_interface.get_utxos_used_by_bitpost()
# for convenience we exclude all pending txs, instead of just RBF pending txs
unspents = [utxo for utxo in unspents if
            utxo not in used_utxos and utxo.confirmations > 0]

stripped_balance = 0
for unspent in unspents:
    stripped_balance += unspent.amount

RBF coin selection

This step refers to the selection of pending (or to-be broadcasted) replaceable UTXOs that, together with coin selection of confirmed/non-RBF UTXOs, will fund the PSTs relayed to Bitpost. RBF coin selection can be done before or after coin selection. However, for the sake of this tutorial, we want to favor the creation of child transactions.

The code below uses some simple heuristics to select a group of UTXOs from possible groups retrieved from Bitpost using a wallettoken. Bitpost assumes that the transaction fee is discounted from the change output and uses this to infer which output is the change. Alternatively, you can specify the change output when you create a fee adjustment request with Bitpost, or you can retrieve the whole transaction instead. If there are no scheduled transactions to be broadcasted by Bitpost, Bitpost will reply with an empty array. No UTXOs will be pre-selected, which is the more straightforward scenario presented in the quick start tutorial.

feerates = bitpost_interface.get_feerates(USER_MAX_FEERATE, size=50, target=target)

def rbf_coin_select(utxos_answer, required_change, few_broadcasts=200, max_groupsize=4):
    for reqgroup in utxos_answer:
        if reqgroup['groupsize'] >= max_groupsize:
            continue
        for req in reqgroup['change']:
            if req['broadcasts'] >= few_broadcasts or req['minamount'] < required_change: continue
            return req['utxos'], req['minamount']

sats_required_from_rbf = sats_to_send - stripped_balance + max(feerates)*500 +566
all_rbf_change = bitpost_interface.get_change_UTXOs_from_bitpost()
rbf_selected, min_rbf_sats = rbf_coin_select(all_rbf_change, sats_required_from_rbf)

The code above selects change UTXOs based on three simple rules: the minimum amount of change is higher than the amount needed (often zero), the number of past broadcasts is smaller than 200, and the number of fee adjustment requests in that group is smaller than four. The last two rules make sure the bandwidth cost of the request doesn't reach very high values. Currently, Bitpost limits the amount of relayed PSTs in a single request to 10000.

For a deeper look at to-be broadcasted change UTXOs or transactions that can be used in RBF coin selection, visit the change structure article. For more information on how you can verify that these UTXOs are safe and valid to be spent, visit the verify change article.

Coin selection per se

After locking unsafe UTXOs, if there is still any bitcoin needed to fund the transactions, UTXOs can be selected as usual.

from bit.transaction import select_coins

selected_unspents, _ = select_coins(sats_to_send - min_rbf_sats, max(feerates), [34], min_change=566, unspents=unspents)

Sending request

If the target for confirmation of the child request is before the parent's, the parent request will be confirmed sooner to make up for this. When there are incompatible targets in a group of requests, the most urgent requests take priority.

raw_signed_txs = []
for pre_selection in rbf_selected:
    coin_selection = pre_selection + selected_unspents
    for i in range(0, len(feerates)):
        raw_signed_txs[i] = priv_key.create_transaction([(destination_address, sats_to_send, 'satoshi')], feerates[i], combine=True, replace_by_fee=True, unspents=coin_selection)

request = bitpost_interface.create_bitpost_request(raw_signed_txs, target, delay=0)
request.send_request()

raw_signed_txs = []
for pre_selection in rbf_selected:
    coin_selection = pre_selection + selected_unspents
    for i in range(0, len(feerates)):
        raw_signed_txs[i] = priv_key.create_transaction([(destination_address, sats_to_send, 'satoshi')], feerates[i], combine=True, replace_by_fee=True, unspents=coin_selection)

request = bitpost_interface.create_bitpost_request(raw_signed_txs, target, delay=0)
request.send_request()

WARNING

On testnet, transactions usually confirm very fast - especially during block storms. Therefore you need to create two requests in quick succession to be able to create a child request.

Drawbacks

This technique of creating child requests, can be used sequentially to create a handful of chained requests or with some other disposition - three parent requests consolidating in one child, two parents with one child who has, in turn, two childen, etc. However, as you can probably imagine, this technique is not very sustainable: an original request with 10 transactions can have a child request with 100 child transactions (10 transactions for each parent transaction), which can have a child with 1000 transactions, etc. This exponential growth means that, at some point, we have to reject incoming child requests. This limit (subject to change) is 10000 transactions per request.

Solution

We are currently developing new features that will help anyone that wants to do a high volume of simultaneous transactions. These will make our API much more suitable for custodial businesses that process many withdrawals.

For now, the best solution for many simultaneous transactions/requests is to batch them before you send them to Bitpost.