Any payment gets stuck on a direct channel

[holzeis]

I created an outbound channel with Kraken.

/ # lncli listchannels
{
    "channels":  [
        {
            "active":  true,
            "remote_pubkey":  "02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69",
            "channel_point":  "a0e123f16ebf8a7bb9975b4f3606a96e8c44261dbbbb1fbe6b76593495f7b896:1",
            "chan_id":  "906558332353249281",
            "capacity":  "1000000",
            "local_balance":  "996530",
            "remote_balance":  "0",
            "commit_fee":  "3140",
            "commit_weight":  "772",
            "fee_per_kw":  "2500",
            "unsettled_balance":  "0",
            "total_satoshis_sent":  "0",
            "total_satoshis_received":  "0",
            "num_updates":  "0",
            "pending_htlcs":  [],
            "csv_delay":  144,
            "private":  false,
            "initiator":  true,
            "chan_status_flags":  "ChanStatusDefault",
            "local_chan_reserve_sat":  "10000",
            "remote_chan_reserve_sat":  "10000",
            "static_remote_key":  false,
            "commitment_type":  "ANCHORS",
            "lifetime":  "6166",
            "uptime":  "6161",
            "close_address":  "",
            "push_amount_sat":  "0",
            "thaw_height":  0,
            "local_constraints":  {
                "csv_delay":  144,
                "chan_reserve_sat":  "10000",
                "dust_limit_sat":  "354",
                "max_pending_amt_msat":  "990000000",
                "min_htlc_msat":  "1000",
                "max_accepted_htlcs":  50
            },
            "remote_constraints":  {
                "csv_delay":  144,
                "chan_reserve_sat":  "10000",
                "dust_limit_sat":  "354",
                "max_pending_amt_msat":  "990000000",
                "min_htlc_msat":  "1",
                "max_accepted_htlcs":  483
            },
            "alias_scids":  [],
            "zero_conf":  false,
            "zero_conf_confirmed_scid":  "0",
            "peer_alias":  "Kraken 🐙⚡",
            "peer_scid_alias":  "0",
            "memo":  ""
        }
    ]
}

Trying to send any payment to Kraken got stuck so far. Yesterday I tried to pay an invoice of 500,290 sats, which is still stuck.

{
            "payment_hash":  "c5f8ba568402bb00558dd5b39a8683ff3d1e0b3e14da1809253aa944e32d9252",
            "value":  "500290",
            "creation_date":  "1704486738",
            "fee":  "0",
            "payment_preimage":  "0000000000000000000000000000000000000000000000000000000000000000",
            "value_sat":  "500290",
            "value_msat":  "500290000",
            "payment_request":  "lnbc5002900n1pjeseh3pp5chut545yq2asq4vd6kee4p5rlu73uze7zndpszf98255fcedjffqdqhfdexz6m9dcsygetsdaekjaqcqzysxqrrsssp5dljejqsfy2g46ad53aumyxhxrnmfzp6gx9fuwdwt8xl8jg2ds3fs9qyyssq4l7l70s3gsnnxftk0tc3c5wcpshmrfw7hgus9q3fnka0x6sy6hxrurrnu586synze4mxfzv28va0kwt68kvyn8590w8xr84s554zf9qpw2gdlr",
            "status":  "IN_FLIGHT",
            "fee_sat":  "0",
            "fee_msat":  "0",
            "creation_time_ns":  "1704486738110155002",
            "htlcs":  [
                {
                    "attempt_id":  "1",
                    "status":  "IN_FLIGHT",
                    "route":  {
                        "total_time_lock":  824668,
                        "total_fees":  "0",
                        "total_amt":  "500290",
                        "hops":  [
                            {
                                "chan_id":  "906558332353249281",
                                "chan_capacity":  "1000000",
                                "amt_to_forward":  "500290",
                                "fee":  "0",
                                "expiry":  824668,
                                "amt_to_forward_msat":  "500290000",
                                "fee_msat":  "0",
                                "pub_key":  "02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69",
                                "tlv_payload":  true,
                                "mpp_record":  {
                                    "payment_addr":  "6fe599020922915d75b48f79b21ae61cf69107483153c735cb39be79214d8453",
                                    "total_amt_msat":  "500290000"
                                },
                                "amp_record":  null,
                                "custom_records":  {},
                                "metadata":  ""
                            }
                        ],
                        "total_fees_msat":  "0",
                        "total_amt_msat":  "500290000"
                    },
                    "attempt_time_ns":  "1704486777742530852",
                    "resolve_time_ns":  "0",
                    "failure":  null,
                    "preimage":  ""
                }
            ],
            "payment_index":  "1",
            "failure_reason":  "FAILURE_REASON_NONE"
        },

I presume the payment will timeout eventually, however I would have expected to be able to successfully pay the invoice to my channel counter party as the my balance is sufficient, and the channel constraints are also adhered to.

Any idea what could be wrong here?

[holzeis]

Now, the stuck payments expired and it seems my channel is now in waiting to close.

    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69",
                "channel_point":  "a0e123f16ebf8a7bb9975b4f3606a96e8c44261dbbbb1fbe6b76593495f7b896:1",
                "capacity":  "1000000",
                "local_balance":  "996530",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "10000",
                "remote_chan_reserve_sat":  "10000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "996530",
            "commitments":  {
                "local_txid":  "d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d",
                "remote_txid":  "2fd2fd1f3075aa75f4fe9de568e467bbcbb27dcf42b261b8816cc0ca50d96f66",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "9495",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d"
        }
    ]

The closing_txid did not get broadcasted, and apparently fails to retry

2024-01-07 07:41:09.835 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-07 07:42:09.836 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-07 07:43:09.836 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-07 07:44:09.840 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-07 07:45:09.836 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...

Is this channel now stuck on closing? Any support on how to get out of this would be appreciated.

[holzeis]

hey @bitromortac, thanks for your response. No I do not get any broadcasting errors. Only debug messages that lnd tries to re-broadcast the closing transaction.

2024-01-08 09:41:29.916 [DBG] BTCN: Re-broadcasting 1 transactions
2024-01-08 09:41:29.923 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-08 09:42:29.916 [DBG] BTCN: Re-broadcasting 1 transactions
2024-01-08 09:42:29.923 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-08 09:43:29.916 [DBG] BTCN: Re-broadcasting 1 transactions
2024-01-08 09:43:29.923 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...
2024-01-08 09:44:29.916 [DBG] BTCN: Re-broadcasting 1 transactions
2024-01-08 09:44:29.922 [DBG] BTCN: Re-broadcast of txid=d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d, still pending...

The fee is indeed very low. I presume that's why the transaction gets immediately purged from mempool?

Should I use bumpclosefee to bump that fee? Also it states in the docs that this will create a CPFP transaction, wouldn't a RBF transaction be more efficient?

[bitromortac]

It would be great to know what bitcoin-cli getmempoolinfo (assuming you're running bitcoind?) and bitcoin-cli getmempoolentry (and getmempooldescendants) d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d say. Yes, bumpclosefee can be a part of the solution, but you would first need to propagate the force close, otherwise the CPFP transaction will not be relayed. Right, it creates a CPFP, RBF is not possible because you and your peer would need to re-sign the force close, which is not part of the protocol.

[holzeis]

# bitcoin-cli getmempoolinfo
{
  "loaded": true,
  "size": 34830,
  "bytes": 7568886,
  "usage": 49999440,
  "total_fee": 13.50662017,
  "maxmempool": 50000000,
  "mempoolminfee": 0.00043000,
  "minrelaytxfee": 0.00001000,
  "incrementalrelayfee": 0.00001000,
  "unbroadcastcount": 0,
  "fullrbf": false
}

Since the transaction doesn't get broadcasted, mempool doesn't know about the transaction d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d yet.

# bitcoin-cli getmempoolentry d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d
error code: -5
error message:
Transaction not in mempool

# bitcoin-cli getmempooldescendants d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d
error code: -5
error message:
Transaction not in mempool

[bitromortac]

I wonder if you see any broadcast errors, because of "mempoolminfee": 0.00043000 (43 sat/byte). You could try to set maxmempool=1500 in bitcoin.conf and then restart the node. This may already be enough to propagate the transaction or to have your transaction in your mempool. Once you see the transaction on some explorer, you may try to bump it.

[holzeis]

Setting the maxmempool=1500 did the trick and the transaction got broadcasted. https://mempool.space/tx/d24562a661ebbf840c7fa0eae9a8a63a40c91296b23ceb827e89151087e2458d

The fees are indeed very low, I will see to bump it using bumpclosefee.

Thanks for your support @bitromortac, any idea why the payment on a direct channel got stuck (force closing my channel after expiry) in the first place?

[bitromortac]

Unfortunately it's difficult to say what went wrong there and why the HTLC wasn't resolved. It could have been a communication problem or there's something wrong within the peer. Perhaps you could look for errors regarding the payment and htlc in your logs.
When bumping the fee with bumpclosefee, keep in mind that the fee rate you enter will effectively be lower for the combined transactions, so you'll need to overshoot quite a bit (this call will be improved in the future to be able to set the package fee rate). Also, it would be best to go back to the default mempool size as then you'll have a better mempoolminfee (it's probably very low for you currently).

[bitromortac]

Sorry, I have confused bumpclosefee with bumpfee, so I think it's not necessary to overshoot the fee.